2024-03-18T00:28:45.395 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T00:28:45.395 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} --owner scheduled_pdonnell@teuthology --name pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --archive /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544 -- /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/orig.config.yaml 2024-03-18T00:28:45.451 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T00:28:45.562 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} email: pdonnell@redhat.com first_in_suite: false job_id: '7608544' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: c382027d5c66ef4e0aa1131ea3e8080c658d3184 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHcrMVErWx8rMuitlnLnLqf/0THqc6mmQOVGlNFjE7UMOVSDBdBwKxCn/qjSqBgIJYUAcuF0mbdep72qRPRxmzI= smithi106.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO8hZd4SIuuzDj1ZLb0BOj23GeXKiYy6xYrzbQwJi23Zt956HjsWRSC4ue6hMgic2Cif1ehIpryh3kqf8+tsaUk= smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO/cQoSbhCAXJ2Ioh5Z+nED/+TevQTyQp4fE+LLukTLLljCaNNXqFPFhudd8be5/9kFWDRzy8WOUvnEBFhb5hxM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-17_23:17:27 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T00:28:45.563 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa; will attempt to use it 2024-03-18T00:28:45.563 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_c382027d5c66ef4e0aa1131ea3e8080c658d3184/qa/tasks 2024-03-18T00:28:45.563 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T00:28:45.565 INFO:teuthology.task.internal:Checking packages... 2024-03-18T00:28:45.588 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-18T00:28:45.588 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T00:28:45.588 INFO:teuthology.packaging:ref: None 2024-03-18T00:28:45.588 INFO:teuthology.packaging:tag: None 2024-03-18T00:28:45.588 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-18T00:28:45.588 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:28:45.588 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-18T00:28:45.722 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-18T00:28:45.724 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T00:28:45.731 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T00:28:45.731 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T00:28:45.751 INFO:teuthology.task.internal:Saving configuration 2024-03-18T00:28:45.769 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T00:28:45.777 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T00:28:45.804 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:22:17.613197', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHcrMVErWx8rMuitlnLnLqf/0THqc6mmQOVGlNFjE7UMOVSDBdBwKxCn/qjSqBgIJYUAcuF0mbdep72qRPRxmzI='} 2024-03-18T00:28:45.827 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:22:17.615069', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO8hZd4SIuuzDj1ZLb0BOj23GeXKiYy6xYrzbQwJi23Zt956HjsWRSC4ue6hMgic2Cif1ehIpryh3kqf8+tsaUk='} 2024-03-18T00:28:45.848 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:22:17.611476', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO/cQoSbhCAXJ2Ioh5Z+nED/+TevQTyQp4fE+LLukTLLljCaNNXqFPFhudd8be5/9kFWDRzy8WOUvnEBFhb5hxM='} 2024-03-18T00:28:45.848 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T00:28:45.856 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-18T00:28:45.856 INFO:teuthology.task.internal:roles: ubuntu@smithi106.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-18T00:28:45.856 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-18T00:28:45.856 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T00:28:45.977 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff5c18e3160>, signals=[15]) 2024-03-18T00:28:45.977 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T00:28:45.984 INFO:teuthology.task.internal:Opening connections... 2024-03-18T00:28:45.984 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2024-03-18T00:28:45.985 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:28:46.092 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi106.front.sepia.ceph.com 2024-03-18T00:28:46.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:28:46.180 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-03-18T00:28:46.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:28:46.269 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T00:28:46.275 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2024-03-18T00:28:46.293 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2024-03-18T00:28:46.294 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-03-18T00:28:46.351 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2024-03-18T00:28:46.351 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2024-03-18T00:28:46.351 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T00:28:46.352 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T00:28:46.353 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-03-18T00:28:46.372 DEBUG:teuthology.orchestra.run.smithi106:> uname -m 2024-03-18T00:28:46.390 INFO:teuthology.orchestra.run.smithi106.stdout:x86_64 2024-03-18T00:28:46.390 DEBUG:teuthology.orchestra.run.smithi106:> cat /etc/os-release 2024-03-18T00:28:46.446 INFO:teuthology.orchestra.run.smithi106.stdout:NAME="CentOS Stream" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION="9" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:ID="centos" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:ID_LIKE="rhel fedora" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:VERSION_ID="9" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:PLATFORM_ID="platform:el9" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:ANSI_COLOR="0;31" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:LOGO="fedora-logo-icon" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:HOME_URL="https://centos.org/" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T00:28:46.447 INFO:teuthology.orchestra.run.smithi106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T00:28:46.448 INFO:teuthology.lock.ops:Updating smithi106.front.sepia.ceph.com on lock server 2024-03-18T00:28:46.472 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-03-18T00:28:46.489 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-03-18T00:28:46.489 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-03-18T00:28:46.547 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T00:28:46.548 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-03-18T00:28:46.548 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T00:28:46.548 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T00:28:46.548 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T00:28:46.548 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-03-18T00:28:46.571 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T00:28:46.583 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T00:28:46.608 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T00:28:46.609 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2024-03-18T00:28:46.611 DEBUG:teuthology.orchestra.run.smithi106:> test '!' -e /home/ubuntu/cephtest 2024-03-18T00:28:46.613 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-03-18T00:28:46.630 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T00:28:46.641 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T00:28:46.641 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2024-03-18T00:28:46.668 DEBUG:teuthology.orchestra.run.smithi106:> test -z $(ls -A /var/lib/ceph) 2024-03-18T00:28:46.671 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-03-18T00:28:46.776 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T00:28:46.859 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T00:28:46.876 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-18T00:28:46.876 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-18T00:28:46.877 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-18T00:28:46.877 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T00:28:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:28:46.894 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-03-18T00:28:46.953 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-361.el9.x86_64 2024-03-18T00:28:46.953 INFO:teuthology.task.kernel:Running kernel on smithi049: 5.14.0-361.el9.x86_64 2024-03-18T00:28:46.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2024-03-18T00:28:49.005 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-18T00:28:50.447 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-18T00:28:52.021 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 8.0 MB 00:01 2024-03-18T00:28:53.983 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-18T00:28:58.433 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 6.1 MB 00:00 2024-03-18T00:29:00.091 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-18T00:29:03.524 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 21 MB 00:03 2024-03-18T00:29:08.608 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-18T00:29:10.864 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-18T00:29:10.865 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-18T00:29:10.952 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-03-18T00:29:10.961 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-03-18T00:29:10.961 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-03-18T00:29:10.961 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-03-18T00:29:10.961 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-03-18T00:29:10.961 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-18T00:29:10.962 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-03-18T00:29:10.962 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-18T00:29:10.962 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-18T00:29:10.962 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-18T00:29:10.962 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:29:10.962 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-03-18T00:29:10.962 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-03-18T00:29:10.962 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2024-03-18T00:29:10.962 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:29:10.963 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 100 M 2024-03-18T00:29:10.963 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 123 M 2024-03-18T00:29:10.963 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-03-18T00:29:12.709 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.2 MB/s | 6.0 MB 00:01 2024-03-18T00:29:14.684 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 6.1 MB/s | 21 MB 00:03 2024-03-18T00:29:15.784 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 8.7 MB/s | 39 MB 00:04 2024-03-18T00:29:16.409 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 9.0 MB/s | 33 MB 00:03 2024-03-18T00:29:16.409 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-03-18T00:29:16.409 INFO:teuthology.orchestra.run.smithi049.stdout:Total 18 MB/s | 100 MB 00:05 2024-03-18T00:29:17.330 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-03-18T00:29:17.763 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-03-18T00:29:17.763 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-03-18T00:29:19.193 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-03-18T00:29:19.194 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-03-18T00:29:22.057 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-03-18T00:29:22.734 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-18T00:29:22.885 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T00:29:25.832 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T00:29:26.013 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T00:29:31.119 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T00:29:31.250 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T00:29:36.377 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T00:30:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T00:30:07.074 INFO:teuthology.orchestra.run.smithi049.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T00:30:07.074 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:30:07.161 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-18T00:30:07.602 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-18T00:30:07.603 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-18T00:30:07.603 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-18T00:30:07.603 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-18T00:30:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-18T00:30:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:30:08.086 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-03-18T00:30:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-18T00:30:08.086 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-18T00:30:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-18T00:30:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-18T00:30:08.087 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:30:08.087 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-03-18T00:30:08.591 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T00:30:08.740 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T00:30:08.740 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T00:30:08.740 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-18T00:30:08.741 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-18T00:30:08.741 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-18T00:30:08.741 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-18T00:30:08.741 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2024-03-18T00:30:08.741 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-03-18T00:30:08.867 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-18T00:30:08.867 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-18T00:30:08.868 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2024-03-18T00:30:08.912 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.06 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 68.el9 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Group : Unspecified 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 8264254 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T00:30:08.913 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : builder@centos.org 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:30:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T00:30:08.915 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T00:30:08.915 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 5.14.0-427.el9.x86_64 2024-03-18T00:30:08.915 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T00:30:09.482 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2024-03-18T00:30:10.844 INFO:teuthology.orchestra.run.smithi049.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T00:30:10.880 INFO:teuthology.orchestra.run.smithi049.stderr:done 2024-03-18T00:30:10.882 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-03-18T00:30:10.897 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.8HGbrtUVff 2024-03-18T00:30:10.898 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8HGbrtUVff 2024-03-18T00:30:10.989 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.8HGbrtUVff 2024-03-18T00:30:11.092 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.8HGbrtUVff is 6KB 2024-03-18T00:30:11.140 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.8HGbrtUVff 2024-03-18T00:30:11.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T00:30:11.233 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T00:30:11.233 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T00:30:11.233 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T00:30:11.233 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-18T00:30:11.234 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-18T00:30:11.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2024-03-18T00:30:41.390 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-18T00:30:41.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-03-18T00:30:41.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:31:41.396 DEBUG:teuthology.orchestra.remote:timed out 2024-03-18T00:31:50.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-03-18T00:31:50.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:31:53.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-03-18T00:32:05.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-03-18T00:32:05.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:32:37.229 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-03-18T00:32:37.707 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-03-18T00:32:37.707 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-18T00:32:37.708 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-03-18T00:32:37.722 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-427.el9.x86_64 2024-03-18T00:32:37.722 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-18T00:32:37.722 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T00:32:37.723 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T00:32:38.724 DEBUG:teuthology.parallel:result is None 2024-03-18T00:32:38.724 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T00:32:38.733 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T00:32:38.733 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T00:32:38.735 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T00:32:38.738 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T00:32:38.756 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T00:32:38.763 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T00:32:38.788 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T00:32:38.788 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T00:32:38.795 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T00:32:38.798 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T00:32:38.933 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T00:32:38.941 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T00:32:38.941 DEBUG:teuthology.orchestra.run.smithi106:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T00:32:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:32:38.958 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T00:32:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:32:38.974 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T00:32:38.976 DEBUG:teuthology.orchestra.run.smithi106:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T00:32:39.000 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T00:32:39.045 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T00:32:39.052 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T00:32:39.064 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T00:32:39.074 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T00:32:39.083 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T00:32:39.116 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T00:32:39.118 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T00:32:39.126 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T00:32:39.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T00:32:39.128 DEBUG:teuthology.orchestra.run.smithi106:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T00:32:39.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T00:32:39.160 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T00:32:39.173 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T00:32:39.173 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T00:32:39.200 DEBUG:teuthology.orchestra.run.smithi106:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T00:32:39.202 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T00:32:39.218 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T00:32:39.296 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T00:32:39.351 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T00:32:39.435 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T00:32:39.480 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:32:39.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T00:32:39.551 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T00:32:39.596 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T00:32:39.685 DEBUG:teuthology.orchestra.run.smithi106:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T00:32:39.728 DEBUG:teuthology.orchestra.run.smithi106:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T00:32:39.773 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:32:39.773 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T00:32:39.841 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T00:32:39.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T00:32:39.950 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T00:32:39.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T00:32:40.038 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:32:40.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T00:32:40.107 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2024-03-18T00:32:40.109 DEBUG:teuthology.orchestra.run.smithi106:> sudo service rsyslog restart 2024-03-18T00:32:40.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-03-18T00:32:40.152 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T00:32:40.161 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T00:32:40.183 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T00:32:40.704 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T00:32:40.714 INFO:teuthology.task.internal:Starting timer... 2024-03-18T00:32:40.714 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T00:32:40.763 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T00:32:40.792 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-18T00:32:40.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2024-03-18T00:32:40.876 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2024-03-18T00:32:40.879 DEBUG:teuthology.orchestra.run.smithi106:> sudo service auditd rotate 2024-03-18T00:32:40.932 INFO:teuthology.orchestra.run.smithi106.stdout:Rotating logs: 2024-03-18T00:32:40.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-03-18T00:32:40.996 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-03-18T00:32:40.998 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T00:32:40.998 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2024-03-18T00:32:41.035 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2024-03-18T00:32:41.036 DEBUG:teuthology.orchestra.run.smithi106:> /usr/sbin/getenforce 2024-03-18T00:32:41.065 INFO:teuthology.orchestra.run.smithi106.stdout:Permissive 2024-03-18T00:32:41.065 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-03-18T00:32:41.082 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-03-18T00:32:41.082 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi106.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi107.front.sepia.ceph.com': 'permissive'} 2024-03-18T00:32:41.083 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T00:32:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:32:41.115 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T00:32:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:32:41.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T00:32:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:32:41.170 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T00:32:41.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2024-03-18T00:32:41.205 DEBUG:teuthology.orchestra.run.smithi106:> sudo /usr/sbin/setenforce permissive 2024-03-18T00:32:41.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-03-18T00:32:41.276 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T00:32:41.286 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T00:32:41.303 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T00:32:41.304 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com,smithi106.front.sepia.ceph.com,smithi107.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T00:37:18.510 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com'), Remote(name='ubuntu@smithi106.front.sepia.ceph.com'), Remote(name='ubuntu@smithi107.front.sepia.ceph.com')] 2024-03-18T00:37:18.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-03-18T00:37:18.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:37:18.620 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-03-18T00:37:18.703 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-03-18T00:37:18.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-18T00:37:18.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:37:18.794 DEBUG:teuthology.orchestra.run.smithi106:> true 2024-03-18T00:37:18.862 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi106.front.sepia.ceph.com' 2024-03-18T00:37:18.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-18T00:37:18.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T00:37:18.961 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-03-18T00:37:19.035 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-18T00:37:19.035 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T00:37:19.046 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T00:37:19.046 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T00:37:19.047 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T00:37:19.050 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T00:37:19.051 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T00:37:19.053 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T00:37:19.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T00:37:19.083 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T00:37:19.085 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T00:37:19.101 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T00:37:19.106 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T00:37:19.109 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T00:37:19.126 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T00:37:19.132 INFO:teuthology.orchestra.run.smithi106.stderr:sudo: ntpd: command not found 2024-03-18T00:37:19.142 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2024-03-18T00:37:19.145 INFO:teuthology.orchestra.run.smithi106.stdout:506 Cannot talk to daemon 2024-03-18T00:37:19.157 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-03-18T00:37:19.159 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2024-03-18T00:37:19.162 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T00:37:19.171 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-03-18T00:37:19.178 INFO:teuthology.orchestra.run.smithi106.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T00:37:19.179 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T00:37:19.187 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T00:37:19.199 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T00:37:19.203 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T00:37:19.222 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-03-18T00:37:19.225 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T00:37:19.225 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-03-18T00:37:19.226 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.226 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.226 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.226 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.247 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-18T00:37:19.251 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T00:37:19.251 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-18T00:37:19.251 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.251 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.251 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.251 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.257 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-03-18T00:37:19.261 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T00:37:19.262 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-03-18T00:37:19.262 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.262 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.262 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.262 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T00:37:19.262 INFO:teuthology.run_tasks:Running task install... 2024-03-18T00:37:19.273 DEBUG:teuthology.task.install:project ceph 2024-03-18T00:37:19.274 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-18T00:37:19.274 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T00:37:19.274 INFO:teuthology.task.install:Using flavor: default 2024-03-18T00:37:19.286 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T00:37:19.286 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T00:37:19.287 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T00:37:19.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:37:19.290 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T00:37:19.291 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:37:19.292 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T00:37:19.293 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:37:19.449 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T00:37:19.450 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T00:37:19.455 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T00:37:19.455 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T00:37:19.517 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T00:37:19.517 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T00:37:19.624 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T00:37:19.624 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:37:19.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T00:37:19.641 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T00:37:19.641 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:37:19.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T00:37:19.643 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T00:37:19.643 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:37:19.643 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T00:37:19.653 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T00:37:19.653 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T00:37:19.672 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T00:37:19.672 DEBUG:teuthology.orchestra.run.smithi106:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T00:37:19.675 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T00:37:19.675 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T00:37:19.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T00:37:19.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T00:37:19.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T00:37:19.805 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T00:37:19.819 DEBUG:teuthology.orchestra.run.smithi106:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T00:37:19.835 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T00:37:19.864 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2024-03-18T00:37:19.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2024-03-18T00:37:19.872 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-03-18T00:37:19.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-03-18T00:37:19.884 INFO:teuthology.orchestra.run.smithi106.stdout:check_obsoletes = 1 2024-03-18T00:37:19.886 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum clean all 2024-03-18T00:37:20.108 INFO:teuthology.orchestra.run.smithi107.stdout:74 files removed 2024-03-18T00:37:20.116 INFO:teuthology.orchestra.run.smithi049.stdout:74 files removed 2024-03-18T00:37:20.118 INFO:teuthology.orchestra.run.smithi106.stdout:74 files removed 2024-03-18T00:37:20.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T00:37:20.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T00:37:20.145 DEBUG:teuthology.orchestra.run.smithi106:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T00:37:20.776 INFO:teuthology.orchestra.run.smithi106.stdout:ceph packages for x86_64 243 kB/s | 83 kB 00:00 2024-03-18T00:37:20.786 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 234 kB/s | 83 kB 00:00 2024-03-18T00:37:20.945 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 179 kB/s | 83 kB 00:00 2024-03-18T00:37:21.151 INFO:teuthology.orchestra.run.smithi106.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-18T00:37:21.203 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-18T00:37:21.387 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-18T00:37:21.502 INFO:teuthology.orchestra.run.smithi106.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-03-18T00:37:21.595 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-18T00:37:21.746 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-18T00:37:22.039 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-18T00:37:22.071 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-18T00:37:22.407 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-18T00:37:23.423 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-18T00:37:23.491 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-18T00:37:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-18T00:37:24.884 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-18T00:37:25.320 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-03-18T00:37:27.330 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-18T00:37:27.382 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-18T00:37:27.846 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-18T00:37:29.314 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-18T00:37:31.525 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-18T00:37:32.901 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 98 kB/s | 16 kB 00:00 2024-03-18T00:37:33.331 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-18T00:37:33.888 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-18T00:37:34.815 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-18T00:37:35.833 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-18T00:37:37.364 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 6.1 MB 00:06 2024-03-18T00:37:38.831 INFO:teuthology.orchestra.run.smithi106.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-18T00:37:39.121 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-18T00:37:39.766 INFO:teuthology.orchestra.run.smithi106.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-18T00:37:40.890 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-18T00:37:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T00:37:41.759 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T00:37:41.772 INFO:teuthology.orchestra.run.smithi049.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T00:37:41.773 INFO:teuthology.orchestra.run.smithi049.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T00:37:41.774 INFO:teuthology.orchestra.run.smithi049.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T00:37:41.775 INFO:teuthology.orchestra.run.smithi049.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T00:37:41.777 INFO:teuthology.orchestra.run.smithi049.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T00:37:41.779 INFO:teuthology.orchestra.run.smithi049.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T00:37:41.780 INFO:teuthology.orchestra.run.smithi049.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T00:37:41.867 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-03-18T00:37:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================ 2024-03-18T00:37:41.889 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-03-18T00:37:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================ 2024-03-18T00:37:41.889 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-03-18T00:37:41.889 INFO:teuthology.orchestra.run.smithi049.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T00:37:41.889 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T00:37:41.890 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T00:37:41.890 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T00:37:41.890 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T00:37:41.890 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T00:37:41.890 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T00:37:41.891 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T00:37:41.891 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T00:37:41.891 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T00:37:41.891 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T00:37:41.891 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T00:37:41.891 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T00:37:41.891 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T00:37:41.891 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T00:37:41.892 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T00:37:41.892 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T00:37:41.892 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T00:37:41.892 INFO:teuthology.orchestra.run.smithi049.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T00:37:41.892 INFO:teuthology.orchestra.run.smithi049.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T00:37:41.892 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T00:37:41.892 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T00:37:41.892 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T00:37:41.893 INFO:teuthology.orchestra.run.smithi049.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T00:37:41.893 INFO:teuthology.orchestra.run.smithi049.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T00:37:41.893 INFO:teuthology.orchestra.run.smithi049.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T00:37:41.893 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T00:37:41.893 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T00:37:41.893 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T00:37:41.894 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T00:37:41.894 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T00:37:41.894 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T00:37:41.894 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T00:37:41.894 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T00:37:41.894 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T00:37:41.895 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T00:37:41.895 INFO:teuthology.orchestra.run.smithi049.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T00:37:41.895 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2024-03-18T00:37:41.895 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T00:37:41.895 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T00:37:41.895 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T00:37:41.895 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T00:37:41.896 INFO:teuthology.orchestra.run.smithi049.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T00:37:41.896 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T00:37:41.896 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T00:37:41.896 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T00:37:41.896 INFO:teuthology.orchestra.run.smithi049.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T00:37:41.896 INFO:teuthology.orchestra.run.smithi049.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T00:37:41.896 INFO:teuthology.orchestra.run.smithi049.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T00:37:41.897 INFO:teuthology.orchestra.run.smithi049.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T00:37:41.897 INFO:teuthology.orchestra.run.smithi049.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T00:37:41.897 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T00:37:41.897 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T00:37:41.898 INFO:teuthology.orchestra.run.smithi049.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T00:37:41.898 INFO:teuthology.orchestra.run.smithi049.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T00:37:41.898 INFO:teuthology.orchestra.run.smithi049.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T00:37:41.898 INFO:teuthology.orchestra.run.smithi049.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T00:37:41.898 INFO:teuthology.orchestra.run.smithi049.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T00:37:41.898 INFO:teuthology.orchestra.run.smithi049.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T00:37:41.898 INFO:teuthology.orchestra.run.smithi049.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T00:37:41.898 INFO:teuthology.orchestra.run.smithi049.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T00:37:41.899 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T00:37:41.900 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T00:37:41.900 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T00:37:41.900 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T00:37:41.900 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T00:37:41.900 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T00:37:41.900 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T00:37:41.900 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T00:37:41.900 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T00:37:41.901 INFO:teuthology.orchestra.run.smithi049.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T00:37:41.901 INFO:teuthology.orchestra.run.smithi049.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T00:37:41.901 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T00:37:41.901 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T00:37:41.901 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T00:37:41.901 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T00:37:41.901 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T00:37:41.901 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T00:37:41.901 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T00:37:41.902 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T00:37:41.902 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T00:37:41.902 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T00:37:41.902 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T00:37:41.902 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T00:37:41.902 INFO:teuthology.orchestra.run.smithi049.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T00:37:41.902 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T00:37:41.903 INFO:teuthology.orchestra.run.smithi049.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T00:37:41.903 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T00:37:41.903 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T00:37:41.903 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T00:37:41.903 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T00:37:41.903 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T00:37:41.903 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T00:37:41.904 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T00:37:41.904 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T00:37:41.904 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T00:37:41.904 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T00:37:41.904 INFO:teuthology.orchestra.run.smithi049.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T00:37:41.904 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T00:37:41.905 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T00:37:41.905 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T00:37:41.905 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T00:37:41.905 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T00:37:41.905 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T00:37:41.905 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T00:37:41.906 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T00:37:41.906 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T00:37:41.906 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T00:37:41.906 INFO:teuthology.orchestra.run.smithi049.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T00:37:41.906 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T00:37:41.906 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T00:37:41.906 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T00:37:41.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T00:37:41.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T00:37:41.907 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T00:37:41.908 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T00:37:41.908 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T00:37:41.908 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T00:37:41.908 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T00:37:41.908 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T00:37:41.908 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T00:37:41.908 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T00:37:41.909 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T00:37:41.909 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T00:37:41.910 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T00:37:41.910 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T00:37:41.910 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T00:37:41.910 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T00:37:41.911 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T00:37:41.911 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T00:37:41.911 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T00:37:41.911 INFO:teuthology.orchestra.run.smithi049.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T00:37:41.911 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T00:37:41.911 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T00:37:41.912 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T00:37:41.912 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T00:37:41.912 INFO:teuthology.orchestra.run.smithi049.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T00:37:41.912 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T00:37:41.912 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T00:37:41.912 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T00:37:41.912 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T00:37:41.913 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T00:37:41.913 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T00:37:41.913 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T00:37:41.913 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T00:37:41.913 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T00:37:41.913 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T00:37:41.913 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T00:37:41.914 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout:Installing weak dependencies: 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout:============================================================================================ 2024-03-18T00:37:41.915 INFO:teuthology.orchestra.run.smithi049.stdout:Install 150 Packages 2024-03-18T00:37:41.916 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 24 Packages 2024-03-18T00:37:41.916 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:37:41.916 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 224 M 2024-03-18T00:37:41.916 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-03-18T00:37:42.355 INFO:teuthology.orchestra.run.smithi049.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-18T00:37:42.539 INFO:teuthology.orchestra.run.smithi049.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 4.8 MB/s | 890 kB 00:00 2024-03-18T00:37:42.598 INFO:teuthology.orchestra.run.smithi049.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 2.4 MB/s | 145 kB 00:00 2024-03-18T00:37:42.823 INFO:teuthology.orchestra.run.smithi049.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 8.4 MB/s | 5.4 MB 00:00 2024-03-18T00:37:42.890 INFO:teuthology.orchestra.run.smithi049.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 8.1 MB/s | 2.4 MB 00:00 2024-03-18T00:37:42.998 INFO:teuthology.orchestra.run.smithi049.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 8.5 MB/s | 1.5 MB 00:00 2024-03-18T00:37:43.337 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T00:37:43.338 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T00:37:43.352 INFO:teuthology.orchestra.run.smithi107.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T00:37:43.353 INFO:teuthology.orchestra.run.smithi107.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T00:37:43.354 INFO:teuthology.orchestra.run.smithi107.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T00:37:43.355 INFO:teuthology.orchestra.run.smithi107.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T00:37:43.357 INFO:teuthology.orchestra.run.smithi107.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T00:37:43.360 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T00:37:43.361 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T00:37:43.390 INFO:teuthology.orchestra.run.smithi049.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 9.4 MB/s | 4.7 MB 00:00 2024-03-18T00:37:43.442 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-18T00:37:43.464 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-18T00:37:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-03-18T00:37:43.464 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-18T00:37:43.464 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-18T00:37:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T00:37:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T00:37:43.464 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T00:37:43.465 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T00:37:43.466 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T00:37:43.467 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T00:37:43.468 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T00:37:43.469 INFO:teuthology.orchestra.run.smithi107.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T00:37:43.469 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T00:37:43.469 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T00:37:43.469 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T00:37:43.469 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T00:37:43.469 INFO:teuthology.orchestra.run.smithi107.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T00:37:43.469 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T00:37:43.469 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T00:37:43.470 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T00:37:43.471 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T00:37:43.472 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T00:37:43.472 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T00:37:43.472 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T00:37:43.473 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T00:37:43.474 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T00:37:43.475 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T00:37:43.476 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T00:37:43.477 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T00:37:43.478 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================ 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout:Install 150 Packages 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 24 Packages 2024-03-18T00:37:43.479 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:37:43.480 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 224 M 2024-03-18T00:37:43.480 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-18T00:37:44.382 INFO:teuthology.orchestra.run.smithi049.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 9.9 MB/s | 9.8 MB 00:00 2024-03-18T00:37:44.566 INFO:teuthology.orchestra.run.smithi049.stdout:(9/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 11 MB/s | 17 MB 00:01 2024-03-18T00:37:44.583 INFO:teuthology.orchestra.run.smithi049.stdout:(10/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el 126 kB/s | 25 kB 00:00 2024-03-18T00:37:44.641 INFO:teuthology.orchestra.run.smithi049.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 3.7 MB/s | 221 kB 00:00 2024-03-18T00:37:44.642 INFO:teuthology.orchestra.run.smithi107.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-03-18T00:37:44.683 INFO:teuthology.orchestra.run.smithi106.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-18T00:37:44.692 INFO:teuthology.orchestra.run.smithi049.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 540 kB/s | 27 kB 00:00 2024-03-18T00:37:44.742 INFO:teuthology.orchestra.run.smithi049.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 637 kB/s | 32 kB 00:00 2024-03-18T00:37:44.867 INFO:teuthology.orchestra.run.smithi049.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 5.8 MB/s | 747 kB 00:00 2024-03-18T00:37:44.934 INFO:teuthology.orchestra.run.smithi049.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.4 MB/s | 163 kB 00:00 2024-03-18T00:37:45.001 INFO:teuthology.orchestra.run.smithi049.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.9 MB/s | 127 kB 00:00 2024-03-18T00:37:45.084 INFO:teuthology.orchestra.run.smithi049.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 5.9 MB/s | 501 kB 00:00 2024-03-18T00:37:45.142 INFO:teuthology.orchestra.run.smithi107.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.7 MB/s | 890 kB 00:00 2024-03-18T00:37:45.300 INFO:teuthology.orchestra.run.smithi107.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 915 kB/s | 145 kB 00:00 2024-03-18T00:37:45.685 INFO:teuthology.orchestra.run.smithi049.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 8.6 MB/s | 5.1 MB 00:00 2024-03-18T00:37:45.760 INFO:teuthology.orchestra.run.smithi049.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 603 kB/s | 45 kB 00:00 2024-03-18T00:37:45.843 INFO:teuthology.orchestra.run.smithi049.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-18T00:37:45.952 INFO:teuthology.orchestra.run.smithi049.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.5 MB/s | 162 kB 00:00 2024-03-18T00:37:46.044 INFO:teuthology.orchestra.run.smithi049.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.4 MB/s | 320 kB 00:00 2024-03-18T00:37:46.136 INFO:teuthology.orchestra.run.smithi049.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 3.2 MB/s | 298 kB 00:00 2024-03-18T00:37:46.217 INFO:teuthology.orchestra.run.smithi107.stdout:(4/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 2.6 MB/s | 2.4 MB 00:00 2024-03-18T00:37:46.219 INFO:teuthology.orchestra.run.smithi049.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.2 MB/s | 99 kB 00:00 2024-03-18T00:37:46.292 INFO:teuthology.orchestra.run.smithi107.stdout:(5/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 2.9 MB/s | 5.4 MB 00:01 2024-03-18T00:37:46.295 INFO:teuthology.orchestra.run.smithi049.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-18T00:37:46.720 INFO:teuthology.orchestra.run.smithi049.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 7.3 MB/s | 3.1 MB 00:00 2024-03-18T00:37:46.759 INFO:teuthology.orchestra.run.smithi107.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-03-18T00:37:46.804 INFO:teuthology.orchestra.run.smithi049.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-18T00:37:46.887 INFO:teuthology.orchestra.run.smithi049.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 409 kB/s | 34 kB 00:00 2024-03-18T00:37:46.980 INFO:teuthology.orchestra.run.smithi049.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.5 MB/s | 139 kB 00:00 2024-03-18T00:37:47.076 INFO:teuthology.orchestra.run.smithi107.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 6.0 MB/s | 4.7 MB 00:00 2024-03-18T00:37:47.130 INFO:teuthology.orchestra.run.smithi106.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T00:37:47.131 INFO:teuthology.orchestra.run.smithi106.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T00:37:47.145 INFO:teuthology.orchestra.run.smithi106.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T00:37:47.146 INFO:teuthology.orchestra.run.smithi106.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T00:37:47.147 INFO:teuthology.orchestra.run.smithi106.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T00:37:47.148 INFO:teuthology.orchestra.run.smithi106.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T00:37:47.150 INFO:teuthology.orchestra.run.smithi106.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T00:37:47.153 INFO:teuthology.orchestra.run.smithi106.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T00:37:47.155 INFO:teuthology.orchestra.run.smithi106.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T00:37:47.237 INFO:teuthology.orchestra.run.smithi106.stdout:Dependencies resolved. 2024-03-18T00:37:47.249 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-03-18T00:37:47.249 INFO:teuthology.orchestra.run.smithi106.stdout: Package Arch Version Repository Size 2024-03-18T00:37:47.249 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-03-18T00:37:47.249 INFO:teuthology.orchestra.run.smithi106.stdout:Installing: 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T00:37:47.250 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T00:37:47.251 INFO:teuthology.orchestra.run.smithi106.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T00:37:47.252 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrading: 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T00:37:47.253 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T00:37:47.254 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout:Installing dependencies: 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T00:37:47.255 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T00:37:47.256 INFO:teuthology.orchestra.run.smithi106.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T00:37:47.257 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T00:37:47.258 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T00:37:47.259 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T00:37:47.260 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T00:37:47.261 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T00:37:47.262 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T00:37:47.263 INFO:teuthology.orchestra.run.smithi106.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T00:37:47.264 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T00:37:47.265 INFO:teuthology.orchestra.run.smithi106.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T00:37:47.265 INFO:teuthology.orchestra.run.smithi106.stdout:Installing weak dependencies: 2024-03-18T00:37:47.265 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T00:37:47.265 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:37:47.265 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction Summary 2024-03-18T00:37:47.265 INFO:teuthology.orchestra.run.smithi106.stdout:============================================================================================ 2024-03-18T00:37:47.265 INFO:teuthology.orchestra.run.smithi106.stdout:Install 150 Packages 2024-03-18T00:37:47.265 INFO:teuthology.orchestra.run.smithi106.stdout:Upgrade 24 Packages 2024-03-18T00:37:47.265 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:37:47.266 INFO:teuthology.orchestra.run.smithi106.stdout:Total download size: 224 M 2024-03-18T00:37:47.266 INFO:teuthology.orchestra.run.smithi106.stdout:Downloading Packages: 2024-03-18T00:37:47.271 INFO:teuthology.orchestra.run.smithi049.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 6.2 MB/s | 1.8 MB 00:00 2024-03-18T00:37:48.117 INFO:teuthology.orchestra.run.smithi106.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-03-18T00:37:48.601 INFO:teuthology.orchestra.run.smithi107.stdout:(8/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 6.4 MB/s | 9.8 MB 00:01 2024-03-18T00:37:48.684 INFO:teuthology.orchestra.run.smithi107.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 302 kB/s | 25 kB 00:00 2024-03-18T00:37:49.142 INFO:teuthology.orchestra.run.smithi106.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 869 kB/s | 890 kB 00:01 2024-03-18T00:37:49.351 INFO:teuthology.orchestra.run.smithi106.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 695 kB/s | 145 kB 00:00 2024-03-18T00:37:50.005 INFO:teuthology.orchestra.run.smithi049.stdout:(31/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 8.8 MB/s | 48 MB 00:05 2024-03-18T00:37:50.105 INFO:teuthology.orchestra.run.smithi049.stdout:(32/174): ceph-mgr-diskprediction-local-19.0.0- 2.6 MB/s | 7.4 MB 00:02 2024-03-18T00:37:50.172 INFO:teuthology.orchestra.run.smithi049.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 742 kB/s | 49 kB 00:00 2024-03-18T00:37:50.185 INFO:teuthology.orchestra.run.smithi106.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 2.2 MB/s | 5.4 MB 00:02 2024-03-18T00:37:50.249 INFO:teuthology.orchestra.run.smithi049.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 219 kB/s | 16 kB 00:00 2024-03-18T00:37:50.305 INFO:teuthology.orchestra.run.smithi049.stdout:(35/174): ceph-mgr-modules-core-19.0.0-2162.g3d 827 kB/s | 248 kB 00:00 2024-03-18T00:37:50.389 INFO:teuthology.orchestra.run.smithi049.stdout:(36/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.8 MB/s | 264 kB 00:00 2024-03-18T00:37:50.490 INFO:teuthology.orchestra.run.smithi049.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 182 kB/s | 18 kB 00:00 2024-03-18T00:37:50.506 INFO:teuthology.orchestra.run.smithi106.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 4.7 MB/s | 1.5 MB 00:00 2024-03-18T00:37:50.565 INFO:teuthology.orchestra.run.smithi106.stdout:(6/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 1.9 MB/s | 2.4 MB 00:01 2024-03-18T00:37:50.623 INFO:teuthology.orchestra.run.smithi049.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-03-18T00:37:50.682 INFO:teuthology.orchestra.run.smithi049.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-03-18T00:37:50.723 INFO:teuthology.orchestra.run.smithi049.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-18T00:37:50.765 INFO:teuthology.orchestra.run.smithi049.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-18T00:37:50.799 INFO:teuthology.orchestra.run.smithi049.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 951 kB/s | 32 kB 00:00 2024-03-18T00:37:50.891 INFO:teuthology.orchestra.run.smithi049.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-18T00:37:50.925 INFO:teuthology.orchestra.run.smithi049.stdout:(44/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 1.2 MB/s | 748 kB 00:00 2024-03-18T00:37:50.942 INFO:teuthology.orchestra.run.smithi049.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 341 kB/s | 17 kB 00:00 2024-03-18T00:37:50.958 INFO:teuthology.orchestra.run.smithi049.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 302 kB/s | 10 kB 00:00 2024-03-18T00:37:50.984 INFO:teuthology.orchestra.run.smithi049.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-18T00:37:51.001 INFO:teuthology.orchestra.run.smithi049.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-18T00:37:51.034 INFO:teuthology.orchestra.run.smithi049.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T00:37:51.060 INFO:teuthology.orchestra.run.smithi049.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-18T00:37:51.077 INFO:teuthology.orchestra.run.smithi049.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-18T00:37:51.102 INFO:teuthology.orchestra.run.smithi049.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-18T00:37:51.127 INFO:teuthology.orchestra.run.smithi049.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-18T00:37:51.153 INFO:teuthology.orchestra.run.smithi049.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-18T00:37:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 827 kB/s | 34 kB 00:00 2024-03-18T00:37:51.270 INFO:teuthology.orchestra.run.smithi049.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-03-18T00:37:51.298 INFO:teuthology.orchestra.run.smithi106.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 5.9 MB/s | 4.7 MB 00:00 2024-03-18T00:37:51.312 INFO:teuthology.orchestra.run.smithi049.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-18T00:37:51.337 INFO:teuthology.orchestra.run.smithi049.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-18T00:37:51.354 INFO:teuthology.orchestra.run.smithi049.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 3.2 MB/s | 134 kB 00:00 2024-03-18T00:37:51.480 INFO:teuthology.orchestra.run.smithi049.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-18T00:37:51.513 INFO:teuthology.orchestra.run.smithi049.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-18T00:37:51.538 INFO:teuthology.orchestra.run.smithi049.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-18T00:37:51.616 INFO:teuthology.orchestra.run.smithi106.stdout:(8/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 5.3 MB/s | 21 MB 00:03 2024-03-18T00:37:51.658 INFO:teuthology.orchestra.run.smithi106.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 608 kB/s | 25 kB 00:00 2024-03-18T00:37:51.722 INFO:teuthology.orchestra.run.smithi049.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-18T00:37:51.747 INFO:teuthology.orchestra.run.smithi049.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 117 kB/s | 24 kB 00:00 2024-03-18T00:37:51.768 INFO:teuthology.orchestra.run.smithi107.stdout:(10/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 3.4 MB/s | 17 MB 00:05 2024-03-18T00:37:51.798 INFO:teuthology.orchestra.run.smithi049.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-18T00:37:51.834 INFO:teuthology.orchestra.run.smithi049.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 241 kB/s | 18 kB 00:00 2024-03-18T00:37:51.850 INFO:teuthology.orchestra.run.smithi049.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-03-18T00:37:51.926 INFO:teuthology.orchestra.run.smithi107.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 1.4 MB/s | 221 kB 00:00 2024-03-18T00:37:51.965 INFO:teuthology.orchestra.run.smithi049.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-18T00:37:51.991 INFO:teuthology.orchestra.run.smithi049.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-18T00:37:52.016 INFO:teuthology.orchestra.run.smithi049.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-18T00:37:52.018 INFO:teuthology.orchestra.run.smithi107.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 296 kB/s | 27 kB 00:00 2024-03-18T00:37:52.041 INFO:teuthology.orchestra.run.smithi049.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-18T00:37:52.066 INFO:teuthology.orchestra.run.smithi049.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 820 kB/s | 41 kB 00:00 2024-03-18T00:37:52.091 INFO:teuthology.orchestra.run.smithi049.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-18T00:37:52.110 INFO:teuthology.orchestra.run.smithi107.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 346 kB/s | 32 kB 00:00 2024-03-18T00:37:52.116 INFO:teuthology.orchestra.run.smithi049.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-18T00:37:52.145 INFO:teuthology.orchestra.run.smithi049.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-18T00:37:52.159 INFO:teuthology.orchestra.run.smithi049.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-18T00:37:52.175 INFO:teuthology.orchestra.run.smithi049.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-18T00:37:52.267 INFO:teuthology.orchestra.run.smithi049.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 426 kB/s | 39 kB 00:00 2024-03-18T00:37:52.427 INFO:teuthology.orchestra.run.smithi107.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 2.3 MB/s | 747 kB 00:00 2024-03-18T00:37:52.534 INFO:teuthology.orchestra.run.smithi049.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-18T00:37:52.551 INFO:teuthology.orchestra.run.smithi049.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 146 kB/s | 41 kB 00:00 2024-03-18T00:37:52.577 INFO:teuthology.orchestra.run.smithi107.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.1 MB/s | 163 kB 00:00 2024-03-18T00:37:52.585 INFO:teuthology.orchestra.run.smithi049.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-18T00:37:52.602 INFO:teuthology.orchestra.run.smithi049.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-18T00:37:52.620 INFO:teuthology.orchestra.run.smithi049.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-18T00:37:52.636 INFO:teuthology.orchestra.run.smithi049.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T00:37:52.652 INFO:teuthology.orchestra.run.smithi107.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.7 MB/s | 127 kB 00:00 2024-03-18T00:37:52.661 INFO:teuthology.orchestra.run.smithi049.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-18T00:37:52.687 INFO:teuthology.orchestra.run.smithi049.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-18T00:37:52.720 INFO:teuthology.orchestra.run.smithi049.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-18T00:37:52.745 INFO:teuthology.orchestra.run.smithi049.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-18T00:37:52.770 INFO:teuthology.orchestra.run.smithi049.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-18T00:37:52.796 INFO:teuthology.orchestra.run.smithi049.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-18T00:37:52.816 INFO:teuthology.orchestra.run.smithi106.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 6.5 MB/s | 9.8 MB 00:01 2024-03-18T00:37:52.861 INFO:teuthology.orchestra.run.smithi107.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.4 MB/s | 501 kB 00:00 2024-03-18T00:37:52.909 INFO:teuthology.orchestra.run.smithi106.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 2.4 MB/s | 221 kB 00:00 2024-03-18T00:37:52.996 INFO:teuthology.orchestra.run.smithi049.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 832 kB/s | 187 kB 00:00 2024-03-18T00:37:53.000 INFO:teuthology.orchestra.run.smithi106.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 295 kB/s | 27 kB 00:00 2024-03-18T00:37:53.055 INFO:teuthology.orchestra.run.smithi049.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2024-03-18T00:37:53.088 INFO:teuthology.orchestra.run.smithi049.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-03-18T00:37:53.092 INFO:teuthology.orchestra.run.smithi106.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 346 kB/s | 32 kB 00:00 2024-03-18T00:37:53.130 INFO:teuthology.orchestra.run.smithi049.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-18T00:37:53.147 INFO:teuthology.orchestra.run.smithi049.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 367 kB/s | 21 kB 00:00 2024-03-18T00:37:53.181 INFO:teuthology.orchestra.run.smithi049.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 506 kB/s | 25 kB 00:00 2024-03-18T00:37:53.248 INFO:teuthology.orchestra.run.smithi049.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-03-18T00:37:53.284 INFO:teuthology.orchestra.run.smithi106.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 3.8 MB/s | 747 kB 00:00 2024-03-18T00:37:53.384 INFO:teuthology.orchestra.run.smithi106.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 1.6 MB/s | 163 kB 00:00 2024-03-18T00:37:53.448 INFO:teuthology.orchestra.run.smithi049.stdout:(98/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 1.8 MB/s | 21 MB 00:11 2024-03-18T00:37:53.476 INFO:teuthology.orchestra.run.smithi106.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.4 MB/s | 127 kB 00:00 2024-03-18T00:37:53.526 INFO:teuthology.orchestra.run.smithi049.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-03-18T00:37:53.548 INFO:teuthology.orchestra.run.smithi049.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 825 kB/s | 248 kB 00:00 2024-03-18T00:37:53.574 INFO:teuthology.orchestra.run.smithi049.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.7 MB/s | 216 kB 00:00 2024-03-18T00:37:53.599 INFO:teuthology.orchestra.run.smithi049.stdout:(102/174): qatlib-service-23.11.0-1.el9.x86_64. 462 kB/s | 35 kB 00:00 2024-03-18T00:37:53.624 INFO:teuthology.orchestra.run.smithi049.stdout:(103/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 590 kB/s | 44 kB 00:00 2024-03-18T00:37:53.650 INFO:teuthology.orchestra.run.smithi049.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-03-18T00:37:53.660 INFO:teuthology.orchestra.run.smithi106.stdout:(17/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 5.4 MB/s | 17 MB 00:03 2024-03-18T00:37:53.694 INFO:teuthology.orchestra.run.smithi106.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 2.3 MB/s | 501 kB 00:00 2024-03-18T00:37:53.708 INFO:teuthology.orchestra.run.smithi049.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-03-18T00:37:53.794 INFO:teuthology.orchestra.run.smithi106.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 452 kB/s | 45 kB 00:00 2024-03-18T00:37:53.825 INFO:teuthology.orchestra.run.smithi049.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.2 MB/s | 863 kB 00:00 2024-03-18T00:37:53.850 INFO:teuthology.orchestra.run.smithi049.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.1 MB/s | 309 kB 00:00 2024-03-18T00:37:53.884 INFO:teuthology.orchestra.run.smithi049.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-03-18T00:37:53.885 INFO:teuthology.orchestra.run.smithi106.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.4 MB/s | 131 kB 00:00 2024-03-18T00:37:53.909 INFO:teuthology.orchestra.run.smithi049.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 231 kB/s | 19 kB 00:00 2024-03-18T00:37:53.937 INFO:teuthology.orchestra.run.smithi049.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 516 kB/s | 26 kB 00:00 2024-03-18T00:37:53.976 INFO:teuthology.orchestra.run.smithi049.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-03-18T00:37:53.986 INFO:teuthology.orchestra.run.smithi106.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.6 MB/s | 162 kB 00:00 2024-03-18T00:37:54.002 INFO:teuthology.orchestra.run.smithi049.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-18T00:37:54.044 INFO:teuthology.orchestra.run.smithi049.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-18T00:37:54.094 INFO:teuthology.orchestra.run.smithi106.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.9 MB/s | 320 kB 00:00 2024-03-18T00:37:54.111 INFO:teuthology.orchestra.run.smithi049.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-18T00:37:54.136 INFO:teuthology.orchestra.run.smithi049.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-03-18T00:37:54.161 INFO:teuthology.orchestra.run.smithi049.stdout:(116/174): python3-asyncssh-2.13.2-4.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-03-18T00:37:54.178 INFO:teuthology.orchestra.run.smithi049.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 766 kB/s | 32 kB 00:00 2024-03-18T00:37:54.195 INFO:teuthology.orchestra.run.smithi106.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.9 MB/s | 298 kB 00:00 2024-03-18T00:37:54.196 INFO:teuthology.orchestra.run.smithi049.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 449 kB/s | 15 kB 00:00 2024-03-18T00:37:54.220 INFO:teuthology.orchestra.run.smithi049.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-03-18T00:37:54.246 INFO:teuthology.orchestra.run.smithi049.stdout:(120/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-18T00:37:54.271 INFO:teuthology.orchestra.run.smithi049.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-03-18T00:37:54.288 INFO:teuthology.orchestra.run.smithi049.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 844 kB/s | 35 kB 00:00 2024-03-18T00:37:54.295 INFO:teuthology.orchestra.run.smithi106.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 996 kB/s | 99 kB 00:00 2024-03-18T00:37:54.313 INFO:teuthology.orchestra.run.smithi049.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-18T00:37:54.338 INFO:teuthology.orchestra.run.smithi049.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-18T00:37:54.372 INFO:teuthology.orchestra.run.smithi049.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-03-18T00:37:54.389 INFO:teuthology.orchestra.run.smithi049.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-03-18T00:37:54.406 INFO:teuthology.orchestra.run.smithi049.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 526 kB/s | 18 kB 00:00 2024-03-18T00:37:54.423 INFO:teuthology.orchestra.run.smithi049.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 691 kB/s | 23 kB 00:00 2024-03-18T00:37:54.437 INFO:teuthology.orchestra.run.smithi106.stdout:(25/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 6.6 MB/s | 5.1 MB 00:00 2024-03-18T00:37:54.439 INFO:teuthology.orchestra.run.smithi049.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 584 kB/s | 19 kB 00:00 2024-03-18T00:37:54.463 INFO:teuthology.orchestra.run.smithi106.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 511 kB/s | 85 kB 00:00 2024-03-18T00:37:54.465 INFO:teuthology.orchestra.run.smithi049.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 469 kB/s | 20 kB 00:00 2024-03-18T00:37:54.477 INFO:teuthology.orchestra.run.smithi107.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 3.2 MB/s | 5.1 MB 00:01 2024-03-18T00:37:54.515 INFO:teuthology.orchestra.run.smithi049.stdout:(131/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-03-18T00:37:54.529 INFO:teuthology.orchestra.run.smithi106.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-18T00:37:54.540 INFO:teuthology.orchestra.run.smithi049.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 617 kB/s | 46 kB 00:00 2024-03-18T00:37:54.553 INFO:teuthology.orchestra.run.smithi107.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 603 kB/s | 45 kB 00:00 2024-03-18T00:37:54.591 INFO:teuthology.orchestra.run.smithi049.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.0 MB/s | 79 kB 00:00 2024-03-18T00:37:54.613 INFO:teuthology.orchestra.run.smithi106.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 407 kB/s | 34 kB 00:00 2024-03-18T00:37:54.616 INFO:teuthology.orchestra.run.smithi049.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 768 kB/s | 58 kB 00:00 2024-03-18T00:37:54.641 INFO:teuthology.orchestra.run.smithi049.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-18T00:37:54.653 INFO:teuthology.orchestra.run.smithi107.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.3 MB/s | 131 kB 00:00 2024-03-18T00:37:54.666 INFO:teuthology.orchestra.run.smithi049.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-03-18T00:37:54.692 INFO:teuthology.orchestra.run.smithi049.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-03-18T00:37:54.717 INFO:teuthology.orchestra.run.smithi049.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-03-18T00:37:54.720 INFO:teuthology.orchestra.run.smithi107.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.4 MB/s | 162 kB 00:00 2024-03-18T00:37:54.730 INFO:teuthology.orchestra.run.smithi106.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.2 MB/s | 139 kB 00:00 2024-03-18T00:37:54.742 INFO:teuthology.orchestra.run.smithi049.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-03-18T00:37:54.776 INFO:teuthology.orchestra.run.smithi049.stdout:(140/174): python3-kubernetes-26.1.0-2.el9.noar 3.0 MB/s | 1.0 MB 00:00 2024-03-18T00:37:54.801 INFO:teuthology.orchestra.run.smithi049.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-18T00:37:54.818 INFO:teuthology.orchestra.run.smithi049.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-03-18T00:37:54.835 INFO:teuthology.orchestra.run.smithi049.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 611 kB/s | 36 kB 00:00 2024-03-18T00:37:54.845 INFO:teuthology.orchestra.run.smithi107.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 2.5 MB/s | 320 kB 00:00 2024-03-18T00:37:54.860 INFO:teuthology.orchestra.run.smithi049.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 1.2 MB/s | 51 kB 00:00 2024-03-18T00:37:54.886 INFO:teuthology.orchestra.run.smithi049.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-18T00:37:54.888 INFO:teuthology.orchestra.run.smithi106.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 6.9 MB/s | 3.1 MB 00:00 2024-03-18T00:37:54.911 INFO:teuthology.orchestra.run.smithi049.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-18T00:37:54.928 INFO:teuthology.orchestra.run.smithi049.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 477 kB/s | 20 kB 00:00 2024-03-18T00:37:54.953 INFO:teuthology.orchestra.run.smithi049.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-18T00:37:54.978 INFO:teuthology.orchestra.run.smithi049.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-18T00:37:54.987 INFO:teuthology.orchestra.run.smithi107.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.1 MB/s | 298 kB 00:00 2024-03-18T00:37:55.087 INFO:teuthology.orchestra.run.smithi107.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 996 kB/s | 99 kB 00:00 2024-03-18T00:37:55.089 INFO:teuthology.orchestra.run.smithi106.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 5.0 MB/s | 1.8 MB 00:00 2024-03-18T00:37:55.104 INFO:teuthology.orchestra.run.smithi049.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-18T00:37:55.146 INFO:teuthology.orchestra.run.smithi049.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.5 MB/s | 321 kB 00:00 2024-03-18T00:37:55.146 INFO:teuthology.orchestra.run.smithi107.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-18T00:37:55.189 INFO:teuthology.orchestra.run.smithi106.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.4 MB/s | 248 kB 00:00 2024-03-18T00:37:55.298 INFO:teuthology.orchestra.run.smithi106.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 456 kB/s | 49 kB 00:00 2024-03-18T00:37:55.362 INFO:teuthology.orchestra.run.smithi049.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 11 MB/s | 2.3 MB 00:00 2024-03-18T00:37:55.373 INFO:teuthology.orchestra.run.smithi106.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 219 kB/s | 16 kB 00:00 2024-03-18T00:37:55.471 INFO:teuthology.orchestra.run.smithi049.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 17 MB/s | 1.8 MB 00:00 2024-03-18T00:37:55.473 INFO:teuthology.orchestra.run.smithi106.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 2.6 MB/s | 264 kB 00:00 2024-03-18T00:37:55.529 INFO:teuthology.orchestra.run.smithi049.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 727 kB/s | 42 kB 00:00 2024-03-18T00:37:55.588 INFO:teuthology.orchestra.run.smithi049.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.2 MB/s | 248 kB 00:00 2024-03-18T00:37:55.640 INFO:teuthology.orchestra.run.smithi106.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-18T00:37:55.655 INFO:teuthology.orchestra.run.smithi049.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-18T00:37:55.705 INFO:teuthology.orchestra.run.smithi107.stdout:(26/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 6.8 MB/s | 48 MB 00:07 2024-03-18T00:37:55.714 INFO:teuthology.orchestra.run.smithi049.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 625 kB/s | 36 kB 00:00 2024-03-18T00:37:55.716 INFO:teuthology.orchestra.run.smithi106.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 243 kB/s | 18 kB 00:00 2024-03-18T00:37:55.763 INFO:teuthology.orchestra.run.smithi107.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-18T00:37:55.772 INFO:teuthology.orchestra.run.smithi049.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-18T00:37:55.805 INFO:teuthology.orchestra.run.smithi107.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 821 kB/s | 34 kB 00:00 2024-03-18T00:37:55.906 INFO:teuthology.orchestra.run.smithi106.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 7.3 MB/s | 7.4 MB 00:01 2024-03-18T00:37:55.906 INFO:teuthology.orchestra.run.smithi049.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.9 MB/s | 256 kB 00:00 2024-03-18T00:37:55.922 INFO:teuthology.orchestra.run.smithi107.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 1.2 MB/s | 139 kB 00:00 2024-03-18T00:37:55.998 INFO:teuthology.orchestra.run.smithi049.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 251 kB/s | 23 kB 00:00 2024-03-18T00:37:56.000 INFO:teuthology.orchestra.run.smithi106.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 662 kB 00:00 2024-03-18T00:37:56.034 INFO:teuthology.orchestra.run.smithi106.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 7.1 MB/s | 920 kB 00:00 2024-03-18T00:37:56.056 INFO:teuthology.orchestra.run.smithi049.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-03-18T00:37:56.059 INFO:teuthology.orchestra.run.smithi106.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-18T00:37:56.064 INFO:teuthology.orchestra.run.smithi107.stdout:(30/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 3.4 MB/s | 3.1 MB 00:00 2024-03-18T00:37:56.084 INFO:teuthology.orchestra.run.smithi106.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-18T00:37:56.109 INFO:teuthology.orchestra.run.smithi106.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-03-18T00:37:56.132 INFO:teuthology.orchestra.run.smithi049.stdout:(162/174): librados2-19.0.0-2162.g3d78f5ba.el9. 2.9 MB/s | 3.4 MB 00:01 2024-03-18T00:37:56.148 INFO:teuthology.orchestra.run.smithi107.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 8.0 MB/s | 1.8 MB 00:00 2024-03-18T00:37:56.157 INFO:teuthology.orchestra.run.smithi049.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-18T00:37:56.193 INFO:teuthology.orchestra.run.smithi106.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-18T00:37:56.199 INFO:teuthology.orchestra.run.smithi049.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-03-18T00:37:56.206 INFO:teuthology.orchestra.run.smithi107.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 4.2 MB/s | 248 kB 00:00 2024-03-18T00:37:56.218 INFO:teuthology.orchestra.run.smithi106.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 162 kB/s | 17 kB 00:00 2024-03-18T00:37:56.243 INFO:teuthology.orchestra.run.smithi106.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-18T00:37:56.248 INFO:teuthology.orchestra.run.smithi107.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 1.2 MB/s | 49 kB 00:00 2024-03-18T00:37:56.258 INFO:teuthology.orchestra.run.smithi049.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 637 kB/s | 64 kB 00:00 2024-03-18T00:37:56.269 INFO:teuthology.orchestra.run.smithi106.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-18T00:37:56.283 INFO:teuthology.orchestra.run.smithi049.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 344 kB/s | 29 kB 00:00 2024-03-18T00:37:56.290 INFO:teuthology.orchestra.run.smithi107.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 396 kB/s | 16 kB 00:00 2024-03-18T00:37:56.294 INFO:teuthology.orchestra.run.smithi106.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-18T00:37:56.319 INFO:teuthology.orchestra.run.smithi106.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T00:37:56.340 INFO:teuthology.orchestra.run.smithi107.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 5.2 MB/s | 264 kB 00:00 2024-03-18T00:37:56.341 INFO:teuthology.orchestra.run.smithi049.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 15 MB/s | 1.2 MB 00:00 2024-03-18T00:37:56.353 INFO:teuthology.orchestra.run.smithi106.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-18T00:37:56.378 INFO:teuthology.orchestra.run.smithi106.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 806 kB/s | 46 kB 00:00 2024-03-18T00:37:56.404 INFO:teuthology.orchestra.run.smithi106.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-18T00:37:56.437 INFO:teuthology.orchestra.run.smithi106.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2024-03-18T00:37:56.458 INFO:teuthology.orchestra.run.smithi049.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 20 MB/s | 2.3 MB 00:00 2024-03-18T00:37:56.466 INFO:teuthology.orchestra.run.smithi107.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 5.9 MB/s | 748 kB 00:00 2024-03-18T00:37:56.472 INFO:teuthology.orchestra.run.smithi106.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.1 MB/s | 217 kB 00:00 2024-03-18T00:37:56.497 INFO:teuthology.orchestra.run.smithi106.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 575 kB/s | 34 kB 00:00 2024-03-18T00:37:56.507 INFO:teuthology.orchestra.run.smithi107.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 440 kB/s | 18 kB 00:00 2024-03-18T00:37:56.509 INFO:teuthology.orchestra.run.smithi049.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.6 MB/s | 2.2 MB 00:00 2024-03-18T00:37:56.576 INFO:teuthology.orchestra.run.smithi049.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.9 MB/s | 465 kB 00:00 2024-03-18T00:37:56.598 INFO:teuthology.orchestra.run.smithi106.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-18T00:37:56.601 INFO:teuthology.orchestra.run.smithi049.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 339 kB/s | 31 kB 00:00 2024-03-18T00:37:56.640 INFO:teuthology.orchestra.run.smithi106.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-18T00:37:56.668 INFO:teuthology.orchestra.run.smithi049.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 12 MB/s | 1.1 MB 00:00 2024-03-18T00:37:56.674 INFO:teuthology.orchestra.run.smithi107.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-18T00:37:56.682 INFO:teuthology.orchestra.run.smithi106.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.8 MB/s | 811 kB 00:00 2024-03-18T00:37:56.707 INFO:teuthology.orchestra.run.smithi106.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-18T00:37:56.718 INFO:teuthology.orchestra.run.smithi049.stdout:(173/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 1.8 MB/s | 3.1 MB 00:01 2024-03-18T00:37:56.743 INFO:teuthology.orchestra.run.smithi049.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 524 kB/s | 74 kB 00:00 2024-03-18T00:37:56.749 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-03-18T00:37:56.749 INFO:teuthology.orchestra.run.smithi049.stdout:Total 15 MB/s | 224 MB 00:14 2024-03-18T00:37:56.758 INFO:teuthology.orchestra.run.smithi107.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 920 kB 00:00 2024-03-18T00:37:56.832 INFO:teuthology.orchestra.run.smithi106.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-03-18T00:37:56.866 INFO:teuthology.orchestra.run.smithi106.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.5 MB/s | 1.0 MB 00:00 2024-03-18T00:37:56.891 INFO:teuthology.orchestra.run.smithi106.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-18T00:37:56.959 INFO:teuthology.orchestra.run.smithi107.stdout:(40/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 1.7 MB/s | 21 MB 00:12 2024-03-18T00:37:56.984 INFO:teuthology.orchestra.run.smithi107.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 470 kB/s | 106 kB 00:00 2024-03-18T00:37:57.010 INFO:teuthology.orchestra.run.smithi106.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 40 MB/s | 5.8 MB 00:00 2024-03-18T00:37:57.016 INFO:teuthology.orchestra.run.smithi107.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-18T00:37:57.036 INFO:teuthology.orchestra.run.smithi106.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 172 kB/s | 24 kB 00:00 2024-03-18T00:37:57.050 INFO:teuthology.orchestra.run.smithi107.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 482 kB/s | 32 kB 00:00 2024-03-18T00:37:57.061 INFO:teuthology.orchestra.run.smithi106.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-18T00:37:57.086 INFO:teuthology.orchestra.run.smithi106.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-18T00:37:57.111 INFO:teuthology.orchestra.run.smithi106.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-18T00:37:57.134 INFO:teuthology.orchestra.run.smithi107.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-03-18T00:37:57.159 INFO:teuthology.orchestra.run.smithi107.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 161 kB/s | 17 kB 00:00 2024-03-18T00:37:57.185 INFO:teuthology.orchestra.run.smithi107.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-18T00:37:57.210 INFO:teuthology.orchestra.run.smithi107.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-18T00:37:57.228 INFO:teuthology.orchestra.run.smithi106.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-18T00:37:57.235 INFO:teuthology.orchestra.run.smithi107.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-18T00:37:57.242 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-03-18T00:37:57.261 INFO:teuthology.orchestra.run.smithi107.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T00:37:57.262 INFO:teuthology.orchestra.run.smithi106.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-18T00:37:57.286 INFO:teuthology.orchestra.run.smithi107.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-18T00:37:57.287 INFO:teuthology.orchestra.run.smithi106.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-18T00:37:57.311 INFO:teuthology.orchestra.run.smithi107.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 46 kB 00:00 2024-03-18T00:37:57.313 INFO:teuthology.orchestra.run.smithi106.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-18T00:37:57.336 INFO:teuthology.orchestra.run.smithi107.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-18T00:37:57.338 INFO:teuthology.orchestra.run.smithi106.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 820 kB/s | 41 kB 00:00 2024-03-18T00:37:57.362 INFO:teuthology.orchestra.run.smithi107.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-18T00:37:57.363 INFO:teuthology.orchestra.run.smithi106.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-18T00:37:57.387 INFO:teuthology.orchestra.run.smithi107.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-18T00:37:57.388 INFO:teuthology.orchestra.run.smithi106.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-18T00:37:57.413 INFO:teuthology.orchestra.run.smithi107.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 689 kB/s | 34 kB 00:00 2024-03-18T00:37:57.414 INFO:teuthology.orchestra.run.smithi106.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-18T00:37:57.431 INFO:teuthology.orchestra.run.smithi106.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-18T00:37:57.448 INFO:teuthology.orchestra.run.smithi106.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.2 MB/s | 43 kB 00:00 2024-03-18T00:37:57.512 INFO:teuthology.orchestra.run.smithi107.stdout:(56/174): ceph-mgr-diskprediction-local-19.0.0- 5.1 MB/s | 7.4 MB 00:01 2024-03-18T00:37:57.564 INFO:teuthology.orchestra.run.smithi106.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 331 kB/s | 39 kB 00:00 2024-03-18T00:37:57.579 INFO:teuthology.orchestra.run.smithi107.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2024-03-18T00:37:57.583 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-03-18T00:37:57.584 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-03-18T00:37:57.621 INFO:teuthology.orchestra.run.smithi107.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 7.4 MB/s | 1.5 MB 00:00 2024-03-18T00:37:57.655 INFO:teuthology.orchestra.run.smithi107.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.7 MB/s | 811 kB 00:00 2024-03-18T00:37:57.680 INFO:teuthology.orchestra.run.smithi107.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-03-18T00:37:57.772 INFO:teuthology.orchestra.run.smithi107.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 8.7 MB/s | 1.0 MB 00:00 2024-03-18T00:37:57.847 INFO:teuthology.orchestra.run.smithi107.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-18T00:37:57.864 INFO:teuthology.orchestra.run.smithi107.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 427 kB/s | 78 kB 00:00 2024-03-18T00:37:57.972 INFO:teuthology.orchestra.run.smithi107.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2024-03-18T00:37:57.998 INFO:teuthology.orchestra.run.smithi107.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 187 kB/s | 24 kB 00:00 2024-03-18T00:37:58.023 INFO:teuthology.orchestra.run.smithi107.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 116 kB/s | 18 kB 00:00 2024-03-18T00:37:58.048 INFO:teuthology.orchestra.run.smithi107.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-18T00:37:58.048 INFO:teuthology.orchestra.run.smithi106.stdout:(79/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 7.4 MB/s | 48 MB 00:06 2024-03-18T00:37:58.073 INFO:teuthology.orchestra.run.smithi107.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 421 kB/s | 32 kB 00:00 2024-03-18T00:37:58.157 INFO:teuthology.orchestra.run.smithi107.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-18T00:37:58.182 INFO:teuthology.orchestra.run.smithi107.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-18T00:37:58.199 INFO:teuthology.orchestra.run.smithi106.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-03-18T00:37:58.207 INFO:teuthology.orchestra.run.smithi107.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 174 kB 00:00 2024-03-18T00:37:58.215 INFO:teuthology.orchestra.run.smithi106.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 63 kB/s | 41 kB 00:00 2024-03-18T00:37:58.233 INFO:teuthology.orchestra.run.smithi107.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-03-18T00:37:58.249 INFO:teuthology.orchestra.run.smithi106.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-18T00:37:58.258 INFO:teuthology.orchestra.run.smithi107.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-18T00:37:58.274 INFO:teuthology.orchestra.run.smithi106.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 976 kB/s | 56 kB 00:00 2024-03-18T00:37:58.283 INFO:teuthology.orchestra.run.smithi107.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-18T00:37:58.291 INFO:teuthology.orchestra.run.smithi106.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 840 kB/s | 63 kB 00:00 2024-03-18T00:37:58.308 INFO:teuthology.orchestra.run.smithi106.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 753 kB/s | 44 kB 00:00 2024-03-18T00:37:58.337 INFO:teuthology.orchestra.run.smithi107.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-18T00:37:58.341 INFO:teuthology.orchestra.run.smithi106.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-18T00:37:58.367 INFO:teuthology.orchestra.run.smithi106.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-18T00:37:58.367 INFO:teuthology.orchestra.run.smithi107.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-03-18T00:37:58.384 INFO:teuthology.orchestra.run.smithi107.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 472 kB/s | 47 kB 00:00 2024-03-18T00:37:58.402 INFO:teuthology.orchestra.run.smithi107.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 685 kB/s | 43 kB 00:00 2024-03-18T00:37:58.409 INFO:teuthology.orchestra.run.smithi106.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-18T00:37:58.434 INFO:teuthology.orchestra.run.smithi106.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-18T00:37:58.451 INFO:teuthology.orchestra.run.smithi107.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 582 kB/s | 39 kB 00:00 2024-03-18T00:37:58.459 INFO:teuthology.orchestra.run.smithi106.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-18T00:37:58.484 INFO:teuthology.orchestra.run.smithi106.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-18T00:37:58.560 INFO:teuthology.orchestra.run.smithi107.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 261 kB/s | 41 kB 00:00 2024-03-18T00:37:58.601 INFO:teuthology.orchestra.run.smithi106.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-18T00:37:58.635 INFO:teuthology.orchestra.run.smithi106.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-18T00:37:58.660 INFO:teuthology.orchestra.run.smithi106.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-18T00:37:58.702 INFO:teuthology.orchestra.run.smithi106.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-03-18T00:37:58.735 INFO:teuthology.orchestra.run.smithi107.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-18T00:37:58.760 INFO:teuthology.orchestra.run.smithi107.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 979 kB/s | 300 kB 00:00 2024-03-18T00:37:58.777 INFO:teuthology.orchestra.run.smithi107.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 263 kB/s | 56 kB 00:00 2024-03-18T00:37:58.794 INFO:teuthology.orchestra.run.smithi107.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-18T00:37:58.794 INFO:teuthology.orchestra.run.smithi106.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 191 kB/s | 25 kB 00:00 2024-03-18T00:37:58.811 INFO:teuthology.orchestra.run.smithi107.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T00:37:58.819 INFO:teuthology.orchestra.run.smithi106.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 117 kB/s | 21 kB 00:00 2024-03-18T00:37:58.836 INFO:teuthology.orchestra.run.smithi107.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-18T00:37:58.862 INFO:teuthology.orchestra.run.smithi107.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-18T00:37:58.895 INFO:teuthology.orchestra.run.smithi107.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-18T00:37:58.920 INFO:teuthology.orchestra.run.smithi106.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-03-18T00:37:58.920 INFO:teuthology.orchestra.run.smithi107.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-18T00:37:58.945 INFO:teuthology.orchestra.run.smithi107.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-18T00:37:58.970 INFO:teuthology.orchestra.run.smithi107.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-18T00:37:59.003 INFO:teuthology.orchestra.run.smithi106.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.5 MB/s | 216 kB 00:00 2024-03-18T00:37:59.029 INFO:teuthology.orchestra.run.smithi106.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.2 MB/s | 248 kB 00:00 2024-03-18T00:37:59.062 INFO:teuthology.orchestra.run.smithi106.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 595 kB/s | 35 kB 00:00 2024-03-18T00:37:59.088 INFO:teuthology.orchestra.run.smithi106.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 760 kB/s | 44 kB 00:00 2024-03-18T00:37:59.104 INFO:teuthology.orchestra.run.smithi107.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-18T00:37:59.121 INFO:teuthology.orchestra.run.smithi106.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-18T00:37:59.154 INFO:teuthology.orchestra.run.smithi107.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-18T00:37:59.188 INFO:teuthology.orchestra.run.smithi107.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-18T00:37:59.205 INFO:teuthology.orchestra.run.smithi106.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-03-18T00:37:59.246 INFO:teuthology.orchestra.run.smithi107.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-18T00:37:59.263 INFO:teuthology.orchestra.run.smithi107.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 197 kB/s | 21 kB 00:00 2024-03-18T00:37:59.280 INFO:teuthology.orchestra.run.smithi107.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 277 kB/s | 25 kB 00:00 2024-03-18T00:37:59.372 INFO:teuthology.orchestra.run.smithi107.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-18T00:37:59.422 INFO:teuthology.orchestra.run.smithi107.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-18T00:37:59.455 INFO:teuthology.orchestra.run.smithi107.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.4 MB/s | 248 kB 00:00 2024-03-18T00:37:59.472 INFO:teuthology.orchestra.run.smithi107.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 705 kB/s | 35 kB 00:00 2024-03-18T00:37:59.489 INFO:teuthology.orchestra.run.smithi107.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-18T00:37:59.514 INFO:teuthology.orchestra.run.smithi107.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-18T00:37:59.573 INFO:teuthology.orchestra.run.smithi107.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 13 MB/s | 4.1 MB 00:00 2024-03-18T00:37:59.674 INFO:teuthology.orchestra.run.smithi106.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-18T00:37:59.702 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-03-18T00:37:59.703 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-03-18T00:37:59.772 INFO:teuthology.orchestra.run.smithi106.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-03-18T00:37:59.814 INFO:teuthology.orchestra.run.smithi107.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.2 MB/s | 548 kB 00:00 2024-03-18T00:37:59.847 INFO:teuthology.orchestra.run.smithi106.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-18T00:37:59.856 INFO:teuthology.orchestra.run.smithi107.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 7.3 MB/s | 309 kB 00:00 2024-03-18T00:37:59.898 INFO:teuthology.orchestra.run.smithi107.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 478 kB/s | 19 kB 00:00 2024-03-18T00:38:00.114 INFO:teuthology.orchestra.run.smithi106.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-18T00:38:00.123 INFO:teuthology.orchestra.run.smithi107.stdout:(108/174): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-03-18T00:38:00.173 INFO:teuthology.orchestra.run.smithi106.stdout:(109/174): openssl-devel-3.0.7-27.el9.x86_64.rp 2.8 MB/s | 4.1 MB 00:01 2024-03-18T00:38:00.198 INFO:teuthology.orchestra.run.smithi106.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-03-18T00:38:00.274 INFO:teuthology.orchestra.run.smithi106.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 902 kB/s | 67 kB 00:00 2024-03-18T00:38:00.282 INFO:teuthology.orchestra.run.smithi107.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-18T00:38:00.308 INFO:teuthology.orchestra.run.smithi107.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2024-03-18T00:38:00.333 INFO:teuthology.orchestra.run.smithi107.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 970 kB/s | 49 kB 00:00 2024-03-18T00:38:00.349 INFO:teuthology.orchestra.run.smithi106.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-18T00:38:00.374 INFO:teuthology.orchestra.run.smithi107.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-18T00:38:00.400 INFO:teuthology.orchestra.run.smithi107.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-18T00:38:00.441 INFO:teuthology.orchestra.run.smithi106.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.0 MB/s | 838 kB 00:00 2024-03-18T00:38:00.466 INFO:teuthology.orchestra.run.smithi107.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 8.1 MB/s | 548 kB 00:00 2024-03-18T00:38:00.474 INFO:teuthology.orchestra.run.smithi106.stdout:(114/174): liboath-2.6.7-2.el9.x86_64.rpm 161 kB/s | 49 kB 00:00 2024-03-18T00:38:00.492 INFO:teuthology.orchestra.run.smithi107.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-18T00:38:00.516 INFO:teuthology.orchestra.run.smithi106.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-03-18T00:38:00.517 INFO:teuthology.orchestra.run.smithi107.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 865 kB/s | 43 kB 00:00 2024-03-18T00:38:00.542 INFO:teuthology.orchestra.run.smithi107.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-18T00:38:00.566 INFO:teuthology.orchestra.run.smithi106.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 472 kB/s | 43 kB 00:00 2024-03-18T00:38:00.576 INFO:teuthology.orchestra.run.smithi107.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 254 kB/s | 15 kB 00:00 2024-03-18T00:38:00.591 INFO:teuthology.orchestra.run.smithi106.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-03-18T00:38:00.601 INFO:teuthology.orchestra.run.smithi107.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-18T00:38:00.633 INFO:teuthology.orchestra.run.smithi106.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-18T00:38:00.643 INFO:teuthology.orchestra.run.smithi107.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-03-18T00:38:00.667 INFO:teuthology.orchestra.run.smithi106.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-18T00:38:00.668 INFO:teuthology.orchestra.run.smithi107.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-03-18T00:38:00.695 INFO:teuthology.orchestra.run.smithi107.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 702 kB/s | 35 kB 00:00 2024-03-18T00:38:00.719 INFO:teuthology.orchestra.run.smithi107.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-03-18T00:38:00.751 INFO:teuthology.orchestra.run.smithi106.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.6 MB/s | 303 kB 00:00 2024-03-18T00:38:00.761 INFO:teuthology.orchestra.run.smithi107.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-03-18T00:38:00.803 INFO:teuthology.orchestra.run.smithi107.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-03-18T00:38:00.818 INFO:teuthology.orchestra.run.smithi106.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 531 kB/s | 35 kB 00:00 2024-03-18T00:38:00.845 INFO:teuthology.orchestra.run.smithi107.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 425 kB/s | 18 kB 00:00 2024-03-18T00:38:00.851 INFO:teuthology.orchestra.run.smithi106.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 1.6 MB/s | 358 kB 00:00 2024-03-18T00:38:00.907 INFO:teuthology.orchestra.run.smithi107.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.7 MB/s | 2.0 MB 00:00 2024-03-18T00:38:00.920 INFO:teuthology.orchestra.run.smithi107.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 309 kB/s | 23 kB 00:00 2024-03-18T00:38:01.021 INFO:teuthology.orchestra.run.smithi107.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-03-18T00:38:01.035 INFO:teuthology.orchestra.run.smithi106.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 933 kB/s | 203 kB 00:00 2024-03-18T00:38:01.038 INFO:teuthology.orchestra.run.smithi107.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 168 kB/s | 20 kB 00:00 2024-03-18T00:38:01.071 INFO:teuthology.orchestra.run.smithi107.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.4 MB/s | 46 kB 00:00 2024-03-18T00:38:01.110 INFO:teuthology.orchestra.run.smithi106.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-03-18T00:38:01.113 INFO:teuthology.orchestra.run.smithi107.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-03-18T00:38:01.147 INFO:teuthology.orchestra.run.smithi107.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-03-18T00:38:01.178 INFO:teuthology.orchestra.run.smithi106.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-18T00:38:01.189 INFO:teuthology.orchestra.run.smithi107.stdout:(134/174): protobuf-compiler-3.14.0-13.el9.x86_ 516 kB/s | 863 kB 00:01 2024-03-18T00:38:01.206 INFO:teuthology.orchestra.run.smithi107.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 627 kB/s | 58 kB 00:00 2024-03-18T00:38:01.219 INFO:teuthology.orchestra.run.smithi106.stdout:(126/174): python3-grpcio-1.46.7-10.el9.x86_64. 5.5 MB/s | 2.0 MB 00:00 2024-03-18T00:38:01.231 INFO:teuthology.orchestra.run.smithi107.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-18T00:38:01.244 INFO:teuthology.orchestra.run.smithi106.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 265 kB/s | 18 kB 00:00 2024-03-18T00:38:01.273 INFO:teuthology.orchestra.run.smithi107.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-03-18T00:38:01.287 INFO:teuthology.orchestra.run.smithi106.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 346 kB/s | 23 kB 00:00 2024-03-18T00:38:01.290 INFO:teuthology.orchestra.run.smithi107.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 880 kB/s | 28 kB 00:00 2024-03-18T00:38:01.307 INFO:teuthology.orchestra.run.smithi107.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 917 kB/s | 31 kB 00:00 2024-03-18T00:38:01.312 INFO:teuthology.orchestra.run.smithi106.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 290 kB/s | 19 kB 00:00 2024-03-18T00:38:01.340 INFO:teuthology.orchestra.run.smithi107.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-03-18T00:38:01.354 INFO:teuthology.orchestra.run.smithi106.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 294 kB/s | 20 kB 00:00 2024-03-18T00:38:01.374 INFO:teuthology.orchestra.run.smithi107.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-18T00:38:01.379 INFO:teuthology.orchestra.run.smithi106.stdout:(131/174): protobuf-compiler-3.14.0-13.el9.x86_ 382 kB/s | 863 kB 00:02 2024-03-18T00:38:01.391 INFO:teuthology.orchestra.run.smithi107.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 715 kB/s | 36 kB 00:00 2024-03-18T00:38:01.407 INFO:teuthology.orchestra.run.smithi107.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-03-18T00:38:01.421 INFO:teuthology.orchestra.run.smithi106.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-03-18T00:38:01.433 INFO:teuthology.orchestra.run.smithi107.stdout:(144/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 397 kB/s | 90 kB 00:00 2024-03-18T00:38:01.438 INFO:teuthology.orchestra.run.smithi106.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 553 kB/s | 46 kB 00:00 2024-03-18T00:38:01.458 INFO:teuthology.orchestra.run.smithi107.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-03-18T00:38:01.483 INFO:teuthology.orchestra.run.smithi107.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-18T00:38:01.488 INFO:teuthology.orchestra.run.smithi106.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 869 kB/s | 58 kB 00:00 2024-03-18T00:38:01.500 INFO:teuthology.orchestra.run.smithi107.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 477 kB/s | 20 kB 00:00 2024-03-18T00:38:01.526 INFO:teuthology.orchestra.run.smithi107.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-03-18T00:38:01.547 INFO:teuthology.orchestra.run.smithi106.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-18T00:38:01.551 INFO:teuthology.orchestra.run.smithi107.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2024-03-18T00:38:01.564 INFO:teuthology.orchestra.run.smithi106.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-18T00:38:01.622 INFO:teuthology.orchestra.run.smithi106.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-03-18T00:38:01.648 INFO:teuthology.orchestra.run.smithi106.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 338 kB/s | 28 kB 00:00 2024-03-18T00:38:01.698 INFO:teuthology.orchestra.run.smithi106.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 410 kB/s | 31 kB 00:00 2024-03-18T00:38:01.723 INFO:teuthology.orchestra.run.smithi106.stdout:(140/174): python3-more-itertools-8.12.0-2.el9. 229 kB/s | 79 kB 00:00 2024-03-18T00:38:01.743 INFO:teuthology.orchestra.run.smithi107.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-03-18T00:38:01.748 INFO:teuthology.orchestra.run.smithi106.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-03-18T00:38:01.785 INFO:teuthology.orchestra.run.smithi107.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-18T00:38:01.790 INFO:teuthology.orchestra.run.smithi106.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-03-18T00:38:01.815 INFO:teuthology.orchestra.run.smithi106.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-03-18T00:38:01.841 INFO:teuthology.orchestra.run.smithi106.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-03-18T00:38:01.874 INFO:teuthology.orchestra.run.smithi106.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-18T00:38:01.917 INFO:teuthology.orchestra.run.smithi106.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.6 MB/s | 427 kB 00:00 2024-03-18T00:38:01.942 INFO:teuthology.orchestra.run.smithi106.stdout:(147/174): python3-websocket-client-1.2.3-2.el9 713 kB/s | 90 kB 00:00 2024-03-18T00:38:01.967 INFO:teuthology.orchestra.run.smithi106.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-03-18T00:38:01.992 INFO:teuthology.orchestra.run.smithi106.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-18T00:38:02.027 INFO:teuthology.orchestra.run.smithi107.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 6.8 MB/s | 3.4 MB 00:00 2024-03-18T00:38:02.251 INFO:teuthology.orchestra.run.smithi106.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 5.1 MB/s | 1.6 MB 00:00 2024-03-18T00:38:02.359 INFO:teuthology.orchestra.run.smithi106.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.9 MB/s | 321 kB 00:00 2024-03-18T00:38:02.419 INFO:teuthology.orchestra.run.smithi107.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 3.6 MB/s | 2.3 MB 00:00 2024-03-18T00:38:02.486 INFO:teuthology.orchestra.run.smithi107.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 637 kB/s | 42 kB 00:00 2024-03-18T00:38:02.503 INFO:teuthology.orchestra.run.smithi106.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 6.3 MB/s | 3.4 MB 00:00 2024-03-18T00:38:02.577 INFO:teuthology.orchestra.run.smithi107.stdout:(155/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 3.1 MB/s | 3.1 MB 00:01 2024-03-18T00:38:02.603 INFO:teuthology.orchestra.run.smithi107.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-18T00:38:02.610 INFO:teuthology.orchestra.run.smithi106.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 9.1 MB/s | 2.3 MB 00:00 2024-03-18T00:38:02.644 INFO:teuthology.orchestra.run.smithi106.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 42 kB 00:00 2024-03-18T00:38:02.670 INFO:teuthology.orchestra.run.smithi107.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 546 kB/s | 36 kB 00:00 2024-03-18T00:38:02.686 INFO:teuthology.orchestra.run.smithi106.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-03-18T00:38:02.695 INFO:teuthology.orchestra.run.smithi107.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 609 kB/s | 71 kB 00:00 2024-03-18T00:38:02.737 INFO:teuthology.orchestra.run.smithi107.stdout:(159/174): NetworkManager-libnm-1.46.0-1.el9.x8 2.6 MB/s | 1.8 MB 00:00 2024-03-18T00:38:02.761 INFO:teuthology.orchestra.run.smithi106.stdout:(156/174): NetworkManager-libnm-1.46.0-1.el9.x8 7.1 MB/s | 1.8 MB 00:00 2024-03-18T00:38:02.762 INFO:teuthology.orchestra.run.smithi107.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.0 MB/s | 192 kB 00:00 2024-03-18T00:38:02.786 INFO:teuthology.orchestra.run.smithi106.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 712 kB/s | 71 kB 00:00 2024-03-18T00:38:02.787 INFO:teuthology.orchestra.run.smithi107.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-18T00:38:02.803 INFO:teuthology.orchestra.run.smithi106.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 868 kB/s | 36 kB 00:00 2024-03-18T00:38:02.812 INFO:teuthology.orchestra.run.smithi107.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 306 kB/s | 23 kB 00:00 2024-03-18T00:38:02.828 INFO:teuthology.orchestra.run.smithi106.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 192 kB 00:00 2024-03-18T00:38:02.829 INFO:teuthology.orchestra.run.smithi107.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 108 kB 00:00 2024-03-18T00:38:02.854 INFO:teuthology.orchestra.run.smithi106.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-18T00:38:02.863 INFO:teuthology.orchestra.run.smithi107.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-18T00:38:02.871 INFO:teuthology.orchestra.run.smithi106.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 550 kB/s | 23 kB 00:00 2024-03-18T00:38:02.888 INFO:teuthology.orchestra.run.smithi107.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-18T00:38:02.896 INFO:teuthology.orchestra.run.smithi106.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-18T00:38:02.913 INFO:teuthology.orchestra.run.smithi107.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 573 kB/s | 29 kB 00:00 2024-03-18T00:38:02.921 INFO:teuthology.orchestra.run.smithi106.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-18T00:38:02.950 INFO:teuthology.orchestra.run.smithi106.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-18T00:38:02.963 INFO:teuthology.orchestra.run.smithi106.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-03-18T00:38:02.981 INFO:teuthology.orchestra.run.smithi106.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 858 kB/s | 29 kB 00:00 2024-03-18T00:38:03.023 INFO:teuthology.orchestra.run.smithi106.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 21 MB/s | 1.2 MB 00:00 2024-03-18T00:38:03.072 INFO:teuthology.orchestra.run.smithi107.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 6.6 MB/s | 1.2 MB 00:00 2024-03-18T00:38:03.081 INFO:teuthology.orchestra.run.smithi106.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 22 MB/s | 2.2 MB 00:00 2024-03-18T00:38:03.097 INFO:teuthology.orchestra.run.smithi107.stdout:(168/174): libfdisk-2.37.4-18.el9.x86_64.rpm 500 kB/s | 155 kB 00:00 2024-03-18T00:38:03.131 INFO:teuthology.orchestra.run.smithi106.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 21 MB/s | 2.3 MB 00:00 2024-03-18T00:38:03.157 INFO:teuthology.orchestra.run.smithi106.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.0 MB/s | 465 kB 00:00 2024-03-18T00:38:03.182 INFO:teuthology.orchestra.run.smithi106.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 622 kB/s | 31 kB 00:00 2024-03-18T00:38:03.215 INFO:teuthology.orchestra.run.smithi106.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-18T00:38:03.231 INFO:teuthology.orchestra.run.smithi107.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.8 MB/s | 2.2 MB 00:00 2024-03-18T00:38:03.281 INFO:teuthology.orchestra.run.smithi107.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 626 kB/s | 31 kB 00:00 2024-03-18T00:38:03.324 INFO:teuthology.orchestra.run.smithi106.stdout:(173/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-03-18T00:38:03.349 INFO:teuthology.orchestra.run.smithi106.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 445 kB/s | 74 kB 00:00 2024-03-18T00:38:03.356 INFO:teuthology.orchestra.run.smithi106.stdout:-------------------------------------------------------------------------------- 2024-03-18T00:38:03.356 INFO:teuthology.orchestra.run.smithi106.stdout:Total 14 MB/s | 224 MB 00:16 2024-03-18T00:38:03.390 INFO:teuthology.orchestra.run.smithi107.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.2 MB/s | 2.3 MB 00:00 2024-03-18T00:38:03.474 INFO:teuthology.orchestra.run.smithi107.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 893 kB/s | 74 kB 00:00 2024-03-18T00:38:03.507 INFO:teuthology.orchestra.run.smithi107.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.8 MB/s | 1.1 MB 00:00 2024-03-18T00:38:03.574 INFO:teuthology.orchestra.run.smithi107.stdout:(174/174): util-linux-core-2.37.4-18.el9.x86_64 977 kB/s | 465 kB 00:00 2024-03-18T00:38:03.581 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-18T00:38:03.581 INFO:teuthology.orchestra.run.smithi107.stdout:Total 11 MB/s | 224 MB 00:20 2024-03-18T00:38:03.825 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction check 2024-03-18T00:38:04.058 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-18T00:38:04.094 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction check succeeded. 2024-03-18T00:38:04.094 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction test 2024-03-18T00:38:04.326 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-18T00:38:04.327 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-18T00:38:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:Transaction test succeeded. 2024-03-18T00:38:05.806 INFO:teuthology.orchestra.run.smithi106.stdout:Running transaction 2024-03-18T00:38:06.035 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-18T00:38:06.035 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-18T00:38:07.056 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-03-18T00:38:07.157 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T00:38:07.191 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T00:38:07.229 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T00:38:07.239 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T00:38:07.293 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T00:38:07.353 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T00:38:07.394 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T00:38:07.419 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T00:38:07.469 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T00:38:07.817 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T00:38:07.861 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T00:38:08.003 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T00:38:08.029 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T00:38:08.110 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T00:38:08.166 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T00:38:08.218 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T00:38:08.402 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T00:38:08.456 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T00:38:08.484 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T00:38:08.597 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T00:38:08.626 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T00:38:08.692 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T00:38:08.730 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T00:38:08.753 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T00:38:08.770 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T00:38:08.955 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T00:38:08.993 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T00:38:09.025 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T00:38:09.177 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T00:38:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T00:38:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T00:38:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T00:38:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T00:38:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T00:38:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T00:38:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T00:38:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T00:38:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T00:38:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T00:38:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T00:38:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T00:38:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T00:38:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T00:38:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T00:38:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T00:38:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T00:38:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T00:38:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T00:38:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T00:38:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T00:38:10.643 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T00:38:10.716 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T00:38:10.758 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T00:38:10.835 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T00:38:10.980 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T00:38:11.036 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T00:38:11.597 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T00:38:11.683 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T00:38:11.767 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T00:38:11.808 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T00:38:11.817 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T00:38:11.858 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T00:38:11.905 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T00:38:11.943 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T00:38:11.973 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T00:38:12.508 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T00:38:12.670 INFO:teuthology.orchestra.run.smithi106.stdout: Preparing : 1/1 2024-03-18T00:38:12.684 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T00:38:12.760 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T00:38:12.798 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T00:38:12.805 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T00:38:12.829 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T00:38:12.868 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T00:38:12.877 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T00:38:12.926 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T00:38:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T00:38:12.997 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T00:38:13.005 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T00:38:13.038 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T00:38:13.054 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T00:38:13.064 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T00:38:13.067 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-18T00:38:13.130 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T00:38:13.166 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T00:38:13.177 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T00:38:13.203 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T00:38:13.206 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T00:38:13.211 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T00:38:13.250 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T00:38:13.285 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T00:38:13.329 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T00:38:13.333 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T00:38:13.343 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T00:38:13.371 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T00:38:13.510 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T00:38:13.513 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T00:38:13.559 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T00:38:13.564 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T00:38:13.589 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T00:38:13.604 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T00:38:13.631 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T00:38:13.636 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T00:38:13.637 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T00:38:13.681 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T00:38:13.698 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T00:38:13.730 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T00:38:13.764 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T00:38:13.805 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T00:38:13.818 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T00:38:13.829 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T00:38:13.841 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T00:38:13.854 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T00:38:13.932 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T00:38:13.936 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T00:38:13.962 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T00:38:14.007 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T00:38:14.007 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T00:38:14.063 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T00:38:14.068 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T00:38:14.177 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T00:38:14.180 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T00:38:14.182 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T00:38:14.222 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T00:38:14.234 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T00:38:14.238 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T00:38:14.271 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T00:38:14.301 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T00:38:14.312 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T00:38:14.342 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T00:38:14.357 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T00:38:14.362 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T00:38:14.380 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T00:38:14.389 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T00:38:14.438 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T00:38:14.484 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T00:38:14.507 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T00:38:14.522 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T00:38:14.550 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T00:38:14.568 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T00:38:14.599 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T00:38:14.617 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T00:38:14.627 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T00:38:14.713 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T00:38:14.714 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T00:38:14.736 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T00:38:14.754 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T00:38:14.762 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T00:38:14.783 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T00:38:14.786 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T00:38:14.799 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T00:38:14.831 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T00:38:14.838 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T00:38:14.854 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T00:38:14.869 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T00:38:14.870 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T00:38:14.890 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T00:38:14.906 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T00:38:14.941 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T00:38:14.980 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T00:38:14.988 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T00:38:15.012 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T00:38:15.037 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T00:38:15.050 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T00:38:15.068 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T00:38:15.069 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T00:38:15.100 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T00:38:15.112 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T00:38:15.147 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T00:38:15.175 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T00:38:15.177 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T00:38:15.196 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T00:38:15.210 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T00:38:15.252 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T00:38:15.295 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T00:38:15.302 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T00:38:15.346 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T00:38:15.380 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T00:38:15.396 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T00:38:15.420 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T00:38:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T00:38:15.472 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T00:38:15.518 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T00:38:15.519 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T00:38:15.549 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T00:38:15.597 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T00:38:15.631 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T00:38:15.667 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T00:38:15.675 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T00:38:15.698 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T00:38:15.709 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T00:38:15.734 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T00:38:15.760 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T00:38:15.793 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T00:38:15.808 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T00:38:15.870 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T00:38:15.888 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T00:38:15.908 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T00:38:15.927 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T00:38:15.953 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T00:38:15.973 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T00:38:15.989 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T00:38:16.007 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T00:38:16.052 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T00:38:16.063 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T00:38:16.083 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T00:38:16.111 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T00:38:16.116 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T00:38:16.155 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T00:38:16.180 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T00:38:16.216 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T00:38:16.220 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T00:38:16.254 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T00:38:16.260 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T00:38:16.297 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T00:38:16.297 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T00:38:16.323 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T00:38:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T00:38:16.365 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T00:38:16.411 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T00:38:16.420 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T00:38:16.455 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T00:38:16.499 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T00:38:16.501 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T00:38:16.527 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T00:38:16.538 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T00:38:16.589 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T00:38:16.597 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T00:38:16.631 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T00:38:16.634 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T00:38:16.642 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T00:38:16.681 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T00:38:16.688 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T00:38:16.716 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T00:38:16.772 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T00:38:16.816 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T00:38:16.852 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T00:38:16.902 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T00:38:17.000 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T00:38:17.029 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T00:38:17.109 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T00:38:17.200 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T00:38:17.284 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T00:38:17.359 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T00:38:17.404 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T00:38:17.412 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T00:38:17.428 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T00:38:17.460 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T00:38:17.501 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T00:38:17.506 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T00:38:17.538 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T00:38:17.568 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T00:38:17.597 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T00:38:17.644 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T00:38:17.653 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T00:38:17.684 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T00:38:17.725 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T00:38:17.763 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T00:38:17.792 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T00:38:17.802 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T00:38:17.805 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T00:38:17.915 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T00:38:17.976 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T00:38:18.025 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T00:38:18.028 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T00:38:18.072 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T00:38:18.193 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T00:38:18.209 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T00:38:18.262 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T00:38:18.267 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T00:38:18.307 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T00:38:18.321 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T00:38:18.386 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T00:38:18.418 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T00:38:18.437 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T00:38:18.454 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T00:38:18.485 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T00:38:18.501 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T00:38:18.537 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T00:38:18.540 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T00:38:18.588 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T00:38:18.610 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T00:38:18.639 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T00:38:18.649 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T00:38:18.686 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T00:38:18.733 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T00:38:18.769 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T00:38:18.771 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T00:38:18.812 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T00:38:18.854 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T00:38:18.903 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T00:38:18.933 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T00:38:18.974 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T00:38:18.988 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T00:38:19.035 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T00:38:19.065 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T00:38:19.068 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T00:38:19.107 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T00:38:19.116 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T00:38:19.165 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T00:38:19.191 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T00:38:19.201 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T00:38:19.211 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T00:38:19.215 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T00:38:19.248 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T00:38:19.287 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T00:38:19.288 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T00:38:19.297 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T00:38:19.314 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T00:38:19.328 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T00:38:19.339 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T00:38:19.358 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T00:38:19.386 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T00:38:19.411 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T00:38:19.415 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T00:38:19.514 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T00:38:19.533 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T00:38:19.548 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T00:38:19.605 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T00:38:19.641 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T00:38:19.668 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T00:38:19.681 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T00:38:19.715 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T00:38:19.729 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T00:38:19.741 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T00:38:19.793 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T00:38:19.832 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T00:38:19.852 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T00:38:19.862 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T00:38:19.899 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T00:38:19.902 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T00:38:19.980 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T00:38:20.029 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T00:38:20.035 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T00:38:20.076 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T00:38:20.107 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T00:38:20.138 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T00:38:20.165 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T00:38:20.166 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T00:38:20.200 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T00:38:20.201 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T00:38:20.239 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T00:38:20.240 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T00:38:20.271 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T00:38:20.271 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T00:38:20.299 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T00:38:20.334 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T00:38:20.335 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T00:38:20.374 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T00:38:20.385 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T00:38:20.410 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T00:38:20.413 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T00:38:20.428 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T00:38:20.443 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T00:38:20.453 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T00:38:20.496 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T00:38:20.499 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T00:38:20.519 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T00:38:20.529 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T00:38:20.536 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T00:38:20.554 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T00:38:20.571 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T00:38:20.587 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T00:38:20.629 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T00:38:20.659 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T00:38:20.663 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T00:38:20.749 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T00:38:20.753 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T00:38:20.802 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T00:38:20.842 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T00:38:20.842 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T00:38:20.886 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T00:38:20.897 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T00:38:20.934 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T00:38:20.981 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T00:38:20.993 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T00:38:21.043 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T00:38:21.078 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T00:38:21.128 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T00:38:21.762 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T00:38:21.815 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T00:38:21.846 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T00:38:21.857 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T00:38:21.891 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T00:38:21.914 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T00:38:21.925 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T00:38:21.973 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T00:38:22.001 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T00:38:22.033 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T00:38:22.061 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T00:38:22.093 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T00:38:22.209 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T00:38:22.248 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T00:38:22.270 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T00:38:22.299 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T00:38:22.333 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T00:38:22.377 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T00:38:22.893 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T00:38:22.925 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T00:38:22.958 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T00:38:23.038 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T00:38:23.068 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T00:38:23.146 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T00:38:23.497 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T00:38:23.537 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T00:38:23.623 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T00:38:23.623 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T00:38:23.623 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T00:38:23.623 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:23.663 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T00:38:24.256 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T00:38:24.347 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T00:38:24.348 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T00:38:24.379 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T00:38:24.410 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T00:38:24.434 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T00:38:24.437 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T00:38:24.465 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T00:38:24.497 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T00:38:24.524 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T00:38:24.747 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T00:38:24.775 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T00:38:24.835 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T00:38:24.866 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T00:38:24.871 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T00:38:24.905 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T00:38:24.941 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T00:38:24.952 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T00:38:24.984 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T00:38:25.020 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T00:38:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T00:38:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T00:38:25.478 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T00:38:25.499 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T00:38:25.507 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T00:38:25.522 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T00:38:25.561 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T00:38:25.584 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T00:38:27.685 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T00:38:27.718 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T00:38:27.764 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T00:38:27.805 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T00:38:28.251 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T00:38:28.298 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T00:38:28.339 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T00:38:28.353 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T00:38:28.353 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T00:38:28.353 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T00:38:28.353 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:28.381 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T00:38:28.392 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T00:38:28.473 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T00:38:28.474 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T00:38:28.474 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T00:38:28.474 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:28.502 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T00:38:35.671 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T00:38:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2024-03-18T00:38:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2024-03-18T00:38:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2024-03-18T00:38:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2024-03-18T00:38:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2024-03-18T00:38:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /root 2024-03-18T00:38:35.671 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2024-03-18T00:38:35.671 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:35.842 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T00:38:35.859 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T00:38:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:35.859 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T00:38:35.859 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:37.412 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T00:38:37.440 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T00:38:37.440 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:37.440 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T00:38:37.440 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T00:38:37.440 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T00:38:37.440 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:37.671 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T00:38:37.695 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T00:38:37.695 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:37.695 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T00:38:37.695 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T00:38:37.695 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T00:38:37.695 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:38.155 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T00:38:38.179 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T00:38:38.179 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:38.179 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T00:38:38.180 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T00:38:38.180 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T00:38:38.180 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:38.274 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T00:38:38.710 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T00:38:38.739 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T00:38:38.772 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T00:38:38.833 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T00:38:38.862 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T00:38:39.108 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T00:38:39.138 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T00:38:39.727 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T00:38:39.727 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /sys 2024-03-18T00:38:39.727 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /proc 2024-03-18T00:38:39.727 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /mnt 2024-03-18T00:38:39.727 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /var/tmp 2024-03-18T00:38:39.727 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /home 2024-03-18T00:38:39.727 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /root 2024-03-18T00:38:39.727 INFO:teuthology.orchestra.run.smithi106.stdout:skipping the directory /tmp 2024-03-18T00:38:39.727 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:39.903 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T00:38:39.918 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T00:38:39.918 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:39.918 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T00:38:39.919 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:40.155 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T00:38:40.197 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T00:38:40.229 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T00:38:40.229 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-03-18T00:38:40.229 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-03-18T00:38:40.229 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-03-18T00:38:40.229 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-03-18T00:38:40.229 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-03-18T00:38:40.229 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-03-18T00:38:40.229 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-03-18T00:38:40.229 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:40.303 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T00:38:40.392 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T00:38:40.408 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T00:38:40.408 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:40.408 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T00:38:40.408 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:40.466 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T00:38:40.490 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T00:38:40.515 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T00:38:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T00:38:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T00:38:40.515 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T00:38:40.516 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:40.566 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T00:38:40.577 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T00:38:40.630 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T00:38:41.365 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T00:38:41.390 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T00:38:41.390 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:41.391 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T00:38:41.391 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T00:38:41.391 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T00:38:41.391 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:41.543 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T00:38:41.576 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T00:38:41.601 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T00:38:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T00:38:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T00:38:41.601 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T00:38:41.601 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:41.624 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T00:38:41.643 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T00:38:41.646 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T00:38:41.646 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:41.646 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T00:38:41.646 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T00:38:41.646 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T00:38:41.647 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:41.667 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T00:38:41.668 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:41.668 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T00:38:41.668 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:41.719 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T00:38:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T00:38:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T00:38:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T00:38:41.743 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T00:38:41.743 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:41.896 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T00:38:41.921 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T00:38:41.921 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:41.921 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T00:38:41.921 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T00:38:41.921 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T00:38:41.921 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:42.070 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T00:38:42.091 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T00:38:42.094 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T00:38:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T00:38:42.094 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T00:38:42.095 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T00:38:42.095 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:42.114 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T00:38:42.114 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:42.114 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T00:38:42.114 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T00:38:42.114 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T00:38:42.114 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:42.146 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T00:38:42.169 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T00:38:42.169 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:42.169 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T00:38:42.169 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T00:38:42.169 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T00:38:42.169 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:42.209 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T00:38:42.593 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T00:38:42.613 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T00:38:42.624 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T00:38:42.635 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T00:38:42.636 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:42.636 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T00:38:42.636 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T00:38:42.636 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T00:38:42.636 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:42.656 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T00:38:42.716 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T00:38:42.732 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T00:38:42.739 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T00:38:42.972 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T00:38:42.999 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T00:38:43.141 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T00:38:43.172 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T00:38:43.205 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T00:38:43.265 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T00:38:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T00:38:43.533 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T00:38:43.563 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T00:38:43.958 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T00:38:43.991 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T00:38:44.090 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T00:38:44.244 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T00:38:44.284 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T00:38:44.307 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T00:38:44.308 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:44.308 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T00:38:44.308 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T00:38:44.308 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T00:38:44.308 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:44.352 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T00:38:44.362 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T00:38:44.416 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T00:38:44.542 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T00:38:44.572 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T00:38:44.673 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T00:38:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T00:38:44.874 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T00:38:44.897 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T00:38:44.897 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:44.897 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T00:38:44.897 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T00:38:44.898 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T00:38:44.898 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:44.941 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T00:38:44.952 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T00:38:45.006 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T00:38:45.284 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T00:38:45.312 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T00:38:45.335 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T00:38:45.335 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:45.335 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T00:38:45.335 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T00:38:45.335 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T00:38:45.335 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:45.380 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T00:38:45.401 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T00:38:45.401 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:45.402 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T00:38:45.402 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:45.447 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T00:38:45.469 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T00:38:45.469 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:45.469 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T00:38:45.469 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T00:38:45.469 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T00:38:45.469 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:45.790 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T00:38:45.812 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T00:38:45.812 INFO:teuthology.orchestra.run.smithi106.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:45.812 INFO:teuthology.orchestra.run.smithi106.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T00:38:45.812 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T00:38:45.812 INFO:teuthology.orchestra.run.smithi106.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T00:38:45.813 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:38:45.885 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T00:38:45.918 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T00:38:45.940 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T00:38:45.940 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:45.941 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T00:38:45.941 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T00:38:45.941 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T00:38:45.941 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:46.003 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T00:38:46.024 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T00:38:46.025 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:46.025 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T00:38:46.025 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:46.070 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T00:38:46.093 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T00:38:46.093 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:46.093 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T00:38:46.093 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T00:38:46.093 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T00:38:46.093 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T00:38:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T00:38:46.436 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T00:38:46.436 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T00:38:46.436 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T00:38:46.436 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T00:38:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:38:46.824 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T00:38:46.872 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T00:38:46.913 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T00:38:46.940 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T00:38:47.130 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T00:38:47.257 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T00:38:47.296 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T00:38:47.349 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T00:38:47.413 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T00:38:47.448 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T00:38:47.480 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T00:38:47.512 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T00:38:47.542 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T00:38:47.606 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T00:38:47.684 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T00:38:47.723 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T00:38:48.332 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T00:38:48.372 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T00:38:48.392 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T00:38:49.271 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T00:38:49.350 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T00:38:49.392 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T00:38:49.485 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T00:38:49.520 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T00:38:49.545 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T00:38:49.570 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T00:38:49.597 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T00:38:49.597 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T00:38:49.668 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T00:38:49.698 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T00:38:49.698 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T00:38:49.783 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T00:38:49.809 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T00:38:49.842 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T00:38:49.867 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T00:38:49.892 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T00:38:49.927 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T00:38:49.974 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T00:38:50.005 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T00:38:50.030 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T00:38:50.056 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T00:38:50.081 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T00:38:50.106 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T00:38:50.113 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T00:38:50.113 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T00:38:50.301 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T00:38:50.301 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T00:38:50.368 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T00:38:50.384 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T00:38:50.408 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T00:38:50.434 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T00:38:50.439 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T00:38:50.467 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T00:38:50.473 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T00:38:50.493 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T00:38:50.500 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T00:38:50.561 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T00:38:50.688 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T00:38:50.819 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T00:38:50.865 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T00:38:50.904 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T00:38:50.917 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T00:38:50.948 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T00:38:50.957 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T00:38:50.988 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T00:38:50.991 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T00:38:51.016 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T00:38:51.016 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T00:38:51.065 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T00:38:51.095 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T00:38:51.160 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T00:38:51.194 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T00:38:51.222 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T00:38:51.259 INFO:teuthology.orchestra.run.smithi106.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T00:38:51.300 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T00:38:51.339 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T00:38:51.378 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T00:38:51.431 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T00:38:51.466 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T00:38:51.498 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T00:38:51.539 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T00:38:51.569 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T00:38:51.614 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T00:38:51.667 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T00:38:51.692 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T00:38:51.844 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T00:38:51.883 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T00:38:51.902 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T00:38:52.261 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T00:38:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T00:38:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T00:38:52.717 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T00:38:52.819 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T00:38:52.859 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T00:38:52.940 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T00:38:52.981 INFO:teuthology.orchestra.run.smithi106.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T00:38:53.006 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T00:38:53.031 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T00:38:53.074 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T00:38:53.074 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T00:38:53.141 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T00:38:53.170 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T00:38:53.201 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T00:38:53.201 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T00:38:53.222 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T00:38:53.263 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T00:38:53.277 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T00:38:53.303 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T00:38:53.336 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T00:38:53.344 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T00:38:53.362 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T00:38:53.386 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T00:38:53.392 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T00:38:53.418 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T00:38:53.427 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T00:38:53.443 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T00:38:53.467 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T00:38:53.469 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T00:38:53.469 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T00:38:53.500 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T00:38:53.525 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T00:38:53.550 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T00:38:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T00:38:53.575 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T00:38:53.587 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T00:38:53.587 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T00:38:53.600 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T00:38:53.607 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T00:38:53.607 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T00:38:53.672 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T00:38:53.715 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T00:38:53.748 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T00:38:53.774 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T00:38:53.787 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T00:38:53.787 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T00:38:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T00:38:53.831 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T00:38:53.871 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T00:38:53.887 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T00:38:53.911 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T00:38:53.912 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T00:38:53.937 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T00:38:53.937 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T00:38:53.962 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T00:38:53.969 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T00:38:53.987 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T00:38:53.996 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T00:38:54.013 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T00:38:54.020 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T00:38:54.020 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T00:38:54.059 INFO:teuthology.orchestra.run.smithi106.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T00:38:54.199 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T00:38:54.200 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T00:38:54.282 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T00:38:54.307 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T00:38:54.332 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T00:38:54.365 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T00:38:54.392 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T00:38:54.456 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T00:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T00:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T00:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T00:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T00:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T00:38:55.290 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T00:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T00:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T00:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T00:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T00:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T00:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T00:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T00:38:55.291 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T00:38:55.292 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T00:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T00:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T00:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T00:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T00:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T00:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T00:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T00:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T00:38:55.293 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T00:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T00:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T00:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T00:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T00:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T00:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T00:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T00:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T00:38:55.294 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T00:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T00:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T00:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T00:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T00:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T00:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T00:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T00:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T00:38:55.295 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T00:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T00:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T00:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T00:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T00:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T00:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T00:38:55.296 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T00:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T00:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T00:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T00:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T00:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T00:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T00:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T00:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T00:38:55.297 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T00:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T00:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T00:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T00:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T00:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T00:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T00:38:55.298 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T00:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T00:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T00:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T00:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T00:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T00:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T00:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T00:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T00:38:55.299 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T00:38:55.300 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T00:38:55.301 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T00:38:55.302 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T00:38:55.303 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T00:38:55.304 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T00:38:55.305 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T00:38:55.306 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T00:38:55.307 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T00:38:55.308 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T00:38:55.309 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T00:38:55.309 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T00:38:55.309 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T00:38:55.309 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T00:38:55.309 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T00:38:55.309 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T00:38:58.508 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T00:38:58.508 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T00:38:58.509 INFO:teuthology.orchestra.run.smithi049.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T00:38:58.510 INFO:teuthology.orchestra.run.smithi049.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.511 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.512 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T00:38:58.513 INFO:teuthology.orchestra.run.smithi049.stdout: jq-1.6-16.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T00:38:58.514 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T00:38:58.515 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T00:38:58.516 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T00:38:58.517 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T00:38:58.518 INFO:teuthology.orchestra.run.smithi049.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T00:38:58.518 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T00:38:58.518 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T00:38:58.518 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T00:38:58.518 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T00:38:58.518 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T00:38:58.518 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T00:38:58.518 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T00:38:58.519 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.520 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T00:38:58.521 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T00:38:58.522 INFO:teuthology.orchestra.run.smithi049.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T00:38:58.523 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T00:38:58.523 INFO:teuthology.orchestra.run.smithi049.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T00:38:58.523 INFO:teuthology.orchestra.run.smithi049.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T00:38:58.523 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T00:38:58.523 INFO:teuthology.orchestra.run.smithi049.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T00:38:58.523 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:38:58.523 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-03-18T00:38:58.985 DEBUG:teuthology.parallel:result is None 2024-03-18T00:38:59.100 INFO:teuthology.orchestra.run.smithi106.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T00:38:59.101 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T00:38:59.101 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T00:38:59.101 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T00:38:59.101 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T00:38:59.101 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T00:38:59.101 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T00:38:59.101 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T00:38:59.101 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T00:38:59.101 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T00:38:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T00:38:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T00:38:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T00:38:59.119 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T00:38:59.120 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T00:38:59.121 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T00:38:59.122 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T00:38:59.123 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T00:38:59.124 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T00:38:59.125 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T00:38:59.126 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T00:38:59.127 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T00:38:59.128 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T00:38:59.129 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T00:38:59.130 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T00:38:59.131 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T00:38:59.132 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T00:38:59.133 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T00:38:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T00:38:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T00:38:59.134 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T00:38:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T00:38:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T00:38:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T00:38:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T00:38:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T00:38:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T00:38:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T00:38:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T00:38:59.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T00:38:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T00:38:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T00:38:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T00:38:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T00:38:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T00:38:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T00:38:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T00:38:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T00:38:59.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T00:38:59.512 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T00:38:59.513 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T00:38:59.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T00:38:59.515 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T00:38:59.515 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T00:38:59.515 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T00:38:59.515 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T00:38:59.515 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T00:38:59.515 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T00:38:59.515 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T00:38:59.515 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T00:38:59.515 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T00:38:59.516 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T00:38:59.517 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T00:38:59.518 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T00:38:59.519 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T00:38:59.520 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T00:38:59.521 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T00:38:59.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T00:38:59.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T00:38:59.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T00:38:59.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T00:38:59.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T00:38:59.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T00:38:59.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T00:38:59.522 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T00:38:59.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T00:38:59.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T00:38:59.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T00:38:59.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T00:38:59.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T00:38:59.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T00:38:59.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T00:38:59.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T00:38:59.523 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T00:38:59.524 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T00:38:59.524 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T00:38:59.524 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T00:38:59.524 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T00:38:59.524 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T00:38:59.524 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T00:38:59.524 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T00:38:59.524 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T00:38:59.524 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T00:38:59.525 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T00:38:59.526 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T00:38:59.526 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T00:38:59.526 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T00:38:59.526 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T00:38:59.526 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T00:38:59.526 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T00:38:59.526 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T00:38:59.526 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T00:38:59.526 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T00:38:59.527 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T00:38:59.528 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T00:38:59.529 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T00:38:59.529 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T00:38:59.529 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T00:38:59.529 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout:Upgraded: 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T00:39:02.095 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.096 INFO:teuthology.orchestra.run.smithi106.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout:Installed: 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.097 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.098 INFO:teuthology.orchestra.run.smithi106.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: jq-1.6-16.el9.x86_64 2024-03-18T00:39:02.099 INFO:teuthology.orchestra.run.smithi106.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T00:39:02.100 INFO:teuthology.orchestra.run.smithi106.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T00:39:02.101 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.102 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T00:39:02.103 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T00:39:02.104 INFO:teuthology.orchestra.run.smithi106.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.105 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T00:39:02.106 INFO:teuthology.orchestra.run.smithi106.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T00:39:02.107 INFO:teuthology.orchestra.run.smithi106.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T00:39:02.108 INFO:teuthology.orchestra.run.smithi106.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T00:39:02.108 INFO:teuthology.orchestra.run.smithi106.stdout: 2024-03-18T00:39:02.108 INFO:teuthology.orchestra.run.smithi106.stdout:Complete! 2024-03-18T00:39:02.579 DEBUG:teuthology.parallel:result is None 2024-03-18T00:39:02.615 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T00:39:02.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:39:02.615 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-03-18T00:39:02.615 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T00:39:02.615 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.616 INFO:teuthology.orchestra.run.smithi107.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T00:39:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.618 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T00:39:02.619 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: jq-1.6-16.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T00:39:02.620 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T00:39:02.621 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T00:39:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T00:39:02.623 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T00:39:02.624 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T00:39:02.625 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T00:39:02.626 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T00:39:02.627 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T00:39:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T00:39:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T00:39:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T00:39:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T00:39:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T00:39:02.628 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:39:02.628 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-18T00:39:03.104 DEBUG:teuthology.parallel:result is None 2024-03-18T00:39:03.105 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:39:03.250 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T00:39:03.276 INFO:teuthology.orchestra.run.smithi049.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T00:39:03.277 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T00:39:03.277 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T00:39:03.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:39:03.499 DEBUG:teuthology.orchestra.run.smithi106:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T00:39:03.524 INFO:teuthology.orchestra.run.smithi106.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T00:39:03.524 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T00:39:03.524 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T00:39:03.526 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:39:03.665 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T00:39:03.687 INFO:teuthology.orchestra.run.smithi107.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T00:39:03.688 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T00:39:03.688 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T00:39:03.689 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T00:39:03.690 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:39:03.690 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T00:39:03.729 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:39:03.729 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T00:39:03.765 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:39:03.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T00:39:03.798 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T00:39:03.799 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:39:03.799 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T00:39:03.835 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T00:39:03.908 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:39:03.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T00:39:03.941 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T00:39:04.010 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:39:04.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T00:39:04.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T00:39:04.104 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T00:39:04.105 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:39:04.105 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T00:39:04.139 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T00:39:04.215 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:39:04.215 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T00:39:04.246 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T00:39:04.319 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:39:04.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T00:39:04.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T00:39:04.412 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T00:39:04.412 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:39:04.412 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T00:39:04.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T00:39:04.520 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:39:04.520 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T00:39:04.551 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T00:39:04.621 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:39:04.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T00:39:04.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T00:39:04.714 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-18T00:39:04.810 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-18T00:39:04.810 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:39:04.811 INFO:tasks.cephadm:Cluster fsid is ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:39:04.811 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-18T00:39:04.811 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.49', 'mon.b': '172.21.15.106', 'mon.c': '172.21.15.107'} 2024-03-18T00:39:04.811 INFO:tasks.cephadm:First mon is mon.a on smithi049 2024-03-18T00:39:04.811 INFO:tasks.cephadm:First mgr is x 2024-03-18T00:39:04.812 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-18T00:39:04.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo hostname $(hostname -s) 2024-03-18T00:39:04.848 DEBUG:teuthology.orchestra.run.smithi106:> sudo hostname $(hostname -s) 2024-03-18T00:39:04.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-03-18T00:39:04.906 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-18T00:39:04.907 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:39:05.063 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-18T00:39:05.329 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:39:05.330 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-18T00:39:05.330 DEBUG:teuthology.orchestra.run.smithi049:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T00:39:06.980 INFO:teuthology.orchestra.run.smithi049.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 00:39 /home/ubuntu/cephtest/cephadm 2024-03-18T00:39:06.980 DEBUG:teuthology.orchestra.run.smithi106:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T00:39:08.522 INFO:teuthology.orchestra.run.smithi106.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 00:39 /home/ubuntu/cephtest/cephadm 2024-03-18T00:39:08.522 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T00:39:09.787 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 00:39 /home/ubuntu/cephtest/cephadm 2024-03-18T00:39:09.788 DEBUG:teuthology.orchestra.run.smithi049:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T00:39:09.808 DEBUG:teuthology.orchestra.run.smithi106:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T00:39:09.826 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T00:39:09.861 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-18T00:39:09.861 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T00:39:09.863 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T00:39:09.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T00:39:10.711 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T00:39:10.749 INFO:teuthology.orchestra.run.smithi106.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T00:39:10.785 INFO:teuthology.orchestra.run.smithi049.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T00:39:40.109 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-18T00:39:40.109 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T00:39:40.109 INFO:teuthology.orchestra.run.smithi106.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T00:39:40.109 INFO:teuthology.orchestra.run.smithi106.stdout: "repo_digests": [ 2024-03-18T00:39:40.110 INFO:teuthology.orchestra.run.smithi106.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T00:39:40.110 INFO:teuthology.orchestra.run.smithi106.stdout: ] 2024-03-18T00:39:40.110 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-18T00:39:51.459 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-03-18T00:39:51.460 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T00:39:51.460 INFO:teuthology.orchestra.run.smithi107.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T00:39:51.460 INFO:teuthology.orchestra.run.smithi107.stdout: "repo_digests": [ 2024-03-18T00:39:51.460 INFO:teuthology.orchestra.run.smithi107.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T00:39:51.460 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-03-18T00:39:51.460 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-03-18T00:39:54.062 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:39:54.062 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T00:39:54.062 INFO:teuthology.orchestra.run.smithi049.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T00:39:54.062 INFO:teuthology.orchestra.run.smithi049.stdout: "repo_digests": [ 2024-03-18T00:39:54.062 INFO:teuthology.orchestra.run.smithi049.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T00:39:54.063 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T00:39:54.063 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:39:54.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph 2024-03-18T00:39:54.123 DEBUG:teuthology.orchestra.run.smithi106:> sudo mkdir -p /etc/ceph 2024-03-18T00:39:54.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-03-18T00:39:54.194 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /etc/ceph 2024-03-18T00:39:54.225 DEBUG:teuthology.orchestra.run.smithi106:> sudo chmod 777 /etc/ceph 2024-03-18T00:39:54.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-03-18T00:39:54.283 INFO:tasks.cephadm:Writing seed config... 2024-03-18T00:39:54.284 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-18T00:39:54.284 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-18T00:39:54.284 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-18T00:39:54.284 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-18T00:39:54.284 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-18T00:39:54.285 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-18T00:39:54.286 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-18T00:39:54.287 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-18T00:39:54.287 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-18T00:39:54.287 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-18T00:39:54.287 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-18T00:39:54.287 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-18T00:39:54.287 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-18T00:39:54.287 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-18T00:39:54.287 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-18T00:39:54.287 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-18T00:39:54.288 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:39:54.288 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-18T00:39:54.303 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ec390cde-e4bf-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-18T00:39:54.303 DEBUG:teuthology.orchestra.run.smithi049:mon.a> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a.service 2024-03-18T00:39:54.346 DEBUG:teuthology.orchestra.run.smithi049:mgr.x> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.x.service 2024-03-18T00:39:54.388 INFO:tasks.cephadm:Bootstrapping... 2024-03-18T00:39:54.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.49 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-18T00:39:54.589 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-03-18T00:39:54.589 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', 'ec390cde-e4bf-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.49', '--skip-admin-label'] 2024-03-18T00:39:54.615 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T00:39:54.615 INFO:teuthology.orchestra.run.smithi049.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-18T00:39:54.616 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying podman|docker is present... 2024-03-18T00:39:54.639 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T00:39:54.639 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying lvm2 is present... 2024-03-18T00:39:54.639 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying time synchronization is in place... 2024-03-18T00:39:54.647 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T00:39:54.647 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T00:39:54.653 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T00:39:54.653 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-03-18T00:39:54.659 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2024-03-18T00:39:54.665 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2024-03-18T00:39:54.665 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2024-03-18T00:39:54.666 INFO:teuthology.orchestra.run.smithi049.stdout:Repeating the final host check... 2024-03-18T00:39:54.690 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T00:39:54.690 INFO:teuthology.orchestra.run.smithi049.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-18T00:39:54.690 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl is present 2024-03-18T00:39:54.690 INFO:teuthology.orchestra.run.smithi049.stdout:lvcreate is present 2024-03-18T00:39:54.696 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T00:39:54.696 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T00:39:54.702 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T00:39:54.702 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-03-18T00:39:54.709 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout enabled 2024-03-18T00:39:54.715 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout active 2024-03-18T00:39:54.715 INFO:teuthology.orchestra.run.smithi049.stdout:Unit chronyd.service is enabled and running 2024-03-18T00:39:54.715 INFO:teuthology.orchestra.run.smithi049.stdout:Host looks OK 2024-03-18T00:39:54.715 INFO:teuthology.orchestra.run.smithi049.stdout:Cluster fsid: ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:39:54.715 INFO:teuthology.orchestra.run.smithi049.stdout:Acquiring lock 139836653399968 on /run/cephadm/ec390cde-e4bf-11ee-95c9-87774f69a715.lock 2024-03-18T00:39:54.715 INFO:teuthology.orchestra.run.smithi049.stdout:Lock 139836653399968 acquired on /run/cephadm/ec390cde-e4bf-11ee-95c9-87774f69a715.lock 2024-03-18T00:39:54.716 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 3300 ... 2024-03-18T00:39:54.716 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying IP 172.21.15.49 port 6789 ... 2024-03-18T00:39:54.716 INFO:teuthology.orchestra.run.smithi049.stdout:Base mon IP(s) is [172.21.15.49:3300, 172.21.15.49:6789], mon addrv is [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2024-03-18T00:39:54.719 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.49 metric 100 2024-03-18T00:39:54.719 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.49 metric 100 2024-03-18T00:39:54.722 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-18T00:39:54.722 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-18T00:39:54.722 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-18T00:39:54.724 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-18T00:39:54.725 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-18T00:39:54.725 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T00:39:54.725 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-18T00:39:54.725 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1566/64 scope link noprefixroute 2024-03-18T00:39:54.725 INFO:teuthology.orchestra.run.smithi049.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T00:39:54.725 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2024-03-18T00:39:54.725 INFO:teuthology.orchestra.run.smithi049.stdout:Mon IP `172.21.15.49` is in CIDR network `172.21.0.0/20` 2024-03-18T00:39:54.725 INFO:teuthology.orchestra.run.smithi049.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-18T00:39:54.726 INFO:teuthology.orchestra.run.smithi049.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-18T00:39:54.727 INFO:teuthology.orchestra.run.smithi049.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T00:39:55.156 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T00:39:55.156 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T00:39:55.157 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Getting image source signatures 2024-03-18T00:39:55.157 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-18T00:39:55.157 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-18T00:39:55.157 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T00:39:55.157 INFO:teuthology.orchestra.run.smithi049.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-18T00:39:56.112 INFO:teuthology.orchestra.run.smithi049.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T00:39:56.112 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T00:39:56.112 INFO:teuthology.orchestra.run.smithi049.stdout:Extracting ceph user uid/gid from container image... 2024-03-18T00:39:57.025 INFO:teuthology.orchestra.run.smithi049.stdout:stat: stdout 167 167 2024-03-18T00:39:57.025 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial keys... 2024-03-18T00:39:58.048 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQBdjfdleC6NFhAAt0Nbawo1niliQridUyMlcg== 2024-03-18T00:39:58.996 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQBejfdl5q/MFBAAphac834OPtkXIkG+B7MshA== 2024-03-18T00:39:59.850 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph-authtool: stdout AQBfjfdlxLGsERAAVA5j0JKnXkMDlvUz6n7TZw== 2024-03-18T00:39:59.851 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial monmap... 2024-03-18T00:40:00.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T00:40:00.615 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-18T00:40:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T00:40:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool for a [v2:172.21.15.49:3300,v1:172.21.15.49:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T00:40:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = quincy 2024-03-18T00:40:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: set fsid to ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T00:40:00.616 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:40:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mon... 2024-03-18T00:40:01.604 INFO:teuthology.orchestra.run.smithi049.stdout:create mon.a on 2024-03-18T00:40:01.806 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-18T00:40:01.947 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-18T00:40:02.115 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715.target. 2024-03-18T00:40:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715.target. 2024-03-18T00:40:02.354 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a 2024-03-18T00:40:02.354 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-18T00:40:02.534 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715.target.wants/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@.service. 2024-03-18T00:40:02.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:02 smithi049 podman[31741]: 2024-03-18 00:40:02.691144667 +0000 UTC m=+0.018373698 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:40:03.061 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T00:40:03.061 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-03-18T00:40:03.068 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T00:40:03.068 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-03-18T00:40:03.068 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-03-18T00:40:03.068 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T00:40:03.069 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon to start... 2024-03-18T00:40:03.069 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mon... 2024-03-18T00:40:03.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:02 smithi049 podman[31741]: 2024-03-18 00:40:02.965116942 +0000 UTC m=+0.292345975 container init 75f1adec3b855ac54c6d6b63da375abf4a2fc4ad5af09043b99da27a7305d5e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T00:40:03.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:02 smithi049 podman[31741]: 2024-03-18 00:40:02.968356136 +0000 UTC m=+0.295585153 container start 75f1adec3b855ac54c6d6b63da375abf4a2fc4ad5af09043b99da27a7305d5e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T00:40:03.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:03 smithi049 ceph-mon[31762]: mkfs ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:03.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:03 smithi049 ceph-mon[31762]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T00:40:03.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:03 smithi049 bash[31741]: 75f1adec3b855ac54c6d6b63da375abf4a2fc4ad5af09043b99da27a7305d5e6 2024-03-18T00:40:03.166 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:03 smithi049 systemd[1]: Started Ceph mon.a for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:40:04.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T00:40:04.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: monmap epoch 1 2024-03-18T00:40:04.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:04.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: last_changed 2024-03-18T00:40:00.185909+0000 2024-03-18T00:40:04.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:40:04.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: min_mon_release 19 (squid) 2024-03-18T00:40:04.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: election_strategy: 1 2024-03-18T00:40:04.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:40:04.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: fsmap 2024-03-18T00:40:04.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T00:40:04.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: mgrmap e1: no daemons active 2024-03-18T00:40:04.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:04 smithi049 ceph-mon[31762]: from='client.? 172.21.15.49:0/916024734' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-18T00:40:04.132 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout cluster: 2024-03-18T00:40:04.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout id: ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:04.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-18T00:40:04.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:04.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout services: 2024-03-18T00:40:04.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.514369s) 2024-03-18T00:40:04.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-18T00:40:04.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-18T00:40:04.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:04.138 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout data: 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout pgs: 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.469+0000 7f9830548700 1 Processor -- start 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.469+0000 7f9830548700 1 -- start start 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.469+0000 7f9830548700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f9828106e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.469+0000 7f9830548700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f98281073d0 con 0x7f98281069e0 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.470+0000 7f982e2e4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f9828106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.470+0000 7f982e2e4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f9828106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40318/0 (socket says 172.21.15.49:40318) 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.470+0000 7f982e2e4700 1 -- 172.21.15.49:0/2543390632 learned_addr learned my addr 172.21.15.49:0/2543390632 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:04.139 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.470+0000 7f982e2e4700 1 -- 172.21.15.49:0/2543390632 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9828107be0 con 0x7f98281069e0 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.470+0000 7f982e2e4700 1 --2- 172.21.15.49:0/2543390632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f9828106e00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9818009e80 tx=0x7f981800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e185fe98c45f6d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.470+0000 7f982d2e2700 1 -- 172.21.15.49:0/2543390632 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9818017070 con 0x7f98281069e0 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.470+0000 7f982d2e2700 1 -- 172.21.15.49:0/2543390632 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9818003a20 con 0x7f98281069e0 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.470+0000 7f982d2e2700 1 -- 172.21.15.49:0/2543390632 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9818003d20 con 0x7f98281069e0 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.471+0000 7f9830548700 1 -- 172.21.15.49:0/2543390632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 msgr2=0x7f9828106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.471+0000 7f9830548700 1 --2- 172.21.15.49:0/2543390632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f9828106e00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9818009e80 tx=0x7f981800b330 comp rx=0 tx=0).stop 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.471+0000 7f9830548700 1 -- 172.21.15.49:0/2543390632 shutdown_connections 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.471+0000 7f9830548700 1 --2- 172.21.15.49:0/2543390632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f9828106e00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.471+0000 7f9830548700 1 -- 172.21.15.49:0/2543390632 >> 172.21.15.49:0/2543390632 conn(0x7f9828101f80 msgr2=0x7f98281043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.471+0000 7f9830548700 1 -- 172.21.15.49:0/2543390632 shutdown_connections 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.471+0000 7f9830548700 1 -- 172.21.15.49:0/2543390632 wait complete. 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.472+0000 7f9830548700 1 Processor -- start 2024-03-18T00:40:04.140 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.472+0000 7f9830548700 1 -- start start 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.472+0000 7f9830548700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f982818f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.472+0000 7f9830548700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9828108110 con 0x7f98281069e0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.472+0000 7f982e2e4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f982818f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.472+0000 7f982e2e4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f982818f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40332/0 (socket says 172.21.15.49:40332) 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.472+0000 7f982e2e4700 1 -- 172.21.15.49:0/916024734 learned_addr learned my addr 172.21.15.49:0/916024734 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.472+0000 7f982e2e4700 1 -- 172.21.15.49:0/916024734 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982818fae0 con 0x7f98281069e0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.472+0000 7f982e2e4700 1 --2- 172.21.15.49:0/916024734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f982818f5a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9818004000 tx=0x7f9818004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.473+0000 7f98277fe700 1 -- 172.21.15.49:0/916024734 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9818022020 con 0x7f98281069e0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.473+0000 7f98277fe700 1 -- 172.21.15.49:0/916024734 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9818018040 con 0x7f98281069e0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.473+0000 7f9830548700 1 -- 172.21.15.49:0/916024734 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982818dcd0 con 0x7f98281069e0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.473+0000 7f98277fe700 1 -- 172.21.15.49:0/916024734 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9818017070 con 0x7f98281069e0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.473+0000 7f9830548700 1 -- 172.21.15.49:0/916024734 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f982818e210 con 0x7f98281069e0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.473+0000 7f98277fe700 1 -- 172.21.15.49:0/916024734 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9818025020 con 0x7f98281069e0 2024-03-18T00:40:04.141 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.473+0000 7f98277fe700 1 -- 172.21.15.49:0/916024734 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f981800f050 con 0x7f98281069e0 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.474+0000 7f9830548700 1 -- 172.21.15.49:0/916024734 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9828063900 con 0x7f98281069e0 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.476+0000 7f98277fe700 1 -- 172.21.15.49:0/916024734 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f981802c030 con 0x7f98281069e0 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.549+0000 7f9830548700 1 -- 172.21.15.49:0/916024734 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f9828194a80 con 0x7f98281069e0 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.549+0000 7f98277fe700 1 -- 172.21.15.49:0/916024734 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f981801e480 con 0x7f98281069e0 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.550+0000 7f9830548700 1 -- 172.21.15.49:0/916024734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 msgr2=0x7f982818f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.550+0000 7f9830548700 1 --2- 172.21.15.49:0/916024734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f982818f5a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9818004000 tx=0x7f9818004ad0 comp rx=0 tx=0).stop 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.550+0000 7f9830548700 1 -- 172.21.15.49:0/916024734 shutdown_connections 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.550+0000 7f9830548700 1 --2- 172.21.15.49:0/916024734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98281069e0 0x7f982818f5a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.550+0000 7f9830548700 1 -- 172.21.15.49:0/916024734 >> 172.21.15.49:0/916024734 conn(0x7f9828101f80 msgr2=0x7f9828191e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.550+0000 7f9830548700 1 -- 172.21.15.49:0/916024734 shutdown_connections 2024-03-18T00:40:04.142 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:03.550+0000 7f9830548700 1 -- 172.21.15.49:0/916024734 wait complete. 2024-03-18T00:40:04.143 INFO:teuthology.orchestra.run.smithi049.stdout:mon is available 2024-03-18T00:40:04.143 INFO:teuthology.orchestra.run.smithi049.stdout:Assimilating anything we can from ceph.conf... 2024-03-18T00:40:05.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:05.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T00:40:05.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:05.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.49:3300,v1:172.21.15.49:6789] 2024-03-18T00:40:05.212 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T00:40:05.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.566+0000 7f8332577700 1 Processor -- start 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.567+0000 7f8332577700 1 -- start start 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.567+0000 7f8332577700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c107f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.567+0000 7f8332577700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f832c108530 con 0x7f832c107b40 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.567+0000 7f832bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c107f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.567+0000 7f832bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c107f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40340/0 (socket says 172.21.15.49:40340) 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.567+0000 7f832bfff700 1 -- 172.21.15.49:0/1807487739 learned_addr learned my addr 172.21.15.49:0/1807487739 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.567+0000 7f832bfff700 1 -- 172.21.15.49:0/1807487739 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832c108d40 con 0x7f832c107b40 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.567+0000 7f832bfff700 1 --2- 172.21.15.49:0/1807487739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c107f60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f831c0098a0 tx=0x7f831c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a3f2f7a6b939e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f832affd700 1 -- 172.21.15.49:0/1807487739 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f831c004160 con 0x7f832c107b40 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f832affd700 1 -- 172.21.15.49:0/1807487739 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831c004300 con 0x7f832c107b40 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f832affd700 1 -- 172.21.15.49:0/1807487739 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f831c0036a0 con 0x7f832c107b40 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f8332577700 1 -- 172.21.15.49:0/1807487739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 msgr2=0x7f832c107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:05.214 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f8332577700 1 --2- 172.21.15.49:0/1807487739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c107f60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f831c0098a0 tx=0x7f831c009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f8332577700 1 -- 172.21.15.49:0/1807487739 shutdown_connections 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f8332577700 1 --2- 172.21.15.49:0/1807487739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c107f60 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f8332577700 1 -- 172.21.15.49:0/1807487739 >> 172.21.15.49:0/1807487739 conn(0x7f832c103140 msgr2=0x7f832c105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f8332577700 1 -- 172.21.15.49:0/1807487739 shutdown_connections 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.568+0000 7f8332577700 1 -- 172.21.15.49:0/1807487739 wait complete. 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.569+0000 7f8332577700 1 Processor -- start 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.569+0000 7f8332577700 1 -- start start 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.569+0000 7f8332577700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.569+0000 7f8332577700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f832c109270 con 0x7f832c107b40 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.569+0000 7f832bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.569+0000 7f832bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c19b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40354/0 (socket says 172.21.15.49:40354) 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.569+0000 7f832bfff700 1 -- 172.21.15.49:0/1792319722 learned_addr learned my addr 172.21.15.49:0/1792319722 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.570+0000 7f832bfff700 1 -- 172.21.15.49:0/1792319722 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832c19b7d0 con 0x7f832c107b40 2024-03-18T00:40:05.215 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.570+0000 7f832bfff700 1 --2- 172.21.15.49:0/1792319722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c19b290 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f831c000c00 tx=0x7f831c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.570+0000 7f83297fa700 1 -- 172.21.15.49:0/1792319722 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f831c003db0 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.570+0000 7f83297fa700 1 -- 172.21.15.49:0/1792319722 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f831c011630 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.570+0000 7f8332577700 1 -- 172.21.15.49:0/1792319722 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832c19ba60 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.570+0000 7f83297fa700 1 -- 172.21.15.49:0/1792319722 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f831c011950 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.570+0000 7f8332577700 1 -- 172.21.15.49:0/1792319722 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f832c19e750 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.570+0000 7f83297fa700 1 -- 172.21.15.49:0/1792319722 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f831c01a460 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.570+0000 7f83297fa700 1 -- 172.21.15.49:0/1792319722 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f831c01b780 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.571+0000 7f8332577700 1 -- 172.21.15.49:0/1792319722 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f832c063900 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.573+0000 7f83297fa700 1 -- 172.21.15.49:0/1792319722 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f831c031020 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.644+0000 7f8332577700 1 -- 172.21.15.49:0/1792319722 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f832c19bf40 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.647+0000 7f83297fa700 1 -- 172.21.15.49:0/1792319722 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f832c19bf40 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.648+0000 7f83297fa700 1 -- 172.21.15.49:0/1792319722 <== mon.0 v2:172.21.15.49:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f831c01be40 con 0x7f832c107b40 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.649+0000 7f8332577700 1 -- 172.21.15.49:0/1792319722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 msgr2=0x7f832c19b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:05.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.649+0000 7f8332577700 1 --2- 172.21.15.49:0/1792319722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c19b290 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f831c000c00 tx=0x7f831c004750 comp rx=0 tx=0).stop 2024-03-18T00:40:05.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.649+0000 7f8332577700 1 -- 172.21.15.49:0/1792319722 shutdown_connections 2024-03-18T00:40:05.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.649+0000 7f8332577700 1 --2- 172.21.15.49:0/1792319722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f832c107b40 0x7f832c19b290 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:05.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.649+0000 7f8332577700 1 -- 172.21.15.49:0/1792319722 >> 172.21.15.49:0/1792319722 conn(0x7f832c103140 msgr2=0x7f832c191ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:05.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.649+0000 7f8332577700 1 -- 172.21.15.49:0/1792319722 shutdown_connections 2024-03-18T00:40:05.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:04.649+0000 7f8332577700 1 -- 172.21.15.49:0/1792319722 wait complete. 2024-03-18T00:40:05.217 INFO:teuthology.orchestra.run.smithi049.stdout:Generating new minimal ceph.conf... 2024-03-18T00:40:05.328 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:05 smithi049 ceph-mon[31762]: from='client.? 172.21.15.49:0/1792319722' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T00:40:05.329 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:05 smithi049 ceph-mon[31762]: from='client.? 172.21.15.49:0/1792319722' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T00:40:06.211 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:06 smithi049 ceph-mon[31762]: from='client.? 172.21.15.49:0/2049439020' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:40:06.213 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.619+0000 7fb42119d700 1 Processor -- start 2024-03-18T00:40:06.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.619+0000 7fb42119d700 1 -- start start 2024-03-18T00:40:06.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.619+0000 7fb42119d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:06.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.619+0000 7fb42119d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c07c890 con 0x7fb41c07de60 2024-03-18T00:40:06.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.619+0000 7fb41ad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:06.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.619+0000 7fb41ad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40362/0 (socket says 172.21.15.49:40362) 2024-03-18T00:40:06.221 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.619+0000 7fb41ad9d700 1 -- 172.21.15.49:0/2328479036 learned_addr learned my addr 172.21.15.49:0/2328479036 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:06.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.619+0000 7fb41ad9d700 1 -- 172.21.15.49:0/2328479036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c07ca10 con 0x7fb41c07de60 2024-03-18T00:40:06.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb41ad9d700 1 --2- 172.21.15.49:0/2328479036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c07c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb40c0098a0 tx=0x7fb40c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef7d654265247e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:06.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb419d9b700 1 -- 172.21.15.49:0/2328479036 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb40c004160 con 0x7fb41c07de60 2024-03-18T00:40:06.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb419d9b700 1 -- 172.21.15.49:0/2328479036 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb40c004300 con 0x7fb41c07de60 2024-03-18T00:40:06.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb419d9b700 1 -- 172.21.15.49:0/2328479036 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb40c011b50 con 0x7fb41c07de60 2024-03-18T00:40:06.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb42119d700 1 -- 172.21.15.49:0/2328479036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 msgr2=0x7fb41c07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:06.222 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb42119d700 1 --2- 172.21.15.49:0/2328479036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c07c2c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb40c0098a0 tx=0x7fb40c009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb42119d700 1 -- 172.21.15.49:0/2328479036 shutdown_connections 2024-03-18T00:40:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb42119d700 1 --2- 172.21.15.49:0/2328479036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c07c2c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb42119d700 1 -- 172.21.15.49:0/2328479036 >> 172.21.15.49:0/2328479036 conn(0x7fb41c103140 msgr2=0x7fb41c105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.620+0000 7fb42119d700 1 -- 172.21.15.49:0/2328479036 shutdown_connections 2024-03-18T00:40:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.621+0000 7fb42119d700 1 -- 172.21.15.49:0/2328479036 wait complete. 2024-03-18T00:40:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.621+0000 7fb42119d700 1 Processor -- start 2024-03-18T00:40:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.621+0000 7fb42119d700 1 -- start start 2024-03-18T00:40:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb42119d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:06.223 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb42119d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb41c10e150 con 0x7fb41c07de60 2024-03-18T00:40:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb41ad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb41ad9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c19dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40368/0 (socket says 172.21.15.49:40368) 2024-03-18T00:40:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb41ad9d700 1 -- 172.21.15.49:0/2049439020 learned_addr learned my addr 172.21.15.49:0/2049439020 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb41ad9d700 1 -- 172.21.15.49:0/2049439020 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41c19e2a0 con 0x7fb41c07de60 2024-03-18T00:40:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb41ad9d700 1 --2- 172.21.15.49:0/2049439020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c19dd60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009fb0 tx=0x7fb40c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb403fff700 1 -- 172.21.15.49:0/2049439020 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb40c0038b0 con 0x7fb41c07de60 2024-03-18T00:40:06.224 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb403fff700 1 -- 172.21.15.49:0/2049439020 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb40c011e50 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb42119d700 1 -- 172.21.15.49:0/2049439020 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb41c19e530 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb403fff700 1 -- 172.21.15.49:0/2049439020 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb40c019550 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.622+0000 7fb42119d700 1 -- 172.21.15.49:0/2049439020 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb41c1a1220 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.623+0000 7fb403fff700 1 -- 172.21.15.49:0/2049439020 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb40c019be0 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.623+0000 7fb403fff700 1 -- 172.21.15.49:0/2049439020 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb40c024730 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.623+0000 7fb42119d700 1 -- 172.21.15.49:0/2049439020 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb41c063900 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.626+0000 7fb403fff700 1 -- 172.21.15.49:0/2049439020 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb40c003a50 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.696+0000 7fb42119d700 1 -- 172.21.15.49:0/2049439020 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fb41c1a17f0 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.697+0000 7fb403fff700 1 -- 172.21.15.49:0/2049439020 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fb40c0249d0 con 0x7fb41c07de60 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.697+0000 7fb42119d700 1 -- 172.21.15.49:0/2049439020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 msgr2=0x7fb41c19dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:06.225 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.697+0000 7fb42119d700 1 --2- 172.21.15.49:0/2049439020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c19dd60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009fb0 tx=0x7fb40c011bf0 comp rx=0 tx=0).stop 2024-03-18T00:40:06.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.697+0000 7fb42119d700 1 -- 172.21.15.49:0/2049439020 shutdown_connections 2024-03-18T00:40:06.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.697+0000 7fb42119d700 1 --2- 172.21.15.49:0/2049439020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb41c07de60 0x7fb41c19dd60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:06.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.697+0000 7fb42119d700 1 -- 172.21.15.49:0/2049439020 >> 172.21.15.49:0/2049439020 conn(0x7fb41c103140 msgr2=0x7fb41c103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:06.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.698+0000 7fb42119d700 1 -- 172.21.15.49:0/2049439020 shutdown_connections 2024-03-18T00:40:06.226 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:05.698+0000 7fb42119d700 1 -- 172.21.15.49:0/2049439020 wait complete. 2024-03-18T00:40:06.226 INFO:teuthology.orchestra.run.smithi049.stdout:Restarting the monitor... 2024-03-18T00:40:06.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:06 smithi049 systemd[1]: Stopping Ceph mon.a for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:40:06.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:06 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a[31758]: 2024-03-18T00:40:06.481+0000 7f38267da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T00:40:06.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:06 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a[31758]: 2024-03-18T00:40:06.481+0000 7f38267da700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T00:40:06.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:06 smithi049 podman[32005]: 2024-03-18 00:40:06.685980549 +0000 UTC m=+0.307906765 container died 75f1adec3b855ac54c6d6b63da375abf4a2fc4ad5af09043b99da27a7305d5e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307) 2024-03-18T00:40:07.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:06 smithi049 podman[32005]: 2024-03-18 00:40:06.997678356 +0000 UTC m=+0.619604581 container cleanup 75f1adec3b855ac54c6d6b63da375abf4a2fc4ad5af09043b99da27a7305d5e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-18T00:40:07.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:06 smithi049 bash[32005]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a 2024-03-18T00:40:07.356 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 podman[32019]: 2024-03-18 00:40:07.157022366 +0000 UTC m=+0.470766863 container remove 75f1adec3b855ac54c6d6b63da375abf4a2fc4ad5af09043b99da27a7305d5e6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T00:40:07.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 systemd[1]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T00:40:07.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 systemd[1]: Stopped Ceph mon.a for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:40:07.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 systemd[1]: Starting Ceph mon.a for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:40:07.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 podman[32111]: 2024-03-18 00:40:07.496037378 +0000 UTC m=+0.019392233 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T00:40:07.705 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 podman[32111]: 2024-03-18 00:40:07.599450455 +0000 UTC m=+0.122805309 container create fdaeca8c59356a016730a979fb4d4f2bee1498b4002376b708941295218b679f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T00:40:07.897 INFO:teuthology.orchestra.run.smithi049.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 podman[32111]: 2024-03-18 00:40:07.802039998 +0000 UTC m=+0.325394877 container init fdaeca8c59356a016730a979fb4d4f2bee1498b4002376b708941295218b679f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 podman[32111]: 2024-03-18 00:40:07.805034066 +0000 UTC m=+0.328388921 container start fdaeca8c59356a016730a979fb4d4f2bee1498b4002376b708941295218b679f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: pidfile_write: ignore empty --pid-file 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: load: jerasure load: lrc 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: RocksDB version: 7.9.2 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Git sha 0 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T00:40:08.020 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: DB SUMMARY 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: DB Session ID: 2ZXNVDJH2BXC0YH72ZBN 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: CURRENT file: CURRENT 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103744 ; 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.error_if_exists: 0 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.create_if_missing: 0 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.paranoid_checks: 1 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T00:40:08.021 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T00:40:08.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.env: 0x558bc3a97c60 2024-03-18T00:40:08.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T00:40:08.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.info_log: 0x558bc47cdac0 2024-03-18T00:40:08.022 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T00:40:08.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.statistics: (nil) 2024-03-18T00:40:08.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.use_fsync: 0 2024-03-18T00:40:08.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_log_file_size: 0 2024-03-18T00:40:08.023 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.allow_fallocate: 1 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.use_direct_reads: 0 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.db_log_dir: 2024-03-18T00:40:08.024 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.wal_dir: 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.write_buffer_manager: 0x558bc47a01e0 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T00:40:08.025 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.unordered_write: 0 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T00:40:08.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.row_cache: None 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.wal_filter: None 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.two_write_queues: 0 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.wal_compression: 0 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.atomic_flush: 0 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T00:40:08.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.log_readahead_size: 0 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_background_jobs: 2 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_background_compactions: -1 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_subcompactions: 1 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T00:40:08.028 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_open_files: -1 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_background_flushes: -1 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Compression algorithms supported: 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: kZSTD supported: 0 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: kXpressCompression supported: 0 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T00:40:08.029 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: kZlibCompression supported: 1 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: kSnappyCompression supported: 1 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: kLZ4Compression supported: 1 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: kBZip2Compression supported: 0 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.merge_operator: 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_filter: None 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T00:40:08.030 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558bc47cdbc0) 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks: 1 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: pin_top_level_index_and_filter: 1 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_type: 0 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_index_type: 0 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_shortening: 1 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: checksum: 4 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: no_block_cache: 0 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache: 0x558bc47d3090 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_name: BinnedLRUCache 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_options: 2024-03-18T00:40:08.031 INFO:journalctl@ceph.mon.a.smithi049.stdout: capacity : 536870912 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_shard_bits : 4 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: strict_capacity_limit : 0 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: high_pri_pool_ratio: 0.000 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_cache_compressed: (nil) 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: persistent_cache: (nil) 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size: 4096 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_size_deviation: 10 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_restart_interval: 16 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: index_block_restart_interval: 1 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: metadata_block_size: 4096 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: partition_filters: 0 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: use_delta_encoding: 1 2024-03-18T00:40:08.032 INFO:journalctl@ceph.mon.a.smithi049.stdout: filter_policy: bloomfilter 2024-03-18T00:40:08.033 INFO:journalctl@ceph.mon.a.smithi049.stdout: whole_key_filtering: 1 2024-03-18T00:40:08.033 INFO:journalctl@ceph.mon.a.smithi049.stdout: verify_compression: 0 2024-03-18T00:40:08.033 INFO:journalctl@ceph.mon.a.smithi049.stdout: read_amp_bytes_per_bit: 0 2024-03-18T00:40:08.033 INFO:journalctl@ceph.mon.a.smithi049.stdout: format_version: 5 2024-03-18T00:40:08.033 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_index_compression: 1 2024-03-18T00:40:08.033 INFO:journalctl@ceph.mon.a.smithi049.stdout: block_align: 0 2024-03-18T00:40:08.033 INFO:journalctl@ceph.mon.a.smithi049.stdout: max_auto_readahead_size: 262144 2024-03-18T00:40:08.033 INFO:journalctl@ceph.mon.a.smithi049.stdout: prepopulate_block_cache: 0 2024-03-18T00:40:08.034 INFO:journalctl@ceph.mon.a.smithi049.stdout: initial_auto_readahead_size: 8192 2024-03-18T00:40:08.034 INFO:journalctl@ceph.mon.a.smithi049.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T00:40:08.034 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T00:40:08.034 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T00:40:08.034 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression: NoCompression 2024-03-18T00:40:08.034 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T00:40:08.034 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.num_levels: 7 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T00:40:08.035 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T00:40:08.036 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T00:40:08.037 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T00:40:08.038 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.table_properties_collectors: 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.inplace_update_support: 0 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.bloom_locality: 0 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.max_successive_merges: 0 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T00:40:08.039 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.ttl: 2592000 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.enable_blob_files: false 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.min_blob_size: 0 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-18T00:40:08.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: db52601f-f6ef-456a-83bc-8d610b7ce39e 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710722407833884, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710722407834977, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98948, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 96927, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13834, "raw_average_key_size": 49, "raw_value_size": 89385, "raw_average_value_size": 320, "num_data_blocks": 12, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710722407, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "db52601f-f6ef-456a-83bc-8d610b7ce39e", "db_session_id": "2ZXNVDJH2BXC0YH72ZBN", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710722407835067, "job": 1, "event": "recovery_finished"} 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558bc48c6000 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: DB pointer 0x558bc48b0000 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T00:40:08.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** DB Stats ** 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T00:40:08.042 INFO:journalctl@ceph.mon.a.smithi049.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: L0 2/0 98.46 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: Sum 2/0 98.46 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** Compaction Stats [default] ** 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T00:40:08.043 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T00:40:08.044 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T00:40:08.044 INFO:journalctl@ceph.mon.a.smithi049.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T00:40:08.044 INFO:journalctl@ceph.mon.a.smithi049.stdout: Cumulative compaction: 0.00 GB write, 29.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T00:40:08.044 INFO:journalctl@ceph.mon.a.smithi049.stdout: Interval compaction: 0.00 GB write, 29.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T00:40:08.044 INFO:journalctl@ceph.mon.a.smithi049.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T00:40:08.044 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache BinnedLRUCache@0x558bc47d3090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-18T00:40:08.044 INFO:journalctl@ceph.mon.a.smithi049.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-18T00:40:08.044 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-03-18T00:40:08.044 INFO:journalctl@ceph.mon.a.smithi049.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: starting mon.a rank 0 at public addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] at bind addrs [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mon.a@-1(???) e1 preinit fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mon.a@-1(???).mds e1 new map 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mon.a@-1(???).mds e1 print_map 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout: e1 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout: legacy client fscid: -1 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout: 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout: No filesystems configured 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T00:40:08.045 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 bash[32111]: fdaeca8c59356a016730a979fb4d4f2bee1498b4002376b708941295218b679f 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 systemd[1]: Started Ceph mon.a for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: monmap epoch 1 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: last_changed 2024-03-18T00:40:00.185909+0000 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: min_mon_release 19 (squid) 2024-03-18T00:40:08.046 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: election_strategy: 1 2024-03-18T00:40:08.047 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:40:08.047 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: fsmap 2024-03-18T00:40:08.047 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T00:40:08.047 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:07 smithi049 ceph-mon[32130]: mgrmap e1: no daemons active 2024-03-18T00:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.343+0000 7f570cd06700 1 Processor -- start 2024-03-18T00:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.344+0000 7f570cd06700 1 -- start start 2024-03-18T00:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.344+0000 7f570cd06700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f5708107f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.344+0000 7f570cd06700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5708108530 con 0x7f5708107b40 2024-03-18T00:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.344+0000 7f570659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f5708107f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.344+0000 7f570659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f5708107f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49150/0 (socket says 172.21.15.49:49150) 2024-03-18T00:40:08.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.344+0000 7f570659c700 1 -- 172.21.15.49:0/3640092019 learned_addr learned my addr 172.21.15.49:0/3640092019 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.345+0000 7f570659c700 1 -- 172.21.15.49:0/3640092019 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5708108d40 con 0x7f5708107b40 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.345+0000 7f570659c700 1 --2- 172.21.15.49:0/3640092019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f5708107f60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f56f80098a0 tx=0x7f56f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36432d960ff68b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.345+0000 7f570559a700 1 -- 172.21.15.49:0/3640092019 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56f8004160 con 0x7f5708107b40 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.345+0000 7f570559a700 1 -- 172.21.15.49:0/3640092019 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f56f8004300 con 0x7f5708107b40 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.345+0000 7f570559a700 1 -- 172.21.15.49:0/3640092019 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56f8011b50 con 0x7f5708107b40 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.345+0000 7f570cd06700 1 -- 172.21.15.49:0/3640092019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 msgr2=0x7f5708107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.345+0000 7f570cd06700 1 --2- 172.21.15.49:0/3640092019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f5708107f60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f56f80098a0 tx=0x7f56f8009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.346+0000 7f570cd06700 1 -- 172.21.15.49:0/3640092019 shutdown_connections 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.346+0000 7f570cd06700 1 --2- 172.21.15.49:0/3640092019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f5708107f60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.346+0000 7f570cd06700 1 -- 172.21.15.49:0/3640092019 >> 172.21.15.49:0/3640092019 conn(0x7f5708103140 msgr2=0x7f5708105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.346+0000 7f570cd06700 1 -- 172.21.15.49:0/3640092019 shutdown_connections 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.346+0000 7f570cd06700 1 -- 172.21.15.49:0/3640092019 wait complete. 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.346+0000 7f570cd06700 1 Processor -- start 2024-03-18T00:40:08.907 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f570cd06700 1 -- start start 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f570cd06700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f570819bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f570cd06700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f570810c0f0 con 0x7f5708107b40 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f570659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f570819bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f570659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f570819bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49158/0 (socket says 172.21.15.49:49158) 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f570659c700 1 -- 172.21.15.49:0/1747284445 learned_addr learned my addr 172.21.15.49:0/1747284445 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f570659c700 1 -- 172.21.15.49:0/1747284445 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570819c0c0 con 0x7f5708107b40 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f570659c700 1 --2- 172.21.15.49:0/1747284445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f570819bb80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009fb0 tx=0x7f56f8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f56f77fe700 1 -- 172.21.15.49:0/1747284445 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56f80038b0 con 0x7f5708107b40 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f56f77fe700 1 -- 172.21.15.49:0/1747284445 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f56f8011e50 con 0x7f5708107b40 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.347+0000 7f570cd06700 1 -- 172.21.15.49:0/1747284445 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570819c350 con 0x7f5708107b40 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.348+0000 7f56f77fe700 1 -- 172.21.15.49:0/1747284445 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f56f8019550 con 0x7f5708107b40 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.348+0000 7f570cd06700 1 -- 172.21.15.49:0/1747284445 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570819f040 con 0x7f5708107b40 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.348+0000 7f56f77fe700 1 -- 172.21.15.49:0/1747284445 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f56f8019b90 con 0x7f5708107b40 2024-03-18T00:40:08.908 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.348+0000 7f56f77fe700 1 -- 172.21.15.49:0/1747284445 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f56f8024730 con 0x7f5708107b40 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.348+0000 7f570cd06700 1 -- 172.21.15.49:0/1747284445 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5708063900 con 0x7f5708107b40 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.351+0000 7f56f77fe700 1 -- 172.21.15.49:0/1747284445 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f56f802dda0 con 0x7f5708107b40 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.422+0000 7f570cd06700 1 -- 172.21.15.49:0/1747284445 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f570819f210 con 0x7f5708107b40 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.423+0000 7f56f77fe700 1 -- 172.21.15.49:0/1747284445 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f56f8032030 con 0x7f5708107b40 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.424+0000 7f570cd06700 1 -- 172.21.15.49:0/1747284445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 msgr2=0x7f570819bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.424+0000 7f570cd06700 1 --2- 172.21.15.49:0/1747284445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f570819bb80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009fb0 tx=0x7f56f8011bf0 comp rx=0 tx=0).stop 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.424+0000 7f570cd06700 1 -- 172.21.15.49:0/1747284445 shutdown_connections 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.424+0000 7f570cd06700 1 --2- 172.21.15.49:0/1747284445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5708107b40 0x7f570819bb80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.424+0000 7f570cd06700 1 -- 172.21.15.49:0/1747284445 >> 172.21.15.49:0/1747284445 conn(0x7f5708103140 msgr2=0x7f5708103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.424+0000 7f570cd06700 1 -- 172.21.15.49:0/1747284445 shutdown_connections 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:08.424+0000 7f570cd06700 1 -- 172.21.15.49:0/1747284445 wait complete. 2024-03-18T00:40:08.909 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-18T00:40:08.933 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-18T00:40:08.933 INFO:teuthology.orchestra.run.smithi049.stdout:Creating mgr... 2024-03-18T00:40:08.933 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-18T00:40:08.933 INFO:teuthology.orchestra.run.smithi049.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-18T00:40:09.182 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.x 2024-03-18T00:40:09.182 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-18T00:40:09.339 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715.target.wants/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@.service. 2024-03-18T00:40:09.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:09 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1747284445' entity='client.admin' 2024-03-18T00:40:09.875 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:09 smithi049 podman[32354]: 2024-03-18 00:40:09.799041897 +0000 UTC m=+0.334401053 container start 440c098891eb0f0fb99964bff7cd5d8463034c8989277afefdf55ef266a7e049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-18T00:40:09.893 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T00:40:09.893 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-03-18T00:40:09.900 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T00:40:09.900 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-03-18T00:40:09.900 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-03-18T00:40:09.900 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T00:40:09.909 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T00:40:09.909 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-03-18T00:40:09.916 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T00:40:09.917 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-03-18T00:40:09.917 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-03-18T00:40:09.917 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-18T00:40:09.917 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr to start... 2024-03-18T00:40:09.917 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr... 2024-03-18T00:40:10.145 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:09 smithi049 bash[32354]: 440c098891eb0f0fb99964bff7cd5d8463034c8989277afefdf55ef266a7e049 2024-03-18T00:40:10.145 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:09 smithi049 systemd[1]: Started Ceph mgr.x for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:40:10.424 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:10.372+0000 7fb6415c0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T00:40:10.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:10 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3718192981' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T00:40:10.875 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:10.473+0000 7fb6415c0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T00:40:10.890 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:10.891 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T00:40:10.899 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T00:40:10.900 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T00:40:10.901 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T00:40:03.037357+0000", 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.344+0000 7fa35d777700 1 Processor -- start 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.344+0000 7fa35d777700 1 -- start start 2024-03-18T00:40:10.902 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.344+0000 7fa35d777700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa35810a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.344+0000 7fa35d777700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa35807be10 con 0x7fa3581085e0 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.344+0000 7fa356ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa35810a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.344+0000 7fa356ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa35810a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49182/0 (socket says 172.21.15.49:49182) 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.344+0000 7fa356ffd700 1 -- 172.21.15.49:0/2596811840 learned_addr learned my addr 172.21.15.49:0/2596811840 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.344+0000 7fa356ffd700 1 -- 172.21.15.49:0/2596811840 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa35810af10 con 0x7fa3581085e0 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.345+0000 7fa356ffd700 1 --2- 172.21.15.49:0/2596811840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa35810a9d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa3480098a0 tx=0x7fa348009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b97a2d4928b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.345+0000 7fa355ffb700 1 -- 172.21.15.49:0/2596811840 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa348004160 con 0x7fa3581085e0 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.345+0000 7fa355ffb700 1 -- 172.21.15.49:0/2596811840 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa348004300 con 0x7fa3581085e0 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.345+0000 7fa355ffb700 1 -- 172.21.15.49:0/2596811840 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa348011b50 con 0x7fa3581085e0 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.345+0000 7fa35d777700 1 -- 172.21.15.49:0/2596811840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 msgr2=0x7fa35810a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.345+0000 7fa35d777700 1 --2- 172.21.15.49:0/2596811840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa35810a9d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa3480098a0 tx=0x7fa348009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.346+0000 7fa35d777700 1 -- 172.21.15.49:0/2596811840 shutdown_connections 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.346+0000 7fa35d777700 1 --2- 172.21.15.49:0/2596811840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa35810a9d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:10.903 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.346+0000 7fa35d777700 1 -- 172.21.15.49:0/2596811840 >> 172.21.15.49:0/2596811840 conn(0x7fa358101fd0 msgr2=0x7fa3581043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.346+0000 7fa35d777700 1 -- 172.21.15.49:0/2596811840 shutdown_connections 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.346+0000 7fa35d777700 1 -- 172.21.15.49:0/2596811840 wait complete. 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.346+0000 7fa35d777700 1 Processor -- start 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.346+0000 7fa35d777700 1 -- start start 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa35d777700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa358107e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa35d777700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa35810bee0 con 0x7fa3581085e0 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa356ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa358107e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa356ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa358107e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49190/0 (socket says 172.21.15.49:49190) 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa356ffd700 1 -- 172.21.15.49:0/3718192981 learned_addr learned my addr 172.21.15.49:0/3718192981 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa356ffd700 1 -- 172.21.15.49:0/3718192981 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3581083a0 con 0x7fa3581085e0 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa356ffd700 1 --2- 172.21.15.49:0/3718192981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa358107e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa348011670 tx=0x7fa348011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa33ffff700 1 -- 172.21.15.49:0/3718192981 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa3480039c0 con 0x7fa3581085e0 2024-03-18T00:40:10.904 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa35d777700 1 -- 172.21.15.49:0/3718192981 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa358106590 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa33ffff700 1 -- 172.21.15.49:0/3718192981 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa348011e50 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa33ffff700 1 -- 172.21.15.49:0/3718192981 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa34801a5c0 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.347+0000 7fa35d777700 1 -- 172.21.15.49:0/3718192981 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa358106ad0 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.348+0000 7fa33ffff700 1 -- 172.21.15.49:0/3718192981 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa34801ac50 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.348+0000 7fa33ffff700 1 -- 172.21.15.49:0/3718192981 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa34802d790 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.348+0000 7fa35d777700 1 -- 172.21.15.49:0/3718192981 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa358063900 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.351+0000 7fa33ffff700 1 -- 172.21.15.49:0/3718192981 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa348003b60 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.424+0000 7fa35d777700 1 -- 172.21.15.49:0/3718192981 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa358106dd0 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.425+0000 7fa33ffff700 1 -- 172.21.15.49:0/3718192981 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa34802d990 con 0x7fa3581085e0 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.426+0000 7fa35d777700 1 -- 172.21.15.49:0/3718192981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 msgr2=0x7fa358107e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.426+0000 7fa35d777700 1 --2- 172.21.15.49:0/3718192981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa358107e60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa348011670 tx=0x7fa348011b60 comp rx=0 tx=0).stop 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.426+0000 7fa35d777700 1 -- 172.21.15.49:0/3718192981 shutdown_connections 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.426+0000 7fa35d777700 1 --2- 172.21.15.49:0/3718192981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3581085e0 0x7fa358107e60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:10.905 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.426+0000 7fa35d777700 1 -- 172.21.15.49:0/3718192981 >> 172.21.15.49:0/3718192981 conn(0x7fa358101fd0 msgr2=0x7fa358102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:10.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.426+0000 7fa35d777700 1 -- 172.21.15.49:0/3718192981 shutdown_connections 2024-03-18T00:40:10.906 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:10.426+0000 7fa35d777700 1 -- 172.21.15.49:0/3718192981 wait complete. 2024-03-18T00:40:10.906 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (1/15)... 2024-03-18T00:40:11.489 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:11.148+0000 7fb6415c0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T00:40:11.489 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:11.244+0000 7fb6415c0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T00:40:11.489 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:11.390+0000 7fb6415c0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T00:40:11.489 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:11.489+0000 7fb6415c0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T00:40:11.849 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:11.575+0000 7fb6415c0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T00:40:12.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:11.849+0000 7fb6415c0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T00:40:12.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:11.962+0000 7fb6415c0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T00:40:12.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:12 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:12.050+0000 7fb6415c0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T00:40:12.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:12 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:12.147+0000 7fb6415c0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T00:40:12.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:12 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:12.329+0000 7fb6415c0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T00:40:13.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:13.358+0000 7fb6415c0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T00:40:13.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:13.492+0000 7fb6415c0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T00:40:14.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:13.839+0000 7fb6415c0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T00:40:14.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:13.931+0000 7fb6415c0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T00:40:14.616 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:14.132+0000 7fb6415c0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:40:14.616 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:14.323+0000 7fb6415c0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T00:40:14.874 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:14.616+0000 7fb6415c0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:40:15.375 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:14.884+0000 7fb6415c0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T00:40:15.375 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:15 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:15.131+0000 7fb6415c0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T00:40:16.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:16 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1775650342' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T00:40:16.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:15 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:15.972+0000 7fb6415c0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T00:40:16.874 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:16.566+0000 7fb6415c0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T00:40:16.875 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:16.667+0000 7fb6415c0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T00:40:17.375 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:16.890+0000 7fb6415c0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T00:40:17.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: Activating manager daemon x 2024-03-18T00:40:17.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: mgrmap e2: x(active, starting, since 0.00302422s) 2024-03-18T00:40:17.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T00:40:17.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T00:40:17.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T00:40:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:40:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T00:40:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: Manager daemon x is now available 2024-03-18T00:40:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T00:40:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T00:40:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' 2024-03-18T00:40:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' 2024-03-18T00:40:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:17 smithi049 ceph-mon[32130]: from='mgr.14100 172.21.15.49:0/2865468370' entity='mgr.x' 2024-03-18T00:40:19.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:18 smithi049 ceph-mon[32130]: mgrmap e3: x(active, since 1.00789s) 2024-03-18T00:40:21.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:20 smithi049 ceph-mon[32130]: mgrmap e4: x(active, since 3s) 2024-03-18T00:40:21.861 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:21.861 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-03-18T00:40:21.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:40:21.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T00:40:21.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T00:40:21.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T00:40:21.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T00:40:21.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:21.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T00:40:21.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T00:40:21.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-03-18T00:40:21.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-03-18T00:40:21.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T00:40:21.874 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T00:40:21.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-03-18T00:40:21.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-18T00:40:21.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T00:40:21.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:21.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T00:40:21.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T00:40:21.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:21.875 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T00:40:21.876 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:21.877 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:21.878 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T00:40:03.037357+0000", 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.936+0000 7ff2f20ba700 1 Processor -- start 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.936+0000 7ff2f20ba700 1 -- start start 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.936+0000 7ff2f20ba700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec07c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.936+0000 7ff2f20ba700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ec07c850 con 0x7ff2ec07de20 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.936+0000 7ff2eb7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec07c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.936+0000 7ff2eb7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec07c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49198/0 (socket says 172.21.15.49:49198) 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.936+0000 7ff2eb7fe700 1 -- 172.21.15.49:0/409291839 learned_addr learned my addr 172.21.15.49:0/409291839 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.936+0000 7ff2eb7fe700 1 -- 172.21.15.49:0/409291839 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ec07c9d0 con 0x7ff2ec07de20 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2eb7fe700 1 --2- 172.21.15.49:0/409291839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec07c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0098a0 tx=0x7ff2dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b899fe6629b8a07 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:21.879 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2ea7fc700 1 -- 172.21.15.49:0/409291839 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2dc004160 con 0x7ff2ec07de20 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2ea7fc700 1 -- 172.21.15.49:0/409291839 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff2dc004300 con 0x7ff2ec07de20 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2ea7fc700 1 -- 172.21.15.49:0/409291839 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2dc011b50 con 0x7ff2ec07de20 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2f20ba700 1 -- 172.21.15.49:0/409291839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 msgr2=0x7ff2ec07c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2f20ba700 1 --2- 172.21.15.49:0/409291839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec07c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0098a0 tx=0x7ff2dc009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2f20ba700 1 -- 172.21.15.49:0/409291839 shutdown_connections 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2f20ba700 1 --2- 172.21.15.49:0/409291839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec07c280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2f20ba700 1 -- 172.21.15.49:0/409291839 >> 172.21.15.49:0/409291839 conn(0x7ff2ec1030e0 msgr2=0x7ff2ec105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2f20ba700 1 -- 172.21.15.49:0/409291839 shutdown_connections 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.937+0000 7ff2f20ba700 1 -- 172.21.15.49:0/409291839 wait complete. 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.938+0000 7ff2f20ba700 1 Processor -- start 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.938+0000 7ff2f20ba700 1 -- start start 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.938+0000 7ff2f20ba700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec19dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.938+0000 7ff2f20ba700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff2ec10e110 con 0x7ff2ec07de20 2024-03-18T00:40:21.880 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.938+0000 7ff2eb7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec19dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.938+0000 7ff2eb7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec19dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49202/0 (socket says 172.21.15.49:49202) 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.938+0000 7ff2eb7fe700 1 -- 172.21.15.49:0/1775650342 learned_addr learned my addr 172.21.15.49:0/1775650342 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.939+0000 7ff2eb7fe700 1 -- 172.21.15.49:0/1775650342 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2ec19e1f0 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.939+0000 7ff2eb7fe700 1 --2- 172.21.15.49:0/1775650342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec19dcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc011b60 tx=0x7ff2dc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.939+0000 7ff2e8ff9700 1 -- 172.21.15.49:0/1775650342 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2dc003b20 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.939+0000 7ff2f20ba700 1 -- 172.21.15.49:0/1775650342 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2ec19e480 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.939+0000 7ff2e8ff9700 1 -- 172.21.15.49:0/1775650342 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff2dc003cc0 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.939+0000 7ff2e8ff9700 1 -- 172.21.15.49:0/1775650342 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2dc024840 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.939+0000 7ff2f20ba700 1 -- 172.21.15.49:0/1775650342 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2ec1a1170 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.939+0000 7ff2e8ff9700 1 -- 172.21.15.49:0/1775650342 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff2dc022020 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.939+0000 7ff2e8ff9700 1 -- 172.21.15.49:0/1775650342 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff2dc011e00 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.940+0000 7ff2f20ba700 1 -- 172.21.15.49:0/1775650342 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2ec063900 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:15.942+0000 7ff2e8ff9700 1 -- 172.21.15.49:0/1775650342 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff2dc02d3f0 con 0x7ff2ec07de20 2024-03-18T00:40:21.881 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:16.016+0000 7ff2f20ba700 1 -- 172.21.15.49:0/1775650342 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff2ec1a1340 con 0x7ff2ec07de20 2024-03-18T00:40:21.882 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:16.016+0000 7ff2e8ff9700 1 -- 172.21.15.49:0/1775650342 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff2dc032030 con 0x7ff2ec07de20 2024-03-18T00:40:21.882 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:16.017+0000 7ff2f20ba700 1 -- 172.21.15.49:0/1775650342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 msgr2=0x7ff2ec19dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:21.882 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:16.017+0000 7ff2f20ba700 1 --2- 172.21.15.49:0/1775650342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec19dcb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc011b60 tx=0x7ff2dc011c40 comp rx=0 tx=0).stop 2024-03-18T00:40:21.882 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:16.017+0000 7ff2f20ba700 1 -- 172.21.15.49:0/1775650342 shutdown_connections 2024-03-18T00:40:21.882 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:16.017+0000 7ff2f20ba700 1 --2- 172.21.15.49:0/1775650342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2ec07de20 0x7ff2ec19dcb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:21.882 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:16.017+0000 7ff2f20ba700 1 -- 172.21.15.49:0/1775650342 >> 172.21.15.49:0/1775650342 conn(0x7ff2ec1030e0 msgr2=0x7ff2ec103c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:21.882 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:16.017+0000 7ff2f20ba700 1 -- 172.21.15.49:0/1775650342 shutdown_connections 2024-03-18T00:40:21.882 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:16.017+0000 7ff2f20ba700 1 -- 172.21.15.49:0/1775650342 wait complete. 2024-03-18T00:40:21.882 INFO:teuthology.orchestra.run.smithi049.stdout:mgr not available, waiting (2/15)... 2024-03-18T00:40:24.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:24 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3632851041' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T00:40:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-03-18T00:40:25.103 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:40:25.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T00:40:25.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T00:40:25.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T00:40:25.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T00:40:25.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:25.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T00:40:25.104 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T00:40:25.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 0 2024-03-18T00:40:25.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-03-18T00:40:25.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T00:40:25.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T00:40:25.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-03-18T00:40:25.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-18T00:40:25.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T00:40:25.105 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:25.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T00:40:25.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T00:40:25.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:25.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T00:40:25.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:25.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T00:40:25.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T00:40:25.106 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T00:40:25.114 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T00:40:25.114 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T00:40:25.114 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T00:40:25.114 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T00:40:25.115 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ], 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T00:40:25.116 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T00:40:03.037357+0000", 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout }, 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.285+0000 7eff3eaf9700 1 Processor -- start 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.285+0000 7eff3eaf9700 1 -- start start 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.285+0000 7eff3eaf9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3810a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.285+0000 7eff3eaf9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7eff3807be30 con 0x7eff381085c0 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.285+0000 7eff3c895700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3810a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.285+0000 7eff3c895700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3810a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54898/0 (socket says 172.21.15.49:54898) 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.285+0000 7eff3c895700 1 -- 172.21.15.49:0/3829187246 learned_addr learned my addr 172.21.15.49:0/3829187246 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.285+0000 7eff3c895700 1 -- 172.21.15.49:0/3829187246 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3810aef0 con 0x7eff381085c0 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.285+0000 7eff3c895700 1 --2- 172.21.15.49:0/3829187246 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3810a9b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7eff280098a0 tx=0x7eff28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cf773dd9fd38e8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff377fe700 1 -- 172.21.15.49:0/3829187246 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff28004160 con 0x7eff381085c0 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff377fe700 1 -- 172.21.15.49:0/3829187246 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7eff28004300 con 0x7eff381085c0 2024-03-18T00:40:25.117 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff377fe700 1 -- 172.21.15.49:0/3829187246 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff28011b50 con 0x7eff381085c0 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3829187246 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 msgr2=0x7eff3810a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff3eaf9700 1 --2- 172.21.15.49:0/3829187246 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3810a9b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7eff280098a0 tx=0x7eff28009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3829187246 shutdown_connections 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff3eaf9700 1 --2- 172.21.15.49:0/3829187246 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3810a9b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3829187246 >> 172.21.15.49:0/3829187246 conn(0x7eff38101fd0 msgr2=0x7eff381043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3829187246 shutdown_connections 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.286+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3829187246 wait complete. 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.287+0000 7eff3eaf9700 1 Processor -- start 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.287+0000 7eff3eaf9700 1 -- start start 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.287+0000 7eff3eaf9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3819ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.287+0000 7eff3eaf9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7eff3810bf10 con 0x7eff381085c0 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.287+0000 7eff3c895700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3819ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.287+0000 7eff3c895700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3819ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54914/0 (socket says 172.21.15.49:54914) 2024-03-18T00:40:25.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.287+0000 7eff3c895700 1 -- 172.21.15.49:0/3632851041 learned_addr learned my addr 172.21.15.49:0/3632851041 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.288+0000 7eff3c895700 1 -- 172.21.15.49:0/3632851041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3819bfa0 con 0x7eff381085c0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.288+0000 7eff3c895700 1 --2- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3819ba60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff28011670 tx=0x7eff28011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.288+0000 7eff35ffb700 1 -- 172.21.15.49:0/3632851041 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff280039c0 con 0x7eff381085c0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.288+0000 7eff35ffb700 1 -- 172.21.15.49:0/3632851041 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7eff28011e50 con 0x7eff381085c0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.288+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff3819c230 con 0x7eff381085c0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.288+0000 7eff35ffb700 1 -- 172.21.15.49:0/3632851041 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff2801a5c0 con 0x7eff381085c0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.288+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff3819ef20 con 0x7eff381085c0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.289+0000 7eff35ffb700 1 -- 172.21.15.49:0/3632851041 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7eff2801a870 con 0x7eff381085c0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.289+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff38063900 con 0x7eff381085c0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.289+0000 7eff35ffb700 1 --2- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7eff20038ed0 0x7eff2003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.289+0000 7eff35ffb700 1 -- 172.21.15.49:0/3632851041 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7eff2804d0c0 con 0x7eff381085c0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.290+0000 7eff37fff700 1 --2- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7eff20038ed0 0x7eff2003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:25.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.290+0000 7eff37fff700 1 --2- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7eff20038ed0 0x7eff2003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff2c006fd0 tx=0x7eff2c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.295+0000 7eff35ffb700 1 -- 172.21.15.49:0/3632851041 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7eff28016260 con 0x7eff381085c0 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.582+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7eff3819f0f0 con 0x7eff381085c0 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.583+0000 7eff35ffb700 1 -- 172.21.15.49:0/3632851041 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7eff3819f0f0 con 0x7eff381085c0 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.585+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7eff20038ed0 msgr2=0x7eff2003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.585+0000 7eff3eaf9700 1 --2- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7eff20038ed0 0x7eff2003b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff2c006fd0 tx=0x7eff2c006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.585+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 msgr2=0x7eff3819ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.585+0000 7eff3eaf9700 1 --2- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3819ba60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff28011670 tx=0x7eff28011b60 comp rx=0 tx=0).stop 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.585+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 shutdown_connections 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.585+0000 7eff3eaf9700 1 --2- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7eff20038ed0 0x7eff2003b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.585+0000 7eff3eaf9700 1 --2- 172.21.15.49:0/3632851041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff381085c0 0x7eff3819ba60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.585+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 >> 172.21.15.49:0/3632851041 conn(0x7eff38101fd0 msgr2=0x7eff38102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.586+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 shutdown_connections 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:24.586+0000 7eff3eaf9700 1 -- 172.21.15.49:0/3632851041 wait complete. 2024-03-18T00:40:25.120 INFO:teuthology.orchestra.run.smithi049.stdout:mgr is available 2024-03-18T00:40:26.104 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:25 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/357697436' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T00:40:26.397 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:26.397 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T00:40:26.397 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout fsid = ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T00:40:26.398 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.525+0000 7fbb9bd58700 1 Processor -- start 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.526+0000 7fbb9bd58700 1 -- start start 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.526+0000 7fbb9bd58700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb94106810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.526+0000 7fbb9bd58700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9407bdf0 con 0x7fbb941063f0 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.526+0000 7fbb99af4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb94106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.526+0000 7fbb99af4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb94106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54916/0 (socket says 172.21.15.49:54916) 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.526+0000 7fbb99af4700 1 -- 172.21.15.49:0/1342370694 learned_addr learned my addr 172.21.15.49:0/1342370694 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.526+0000 7fbb99af4700 1 -- 172.21.15.49:0/1342370694 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb94106d50 con 0x7fbb941063f0 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.527+0000 7fbb99af4700 1 --2- 172.21.15.49:0/1342370694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb94106810 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbb900098a0 tx=0x7fbb90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38d6d0d85b2d7e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.527+0000 7fbb98af2700 1 -- 172.21.15.49:0/1342370694 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb90004160 con 0x7fbb941063f0 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.527+0000 7fbb98af2700 1 -- 172.21.15.49:0/1342370694 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb90004300 con 0x7fbb941063f0 2024-03-18T00:40:26.399 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.527+0000 7fbb98af2700 1 -- 172.21.15.49:0/1342370694 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb90011b50 con 0x7fbb941063f0 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.527+0000 7fbb9bd58700 1 -- 172.21.15.49:0/1342370694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 msgr2=0x7fbb94106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.527+0000 7fbb9bd58700 1 --2- 172.21.15.49:0/1342370694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb94106810 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbb900098a0 tx=0x7fbb90009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.528+0000 7fbb9bd58700 1 -- 172.21.15.49:0/1342370694 shutdown_connections 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.528+0000 7fbb9bd58700 1 --2- 172.21.15.49:0/1342370694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb94106810 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.528+0000 7fbb9bd58700 1 -- 172.21.15.49:0/1342370694 >> 172.21.15.49:0/1342370694 conn(0x7fbb94101fd0 msgr2=0x7fbb941043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.528+0000 7fbb9bd58700 1 -- 172.21.15.49:0/1342370694 shutdown_connections 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.528+0000 7fbb9bd58700 1 -- 172.21.15.49:0/1342370694 wait complete. 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.528+0000 7fbb9bd58700 1 Processor -- start 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.528+0000 7fbb9bd58700 1 -- start start 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb9bd58700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb9419dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb9bd58700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbb94107de0 con 0x7fbb941063f0 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb99af4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb9419dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:26.400 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb99af4700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb9419dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54920/0 (socket says 172.21.15.49:54920) 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb99af4700 1 -- 172.21.15.49:0/357697436 learned_addr learned my addr 172.21.15.49:0/357697436 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb99af4700 1 -- 172.21.15.49:0/357697436 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9419e1c0 con 0x7fbb941063f0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb99af4700 1 --2- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb9419dc80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbb90011670 tx=0x7fbb90011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb8affd700 1 -- 172.21.15.49:0/357697436 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb900039c0 con 0x7fbb941063f0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb8affd700 1 -- 172.21.15.49:0/357697436 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbb90011e50 con 0x7fbb941063f0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9419e450 con 0x7fbb941063f0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb8affd700 1 -- 172.21.15.49:0/357697436 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbb9001a5c0 con 0x7fbb941063f0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.529+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb941a1140 con 0x7fbb941063f0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.530+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb94063900 con 0x7fbb941063f0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.531+0000 7fbb8affd700 1 -- 172.21.15.49:0/357697436 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fbb90024430 con 0x7fbb941063f0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.531+0000 7fbb8affd700 1 --2- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fbb80038e80 0x7fbb8003b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.531+0000 7fbb8affd700 1 -- 172.21.15.49:0/357697436 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbb9004d140 con 0x7fbb941063f0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.531+0000 7fbb992f3700 1 --2- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fbb80038e80 0x7fbb8003b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:26.401 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.531+0000 7fbb992f3700 1 --2- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fbb80038e80 0x7fbb8003b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbb84006fd0 tx=0x7fbb84006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.536+0000 7fbb8affd700 1 -- 172.21.15.49:0/357697436 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbb90017ca0 con 0x7fbb941063f0 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.742+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fbb940008d0 con 0x7fbb941063f0 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.743+0000 7fbb8affd700 1 -- 172.21.15.49:0/357697436 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fbb9000bc90 con 0x7fbb941063f0 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fbb80038e80 msgr2=0x7fbb8003b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 --2- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fbb80038e80 0x7fbb8003b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbb84006fd0 tx=0x7fbb84006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 msgr2=0x7fbb9419dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 --2- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb9419dc80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbb90011670 tx=0x7fbb90011b60 comp rx=0 tx=0).stop 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 shutdown_connections 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 --2- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fbb80038e80 0x7fbb8003b340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:26.402 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 --2- 172.21.15.49:0/357697436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb941063f0 0x7fbb9419dc80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:26.403 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 >> 172.21.15.49:0/357697436 conn(0x7fbb94101fd0 msgr2=0x7fbb94194860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:26.403 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 shutdown_connections 2024-03-18T00:40:26.403 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:25.745+0000 7fbb9bd58700 1 -- 172.21.15.49:0/357697436 wait complete. 2024-03-18T00:40:26.403 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling cephadm module... 2024-03-18T00:40:27.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:27 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3024046897' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T00:40:27.376 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:27 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: ignoring --setuser ceph since I am not root 2024-03-18T00:40:27.376 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:27 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: ignoring --setgroup ceph since I am not root 2024-03-18T00:40:28.098 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:27 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:27.671+0000 7f6cdae9f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T00:40:28.098 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:27 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:27.767+0000 7f6cdae9f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T00:40:28.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:28 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3024046897' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T00:40:28.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:28 smithi049 ceph-mon[32130]: mgrmap e5: x(active, since 10s) 2024-03-18T00:40:28.356 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.770+0000 7fe725c32700 1 Processor -- start 2024-03-18T00:40:28.356 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.770+0000 7fe725c32700 1 -- start start 2024-03-18T00:40:28.356 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.770+0000 7fe725c32700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72010a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:28.356 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.770+0000 7fe725c32700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe72007be10 con 0x7fe7201085e0 2024-03-18T00:40:28.356 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.770+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72010a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:28.356 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.770+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72010a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54928/0 (socket says 172.21.15.49:54928) 2024-03-18T00:40:28.356 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.771+0000 7fe71f7fe700 1 -- 172.21.15.49:0/4002961502 learned_addr learned my addr 172.21.15.49:0/4002961502 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.771+0000 7fe71f7fe700 1 -- 172.21.15.49:0/4002961502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72010af10 con 0x7fe7201085e0 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.771+0000 7fe71f7fe700 1 --2- 172.21.15.49:0/4002961502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72010a9d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe7100098a0 tx=0x7fe710009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618ae852470aca00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.771+0000 7fe71e7fc700 1 -- 172.21.15.49:0/4002961502 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe710004160 con 0x7fe7201085e0 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.771+0000 7fe71e7fc700 1 -- 172.21.15.49:0/4002961502 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe710004300 con 0x7fe7201085e0 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.771+0000 7fe71e7fc700 1 -- 172.21.15.49:0/4002961502 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe710011b50 con 0x7fe7201085e0 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.772+0000 7fe725c32700 1 -- 172.21.15.49:0/4002961502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 msgr2=0x7fe72010a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.772+0000 7fe725c32700 1 --2- 172.21.15.49:0/4002961502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72010a9d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe7100098a0 tx=0x7fe710009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.772+0000 7fe725c32700 1 -- 172.21.15.49:0/4002961502 shutdown_connections 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.772+0000 7fe725c32700 1 --2- 172.21.15.49:0/4002961502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72010a9d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.772+0000 7fe725c32700 1 -- 172.21.15.49:0/4002961502 >> 172.21.15.49:0/4002961502 conn(0x7fe720101fd0 msgr2=0x7fe7201043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.772+0000 7fe725c32700 1 -- 172.21.15.49:0/4002961502 shutdown_connections 2024-03-18T00:40:28.357 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.772+0000 7fe725c32700 1 -- 172.21.15.49:0/4002961502 wait complete. 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.772+0000 7fe725c32700 1 Processor -- start 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.773+0000 7fe725c32700 1 -- start start 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.773+0000 7fe725c32700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.773+0000 7fe725c32700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe72010bee0 con 0x7fe7201085e0 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.773+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.773+0000 7fe71f7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72019ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54940/0 (socket says 172.21.15.49:54940) 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.773+0000 7fe71f7fe700 1 -- 172.21.15.49:0/3024046897 learned_addr learned my addr 172.21.15.49:0/3024046897 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.773+0000 7fe71f7fe700 1 -- 172.21.15.49:0/3024046897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72019bf90 con 0x7fe7201085e0 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.773+0000 7fe71f7fe700 1 --2- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72019ba50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe710011670 tx=0x7fe710011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:28.358 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.774+0000 7fe71cff9700 1 -- 172.21.15.49:0/3024046897 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7100039c0 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.774+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72019c220 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.774+0000 7fe71cff9700 1 -- 172.21.15.49:0/3024046897 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe710011e50 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.774+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe72019ef10 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.774+0000 7fe71cff9700 1 -- 172.21.15.49:0/3024046897 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe71001a5c0 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.775+0000 7fe71cff9700 1 -- 172.21.15.49:0/3024046897 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fe71001a8c0 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.775+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe720063900 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.775+0000 7fe71cff9700 1 --2- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fe708038e80 0x7fe70803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.775+0000 7fe71cff9700 1 -- 172.21.15.49:0/3024046897 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe71004d2b0 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.775+0000 7fe71effd700 1 --2- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fe708038e80 0x7fe70803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.775+0000 7fe71effd700 1 --2- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fe708038e80 0x7fe70803b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe714006fd0 tx=0x7fe714006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:26.780+0000 7fe71cff9700 1 -- 172.21.15.49:0/3024046897 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe710019c70 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.041+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fe72019f0e0 con 0x7fe7201085e0 2024-03-18T00:40:28.359 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.093+0000 7fe71cff9700 1 -- 172.21.15.49:0/3024046897 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fe72019f0e0 con 0x7fe7201085e0 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.094+0000 7fe71cff9700 1 -- 172.21.15.49:0/3024046897 <== mon.0 v2:172.21.15.49:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fe71002d910 con 0x7fe7201085e0 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fe708038e80 msgr2=0x7fe70803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 --2- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fe708038e80 0x7fe70803b340 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe714006fd0 tx=0x7fe714006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 msgr2=0x7fe72019ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 --2- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72019ba50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe710011670 tx=0x7fe710011b60 comp rx=0 tx=0).stop 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 shutdown_connections 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 --2- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7fe708038e80 0x7fe70803b340 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 --2- 172.21.15.49:0/3024046897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7201085e0 0x7fe72019ba50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 >> 172.21.15.49:0/3024046897 conn(0x7fe720101fd0 msgr2=0x7fe720102a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 shutdown_connections 2024-03-18T00:40:28.360 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:27.096+0000 7fe725c32700 1 -- 172.21.15.49:0/3024046897 wait complete. 2024-03-18T00:40:28.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:28 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:28.440+0000 7f6cdae9f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T00:40:28.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:28 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:28.540+0000 7f6cdae9f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T00:40:28.969 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:28 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:28.689+0000 7f6cdae9f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T00:40:28.969 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:28 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:28.791+0000 7f6cdae9f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T00:40:28.969 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:28 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:28.878+0000 7f6cdae9f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T00:40:29.270 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:29 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2361005010' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T00:40:29.270 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:29 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:29.155+0000 7f6cdae9f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T00:40:29.270 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:29 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:29.269+0000 7f6cdae9f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T00:40:29.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:29 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:29.359+0000 7f6cdae9f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T00:40:29.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:29 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:29.458+0000 7f6cdae9f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T00:40:29.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-03-18T00:40:29.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-18T00:40:29.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T00:40:29.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T00:40:29.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T00:40:29.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-03-18T00:40:29.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.762+0000 7f94a7490700 1 Processor -- start 2024-03-18T00:40:29.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.763+0000 7f94a7490700 1 -- start start 2024-03-18T00:40:29.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.763+0000 7f94a7490700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a0106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.763+0000 7f94a7490700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f94a0107400 con 0x7f94a0106a10 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.763+0000 7f94a522c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a0106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.763+0000 7f94a522c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a0106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35684/0 (socket says 172.21.15.49:35684) 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.763+0000 7f94a522c700 1 -- 172.21.15.49:0/2998046066 learned_addr learned my addr 172.21.15.49:0/2998046066 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.763+0000 7f94a522c700 1 -- 172.21.15.49:0/2998046066 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a0107c10 con 0x7f94a0106a10 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.764+0000 7f94a522c700 1 --2- 172.21.15.49:0/2998046066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a0106e30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f949c00bb40 tx=0x7f949c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7406867860b44797 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.764+0000 7f9497fff700 1 -- 172.21.15.49:0/2998046066 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f949c00dbe0 con 0x7f94a0106a10 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.764+0000 7f9497fff700 1 -- 172.21.15.49:0/2998046066 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f949c0044f0 con 0x7f94a0106a10 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.764+0000 7f9497fff700 1 -- 172.21.15.49:0/2998046066 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f949c010470 con 0x7f94a0106a10 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.764+0000 7f94a7490700 1 -- 172.21.15.49:0/2998046066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 msgr2=0x7f94a0106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.764+0000 7f94a7490700 1 --2- 172.21.15.49:0/2998046066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a0106e30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f949c00bb40 tx=0x7f949c00be50 comp rx=0 tx=0).stop 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.764+0000 7f94a7490700 1 -- 172.21.15.49:0/2998046066 shutdown_connections 2024-03-18T00:40:29.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.764+0000 7f94a7490700 1 --2- 172.21.15.49:0/2998046066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a0106e30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.764+0000 7f94a7490700 1 -- 172.21.15.49:0/2998046066 >> 172.21.15.49:0/2998046066 conn(0x7f94a0101fd0 msgr2=0x7f94a01043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.765+0000 7f94a7490700 1 -- 172.21.15.49:0/2998046066 shutdown_connections 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.765+0000 7f94a7490700 1 -- 172.21.15.49:0/2998046066 wait complete. 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.765+0000 7f94a7490700 1 Processor -- start 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.765+0000 7f94a7490700 1 -- start start 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.765+0000 7f94a7490700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a01998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.765+0000 7f94a7490700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f94a007af90 con 0x7f94a0106a10 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.765+0000 7f94a522c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a01998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.765+0000 7f94a522c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a01998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35700/0 (socket says 172.21.15.49:35700) 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.765+0000 7f94a522c700 1 -- 172.21.15.49:0/2361005010 learned_addr learned my addr 172.21.15.49:0/2361005010 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.766+0000 7f94a522c700 1 -- 172.21.15.49:0/2361005010 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a0199df0 con 0x7f94a0106a10 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.766+0000 7f94a522c700 1 --2- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a01998b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f949c00b340 tx=0x7f949c0040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.766+0000 7f94967fc700 1 -- 172.21.15.49:0/2361005010 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f949c027070 con 0x7f94a0106a10 2024-03-18T00:40:29.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.766+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a019a080 con 0x7f94a0106a10 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.766+0000 7f94967fc700 1 -- 172.21.15.49:0/2361005010 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f949c010be0 con 0x7f94a0106a10 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.766+0000 7f94967fc700 1 -- 172.21.15.49:0/2361005010 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f949c022c40 con 0x7f94a0106a10 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.766+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a019cd70 con 0x7f94a0106a10 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.767+0000 7f94967fc700 1 -- 172.21.15.49:0/2361005010 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f949c02a070 con 0x7f94a0106a10 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.767+0000 7f94967fc700 1 --2- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f948c038ed0 0x7f948c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.767+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94a0063900 con 0x7f94a0106a10 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.767+0000 7f94967fc700 1 -- 172.21.15.49:0/2361005010 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f949c01f070 con 0x7f94a0106a10 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.767+0000 7f94a4a2b700 1 -- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f948c038ed0 msgr2=0x7f948c03b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1672783865 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.767+0000 7f94a4a2b700 1 --2- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f948c038ed0 0x7f948c03b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.772+0000 7f94967fc700 1 -- 172.21.15.49:0/2361005010 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f949c015070 con 0x7f94a0106a10 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.968+0000 7f94a4a2b700 1 -- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f948c038ed0 msgr2=0x7f948c03b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1672783865 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:28.968+0000 7f94a4a2b700 1 --2- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f948c038ed0 0x7f948c03b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.040+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f94a0192e10 con 0x7f94a0106a10 2024-03-18T00:40:29.674 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.041+0000 7f94967fc700 1 -- 172.21.15.49:0/2361005010 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f94a0192e10 con 0x7f94a0106a10 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f948c038ed0 msgr2=0x7f948c03b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 --2- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f948c038ed0 0x7f948c03b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 msgr2=0x7f94a01998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 --2- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a01998b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f949c00b340 tx=0x7f949c0040b0 comp rx=0 tx=0).stop 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 shutdown_connections 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 --2- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f948c038ed0 0x7f948c03b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 --2- 172.21.15.49:0/2361005010 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f94a0106a10 0x7f94a01998b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 >> 172.21.15.49:0/2361005010 conn(0x7f94a0101fd0 msgr2=0x7f94a018fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 shutdown_connections 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:29.044+0000 7f94a7490700 1 -- 172.21.15.49:0/2361005010 wait complete. 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2024-03-18T00:40:29.675 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 5... 2024-03-18T00:40:29.932 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:29 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:29.639+0000 7f6cdae9f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T00:40:31.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:30 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:30.667+0000 7f6cdae9f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T00:40:31.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:30 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:30.802+0000 7f6cdae9f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T00:40:31.441 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:31 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:31.146+0000 7f6cdae9f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T00:40:31.441 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:31 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:31.239+0000 7f6cdae9f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T00:40:31.442 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:31 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:31.441+0000 7f6cdae9f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:40:31.875 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:31 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:31.634+0000 7f6cdae9f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T00:40:32.199 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:31 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:31.932+0000 7f6cdae9f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:40:32.199 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:32 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:32.199+0000 7f6cdae9f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T00:40:32.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:32 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:32.446+0000 7f6cdae9f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T00:40:33.375 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:33 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:33.285+0000 7f6cdae9f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T00:40:34.204 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:33 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:33.877+0000 7f6cdae9f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T00:40:34.204 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:33 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:33.978+0000 7f6cdae9f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T00:40:34.204 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:34.204+0000 7f6cdae9f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: Active manager daemon x restarted 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: Activating manager daemon x 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: mgrmap e6: x(active, starting, since 0.0033343s) 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T00:40:34.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: Manager daemon x is now available 2024-03-18T00:40:34.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:34.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:34.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:40:34.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:34 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:40:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:35 smithi049 ceph-mon[32130]: Found migration_current of "None". Setting to last migration. 2024-03-18T00:40:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:35 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:40:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:35 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T00:40:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:35 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T00:40:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:35 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:35 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:35.535 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:35 smithi049 ceph-mon[32130]: mgrmap e7: x(active, since 1.00528s) 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.068+0000 7f4220b77700 1 Processor -- start 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.068+0000 7f4220b77700 1 -- start start 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.069+0000 7f4220b77700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.069+0000 7f4220b77700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f421c107400 con 0x7f421c106a10 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.069+0000 7f421a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.069+0000 7f421a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35706/0 (socket says 172.21.15.49:35706) 2024-03-18T00:40:35.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.069+0000 7f421a59c700 1 -- 172.21.15.49:0/2978557229 learned_addr learned my addr 172.21.15.49:0/2978557229 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.069+0000 7f421a59c700 1 -- 172.21.15.49:0/2978557229 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c107c10 con 0x7f421c106a10 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.069+0000 7f421a59c700 1 --2- 172.21.15.49:0/2978557229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c106e30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f420c00d180 tx=0x7f420c00d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25908079cdbf364 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.070+0000 7f421959a700 1 -- 172.21.15.49:0/2978557229 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f420c019070 con 0x7f421c106a10 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.070+0000 7f421959a700 1 -- 172.21.15.49:0/2978557229 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f420c00de20 con 0x7f421c106a10 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.070+0000 7f421959a700 1 -- 172.21.15.49:0/2978557229 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f420c009b90 con 0x7f421c106a10 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.070+0000 7f4220b77700 1 -- 172.21.15.49:0/2978557229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 msgr2=0x7f421c106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.070+0000 7f4220b77700 1 --2- 172.21.15.49:0/2978557229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c106e30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f420c00d180 tx=0x7f420c00d490 comp rx=0 tx=0).stop 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.070+0000 7f4220b77700 1 -- 172.21.15.49:0/2978557229 shutdown_connections 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.070+0000 7f4220b77700 1 --2- 172.21.15.49:0/2978557229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c106e30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.070+0000 7f4220b77700 1 -- 172.21.15.49:0/2978557229 >> 172.21.15.49:0/2978557229 conn(0x7f421c101fd0 msgr2=0x7f421c1043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:35.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.071+0000 7f4220b77700 1 -- 172.21.15.49:0/2978557229 shutdown_connections 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.071+0000 7f4220b77700 1 -- 172.21.15.49:0/2978557229 wait complete. 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.071+0000 7f4220b77700 1 Processor -- start 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.071+0000 7f4220b77700 1 -- start start 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f4220b77700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c19dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f4220b77700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f421c07af90 con 0x7f421c106a10 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f421a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c19dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f421a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c19dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35710/0 (socket says 172.21.15.49:35710) 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f421a59c700 1 -- 172.21.15.49:0/250001148 learned_addr learned my addr 172.21.15.49:0/250001148 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f421a59c700 1 -- 172.21.15.49:0/250001148 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f421c19e1a0 con 0x7f421c106a10 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f421a59c700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c19dc60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f420c009370 tx=0x7f420c009450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f420c019040 con 0x7f421c106a10 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f420c003d60 con 0x7f421c106a10 2024-03-18T00:40:35.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.072+0000 7f4220b77700 1 -- 172.21.15.49:0/250001148 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f421c19e430 con 0x7f421c106a10 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.073+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f420c01f820 con 0x7f421c106a10 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.073+0000 7f4220b77700 1 -- 172.21.15.49:0/250001148 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f421c1a1120 con 0x7f421c106a10 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.073+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f420c01fa90 con 0x7f421c106a10 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.074+0000 7f42137fe700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 0x7f420803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.074+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 --> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f420803baf0 con 0x7f4208038f20 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.074+0000 7f4219d9b700 1 -- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 msgr2=0x7f420803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1672783865 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.074+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f420c011050 con 0x7f421c106a10 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.074+0000 7f4219d9b700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 0x7f420803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.275+0000 7f4219d9b700 1 -- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 msgr2=0x7f420803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1672783865 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.275+0000 7f4219d9b700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 0x7f420803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.676+0000 7f4219d9b700 1 -- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 msgr2=0x7f420803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1672783865 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:30.676+0000 7f4219d9b700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 0x7f420803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T00:40:35.840 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:31.477+0000 7f4219d9b700 1 -- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 msgr2=0x7f420803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1672783865 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:31.477+0000 7f4219d9b700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 0x7f420803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:33.079+0000 7f4219d9b700 1 -- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 msgr2=0x7f420803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1672783865 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:33.079+0000 7f4219d9b700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 0x7f420803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:34.219+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7f420c014950 con 0x7f421c106a10 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:34.219+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 msgr2=0x7f420803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:34.219+0000 7f42137fe700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 0x7f420803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.220+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f420c04f120 con 0x7f421c106a10 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.220+0000 7f42137fe700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f420803c810 0x7f420803ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.220+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f420c00aea0 con 0x7f420803c810 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.220+0000 7f4219d9b700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f420803c810 0x7f420803ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.221+0000 7f4219d9b700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f420803c810 0x7f420803ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4204003a10 tx=0x7f42040092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.222+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 <== mgr.14118 v2:172.21.15.49:6800/1899027445 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f420c00aea0 con 0x7f420803c810 2024-03-18T00:40:35.841 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.228+0000 7f4220b77700 1 -- 172.21.15.49:0/250001148 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f421c063900 con 0x7f420803c810 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.228+0000 7f42137fe700 1 -- 172.21.15.49:0/250001148 <== mgr.14118 v2:172.21.15.49:6800/1899027445 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f421c063900 con 0x7f420803c810 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 -- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f420803c810 msgr2=0x7f420803ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f420803c810 0x7f420803ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4204003a10 tx=0x7f42040092b0 comp rx=0 tx=0).stop 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 -- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 msgr2=0x7f421c19dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c19dc60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f420c009370 tx=0x7f420c009450 comp rx=0 tx=0).stop 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 -- 172.21.15.49:0/250001148 shutdown_connections 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f420803c810 0x7f420803ec00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f421c106a10 0x7f421c19dc60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 --2- 172.21.15.49:0/250001148 >> [v2:172.21.15.49:6800/1672783865,v1:172.21.15.49:6801/1672783865] conn(0x7f4208038f20 0x7f420803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:35.842 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 -- 172.21.15.49:0/250001148 >> 172.21.15.49:0/250001148 conn(0x7f421c101fd0 msgr2=0x7f421c18fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 -- 172.21.15.49:0/250001148 shutdown_connections 2024-03-18T00:40:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:35.229+0000 7f4220b77700 1 -- 172.21.15.49:0/250001148 wait complete. 2024-03-18T00:40:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 5 is available 2024-03-18T00:40:35.843 INFO:teuthology.orchestra.run.smithi049.stdout:Setting orchestrator backend to cephadm... 2024-03-18T00:40:36.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:36 smithi049 ceph-mon[32130]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T00:40:36.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:36 smithi049 ceph-mon[32130]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T00:40:36.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:36 smithi049 ceph-mon[32130]: [18/Mar/2024:00:40:35] ENGINE Bus STARTING 2024-03-18T00:40:36.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:36 smithi049 ceph-mon[32130]: [18/Mar/2024:00:40:35] ENGINE Serving on http://172.21.15.49:8765 2024-03-18T00:40:36.492 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:36 smithi049 ceph-mon[32130]: [18/Mar/2024:00:40:35] ENGINE Serving on https://172.21.15.49:7150 2024-03-18T00:40:36.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:36 smithi049 ceph-mon[32130]: [18/Mar/2024:00:40:35] ENGINE Bus STARTED 2024-03-18T00:40:36.493 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:36 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:40:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.259+0000 7f7e73c3a700 1 Processor -- start 2024-03-18T00:40:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.259+0000 7f7e73c3a700 1 -- start start 2024-03-18T00:40:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.259+0000 7f7e73c3a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c10a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.259+0000 7f7e73c3a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c07be50 con 0x7f7e6c1085a0 2024-03-18T00:40:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.259+0000 7f7e719d6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c10a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:37.067 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.259+0000 7f7e719d6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c10a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35786/0 (socket says 172.21.15.49:35786) 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.259+0000 7f7e719d6700 1 -- 172.21.15.49:0/3095592326 learned_addr learned my addr 172.21.15.49:0/3095592326 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.260+0000 7f7e719d6700 1 -- 172.21.15.49:0/3095592326 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c10aed0 con 0x7f7e6c1085a0 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.260+0000 7f7e719d6700 1 --2- 172.21.15.49:0/3095592326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c10a990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7e680098a0 tx=0x7f7e68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8e805a0adf5ce66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.260+0000 7f7e709d4700 1 -- 172.21.15.49:0/3095592326 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e68004160 con 0x7f7e6c1085a0 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.260+0000 7f7e709d4700 1 -- 172.21.15.49:0/3095592326 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7e68004300 con 0x7f7e6c1085a0 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.260+0000 7f7e709d4700 1 -- 172.21.15.49:0/3095592326 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e68011b50 con 0x7f7e6c1085a0 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.261+0000 7f7e73c3a700 1 -- 172.21.15.49:0/3095592326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 msgr2=0x7f7e6c10a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.261+0000 7f7e73c3a700 1 --2- 172.21.15.49:0/3095592326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c10a990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7e680098a0 tx=0x7f7e68009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.261+0000 7f7e73c3a700 1 -- 172.21.15.49:0/3095592326 shutdown_connections 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.261+0000 7f7e73c3a700 1 --2- 172.21.15.49:0/3095592326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c10a990 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.261+0000 7f7e73c3a700 1 -- 172.21.15.49:0/3095592326 >> 172.21.15.49:0/3095592326 conn(0x7f7e6c101fd0 msgr2=0x7f7e6c1043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:37.068 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.261+0000 7f7e73c3a700 1 -- 172.21.15.49:0/3095592326 shutdown_connections 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.261+0000 7f7e73c3a700 1 -- 172.21.15.49:0/3095592326 wait complete. 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.261+0000 7f7e73c3a700 1 Processor -- start 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.261+0000 7f7e73c3a700 1 -- start start 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e73c3a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c107e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e73c3a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c10be80 con 0x7f7e6c1085a0 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e719d6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e719d6700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35796/0 (socket says 172.21.15.49:35796) 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e719d6700 1 -- 172.21.15.49:0/2777185607 learned_addr learned my addr 172.21.15.49:0/2777185607 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e719d6700 1 -- 172.21.15.49:0/2777185607 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c108360 con 0x7f7e6c1085a0 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e719d6700 1 --2- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c107e20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7e68011670 tx=0x7f7e68011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e62ffd700 1 -- 172.21.15.49:0/2777185607 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e680039c0 con 0x7f7e6c1085a0 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e62ffd700 1 -- 172.21.15.49:0/2777185607 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7e68011e50 con 0x7f7e6c1085a0 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6c106550 con 0x7f7e6c1085a0 2024-03-18T00:40:37.069 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e62ffd700 1 -- 172.21.15.49:0/2777185607 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e6801a5c0 con 0x7f7e6c1085a0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.262+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e6c106a90 con 0x7f7e6c1085a0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.263+0000 7f7e62ffd700 1 -- 172.21.15.49:0/2777185607 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7f7e6801a8c0 con 0x7f7e6c1085a0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.263+0000 7f7e62ffd700 1 --2- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7e58038dc0 0x7f7e5803b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.263+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e6c063900 con 0x7f7e6c1085a0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.264+0000 7f7e62ffd700 1 -- 172.21.15.49:0/2777185607 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7e6804d040 con 0x7f7e6c1085a0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.264+0000 7f7e711d5700 1 --2- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7e58038dc0 0x7f7e5803b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.264+0000 7f7e711d5700 1 --2- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7e58038dc0 0x7f7e5803b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c006fd0 tx=0x7f7e5c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.269+0000 7f7e62ffd700 1 -- 172.21.15.49:0/2777185607 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7e68019c70 con 0x7f7e6c1085a0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.271+0000 7f7e62ffd700 1 -- 172.21.15.49:0/2777185607 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f7e6804bc50 con 0x7f7e6c1085a0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.492+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f7e6c1031b0 con 0x7f7e58038dc0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.498+0000 7f7e62ffd700 1 -- 172.21.15.49:0/2777185607 <== mgr.14118 v2:172.21.15.49:6800/1899027445 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7e6c1031b0 con 0x7f7e58038dc0 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.500+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7e58038dc0 msgr2=0x7f7e5803b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.500+0000 7f7e73c3a700 1 --2- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7e58038dc0 0x7f7e5803b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c006fd0 tx=0x7f7e5c006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:37.070 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.500+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 msgr2=0x7f7e6c107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.500+0000 7f7e73c3a700 1 --2- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c107e20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7e68011670 tx=0x7f7e68011b60 comp rx=0 tx=0).stop 2024-03-18T00:40:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.500+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 shutdown_connections 2024-03-18T00:40:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.500+0000 7f7e73c3a700 1 --2- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7e58038dc0 0x7f7e5803b280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.500+0000 7f7e73c3a700 1 --2- 172.21.15.49:0/2777185607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7e6c1085a0 0x7f7e6c107e20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.500+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 >> 172.21.15.49:0/2777185607 conn(0x7f7e6c101fd0 msgr2=0x7f7e6c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.501+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 shutdown_connections 2024-03-18T00:40:37.071 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:36.501+0000 7f7e73c3a700 1 -- 172.21.15.49:0/2777185607 wait complete. 2024-03-18T00:40:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:37 smithi049 ceph-mon[32130]: mgrmap e8: x(active, since 2s) 2024-03-18T00:40:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:37 smithi049 ceph-mon[32130]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:40:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:37 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:37 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:40:38.232 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.503+0000 7fa25cb19700 1 Processor -- start 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.503+0000 7fa25cb19700 1 -- start start 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.503+0000 7fa25cb19700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25807c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.503+0000 7fa25cb19700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa25807c890 con 0x7fa25807de60 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.504+0000 7fa25659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25807c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.504+0000 7fa25659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25807c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35802/0 (socket says 172.21.15.49:35802) 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.504+0000 7fa25659c700 1 -- 172.21.15.49:0/1406865232 learned_addr learned my addr 172.21.15.49:0/1406865232 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.504+0000 7fa25659c700 1 -- 172.21.15.49:0/1406865232 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25807ca10 con 0x7fa25807de60 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.504+0000 7fa25659c700 1 --2- 172.21.15.49:0/1406865232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25807c2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa250009e80 tx=0x7fa25000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24693ac1df7ca914 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.504+0000 7fa25559a700 1 -- 172.21.15.49:0/1406865232 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa250017070 con 0x7fa25807de60 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.504+0000 7fa25559a700 1 -- 172.21.15.49:0/1406865232 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa25000b8c0 con 0x7fa25807de60 2024-03-18T00:40:38.233 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.504+0000 7fa25559a700 1 -- 172.21.15.49:0/1406865232 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa2500045c0 con 0x7fa25807de60 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.505+0000 7fa25cb19700 1 -- 172.21.15.49:0/1406865232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 msgr2=0x7fa25807c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.505+0000 7fa25cb19700 1 --2- 172.21.15.49:0/1406865232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25807c2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa250009e80 tx=0x7fa25000b330 comp rx=0 tx=0).stop 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.505+0000 7fa25cb19700 1 -- 172.21.15.49:0/1406865232 shutdown_connections 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.505+0000 7fa25cb19700 1 --2- 172.21.15.49:0/1406865232 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25807c2c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.505+0000 7fa25cb19700 1 -- 172.21.15.49:0/1406865232 >> 172.21.15.49:0/1406865232 conn(0x7fa258103140 msgr2=0x7fa258105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.505+0000 7fa25cb19700 1 -- 172.21.15.49:0/1406865232 shutdown_connections 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.505+0000 7fa25cb19700 1 -- 172.21.15.49:0/1406865232 wait complete. 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.506+0000 7fa25cb19700 1 Processor -- start 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.506+0000 7fa25cb19700 1 -- start start 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.506+0000 7fa25cb19700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.506+0000 7fa25cb19700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa25810e150 con 0x7fa25807de60 2024-03-18T00:40:38.234 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.506+0000 7fa25659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.506+0000 7fa25659c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:35808/0 (socket says 172.21.15.49:35808) 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.506+0000 7fa25659c700 1 -- 172.21.15.49:0/3446988737 learned_addr learned my addr 172.21.15.49:0/3446988737 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.506+0000 7fa25659c700 1 -- 172.21.15.49:0/3446988737 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa25819e250 con 0x7fa25807de60 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.507+0000 7fa25659c700 1 --2- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25819dd10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa250004210 tx=0x7fa25000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.507+0000 7fa24f7fe700 1 -- 172.21.15.49:0/3446988737 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa250020020 con 0x7fa25807de60 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.507+0000 7fa24f7fe700 1 -- 172.21.15.49:0/3446988737 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa250003b50 con 0x7fa25807de60 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.507+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa25819e4e0 con 0x7fa25807de60 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.507+0000 7fa24f7fe700 1 -- 172.21.15.49:0/3446988737 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa250017040 con 0x7fa25807de60 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.507+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2581a11c0 con 0x7fa25807de60 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.508+0000 7fa24f7fe700 1 -- 172.21.15.49:0/3446988737 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fa250023020 con 0x7fa25807de60 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.508+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa258063900 con 0x7fa25807de60 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.508+0000 7fa24f7fe700 1 --2- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fa244038f20 0x7fa24403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:38.235 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.508+0000 7fa24f7fe700 1 -- 172.21.15.49:0/3446988737 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa25000f050 con 0x7fa25807de60 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.508+0000 7fa255d9b700 1 --2- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fa244038f20 0x7fa24403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.508+0000 7fa255d9b700 1 --2- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fa244038f20 0x7fa24403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa240006fd0 tx=0x7fa240006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.513+0000 7fa24f7fe700 1 -- 172.21.15.49:0/3446988737 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa25002a020 con 0x7fa25807de60 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.718+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fa258104380 con 0x7fa244038f20 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.719+0000 7fa24f7fe700 1 -- 172.21.15.49:0/3446988737 <== mgr.14118 v2:172.21.15.49:6800/1899027445 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fa258104380 con 0x7fa244038f20 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fa244038f20 msgr2=0x7fa24403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 --2- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fa244038f20 0x7fa24403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa240006fd0 tx=0x7fa240006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 msgr2=0x7fa25819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 --2- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25819dd10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa250004210 tx=0x7fa25000bd50 comp rx=0 tx=0).stop 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 shutdown_connections 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 --2- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fa244038f20 0x7fa24403b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 --2- 172.21.15.49:0/3446988737 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa25807de60 0x7fa25819dd10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 >> 172.21.15.49:0/3446988737 conn(0x7fa258103140 msgr2=0x7fa258103c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:38.236 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 shutdown_connections 2024-03-18T00:40:38.237 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:37.721+0000 7fa25cb19700 1 -- 172.21.15.49:0/3446988737 wait complete. 2024-03-18T00:40:38.237 INFO:teuthology.orchestra.run.smithi049.stdout:Generating ssh key... 2024-03-18T00:40:38.477 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-mon[32130]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:40:39.355 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: Generating public/private rsa key pair. 2024-03-18T00:40:39.355 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: Your identification has been saved in /tmp/tmpxlzhhdny/key. 2024-03-18T00:40:39.355 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: Your public key has been saved in /tmp/tmpxlzhhdny/key.pub. 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: The key fingerprint is: 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: SHA256:zvDVa2n2rCOwhENdjsUIlo1brNzPY/ETQlv+Z3e8UjY ceph-ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: The key's randomart image is: 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: +---[RSA 3072]----+ 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: | o* o | 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: | .o = = . | 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: | . * * + | 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: | = + =.o | 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: | ...So.+.o . | 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: | o=o.= oo.E*| 2024-03-18T00:40:39.356 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: | o++ .*.oo=| 2024-03-18T00:40:39.357 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: | . .+.+ . | 2024-03-18T00:40:39.357 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: | ..o+ | 2024-03-18T00:40:39.357 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: +----[SHA256]-----+ 2024-03-18T00:40:39.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.700+0000 7f88bf5da700 1 Processor -- start 2024-03-18T00:40:39.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.700+0000 7f88bf5da700 1 -- start start 2024-03-18T00:40:39.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.700+0000 7f88bf5da700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b81067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:39.622 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.700+0000 7f88bf5da700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f88b807be10 con 0x7f88b81063d0 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.700+0000 7f88bd376700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b81067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.700+0000 7f88bd376700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b81067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44110/0 (socket says 172.21.15.49:44110) 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.700+0000 7f88bd376700 1 -- 172.21.15.49:0/3375480775 learned_addr learned my addr 172.21.15.49:0/3375480775 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88bd376700 1 -- 172.21.15.49:0/3375480775 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b8106d30 con 0x7f88b81063d0 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88bd376700 1 --2- 172.21.15.49:0/3375480775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b81067f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88b40098a0 tx=0x7f88b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc102f66a51bc1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88affff700 1 -- 172.21.15.49:0/3375480775 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b4004160 con 0x7f88b81063d0 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88affff700 1 -- 172.21.15.49:0/3375480775 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f88b4004300 con 0x7f88b81063d0 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88affff700 1 -- 172.21.15.49:0/3375480775 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b4011b50 con 0x7f88b81063d0 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88bf5da700 1 -- 172.21.15.49:0/3375480775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 msgr2=0x7f88b81067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88bf5da700 1 --2- 172.21.15.49:0/3375480775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b81067f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f88b40098a0 tx=0x7f88b4009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88bf5da700 1 -- 172.21.15.49:0/3375480775 shutdown_connections 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88bf5da700 1 --2- 172.21.15.49:0/3375480775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b81067f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.701+0000 7f88bf5da700 1 -- 172.21.15.49:0/3375480775 >> 172.21.15.49:0/3375480775 conn(0x7f88b8102030 msgr2=0x7f88b8104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:39.623 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.702+0000 7f88bf5da700 1 -- 172.21.15.49:0/3375480775 shutdown_connections 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.702+0000 7f88bf5da700 1 -- 172.21.15.49:0/3375480775 wait complete. 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.702+0000 7f88bf5da700 1 Processor -- start 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.702+0000 7f88bf5da700 1 -- start start 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.702+0000 7f88bf5da700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b819dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.702+0000 7f88bf5da700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f88b8107dc0 con 0x7f88b81063d0 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88bd376700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b819dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88bd376700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b819dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44116/0 (socket says 172.21.15.49:44116) 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88bd376700 1 -- 172.21.15.49:0/1514549989 learned_addr learned my addr 172.21.15.49:0/1514549989 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88bd376700 1 -- 172.21.15.49:0/1514549989 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88b819e200 con 0x7f88b81063d0 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88bd376700 1 --2- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b819dcc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009fb0 tx=0x7f88b4011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88ae7fc700 1 -- 172.21.15.49:0/1514549989 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b40038b0 con 0x7f88b81063d0 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88ae7fc700 1 -- 172.21.15.49:0/1514549989 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f88b4011e50 con 0x7f88b81063d0 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88b819e490 con 0x7f88b81063d0 2024-03-18T00:40:39.624 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88ae7fc700 1 -- 172.21.15.49:0/1514549989 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f88b4019420 con 0x7f88b81063d0 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.703+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88b81a1180 con 0x7f88b81063d0 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.704+0000 7f88ae7fc700 1 -- 172.21.15.49:0/1514549989 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f88b40196d0 con 0x7f88b81063d0 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.704+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88b8063900 con 0x7f88b81063d0 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.704+0000 7f88ae7fc700 1 --2- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f88a4038f20 0x7f88a403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.704+0000 7f88ae7fc700 1 -- 172.21.15.49:0/1514549989 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f88b404d340 con 0x7f88b81063d0 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.704+0000 7f88bcb75700 1 --2- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f88a4038f20 0x7f88a403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.705+0000 7f88bcb75700 1 --2- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f88a4038f20 0x7f88a403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f88a8006fd0 tx=0x7f88a8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.709+0000 7f88ae7fc700 1 -- 172.21.15.49:0/1514549989 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f88b400bc70 con 0x7f88b81063d0 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.917+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f88b80008d0 con 0x7f88a4038f20 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.982+0000 7f88ae7fc700 1 -- 172.21.15.49:0/1514549989 <== mgr.14118 v2:172.21.15.49:6800/1899027445 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f88b80008d0 con 0x7f88a4038f20 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f88a4038f20 msgr2=0x7f88a403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 --2- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f88a4038f20 0x7f88a403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f88a8006fd0 tx=0x7f88a8006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 msgr2=0x7f88b819dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 --2- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b819dcc0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f88b4009fb0 tx=0x7f88b4011bf0 comp rx=0 tx=0).stop 2024-03-18T00:40:39.625 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 shutdown_connections 2024-03-18T00:40:39.626 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 --2- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f88a4038f20 0x7f88a403b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:39.626 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 --2- 172.21.15.49:0/1514549989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f88b81063d0 0x7f88b819dcc0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:39.626 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 >> 172.21.15.49:0/1514549989 conn(0x7f88b8102030 msgr2=0x7f88b8194930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:39.626 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 shutdown_connections 2024-03-18T00:40:39.626 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:38.984+0000 7f88bf5da700 1 -- 172.21.15.49:0/1514549989 wait complete. 2024-03-18T00:40:40.266 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:39 smithi049 ceph-mon[32130]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:40:40.266 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:39 smithi049 ceph-mon[32130]: Generating ssh key... 2024-03-18T00:40:40.266 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:39 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:40.266 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:39 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwjYgkn9eaHBeNkCJ09UAkoQ+HAGO0WMi5hAtHqQ8PxgVCPYs20fljjIHw32TDMGR617jNqRBwyQz/c9ih1wwAM2/rHmPxo/H6awBVqsAxTq4CQCNWuaFsFkaoRNs5VdXDLpn46LyIIQD3oaxVlNP6x4Mx2K4Kr1T4SWYRWx5RGe1hbjBi+pSLLtG6JqOkG7owKiYUmq4oAFjAjh2NPGqdHLXQGXhSPC/lAFG1oRTD4oOgYRhgBPhIP1aVANJLMn3zTulLS7W3Qo6kDZu8j/oedvHpvYvecPuta4nO5AGwq/C7OBp9yvFbj7w8+jTtfr8HgxxxYOxiyX9j0XbBuUx0+3SOtH8XBt7qUmWfUbrCXPxgeC9PW2sU394Udotzysheto+MuLvajDu1bL/GAILq8j6cxOb0vlp3ubjFf8K74f3Yk9apm/9KU52ElsuS2CqVPHkaTew259lmnz7LhjJMi3Jq4ajF33tniMHytsknnfKMuemEBnuuJzg1IO98hZM= ceph-ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.050+0000 7fdf04c5d700 1 Processor -- start 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.050+0000 7fdf04c5d700 1 -- start start 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.050+0000 7fdf04c5d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0007c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.050+0000 7fdf04c5d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0007c890 con 0x7fdf0007de60 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.050+0000 7fdefe59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0007c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.050+0000 7fdefe59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0007c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44130/0 (socket says 172.21.15.49:44130) 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.050+0000 7fdefe59c700 1 -- 172.21.15.49:0/960129408 learned_addr learned my addr 172.21.15.49:0/960129408 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.051+0000 7fdefe59c700 1 -- 172.21.15.49:0/960129408 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0007ca10 con 0x7fdf0007de60 2024-03-18T00:40:40.862 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.051+0000 7fdefe59c700 1 --2- 172.21.15.49:0/960129408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0007c2c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdef00098a0 tx=0x7fdef0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9a38160d4b2fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.051+0000 7fdefd59a700 1 -- 172.21.15.49:0/960129408 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdef0004160 con 0x7fdf0007de60 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.051+0000 7fdefd59a700 1 -- 172.21.15.49:0/960129408 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdef0004300 con 0x7fdf0007de60 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.051+0000 7fdefd59a700 1 -- 172.21.15.49:0/960129408 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdef0011b50 con 0x7fdf0007de60 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.052+0000 7fdf04c5d700 1 -- 172.21.15.49:0/960129408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 msgr2=0x7fdf0007c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.052+0000 7fdf04c5d700 1 --2- 172.21.15.49:0/960129408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0007c2c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdef00098a0 tx=0x7fdef0009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.052+0000 7fdf04c5d700 1 -- 172.21.15.49:0/960129408 shutdown_connections 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.052+0000 7fdf04c5d700 1 --2- 172.21.15.49:0/960129408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0007c2c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.052+0000 7fdf04c5d700 1 -- 172.21.15.49:0/960129408 >> 172.21.15.49:0/960129408 conn(0x7fdf00103140 msgr2=0x7fdf00105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.052+0000 7fdf04c5d700 1 -- 172.21.15.49:0/960129408 shutdown_connections 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.052+0000 7fdf04c5d700 1 -- 172.21.15.49:0/960129408 wait complete. 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.052+0000 7fdf04c5d700 1 Processor -- start 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdf04c5d700 1 -- start start 2024-03-18T00:40:40.863 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdf04c5d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdf04c5d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdf0010e150 con 0x7fdf0007de60 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdefe59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdefe59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0019dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44136/0 (socket says 172.21.15.49:44136) 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdefe59c700 1 -- 172.21.15.49:0/2463452753 learned_addr learned my addr 172.21.15.49:0/2463452753 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdefe59c700 1 -- 172.21.15.49:0/2463452753 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf0019e2a0 con 0x7fdf0007de60 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdefe59c700 1 --2- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0019dd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdef0011850 tx=0x7fdef0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdef77fe700 1 -- 172.21.15.49:0/2463452753 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdef0003af0 con 0x7fdf0007de60 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.053+0000 7fdef77fe700 1 -- 172.21.15.49:0/2463452753 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fdef0020070 con 0x7fdf0007de60 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.054+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf0019e530 con 0x7fdf0007de60 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.054+0000 7fdef77fe700 1 -- 172.21.15.49:0/2463452753 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdef00195a0 con 0x7fdf0007de60 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.054+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf001a1220 con 0x7fdf0007de60 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.055+0000 7fdef77fe700 1 -- 172.21.15.49:0/2463452753 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fdef0019850 con 0x7fdf0007de60 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.055+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf00063900 con 0x7fdf0007de60 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.055+0000 7fdef77fe700 1 --2- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fdeec038f70 0x7fdeec03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:40.864 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.055+0000 7fdef77fe700 1 -- 172.21.15.49:0/2463452753 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdef0011e00 con 0x7fdf0007de60 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.055+0000 7fdefdd9b700 1 --2- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fdeec038f70 0x7fdeec03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.055+0000 7fdefdd9b700 1 --2- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fdeec038f70 0x7fdeec03b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006fd0 tx=0x7fdee8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.060+0000 7fdef77fe700 1 -- 172.21.15.49:0/2463452753 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdef0003c90 con 0x7fdf0007de60 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.265+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fdf001043d0 con 0x7fdeec038f70 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.266+0000 7fdef77fe700 1 -- 172.21.15.49:0/2463452753 <== mgr.14118 v2:172.21.15.49:6800/1899027445 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fdf001043d0 con 0x7fdeec038f70 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.268+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fdeec038f70 msgr2=0x7fdeec03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.268+0000 7fdf04c5d700 1 --2- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fdeec038f70 0x7fdeec03b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdee8006fd0 tx=0x7fdee8006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.268+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 msgr2=0x7fdf0019dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.268+0000 7fdf04c5d700 1 --2- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0019dd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdef0011850 tx=0x7fdef0011c10 comp rx=0 tx=0).stop 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.269+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 shutdown_connections 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.269+0000 7fdf04c5d700 1 --2- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fdeec038f70 0x7fdeec03b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.269+0000 7fdf04c5d700 1 --2- 172.21.15.49:0/2463452753 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf0007de60 0x7fdf0019dd60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.269+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 >> 172.21.15.49:0/2463452753 conn(0x7fdf00103140 msgr2=0x7fdf00103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:40.865 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.269+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 shutdown_connections 2024-03-18T00:40:40.866 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:40.269+0000 7fdf04c5d700 1 -- 172.21.15.49:0/2463452753 wait complete. 2024-03-18T00:40:40.866 INFO:teuthology.orchestra.run.smithi049.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-18T00:40:40.866 INFO:teuthology.orchestra.run.smithi049.stdout:Adding key to root@localhost authorized_keys... 2024-03-18T00:40:40.866 INFO:teuthology.orchestra.run.smithi049.stdout:Adding host smithi049... 2024-03-18T00:40:41.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:41 smithi049 ceph-mon[32130]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:40:42.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:42 smithi049 ceph-mon[32130]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi049", "addr": "172.21.15.49", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:40:43.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:43 smithi049 ceph-mon[32130]: Deploying cephadm binary to smithi049 2024-03-18T00:40:46.627 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Added host 'smithi049' with addr '172.21.15.49' 2024-03-18T00:40:46.627 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.257+0000 7f1296e6a700 1 Processor -- start 2024-03-18T00:40:46.627 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.257+0000 7f1296e6a700 1 -- start start 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.257+0000 7f1296e6a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129007c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.257+0000 7f1296e6a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f129007c8b0 con 0x7f129007de80 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.257+0000 7f1294c06700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129007c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.257+0000 7f1294c06700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129007c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44148/0 (socket says 172.21.15.49:44148) 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.257+0000 7f1294c06700 1 -- 172.21.15.49:0/3799034297 learned_addr learned my addr 172.21.15.49:0/3799034297 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.258+0000 7f1294c06700 1 -- 172.21.15.49:0/3799034297 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129007ca30 con 0x7f129007de80 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.258+0000 7f1294c06700 1 --2- 172.21.15.49:0/3799034297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129007c2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f12840098a0 tx=0x7f1284009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09002dcfb1b514e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.258+0000 7f128f7fe700 1 -- 172.21.15.49:0/3799034297 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1284004160 con 0x7f129007de80 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.258+0000 7f128f7fe700 1 -- 172.21.15.49:0/3799034297 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1284004300 con 0x7f129007de80 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.258+0000 7f128f7fe700 1 -- 172.21.15.49:0/3799034297 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1284011b50 con 0x7f129007de80 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.259+0000 7f1296e6a700 1 -- 172.21.15.49:0/3799034297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 msgr2=0x7f129007c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.259+0000 7f1296e6a700 1 --2- 172.21.15.49:0/3799034297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129007c2e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f12840098a0 tx=0x7f1284009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:46.628 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.259+0000 7f1296e6a700 1 -- 172.21.15.49:0/3799034297 shutdown_connections 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.259+0000 7f1296e6a700 1 --2- 172.21.15.49:0/3799034297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129007c2e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.259+0000 7f1296e6a700 1 -- 172.21.15.49:0/3799034297 >> 172.21.15.49:0/3799034297 conn(0x7f1290103140 msgr2=0x7f1290105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.259+0000 7f1296e6a700 1 -- 172.21.15.49:0/3799034297 shutdown_connections 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.259+0000 7f1296e6a700 1 -- 172.21.15.49:0/3799034297 wait complete. 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.259+0000 7f1296e6a700 1 Processor -- start 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f1296e6a700 1 -- start start 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f1296e6a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129019dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f1296e6a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f129010e130 con 0x7f129007de80 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f1294c06700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129019dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f1294c06700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129019dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44160/0 (socket says 172.21.15.49:44160) 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f1294c06700 1 -- 172.21.15.49:0/3540063843 learned_addr learned my addr 172.21.15.49:0/3540063843 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f1294c06700 1 -- 172.21.15.49:0/3540063843 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f129019e280 con 0x7f129007de80 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f1294c06700 1 --2- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129019dd40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1284011850 tx=0x7f1284011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:46.629 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f128dffb700 1 -- 172.21.15.49:0/3540063843 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1284003af0 con 0x7f129007de80 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.260+0000 7f128dffb700 1 -- 172.21.15.49:0/3540063843 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1284020070 con 0x7f129007de80 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.261+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f129019e510 con 0x7f129007de80 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.261+0000 7f128dffb700 1 -- 172.21.15.49:0/3540063843 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12840195a0 con 0x7f129007de80 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.261+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12901a1200 con 0x7f129007de80 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.262+0000 7f128dffb700 1 -- 172.21.15.49:0/3540063843 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f1284019850 con 0x7f129007de80 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.262+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1290063900 con 0x7f129007de80 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.262+0000 7f128dffb700 1 --2- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f1280038f70 0x7f128003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.262+0000 7f128dffb700 1 -- 172.21.15.49:0/3540063843 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1284011e00 con 0x7f129007de80 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.262+0000 7f128ffff700 1 --2- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f1280038f70 0x7f128003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.263+0000 7f128ffff700 1 --2- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f1280038f70 0x7f128003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f127c006fd0 tx=0x7f127c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.267+0000 7f128dffb700 1 -- 172.21.15.49:0/3540063843 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1284003c90 con 0x7f129007de80 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:41.483+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi049", "addr": "172.21.15.49", "target": ["mon-mgr", ""]}) -- 0x7f12901043a0 con 0x7f1280038f70 2024-03-18T00:40:46.630 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.050+0000 7f128dffb700 1 -- 172.21.15.49:0/3540063843 <== mgr.14118 v2:172.21.15.49:6800/1899027445 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f12901043a0 con 0x7f1280038f70 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f1280038f70 msgr2=0x7f128003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 --2- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f1280038f70 0x7f128003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f127c006fd0 tx=0x7f127c006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 msgr2=0x7f129019dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 --2- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129019dd40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1284011850 tx=0x7f1284011c10 comp rx=0 tx=0).stop 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 shutdown_connections 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 --2- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f1280038f70 0x7f128003b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 --2- 172.21.15.49:0/3540063843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f129007de80 0x7f129019dd40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 >> 172.21.15.49:0/3540063843 conn(0x7f1290103140 msgr2=0x7f1290103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 shutdown_connections 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:46.053+0000 7f1296e6a700 1 -- 172.21.15.49:0/3540063843 wait complete. 2024-03-18T00:40:46.631 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mon service... 2024-03-18T00:40:47.279 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:47 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:47.280 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:47 smithi049 ceph-mon[32130]: Added host smithi049 2024-03-18T00:40:47.280 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:47 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:40:47.828 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-18T00:40:47.828 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.052+0000 7fe7583cb700 1 Processor -- start 2024-03-18T00:40:47.828 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.052+0000 7fe7583cb700 1 -- start start 2024-03-18T00:40:47.828 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.052+0000 7fe7583cb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe750106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:47.828 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.052+0000 7fe7583cb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe750107400 con 0x7fe750106a10 2024-03-18T00:40:47.828 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.052+0000 7fe756167700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe750106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:47.828 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.052+0000 7fe756167700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe750106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44166/0 (socket says 172.21.15.49:44166) 2024-03-18T00:40:47.828 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.052+0000 7fe756167700 1 -- 172.21.15.49:0/3706870265 learned_addr learned my addr 172.21.15.49:0/3706870265 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.053+0000 7fe756167700 1 -- 172.21.15.49:0/3706870265 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe750107c10 con 0x7fe750106a10 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.053+0000 7fe756167700 1 --2- 172.21.15.49:0/3706870265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe750106e30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0098a0 tx=0x7fe74c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17fc2cd679d713c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.053+0000 7fe755165700 1 -- 172.21.15.49:0/3706870265 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe74c004160 con 0x7fe750106a10 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.053+0000 7fe755165700 1 -- 172.21.15.49:0/3706870265 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe74c004300 con 0x7fe750106a10 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.053+0000 7fe755165700 1 -- 172.21.15.49:0/3706870265 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe74c011b50 con 0x7fe750106a10 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.054+0000 7fe7583cb700 1 -- 172.21.15.49:0/3706870265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 msgr2=0x7fe750106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.054+0000 7fe7583cb700 1 --2- 172.21.15.49:0/3706870265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe750106e30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe74c0098a0 tx=0x7fe74c009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.054+0000 7fe7583cb700 1 -- 172.21.15.49:0/3706870265 shutdown_connections 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.054+0000 7fe7583cb700 1 --2- 172.21.15.49:0/3706870265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe750106e30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.054+0000 7fe7583cb700 1 -- 172.21.15.49:0/3706870265 >> 172.21.15.49:0/3706870265 conn(0x7fe750102030 msgr2=0x7fe750104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.054+0000 7fe7583cb700 1 -- 172.21.15.49:0/3706870265 shutdown_connections 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.054+0000 7fe7583cb700 1 -- 172.21.15.49:0/3706870265 wait complete. 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.054+0000 7fe7583cb700 1 Processor -- start 2024-03-18T00:40:47.829 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.054+0000 7fe7583cb700 1 -- start start 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe7583cb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe75019bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe7583cb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe75010afc0 con 0x7fe750106a10 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe756167700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe75019bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe756167700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe75019bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44182/0 (socket says 172.21.15.49:44182) 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe756167700 1 -- 172.21.15.49:0/3551287783 learned_addr learned my addr 172.21.15.49:0/3551287783 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe756167700 1 -- 172.21.15.49:0/3551287783 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75019c0d0 con 0x7fe750106a10 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe756167700 1 --2- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe75019bb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe74c011850 tx=0x7fe74c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe7477fe700 1 -- 172.21.15.49:0/3551287783 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe74c003af0 con 0x7fe750106a10 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe7477fe700 1 -- 172.21.15.49:0/3551287783 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe74c020070 con 0x7fe750106a10 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe7477fe700 1 -- 172.21.15.49:0/3551287783 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe74c0195a0 con 0x7fe750106a10 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe75019c360 con 0x7fe750106a10 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.055+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe75019f050 con 0x7fe750106a10 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.056+0000 7fe7477fe700 1 -- 172.21.15.49:0/3551287783 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fe74c019850 con 0x7fe750106a10 2024-03-18T00:40:47.830 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.056+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe750063900 con 0x7fe750106a10 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.056+0000 7fe7477fe700 1 --2- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fe73c038f70 0x7fe73c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.056+0000 7fe7477fe700 1 -- 172.21.15.49:0/3551287783 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe74c011e00 con 0x7fe750106a10 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.056+0000 7fe755966700 1 --2- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fe73c038f70 0x7fe73c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.057+0000 7fe755966700 1 --2- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fe73c038f70 0x7fe73c03b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe740006fd0 tx=0x7fe740006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.061+0000 7fe7477fe700 1 -- 172.21.15.49:0/3551287783 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe74c003c90 con 0x7fe750106a10 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.279+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fe7500008d0 con 0x7fe73c038f70 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.284+0000 7fe7477fe700 1 -- 172.21.15.49:0/3551287783 <== mgr.14118 v2:172.21.15.49:6800/1899027445 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe7500008d0 con 0x7fe73c038f70 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.286+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fe73c038f70 msgr2=0x7fe73c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.286+0000 7fe7583cb700 1 --2- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fe73c038f70 0x7fe73c03b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe740006fd0 tx=0x7fe740006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.286+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 msgr2=0x7fe75019bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.286+0000 7fe7583cb700 1 --2- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe75019bb90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe74c011850 tx=0x7fe74c011c10 comp rx=0 tx=0).stop 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.286+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 shutdown_connections 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.286+0000 7fe7583cb700 1 --2- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7fe73c038f70 0x7fe73c03b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:47.831 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.287+0000 7fe7583cb700 1 --2- 172.21.15.49:0/3551287783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe750106a10 0x7fe75019bb90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:47.832 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.287+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 >> 172.21.15.49:0/3551287783 conn(0x7fe750102030 msgr2=0x7fe750102f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:47.832 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.287+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 shutdown_connections 2024-03-18T00:40:47.832 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:47.287+0000 7fe7583cb700 1 -- 172.21.15.49:0/3551287783 wait complete. 2024-03-18T00:40:47.832 INFO:teuthology.orchestra.run.smithi049.stdout:Deploying unmanaged mgr service... 2024-03-18T00:40:48.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:48 smithi049 ceph-mon[32130]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:40:48.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:48 smithi049 ceph-mon[32130]: Saving service mon spec with placement count:5 2024-03-18T00:40:48.509 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:48 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:49.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:49 smithi049 ceph-mon[32130]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:40:49.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:49 smithi049 ceph-mon[32130]: Saving service mgr spec with placement count:2 2024-03-18T00:40:49.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:49 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:49.808 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:49 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:49.815 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-18T00:40:49.815 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.278+0000 7ff2bdbd1700 1 Processor -- start 2024-03-18T00:40:49.815 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.278+0000 7ff2bdbd1700 1 -- start start 2024-03-18T00:40:49.815 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2bdbd1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b810a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:49.815 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2bdbd1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b807be30 con 0x7ff2b81085e0 2024-03-18T00:40:49.815 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2b77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b810a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:49.815 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2b77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b810a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53654/0 (socket says 172.21.15.49:53654) 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2b77fe700 1 -- 172.21.15.49:0/48491480 learned_addr learned my addr 172.21.15.49:0/48491480 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2b77fe700 1 -- 172.21.15.49:0/48491480 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b810af10 con 0x7ff2b81085e0 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2b77fe700 1 --2- 172.21.15.49:0/48491480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b810a9d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80098a0 tx=0x7ff2a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e093842fe74c3cb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2b67fc700 1 -- 172.21.15.49:0/48491480 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8004160 con 0x7ff2b81085e0 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2b67fc700 1 -- 172.21.15.49:0/48491480 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff2a8004300 con 0x7ff2b81085e0 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.279+0000 7ff2b67fc700 1 -- 172.21.15.49:0/48491480 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8011b50 con 0x7ff2b81085e0 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.280+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/48491480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 msgr2=0x7ff2b810a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.280+0000 7ff2bdbd1700 1 --2- 172.21.15.49:0/48491480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b810a9d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80098a0 tx=0x7ff2a8009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.280+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/48491480 shutdown_connections 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.280+0000 7ff2bdbd1700 1 --2- 172.21.15.49:0/48491480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b810a9d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.280+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/48491480 >> 172.21.15.49:0/48491480 conn(0x7ff2b8102030 msgr2=0x7ff2b8104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.280+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/48491480 shutdown_connections 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.280+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/48491480 wait complete. 2024-03-18T00:40:49.816 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.281+0000 7ff2bdbd1700 1 Processor -- start 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.281+0000 7ff2bdbd1700 1 -- start start 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.281+0000 7ff2bdbd1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b8107e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.281+0000 7ff2bdbd1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b810bf30 con 0x7ff2b81085e0 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.281+0000 7ff2b77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b8107e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.281+0000 7ff2b77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b8107e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53656/0 (socket says 172.21.15.49:53656) 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.281+0000 7ff2b77fe700 1 -- 172.21.15.49:0/1173051831 learned_addr learned my addr 172.21.15.49:0/1173051831 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.281+0000 7ff2b77fe700 1 -- 172.21.15.49:0/1173051831 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b81083a0 con 0x7ff2b81085e0 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.282+0000 7ff2b77fe700 1 --2- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b8107e60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009fb0 tx=0x7ff2a8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.282+0000 7ff2b4ff9700 1 -- 172.21.15.49:0/1173051831 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a80038b0 con 0x7ff2b81085e0 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.282+0000 7ff2b4ff9700 1 -- 172.21.15.49:0/1173051831 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff2a8011e50 con 0x7ff2b81085e0 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.282+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b8106590 con 0x7ff2b81085e0 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.282+0000 7ff2b4ff9700 1 -- 172.21.15.49:0/1173051831 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff2a8019420 con 0x7ff2b81085e0 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.282+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b8106ad0 con 0x7ff2b81085e0 2024-03-18T00:40:49.817 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.283+0000 7ff2b4ff9700 1 -- 172.21.15.49:0/1173051831 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7ff2a8003a50 con 0x7ff2b81085e0 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.283+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2b8063900 con 0x7ff2b81085e0 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.283+0000 7ff2b4ff9700 1 --2- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7ff2a0038f20 0x7ff2a003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.283+0000 7ff2b6ffd700 1 --2- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7ff2a0038f20 0x7ff2a003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.284+0000 7ff2b6ffd700 1 --2- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7ff2a0038f20 0x7ff2a003b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac006fd0 tx=0x7ff2ac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.285+0000 7ff2b4ff9700 1 -- 172.21.15.49:0/1173051831 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff2a80196f0 con 0x7ff2b81085e0 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.290+0000 7ff2b4ff9700 1 -- 172.21.15.49:0/1173051831 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff2a8023390 con 0x7ff2b81085e0 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.519+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7ff2b8107330 con 0x7ff2a0038f20 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.524+0000 7ff2b4ff9700 1 -- 172.21.15.49:0/1173051831 <== mgr.14118 v2:172.21.15.49:6800/1899027445 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff2b8107330 con 0x7ff2a0038f20 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7ff2a0038f20 msgr2=0x7ff2a003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 --2- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7ff2a0038f20 0x7ff2a003b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac006fd0 tx=0x7ff2ac006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 msgr2=0x7ff2b8107e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 --2- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b8107e60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009fb0 tx=0x7ff2a8011bf0 comp rx=0 tx=0).stop 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 shutdown_connections 2024-03-18T00:40:49.818 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 --2- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7ff2a0038f20 0x7ff2a003b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:49.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 --2- 172.21.15.49:0/1173051831 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff2b81085e0 0x7ff2b8107e60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:49.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 >> 172.21.15.49:0/1173051831 conn(0x7ff2b8102030 msgr2=0x7ff2b8102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:49.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 shutdown_connections 2024-03-18T00:40:49.819 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:48.526+0000 7ff2bdbd1700 1 -- 172.21.15.49:0/1173051831 wait complete. 2024-03-18T00:40:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:50 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:50 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/991262172' entity='client.admin' 2024-03-18T00:40:51.750 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.267+0000 7f7d51cdb700 1 Processor -- start 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.267+0000 7f7d51cdb700 1 -- start start 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.267+0000 7f7d51cdb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c10a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.267+0000 7f7d51cdb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c07be50 con 0x7f7d4c1085a0 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.267+0000 7f7d4b7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c10a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.267+0000 7f7d4b7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c10a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53668/0 (socket says 172.21.15.49:53668) 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.268+0000 7f7d4b7fe700 1 -- 172.21.15.49:0/2116235038 learned_addr learned my addr 172.21.15.49:0/2116235038 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.268+0000 7f7d4b7fe700 1 -- 172.21.15.49:0/2116235038 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4c10aed0 con 0x7f7d4c1085a0 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.268+0000 7f7d4b7fe700 1 --2- 172.21.15.49:0/2116235038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c10a990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7d400098a0 tx=0x7f7d40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbc676ad56477cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.268+0000 7f7d4a7fc700 1 -- 172.21.15.49:0/2116235038 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d40004160 con 0x7f7d4c1085a0 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.268+0000 7f7d4a7fc700 1 -- 172.21.15.49:0/2116235038 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d40004300 con 0x7f7d4c1085a0 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.268+0000 7f7d4a7fc700 1 -- 172.21.15.49:0/2116235038 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d40011b20 con 0x7f7d4c1085a0 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.269+0000 7f7d51cdb700 1 -- 172.21.15.49:0/2116235038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 msgr2=0x7f7d4c10a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:51.751 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.269+0000 7f7d51cdb700 1 --2- 172.21.15.49:0/2116235038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c10a990 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7d400098a0 tx=0x7f7d40009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.269+0000 7f7d51cdb700 1 -- 172.21.15.49:0/2116235038 shutdown_connections 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.269+0000 7f7d51cdb700 1 --2- 172.21.15.49:0/2116235038 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c10a990 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.269+0000 7f7d51cdb700 1 -- 172.21.15.49:0/2116235038 >> 172.21.15.49:0/2116235038 conn(0x7f7d4c102030 msgr2=0x7f7d4c104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.269+0000 7f7d51cdb700 1 -- 172.21.15.49:0/2116235038 shutdown_connections 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.269+0000 7f7d51cdb700 1 -- 172.21.15.49:0/2116235038 wait complete. 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.270+0000 7f7d51cdb700 1 Processor -- start 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.270+0000 7f7d51cdb700 1 -- start start 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.270+0000 7f7d51cdb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.270+0000 7f7d51cdb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4c10bf30 con 0x7f7d4c1085a0 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.270+0000 7f7d4b7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.270+0000 7f7d4b7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53682/0 (socket says 172.21.15.49:53682) 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.270+0000 7f7d4b7fe700 1 -- 172.21.15.49:0/991262172 learned_addr learned my addr 172.21.15.49:0/991262172 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.271+0000 7f7d4b7fe700 1 -- 172.21.15.49:0/991262172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d4c19bf60 con 0x7f7d4c1085a0 2024-03-18T00:40:51.752 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.271+0000 7f7d4b7fe700 1 --2- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c19ba20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7d400116a0 tx=0x7f7d40011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.271+0000 7f7d48ff9700 1 -- 172.21.15.49:0/991262172 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d40003a40 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.271+0000 7f7d48ff9700 1 -- 172.21.15.49:0/991262172 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d40003be0 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.271+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d4c19c250 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.271+0000 7f7d48ff9700 1 -- 172.21.15.49:0/991262172 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d4001a720 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.271+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d4c19eee0 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.272+0000 7f7d48ff9700 1 -- 172.21.15.49:0/991262172 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f7d4001a9d0 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.272+0000 7f7d48ff9700 1 --2- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7d34038f20 0x7f7d3403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.272+0000 7f7d48ff9700 1 -- 172.21.15.49:0/991262172 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7d4004e2a0 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.272+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d4c063900 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.272+0000 7f7d4affd700 1 --2- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7d34038f20 0x7f7d3403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.273+0000 7f7d4affd700 1 --2- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7d34038f20 0x7f7d3403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006fd0 tx=0x7f7d3c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.277+0000 7f7d48ff9700 1 -- 172.21.15.49:0/991262172 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d40011e20 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.492+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f7d4c19f420 con 0x7f7d4c1085a0 2024-03-18T00:40:51.753 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.494+0000 7f7d48ff9700 1 -- 172.21.15.49:0/991262172 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f7d4000bc40 con 0x7f7d4c1085a0 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7d34038f20 msgr2=0x7f7d3403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 --2- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7d34038f20 0x7f7d3403b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006fd0 tx=0x7f7d3c006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 msgr2=0x7f7d4c19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 --2- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c19ba20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7d400116a0 tx=0x7f7d40011ba0 comp rx=0 tx=0).stop 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 shutdown_connections 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 --2- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f7d34038f20 0x7f7d3403b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 --2- 172.21.15.49:0/991262172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d4c1085a0 0x7f7d4c19ba20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 >> 172.21.15.49:0/991262172 conn(0x7f7d4c102030 msgr2=0x7f7d4c102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 shutdown_connections 2024-03-18T00:40:51.754 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:50.497+0000 7f7d51cdb700 1 -- 172.21.15.49:0/991262172 wait complete. 2024-03-18T00:40:52.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:52 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:52.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:52 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1495574689' entity='client.admin' 2024-03-18T00:40:53.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.142+0000 7f98b58c8700 1 Processor -- start 2024-03-18T00:40:53.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.142+0000 7f98b58c8700 1 -- start start 2024-03-18T00:40:53.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.142+0000 7f98b58c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0106220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:53.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.142+0000 7f98b58c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f98b01067f0 con 0x7f98b0105e00 2024-03-18T00:40:53.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.143+0000 7f98aeffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0106220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:53.216 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.143+0000 7f98aeffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0106220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53686/0 (socket says 172.21.15.49:53686) 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.143+0000 7f98aeffd700 1 -- 172.21.15.49:0/43207143 learned_addr learned my addr 172.21.15.49:0/43207143 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.143+0000 7f98aeffd700 1 -- 172.21.15.49:0/43207143 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b0108d90 con 0x7f98b0105e00 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.143+0000 7f98aeffd700 1 --2- 172.21.15.49:0/43207143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0106220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f98a00098a0 tx=0x7f98a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39fa71fc6ab9e742 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.143+0000 7f98ae7fc700 1 -- 172.21.15.49:0/43207143 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98a0004160 con 0x7f98b0105e00 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.143+0000 7f98ae7fc700 1 -- 172.21.15.49:0/43207143 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98a0004300 con 0x7f98b0105e00 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.143+0000 7f98ae7fc700 1 -- 172.21.15.49:0/43207143 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98a0011b20 con 0x7f98b0105e00 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.144+0000 7f98b58c8700 1 -- 172.21.15.49:0/43207143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 msgr2=0x7f98b0106220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.144+0000 7f98b58c8700 1 --2- 172.21.15.49:0/43207143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0106220 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f98a00098a0 tx=0x7f98a0009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.144+0000 7f98b58c8700 1 -- 172.21.15.49:0/43207143 shutdown_connections 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.144+0000 7f98b58c8700 1 --2- 172.21.15.49:0/43207143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0106220 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.144+0000 7f98b58c8700 1 -- 172.21.15.49:0/43207143 >> 172.21.15.49:0/43207143 conn(0x7f98b0101710 msgr2=0x7f98b0103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.144+0000 7f98b58c8700 1 -- 172.21.15.49:0/43207143 shutdown_connections 2024-03-18T00:40:53.217 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.144+0000 7f98b58c8700 1 -- 172.21.15.49:0/43207143 wait complete. 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.145+0000 7f98b58c8700 1 Processor -- start 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.145+0000 7f98b58c8700 1 -- start start 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.145+0000 7f98b58c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0108610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.145+0000 7f98b58c8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f98b010c140 con 0x7f98b0105e00 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.145+0000 7f98aeffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0108610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.145+0000 7f98aeffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0108610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53694/0 (socket says 172.21.15.49:53694) 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.145+0000 7f98aeffd700 1 -- 172.21.15.49:0/1495574689 learned_addr learned my addr 172.21.15.49:0/1495574689 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.145+0000 7f98aeffd700 1 -- 172.21.15.49:0/1495574689 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b0108b50 con 0x7f98b0105e00 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.146+0000 7f98aeffd700 1 --2- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0108610 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f98a00117e0 tx=0x7f98a0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.146+0000 7f98b48c6700 1 -- 172.21.15.49:0/1495574689 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98a0003bf0 con 0x7f98b0105e00 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.146+0000 7f98b48c6700 1 -- 172.21.15.49:0/1495574689 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98a001a910 con 0x7f98b0105e00 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.146+0000 7f98b48c6700 1 -- 172.21.15.49:0/1495574689 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98a00249a0 con 0x7f98b0105e00 2024-03-18T00:40:53.218 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.146+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b0106d40 con 0x7f98b0105e00 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.146+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b0107200 con 0x7f98b0105e00 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.147+0000 7f98b48c6700 1 -- 172.21.15.49:0/1495574689 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f98a0022020 con 0x7f98b0105e00 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.147+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b0063900 con 0x7f98b0105e00 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.147+0000 7f98b48c6700 1 --2- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f989c038ed0 0x7f989c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.147+0000 7f98b48c6700 1 -- 172.21.15.49:0/1495574689 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f98a004d1a0 con 0x7f98b0105e00 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.147+0000 7f98a7fff700 1 --2- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f989c038ed0 0x7f989c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.147+0000 7f98a7fff700 1 --2- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f989c038ed0 0x7f989c03b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9898006fd0 tx=0x7f9898006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.152+0000 7f98b48c6700 1 -- 172.21.15.49:0/1495574689 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98a0011e20 con 0x7f98b0105e00 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.358+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f98b0107480 con 0x7f98b0105e00 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.360+0000 7f98b48c6700 1 -- 172.21.15.49:0/1495574689 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f98a000b9d0 con 0x7f98b0105e00 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f989c038ed0 msgr2=0x7f989c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:53.219 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 --2- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f989c038ed0 0x7f989c03b390 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9898006fd0 tx=0x7f9898006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:53.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 msgr2=0x7f98b0108610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:53.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 --2- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0108610 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f98a00117e0 tx=0x7f98a0011c10 comp rx=0 tx=0).stop 2024-03-18T00:40:53.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 shutdown_connections 2024-03-18T00:40:53.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 --2- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f989c038ed0 0x7f989c03b390 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:53.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 --2- 172.21.15.49:0/1495574689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f98b0105e00 0x7f98b0108610 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:53.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 >> 172.21.15.49:0/1495574689 conn(0x7f98b0101710 msgr2=0x7f98b0102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:53.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 shutdown_connections 2024-03-18T00:40:53.220 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:52.363+0000 7f98b58c8700 1 -- 172.21.15.49:0/1495574689 wait complete. 2024-03-18T00:40:53.220 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling the dashboard module... 2024-03-18T00:40:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:54 smithi049 ceph-mon[32130]: from='mgr.14118 172.21.15.49:0/795676255' entity='mgr.x' 2024-03-18T00:40:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:54 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/463952946' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T00:40:54.375 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: ignoring --setuser ceph since I am not root 2024-03-18T00:40:54.375 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: ignoring --setgroup ceph since I am not root 2024-03-18T00:40:54.952 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:54.742+0000 7fc7fb5ec200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T00:40:54.952 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:54.839+0000 7fc7fb5ec200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T00:40:55.313 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:55 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/463952946' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T00:40:55.313 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:55 smithi049 ceph-mon[32130]: mgrmap e9: x(active, since 19s) 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.632+0000 7f42a18f9700 1 Processor -- start 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.632+0000 7f42a18f9700 1 -- start start 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.632+0000 7f42a18f9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c10a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.632+0000 7f42a18f9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f429c07be70 con 0x7f429c108580 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.633+0000 7f429affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c10a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.633+0000 7f429affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c10a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53710/0 (socket says 172.21.15.49:53710) 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.633+0000 7f429affd700 1 -- 172.21.15.49:0/1126901152 learned_addr learned my addr 172.21.15.49:0/1126901152 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.633+0000 7f429affd700 1 -- 172.21.15.49:0/1126901152 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429c10aeb0 con 0x7f429c108580 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.633+0000 7f429affd700 1 --2- 172.21.15.49:0/1126901152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c10a970 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f428c0098a0 tx=0x7f428c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9544d3d05e4b506d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:55.320 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.634+0000 7f4299ffb700 1 -- 172.21.15.49:0/1126901152 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c004160 con 0x7f429c108580 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.634+0000 7f4299ffb700 1 -- 172.21.15.49:0/1126901152 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f428c004300 con 0x7f429c108580 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.634+0000 7f4299ffb700 1 -- 172.21.15.49:0/1126901152 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c011b20 con 0x7f429c108580 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.635+0000 7f42a18f9700 1 -- 172.21.15.49:0/1126901152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 msgr2=0x7f429c10a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.635+0000 7f42a18f9700 1 --2- 172.21.15.49:0/1126901152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c10a970 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f428c0098a0 tx=0x7f428c009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.635+0000 7f42a18f9700 1 -- 172.21.15.49:0/1126901152 shutdown_connections 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.635+0000 7f42a18f9700 1 --2- 172.21.15.49:0/1126901152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c10a970 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.635+0000 7f42a18f9700 1 -- 172.21.15.49:0/1126901152 >> 172.21.15.49:0/1126901152 conn(0x7f429c101fd0 msgr2=0x7f429c1043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.635+0000 7f42a18f9700 1 -- 172.21.15.49:0/1126901152 shutdown_connections 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.635+0000 7f42a18f9700 1 -- 172.21.15.49:0/1126901152 wait complete. 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.635+0000 7f42a18f9700 1 Processor -- start 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.636+0000 7f42a18f9700 1 -- start start 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.636+0000 7f42a18f9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c107e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.636+0000 7f42a18f9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f429c10bf20 con 0x7f429c108580 2024-03-18T00:40:55.321 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.636+0000 7f429affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c107e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.636+0000 7f429affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c107e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53724/0 (socket says 172.21.15.49:53724) 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.636+0000 7f429affd700 1 -- 172.21.15.49:0/463952946 learned_addr learned my addr 172.21.15.49:0/463952946 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.636+0000 7f429affd700 1 -- 172.21.15.49:0/463952946 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f429c108340 con 0x7f429c108580 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.637+0000 7f429affd700 1 --2- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c107e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f428c0117e0 tx=0x7f428c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.637+0000 7f42a08f7700 1 -- 172.21.15.49:0/463952946 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c003bf0 con 0x7f429c108580 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.637+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f429c106530 con 0x7f429c108580 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.637+0000 7f42a08f7700 1 -- 172.21.15.49:0/463952946 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f428c01a910 con 0x7f429c108580 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.637+0000 7f42a08f7700 1 -- 172.21.15.49:0/463952946 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f428c024970 con 0x7f429c108580 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.637+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f429c1069f0 con 0x7f429c108580 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.638+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f429c063900 con 0x7f429c108580 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.638+0000 7f42a08f7700 1 -- 172.21.15.49:0/463952946 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f428c022020 con 0x7f429c108580 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.638+0000 7f42a08f7700 1 --2- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f4284038f20 0x7f428403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:55.322 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.638+0000 7f42a08f7700 1 -- 172.21.15.49:0/463952946 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f428c04d180 con 0x7f429c108580 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.638+0000 7f429a7fc700 1 --2- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f4284038f20 0x7f428403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.639+0000 7f429a7fc700 1 --2- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f4284038f20 0x7f428403b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4290006fd0 tx=0x7f4290006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.648+0000 7f42a08f7700 1 -- 172.21.15.49:0/463952946 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f428c011e20 con 0x7f429c108580 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:53.920+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f429c106c70 con 0x7f429c108580 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.156+0000 7f42a08f7700 1 -- 172.21.15.49:0/463952946 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f428c00ba30 con 0x7f429c108580 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.157+0000 7f42a08f7700 1 -- 172.21.15.49:0/463952946 <== mon.0 v2:172.21.15.49:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f428c02d910 con 0x7f429c108580 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f4284038f20 msgr2=0x7f428403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 --2- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f4284038f20 0x7f428403b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4290006fd0 tx=0x7f4290006e40 comp rx=0 tx=0).stop 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 msgr2=0x7f429c107e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 --2- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c107e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f428c0117e0 tx=0x7f428c011c10 comp rx=0 tx=0).stop 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 shutdown_connections 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 --2- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f4284038f20 0x7f428403b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 --2- 172.21.15.49:0/463952946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f429c108580 0x7f429c107e00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:55.323 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 >> 172.21.15.49:0/463952946 conn(0x7f429c101fd0 msgr2=0x7f429c1029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:55.324 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 shutdown_connections 2024-03-18T00:40:55.324 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:54.159+0000 7f42a18f9700 1 -- 172.21.15.49:0/463952946 wait complete. 2024-03-18T00:40:55.562 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:55.518+0000 7fc7fb5ec200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T00:40:55.868 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:55.618+0000 7fc7fb5ec200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T00:40:55.868 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:55.766+0000 7fc7fb5ec200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T00:40:55.868 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:55.868+0000 7fc7fb5ec200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T00:40:56.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:55.954+0000 7fc7fb5ec200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T00:40:56.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:40:56 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2742610882' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T00:40:56.441 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:56.236+0000 7fc7fb5ec200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T00:40:56.441 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:56.352+0000 7fc7fb5ec200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T00:40:56.728 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:56.448+0000 7fc7fb5ec200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T00:40:56.729 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:56.554+0000 7fc7fb5ec200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T00:40:56.732 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-03-18T00:40:56.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-18T00:40:56.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T00:40:56.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T00:40:56.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T00:40:56.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-03-18T00:40:56.738 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.779+0000 7f8f8d7dc700 1 Processor -- start 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.779+0000 7f8f8d7dc700 1 -- start start 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.779+0000 7f8f8d7dc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88106e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.779+0000 7f8f8d7dc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88107400 con 0x7f8f88106a10 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.780+0000 7f8f86ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88106e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.780+0000 7f8f86ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88106e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53746/0 (socket says 172.21.15.49:53746) 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.780+0000 7f8f86ffd700 1 -- 172.21.15.49:0/934541923 learned_addr learned my addr 172.21.15.49:0/934541923 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.780+0000 7f8f86ffd700 1 -- 172.21.15.49:0/934541923 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f88107c10 con 0x7f8f88106a10 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.780+0000 7f8f86ffd700 1 --2- 172.21.15.49:0/934541923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88106e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f780098a0 tx=0x7f8f78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d50d07eb40f5c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.780+0000 7f8f85ffb700 1 -- 172.21.15.49:0/934541923 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f78004160 con 0x7f8f88106a10 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.780+0000 7f8f85ffb700 1 -- 172.21.15.49:0/934541923 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f78004300 con 0x7f8f88106a10 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.780+0000 7f8f85ffb700 1 -- 172.21.15.49:0/934541923 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f78011b20 con 0x7f8f88106a10 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.781+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/934541923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 msgr2=0x7f8f88106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.781+0000 7f8f8d7dc700 1 --2- 172.21.15.49:0/934541923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88106e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8f780098a0 tx=0x7f8f78009bb0 comp rx=0 tx=0).stop 2024-03-18T00:40:56.739 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.781+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/934541923 shutdown_connections 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.781+0000 7f8f8d7dc700 1 --2- 172.21.15.49:0/934541923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88106e30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.781+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/934541923 >> 172.21.15.49:0/934541923 conn(0x7f8f88101fd0 msgr2=0x7f8f881043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.781+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/934541923 shutdown_connections 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.781+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/934541923 wait complete. 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.782+0000 7f8f8d7dc700 1 Processor -- start 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.782+0000 7f8f8d7dc700 1 -- start start 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.782+0000 7f8f8d7dc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88191bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.782+0000 7f8f8d7dc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8810afc0 con 0x7f8f88106a10 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.782+0000 7f8f86ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88191bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.782+0000 7f8f86ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88191bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53756/0 (socket says 172.21.15.49:53756) 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.782+0000 7f8f86ffd700 1 -- 172.21.15.49:0/2742610882 learned_addr learned my addr 172.21.15.49:0/2742610882 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.782+0000 7f8f86ffd700 1 -- 172.21.15.49:0/2742610882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f88192100 con 0x7f8f88106a10 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.783+0000 7f8f86ffd700 1 --2- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88191bc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8f780117e0 tx=0x7f8f78011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:40:56.740 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.783+0000 7f8f6ffff700 1 -- 172.21.15.49:0/2742610882 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f78003bf0 con 0x7f8f88106a10 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.783+0000 7f8f6ffff700 1 -- 172.21.15.49:0/2742610882 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f7801a910 con 0x7f8f88106a10 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.783+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f881902f0 con 0x7f8f88106a10 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.783+0000 7f8f6ffff700 1 -- 172.21.15.49:0/2742610882 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8f780249a0 con 0x7f8f88106a10 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.783+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f881907b0 con 0x7f8f88106a10 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.784+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f88063900 con 0x7f8f88106a10 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.784+0000 7f8f6ffff700 1 -- 172.21.15.49:0/2742610882 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f8f78022020 con 0x7f8f88106a10 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.784+0000 7f8f6ffff700 1 --2- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f8f70038f80 0x7f8f7003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.784+0000 7f8f6ffff700 1 -- 172.21.15.49:0/2742610882 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8f7804d300 con 0x7f8f88106a10 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.784+0000 7f8f867fc700 1 -- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f8f70038f80 msgr2=0x7f8f7003b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1899027445 2024-03-18T00:40:56.741 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.784+0000 7f8f867fc700 1 --2- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f8f70038f80 0x7f8f7003b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.789+0000 7f8f6ffff700 1 -- 172.21.15.49:0/2742610882 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f78011e20 con 0x7f8f88106a10 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.985+0000 7f8f867fc700 1 -- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f8f70038f80 msgr2=0x7f8f7003b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1899027445 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:55.985+0000 7f8f867fc700 1 --2- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f8f70038f80 0x7f8f7003b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.059+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f8f88195910 con 0x7f8f88106a10 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.060+0000 7f8f6ffff700 1 -- 172.21.15.49:0/2742610882 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f8f7802c350 con 0x7f8f88106a10 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f8f70038f80 msgr2=0x7f8f7003b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 --2- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f8f70038f80 0x7f8f7003b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 msgr2=0x7f8f88191bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 --2- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88191bc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8f780117e0 tx=0x7f8f78011c10 comp rx=0 tx=0).stop 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 shutdown_connections 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 --2- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f8f70038f80 0x7f8f7003b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 --2- 172.21.15.49:0/2742610882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f88106a10 0x7f8f88191bc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:40:56.742 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 >> 172.21.15.49:0/2742610882 conn(0x7f8f88101fd0 msgr2=0x7f8f88102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:40:56.743 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 shutdown_connections 2024-03-18T00:40:56.743 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:56.062+0000 7f8f8d7dc700 1 -- 172.21.15.49:0/2742610882 wait complete. 2024-03-18T00:40:56.743 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for the mgr to restart... 2024-03-18T00:40:56.743 INFO:teuthology.orchestra.run.smithi049.stdout:Waiting for mgr epoch 9... 2024-03-18T00:40:57.039 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:56.742+0000 7fc7fb5ec200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T00:40:58.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:57 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:57.779+0000 7fc7fb5ec200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T00:40:58.125 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:57 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:57.915+0000 7fc7fb5ec200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T00:40:58.570 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:58 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:58.271+0000 7fc7fb5ec200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T00:40:58.570 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:58 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:58.365+0000 7fc7fb5ec200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T00:40:58.570 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:58 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:58.570+0000 7fc7fb5ec200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:40:58.874 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:58 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:58.763+0000 7fc7fb5ec200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T00:40:59.328 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:59 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:59.060+0000 7fc7fb5ec200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:40:59.328 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:59 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:59.328+0000 7fc7fb5ec200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T00:40:59.624 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:40:59 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:40:59.576+0000 7fc7fb5ec200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T00:41:00.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:41:00 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:41:00.421+0000 7fc7fb5ec200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T00:41:01.362 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:41:01.033+0000 7fc7fb5ec200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T00:41:01.363 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:41:01.135+0000 7fc7fb5ec200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T00:41:01.363 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:41:01.361+0000 7fc7fb5ec200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T00:41:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: Active manager daemon x restarted 2024-03-18T00:41:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: Activating manager daemon x 2024-03-18T00:41:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T00:41:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: mgrmap e10: x(active, starting, since 0.00321314s) 2024-03-18T00:41:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:41:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T00:41:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T00:41:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T00:41:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T00:41:01.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: Manager daemon x is now available 2024-03-18T00:41:01.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:01.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:02.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T00:41:02.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T00:41:02.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:02 smithi049 ceph-mon[32130]: [18/Mar/2024:00:41:01] ENGINE Bus STARTING 2024-03-18T00:41:02.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:02 smithi049 ceph-mon[32130]: [18/Mar/2024:00:41:01] ENGINE Serving on http://172.21.15.49:8765 2024-03-18T00:41:02.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:02 smithi049 ceph-mon[32130]: [18/Mar/2024:00:41:01] ENGINE Serving on https://172.21.15.49:7150 2024-03-18T00:41:02.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:02 smithi049 ceph-mon[32130]: [18/Mar/2024:00:41:01] ENGINE Bus STARTED 2024-03-18T00:41:02.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:02 smithi049 ceph-mon[32130]: mgrmap e11: x(active, since 1.00655s) 2024-03-18T00:41:02.945 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout { 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout } 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.177+0000 7f00ed396700 1 Processor -- start 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.177+0000 7f00ed396700 1 -- start start 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.178+0000 7f00ed396700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8106df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.178+0000 7f00ed396700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f00e81073c0 con 0x7f00e81069d0 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.178+0000 7f00e6ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8106df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.178+0000 7f00e6ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8106df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53758/0 (socket says 172.21.15.49:53758) 2024-03-18T00:41:02.946 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.178+0000 7f00e6ffd700 1 -- 172.21.15.49:0/2220343470 learned_addr learned my addr 172.21.15.49:0/2220343470 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.179+0000 7f00e6ffd700 1 -- 172.21.15.49:0/2220343470 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e8107bd0 con 0x7f00e81069d0 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.179+0000 7f00e6ffd700 1 --2- 172.21.15.49:0/2220343470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8106df0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00d800bb40 tx=0x7f00d800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9556d5689a508e6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.179+0000 7f00e5ffb700 1 -- 172.21.15.49:0/2220343470 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00d800dbe0 con 0x7f00e81069d0 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.180+0000 7f00e5ffb700 1 -- 172.21.15.49:0/2220343470 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00d8004510 con 0x7f00e81069d0 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.180+0000 7f00e5ffb700 1 -- 172.21.15.49:0/2220343470 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00d8010470 con 0x7f00e81069d0 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.180+0000 7f00ed396700 1 -- 172.21.15.49:0/2220343470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 msgr2=0x7f00e8106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.180+0000 7f00ed396700 1 --2- 172.21.15.49:0/2220343470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8106df0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00d800bb40 tx=0x7f00d800be50 comp rx=0 tx=0).stop 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.180+0000 7f00ed396700 1 -- 172.21.15.49:0/2220343470 shutdown_connections 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.180+0000 7f00ed396700 1 --2- 172.21.15.49:0/2220343470 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8106df0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.180+0000 7f00ed396700 1 -- 172.21.15.49:0/2220343470 >> 172.21.15.49:0/2220343470 conn(0x7f00e8101f90 msgr2=0x7f00e81043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.181+0000 7f00ed396700 1 -- 172.21.15.49:0/2220343470 shutdown_connections 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.181+0000 7f00ed396700 1 -- 172.21.15.49:0/2220343470 wait complete. 2024-03-18T00:41:02.947 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.182+0000 7f00ed396700 1 Processor -- start 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.182+0000 7f00ed396700 1 -- start start 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.183+0000 7f00ed396700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.183+0000 7f00ed396700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f00e807af90 con 0x7f00e81069d0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.183+0000 7f00e6ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.183+0000 7f00e6ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53766/0 (socket says 172.21.15.49:53766) 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.183+0000 7f00e6ffd700 1 -- 172.21.15.49:0/1645342118 learned_addr learned my addr 172.21.15.49:0/1645342118 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.183+0000 7f00e6ffd700 1 -- 172.21.15.49:0/1645342118 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e8199db0 con 0x7f00e81069d0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.183+0000 7f00e6ffd700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8199870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00d800b340 tx=0x7f00d8003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.184+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00d8027070 con 0x7f00e81069d0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.184+0000 7f00ed396700 1 -- 172.21.15.49:0/1645342118 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e819a040 con 0x7f00e81069d0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.184+0000 7f00ed396700 1 -- 172.21.15.49:0/1645342118 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e819cd30 con 0x7f00e81069d0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.184+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00d8010c50 con 0x7f00e81069d0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.184+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00d8022be0 con 0x7f00e81069d0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.185+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f00d802a070 con 0x7f00e81069d0 2024-03-18T00:41:02.948 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.185+0000 7f00cffff700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 0x7f00d003b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.186+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 --> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f00d003bb50 con 0x7f00d0038f80 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.186+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f00d801f070 con 0x7f00e81069d0 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.186+0000 7f00e67fc700 1 -- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 msgr2=0x7f00d003b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1899027445 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.186+0000 7f00e67fc700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 0x7f00d003b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.387+0000 7f00e67fc700 1 -- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 msgr2=0x7f00d003b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1899027445 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.387+0000 7f00e67fc700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 0x7f00d003b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.788+0000 7f00e67fc700 1 -- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 msgr2=0x7f00d003b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1899027445 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:57.788+0000 7f00e67fc700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 0x7f00d003b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:58.589+0000 7f00e67fc700 1 -- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 msgr2=0x7f00d003b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1899027445 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:40:58.589+0000 7f00e67fc700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 0x7f00d003b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:00.189+0000 7f00e67fc700 1 -- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 msgr2=0x7f00d003b440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.49:6800/1899027445 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:00.189+0000 7f00e67fc700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 0x7f00d003b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:01.377+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f00d8010770 con 0x7f00e81069d0 2024-03-18T00:41:02.949 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:01.377+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 msgr2=0x7f00d003b440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:01.377+0000 7f00cffff700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 0x7f00d003b440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.379+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f00d8044120 con 0x7f00e81069d0 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.379+0000 7f00cffff700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f00d003ca30 0x7f00d003ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.379+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f00d8052e50 con 0x7f00d003ca30 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.379+0000 7f00e67fc700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f00d003ca30 0x7f00d003ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.380+0000 7f00e67fc700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f00d003ca30 0x7f00d003ee20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f00dc003a10 tx=0x7f00dc009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.381+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f00d8052e50 con 0x7f00d003ca30 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.387+0000 7f00ed396700 1 -- 172.21.15.49:0/1645342118 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f00e8063900 con 0x7f00d003ca30 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00cffff700 1 -- 172.21.15.49:0/1645342118 <== mgr.14150 v2:172.21.15.49:6800/1246732627 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f00e8063900 con 0x7f00d003ca30 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 -- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f00d003ca30 msgr2=0x7f00d003ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f00d003ca30 0x7f00d003ee20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f00dc003a10 tx=0x7f00dc009270 comp rx=0 tx=0).stop 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 -- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 msgr2=0x7f00e8199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8199870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00d800b340 tx=0x7f00d8003e80 comp rx=0 tx=0).stop 2024-03-18T00:41:02.950 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 -- 172.21.15.49:0/1645342118 shutdown_connections 2024-03-18T00:41:02.951 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f00e81069d0 0x7f00e8199870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:02.951 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f00d003ca30 0x7f00d003ee20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:02.951 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 --2- 172.21.15.49:0/1645342118 >> [v2:172.21.15.49:6800/1899027445,v1:172.21.15.49:6801/1899027445] conn(0x7f00d0038f80 0x7f00d003b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:02.951 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 -- 172.21.15.49:0/1645342118 >> 172.21.15.49:0/1645342118 conn(0x7f00e8101f90 msgr2=0x7f00e818feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:02.951 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 -- 172.21.15.49:0/1645342118 shutdown_connections 2024-03-18T00:41:02.951 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:02.388+0000 7f00ed396700 1 -- 172.21.15.49:0/1645342118 wait complete. 2024-03-18T00:41:02.951 INFO:teuthology.orchestra.run.smithi049.stdout:mgr epoch 9 is available 2024-03-18T00:41:02.951 INFO:teuthology.orchestra.run.smithi049.stdout:Generating a dashboard self-signed certificate... 2024-03-18T00:41:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:03 smithi049 ceph-mon[32130]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T00:41:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:03 smithi049 ceph-mon[32130]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T00:41:03.707 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:04.669 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-18T00:41:04.669 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.728+0000 7fe7d2c62700 1 Processor -- start 2024-03-18T00:41:04.669 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.728+0000 7fe7d2c62700 1 -- start start 2024-03-18T00:41:04.669 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.728+0000 7fe7d2c62700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc10a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.728+0000 7fe7d2c62700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc07be70 con 0x7fe7cc1085a0 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.728+0000 7fe7d09fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc10a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.728+0000 7fe7d09fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc10a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42864/0 (socket says 172.21.15.49:42864) 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.728+0000 7fe7d09fe700 1 -- 172.21.15.49:0/318521978 learned_addr learned my addr 172.21.15.49:0/318521978 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.729+0000 7fe7d09fe700 1 -- 172.21.15.49:0/318521978 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc10aed0 con 0x7fe7cc1085a0 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.729+0000 7fe7d09fe700 1 --2- 172.21.15.49:0/318521978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc10a990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00098a0 tx=0x7fe7c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54de2bc0389d84cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.729+0000 7fe7cb7fe700 1 -- 172.21.15.49:0/318521978 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7c0004160 con 0x7fe7cc1085a0 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.729+0000 7fe7cb7fe700 1 -- 172.21.15.49:0/318521978 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c0004300 con 0x7fe7cc1085a0 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.729+0000 7fe7cb7fe700 1 -- 172.21.15.49:0/318521978 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7c0011b20 con 0x7fe7cc1085a0 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.730+0000 7fe7d2c62700 1 -- 172.21.15.49:0/318521978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 msgr2=0x7fe7cc10a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.730+0000 7fe7d2c62700 1 --2- 172.21.15.49:0/318521978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc10a990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00098a0 tx=0x7fe7c0009bb0 comp rx=0 tx=0).stop 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.730+0000 7fe7d2c62700 1 -- 172.21.15.49:0/318521978 shutdown_connections 2024-03-18T00:41:04.670 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.730+0000 7fe7d2c62700 1 --2- 172.21.15.49:0/318521978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc10a990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.730+0000 7fe7d2c62700 1 -- 172.21.15.49:0/318521978 >> 172.21.15.49:0/318521978 conn(0x7fe7cc102030 msgr2=0x7fe7cc104450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.730+0000 7fe7d2c62700 1 -- 172.21.15.49:0/318521978 shutdown_connections 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.730+0000 7fe7d2c62700 1 -- 172.21.15.49:0/318521978 wait complete. 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.731+0000 7fe7d2c62700 1 Processor -- start 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.731+0000 7fe7d2c62700 1 -- start start 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.731+0000 7fe7d2c62700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc107e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.731+0000 7fe7d2c62700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe7cc10bf40 con 0x7fe7cc1085a0 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.731+0000 7fe7d09fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc107e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.731+0000 7fe7d09fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc107e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42874/0 (socket says 172.21.15.49:42874) 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.731+0000 7fe7d09fe700 1 -- 172.21.15.49:0/2650885556 learned_addr learned my addr 172.21.15.49:0/2650885556 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.731+0000 7fe7d09fe700 1 -- 172.21.15.49:0/2650885556 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7cc108360 con 0x7fe7cc1085a0 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.731+0000 7fe7d09fe700 1 --2- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc107e20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00116a0 tx=0x7fe7c0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.732+0000 7fe7c9ffb700 1 -- 172.21.15.49:0/2650885556 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7c0003a40 con 0x7fe7cc1085a0 2024-03-18T00:41:04.671 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.732+0000 7fe7c9ffb700 1 -- 172.21.15.49:0/2650885556 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7c0003be0 con 0x7fe7cc1085a0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.732+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7cc106550 con 0x7fe7cc1085a0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.732+0000 7fe7c9ffb700 1 -- 172.21.15.49:0/2650885556 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe7c001a750 con 0x7fe7cc1085a0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.732+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7cc106a10 con 0x7fe7cc1085a0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.733+0000 7fe7c9ffb700 1 -- 172.21.15.49:0/2650885556 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fe7c001aa00 con 0x7fe7cc1085a0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.733+0000 7fe7c9ffb700 1 --2- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7bc038e60 0x7fe7bc03b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.733+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7cc063900 con 0x7fe7cc1085a0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.733+0000 7fe7c9ffb700 1 -- 172.21.15.49:0/2650885556 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe7c0024aa0 con 0x7fe7cc1085a0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.733+0000 7fe7cbfff700 1 --2- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7bc038e60 0x7fe7bc03b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.734+0000 7fe7cbfff700 1 --2- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7bc038e60 0x7fe7bc03b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8006fd0 tx=0x7fe7b8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.738+0000 7fe7c9ffb700 1 -- 172.21.15.49:0/2650885556 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe7c0011e20 con 0x7fe7cc1085a0 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:03.962+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fe7cc1071f0 con 0x7fe7bc038e60 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.079+0000 7fe7c9ffb700 1 -- 172.21.15.49:0/2650885556 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fe7cc1071f0 con 0x7fe7bc038e60 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.081+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7bc038e60 msgr2=0x7fe7bc03b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:04.672 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.081+0000 7fe7d2c62700 1 --2- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7bc038e60 0x7fe7bc03b320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8006fd0 tx=0x7fe7b8006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:04.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.081+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 msgr2=0x7fe7cc107e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:04.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.081+0000 7fe7d2c62700 1 --2- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc107e20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00116a0 tx=0x7fe7c0011ba0 comp rx=0 tx=0).stop 2024-03-18T00:41:04.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.082+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 shutdown_connections 2024-03-18T00:41:04.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.082+0000 7fe7d2c62700 1 --2- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7bc038e60 0x7fe7bc03b320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:04.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.082+0000 7fe7d2c62700 1 --2- 172.21.15.49:0/2650885556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7cc1085a0 0x7fe7cc107e20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:04.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.082+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 >> 172.21.15.49:0/2650885556 conn(0x7fe7cc102030 msgr2=0x7fe7cc102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:04.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.082+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 shutdown_connections 2024-03-18T00:41:04.673 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:04.082+0000 7fe7d2c62700 1 -- 172.21.15.49:0/2650885556 wait complete. 2024-03-18T00:41:04.673 INFO:teuthology.orchestra.run.smithi049.stdout:Creating initial admin user... 2024-03-18T00:41:05.322 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:05 smithi049 ceph-mon[32130]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:41:05.323 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:05 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:05.323 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:05 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:05.323 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:05 smithi049 ceph-mon[32130]: mgrmap e12: x(active, since 2s) 2024-03-18T00:41:06.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$eeMwnPN1awlTx2kqpRZMlu/vNYzwh1S.dCJ9tFQABa54TGta5w9uG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710722465, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-18T00:41:06.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.082+0000 7ff3f4798700 1 Processor -- start 2024-03-18T00:41:06.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.082+0000 7ff3f4798700 1 -- start start 2024-03-18T00:41:06.118 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.082+0000 7ff3f4798700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec107f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.082+0000 7ff3f4798700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec108510 con 0x7ff3ec107b20 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.082+0000 7ff3f2534700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec107f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.082+0000 7ff3f2534700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec107f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42884/0 (socket says 172.21.15.49:42884) 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.082+0000 7ff3f2534700 1 -- 172.21.15.49:0/3385946934 learned_addr learned my addr 172.21.15.49:0/3385946934 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.083+0000 7ff3f2534700 1 -- 172.21.15.49:0/3385946934 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3ec108d20 con 0x7ff3ec107b20 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.083+0000 7ff3f2534700 1 --2- 172.21.15.49:0/3385946934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec107f40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80098a0 tx=0x7ff3e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f59e9d872b1f57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.083+0000 7ff3f1532700 1 -- 172.21.15.49:0/3385946934 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3e8004160 con 0x7ff3ec107b20 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.083+0000 7ff3f1532700 1 -- 172.21.15.49:0/3385946934 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3e8004300 con 0x7ff3ec107b20 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.083+0000 7ff3f1532700 1 -- 172.21.15.49:0/3385946934 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3e8011b20 con 0x7ff3ec107b20 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.083+0000 7ff3f4798700 1 -- 172.21.15.49:0/3385946934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 msgr2=0x7ff3ec107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 --2- 172.21.15.49:0/3385946934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec107f40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80098a0 tx=0x7ff3e8009bb0 comp rx=0 tx=0).stop 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 -- 172.21.15.49:0/3385946934 shutdown_connections 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 --2- 172.21.15.49:0/3385946934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec107f40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:06.119 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 -- 172.21.15.49:0/3385946934 >> 172.21.15.49:0/3385946934 conn(0x7ff3ec1030e0 msgr2=0x7ff3ec105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 -- 172.21.15.49:0/3385946934 shutdown_connections 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 -- 172.21.15.49:0/3385946934 wait complete. 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 Processor -- start 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 -- start start 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec19dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.084+0000 7ff3f4798700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec10c0d0 con 0x7ff3ec107b20 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3f2534700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec19dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3f2534700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec19dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42890/0 (socket says 172.21.15.49:42890) 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3f2534700 1 -- 172.21.15.49:0/1430814833 learned_addr learned my addr 172.21.15.49:0/1430814833 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3f2534700 1 -- 172.21.15.49:0/1430814833 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3ec19e1f0 con 0x7ff3ec107b20 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3f2534700 1 --2- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec19dcb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80117e0 tx=0x7ff3e8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3e37fe700 1 -- 172.21.15.49:0/1430814833 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3e8003bf0 con 0x7ff3ec107b20 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3e37fe700 1 -- 172.21.15.49:0/1430814833 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3e801a910 con 0x7ff3ec107b20 2024-03-18T00:41:06.120 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3e37fe700 1 -- 172.21.15.49:0/1430814833 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff3e80249a0 con 0x7ff3ec107b20 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3ec19e480 con 0x7ff3ec107b20 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.085+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3ec19e8e0 con 0x7ff3ec107b20 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.086+0000 7ff3e37fe700 1 -- 172.21.15.49:0/1430814833 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7ff3e8022020 con 0x7ff3ec107b20 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.086+0000 7ff3e37fe700 1 --2- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3d8038f20 0x7ff3d803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.086+0000 7ff3e37fe700 1 -- 172.21.15.49:0/1430814833 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff3e804d2d0 con 0x7ff3ec107b20 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.086+0000 7ff3f1d33700 1 --2- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3d8038f20 0x7ff3d803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.086+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3ec063900 con 0x7ff3ec107b20 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.087+0000 7ff3f1d33700 1 --2- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3d8038f20 0x7ff3d803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc006fd0 tx=0x7ff3dc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.091+0000 7ff3e37fe700 1 -- 172.21.15.49:0/1430814833 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff3e8011e20 con 0x7ff3ec107b20 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.321+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7ff3ec104210 con 0x7ff3d8038f20 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.569+0000 7ff3e37fe700 1 -- 172.21.15.49:0/1430814833 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7ff3ec104210 con 0x7ff3d8038f20 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3d8038f20 msgr2=0x7ff3d803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:06.121 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 --2- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3d8038f20 0x7ff3d803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc006fd0 tx=0x7ff3dc006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 msgr2=0x7ff3ec19dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 --2- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec19dcb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff3e80117e0 tx=0x7ff3e8011c10 comp rx=0 tx=0).stop 2024-03-18T00:41:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 shutdown_connections 2024-03-18T00:41:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 --2- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3d8038f20 0x7ff3d803b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 --2- 172.21.15.49:0/1430814833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff3ec107b20 0x7ff3ec19dcb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 >> 172.21.15.49:0/1430814833 conn(0x7ff3ec1030e0 msgr2=0x7ff3ec103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 shutdown_connections 2024-03-18T00:41:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:05.571+0000 7ff3f4798700 1 -- 172.21.15.49:0/1430814833 wait complete. 2024-03-18T00:41:06.122 INFO:teuthology.orchestra.run.smithi049.stdout:Fetching dashboard port number... 2024-03-18T00:41:06.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:06 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:06.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:06 smithi049 ceph-mon[32130]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:41:06.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:06 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:07.862 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2627104336' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T00:41:08.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stdout 8443 2024-03-18T00:41:08.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.144+0000 7fc97b1a1700 1 Processor -- start 2024-03-18T00:41:08.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.144+0000 7fc97b1a1700 1 -- start start 2024-03-18T00:41:08.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.145+0000 7fc97b1a1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc974107f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:08.152 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.145+0000 7fc97b1a1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc974108510 con 0x7fc974107b20 2024-03-18T00:41:08.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.145+0000 7fc978f3d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc974107f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:08.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.145+0000 7fc978f3d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc974107f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42892/0 (socket says 172.21.15.49:42892) 2024-03-18T00:41:08.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.145+0000 7fc978f3d700 1 -- 172.21.15.49:0/801075154 learned_addr learned my addr 172.21.15.49:0/801075154 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:08.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.145+0000 7fc978f3d700 1 -- 172.21.15.49:0/801075154 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc974108d20 con 0x7fc974107b20 2024-03-18T00:41:08.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.145+0000 7fc978f3d700 1 --2- 172.21.15.49:0/801075154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc974107f40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc9640098a0 tx=0x7fc964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea90a1d950a49677 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:08.153 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.145+0000 7fc9737fe700 1 -- 172.21.15.49:0/801075154 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc964004160 con 0x7fc974107b20 2024-03-18T00:41:08.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.145+0000 7fc9737fe700 1 -- 172.21.15.49:0/801075154 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc964004300 con 0x7fc974107b20 2024-03-18T00:41:08.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.146+0000 7fc9737fe700 1 -- 172.21.15.49:0/801075154 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc964011b20 con 0x7fc974107b20 2024-03-18T00:41:08.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.146+0000 7fc97b1a1700 1 -- 172.21.15.49:0/801075154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 msgr2=0x7fc974107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:08.154 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.146+0000 7fc97b1a1700 1 --2- 172.21.15.49:0/801075154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc974107f40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc9640098a0 tx=0x7fc964009bb0 comp rx=0 tx=0).stop 2024-03-18T00:41:08.155 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.146+0000 7fc97b1a1700 1 -- 172.21.15.49:0/801075154 shutdown_connections 2024-03-18T00:41:08.155 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.146+0000 7fc97b1a1700 1 --2- 172.21.15.49:0/801075154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc974107f40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:08.155 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.146+0000 7fc97b1a1700 1 -- 172.21.15.49:0/801075154 >> 172.21.15.49:0/801075154 conn(0x7fc9741030e0 msgr2=0x7fc974105500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:08.155 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.146+0000 7fc97b1a1700 1 -- 172.21.15.49:0/801075154 shutdown_connections 2024-03-18T00:41:08.155 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.146+0000 7fc97b1a1700 1 -- 172.21.15.49:0/801075154 wait complete. 2024-03-18T00:41:08.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.147+0000 7fc97b1a1700 1 Processor -- start 2024-03-18T00:41:08.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.147+0000 7fc97b1a1700 1 -- start start 2024-03-18T00:41:08.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.147+0000 7fc97b1a1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc97419ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:08.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.147+0000 7fc97b1a1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc97410c0d0 con 0x7fc974107b20 2024-03-18T00:41:08.156 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.147+0000 7fc978f3d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc97419ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:08.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.147+0000 7fc978f3d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc97419ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42900/0 (socket says 172.21.15.49:42900) 2024-03-18T00:41:08.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.147+0000 7fc978f3d700 1 -- 172.21.15.49:0/2627104336 learned_addr learned my addr 172.21.15.49:0/2627104336 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:08.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.147+0000 7fc978f3d700 1 -- 172.21.15.49:0/2627104336 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97419bf90 con 0x7fc974107b20 2024-03-18T00:41:08.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.147+0000 7fc978f3d700 1 --2- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc97419ba50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc9640117e0 tx=0x7fc964011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:08.157 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.148+0000 7fc971ffb700 1 -- 172.21.15.49:0/2627104336 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc964003bf0 con 0x7fc974107b20 2024-03-18T00:41:08.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.148+0000 7fc971ffb700 1 -- 172.21.15.49:0/2627104336 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc96401a910 con 0x7fc974107b20 2024-03-18T00:41:08.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.148+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97419c220 con 0x7fc974107b20 2024-03-18T00:41:08.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.148+0000 7fc971ffb700 1 -- 172.21.15.49:0/2627104336 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9640249a0 con 0x7fc974107b20 2024-03-18T00:41:08.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.148+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97419c680 con 0x7fc974107b20 2024-03-18T00:41:08.158 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.149+0000 7fc971ffb700 1 -- 172.21.15.49:0/2627104336 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fc964022020 con 0x7fc974107b20 2024-03-18T00:41:08.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.149+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc974063900 con 0x7fc974107b20 2024-03-18T00:41:08.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.149+0000 7fc971ffb700 1 --2- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc95c038f80 0x7fc95c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:08.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.149+0000 7fc971ffb700 1 -- 172.21.15.49:0/2627104336 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc96404d280 con 0x7fc974107b20 2024-03-18T00:41:08.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.149+0000 7fc973fff700 1 --2- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc95c038f80 0x7fc95c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:08.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.149+0000 7fc973fff700 1 --2- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc95c038f80 0x7fc95c03b440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc968006fd0 tx=0x7fc968006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:08.159 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.154+0000 7fc971ffb700 1 -- 172.21.15.49:0/2627104336 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc964011e20 con 0x7fc974107b20 2024-03-18T00:41:08.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.362+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fc97419f110 con 0x7fc974107b20 2024-03-18T00:41:08.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.362+0000 7fc971ffb700 1 -- 172.21.15.49:0/2627104336 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fc96400ba00 con 0x7fc974107b20 2024-03-18T00:41:08.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc95c038f80 msgr2=0x7fc95c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:08.160 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 --2- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc95c038f80 0x7fc95c03b440 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc968006fd0 tx=0x7fc968006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:08.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 msgr2=0x7fc97419ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:08.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 --2- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc97419ba50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc9640117e0 tx=0x7fc964011c10 comp rx=0 tx=0).stop 2024-03-18T00:41:08.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 shutdown_connections 2024-03-18T00:41:08.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 --2- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc95c038f80 0x7fc95c03b440 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:08.161 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 --2- 172.21.15.49:0/2627104336 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc974107b20 0x7fc97419ba50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:08.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 >> 172.21.15.49:0/2627104336 conn(0x7fc9741030e0 msgr2=0x7fc974103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:08.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 shutdown_connections 2024-03-18T00:41:08.162 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:07.364+0000 7fc97b1a1700 1 -- 172.21.15.49:0/2627104336 wait complete. 2024-03-18T00:41:08.166 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T00:41:08.166 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout disabled 2024-03-18T00:41:08.166 INFO:teuthology.orchestra.run.smithi049.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T00:41:08.167 INFO:teuthology.orchestra.run.smithi049.stdout:systemctl: stdout inactive 2024-03-18T00:41:08.167 INFO:teuthology.orchestra.run.smithi049.stdout:firewalld.service is not enabled 2024-03-18T00:41:08.167 INFO:teuthology.orchestra.run.smithi049.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-18T00:41:08.167 INFO:teuthology.orchestra.run.smithi049.stdout:Ceph Dashboard is now available at: 2024-03-18T00:41:08.167 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:08.167 INFO:teuthology.orchestra.run.smithi049.stdout: URL: https://smithi049.front.sepia.ceph.com:8443/ 2024-03-18T00:41:08.167 INFO:teuthology.orchestra.run.smithi049.stdout: User: admin 2024-03-18T00:41:08.168 INFO:teuthology.orchestra.run.smithi049.stdout: Password: xgamvy0j4q 2024-03-18T00:41:08.168 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:08.168 INFO:teuthology.orchestra.run.smithi049.stdout:Saving cluster configuration to /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config directory 2024-03-18T00:41:08.168 INFO:teuthology.orchestra.run.smithi049.stdout:Enabling autotune for osd_memory_target 2024-03-18T00:41:08.838 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:08 smithi049 ceph-mon[32130]: mgrmap e13: x(active, since 6s) 2024-03-18T00:41:09.368 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.626+0000 7f51f01c3700 1 Processor -- start 2024-03-18T00:41:09.368 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.626+0000 7f51f01c3700 1 -- start start 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.626+0000 7f51f01c3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e8106e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.626+0000 7f51f01c3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f51e81073d0 con 0x7f51e81069e0 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.626+0000 7f51edf5f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e8106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.626+0000 7f51edf5f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e8106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41386/0 (socket says 172.21.15.49:41386) 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.626+0000 7f51edf5f700 1 -- 172.21.15.49:0/1056269145 learned_addr learned my addr 172.21.15.49:0/1056269145 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.627+0000 7f51edf5f700 1 -- 172.21.15.49:0/1056269145 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e8107be0 con 0x7f51e81069e0 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.627+0000 7f51edf5f700 1 --2- 172.21.15.49:0/1056269145 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e8106e00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f51e40098a0 tx=0x7f51e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2e9756c8ea5531 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.627+0000 7f51ecf5d700 1 -- 172.21.15.49:0/1056269145 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e4004160 con 0x7f51e81069e0 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.627+0000 7f51ecf5d700 1 -- 172.21.15.49:0/1056269145 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e4004300 con 0x7f51e81069e0 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.627+0000 7f51ecf5d700 1 -- 172.21.15.49:0/1056269145 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e4011b20 con 0x7f51e81069e0 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.628+0000 7f51f01c3700 1 -- 172.21.15.49:0/1056269145 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 msgr2=0x7f51e8106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.628+0000 7f51f01c3700 1 --2- 172.21.15.49:0/1056269145 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e8106e00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f51e40098a0 tx=0x7f51e4009bb0 comp rx=0 tx=0).stop 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.628+0000 7f51f01c3700 1 -- 172.21.15.49:0/1056269145 shutdown_connections 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.628+0000 7f51f01c3700 1 --2- 172.21.15.49:0/1056269145 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e8106e00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:09.369 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.628+0000 7f51f01c3700 1 -- 172.21.15.49:0/1056269145 >> 172.21.15.49:0/1056269145 conn(0x7f51e8101fa0 msgr2=0x7f51e81043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.628+0000 7f51f01c3700 1 -- 172.21.15.49:0/1056269145 shutdown_connections 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.628+0000 7f51f01c3700 1 -- 172.21.15.49:0/1056269145 wait complete. 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.628+0000 7f51f01c3700 1 Processor -- start 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.628+0000 7f51f01c3700 1 -- start start 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51f01c3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e819ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51f01c3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f51e810af90 con 0x7f51e81069e0 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51edf5f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e819ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51edf5f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e819ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41396/0 (socket says 172.21.15.49:41396) 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51edf5f700 1 -- 172.21.15.49:0/2748412969 learned_addr learned my addr 172.21.15.49:0/2748412969 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51edf5f700 1 -- 172.21.15.49:0/2748412969 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e819bfb0 con 0x7f51e81069e0 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51edf5f700 1 --2- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e819ba70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f51e40117e0 tx=0x7f51e4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51deffd700 1 -- 172.21.15.49:0/2748412969 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e4003bf0 con 0x7f51e81069e0 2024-03-18T00:41:09.370 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51deffd700 1 -- 172.21.15.49:0/2748412969 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e401a910 con 0x7f51e81069e0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e819c240 con 0x7f51e81069e0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51deffd700 1 -- 172.21.15.49:0/2748412969 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e40249a0 con 0x7f51e81069e0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.629+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e819c6a0 con 0x7f51e81069e0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.630+0000 7f51deffd700 1 -- 172.21.15.49:0/2748412969 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f51e4022020 con 0x7f51e81069e0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.630+0000 7f51deffd700 1 --2- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51d4038fc0 0x7f51d403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.630+0000 7f51deffd700 1 -- 172.21.15.49:0/2748412969 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f51e404d3d0 con 0x7f51e81069e0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.630+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51e8063900 con 0x7f51e81069e0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.631+0000 7f51ed75e700 1 --2- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51d4038fc0 0x7f51d403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.631+0000 7f51ed75e700 1 --2- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51d4038fc0 0x7f51d403b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f51d8006fd0 tx=0x7f51d8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.636+0000 7f51deffd700 1 -- 172.21.15.49:0/2748412969 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51e4011e20 con 0x7f51e81069e0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.838+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f51e8062ee0 con 0x7f51e81069e0 2024-03-18T00:41:09.371 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.838+0000 7f51deffd700 1 -- 172.21.15.49:0/2748412969 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f51e4019c70 con 0x7f51e81069e0 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51d4038fc0 msgr2=0x7f51d403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 --2- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51d4038fc0 0x7f51d403b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f51d8006fd0 tx=0x7f51d8006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 msgr2=0x7f51e819ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 --2- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e819ba70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f51e40117e0 tx=0x7f51e4011c10 comp rx=0 tx=0).stop 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 shutdown_connections 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 --2- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51d4038fc0 0x7f51d403b480 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 --2- 172.21.15.49:0/2748412969 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51e81069e0 0x7f51e819ba70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 >> 172.21.15.49:0/2748412969 conn(0x7f51e8101fa0 msgr2=0x7f51e8102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 shutdown_connections 2024-03-18T00:41:09.372 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:08.840+0000 7f51f01c3700 1 -- 172.21.15.49:0/2748412969 wait complete. 2024-03-18T00:41:10.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.822+0000 7f793f0a7700 1 Processor -- start 2024-03-18T00:41:10.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.822+0000 7f793f0a7700 1 -- start start 2024-03-18T00:41:10.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.822+0000 7f793f0a7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793807c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:10.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.822+0000 7f793f0a7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f793807c870 con 0x7f793807de40 2024-03-18T00:41:10.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.822+0000 7f793ce43700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793807c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:10.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.822+0000 7f793ce43700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793807c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41402/0 (socket says 172.21.15.49:41402) 2024-03-18T00:41:10.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.822+0000 7f793ce43700 1 -- 172.21.15.49:0/166927982 learned_addr learned my addr 172.21.15.49:0/166927982 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:10.835 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.822+0000 7f793ce43700 1 -- 172.21.15.49:0/166927982 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793807c9f0 con 0x7f793807de40 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f793ce43700 1 --2- 172.21.15.49:0/166927982 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793807c2a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f792c0098a0 tx=0x7f792c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9187f390a32c31f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f79377fe700 1 -- 172.21.15.49:0/166927982 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f792c004160 con 0x7f793807de40 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f79377fe700 1 -- 172.21.15.49:0/166927982 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f792c004300 con 0x7f793807de40 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f79377fe700 1 -- 172.21.15.49:0/166927982 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f792c011b20 con 0x7f793807de40 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f793f0a7700 1 -- 172.21.15.49:0/166927982 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 msgr2=0x7f793807c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f793f0a7700 1 --2- 172.21.15.49:0/166927982 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793807c2a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f792c0098a0 tx=0x7f792c009bb0 comp rx=0 tx=0).stop 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f793f0a7700 1 -- 172.21.15.49:0/166927982 shutdown_connections 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f793f0a7700 1 --2- 172.21.15.49:0/166927982 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793807c2a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f793f0a7700 1 -- 172.21.15.49:0/166927982 >> 172.21.15.49:0/166927982 conn(0x7f7938103140 msgr2=0x7f7938105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f793f0a7700 1 -- 172.21.15.49:0/166927982 shutdown_connections 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.823+0000 7f793f0a7700 1 -- 172.21.15.49:0/166927982 wait complete. 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.824+0000 7f793f0a7700 1 Processor -- start 2024-03-18T00:41:10.836 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.824+0000 7f793f0a7700 1 -- start start 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.824+0000 7f793f0a7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793819dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.824+0000 7f793f0a7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f793810e1c0 con 0x7f793807de40 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.824+0000 7f793ce43700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793819dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.824+0000 7f793ce43700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793819dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41418/0 (socket says 172.21.15.49:41418) 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.824+0000 7f793ce43700 1 -- 172.21.15.49:0/190792704 learned_addr learned my addr 172.21.15.49:0/190792704 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.825+0000 7f793ce43700 1 -- 172.21.15.49:0/190792704 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793819e1d0 con 0x7f793807de40 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.825+0000 7f793ce43700 1 --2- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793819dc90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f792c011b60 tx=0x7f792c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.825+0000 7f7935ffb700 1 -- 172.21.15.49:0/190792704 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f792c003b60 con 0x7f793807de40 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.825+0000 7f7935ffb700 1 -- 172.21.15.49:0/190792704 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f792c01a910 con 0x7f793807de40 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.825+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f793819e460 con 0x7f793807de40 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.825+0000 7f7935ffb700 1 -- 172.21.15.49:0/190792704 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f792c024930 con 0x7f793807de40 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.825+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f793819e8c0 con 0x7f793807de40 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.826+0000 7f7935ffb700 1 -- 172.21.15.49:0/190792704 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f792c022020 con 0x7f793807de40 2024-03-18T00:41:10.837 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.826+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7938063900 con 0x7f793807de40 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.826+0000 7f7935ffb700 1 --2- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7928039010 0x7f792803b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.826+0000 7f7935ffb700 1 -- 172.21.15.49:0/190792704 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f792c04d270 con 0x7f793807de40 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.826+0000 7f7937fff700 1 --2- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7928039010 0x7f792803b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.827+0000 7f7937fff700 1 --2- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7928039010 0x7f792803b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7924006fd0 tx=0x7f7924006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:09.831+0000 7f7935ffb700 1 -- 172.21.15.49:0/190792704 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f792c00b3e0 con 0x7f793807de40 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.151+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f79381a13f0 con 0x7f793807de40 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.154+0000 7f7935ffb700 1 -- 172.21.15.49:0/190792704 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f792c016260 con 0x7f793807de40 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7928039010 msgr2=0x7f792803b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 --2- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7928039010 0x7f792803b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7924006fd0 tx=0x7f7924006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 msgr2=0x7f793819dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 --2- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793819dc90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f792c011b60 tx=0x7f792c011c40 comp rx=0 tx=0).stop 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 shutdown_connections 2024-03-18T00:41:10.838 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 --2- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7928039010 0x7f792803b4d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 --2- 172.21.15.49:0/190792704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f793807de40 0x7f793819dc90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 >> 172.21.15.49:0/190792704 conn(0x7f7938103140 msgr2=0x7f7938103be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 shutdown_connections 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr 2024-03-18T00:41:10.156+0000 7f793f0a7700 1 -- 172.21.15.49:0/190792704 wait complete. 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout:Or, if you are only running a single cluster on this host: 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:10.839 INFO:teuthology.orchestra.run.smithi049.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-18T00:41:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: ceph telemetry on 2024-03-18T00:41:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:10.840 INFO:teuthology.orchestra.run.smithi049.stdout:For more information see: 2024-03-18T00:41:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-18T00:41:10.840 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:10.840 INFO:teuthology.orchestra.run.smithi049.stdout:Bootstrap complete. 2024-03-18T00:41:10.886 INFO:tasks.cephadm:Fetching config... 2024-03-18T00:41:10.886 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:41:10.886 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-18T00:41:10.903 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-18T00:41:10.903 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:41:10.903 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-18T00:41:10.960 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-18T00:41:10.960 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:41:10.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-18T00:41:11.032 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-18T00:41:11.032 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:41:11.032 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-18T00:41:11.089 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-18T00:41:11.089 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwjYgkn9eaHBeNkCJ09UAkoQ+HAGO0WMi5hAtHqQ8PxgVCPYs20fljjIHw32TDMGR617jNqRBwyQz/c9ih1wwAM2/rHmPxo/H6awBVqsAxTq4CQCNWuaFsFkaoRNs5VdXDLpn46LyIIQD3oaxVlNP6x4Mx2K4Kr1T4SWYRWx5RGe1hbjBi+pSLLtG6JqOkG7owKiYUmq4oAFjAjh2NPGqdHLXQGXhSPC/lAFG1oRTD4oOgYRhgBPhIP1aVANJLMn3zTulLS7W3Qo6kDZu8j/oedvHpvYvecPuta4nO5AGwq/C7OBp9yvFbj7w8+jTtfr8HgxxxYOxiyX9j0XbBuUx0+3SOtH8XBt7qUmWfUbrCXPxgeC9PW2sU394Udotzysheto+MuLvajDu1bL/GAILq8j6cxOb0vlp3ubjFf8K74f3Yk9apm/9KU52ElsuS2CqVPHkaTew259lmnz7LhjJMi3Jq4ajF33tniMHytsknnfKMuemEBnuuJzg1IO98hZM= ceph-ec390cde-e4bf-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T00:41:11.178 INFO:teuthology.orchestra.run.smithi049.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwjYgkn9eaHBeNkCJ09UAkoQ+HAGO0WMi5hAtHqQ8PxgVCPYs20fljjIHw32TDMGR617jNqRBwyQz/c9ih1wwAM2/rHmPxo/H6awBVqsAxTq4CQCNWuaFsFkaoRNs5VdXDLpn46LyIIQD3oaxVlNP6x4Mx2K4Kr1T4SWYRWx5RGe1hbjBi+pSLLtG6JqOkG7owKiYUmq4oAFjAjh2NPGqdHLXQGXhSPC/lAFG1oRTD4oOgYRhgBPhIP1aVANJLMn3zTulLS7W3Qo6kDZu8j/oedvHpvYvecPuta4nO5AGwq/C7OBp9yvFbj7w8+jTtfr8HgxxxYOxiyX9j0XbBuUx0+3SOtH8XBt7qUmWfUbrCXPxgeC9PW2sU394Udotzysheto+MuLvajDu1bL/GAILq8j6cxOb0vlp3ubjFf8K74f3Yk9apm/9KU52ElsuS2CqVPHkaTew259lmnz7LhjJMi3Jq4ajF33tniMHytsknnfKMuemEBnuuJzg1IO98hZM= ceph-ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:41:11.195 DEBUG:teuthology.orchestra.run.smithi106:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwjYgkn9eaHBeNkCJ09UAkoQ+HAGO0WMi5hAtHqQ8PxgVCPYs20fljjIHw32TDMGR617jNqRBwyQz/c9ih1wwAM2/rHmPxo/H6awBVqsAxTq4CQCNWuaFsFkaoRNs5VdXDLpn46LyIIQD3oaxVlNP6x4Mx2K4Kr1T4SWYRWx5RGe1hbjBi+pSLLtG6JqOkG7owKiYUmq4oAFjAjh2NPGqdHLXQGXhSPC/lAFG1oRTD4oOgYRhgBPhIP1aVANJLMn3zTulLS7W3Qo6kDZu8j/oedvHpvYvecPuta4nO5AGwq/C7OBp9yvFbj7w8+jTtfr8HgxxxYOxiyX9j0XbBuUx0+3SOtH8XBt7qUmWfUbrCXPxgeC9PW2sU394Udotzysheto+MuLvajDu1bL/GAILq8j6cxOb0vlp3ubjFf8K74f3Yk9apm/9KU52ElsuS2CqVPHkaTew259lmnz7LhjJMi3Jq4ajF33tniMHytsknnfKMuemEBnuuJzg1IO98hZM= ceph-ec390cde-e4bf-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T00:41:11.243 INFO:teuthology.orchestra.run.smithi106.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwjYgkn9eaHBeNkCJ09UAkoQ+HAGO0WMi5hAtHqQ8PxgVCPYs20fljjIHw32TDMGR617jNqRBwyQz/c9ih1wwAM2/rHmPxo/H6awBVqsAxTq4CQCNWuaFsFkaoRNs5VdXDLpn46LyIIQD3oaxVlNP6x4Mx2K4Kr1T4SWYRWx5RGe1hbjBi+pSLLtG6JqOkG7owKiYUmq4oAFjAjh2NPGqdHLXQGXhSPC/lAFG1oRTD4oOgYRhgBPhIP1aVANJLMn3zTulLS7W3Qo6kDZu8j/oedvHpvYvecPuta4nO5AGwq/C7OBp9yvFbj7w8+jTtfr8HgxxxYOxiyX9j0XbBuUx0+3SOtH8XBt7qUmWfUbrCXPxgeC9PW2sU394Udotzysheto+MuLvajDu1bL/GAILq8j6cxOb0vlp3ubjFf8K74f3Yk9apm/9KU52ElsuS2CqVPHkaTew259lmnz7LhjJMi3Jq4ajF33tniMHytsknnfKMuemEBnuuJzg1IO98hZM= ceph-ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:41:11.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwjYgkn9eaHBeNkCJ09UAkoQ+HAGO0WMi5hAtHqQ8PxgVCPYs20fljjIHw32TDMGR617jNqRBwyQz/c9ih1wwAM2/rHmPxo/H6awBVqsAxTq4CQCNWuaFsFkaoRNs5VdXDLpn46LyIIQD3oaxVlNP6x4Mx2K4Kr1T4SWYRWx5RGe1hbjBi+pSLLtG6JqOkG7owKiYUmq4oAFjAjh2NPGqdHLXQGXhSPC/lAFG1oRTD4oOgYRhgBPhIP1aVANJLMn3zTulLS7W3Qo6kDZu8j/oedvHpvYvecPuta4nO5AGwq/C7OBp9yvFbj7w8+jTtfr8HgxxxYOxiyX9j0XbBuUx0+3SOtH8XBt7qUmWfUbrCXPxgeC9PW2sU394Udotzysheto+MuLvajDu1bL/GAILq8j6cxOb0vlp3ubjFf8K74f3Yk9apm/9KU52ElsuS2CqVPHkaTew259lmnz7LhjJMi3Jq4ajF33tniMHytsknnfKMuemEBnuuJzg1IO98hZM= ceph-ec390cde-e4bf-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T00:41:11.305 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwjYgkn9eaHBeNkCJ09UAkoQ+HAGO0WMi5hAtHqQ8PxgVCPYs20fljjIHw32TDMGR617jNqRBwyQz/c9ih1wwAM2/rHmPxo/H6awBVqsAxTq4CQCNWuaFsFkaoRNs5VdXDLpn46LyIIQD3oaxVlNP6x4Mx2K4Kr1T4SWYRWx5RGe1hbjBi+pSLLtG6JqOkG7owKiYUmq4oAFjAjh2NPGqdHLXQGXhSPC/lAFG1oRTD4oOgYRhgBPhIP1aVANJLMn3zTulLS7W3Qo6kDZu8j/oedvHpvYvecPuta4nO5AGwq/C7OBp9yvFbj7w8+jTtfr8HgxxxYOxiyX9j0XbBuUx0+3SOtH8XBt7qUmWfUbrCXPxgeC9PW2sU394Udotzysheto+MuLvajDu1bL/GAILq8j6cxOb0vlp3ubjFf8K74f3Yk9apm/9KU52ElsuS2CqVPHkaTew259lmnz7LhjJMi3Jq4ajF33tniMHytsknnfKMuemEBnuuJzg1IO98hZM= ceph-ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:41:11.321 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-18T00:41:11.568 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:41:11.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:11 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/190792704' entity='client.admin' 2024-03-18T00:41:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.081+0000 7f51fae95700 1 -- 172.21.15.49:0/3311593705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 msgr2=0x7f51f4101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.081+0000 7f51fae95700 1 --2- 172.21.15.49:0/3311593705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 0x7f51f4101020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009910 tx=0x7f51e8009c20 comp rx=0 tx=0).stop 2024-03-18T00:41:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.081+0000 7f51fae95700 1 -- 172.21.15.49:0/3311593705 shutdown_connections 2024-03-18T00:41:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.081+0000 7f51fae95700 1 --2- 172.21.15.49:0/3311593705 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 0x7f51f4101020 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.081+0000 7f51fae95700 1 -- 172.21.15.49:0/3311593705 >> 172.21.15.49:0/3311593705 conn(0x7f51f40fc1a0 msgr2=0x7f51f40fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.081+0000 7f51fae95700 1 -- 172.21.15.49:0/3311593705 shutdown_connections 2024-03-18T00:41:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.081+0000 7f51fae95700 1 -- 172.21.15.49:0/3311593705 wait complete. 2024-03-18T00:41:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.082+0000 7f51fae95700 1 Processor -- start 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.082+0000 7f51fae95700 1 -- start start 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.082+0000 7f51fae95700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 0x7f51f406a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.082+0000 7f51fae95700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f51f4106c90 con 0x7f51f4100c00 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.082+0000 7f51f8c31700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 0x7f51f406a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.082+0000 7f51f8c31700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 0x7f51f406a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41436/0 (socket says 172.21.15.49:41436) 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.082+0000 7f51f8c31700 1 -- 172.21.15.49:0/3056186061 learned_addr learned my addr 172.21.15.49:0/3056186061 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.082+0000 7f51f8c31700 1 -- 172.21.15.49:0/3056186061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f419a150 con 0x7f51f4100c00 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.082+0000 7f51f8c31700 1 --2- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 0x7f51f406a9d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f51e8006010 tx=0x7f51e8005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.083+0000 7f51f1ffb700 1 -- 172.21.15.49:0/3056186061 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e8027070 con 0x7f51f4100c00 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.083+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f419a320 con 0x7f51f4100c00 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.083+0000 7f51f1ffb700 1 -- 172.21.15.49:0/3056186061 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e800be40 con 0x7f51f4100c00 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.083+0000 7f51f1ffb700 1 -- 172.21.15.49:0/3056186061 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51e800f460 con 0x7f51f4100c00 2024-03-18T00:41:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.083+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f419a6c0 con 0x7f51f4100c00 2024-03-18T00:41:14.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.084+0000 7f51f1ffb700 1 -- 172.21.15.49:0/3056186061 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f51e802a070 con 0x7f51f4100c00 2024-03-18T00:41:14.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.084+0000 7f51f1ffb700 1 --2- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51e4038fc0 0x7f51e403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:14.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.084+0000 7f51f1ffb700 1 -- 172.21.15.49:0/3056186061 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f51e801e030 con 0x7f51f4100c00 2024-03-18T00:41:14.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.084+0000 7f51f3fff700 1 --2- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51e4038fc0 0x7f51e403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:14.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.084+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f4107410 con 0x7f51f4100c00 2024-03-18T00:41:14.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.085+0000 7f51f3fff700 1 --2- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51e4038fc0 0x7f51e403b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f51e0006fd0 tx=0x7f51e0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:14.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.094+0000 7f51f1ffb700 1 -- 172.21.15.49:0/3056186061 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f51e8013070 con 0x7f51f4100c00 2024-03-18T00:41:14.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:14.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:14.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi049", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:41:14.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:14.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:14.183 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:14.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.299+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f51f402dc80 con 0x7f51f4100c00 2024-03-18T00:41:14.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.302+0000 7f51f1ffb700 1 -- 172.21.15.49:0/3056186061 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f51e8031070 con 0x7f51f4100c00 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.304+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51e4038fc0 msgr2=0x7f51e403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.304+0000 7f51fae95700 1 --2- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51e4038fc0 0x7f51e403b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f51e0006fd0 tx=0x7f51e0006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.304+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 msgr2=0x7f51f406a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.304+0000 7f51fae95700 1 --2- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 0x7f51f406a9d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f51e8006010 tx=0x7f51e8005050 comp rx=0 tx=0).stop 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.304+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 shutdown_connections 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.304+0000 7f51fae95700 1 --2- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51e4038fc0 0x7f51e403b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.304+0000 7f51fae95700 1 --2- 172.21.15.49:0/3056186061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f51f4100c00 0x7f51f406a9d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.304+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 >> 172.21.15.49:0/3056186061 conn(0x7f51f40fc1a0 msgr2=0x7f51f40fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.305+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 shutdown_connections 2024-03-18T00:41:14.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:14.305+0000 7f51fae95700 1 -- 172.21.15.49:0/3056186061 wait complete. 2024-03-18T00:41:14.981 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-18T00:41:14.981 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-18T00:41:15.549 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:15 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3056186061' entity='client.admin' 2024-03-18T00:41:15.549 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:15.549 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:15.664 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:41:17.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.074+0000 7f79a2335700 1 -- 172.21.15.49:0/1887430548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 msgr2=0x7f799c102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:17.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.074+0000 7f79a2335700 1 --2- 172.21.15.49:0/1887430548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 0x7f799c102160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f798c009910 tx=0x7f798c009c20 comp rx=0 tx=0).stop 2024-03-18T00:41:17.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.074+0000 7f79a2335700 1 -- 172.21.15.49:0/1887430548 shutdown_connections 2024-03-18T00:41:17.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.074+0000 7f79a2335700 1 --2- 172.21.15.49:0/1887430548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 0x7f799c102160 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:17.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.074+0000 7f79a2335700 1 -- 172.21.15.49:0/1887430548 >> 172.21.15.49:0/1887430548 conn(0x7f799c0fd2c0 msgr2=0x7f799c0ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.074+0000 7f79a2335700 1 -- 172.21.15.49:0/1887430548 shutdown_connections 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.074+0000 7f79a2335700 1 -- 172.21.15.49:0/1887430548 wait complete. 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.074+0000 7f79a2335700 1 Processor -- start 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.074+0000 7f79a2335700 1 -- start start 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.075+0000 7f79a2335700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 0x7f799c197410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.075+0000 7f79a2335700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f799c107dd0 con 0x7f799c101d40 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.075+0000 7f799bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 0x7f799c197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.075+0000 7f799bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 0x7f799c197410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41468/0 (socket says 172.21.15.49:41468) 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.075+0000 7f799bfff700 1 -- 172.21.15.49:0/1736575966 learned_addr learned my addr 172.21.15.49:0/1736575966 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:17.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.075+0000 7f799bfff700 1 -- 172.21.15.49:0/1736575966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f799c197950 con 0x7f799c101d40 2024-03-18T00:41:17.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.075+0000 7f799bfff700 1 --2- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 0x7f799c197410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f798c005850 tx=0x7f798c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:17.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.076+0000 7f79997fa700 1 -- 172.21.15.49:0/1736575966 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f798c026070 con 0x7f799c101d40 2024-03-18T00:41:17.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.076+0000 7f79997fa700 1 -- 172.21.15.49:0/1736575966 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f798c0044a0 con 0x7f799c101d40 2024-03-18T00:41:17.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.076+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799c197be0 con 0x7f799c101d40 2024-03-18T00:41:17.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.076+0000 7f79997fa700 1 -- 172.21.15.49:0/1736575966 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f798c003ab0 con 0x7f799c101d40 2024-03-18T00:41:17.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.076+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f799c19a8d0 con 0x7f799c101d40 2024-03-18T00:41:17.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.077+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f799c108550 con 0x7f799c101d40 2024-03-18T00:41:17.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.077+0000 7f79997fa700 1 -- 172.21.15.49:0/1736575966 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f798c029070 con 0x7f799c101d40 2024-03-18T00:41:17.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.077+0000 7f79997fa700 1 --2- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7984038fc0 0x7f798403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:17.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.077+0000 7f79997fa700 1 -- 172.21.15.49:0/1736575966 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f798c01e070 con 0x7f799c101d40 2024-03-18T00:41:17.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.077+0000 7f799b7fe700 1 --2- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7984038fc0 0x7f798403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:17.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.077+0000 7f799b7fe700 1 --2- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7984038fc0 0x7f798403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7990006fd0 tx=0x7f7990006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:17.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.082+0000 7f79997fa700 1 -- 172.21.15.49:0/1736575966 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f798c013070 con 0x7f799c101d40 2024-03-18T00:41:17.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.292+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f799c02dc80 con 0x7f7984038fc0 2024-03-18T00:41:17.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.296+0000 7f79997fa700 1 -- 172.21.15.49:0/1736575966 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f799c02dc80 con 0x7f7984038fc0 2024-03-18T00:41:17.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7984038fc0 msgr2=0x7f798403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:17.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 --2- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7984038fc0 0x7f798403b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7990006fd0 tx=0x7f7990006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:17.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 msgr2=0x7f799c197410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:17.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 --2- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 0x7f799c197410 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f798c005850 tx=0x7f798c005030 comp rx=0 tx=0).stop 2024-03-18T00:41:17.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 shutdown_connections 2024-03-18T00:41:17.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 --2- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7984038fc0 0x7f798403b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:17.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 --2- 172.21.15.49:0/1736575966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f799c101d40 0x7f799c197410 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:17.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 >> 172.21.15.49:0/1736575966 conn(0x7f799c0fd2c0 msgr2=0x7f799c0fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:17.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 shutdown_connections 2024-03-18T00:41:17.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:17.298+0000 7f79a2335700 1 -- 172.21.15.49:0/1736575966 wait complete. 2024-03-18T00:41:18.019 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi106 2024-03-18T00:41:18.019 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:41:18.019 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/etc/ceph/ceph.conf 2024-03-18T00:41:18.038 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:41:18.038 DEBUG:teuthology.orchestra.run.smithi106:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:41:18.095 INFO:tasks.cephadm:Adding host smithi106 to orchestrator... 2024-03-18T00:41:18.095 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch host add smithi106 2024-03-18T00:41:18.348 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:41:18.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:41:18.566 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:18.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:18.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:18.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:41:18.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: Updating smithi049:/etc/ceph/ceph.conf 2024-03-18T00:41:18.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: Updating smithi049:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:41:18.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: Updating smithi049:/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:41:18.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: Updating smithi049:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T00:41:18.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:18.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:18.568 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:18 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:19.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.945+0000 7fccf6c89700 1 -- 172.21.15.49:0/586275937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 msgr2=0x7fccf01022f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:19.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.945+0000 7fccf6c89700 1 --2- 172.21.15.49:0/586275937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 0x7fccf01022f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009910 tx=0x7fcce0009c20 comp rx=0 tx=0).stop 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.945+0000 7fccf6c89700 1 -- 172.21.15.49:0/586275937 shutdown_connections 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.945+0000 7fccf6c89700 1 --2- 172.21.15.49:0/586275937 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 0x7fccf01022f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.945+0000 7fccf6c89700 1 -- 172.21.15.49:0/586275937 >> 172.21.15.49:0/586275937 conn(0x7fccf00fd7e0 msgr2=0x7fccf00ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.945+0000 7fccf6c89700 1 -- 172.21.15.49:0/586275937 shutdown_connections 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.945+0000 7fccf6c89700 1 -- 172.21.15.49:0/586275937 wait complete. 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.945+0000 7fccf6c89700 1 Processor -- start 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.945+0000 7fccf6c89700 1 -- start start 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccf6c89700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 0x7fccf00745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccf6c89700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fccf010c2e0 con 0x7fccf0101f10 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccf4a25700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 0x7fccf00745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:19.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccf4a25700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 0x7fccf00745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44016/0 (socket says 172.21.15.49:44016) 2024-03-18T00:41:19.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccf4a25700 1 -- 172.21.15.49:0/3733288810 learned_addr learned my addr 172.21.15.49:0/3733288810 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:19.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccf4a25700 1 -- 172.21.15.49:0/3733288810 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf00764c0 con 0x7fccf0101f10 2024-03-18T00:41:19.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccf4a25700 1 --2- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 0x7fccf00745d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcce0005850 tx=0x7fcce0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:19.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccedffb700 1 -- 172.21.15.49:0/3733288810 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcce0026070 con 0x7fccf0101f10 2024-03-18T00:41:19.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccedffb700 1 -- 172.21.15.49:0/3733288810 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcce000bc30 con 0x7fccf0101f10 2024-03-18T00:41:19.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf0074b70 con 0x7fccf0101f10 2024-03-18T00:41:19.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.946+0000 7fccedffb700 1 -- 172.21.15.49:0/3733288810 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcce0003a20 con 0x7fccf0101f10 2024-03-18T00:41:19.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.947+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf0075030 con 0x7fccf0101f10 2024-03-18T00:41:19.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.947+0000 7fccedffb700 1 -- 172.21.15.49:0/3733288810 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fcce0029070 con 0x7fccf0101f10 2024-03-18T00:41:19.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.947+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf01033f0 con 0x7fccf0101f10 2024-03-18T00:41:19.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.948+0000 7fccedffb700 1 --2- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd8039020 0x7fccd803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:19.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.948+0000 7fccedffb700 1 -- 172.21.15.49:0/3733288810 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcce001e070 con 0x7fccf0101f10 2024-03-18T00:41:19.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.948+0000 7fcceffff700 1 --2- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd8039020 0x7fccd803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:19.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.948+0000 7fcceffff700 1 --2- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd8039020 0x7fccd803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcce4006fd0 tx=0x7fcce4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:19.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:19.953+0000 7fccedffb700 1 -- 172.21.15.49:0/3733288810 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcce0013070 con 0x7fccf0101f10 2024-03-18T00:41:20.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:20.163+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi106", "target": ["mon-mgr", ""]}) -- 0x7fccf0075b00 con 0x7fccd8039020 2024-03-18T00:41:20.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:20 smithi049 ceph-mon[32130]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi106", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:41:21.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:21 smithi049 ceph-mon[32130]: Deploying cephadm binary to smithi106 2024-03-18T00:41:22.759 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:22 smithi049 ceph-mon[32130]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:24.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.417+0000 7fccedffb700 1 -- 172.21.15.49:0/3733288810 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fccf0075b00 con 0x7fccd8039020 2024-03-18T00:41:24.418 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi106' with addr '172.21.15.106' 2024-03-18T00:41:24.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.419+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd8039020 msgr2=0x7fccd803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:24.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.419+0000 7fccf6c89700 1 --2- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd8039020 0x7fccd803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcce4006fd0 tx=0x7fcce4006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:24.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.420+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 msgr2=0x7fccf00745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:24.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.420+0000 7fccf6c89700 1 --2- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 0x7fccf00745d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcce0005850 tx=0x7fcce0005030 comp rx=0 tx=0).stop 2024-03-18T00:41:24.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.420+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 shutdown_connections 2024-03-18T00:41:24.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.420+0000 7fccf6c89700 1 --2- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd8039020 0x7fccd803b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:24.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.420+0000 7fccf6c89700 1 --2- 172.21.15.49:0/3733288810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf0101f10 0x7fccf00745d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:24.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.420+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 >> 172.21.15.49:0/3733288810 conn(0x7fccf00fd7e0 msgr2=0x7fccf0104d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:24.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.420+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 shutdown_connections 2024-03-18T00:41:24.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:24.420+0000 7fccf6c89700 1 -- 172.21.15.49:0/3733288810 wait complete. 2024-03-18T00:41:24.728 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:24 smithi049 ceph-mon[32130]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:25.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T00:41:25.276 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:41:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:25 smithi049 ceph-mon[32130]: Added host smithi106 2024-03-18T00:41:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:25.715 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:26.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:26 smithi049 ceph-mon[32130]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:26.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.820+0000 7fba6d286700 1 -- 172.21.15.49:0/3304821303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 msgr2=0x7fba681022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.820+0000 7fba6d286700 1 --2- 172.21.15.49:0/3304821303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 0x7fba681022c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fba58009910 tx=0x7fba58009c20 comp rx=0 tx=0).stop 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.820+0000 7fba6d286700 1 -- 172.21.15.49:0/3304821303 shutdown_connections 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.820+0000 7fba6d286700 1 --2- 172.21.15.49:0/3304821303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 0x7fba681022c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.820+0000 7fba6d286700 1 -- 172.21.15.49:0/3304821303 >> 172.21.15.49:0/3304821303 conn(0x7fba680fd770 msgr2=0x7fba680ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.820+0000 7fba6d286700 1 -- 172.21.15.49:0/3304821303 shutdown_connections 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.820+0000 7fba6d286700 1 -- 172.21.15.49:0/3304821303 wait complete. 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.821+0000 7fba6d286700 1 Processor -- start 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.821+0000 7fba6d286700 1 -- start start 2024-03-18T00:41:26.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.821+0000 7fba6d286700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 0x7fba6819d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:26.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.821+0000 7fba6d286700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fba6810c2b0 con 0x7fba68101ee0 2024-03-18T00:41:26.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.821+0000 7fba66ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 0x7fba6819d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:26.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.821+0000 7fba66ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 0x7fba6819d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44028/0 (socket says 172.21.15.49:44028) 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.821+0000 7fba66ffd700 1 -- 172.21.15.49:0/2493415938 learned_addr learned my addr 172.21.15.49:0/2493415938 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.821+0000 7fba66ffd700 1 -- 172.21.15.49:0/2493415938 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6819d960 con 0x7fba68101ee0 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.822+0000 7fba66ffd700 1 --2- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 0x7fba6819d420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fba58006010 tx=0x7fba58005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.822+0000 7fba5ffff700 1 -- 172.21.15.49:0/2493415938 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba58026070 con 0x7fba68101ee0 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.822+0000 7fba5ffff700 1 -- 172.21.15.49:0/2493415938 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba5800bbd0 con 0x7fba68101ee0 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.822+0000 7fba5ffff700 1 -- 172.21.15.49:0/2493415938 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba58003910 con 0x7fba68101ee0 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.822+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6819dc50 con 0x7fba68101ee0 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.822+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba681975e0 con 0x7fba68101ee0 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.823+0000 7fba5ffff700 1 -- 172.21.15.49:0/2493415938 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fba58029070 con 0x7fba68101ee0 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.823+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba681033c0 con 0x7fba68101ee0 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.823+0000 7fba5ffff700 1 --2- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba54038fc0 0x7fba5403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:26.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.823+0000 7fba5ffff700 1 -- 172.21.15.49:0/2493415938 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fba5801e070 con 0x7fba68101ee0 2024-03-18T00:41:26.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.823+0000 7fba667fc700 1 --2- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba54038fc0 0x7fba5403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:26.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.824+0000 7fba667fc700 1 --2- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba54038fc0 0x7fba5403b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fba50006fd0 tx=0x7fba50006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:26.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:26.828+0000 7fba5ffff700 1 -- 172.21.15.49:0/2493415938 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fba58013070 con 0x7fba68101ee0 2024-03-18T00:41:27.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.044+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fba68198b10 con 0x7fba54038fc0 2024-03-18T00:41:27.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.045+0000 7fba5ffff700 1 -- 172.21.15.49:0/2493415938 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fba68198b10 con 0x7fba54038fc0 2024-03-18T00:41:27.045 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:27.045 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.49", "hostname": "smithi049", "labels": [], "status": ""}, {"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}] 2024-03-18T00:41:27.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba54038fc0 msgr2=0x7fba5403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:27.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 --2- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba54038fc0 0x7fba5403b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fba50006fd0 tx=0x7fba50006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:27.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 msgr2=0x7fba6819d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:27.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 --2- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 0x7fba6819d420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fba58006010 tx=0x7fba58005030 comp rx=0 tx=0).stop 2024-03-18T00:41:27.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 shutdown_connections 2024-03-18T00:41:27.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 --2- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba54038fc0 0x7fba5403b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:27.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 --2- 172.21.15.49:0/2493415938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba68101ee0 0x7fba6819d420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:27.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 >> 172.21.15.49:0/2493415938 conn(0x7fba680fd770 msgr2=0x7fba681038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:27.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 shutdown_connections 2024-03-18T00:41:27.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:27.047+0000 7fba6d286700 1 -- 172.21.15.49:0/2493415938 wait complete. 2024-03-18T00:41:27.672 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi107 2024-03-18T00:41:27.672 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:41:27.672 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/etc/ceph/ceph.conf 2024-03-18T00:41:27.690 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:41:27.691 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:41:27.748 INFO:tasks.cephadm:Adding host smithi107 to orchestrator... 2024-03-18T00:41:27.748 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch host add smithi107 2024-03-18T00:41:27.998 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:41:28.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:27 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:28.222 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:27 smithi049 ceph-mon[32130]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:41:29.139 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:28 smithi049 ceph-mon[32130]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:29.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.326+0000 7f6405a8b700 1 -- 172.21.15.49:0/44347799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 msgr2=0x7f6400103300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:29.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.326+0000 7f6405a8b700 1 --2- 172.21.15.49:0/44347799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 0x7f6400103300 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f63f0009910 tx=0x7f63f0009c20 comp rx=0 tx=0).stop 2024-03-18T00:41:29.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.326+0000 7f6405a8b700 1 -- 172.21.15.49:0/44347799 shutdown_connections 2024-03-18T00:41:29.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.326+0000 7f6405a8b700 1 --2- 172.21.15.49:0/44347799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 0x7f6400103300 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:29.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.326+0000 7f6405a8b700 1 -- 172.21.15.49:0/44347799 >> 172.21.15.49:0/44347799 conn(0x7f6400077810 msgr2=0x7f64000feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:29.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.326+0000 7f6405a8b700 1 -- 172.21.15.49:0/44347799 shutdown_connections 2024-03-18T00:41:29.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.326+0000 7f6405a8b700 1 -- 172.21.15.49:0/44347799 wait complete. 2024-03-18T00:41:29.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.327+0000 7f6405a8b700 1 Processor -- start 2024-03-18T00:41:29.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.327+0000 7f6405a8b700 1 -- start start 2024-03-18T00:41:29.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.327+0000 7f6405a8b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 0x7f640019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:29.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.327+0000 7f6405a8b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f640010c340 con 0x7f6400102f20 2024-03-18T00:41:29.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.327+0000 7f63ff7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 0x7f640019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:29.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.327+0000 7f63ff7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 0x7f640019d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:51802/0 (socket says 172.21.15.49:51802) 2024-03-18T00:41:29.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.327+0000 7f63ff7fe700 1 -- 172.21.15.49:0/277382711 learned_addr learned my addr 172.21.15.49:0/277382711 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:29.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.327+0000 7f63ff7fe700 1 -- 172.21.15.49:0/277382711 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f640019db10 con 0x7f6400102f20 2024-03-18T00:41:29.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.327+0000 7f63ff7fe700 1 --2- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 0x7f640019d5d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f63f0006010 tx=0x7f63f0005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:29.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.328+0000 7f63fcff9700 1 -- 172.21.15.49:0/277382711 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63f0026070 con 0x7f6400102f20 2024-03-18T00:41:29.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.328+0000 7f63fcff9700 1 -- 172.21.15.49:0/277382711 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63f000bc30 con 0x7f6400102f20 2024-03-18T00:41:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.328+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f640019dda0 con 0x7f6400102f20 2024-03-18T00:41:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.328+0000 7f63fcff9700 1 -- 172.21.15.49:0/277382711 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f63f0003ab0 con 0x7f6400102f20 2024-03-18T00:41:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.328+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64001976a0 con 0x7f6400102f20 2024-03-18T00:41:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.329+0000 7f63fcff9700 1 -- 172.21.15.49:0/277382711 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f63f0029070 con 0x7f6400102f20 2024-03-18T00:41:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.329+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6400069ee0 con 0x7f6400102f20 2024-03-18T00:41:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.329+0000 7f63fcff9700 1 --2- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f63e8039020 0x7f63e803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.329+0000 7f63fcff9700 1 -- 172.21.15.49:0/277382711 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f63f001e070 con 0x7f6400102f20 2024-03-18T00:41:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.329+0000 7f63feffd700 1 --2- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f63e8039020 0x7f63e803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:29.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.329+0000 7f63feffd700 1 --2- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f63e8039020 0x7f63e803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f63f4006fd0 tx=0x7f63f4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:29.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.334+0000 7f63fcff9700 1 -- 172.21.15.49:0/277382711 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f63f0013070 con 0x7f6400102f20 2024-03-18T00:41:29.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:29.544+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi107", "target": ["mon-mgr", ""]}) -- 0x7f6400198bd0 con 0x7f63e8039020 2024-03-18T00:41:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:41:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:41:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:41:31.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:31 smithi049 ceph-mon[32130]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-18T00:41:32.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:32 smithi049 ceph-mon[32130]: Deploying cephadm binary to smithi107 2024-03-18T00:41:32.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:32 smithi049 ceph-mon[32130]: Updating smithi106:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:41:32.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:32 smithi049 ceph-mon[32130]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:41:32.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:32 smithi049 ceph-mon[32130]: Updating smithi106:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T00:41:33.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:33 smithi049 ceph-mon[32130]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.902+0000 7f63fcff9700 1 -- 172.21.15.49:0/277382711 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f6400198bd0 con 0x7f63e8039020 2024-03-18T00:41:33.902 INFO:teuthology.orchestra.run.smithi049.stdout:Added host 'smithi107' with addr '172.21.15.107' 2024-03-18T00:41:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f63e8039020 msgr2=0x7f63e803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 --2- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f63e8039020 0x7f63e803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f63f4006fd0 tx=0x7f63f4006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 msgr2=0x7f640019d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 --2- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 0x7f640019d5d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f63f0006010 tx=0x7f63f0005050 comp rx=0 tx=0).stop 2024-03-18T00:41:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 shutdown_connections 2024-03-18T00:41:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 --2- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f63e8039020 0x7f63e803b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 --2- 172.21.15.49:0/277382711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6400102f20 0x7f640019d5d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 >> 172.21.15.49:0/277382711 conn(0x7f6400077810 msgr2=0x7f6400103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:33.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 shutdown_connections 2024-03-18T00:41:33.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:33.905+0000 7f6405a8b700 1 -- 172.21.15.49:0/277382711 wait complete. 2024-03-18T00:41:34.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:34 smithi049 ceph-mon[32130]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:34.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:34.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:34.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:34.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:34.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:34 smithi049 ceph-mon[32130]: Added host smithi107 2024-03-18T00:41:34.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:34.614 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:34.647 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T00:41:34.898 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:41:36.206 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:36.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.435+0000 7f7d7fb86700 1 -- 172.21.15.49:0/3182911783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 msgr2=0x7f7d78101220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:36.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.435+0000 7f7d7fb86700 1 --2- 172.21.15.49:0/3182911783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 0x7f7d78101220 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d68009910 tx=0x7f7d68009c20 comp rx=0 tx=0).stop 2024-03-18T00:41:36.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.435+0000 7f7d7fb86700 1 -- 172.21.15.49:0/3182911783 shutdown_connections 2024-03-18T00:41:36.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.435+0000 7f7d7fb86700 1 --2- 172.21.15.49:0/3182911783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 0x7f7d78101220 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:36.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.435+0000 7f7d7fb86700 1 -- 172.21.15.49:0/3182911783 >> 172.21.15.49:0/3182911783 conn(0x7f7d780fc710 msgr2=0x7f7d780feb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:36.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.435+0000 7f7d7fb86700 1 -- 172.21.15.49:0/3182911783 shutdown_connections 2024-03-18T00:41:36.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.435+0000 7f7d7fb86700 1 -- 172.21.15.49:0/3182911783 wait complete. 2024-03-18T00:41:36.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.435+0000 7f7d7fb86700 1 Processor -- start 2024-03-18T00:41:36.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.435+0000 7f7d7fb86700 1 -- start start 2024-03-18T00:41:36.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d7fb86700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 0x7f7d781028e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d7fb86700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7810b210 con 0x7f7d78100e40 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d7d922700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 0x7f7d781028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d7d922700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 0x7f7d781028e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:51828/0 (socket says 172.21.15.49:51828) 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d7d922700 1 -- 172.21.15.49:0/910840 learned_addr learned my addr 172.21.15.49:0/910840 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d7d922700 1 -- 172.21.15.49:0/910840 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d781047d0 con 0x7f7d78100e40 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d7d922700 1 --2- 172.21.15.49:0/910840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 0x7f7d781028e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7d68006010 tx=0x7f7d68005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d6effd700 1 -- 172.21.15.49:0/910840 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d68026070 con 0x7f7d78100e40 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d6effd700 1 -- 172.21.15.49:0/910840 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d6800bc30 con 0x7f7d78100e40 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.436+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d78102e80 con 0x7f7d78100e40 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.437+0000 7f7d6effd700 1 -- 172.21.15.49:0/910840 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d680039d0 con 0x7f7d78100e40 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.437+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d78103340 con 0x7f7d78100e40 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.437+0000 7f7d6effd700 1 -- 172.21.15.49:0/910840 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f7d68029070 con 0x7f7d78100e40 2024-03-18T00:41:36.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.437+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d78102320 con 0x7f7d78100e40 2024-03-18T00:41:36.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.437+0000 7f7d6effd700 1 --2- 172.21.15.49:0/910840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d64038fc0 0x7f7d6403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:36.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.438+0000 7f7d6effd700 1 -- 172.21.15.49:0/910840 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7d6801e070 con 0x7f7d78100e40 2024-03-18T00:41:36.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.438+0000 7f7d7d121700 1 --2- 172.21.15.49:0/910840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d64038fc0 0x7f7d6403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:36.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.438+0000 7f7d7d121700 1 --2- 172.21.15.49:0/910840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d64038fc0 0x7f7d6403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006fd0 tx=0x7f7d74006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:36.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.443+0000 7f7d6effd700 1 -- 172.21.15.49:0/910840 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d68013070 con 0x7f7d78100e40 2024-03-18T00:41:36.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.652+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7d78103e10 con 0x7f7d64038fc0 2024-03-18T00:41:36.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.653+0000 7f7d6effd700 1 -- 172.21.15.49:0/910840 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f7d78103e10 con 0x7f7d64038fc0 2024-03-18T00:41:36.654 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:41:36.654 INFO:teuthology.orchestra.run.smithi049.stdout:[{"addr": "172.21.15.49", "hostname": "smithi049", "labels": [], "status": ""}, {"addr": "172.21.15.106", "hostname": "smithi106", "labels": [], "status": ""}, {"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}] 2024-03-18T00:41:36.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d64038fc0 msgr2=0x7f7d6403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:36.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 --2- 172.21.15.49:0/910840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d64038fc0 0x7f7d6403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7d74006fd0 tx=0x7f7d74006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:36.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 msgr2=0x7f7d781028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:36.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 --2- 172.21.15.49:0/910840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 0x7f7d781028e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7d68006010 tx=0x7f7d68005030 comp rx=0 tx=0).stop 2024-03-18T00:41:36.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 shutdown_connections 2024-03-18T00:41:36.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 --2- 172.21.15.49:0/910840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d64038fc0 0x7f7d6403b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:36.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 --2- 172.21.15.49:0/910840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d78100e40 0x7f7d781028e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:36.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 >> 172.21.15.49:0/910840 conn(0x7f7d780fc710 msgr2=0x7f7d7806a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:36.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 shutdown_connections 2024-03-18T00:41:36.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:36.655+0000 7f7d7fb86700 1 -- 172.21.15.49:0/910840 wait complete. 2024-03-18T00:41:37.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:36 smithi049 ceph-mon[32130]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:37.227 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:36 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:37.255 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-18T00:41:37.255 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-18T00:41:37.503 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:41:38.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:37 smithi049 ceph-mon[32130]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:41:38.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.874+0000 7fc2aef67700 1 -- 172.21.15.49:0/3534031669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 msgr2=0x7fc2a81022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:38.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.874+0000 7fc2aef67700 1 --2- 172.21.15.49:0/3534031669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 0x7fc2a81022c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009910 tx=0x7fc29c009c20 comp rx=0 tx=0).stop 2024-03-18T00:41:38.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.874+0000 7fc2aef67700 1 -- 172.21.15.49:0/3534031669 shutdown_connections 2024-03-18T00:41:38.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.874+0000 7fc2aef67700 1 --2- 172.21.15.49:0/3534031669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 0x7fc2a81022c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:38.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.874+0000 7fc2aef67700 1 -- 172.21.15.49:0/3534031669 >> 172.21.15.49:0/3534031669 conn(0x7fc2a80fd7b0 msgr2=0x7fc2a80ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:38.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.874+0000 7fc2aef67700 1 -- 172.21.15.49:0/3534031669 shutdown_connections 2024-03-18T00:41:38.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.874+0000 7fc2aef67700 1 -- 172.21.15.49:0/3534031669 wait complete. 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.875+0000 7fc2aef67700 1 Processor -- start 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.875+0000 7fc2aef67700 1 -- start start 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.875+0000 7fc2aef67700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 0x7fc2a819d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.875+0000 7fc2aef67700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a810c2b0 con 0x7fc2a8101ee0 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.875+0000 7fc2acd03700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 0x7fc2a819d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.875+0000 7fc2acd03700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 0x7fc2a819d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33294/0 (socket says 172.21.15.49:33294) 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.875+0000 7fc2acd03700 1 -- 172.21.15.49:0/1364468275 learned_addr learned my addr 172.21.15.49:0/1364468275 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.876+0000 7fc2acd03700 1 -- 172.21.15.49:0/1364468275 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a819daf0 con 0x7fc2a8101ee0 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.876+0000 7fc2acd03700 1 --2- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 0x7fc2a819d5b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc29c005850 tx=0x7fc29c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.876+0000 7fc2a5ffb700 1 -- 172.21.15.49:0/1364468275 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc29c026070 con 0x7fc2a8101ee0 2024-03-18T00:41:38.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.876+0000 7fc2a5ffb700 1 -- 172.21.15.49:0/1364468275 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc29c00bbd0 con 0x7fc2a8101ee0 2024-03-18T00:41:38.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.876+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a819dde0 con 0x7fc2a8101ee0 2024-03-18T00:41:38.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.876+0000 7fc2a5ffb700 1 -- 172.21.15.49:0/1364468275 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc29c003970 con 0x7fc2a8101ee0 2024-03-18T00:41:38.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.876+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a8197850 con 0x7fc2a8101ee0 2024-03-18T00:41:38.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.877+0000 7fc2a5ffb700 1 -- 172.21.15.49:0/1364468275 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fc29c029070 con 0x7fc2a8101ee0 2024-03-18T00:41:38.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.877+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2a81033c0 con 0x7fc2a8101ee0 2024-03-18T00:41:38.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.877+0000 7fc2a5ffb700 1 --2- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc298039020 0x7fc29803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:38.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.877+0000 7fc2a5ffb700 1 -- 172.21.15.49:0/1364468275 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc29c01e070 con 0x7fc2a8101ee0 2024-03-18T00:41:38.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.878+0000 7fc2a7fff700 1 --2- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc298039020 0x7fc29803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:38.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.878+0000 7fc2a7fff700 1 --2- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc298039020 0x7fc29803b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc294006fd0 tx=0x7fc294006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:38.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:38.883+0000 7fc2a5ffb700 1 -- 172.21.15.49:0/1364468275 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc29c013070 con 0x7fc2a8101ee0 2024-03-18T00:41:39.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.095+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc2a8198bb0 con 0x7fc2a8101ee0 2024-03-18T00:41:39.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:38 smithi049 ceph-mon[32130]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:39.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.949+0000 7fc2a5ffb700 1 -- 172.21.15.49:0/1364468275 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fc29c00fe20 con 0x7fc2a8101ee0 2024-03-18T00:41:39.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc298039020 msgr2=0x7fc29803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 --2- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc298039020 0x7fc29803b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc294006fd0 tx=0x7fc294006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 msgr2=0x7fc2a819d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 --2- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 0x7fc2a819d5b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc29c005850 tx=0x7fc29c005030 comp rx=0 tx=0).stop 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 shutdown_connections 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 --2- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc298039020 0x7fc29803b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 --2- 172.21.15.49:0/1364468275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc2a8101ee0 0x7fc2a819d5b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 >> 172.21.15.49:0/1364468275 conn(0x7fc2a80fd7b0 msgr2=0x7fc2a8104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 shutdown_connections 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:41:39.952+0000 7fc2aef67700 1 -- 172.21.15.49:0/1364468275 wait complete. 2024-03-18T00:41:39.954 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2024-03-18T00:41:40.265 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:39 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1364468275' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T00:41:40.265 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:40.265 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:40.265 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:40.265 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:40.265 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:41:40.265 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:40.266 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:41:40.266 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:39 smithi049 ceph-mon[32130]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-18T00:41:40.543 INFO:tasks.cephadm:Adding mon.a on smithi049 2024-03-18T00:41:40.543 INFO:tasks.cephadm:Adding mon.b on smithi106 2024-03-18T00:41:40.543 INFO:tasks.cephadm:Adding mon.c on smithi107 2024-03-18T00:41:40.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi049:172.21.15.49=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c' 2024-03-18T00:41:40.774 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:41:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:40 smithi049 ceph-mon[32130]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:40 smithi049 ceph-mon[32130]: Updating smithi107:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:41:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:40 smithi049 ceph-mon[32130]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:41:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:40 smithi049 ceph-mon[32130]: Updating smithi107:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T00:41:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:40 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1364468275' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T00:41:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:40 smithi049 ceph-mon[32130]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:41:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:42.268 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.267+0000 7efc96cdd700 1 -- 172.21.15.107:0/3198895251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 msgr2=0x7efc901022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.267+0000 7efc96cdd700 1 --2- 172.21.15.107:0/3198895251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 0x7efc901022e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efc84009910 tx=0x7efc84009c20 comp rx=0 tx=0).stop 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.267+0000 7efc96cdd700 1 -- 172.21.15.107:0/3198895251 shutdown_connections 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.267+0000 7efc96cdd700 1 --2- 172.21.15.107:0/3198895251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 0x7efc901022e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.267+0000 7efc96cdd700 1 -- 172.21.15.107:0/3198895251 >> 172.21.15.107:0/3198895251 conn(0x7efc900fd810 msgr2=0x7efc900ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.267+0000 7efc96cdd700 1 -- 172.21.15.107:0/3198895251 shutdown_connections 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.267+0000 7efc96cdd700 1 -- 172.21.15.107:0/3198895251 wait complete. 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.268+0000 7efc96cdd700 1 Processor -- start 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.268+0000 7efc96cdd700 1 -- start start 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.268+0000 7efc96cdd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 0x7efc900745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.268+0000 7efc96cdd700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efc9010c2d0 con 0x7efc90101f00 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.268+0000 7efc94a79700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 0x7efc900745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:42.269 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.268+0000 7efc94a79700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 0x7efc900745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.107:38930/0 (socket says 172.21.15.107:38930) 2024-03-18T00:41:42.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.268+0000 7efc94a79700 1 -- 172.21.15.107:0/261782244 learned_addr learned my addr 172.21.15.107:0/261782244 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:41:42.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.268+0000 7efc94a79700 1 -- 172.21.15.107:0/261782244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc900764c0 con 0x7efc90101f00 2024-03-18T00:41:42.270 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.269+0000 7efc94a79700 1 --2- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 0x7efc900745d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc84005850 tx=0x7efc84005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.269+0000 7efc8dffb700 1 -- 172.21.15.107:0/261782244 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc84026070 con 0x7efc90101f00 2024-03-18T00:41:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.269+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc90074b70 con 0x7efc90101f00 2024-03-18T00:41:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.269+0000 7efc8dffb700 1 -- 172.21.15.107:0/261782244 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc8400bc30 con 0x7efc90101f00 2024-03-18T00:41:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.269+0000 7efc8dffb700 1 -- 172.21.15.107:0/261782244 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efc84003a20 con 0x7efc90101f00 2024-03-18T00:41:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.269+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc90075030 con 0x7efc90101f00 2024-03-18T00:41:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.270+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc901033e0 con 0x7efc90101f00 2024-03-18T00:41:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.270+0000 7efc8dffb700 1 -- 172.21.15.107:0/261782244 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7efc84029070 con 0x7efc90101f00 2024-03-18T00:41:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.270+0000 7efc8dffb700 1 --2- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc80038fc0 0x7efc8003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.270+0000 7efc8dffb700 1 -- 172.21.15.107:0/261782244 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7efc8401e070 con 0x7efc90101f00 2024-03-18T00:41:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.270+0000 7efc8ffff700 1 --2- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc80038fc0 0x7efc8003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:42.272 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.271+0000 7efc8ffff700 1 --2- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc80038fc0 0x7efc8003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc7c006fd0 tx=0x7efc7c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:42.275 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.275+0000 7efc8dffb700 1 -- 172.21.15.107:0/261782244 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efc84013070 con 0x7efc90101f00 2024-03-18T00:41:42.485 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.484+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c", "target": ["mon-mgr", ""]}) -- 0x7efc90075b00 con 0x7efc80038fc0 2024-03-18T00:41:42.491 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.491+0000 7efc8dffb700 1 -- 172.21.15.107:0/261782244 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7efc90075b00 con 0x7efc80038fc0 2024-03-18T00:41:42.491 INFO:teuthology.orchestra.run.smithi107.stdout:Scheduled mon update... 2024-03-18T00:41:42.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc80038fc0 msgr2=0x7efc8003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:42.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 --2- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc80038fc0 0x7efc8003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efc7c006fd0 tx=0x7efc7c006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:42.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 msgr2=0x7efc900745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:42.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 --2- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 0x7efc900745d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efc84005850 tx=0x7efc84005030 comp rx=0 tx=0).stop 2024-03-18T00:41:42.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 shutdown_connections 2024-03-18T00:41:42.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 --2- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc80038fc0 0x7efc8003b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:42.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 --2- 172.21.15.107:0/261782244 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc90101f00 0x7efc900745d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:42.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 >> 172.21.15.107:0/261782244 conn(0x7efc900fd810 msgr2=0x7efc901038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:42.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 shutdown_connections 2024-03-18T00:41:42.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:42.493+0000 7efc96cdd700 1 -- 172.21.15.107:0/261782244 wait complete. 2024-03-18T00:41:43.095 DEBUG:teuthology.orchestra.run.smithi106:mon.b> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.b.service 2024-03-18T00:41:43.098 DEBUG:teuthology.orchestra.run.smithi107:mon.c> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.c.service 2024-03-18T00:41:43.100 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T00:41:43.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T00:41:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:42 smithi049 ceph-mon[32130]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:42 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:42 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:42 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:42 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:41:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:42 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:42 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:41:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:42 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:44.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:43 smithi049 ceph-mon[32130]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:41:44.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:43 smithi049 ceph-mon[32130]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c;count:3 2024-03-18T00:41:44.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:43 smithi049 ceph-mon[32130]: Deploying daemon mon.c on smithi107 2024-03-18T00:41:45.075 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:41:45.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:44 smithi049 ceph-mon[32130]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:46.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:46 smithi107 systemd[1]: Starting Ceph mon.c for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.771+0000 7f8837332700 1 -- 172.21.15.107:0/3144726609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 msgr2=0x7f88301012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.771+0000 7f8837332700 1 --2- 172.21.15.107:0/3144726609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 0x7f88301012c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f882800b0a0 tx=0x7f882800b3b0 comp rx=0 tx=0).stop 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.771+0000 7f8837332700 1 -- 172.21.15.107:0/3144726609 shutdown_connections 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.771+0000 7f8837332700 1 --2- 172.21.15.107:0/3144726609 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 0x7f88301012c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.771+0000 7f8837332700 1 -- 172.21.15.107:0/3144726609 >> 172.21.15.107:0/3144726609 conn(0x7f88300fc770 msgr2=0x7f88300feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.772+0000 7f8837332700 1 -- 172.21.15.107:0/3144726609 shutdown_connections 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.772+0000 7f8837332700 1 -- 172.21.15.107:0/3144726609 wait complete. 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.772+0000 7f8837332700 1 Processor -- start 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.772+0000 7f8837332700 1 -- start start 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.772+0000 7f8837332700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 0x7f883019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.772+0000 7f8837332700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f883010b2b0 con 0x7f8830100ee0 2024-03-18T00:41:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.773+0000 7f88350ce700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 0x7f883019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.773+0000 7f88350ce700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 0x7f883019d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.107:33300/0 (socket says 172.21.15.107:33300) 2024-03-18T00:41:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.773+0000 7f88350ce700 1 -- 172.21.15.107:0/637976017 learned_addr learned my addr 172.21.15.107:0/637976017 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:41:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.773+0000 7f88350ce700 1 -- 172.21.15.107:0/637976017 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883019db10 con 0x7f8830100ee0 2024-03-18T00:41:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.773+0000 7f88350ce700 1 --2- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 0x7f883019d5d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8828015040 tx=0x7f8828003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.773+0000 7f88267fc700 1 -- 172.21.15.107:0/637976017 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8828017040 con 0x7f8830100ee0 2024-03-18T00:41:46.775 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.774+0000 7f88267fc700 1 -- 172.21.15.107:0/637976017 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8828022070 con 0x7f8830100ee0 2024-03-18T00:41:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.774+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f883019dda0 con 0x7f8830100ee0 2024-03-18T00:41:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.774+0000 7f88267fc700 1 -- 172.21.15.107:0/637976017 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f882801d5d0 con 0x7f8830100ee0 2024-03-18T00:41:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.774+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88301976a0 con 0x7f8830100ee0 2024-03-18T00:41:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.775+0000 7f88267fc700 1 -- 172.21.15.107:0/637976017 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f882801d8d0 con 0x7f8830100ee0 2024-03-18T00:41:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.775+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88301023c0 con 0x7f8830100ee0 2024-03-18T00:41:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.775+0000 7f88267fc700 1 --2- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f881c038fc0 0x7f881c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.775+0000 7f88267fc700 1 -- 172.21.15.107:0/637976017 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f882800f050 con 0x7f8830100ee0 2024-03-18T00:41:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.775+0000 7f88348cd700 1 --2- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f881c038fc0 0x7f881c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:46.776 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.776+0000 7f88348cd700 1 --2- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f881c038fc0 0x7f881c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8820006fd0 tx=0x7f8820006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:46.780 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:46.780+0000 7f88267fc700 1 -- 172.21.15.107:0/637976017 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8828007e20 con 0x7f8830100ee0 2024-03-18T00:41:47.062 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.062+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8830198bd0 con 0x7f8830100ee0 2024-03-18T00:41:47.063 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.062+0000 7f88267fc700 1 -- 172.21.15.107:0/637976017 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f882801ce10 con 0x7f8830100ee0 2024-03-18T00:41:47.063 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:41:47.063 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":1,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","modified":"2024-03-18T00:40:00.185909Z","created":"2024-03-18T00:40:00.185909Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.064+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f881c038fc0 msgr2=0x7f881c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.064+0000 7f8837332700 1 --2- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f881c038fc0 0x7f881c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8820006fd0 tx=0x7f8820006e40 comp rx=0 tx=0).stop 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.064+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 msgr2=0x7f883019d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.064+0000 7f8837332700 1 --2- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 0x7f883019d5d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8828015040 tx=0x7f8828003d50 comp rx=0 tx=0).stop 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.064+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 shutdown_connections 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.064+0000 7f8837332700 1 --2- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f881c038fc0 0x7f881c03b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.064+0000 7f8837332700 1 --2- 172.21.15.107:0/637976017 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8830100ee0 0x7f883019d5d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.065+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 >> 172.21.15.107:0/637976017 conn(0x7f88300fc770 msgr2=0x7f883006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.065+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 shutdown_connections 2024-03-18T00:41:47.065 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:47.065+0000 7f8837332700 1 -- 172.21.15.107:0/637976017 wait complete. 2024-03-18T00:41:47.066 INFO:teuthology.orchestra.run.smithi107.stderr:dumped monmap epoch 1 2024-03-18T00:41:47.115 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:46 smithi107 podman[36211]: 2024-03-18 00:41:46.765429445 +0000 UTC m=+0.347086346 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:41:47.116 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:46 smithi107 podman[36211]: 2024-03-18 00:41:46.863278101 +0000 UTC m=+0.444935001 container create 92247214c5cb42348a7fcc0a404471892ead07faa9b3a8519846ded5bc03e235 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-c, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-18T00:41:47.116 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 podman[36211]: 2024-03-18 00:41:47.116213512 +0000 UTC m=+0.697870424 container init 92247214c5cb42348a7fcc0a404471892ead07faa9b3a8519846ded5bc03e235 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:41:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:46 smithi049 ceph-mon[32130]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 podman[36211]: 2024-03-18 00:41:47.119867806 +0000 UTC m=+0.701524707 container start 92247214c5cb42348a7fcc0a404471892ead07faa9b3a8519846ded5bc03e235 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pidfile_write: ignore empty --pid-file 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: load: jerasure load: lrc 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: RocksDB version: 7.9.2 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Git sha 0 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: DB SUMMARY 2024-03-18T00:41:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: DB Session ID: 3Y3324EAI7K9ZHHICGW5 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: CURRENT file: CURRENT 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.error_if_exists: 0 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.create_if_missing: 0 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.paranoid_checks: 1 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.env: 0x55885b929c60 2024-03-18T00:41:47.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.info_log: 0x55885d249ac0 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.statistics: (nil) 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.use_fsync: 0 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_log_file_size: 0 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T00:41:47.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.allow_fallocate: 1 2024-03-18T00:41:47.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T00:41:47.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T00:41:47.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.use_direct_reads: 0 2024-03-18T00:41:47.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T00:41:47.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T00:41:47.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.db_log_dir: 2024-03-18T00:41:47.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.wal_dir: 2024-03-18T00:41:47.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.write_buffer_manager: 0x55885d21c1e0 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T00:41:47.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.unordered_write: 0 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.row_cache: None 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.wal_filter: None 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T00:41:47.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.two_write_queues: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.wal_compression: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.atomic_flush: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.log_readahead_size: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T00:41:47.438 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_background_jobs: 2 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_background_compactions: -1 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_subcompactions: 1 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_open_files: -1 2024-03-18T00:41:47.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_background_flushes: -1 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Compression algorithms supported: 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: kZSTD supported: 0 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: kXpressCompression supported: 0 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: kZlibCompression supported: 1 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: kSnappyCompression supported: 1 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: kLZ4Compression supported: 1 2024-03-18T00:41:47.440 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: kBZip2Compression supported: 0 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.merge_operator: 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_filter: None 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T00:41:47.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55885d249bc0) 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_type: 0 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: data_block_index_type: 0 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_shortening: 1 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: checksum: 4 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: no_block_cache: 0 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache: 0x55885d24f090 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-03-18T00:41:47.442 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_options: 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: capacity : 536870912 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: num_shard_bits : 4 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: strict_capacity_limit : 0 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_compressed: (nil) 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: persistent_cache: (nil) 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_size: 4096 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_size_deviation: 10 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_restart_interval: 16 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_block_restart_interval: 1 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: metadata_block_size: 4096 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: partition_filters: 0 2024-03-18T00:41:47.443 INFO:journalctl@ceph.mon.c.smithi107.stdout: use_delta_encoding: 1 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: filter_policy: bloomfilter 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: whole_key_filtering: 1 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: verify_compression: 0 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: format_version: 5 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: enable_index_compression: 1 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_align: 0 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: max_auto_readahead_size: 262144 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: prepopulate_block_cache: 0 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: initial_auto_readahead_size: 8192 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T00:41:47.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression: NoCompression 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.num_levels: 7 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T00:41:47.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T00:41:47.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T00:41:47.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T00:41:47.448 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.table_properties_collectors: 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.inplace_update_support: 0 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.bloom_locality: 0 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.max_successive_merges: 0 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T00:41:47.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.ttl: 2592000 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.enable_blob_files: false 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.min_blob_size: 0 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T00:41:47.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7615da9e-e859-4c32-b0cb-58884290ad46 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710722507149340, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710722507149765, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710722507, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7615da9e-e859-4c32-b0cb-58884290ad46", "db_session_id": "3Y3324EAI7K9ZHHICGW5", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710722507149843, "job": 1, "event": "recovery_finished"} 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55885d342000 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: DB pointer 0x55885d32c000 2024-03-18T00:41:47.451 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** DB Stats ** 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T00:41:47.452 INFO:journalctl@ceph.mon.c.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T00:41:47.453 INFO:journalctl@ceph.mon.c.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: Block cache BinnedLRUCache@0x55885d24f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-18T00:41:47.454 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: using public_addr v2:172.21.15.107:0/0 -> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: starting mon.c rank -1 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(???) e0 preinit fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).mds e1 new map 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout: e1 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout: legacy client fscid: -1 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout: No filesystems configured 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T00:41:47.455 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi106", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Updating smithi106:/etc/ceph/ceph.conf 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Deploying cephadm binary to smithi107 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Updating smithi106:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Updating smithi106:/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Updating smithi106:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Added host smithi107 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1364468275' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.458 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Updating smithi107:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Updating smithi107:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1364468275' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi049:172.21.15.49=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Saving service mon spec with placement smithi049:172.21.15.49=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c;count:3 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: Deploying daemon mon.c on smithi107 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 ceph-mon[36270]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T00:41:47.460 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 bash[36211]: 92247214c5cb42348a7fcc0a404471892ead07faa9b3a8519846ded5bc03e235 2024-03-18T00:41:47.461 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:47 smithi107 systemd[1]: Started Ceph mon.c for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:41:48.762 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T00:41:48.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T00:41:49.006 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: mon.a calling monitor election 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: mon.c calling monitor election 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: monmap epoch 2 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: last_changed 2024-03-18T00:41:47.172259+0000 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: min_mon_release 19 (squid) 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: election_strategy: 1 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: 1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.c 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: fsmap 2024-03-18T00:41:52.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:41:52.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: mgrmap e13: x(active, since 50s) 2024-03-18T00:41:52.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: overall HEALTH_OK 2024-03-18T00:41:52.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:52.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:52.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:52.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:41:52.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: mon.a calling monitor election 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: mon.c calling monitor election 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: monmap epoch 2 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: last_changed 2024-03-18T00:41:47.172259+0000 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: min_mon_release 19 (squid) 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: election_strategy: 1 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: 1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.c 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: fsmap 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: mgrmap e13: x(active, since 50s) 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: overall HEALTH_OK 2024-03-18T00:41:52.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:52.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:52.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:52.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:41:52.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:53.511 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.511+0000 7f317be6d700 1 -- 172.21.15.107:0/1463586219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31600035c0 msgr2=0x7f3160005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:53.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.511+0000 7f317be6d700 1 --2- 172.21.15.107:0/1463586219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31600035c0 0x7f3160005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3174101ce0 tx=0x7f317000feb0 comp rx=0 tx=0).stop 2024-03-18T00:41:53.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.511+0000 7f317be6d700 1 -- 172.21.15.107:0/1463586219 shutdown_connections 2024-03-18T00:41:53.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.511+0000 7f317be6d700 1 --2- 172.21.15.107:0/1463586219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31600035c0 0x7f3160005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:53.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.511+0000 7f317be6d700 1 -- 172.21.15.107:0/1463586219 >> 172.21.15.107:0/1463586219 conn(0x7f31740fd8a0 msgr2=0x7f31740ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:53.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.511+0000 7f317be6d700 1 -- 172.21.15.107:0/1463586219 shutdown_connections 2024-03-18T00:41:53.512 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.511+0000 7f317be6d700 1 -- 172.21.15.107:0/1463586219 wait complete. 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f317be6d700 1 Processor -- start 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f317be6d700 1 -- start start 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f317be6d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31600035c0 0x7f317419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f317be6d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 0x7f31741977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f317be6d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3174109e40 con 0x7f31600035c0 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f317be6d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3174109cc0 con 0x7f317419dbe0 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f3179408700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 0x7f31741977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f3179408700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 0x7f31741977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60454/0 (socket says 172.21.15.107:60454) 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f3179408700 1 -- 172.21.15.107:0/549856573 learned_addr learned my addr 172.21.15.107:0/549856573 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.512+0000 7f3179c09700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31600035c0 0x7f317419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:53.514 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f3179408700 1 -- 172.21.15.107:0/549856573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 msgr2=0x7f31741977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f3179408700 1 -- 172.21.15.107:0/549856573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 msgr2=0x7f31741977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f3179408700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 0x7f31741977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f3179408700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 0x7f31741977c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f3179c09700 1 -- 172.21.15.107:0/549856573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 msgr2=0x7f31741977c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f3179c09700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 0x7f31741977c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f3179c09700 1 -- 172.21.15.107:0/549856573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f317419e160 con 0x7f31600035c0 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f3179c09700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31600035c0 0x7f317419d6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3170010010 tx=0x7f31700055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f316affd700 1 -- 172.21.15.107:0/549856573 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f3170004b20 con 0x7f31600035c0 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f316affd700 1 -- 172.21.15.107:0/549856573 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3170013e40 con 0x7f31600035c0 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3174197d60 con 0x7f31600035c0 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f316affd700 1 -- 172.21.15.107:0/549856573 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f3170011b60 con 0x7f31600035c0 2024-03-18T00:41:53.515 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.513+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31741982f0 con 0x7f31600035c0 2024-03-18T00:41:53.516 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.514+0000 7f316affd700 1 -- 172.21.15.107:0/549856573 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f3170023070 con 0x7f31600035c0 2024-03-18T00:41:53.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:53 smithi107 ceph-mon[36270]: Deploying daemon mon.b on smithi106 2024-03-18T00:41:53.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:53.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.515+0000 7f316affd700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3158038d60 0x7f315803b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:41:53.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.515+0000 7f316affd700 1 -- 172.21.15.107:0/549856573 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3170020050 con 0x7f31600035c0 2024-03-18T00:41:53.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.515+0000 7f3179408700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3158038d60 0x7f315803b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:41:53.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.515+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31741034f0 con 0x7f31600035c0 2024-03-18T00:41:53.517 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.516+0000 7f3179408700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3158038d60 0x7f315803b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3164009180 tx=0x7f3164008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:41:53.521 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.521+0000 7f316affd700 1 -- 172.21.15.107:0/549856573 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3170028020 con 0x7f31600035c0 2024-03-18T00:41:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:53 smithi049 ceph-mon[32130]: Deploying daemon mon.b on smithi106 2024-03-18T00:41:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:53.798 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.798+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3174198ca0 con 0x7f31600035c0 2024-03-18T00:41:53.799 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.799+0000 7f316affd700 1 -- 172.21.15.107:0/549856573 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f3170011e10 con 0x7f31600035c0 2024-03-18T00:41:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:41:53.799 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":2,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","modified":"2024-03-18T00:41:47.172259Z","created":"2024-03-18T00:40:00.185909Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T00:41:53.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3158038d60 msgr2=0x7f315803b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:53.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3158038d60 0x7f315803b220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3164009180 tx=0x7f3164008040 comp rx=0 tx=0).stop 2024-03-18T00:41:53.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31600035c0 msgr2=0x7f317419d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:41:53.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31600035c0 0x7f317419d6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3170010010 tx=0x7f31700055a0 comp rx=0 tx=0).stop 2024-03-18T00:41:53.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 shutdown_connections 2024-03-18T00:41:53.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31600035c0 0x7f317419d6a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:53.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3158038d60 0x7f315803b220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:53.802 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 --2- 172.21.15.107:0/549856573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f317419dbe0 0x7f31741977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:41:53.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 >> 172.21.15.107:0/549856573 conn(0x7f31740fd8a0 msgr2=0x7f31740fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:41:53.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 shutdown_connections 2024-03-18T00:41:53.803 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:41:53.801+0000 7f317be6d700 1 -- 172.21.15.107:0/549856573 wait complete. 2024-03-18T00:41:53.803 INFO:teuthology.orchestra.run.smithi107.stderr:dumped monmap epoch 2 2024-03-18T00:41:54.378 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:54 smithi107 ceph-mon[36270]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:54.378 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:54 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/549856573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T00:41:54.526 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:54 smithi049 ceph-mon[32130]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:54.526 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:54 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/549856573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T00:41:55.439 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T00:41:55.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T00:41:55.675 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:41:56.452 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: monmap epoch 2 2024-03-18T00:41:56.452 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:41:56.452 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: last_changed 2024-03-18T00:41:47.172259+0000 2024-03-18T00:41:56.452 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:41:56.452 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: min_mon_release 19 (squid) 2024-03-18T00:41:56.452 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: election_strategy: 1 2024-03-18T00:41:56.453 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:41:56.453 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: 1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.c 2024-03-18T00:41:56.727 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: fsmap 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: mgrmap e13: x(active, since 50s) 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: overall HEALTH_OK 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: Deploying daemon mon.b on smithi106 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/549856573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 ceph-mon[35899]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T00:41:56.728 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 bash[35879]: d10ddbeaacf39c8f59374f81656b5250e78b76277c62985423e14fb7b0812c68 2024-03-18T00:41:56.729 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:41:56 smithi106 systemd[1]: Started Ceph mon.b for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:41:56.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:41:56 smithi107 ceph-mon[36270]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:41:56.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:41:56 smithi049 ceph-mon[32130]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:01.493 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.492+0000 7ff62dad0700 1 -- 172.21.15.107:0/348218934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6100035c0 msgr2=0x7ff610005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.492+0000 7ff62dad0700 1 --2- 172.21.15.107:0/348218934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6100035c0 0x7ff610005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff6281013f0 tx=0x7ff61800fe60 comp rx=0 tx=0).stop 2024-03-18T00:42:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.492+0000 7ff62dad0700 1 -- 172.21.15.107:0/348218934 shutdown_connections 2024-03-18T00:42:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.492+0000 7ff62dad0700 1 --2- 172.21.15.107:0/348218934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6100035c0 0x7ff610005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.492+0000 7ff62dad0700 1 -- 172.21.15.107:0/348218934 >> 172.21.15.107:0/348218934 conn(0x7ff6280773c0 msgr2=0x7ff6280777d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.492+0000 7ff62dad0700 1 -- 172.21.15.107:0/348218934 shutdown_connections 2024-03-18T00:42:01.494 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.492+0000 7ff62dad0700 1 -- 172.21.15.107:0/348218934 wait complete. 2024-03-18T00:42:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff62dad0700 1 Processor -- start 2024-03-18T00:42:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff62dad0700 1 -- start start 2024-03-18T00:42:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff62dad0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6100035c0 0x7ff62819d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff62dad0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 0x7ff628197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff62dad0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff628197ed0 0x7ff628198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff62dad0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff628109f00 con 0x7ff6100035c0 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff62dad0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff628109d80 con 0x7ff628197ed0 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff62dad0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff62810a080 con 0x7ff62819dd10 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff626ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 0x7ff628197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.494+0000 7ff6277fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6100035c0 0x7ff62819d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff626ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 0x7ff628197990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:33150/0 (socket says 172.21.15.107:33150) 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff626ffd700 1 -- 172.21.15.107:0/2178747235 learned_addr learned my addr 172.21.15.107:0/2178747235 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff626ffd700 1 -- 172.21.15.107:0/2178747235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 msgr2=0x7ff628197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff626ffd700 1 -- 172.21.15.107:0/2178747235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 msgr2=0x7ff628197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff626ffd700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 0x7ff628197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff626ffd700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 0x7ff628197990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff627fff700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff628197ed0 0x7ff628198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff627fff700 1 -- 172.21.15.107:0/2178747235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff628197ed0 msgr2=0x7ff628198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-18T00:42:01.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff627fff700 1 -- 172.21.15.107:0/2178747235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff628197ed0 msgr2=0x7ff628198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff6277fe700 1 -- 172.21.15.107:0/2178747235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 msgr2=0x7ff628197990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff627fff700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff628197ed0 0x7ff628198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff6277fe700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 0x7ff628197990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff6277fe700 1 -- 172.21.15.107:0/2178747235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff628197ed0 msgr2=0x7ff628198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff627fff700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff628197ed0 0x7ff628198380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff6277fe700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff628197ed0 0x7ff628198380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff6277fe700 1 -- 172.21.15.107:0/2178747235 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6281a46a0 con 0x7ff6100035c0 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff6277fe700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6100035c0 0x7ff62819d7d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff61801f000 tx=0x7ff618005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff624ff9700 1 -- 172.21.15.107:0/2178747235 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff61802b020 con 0x7ff6100035c0 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff624ff9700 1 -- 172.21.15.107:0/2178747235 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff618004110 con 0x7ff6100035c0 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.495+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6281a4870 con 0x7ff6100035c0 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.496+0000 7ff624ff9700 1 -- 172.21.15.107:0/2178747235 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff618022040 con 0x7ff6100035c0 2024-03-18T00:42:01.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.496+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6281a4e00 con 0x7ff6100035c0 2024-03-18T00:42:01.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.497+0000 7ff624ff9700 1 -- 172.21.15.107:0/2178747235 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff61802e020 con 0x7ff6100035c0 2024-03-18T00:42:01.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.497+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff628063900 con 0x7ff6100035c0 2024-03-18T00:42:01.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.497+0000 7ff624ff9700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff614039450 0x7ff61403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:01.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.497+0000 7ff624ff9700 1 -- 172.21.15.107:0/2178747235 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff61804e9c0 con 0x7ff6100035c0 2024-03-18T00:42:01.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.497+0000 7ff626ffd700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff614039450 0x7ff61403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:01.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.498+0000 7ff626ffd700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff614039450 0x7ff61403b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009180 tx=0x7ff61c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:01.502 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.502+0000 7ff624ff9700 1 -- 172.21.15.107:0/2178747235 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff618035020 con 0x7ff6100035c0 2024-03-18T00:42:01.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.789+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff6281a5030 con 0x7ff6100035c0 2024-03-18T00:42:01.790 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.790+0000 7ff624ff9700 1 -- 172.21.15.107:0/2178747235 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7ff618010c90 con 0x7ff6100035c0 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: mon.c calling monitor election 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: mon.a calling monitor election 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: monmap epoch 3 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: last_changed 2024-03-18T00:41:56.455026+0000 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: min_mon_release 19 (squid) 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: election_strategy: 1 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: 1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.c 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: 2: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.b 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: fsmap 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: mgrmap e13: x(active, since 60s) 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T00:42:01.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T00:42:01.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T00:42:01.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: mon.b (rank 2) addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] is down (out of quorum) 2024-03-18T00:42:01.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:01.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:01.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:01.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:01.793 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-18T00:42:01.793 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":3,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","modified":"2024-03-18T00:41:56.455026Z","created":"2024-03-18T00:40:00.185909Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:3300","nonce":0},{"type":"v1","addr":"172.21.15.49:6789","nonce":0}]},"addr":"172.21.15.49:6789/0","public_addr":"172.21.15.49:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:3300","nonce":0},{"type":"v1","addr":"172.21.15.106:6789","nonce":0}]},"addr":"172.21.15.106:6789/0","public_addr":"172.21.15.106:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T00:42:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff614039450 msgr2=0x7ff61403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff614039450 0x7ff61403b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009180 tx=0x7ff61c008040 comp rx=0 tx=0).stop 2024-03-18T00:42:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6100035c0 msgr2=0x7ff62819d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6100035c0 0x7ff62819d7d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff61801f000 tx=0x7ff618005720 comp rx=0 tx=0).stop 2024-03-18T00:42:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 shutdown_connections 2024-03-18T00:42:01.794 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff628197ed0 0x7ff628198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff6100035c0 0x7ff62819d7d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff614039450 0x7ff61403b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 --2- 172.21.15.107:0/2178747235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff62819dd10 0x7ff628197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 >> 172.21.15.107:0/2178747235 conn(0x7ff6280773c0 msgr2=0x7ff6280ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 shutdown_connections 2024-03-18T00:42:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:01.793+0000 7ff62dad0700 1 -- 172.21.15.107:0/2178747235 wait complete. 2024-03-18T00:42:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:dumped monmap epoch 3 2024-03-18T00:42:01.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: mon.c calling monitor election 2024-03-18T00:42:01.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:42:01.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: mon.a calling monitor election 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: monmap epoch 3 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:42:01.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: last_changed 2024-03-18T00:41:56.455026+0000 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: min_mon_release 19 (squid) 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: election_strategy: 1 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: 1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.c 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: 2: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.b 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: fsmap 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: mgrmap e13: x(active, since 60s) 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T00:42:01.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: mon.b (rank 2) addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] is down (out of quorum) 2024-03-18T00:42:01.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:01.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:01.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:01.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:02.386 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-18T00:42:02.387 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-18T00:42:02.649 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:42:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:02 smithi107 ceph-mon[36270]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:02 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/2178747235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T00:42:02.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:02.771 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:02 smithi049 ceph-mon[32130]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:02.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:02.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:02 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/2178747235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T00:42:02.772 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: mon.c calling monitor election 2024-03-18T00:42:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:42:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: mon.a calling monitor election 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: monmap epoch 3 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: last_changed 2024-03-18T00:41:56.455026+0000 2024-03-18T00:42:03.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: min_mon_release 19 (squid) 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: election_strategy: 1 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: 1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.c 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: 2: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.b 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: fsmap 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: mgrmap e13: x(active, since 60s) 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: mon.b (rank 2) addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] is down (out of quorum) 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:03.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:03.949 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:03.949 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:03.949 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:03.949 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:03.949 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/2178747235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T00:42:03.949 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:04.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.265+0000 7fc8f409b700 1 -- 172.21.15.49:0/1508728250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec101f00 msgr2=0x7fc8ec1022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:04.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.265+0000 7fc8f409b700 1 --2- 172.21.15.49:0/1508728250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec101f00 0x7fc8ec1022e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009910 tx=0x7fc8e8009c20 comp rx=0 tx=0).stop 2024-03-18T00:42:04.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.265+0000 7fc8f409b700 1 -- 172.21.15.49:0/1508728250 shutdown_connections 2024-03-18T00:42:04.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.265+0000 7fc8f409b700 1 --2- 172.21.15.49:0/1508728250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec101f00 0x7fc8ec1022e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:04.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.265+0000 7fc8f409b700 1 -- 172.21.15.49:0/1508728250 >> 172.21.15.49:0/1508728250 conn(0x7fc8ec0fd7b0 msgr2=0x7fc8ec0ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:04.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.265+0000 7fc8f409b700 1 -- 172.21.15.49:0/1508728250 shutdown_connections 2024-03-18T00:42:04.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.265+0000 7fc8f409b700 1 -- 172.21.15.49:0/1508728250 wait complete. 2024-03-18T00:42:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f409b700 1 Processor -- start 2024-03-18T00:42:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f409b700 1 -- start start 2024-03-18T00:42:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f409b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec101f00 0x7fc8ec19f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f409b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec19fe10 0x7fc8ec199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f409b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8ec199fd0 0x7fc8ec19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f409b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec10c2d0 con 0x7fc8ec19fe10 2024-03-18T00:42:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f409b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec10c150 con 0x7fc8ec199fd0 2024-03-18T00:42:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f409b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec10c450 con 0x7fc8ec101f00 2024-03-18T00:42:04.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f1636700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec19fe10 0x7fc8ec199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f1636700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec19fe10 0x7fc8ec199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40230/0 (socket says 172.21.15.49:40230) 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f1636700 1 -- 172.21.15.49:0/240343816 learned_addr learned my addr 172.21.15.49:0/240343816 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f1e37700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec101f00 0x7fc8ec19f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f2638700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8ec199fd0 0x7fc8ec19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f1636700 1 -- 172.21.15.49:0/240343816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec101f00 msgr2=0x7fc8ec19f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.266+0000 7fc8f1636700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec101f00 0x7fc8ec19f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8f1636700 1 -- 172.21.15.49:0/240343816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8ec199fd0 msgr2=0x7fc8ec19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8f1636700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8ec199fd0 0x7fc8ec19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8f1636700 1 -- 172.21.15.49:0/240343816 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ec1a67a0 con 0x7fc8ec19fe10 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8f1e37700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec101f00 0x7fc8ec19f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8f2638700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8ec199fd0 0x7fc8ec19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8f1636700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec19fe10 0x7fc8ec199a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00b780 tx=0x7fc8dc00bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8e2ffd700 1 -- 172.21.15.49:0/240343816 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc004230 con 0x7fc8ec19fe10 2024-03-18T00:42:04.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8ec1a69d0 con 0x7fc8ec19fe10 2024-03-18T00:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8e2ffd700 1 -- 172.21.15.49:0/240343816 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8dc0043d0 con 0x7fc8ec19fe10 2024-03-18T00:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8e2ffd700 1 -- 172.21.15.49:0/240343816 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc011c60 con 0x7fc8ec19fe10 2024-03-18T00:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.267+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8ec1a6f10 con 0x7fc8ec19fe10 2024-03-18T00:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.268+0000 7fc8e2ffd700 1 -- 172.21.15.49:0/240343816 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fc8dc01a400 con 0x7fc8ec19fe10 2024-03-18T00:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.268+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8ec063900 con 0x7fc8ec19fe10 2024-03-18T00:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.268+0000 7fc8e2ffd700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc8d8039450 0x7fc8d803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.268+0000 7fc8e2ffd700 1 -- 172.21.15.49:0/240343816 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc8dc04bc50 con 0x7fc8ec19fe10 2024-03-18T00:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.268+0000 7fc8f1e37700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc8d8039450 0x7fc8d803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.269+0000 7fc8f1e37700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc8d8039450 0x7fc8d803b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800b5c0 tx=0x7fc8e80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.273+0000 7fc8e2ffd700 1 -- 172.21.15.49:0/240343816 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc8dc051020 con 0x7fc8ec19fe10 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: mon.b calling monitor election 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: mon.b calling monitor election 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: mon.c calling monitor election 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: mon.a calling monitor election 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: monmap epoch 3 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: last_changed 2024-03-18T00:41:56.455026+0000 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: min_mon_release 19 (squid) 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: election_strategy: 1 2024-03-18T00:42:04.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: 1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.c 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: 2: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.b 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: fsmap 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: mgrmap e13: x(active, since 62s) 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: Cluster is now healthy 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: overall HEALTH_OK 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.275 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:42:04.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T00:42:04.276 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:04.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: mon.b calling monitor election 2024-03-18T00:42:04.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: mon.b calling monitor election 2024-03-18T00:42:04.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: mon.c calling monitor election 2024-03-18T00:42:04.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: mon.a calling monitor election 2024-03-18T00:42:04.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: monmap epoch 3 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: last_changed 2024-03-18T00:41:56.455026+0000 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: min_mon_release 19 (squid) 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: election_strategy: 1 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: 1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.c 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: 2: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.b 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: fsmap 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: mgrmap e13: x(active, since 62s) 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T00:42:04.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: Cluster is now healthy 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: overall HEALTH_OK 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T00:42:04.448 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:04.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.484+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc8ec1a7200 con 0x7fc8ec19fe10 2024-03-18T00:42:04.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.484+0000 7fc8e2ffd700 1 -- 172.21.15.49:0/240343816 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7fc8dc020500 con 0x7fc8ec19fe10 2024-03-18T00:42:04.484 INFO:teuthology.orchestra.run.smithi049.stdout:# minimal ceph.conf for ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:42:04.484 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2024-03-18T00:42:04.484 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:42:04.484 INFO:teuthology.orchestra.run.smithi049.stdout: mon_host = [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2024-03-18T00:42:04.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc8d8039450 msgr2=0x7fc8d803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:04.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc8d8039450 0x7fc8d803b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800b5c0 tx=0x7fc8e80058e0 comp rx=0 tx=0).stop 2024-03-18T00:42:04.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec19fe10 msgr2=0x7fc8ec199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:04.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec19fe10 0x7fc8ec199a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00b780 tx=0x7fc8dc00bb40 comp rx=0 tx=0).stop 2024-03-18T00:42:04.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 shutdown_connections 2024-03-18T00:42:04.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc8d8039450 0x7fc8d803b910 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:04.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc8ec101f00 0x7fc8ec19f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:04.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc8ec199fd0 0x7fc8ec19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:04.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 --2- 172.21.15.49:0/240343816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc8ec19fe10 0x7fc8ec199a90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:04.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 >> 172.21.15.49:0/240343816 conn(0x7fc8ec0fd7b0 msgr2=0x7fc8ec074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:04.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 shutdown_connections 2024-03-18T00:42:04.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:04.486+0000 7fc8f409b700 1 -- 172.21.15.49:0/240343816 wait complete. 2024-03-18T00:42:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: mon.b calling monitor election 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: mon.b calling monitor election 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: mon.c calling monitor election 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: mon.a calling monitor election 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: monmap epoch 3 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: last_changed 2024-03-18T00:41:56.455026+0000 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: created 2024-03-18T00:40:00.185909+0000 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: min_mon_release 19 (squid) 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: election_strategy: 1 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: 0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: 1: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.c 2024-03-18T00:42:04.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: 2: [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] mon.b 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: fsmap 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: mgrmap e13: x(active, since 62s) 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: Cluster is now healthy 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: overall HEALTH_OK 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:04.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:42:04.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T00:42:04.683 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:05.056 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-18T00:42:05.056 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:42:05.057 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T00:42:05.102 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:42:05.102 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:42:05.183 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:42:05.183 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T00:42:05.223 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:42:05.223 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:42:05.299 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:42:05.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T00:42:05.334 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:42:05.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T00:42:05.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:05 smithi049 ceph-mon[32130]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:05.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:05 smithi049 ceph-mon[32130]: Updating smithi107:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:42:05.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:05 smithi049 ceph-mon[32130]: Updating smithi049:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:42:05.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:05 smithi049 ceph-mon[32130]: Updating smithi106:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:42:05.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:05 smithi049 ceph-mon[32130]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T00:42:05.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:05 smithi049 ceph-mon[32130]: Reconfiguring daemon mon.a on smithi049 2024-03-18T00:42:05.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:05 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:05.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:05 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/240343816' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:05.410 INFO:tasks.cephadm:Adding mgr.x on smithi049 2024-03-18T00:42:05.410 INFO:tasks.cephadm:Adding mgr.y on smithi106 2024-03-18T00:42:05.410 INFO:tasks.cephadm:Adding mgr.z on smithi107 2024-03-18T00:42:05.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi049=x;smithi106=y;smithi107=z' 2024-03-18T00:42:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:05 smithi106 ceph-mon[35899]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:05 smithi106 ceph-mon[35899]: Updating smithi107:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:42:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:05 smithi106 ceph-mon[35899]: Updating smithi049:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:42:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:05 smithi106 ceph-mon[35899]: Updating smithi106:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:42:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:05 smithi106 ceph-mon[35899]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T00:42:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:05 smithi106 ceph-mon[35899]: Reconfiguring daemon mon.a on smithi049 2024-03-18T00:42:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:05 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:05 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/240343816' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:05.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:05 smithi107 ceph-mon[36270]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:05.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:05 smithi107 ceph-mon[36270]: Updating smithi107:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:42:05.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:05 smithi107 ceph-mon[36270]: Updating smithi049:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:42:05.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:05 smithi107 ceph-mon[36270]: Updating smithi106:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T00:42:05.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:05 smithi107 ceph-mon[36270]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T00:42:05.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:05 smithi107 ceph-mon[36270]: Reconfiguring daemon mon.a on smithi049 2024-03-18T00:42:05.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:05 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:42:05.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:05 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/240343816' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:05.624 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:42:05 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:42:05.454+0000 7fc7b1791700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-18T00:42:05.683 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.994+0000 7f78cf883700 1 -- 172.21.15.107:0/586471632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8102310 msgr2=0x7f78c8102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.994+0000 7f78cf883700 1 --2- 172.21.15.107:0/586471632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8102310 0x7f78c8102730 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78b8009910 tx=0x7f78b8009c20 comp rx=0 tx=0).stop 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.994+0000 7f78cf883700 1 -- 172.21.15.107:0/586471632 shutdown_connections 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.994+0000 7f78cf883700 1 --2- 172.21.15.107:0/586471632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8102310 0x7f78c8102730 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.994+0000 7f78cf883700 1 -- 172.21.15.107:0/586471632 >> 172.21.15.107:0/586471632 conn(0x7f78c80fba20 msgr2=0x7f78c80fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 -- 172.21.15.107:0/586471632 shutdown_connections 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 -- 172.21.15.107:0/586471632 wait complete. 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 Processor -- start 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 -- start start 2024-03-18T00:42:06.997 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78c8102310 0x7f78c8197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8197b80 0x7f78c819cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78c8198000 0x7f78c819f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f78c8106d50 con 0x7f78c8197b80 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f78c8106bd0 con 0x7f78c8102310 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cf883700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f78c8106ed0 con 0x7f78c8198000 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.995+0000 7f78cce1e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8197b80 0x7f78c819cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cce1e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8197b80 0x7f78c819cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.107:37902/0 (socket says 172.21.15.107:37902) 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cce1e700 1 -- 172.21.15.107:0/357276225 learned_addr learned my addr 172.21.15.107:0/357276225 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cde20700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78c8198000 0x7f78c819f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cd61f700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78c8102310 0x7f78c8197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cce1e700 1 -- 172.21.15.107:0/357276225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78c8198000 msgr2=0x7f78c819f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cce1e700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78c8198000 0x7f78c819f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:06.998 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cce1e700 1 -- 172.21.15.107:0/357276225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78c8102310 msgr2=0x7f78c8197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cce1e700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78c8102310 0x7f78c8197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cce1e700 1 -- 172.21.15.107:0/357276225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c819fa50 con 0x7f78c8197b80 2024-03-18T00:42:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cd61f700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78c8102310 0x7f78c8197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:42:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cce1e700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8197b80 0x7f78c819cbf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78c400b780 tx=0x7f78c400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78be7fc700 1 -- 172.21.15.107:0/357276225 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4004230 con 0x7f78c8197b80 2024-03-18T00:42:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c819fd40 con 0x7f78c8197b80 2024-03-18T00:42:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78be7fc700 1 -- 172.21.15.107:0/357276225 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78c40043d0 con 0x7f78c8197b80 2024-03-18T00:42:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78be7fc700 1 -- 172.21.15.107:0/357276225 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c4011c90 con 0x7f78c8197b80 2024-03-18T00:42:06.999 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.996+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c81a02f0 con 0x7f78c8197b80 2024-03-18T00:42:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.997+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c8063900 con 0x7f78c8197b80 2024-03-18T00:42:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.997+0000 7f78be7fc700 1 -- 172.21.15.107:0/357276225 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f78c40036a0 con 0x7f78c8197b80 2024-03-18T00:42:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.997+0000 7f78be7fc700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78b4039450 0x7f78b403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.997+0000 7f78be7fc700 1 -- 172.21.15.107:0/357276225 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f78c404bc20 con 0x7f78c8197b80 2024-03-18T00:42:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.998+0000 7f78cd61f700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78b4039450 0x7f78b403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:07.000 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:06.998+0000 7f78cd61f700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78b4039450 0x7f78b403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f78b800b5c0 tx=0x7f78b801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:07.003 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.003+0000 7f78be7fc700 1 -- 172.21.15.107:0/357276225 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f78c4003d50 con 0x7f78c8197b80 2024-03-18T00:42:07.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:07 smithi107 ceph-mon[36270]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:07.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:07.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:07 smithi107 ceph-mon[36270]: Reconfiguring mon.b (monmap changed)... 2024-03-18T00:42:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:42:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T00:42:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:07 smithi107 ceph-mon[36270]: Reconfiguring daemon mon.b on smithi106 2024-03-18T00:42:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:07 smithi106 ceph-mon[35899]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:07 smithi106 ceph-mon[35899]: Reconfiguring mon.b (monmap changed)... 2024-03-18T00:42:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:42:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T00:42:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:07 smithi106 ceph-mon[35899]: Reconfiguring daemon mon.b on smithi106 2024-03-18T00:42:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:07.213 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.212+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi049=x;smithi106=y;smithi107=z", "target": ["mon-mgr", ""]}) -- 0x7f78c8100570 con 0x7f78b4039450 2024-03-18T00:42:07.219 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.219+0000 7f78be7fc700 1 -- 172.21.15.107:0/357276225 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f78c8100570 con 0x7f78b4039450 2024-03-18T00:42:07.219 INFO:teuthology.orchestra.run.smithi107.stdout:Scheduled mgr update... 2024-03-18T00:42:07.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78b4039450 msgr2=0x7f78b403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:07.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78b4039450 0x7f78b403b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f78b800b5c0 tx=0x7f78b801a040 comp rx=0 tx=0).stop 2024-03-18T00:42:07.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8197b80 msgr2=0x7f78c819cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:07.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8197b80 0x7f78c819cbf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78c400b780 tx=0x7f78c400ba90 comp rx=0 tx=0).stop 2024-03-18T00:42:07.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 shutdown_connections 2024-03-18T00:42:07.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78b4039450 0x7f78b403b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:07.222 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78c8198000 0x7f78c819f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:07.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78c8102310 0x7f78c8197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:07.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 --2- 172.21.15.107:0/357276225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78c8197b80 0x7f78c819cbf0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:07.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 >> 172.21.15.107:0/357276225 conn(0x7f78c80fba20 msgr2=0x7f78c80ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:07.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 shutdown_connections 2024-03-18T00:42:07.223 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:42:07.222+0000 7f78cf883700 1 -- 172.21.15.107:0/357276225 wait complete. 2024-03-18T00:42:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:07 smithi049 ceph-mon[32130]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:07 smithi049 ceph-mon[32130]: Reconfiguring mon.b (monmap changed)... 2024-03-18T00:42:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:42:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T00:42:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:07 smithi049 ceph-mon[32130]: Reconfiguring daemon mon.b on smithi106 2024-03-18T00:42:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:07.666 DEBUG:teuthology.orchestra.run.smithi106:mgr.y> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.y.service 2024-03-18T00:42:07.668 DEBUG:teuthology.orchestra.run.smithi107:mgr.z> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.z.service 2024-03-18T00:42:07.709 INFO:tasks.cephadm:Deploying OSDs... 2024-03-18T00:42:07.709 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:42:07.709 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T00:42:07.728 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T00:42:07.729 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2024-03-18T00:42:07.787 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T00:42:07.787 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:07.787 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-18T00:42:07.787 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:07.787 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:07.787 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-03-18 00:41:14.399579175 +0000 2024-03-18T00:42:07.787 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-03-18 00:36:43.974017185 +0000 2024-03-18T00:42:07.787 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-03-18 00:36:43.974017185 +0000 2024-03-18T00:42:07.787 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-03-18 00:36:43.974017185 +0000 2024-03-18T00:42:07.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T00:42:07.860 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-03-18T00:42:07.860 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-03-18T00:42:07.860 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000121235 s, 4.2 MB/s 2024-03-18T00:42:07.861 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T00:42:07.919 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2024-03-18T00:42:07.978 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T00:42:07.978 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:07.978 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-18T00:42:07.978 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:07.978 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:07.978 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-03-18 00:41:14.427578393 +0000 2024-03-18T00:42:07.978 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-03-18 00:36:44.331007205 +0000 2024-03-18T00:42:07.979 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-03-18 00:36:44.331007205 +0000 2024-03-18T00:42:07.979 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-03-18 00:36:44.331007205 +0000 2024-03-18T00:42:07.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T00:42:08.050 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-03-18T00:42:08.050 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-03-18T00:42:08.050 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000106509 s, 4.8 MB/s 2024-03-18T00:42:08.052 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T00:42:08.109 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2024-03-18T00:42:08.165 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T00:42:08.165 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:08.165 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-18T00:42:08.165 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:08.165 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:08.165 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-03-18 00:41:14.454577639 +0000 2024-03-18T00:42:08.165 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-03-18 00:36:44.678997476 +0000 2024-03-18T00:42:08.165 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-03-18 00:36:44.678997476 +0000 2024-03-18T00:42:08.166 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-03-18 00:36:44.678997476 +0000 2024-03-18T00:42:08.166 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T00:42:08.237 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-03-18T00:42:08.237 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-03-18T00:42:08.237 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000122337 s, 4.2 MB/s 2024-03-18T00:42:08.239 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T00:42:08.296 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2024-03-18T00:42:08.352 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T00:42:08.352 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:08.352 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-18T00:42:08.352 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:08.352 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:08.353 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-03-18 00:41:14.480576912 +0000 2024-03-18T00:42:08.353 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-03-18 00:36:45.020987916 +0000 2024-03-18T00:42:08.353 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-03-18 00:36:45.020987916 +0000 2024-03-18T00:42:08.353 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-03-18 00:36:45.020987916 +0000 2024-03-18T00:42:08.353 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T00:42:08.423 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-03-18T00:42:08.423 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-03-18T00:42:08.423 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000113325 s, 4.5 MB/s 2024-03-18T00:42:08.424 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T00:42:08.481 DEBUG:teuthology.orchestra.run.smithi106:> set -ex 2024-03-18T00:42:08.481 DEBUG:teuthology.orchestra.run.smithi106:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T00:42:08.499 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T00:42:08.499 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_1 2024-03-18T00:42:08.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:08 smithi107 ceph-mon[36270]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi049=x;smithi106=y;smithi107=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:42:08.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:08 smithi107 ceph-mon[36270]: Saving service mgr spec with placement smithi049=x;smithi106=y;smithi107=z;count:3 2024-03-18T00:42:08.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:08.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:08.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:08.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:42:08.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T00:42:08.542 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:08.556 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T00:42:08.556 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:08.556 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-18T00:42:08.556 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:08.556 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:08.556 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-18 00:41:55.916043665 +0000 2024-03-18T00:42:08.556 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-18 00:36:54.778776137 +0000 2024-03-18T00:42:08.556 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-18 00:36:54.778776137 +0000 2024-03-18T00:42:08.556 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-18 00:36:54.778776137 +0000 2024-03-18T00:42:08.556 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T00:42:08.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:08 smithi106 ceph-mon[35899]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi049=x;smithi106=y;smithi107=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:42:08.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:08 smithi106 ceph-mon[35899]: Saving service mgr spec with placement smithi049=x;smithi106=y;smithi107=z;count:3 2024-03-18T00:42:08.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:08.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:08.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:08.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:42:08.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T00:42:08.622 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:08.624 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-18T00:42:08.624 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-18T00:42:08.624 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000280272 s, 1.8 MB/s 2024-03-18T00:42:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:08 smithi049 ceph-mon[32130]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi049=x;smithi106=y;smithi107=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:42:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:08 smithi049 ceph-mon[32130]: Saving service mgr spec with placement smithi049=x;smithi106=y;smithi107=z;count:3 2024-03-18T00:42:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T00:42:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T00:42:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:08.626 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T00:42:08.683 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_2 2024-03-18T00:42:08.739 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T00:42:08.739 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:08.739 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-18T00:42:08.739 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:08.739 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:08.739 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-18 00:41:55.917043641 +0000 2024-03-18T00:42:08.739 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-18 00:36:55.120767817 +0000 2024-03-18T00:42:08.739 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-18 00:36:55.120767817 +0000 2024-03-18T00:42:08.739 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-18 00:36:55.120767817 +0000 2024-03-18T00:42:08.740 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T00:42:08.805 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-18T00:42:08.805 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-18T00:42:08.805 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000199768 s, 2.6 MB/s 2024-03-18T00:42:08.806 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T00:42:08.863 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_3 2024-03-18T00:42:08.917 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T00:42:08.918 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:08.918 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 698 Links: 1 2024-03-18T00:42:08.918 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:08.918 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:08.918 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-18 00:41:55.917043641 +0000 2024-03-18T00:42:08.918 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-18 00:36:55.454759692 +0000 2024-03-18T00:42:08.918 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-18 00:36:55.454759692 +0000 2024-03-18T00:42:08.918 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-18 00:36:55.454759692 +0000 2024-03-18T00:42:08.918 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T00:42:08.983 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-18T00:42:08.984 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-18T00:42:08.984 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000195769 s, 2.6 MB/s 2024-03-18T00:42:08.984 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T00:42:09.040 DEBUG:teuthology.orchestra.run.smithi106:> stat /dev/vg_nvme/lv_4 2024-03-18T00:42:09.095 INFO:teuthology.orchestra.run.smithi106.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T00:42:09.095 INFO:teuthology.orchestra.run.smithi106.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:09.095 INFO:teuthology.orchestra.run.smithi106.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-18T00:42:09.095 INFO:teuthology.orchestra.run.smithi106.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:09.095 INFO:teuthology.orchestra.run.smithi106.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:09.095 INFO:teuthology.orchestra.run.smithi106.stdout:Access: 2024-03-18 00:41:55.917043641 +0000 2024-03-18T00:42:09.095 INFO:teuthology.orchestra.run.smithi106.stdout:Modify: 2024-03-18 00:36:55.808751080 +0000 2024-03-18T00:42:09.095 INFO:teuthology.orchestra.run.smithi106.stdout:Change: 2024-03-18 00:36:55.808751080 +0000 2024-03-18T00:42:09.095 INFO:teuthology.orchestra.run.smithi106.stdout: Birth: 2024-03-18 00:36:55.808751080 +0000 2024-03-18T00:42:09.095 DEBUG:teuthology.orchestra.run.smithi106:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T00:42:09.160 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records in 2024-03-18T00:42:09.160 INFO:teuthology.orchestra.run.smithi106.stderr:1+0 records out 2024-03-18T00:42:09.160 INFO:teuthology.orchestra.run.smithi106.stderr:512 bytes copied, 0.000282122 s, 1.8 MB/s 2024-03-18T00:42:09.162 DEBUG:teuthology.orchestra.run.smithi106:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T00:42:09.217 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-18T00:42:09.217 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T00:42:09.232 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T00:42:09.232 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-03-18T00:42:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T00:42:09.287 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:09.288 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-18T00:42:09.288 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:09.288 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:09.288 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-18 00:41:46.061764344 +0000 2024-03-18T00:42:09.288 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-18 00:36:52.372140085 +0000 2024-03-18T00:42:09.288 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-18 00:36:52.372140085 +0000 2024-03-18T00:42:09.288 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-18 00:36:52.372140085 +0000 2024-03-18T00:42:09.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T00:42:09.350 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:09 smithi107 ceph-mon[36270]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:09.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:09 smithi107 ceph-mon[36270]: Reconfiguring mon.c (monmap changed)... 2024-03-18T00:42:09.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:09 smithi107 ceph-mon[36270]: Reconfiguring daemon mon.c on smithi107 2024-03-18T00:42:09.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:09 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:09.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:09 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:09.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:09 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:09.353 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-18T00:42:09.353 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-18T00:42:09.353 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000284096 s, 1.8 MB/s 2024-03-18T00:42:09.354 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T00:42:09.411 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-03-18T00:42:09.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:09 smithi106 ceph-mon[35899]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:09.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:09 smithi106 ceph-mon[35899]: Reconfiguring mon.c (monmap changed)... 2024-03-18T00:42:09.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:09 smithi106 ceph-mon[35899]: Reconfiguring daemon mon.c on smithi107 2024-03-18T00:42:09.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:09 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:09.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:09 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:09.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:09 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T00:42:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:09.466 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-18T00:42:09.466 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:09.466 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:09.466 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-18 00:41:46.061764344 +0000 2024-03-18T00:42:09.466 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-18 00:36:52.729134187 +0000 2024-03-18T00:42:09.466 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-18 00:36:52.729134187 +0000 2024-03-18T00:42:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-18 00:36:52.729134187 +0000 2024-03-18T00:42:09.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T00:42:09.531 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-18T00:42:09.531 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-18T00:42:09.531 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000210449 s, 2.4 MB/s 2024-03-18T00:42:09.532 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T00:42:09.587 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-03-18T00:42:09.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:09 smithi049 ceph-mon[32130]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:09.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:09 smithi049 ceph-mon[32130]: Reconfiguring mon.c (monmap changed)... 2024-03-18T00:42:09.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:09 smithi049 ceph-mon[32130]: Reconfiguring daemon mon.c on smithi107 2024-03-18T00:42:09.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:09 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:09.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:09 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:09.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:09 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:09.641 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T00:42:09.641 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:09.641 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-18T00:42:09.641 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:09.642 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:09.642 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-18 00:41:46.061764344 +0000 2024-03-18T00:42:09.642 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-18 00:36:53.072128522 +0000 2024-03-18T00:42:09.642 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-18 00:36:53.072128522 +0000 2024-03-18T00:42:09.642 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-18 00:36:53.072128522 +0000 2024-03-18T00:42:09.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T00:42:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-18T00:42:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-18T00:42:09.707 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000170947 s, 3.0 MB/s 2024-03-18T00:42:09.708 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T00:42:09.763 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-03-18T00:42:09.817 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T00:42:09.817 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T00:42:09.817 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-18T00:42:09.818 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:42:09.818 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T00:42:09.818 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-18 00:41:46.062764328 +0000 2024-03-18T00:42:09.818 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-18 00:36:53.412122905 +0000 2024-03-18T00:42:09.818 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-18 00:36:53.412122905 +0000 2024-03-18T00:42:09.818 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-18 00:36:53.412122905 +0000 2024-03-18T00:42:09.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T00:42:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-18T00:42:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-18T00:42:09.883 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000238171 s, 2.1 MB/s 2024-03-18T00:42:09.885 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T00:42:09.940 INFO:tasks.cephadm:Deploying osd.0 on smithi049 with /dev/vg_nvme/lv_4... 2024-03-18T00:42:09.940 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T00:42:10.182 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:42:10.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:10.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:10.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:10.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T00:42:10.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T00:42:10.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T00:42:10.498 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T00:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T00:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T00:42:10.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:10.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:10.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:10.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:10.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T00:42:10.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T00:42:10.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T00:42:10.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:11.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:11 smithi049 ceph-mon[32130]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:11.370 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:11 smithi049 ceph-mon[32130]: Deploying daemon mgr.z on smithi107 2024-03-18T00:42:11.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:11 smithi107 ceph-mon[36270]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:11.457 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:11 smithi107 ceph-mon[36270]: Deploying daemon mgr.z on smithi107 2024-03-18T00:42:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:11 smithi106 ceph-mon[35899]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:11.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:11 smithi106 ceph-mon[35899]: Deploying daemon mgr.z on smithi107 2024-03-18T00:42:11.835 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:11 smithi107 systemd[1]: Starting Ceph mgr.z for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T00:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T00:42:11.922 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-03-18T00:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-03-18T00:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0081672 s, 1.3 GB/s 2024-03-18T00:42:11.923 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-03-18T00:42:12.138 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:11 smithi107 podman[38114]: 2024-03-18 00:42:11.738208214 +0000 UTC m=+0.018261122 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:42:12.138 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:11 smithi107 podman[38114]: 2024-03-18 00:42:11.853234052 +0000 UTC m=+0.133286964 container create 2d7aad30a0be14c5744966df721e0ddc434248c30a84a2c1f4c6c87c4575ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-18T00:42:12.139 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:12 smithi107 podman[38114]: 2024-03-18 00:42:12.056035772 +0000 UTC m=+0.336088664 container init 2d7aad30a0be14c5744966df721e0ddc434248c30a84a2c1f4c6c87c4575ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-18T00:42:12.139 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:12 smithi107 podman[38114]: 2024-03-18 00:42:12.058552389 +0000 UTC m=+0.338605274 container start 2d7aad30a0be14c5744966df721e0ddc434248c30a84a2c1f4c6c87c4575ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T00:42:12.139 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:12 smithi107 ceph-mgr[38133]: -- 172.21.15.107:0/1442031581 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d66883a5a0 con 0x55d66880e800 2024-03-18T00:42:12.139 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:12 smithi107 bash[38114]: 2d7aad30a0be14c5744966df721e0ddc434248c30a84a2c1f4c6c87c4575ff99 2024-03-18T00:42:12.430 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:12 smithi107 systemd[1]: Started Ceph mgr.z for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:42:12.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi049:vg_nvme/lv_4 2024-03-18T00:42:12.796 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:42:12.930 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:12.598+0000 7f2f87d24200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T00:42:12.930 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:12.690+0000 7f2f87d24200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T00:42:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:13 smithi106 ceph-mon[35899]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T00:42:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T00:42:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T00:42:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:13 smithi106 ceph-mon[35899]: Deploying daemon mgr.y on smithi106 2024-03-18T00:42:13.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:13 smithi049 ceph-mon[32130]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:13.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:13.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:13.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:13.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T00:42:13.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T00:42:13.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T00:42:13.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:13.481 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:13 smithi049 ceph-mon[32130]: Deploying daemon mgr.y on smithi106 2024-03-18T00:42:13.481 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:13.339+0000 7f2f87d24200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:13.434+0000 7f2f87d24200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-mon[36270]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:13.482 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-mon[36270]: Deploying daemon mgr.y on smithi106 2024-03-18T00:42:13.768 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:13.582+0000 7f2f87d24200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T00:42:13.768 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:13.682+0000 7f2f87d24200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T00:42:13.768 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:13.767+0000 7f2f87d24200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T00:42:14.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 -- 172.21.15.49:0/265440443 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102db0 msgr2=0x7f4df4109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/265440443 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102db0 0x7f4df4109660 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00b0d0 tx=0x7f4dec00b3e0 comp rx=0 tx=0).stop 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 -- 172.21.15.49:0/265440443 shutdown_connections 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/265440443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df4101210 0x7f4df4101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/265440443 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102db0 0x7f4df4109660 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/265440443 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df4102410 0x7f4df4102870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 -- 172.21.15.49:0/265440443 >> 172.21.15.49:0/265440443 conn(0x7f4df40fc7d0 msgr2=0x7f4df40febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 -- 172.21.15.49:0/265440443 shutdown_connections 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 -- 172.21.15.49:0/265440443 wait complete. 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 Processor -- start 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.216+0000 7f4dfba0c700 1 -- start start 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4dfba0c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df4101210 0x7f4df4198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4dfba0c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102410 0x7f4df4198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4dfba0c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df4102db0 0x7f4df419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:14.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4dfba0c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4df410ce40 con 0x7f4df4102410 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4dfba0c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4df410ccc0 con 0x7f4df4102db0 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4dfba0c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4df410cfc0 con 0x7f4df4101210 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df8fa7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102410 0x7f4df4198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df8fa7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102410 0x7f4df4198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38642/0 (socket says 172.21.15.49:38642) 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df8fa7700 1 -- 172.21.15.49:0/1264005197 learned_addr learned my addr 172.21.15.49:0/1264005197 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df97a8700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df4101210 0x7f4df4198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df9fa9700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df4102db0 0x7f4df419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df8fa7700 1 -- 172.21.15.49:0/1264005197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df4101210 msgr2=0x7f4df4198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df8fa7700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df4101210 0x7f4df4198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df8fa7700 1 -- 172.21.15.49:0/1264005197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df4102db0 msgr2=0x7f4df419fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df8fa7700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df4102db0 0x7f4df419fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df8fa7700 1 -- 172.21.15.49:0/1264005197 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df41a01d0 con 0x7f4df4102410 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df97a8700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df4101210 0x7f4df4198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:42:14.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.217+0000 7f4df8fa7700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102410 0x7f4df4198ab0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4de400d9b0 tx=0x7f4de400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:14.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.218+0000 7f4dea7fc700 1 -- 172.21.15.49:0/1264005197 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4016070 con 0x7f4df4102410 2024-03-18T00:42:14.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.218+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df41a04c0 con 0x7f4df4102410 2024-03-18T00:42:14.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.218+0000 7f4dea7fc700 1 -- 172.21.15.49:0/1264005197 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de4004d10 con 0x7f4df4102410 2024-03-18T00:42:14.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.218+0000 7f4dea7fc700 1 -- 172.21.15.49:0/1264005197 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4005530 con 0x7f4df4102410 2024-03-18T00:42:14.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.218+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df41a0a00 con 0x7f4df4102410 2024-03-18T00:42:14.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.218+0000 7f4dea7fc700 1 -- 172.21.15.49:0/1264005197 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f4de4004830 con 0x7f4df4102410 2024-03-18T00:42:14.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.219+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df4068f90 con 0x7f4df4102410 2024-03-18T00:42:14.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.219+0000 7f4dea7fc700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4de0039200 0x7f4de003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:14.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.219+0000 7f4dea7fc700 1 -- 172.21.15.49:0/1264005197 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4de404beb0 con 0x7f4df4102410 2024-03-18T00:42:14.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.219+0000 7f4df97a8700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4de0039200 0x7f4de003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:14.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.219+0000 7f4df97a8700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4de0039200 0x7f4de003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4df4102270 tx=0x7f4df0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:14.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.224+0000 7f4dea7fc700 1 -- 172.21.15.49:0/1264005197 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4de401f020 con 0x7f4df4102410 2024-03-18T00:42:14.325 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:14.035+0000 7f2f87d24200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T00:42:14.325 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:14.144+0000 7f2f87d24200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T00:42:14.325 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:14.230+0000 7f2f87d24200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T00:42:14.325 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:14.325+0000 7f2f87d24200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T00:42:14.425 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:14 smithi106 systemd[1]: Starting Ceph mgr.y for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:42:14.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:14.439+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4df406b140 con 0x7f4de0039200 2024-03-18T00:42:14.680 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:14.503+0000 7f2f87d24200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T00:42:14.696 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:14 smithi106 podman[37272]: 2024-03-18 00:42:14.328237932 +0000 UTC m=+0.017360692 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:42:14.696 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:14 smithi106 podman[37272]: 2024-03-18 00:42:14.431618981 +0000 UTC m=+0.120741747 container create c2df8d4435b81f3cba5a8c64947ab1fd563fdaacd5001ab50c6a3c1d7111f15e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:42:14.696 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:14 smithi106 podman[37272]: 2024-03-18 00:42:14.642661858 +0000 UTC m=+0.331784616 container init c2df8d4435b81f3cba5a8c64947ab1fd563fdaacd5001ab50c6a3c1d7111f15e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-18T00:42:14.696 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:14 smithi106 podman[37272]: 2024-03-18 00:42:14.645207434 +0000 UTC m=+0.334330196 container start c2df8d4435b81f3cba5a8c64947ab1fd563fdaacd5001ab50c6a3c1d7111f15e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:42:14.696 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:14 smithi106 ceph-mgr[37292]: -- 172.21.15.106:0/3821402925 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c2e94305a0 con 0x55c2e9405000 2024-03-18T00:42:15.052 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:14 smithi106 bash[37272]: c2df8d4435b81f3cba5a8c64947ab1fd563fdaacd5001ab50c6a3c1d7111f15e 2024-03-18T00:42:15.052 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:14 smithi106 systemd[1]: Started Ceph mgr.y for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:42:15.302 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:15.203+0000 7fa341e46200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-mon[36270]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:15 smithi049 ceph-mon[32130]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:15.572 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-mon[35899]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.573 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.574 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:15.574 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:15.695 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:15.302+0000 7fa341e46200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T00:42:15.868 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:15.491+0000 7f2f87d24200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T00:42:15.868 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:15.642+0000 7f2f87d24200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T00:42:16.111 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:16.026+0000 7fa341e46200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T00:42:16.180 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:15 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:15.981+0000 7f2f87d24200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T00:42:16.180 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:16 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:16.072+0000 7f2f87d24200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T00:42:16.313 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:16 smithi049 ceph-mon[32130]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:42:16.367 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-mon[35899]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:42:16.367 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:16.119+0000 7fa341e46200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T00:42:16.367 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:16.263+0000 7fa341e46200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T00:42:16.367 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:16.366+0000 7fa341e46200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T00:42:16.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:16 smithi107 ceph-mon[36270]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:42:16.680 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:16 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:16.269+0000 7f2f87d24200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:42:16.680 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:16 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:16.462+0000 7f2f87d24200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T00:42:16.695 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:16.452+0000 7fa341e46200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T00:42:16.996 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:16.715+0000 7fa341e46200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T00:42:16.997 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:16.821+0000 7fa341e46200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T00:42:16.997 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:16.904+0000 7fa341e46200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T00:42:16.997 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:16.996+0000 7fa341e46200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T00:42:17.003 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:16 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:16.746+0000 7f2f87d24200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:42:17.430 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:17.003+0000 7f2f87d24200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T00:42:17.430 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:17.239+0000 7f2f87d24200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T00:42:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1120598482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba2a7ba2-663e-47ea-99f4-f190dece19d2"}]: dispatch 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1120598482' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba2a7ba2-663e-47ea-99f4-f190dece19d2"}]': finished 2024-03-18T00:42:17.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T00:42:17.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:17.445 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:17.167+0000 7fa341e46200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T00:42:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:17.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1120598482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba2a7ba2-663e-47ea-99f4-f190dece19d2"}]: dispatch 2024-03-18T00:42:17.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1120598482' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba2a7ba2-663e-47ea-99f4-f190dece19d2"}]': finished 2024-03-18T00:42:17.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T00:42:17.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:17.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:17.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.502 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1120598482' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba2a7ba2-663e-47ea-99f4-f190dece19d2"}]: dispatch 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1120598482' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba2a7ba2-663e-47ea-99f4-f190dece19d2"}]': finished 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T00:42:17.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:18.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:18 smithi107 ceph-mon[36270]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T00:42:18.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:18 smithi107 ceph-mon[36270]: Reconfiguring daemon mgr.x on smithi049 2024-03-18T00:42:18.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:18 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/489053131' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:42:18.430 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:18 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:18.033+0000 7f2f87d24200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T00:42:18.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:18 smithi106 ceph-mon[35899]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T00:42:18.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:18 smithi106 ceph-mon[35899]: Reconfiguring daemon mgr.x on smithi049 2024-03-18T00:42:18.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:18 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/489053131' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:42:18.446 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:18.113+0000 7fa341e46200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T00:42:18.446 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:18.240+0000 7fa341e46200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T00:42:18.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:18 smithi049 ceph-mon[32130]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T00:42:18.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:18 smithi049 ceph-mon[32130]: Reconfiguring daemon mgr.x on smithi049 2024-03-18T00:42:18.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:18 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/489053131' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:42:18.844 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:18.566+0000 7fa341e46200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T00:42:18.844 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:18.652+0000 7fa341e46200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T00:42:18.844 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:18.843+0000 7fa341e46200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:42:18.909 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:18 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:18.596+0000 7f2f87d24200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T00:42:18.909 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:18 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:18.693+0000 7f2f87d24200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T00:42:18.909 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 00:42:18 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z[38129]: 2024-03-18T00:42:18.909+0000 7f2f87d24200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T00:42:19.195 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:19.022+0000 7fa341e46200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T00:42:19.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:19.225+0000 7f4dea7fc700 1 -- 172.21.15.49:0/1264005197 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f4de400b750 con 0x7f4df4102410 2024-03-18T00:42:19.554 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:19.299+0000 7fa341e46200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T00:42:19.555 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:19.554+0000 7fa341e46200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T00:42:19.555 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-mon[35899]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:19.555 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-mon[35899]: Standby manager daemon z started 2024-03-18T00:42:19.555 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-mon[35899]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T00:42:19.555 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-mon[35899]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T00:42:19.555 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-mon[35899]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T00:42:19.555 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-mon[35899]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T00:42:19.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:19 smithi049 ceph-mon[32130]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:19.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:19 smithi049 ceph-mon[32130]: Standby manager daemon z started 2024-03-18T00:42:19.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:19 smithi049 ceph-mon[32130]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T00:42:19.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:19 smithi049 ceph-mon[32130]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T00:42:19.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:19 smithi049 ceph-mon[32130]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T00:42:19.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:19 smithi049 ceph-mon[32130]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T00:42:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:19 smithi107 ceph-mon[36270]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:19 smithi107 ceph-mon[36270]: Standby manager daemon z started 2024-03-18T00:42:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:19 smithi107 ceph-mon[36270]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T00:42:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:19 smithi107 ceph-mon[36270]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T00:42:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:19 smithi107 ceph-mon[36270]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T00:42:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:19 smithi107 ceph-mon[36270]: from='mgr.? 172.21.15.107:0/3862128318' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T00:42:19.945 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:19.787+0000 7fa341e46200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T00:42:20.573 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:20 smithi106 ceph-mon[35899]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-18T00:42:20.574 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T00:42:20.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:20 smithi049 ceph-mon[32130]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-18T00:42:20.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T00:42:20.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:20 smithi107 ceph-mon[36270]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-18T00:42:20.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T00:42:20.945 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:20 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:20.573+0000 7fa341e46200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T00:42:21.438 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:21 smithi106 ceph-mon[35899]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:21.439 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:21 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:21.128+0000 7fa341e46200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T00:42:21.439 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:21 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:21.224+0000 7fa341e46200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T00:42:21.439 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 00:42:21 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y[37288]: 2024-03-18T00:42:21.438+0000 7fa341e46200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T00:42:21.441 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:21 smithi049 ceph-mon[32130]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:21 smithi107 ceph-mon[36270]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:22.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:22.237+0000 7f4dea7fc700 1 -- 172.21.15.49:0/1264005197 <== mon.0 v2:172.21.15.49:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f4de400a250 con 0x7f4df4102410 2024-03-18T00:42:22.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:22 smithi049 ceph-mon[32130]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:22.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:22 smithi049 ceph-mon[32130]: Standby manager daemon y started 2024-03-18T00:42:22.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:22 smithi049 ceph-mon[32130]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T00:42:22.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:22 smithi049 ceph-mon[32130]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T00:42:22.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:22 smithi049 ceph-mon[32130]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T00:42:22.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:22 smithi049 ceph-mon[32130]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T00:42:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:22 smithi107 ceph-mon[36270]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:22 smithi107 ceph-mon[36270]: Standby manager daemon y started 2024-03-18T00:42:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:22 smithi107 ceph-mon[36270]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T00:42:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:22 smithi107 ceph-mon[36270]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T00:42:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:22 smithi107 ceph-mon[36270]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T00:42:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:22 smithi107 ceph-mon[36270]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T00:42:22.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:22 smithi106 ceph-mon[35899]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:22 smithi106 ceph-mon[35899]: Standby manager daemon y started 2024-03-18T00:42:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:22 smithi106 ceph-mon[35899]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T00:42:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:22 smithi106 ceph-mon[35899]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T00:42:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:22 smithi106 ceph-mon[35899]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T00:42:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:22 smithi106 ceph-mon[35899]: from='mgr.? 172.21.15.106:0/4008830335' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T00:42:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:23 smithi049 ceph-mon[32130]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-18T00:42:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T00:42:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:23 smithi107 ceph-mon[36270]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-18T00:42:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T00:42:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:23.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:23.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:23 smithi106 ceph-mon[35899]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-18T00:42:23.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T00:42:23.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:23.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:23.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:23.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:23.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:23.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:24 smithi049 ceph-mon[32130]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:24.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:24 smithi107 ceph-mon[36270]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:24.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:24 smithi106 ceph-mon[35899]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:25.281 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T00:42:25.281 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:25.281 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:25 smithi049 ceph-mon[32130]: Deploying daemon osd.0 on smithi049 2024-03-18T00:42:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T00:42:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:25 smithi107 ceph-mon[36270]: Deploying daemon osd.0 on smithi049 2024-03-18T00:42:25.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T00:42:25.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:25.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:25 smithi106 ceph-mon[35899]: Deploying daemon osd.0 on smithi049 2024-03-18T00:42:26.585 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:26 smithi049 ceph-mon[32130]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:26 smithi107 ceph-mon[36270]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:26.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:26 smithi106 ceph-mon[35899]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:28.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:28 smithi106 ceph-mon[35899]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:28.702 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:28 smithi049 ceph-mon[32130]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:28 smithi107 ceph-mon[36270]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:30.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:30 smithi049 ceph-mon[32130]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:30 smithi107 ceph-mon[36270]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:30.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:30 smithi106 ceph-mon[35899]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:31.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:31.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:31.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:31.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:31.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:31.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:31.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:31.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:31.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:32.827 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:32 smithi049 ceph-mon[32130]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:32 smithi107 ceph-mon[36270]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:32.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:32 smithi106 ceph-mon[35899]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:34.044 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 0 on host 'smithi049' 2024-03-18T00:42:34.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.044+0000 7f4dea7fc700 1 -- 172.21.15.49:0/1264005197 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4df406b140 con 0x7f4de0039200 2024-03-18T00:42:34.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4de0039200 msgr2=0x7f4de003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:34.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4de0039200 0x7f4de003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4df4102270 tx=0x7f4df0009260 comp rx=0 tx=0).stop 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102410 msgr2=0x7f4df4198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102410 0x7f4df4198ab0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4de400d9b0 tx=0x7f4de400dcc0 comp rx=0 tx=0).stop 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 shutdown_connections 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4de0039200 0x7f4de003b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df4101210 0x7f4df4198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4df4102db0 0x7f4df419fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 --2- 172.21.15.49:0/1264005197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4df4102410 0x7f4df4198ab0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 >> 172.21.15.49:0/1264005197 conn(0x7f4df40fc7d0 msgr2=0x7f4df406aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.046+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 shutdown_connections 2024-03-18T00:42:34.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:42:34.047+0000 7f4dfba0c700 1 -- 172.21.15.49:0/1264005197 wait complete. 2024-03-18T00:42:34.866 DEBUG:teuthology.orchestra.run.smithi049:osd.0> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.0.service 2024-03-18T00:42:34.868 INFO:tasks.cephadm:Deploying osd.1 on smithi106 with /dev/vg_nvme/lv_4... 2024-03-18T00:42:34.868 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T00:42:35.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:35.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.026 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:35.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:35.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:35.027 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:34 smithi049 ceph-mon[32130]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T00:42:35.110 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:42:35.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:35.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:35.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:35.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:35.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:34 smithi107 ceph-mon[36270]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T00:42:35.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:35.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:35.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:35.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:35.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:35.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:34 smithi106 ceph-mon[35899]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T00:42:36.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:36 smithi049 ceph-mon[32130]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T00:42:36.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:36 smithi049 ceph-mon[32130]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T00:42:36.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:36 smithi049 ceph-mon[32130]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:42:36.325 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:36 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:36.378 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:36 smithi106 ceph-mon[35899]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T00:42:36.378 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:36 smithi106 ceph-mon[35899]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T00:42:36.378 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:36 smithi106 ceph-mon[35899]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:42:36.378 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:36 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:36.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:36 smithi107 ceph-mon[36270]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T00:42:36.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:36 smithi107 ceph-mon[36270]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T00:42:36.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:36 smithi107 ceph-mon[36270]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:42:36.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:36 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:36.595 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:42:36 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0[46641]: 2024-03-18T00:42:36.522+0000 7f4916d9e700 -1 osd.0 0 waiting for initial osdmap 2024-03-18T00:42:36.595 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:42:36 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0[46641]: 2024-03-18T00:42:36.526+0000 7f490eb89700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:42:36.930 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T00:42:36.930 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T00:42:36.931 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-18T00:42:36.931 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-18T00:42:36.931 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012183 s, 861 MB/s 2024-03-18T00:42:36.931 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-18T00:42:37.248 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:37 smithi106 ceph-mon[35899]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:37.248 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:37 smithi106 ceph-mon[35899]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:42:37.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:37 smithi106 ceph-mon[35899]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T00:42:37.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:37.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:37.249 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:37 smithi106 ceph-mon[35899]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' 2024-03-18T00:42:37.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:37 smithi049 ceph-mon[32130]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:37.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:37 smithi049 ceph-mon[32130]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:42:37.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:37 smithi049 ceph-mon[32130]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T00:42:37.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:37.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:37.303 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:37 smithi049 ceph-mon[32130]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' 2024-03-18T00:42:37.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:37 smithi107 ceph-mon[36270]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:37.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:37 smithi107 ceph-mon[36270]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:42:37.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:37 smithi107 ceph-mon[36270]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T00:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:37.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:37 smithi107 ceph-mon[36270]: from='osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645]' entity='osd.0' 2024-03-18T00:42:37.569 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_4 2024-03-18T00:42:37.809 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:42:38.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:38 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:42:38.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:38 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:42:38.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:38.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:38 smithi106 ceph-mon[35899]: osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] boot 2024-03-18T00:42:38.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:38 smithi106 ceph-mon[35899]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T00:42:38.101 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:38.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:38.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:38.102 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:42:38.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:38 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:42:38.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:38 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:42:38.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:38.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:38 smithi049 ceph-mon[32130]: osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] boot 2024-03-18T00:42:38.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:38 smithi049 ceph-mon[32130]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T00:42:38.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:38.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:38.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:38.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:42:38.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:38 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:42:38.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:38 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:42:38.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:38.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:38 smithi107 ceph-mon[36270]: osd.0 [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] boot 2024-03-18T00:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:38 smithi107 ceph-mon[36270]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T00:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T00:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:38.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:42:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:39 smithi106 ceph-mon[35899]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:39 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi049 2024-03-18T00:42:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:39 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi049 to 17088M 2024-03-18T00:42:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:39.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:39 smithi049 ceph-mon[32130]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:39.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:39 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi049 2024-03-18T00:42:39.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:39 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi049 to 17088M 2024-03-18T00:42:39.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:39.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:39.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:39.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:39.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:39 smithi107 ceph-mon[36270]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T00:42:39.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:39 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi049 2024-03-18T00:42:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:39 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi049 to 17088M 2024-03-18T00:42:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:39.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:39.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 -- 172.21.15.106:0/2717935295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0102300 msgr2=0x7fedd0102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:39.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2717935295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0102300 0x7fedd0102720 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fedcc0099c0 tx=0x7fedcc009cd0 comp rx=0 tx=0).stop 2024-03-18T00:42:39.520 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 -- 172.21.15.106:0/2717935295 shutdown_connections 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2717935295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd0103ea0 0x7fedd010a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2717935295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0102300 0x7fedd0102720 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2717935295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd0103500 0x7fedd0103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 -- 172.21.15.106:0/2717935295 >> 172.21.15.106:0/2717935295 conn(0x7fedd00fd8e0 msgr2=0x7fedd00ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 -- 172.21.15.106:0/2717935295 shutdown_connections 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 -- 172.21.15.106:0/2717935295 wait complete. 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 Processor -- start 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 -- start start 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd0102300 0x7fedd0076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd0103500 0x7fedd00746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0103ea0 0x7fedd0074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fedd010dec0 con 0x7fedd0103500 2024-03-18T00:42:39.521 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fedd010dd40 con 0x7fedd0103ea0 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.519+0000 7fedd77b7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fedd010e040 con 0x7fedd0102300 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5d54700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0103ea0 0x7fedd0074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd4d52700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd0103500 0x7fedd00746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5d54700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0103ea0 0x7fedd0074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:36550/0 (socket says 172.21.15.106:36550) 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5d54700 1 -- 172.21.15.106:0/2135651943 learned_addr learned my addr 172.21.15.106:0/2135651943 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5553700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd0102300 0x7fedd0076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5d54700 1 -- 172.21.15.106:0/2135651943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd0102300 msgr2=0x7fedd0076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5d54700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd0102300 0x7fedd0076070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5d54700 1 -- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd0103500 msgr2=0x7fedd00746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5d54700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd0103500 0x7fedd00746c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5d54700 1 -- 172.21.15.106:0/2135651943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedd00754b0 con 0x7fedd0103ea0 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd4d52700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd0103500 0x7fedd00746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:42:39.522 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd5d54700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0103ea0 0x7fedd0074c00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fedc8007ee0 tx=0x7fedc800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:39.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedc67fc700 1 -- 172.21.15.106:0/2135651943 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8004bb0 con 0x7fedd0103ea0 2024-03-18T00:42:39.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedd01a2630 con 0x7fedd0103ea0 2024-03-18T00:42:39.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedc67fc700 1 -- 172.21.15.106:0/2135651943 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc8004d50 con 0x7fedd0103ea0 2024-03-18T00:42:39.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedc67fc700 1 -- 172.21.15.106:0/2135651943 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc8005960 con 0x7fedd0103ea0 2024-03-18T00:42:39.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.520+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedd01a2b40 con 0x7fedd0103ea0 2024-03-18T00:42:39.523 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.521+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedd0068f90 con 0x7fedd0103ea0 2024-03-18T00:42:39.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.523+0000 7fedc67fc700 1 -- 172.21.15.106:0/2135651943 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fedc801a020 con 0x7fedd0103ea0 2024-03-18T00:42:39.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.523+0000 7fedc67fc700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedbc0a2880 0x7fedbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:42:39.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.524+0000 7fedc67fc700 1 -- 172.21.15.106:0/2135651943 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fedc80ccd50 con 0x7fedd0103ea0 2024-03-18T00:42:39.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.524+0000 7fedd5553700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedbc0a2880 0x7fedbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:42:39.524 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.524+0000 7fedd5553700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedbc0a2880 0x7fedbc0a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fedcc003890 tx=0x7fedcc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:42:39.526 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.527+0000 7fedc67fc700 1 -- 172.21.15.106:0/2135651943 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fedc8010070 con 0x7fedd0103ea0 2024-03-18T00:42:39.726 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:39.727+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fedd0062710 con 0x7fedbc0a2880 2024-03-18T00:42:40.324 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:40 smithi106 ceph-mon[35899]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T00:42:40.325 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:40 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:42:40.325 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:40 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:42:40.325 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:40 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:40.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:40 smithi049 ceph-mon[32130]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T00:42:40.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:42:40.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:42:40.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:40.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:40 smithi107 ceph-mon[36270]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T00:42:40.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:40 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:42:40.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:40 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:42:40.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:40 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:41.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:41 smithi107 ceph-mon[36270]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:41.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:41 smithi107 ceph-mon[36270]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:42:41.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:41 smithi049 ceph-mon[32130]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:41 smithi049 ceph-mon[32130]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:42:41.382 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:41 smithi106 ceph-mon[35899]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:41.382 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:41 smithi106 ceph-mon[35899]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:42:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:43 smithi049 ceph-mon[32130]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:43 smithi049 ceph-mon[32130]: from='client.? 172.21.15.106:0/2134395501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e16bd7a7-f97d-4c1c-b014-c3252101dd2a"}]: dispatch 2024-03-18T00:42:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:43 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e16bd7a7-f97d-4c1c-b014-c3252101dd2a"}]: dispatch 2024-03-18T00:42:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:43 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e16bd7a7-f97d-4c1c-b014-c3252101dd2a"}]': finished 2024-03-18T00:42:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:43 smithi049 ceph-mon[32130]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T00:42:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:42:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:43 smithi049 ceph-mon[32130]: from='client.? 172.21.15.106:0/2093319404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:42:43.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:43 smithi107 ceph-mon[36270]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:43 smithi107 ceph-mon[36270]: from='client.? 172.21.15.106:0/2134395501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e16bd7a7-f97d-4c1c-b014-c3252101dd2a"}]: dispatch 2024-03-18T00:42:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:43 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e16bd7a7-f97d-4c1c-b014-c3252101dd2a"}]: dispatch 2024-03-18T00:42:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:43 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e16bd7a7-f97d-4c1c-b014-c3252101dd2a"}]': finished 2024-03-18T00:42:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:43 smithi107 ceph-mon[36270]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T00:42:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:42:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:43 smithi107 ceph-mon[36270]: from='client.? 172.21.15.106:0/2093319404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:42:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:43 smithi106 ceph-mon[35899]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:43 smithi106 ceph-mon[35899]: from='client.? 172.21.15.106:0/2134395501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e16bd7a7-f97d-4c1c-b014-c3252101dd2a"}]: dispatch 2024-03-18T00:42:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:43 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e16bd7a7-f97d-4c1c-b014-c3252101dd2a"}]: dispatch 2024-03-18T00:42:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:43 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e16bd7a7-f97d-4c1c-b014-c3252101dd2a"}]': finished 2024-03-18T00:42:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:43 smithi106 ceph-mon[35899]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T00:42:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:42:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:43 smithi106 ceph-mon[35899]: from='client.? 172.21.15.106:0/2093319404' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:42:45.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:45 smithi049 ceph-mon[32130]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:45.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:45 smithi107 ceph-mon[36270]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:45.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:45 smithi106 ceph-mon[35899]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:47 smithi106 ceph-mon[35899]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:47.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:47 smithi049 ceph-mon[32130]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:47 smithi107 ceph-mon[36270]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:49.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:49 smithi049 ceph-mon[32130]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:49 smithi107 ceph-mon[36270]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:49.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:49 smithi106 ceph-mon[35899]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:50.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T00:42:50.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:50.429 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T00:42:50.430 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:50.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T00:42:50.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:51.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:51 smithi106 ceph-mon[35899]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:51.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:51 smithi106 ceph-mon[35899]: Deploying daemon osd.1 on smithi106 2024-03-18T00:42:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:51 smithi049 ceph-mon[32130]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:51 smithi049 ceph-mon[32130]: Deploying daemon osd.1 on smithi106 2024-03-18T00:42:51.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:51 smithi107 ceph-mon[36270]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:51.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:51 smithi107 ceph-mon[36270]: Deploying daemon osd.1 on smithi106 2024-03-18T00:42:53.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:53 smithi106 ceph-mon[35899]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:53 smithi049 ceph-mon[32130]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:53.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:53 smithi107 ceph-mon[36270]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:55.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:55 smithi106 ceph-mon[35899]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:55.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:55 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:55.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:55 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:55.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:55 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:55.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:55 smithi049 ceph-mon[32130]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:55.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:55 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:55.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:55 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:55.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:55 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:55 smithi107 ceph-mon[36270]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:55 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:55 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:55 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:42:57.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:57 smithi049 ceph-mon[32130]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:57.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:57 smithi107 ceph-mon[36270]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:57.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:57 smithi106 ceph-mon[35899]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:58 smithi107 ceph-mon[36270]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:58.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:58 smithi106 ceph-mon[35899]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:58.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:58.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:58.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:58.699 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:58.699 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:58.797 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.796+0000 7fedc67fc700 1 -- 172.21.15.106:0/2135651943 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fedd0062710 con 0x7fedbc0a2880 2024-03-18T00:42:58.797 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 1 on host 'smithi106' 2024-03-18T00:42:58.799 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedbc0a2880 msgr2=0x7fedbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:58.799 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedbc0a2880 0x7fedbc0a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fedcc003890 tx=0x7fedcc005fb0 comp rx=0 tx=0).stop 2024-03-18T00:42:58.799 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0103ea0 msgr2=0x7fedd0074c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:42:58.799 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0103ea0 0x7fedd0074c00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fedc8007ee0 tx=0x7fedc800d3b0 comp rx=0 tx=0).stop 2024-03-18T00:42:58.799 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 shutdown_connections 2024-03-18T00:42:58.799 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedbc0a2880 0x7fedbc0a4d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:58.800 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fedd0102300 0x7fedd0076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:58.800 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fedd0103ea0 0x7fedd0074c00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:58.800 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 --2- 172.21.15.106:0/2135651943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fedd0103500 0x7fedd00746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:42:58.800 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 >> 172.21.15.106:0/2135651943 conn(0x7fedd00fd8e0 msgr2=0x7fedd0106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:42:58.800 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 shutdown_connections 2024-03-18T00:42:58.800 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:42:58.799+0000 7fedd77b7700 1 -- 172.21.15.106:0/2135651943 wait complete. 2024-03-18T00:42:58.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:58 smithi049 ceph-mon[32130]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:42:58.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:58.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:58.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:42:58.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:42:58.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:42:59.424 DEBUG:teuthology.orchestra.run.smithi106:osd.1> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.1.service 2024-03-18T00:42:59.426 INFO:tasks.cephadm:Deploying osd.2 on smithi107 with /dev/vg_nvme/lv_4... 2024-03-18T00:42:59.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T00:42:59.668 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:43:00.063 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:59 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:00.063 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:59 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:00.063 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:59 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:00.063 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:59 smithi107 ceph-mon[36270]: from='osd.1 [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T00:43:00.063 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:42:59 smithi107 ceph-mon[36270]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T00:43:00.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:59 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:00.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:59 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:00.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:59 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:00.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:59 smithi049 ceph-mon[32130]: from='osd.1 [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T00:43:00.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:42:59 smithi049 ceph-mon[32130]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T00:43:00.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:59 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:00.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:59 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:00.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:59 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:00.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:59 smithi106 ceph-mon[35899]: from='osd.1 [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T00:43:00.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:42:59 smithi106 ceph-mon[35899]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T00:43:00.858 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:00 smithi106 ceph-mon[35899]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:43:00.859 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:00 smithi106 ceph-mon[35899]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T00:43:00.859 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:00 smithi106 ceph-mon[35899]: from='osd.1 [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:43:00.859 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:00 smithi106 ceph-mon[35899]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T00:43:00.859 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:00.859 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:00 smithi106 ceph-mon[35899]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:43:00.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:00 smithi107 ceph-mon[36270]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:43:00.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:00 smithi107 ceph-mon[36270]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T00:43:00.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:00 smithi107 ceph-mon[36270]: from='osd.1 [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:43:00.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:00 smithi107 ceph-mon[36270]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T00:43:00.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:00.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:00 smithi107 ceph-mon[36270]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:43:01.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:00 smithi049 ceph-mon[32130]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:43:01.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:00 smithi049 ceph-mon[32130]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T00:43:01.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:00 smithi049 ceph-mon[32130]: from='osd.1 [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:43:01.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:00 smithi049 ceph-mon[32130]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T00:43:01.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:01.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:00 smithi049 ceph-mon[32130]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:43:01.417 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T00:43:01.417 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T00:43:01.417 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-03-18T00:43:01.417 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-03-18T00:43:01.417 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113198 s, 926 MB/s 2024-03-18T00:43:01.417 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-03-18T00:43:01.696 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1[40489]: 2024-03-18T00:43:01.234+0000 7fd99accb700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T00:43:01.696 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1[40489]: 2024-03-18T00:43:01.239+0000 7fd992ab6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:43:01.989 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:01 smithi107 ceph-mon[36270]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:43:01.989 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:01 smithi107 ceph-mon[36270]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T00:43:01.989 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:01.990 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:01.990 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:01 smithi107 ceph-mon[36270]: from='osd.1 ' entity='osd.1' 2024-03-18T00:43:02.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_4 2024-03-18T00:43:02.075 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:01 smithi106 ceph-mon[35899]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:43:02.075 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:01 smithi106 ceph-mon[35899]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T00:43:02.075 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:01 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:02.075 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:01 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:02.075 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:01 smithi106 ceph-mon[35899]: from='osd.1 ' entity='osd.1' 2024-03-18T00:43:02.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:01 smithi049 ceph-mon[32130]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:43:02.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:01 smithi049 ceph-mon[32130]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T00:43:02.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:02.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:02.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:01 smithi049 ceph-mon[32130]: from='osd.1 ' entity='osd.1' 2024-03-18T00:43:02.259 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:43:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:43:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:43:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:43:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: osd.1 [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] boot 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:03.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:43:03.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:43:03.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:43:03.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:03.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: osd.1 [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] boot 2024-03-18T00:43:03.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T00:43:03.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:03.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:03.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:03.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:03.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:03.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:03.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:03.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: osd.1 [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] boot 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:03.197 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 -- 172.21.15.107:0/1788047223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102da0 msgr2=0x7f87f8109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/1788047223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102da0 0x7f87f8109650 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f87f000b0a0 tx=0x7f87f000b3b0 comp rx=0 tx=0).stop 2024-03-18T00:43:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 -- 172.21.15.107:0/1788047223 shutdown_connections 2024-03-18T00:43:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/1788047223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102da0 0x7f87f8109650 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/1788047223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87f8101200 0x7f87f8101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:03.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/1788047223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87f8102400 0x7f87f8102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 -- 172.21.15.107:0/1788047223 >> 172.21.15.107:0/1788047223 conn(0x7f87f80fc7a0 msgr2=0x7f87f80febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 -- 172.21.15.107:0/1788047223 shutdown_connections 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 -- 172.21.15.107:0/1788047223 wait complete. 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.648+0000 7f87fdc0b700 1 Processor -- start 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87fdc0b700 1 -- start start 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87fdc0b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87f8101200 0x7f87f8105190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87fdc0b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102400 0x7f87f81037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87fdc0b700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87f8102da0 0x7f87f8103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87fdc0b700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f87f810ce70 con 0x7f87f8101200 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87fdc0b700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f87f810ccf0 con 0x7f87f8102da0 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87fdc0b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f87f810cff0 con 0x7f87f8102400 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f6ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102400 0x7f87f81037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f6ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102400 0x7f87f81037e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:37404/0 (socket says 172.21.15.107:37404) 2024-03-18T00:43:03.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f6ffd700 1 -- 172.21.15.107:0/3652374893 learned_addr learned my addr 172.21.15.107:0/3652374893 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f77fe700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87f8101200 0x7f87f8105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f7fff700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87f8102da0 0x7f87f8103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f6ffd700 1 -- 172.21.15.107:0/3652374893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87f8102da0 msgr2=0x7f87f8103d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f6ffd700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87f8102da0 0x7f87f8103d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f6ffd700 1 -- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87f8101200 msgr2=0x7f87f8105190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f6ffd700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87f8101200 0x7f87f8105190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f6ffd700 1 -- 172.21.15.107:0/3652374893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f81045d0 con 0x7f87f8102400 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f6ffd700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102400 0x7f87f81037e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00d950 tx=0x7f87ec00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.649+0000 7f87f4ff9700 1 -- 172.21.15.107:0/3652374893 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec016070 con 0x7f87f8102400 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.650+0000 7f87f4ff9700 1 -- 172.21.15.107:0/3652374893 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87ec009e20 con 0x7f87f8102400 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.650+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87f81a2700 con 0x7f87f8102400 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.650+0000 7f87f4ff9700 1 -- 172.21.15.107:0/3652374893 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00f620 con 0x7f87f8102400 2024-03-18T00:43:03.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.650+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87f81a2c90 con 0x7f87f8102400 2024-03-18T00:43:03.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.650+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87f8068f90 con 0x7f87f8102400 2024-03-18T00:43:03.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.651+0000 7f87f4ff9700 1 -- 172.21.15.107:0/3652374893 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f87ec019070 con 0x7f87f8102400 2024-03-18T00:43:03.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.652+0000 7f87f4ff9700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87e00a2800 0x7f87e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:03.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.652+0000 7f87f4ff9700 1 -- 172.21.15.107:0/3652374893 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7f87ec0ccd80 con 0x7f87f8102400 2024-03-18T00:43:03.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.652+0000 7f87f77fe700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87e00a2800 0x7f87e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:03.653 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.653+0000 7f87f77fe700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87e00a2800 0x7f87e00a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87f8102260 tx=0x7f87e8009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:43:03.657 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.657+0000 7f87f4ff9700 1 -- 172.21.15.107:0/3652374893 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f87ec09b670 con 0x7f87f8102400 2024-03-18T00:43:03.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:03.862+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f87f80fe910 con 0x7f87e00a2800 2024-03-18T00:43:04.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:03 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi106 2024-03-18T00:43:04.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:03 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi106 to 17080M 2024-03-18T00:43:04.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:03 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi106 2024-03-18T00:43:04.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:03 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi106 to 17080M 2024-03-18T00:43:04.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:03 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi106 2024-03-18T00:43:04.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:03 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi106 to 17080M 2024-03-18T00:43:05.102 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:04 smithi107 ceph-mon[36270]: pgmap v64: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:05.102 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:04 smithi107 ceph-mon[36270]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T00:43:05.102 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:04 smithi107 ceph-mon[36270]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:43:05.102 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:43:05.102 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:43:05.103 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:05.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:04 smithi049 ceph-mon[32130]: pgmap v64: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:05.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:04 smithi049 ceph-mon[32130]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T00:43:05.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:04 smithi049 ceph-mon[32130]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:43:05.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:43:05.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:43:05.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:05.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:04 smithi106 ceph-mon[35899]: pgmap v64: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:05.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:04 smithi106 ceph-mon[35899]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T00:43:05.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:04 smithi106 ceph-mon[35899]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:43:05.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:43:05.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:43:05.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:07.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:06 smithi049 ceph-mon[32130]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:07.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:06 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/3801743488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fc48643-7a8d-4a34-9dcc-8b5b9098abe9"}]: dispatch 2024-03-18T00:43:07.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:06 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fc48643-7a8d-4a34-9dcc-8b5b9098abe9"}]: dispatch 2024-03-18T00:43:07.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:06 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fc48643-7a8d-4a34-9dcc-8b5b9098abe9"}]': finished 2024-03-18T00:43:07.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:06 smithi049 ceph-mon[32130]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T00:43:07.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:06 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:07.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:06 smithi107 ceph-mon[36270]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:07.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:06 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/3801743488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fc48643-7a8d-4a34-9dcc-8b5b9098abe9"}]: dispatch 2024-03-18T00:43:07.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:06 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fc48643-7a8d-4a34-9dcc-8b5b9098abe9"}]: dispatch 2024-03-18T00:43:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:06 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fc48643-7a8d-4a34-9dcc-8b5b9098abe9"}]': finished 2024-03-18T00:43:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:06 smithi107 ceph-mon[36270]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T00:43:07.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:06 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:07.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:06 smithi106 ceph-mon[35899]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:06 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/3801743488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fc48643-7a8d-4a34-9dcc-8b5b9098abe9"}]: dispatch 2024-03-18T00:43:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:06 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2fc48643-7a8d-4a34-9dcc-8b5b9098abe9"}]: dispatch 2024-03-18T00:43:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:06 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2fc48643-7a8d-4a34-9dcc-8b5b9098abe9"}]': finished 2024-03-18T00:43:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:06 smithi106 ceph-mon[35899]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T00:43:07.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:06 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:08.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/3729851819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:43:08.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:07 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/3729851819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:43:08.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:07 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/3729851819' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:43:09.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:08 smithi049 ceph-mon[32130]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:09.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:08 smithi107 ceph-mon[36270]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:09.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:08 smithi106 ceph-mon[35899]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:10.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:10 smithi107 ceph-mon[36270]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:11.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:10 smithi049 ceph-mon[32130]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:11.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:10 smithi106 ceph-mon[35899]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:12 smithi049 ceph-mon[32130]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:13.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:12 smithi107 ceph-mon[36270]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:13.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:12 smithi106 ceph-mon[35899]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T00:43:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:14.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T00:43:14.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:14.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T00:43:14.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:14.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:14 smithi107 ceph-mon[36270]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:14.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:14 smithi107 ceph-mon[36270]: Deploying daemon osd.2 on smithi107 2024-03-18T00:43:15.110 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:14 smithi106 ceph-mon[35899]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:15.111 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:14 smithi106 ceph-mon[35899]: Deploying daemon osd.2 on smithi107 2024-03-18T00:43:15.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:14 smithi049 ceph-mon[32130]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:15.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:14 smithi049 ceph-mon[32130]: Deploying daemon osd.2 on smithi107 2024-03-18T00:43:17.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:16 smithi049 ceph-mon[32130]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:17.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:16 smithi107 ceph-mon[36270]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:17.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:16 smithi106 ceph-mon[35899]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:19.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:18 smithi049 ceph-mon[32130]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:19.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:18 smithi107 ceph-mon[36270]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:19.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:18 smithi106 ceph-mon[35899]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:20.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:19 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:20.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:19 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:20.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:19 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:20.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:19 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:20.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:19 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:20.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:19 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:20.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:19 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:20.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:19 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:20.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:19 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:21.162 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:20 smithi107 ceph-mon[36270]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:21.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:20 smithi106 ceph-mon[35899]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:21.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:20 smithi049 ceph-mon[32130]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:22.631 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.632+0000 7f87f4ff9700 1 -- 172.21.15.107:0/3652374893 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f87f80fe910 con 0x7f87e00a2800 2024-03-18T00:43:22.632 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 2 on host 'smithi107' 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87e00a2800 msgr2=0x7f87e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87e00a2800 0x7f87e00a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87f8102260 tx=0x7f87e8009190 comp rx=0 tx=0).stop 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102400 msgr2=0x7f87f81037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102400 0x7f87f81037e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00d950 tx=0x7f87ec00dc60 comp rx=0 tx=0).stop 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 shutdown_connections 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f87f8102da0 0x7f87f8103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87f8101200 0x7f87f8105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87e00a2800 0x7f87e00a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 --2- 172.21.15.107:0/3652374893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f87f8102400 0x7f87f81037e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:22.634 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.633+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 >> 172.21.15.107:0/3652374893 conn(0x7f87f80fc7a0 msgr2=0x7f87f80fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:43:22.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.634+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 shutdown_connections 2024-03-18T00:43:22.635 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:43:22.634+0000 7f87fdc0b700 1 -- 172.21.15.107:0/3652374893 wait complete. 2024-03-18T00:43:22.812 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:22 smithi107 ceph-mon[36270]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:22.812 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:22.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:22.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:22.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:22.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:22.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:22.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:22.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:22 smithi049 ceph-mon[32130]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:23.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:22 smithi106 ceph-mon[35899]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:23.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:23.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:23.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:23.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:23.216 DEBUG:teuthology.orchestra.run.smithi107:osd.2> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.2.service 2024-03-18T00:43:23.218 INFO:tasks.cephadm:Deploying osd.3 on smithi049 with /dev/vg_nvme/lv_3... 2024-03-18T00:43:23.218 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T00:43:23.478 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:43:23.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:23 smithi049 ceph-mon[32130]: from='osd.2 [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T00:43:23.852 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:23 smithi049 ceph-mon[32130]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T00:43:24.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:23 smithi107 ceph-mon[36270]: from='osd.2 [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T00:43:24.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:23 smithi107 ceph-mon[36270]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T00:43:24.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:23 smithi106 ceph-mon[35899]: from='osd.2 [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T00:43:24.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:23 smithi106 ceph-mon[35899]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T00:43:24.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:24 smithi049 ceph-mon[32130]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:24.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:24 smithi049 ceph-mon[32130]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T00:43:24.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:24 smithi049 ceph-mon[32130]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T00:43:24.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:24 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:24.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:24 smithi049 ceph-mon[32130]: from='osd.2 [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:43:24.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:24 smithi049 ceph-mon[32130]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:43:24.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:24 smithi107 ceph-mon[36270]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:24.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:24 smithi107 ceph-mon[36270]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T00:43:24.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:24 smithi107 ceph-mon[36270]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T00:43:24.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:24 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:24.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:24 smithi107 ceph-mon[36270]: from='osd.2 [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:43:24.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:24 smithi107 ceph-mon[36270]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:43:24.931 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:24 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2[41435]: 2024-03-18T00:43:24.823+0000 7f5ef6c0e700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T00:43:24.931 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:24 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2[41435]: 2024-03-18T00:43:24.828+0000 7f5ef2a01700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:43:25.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:24 smithi106 ceph-mon[35899]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:25.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:24 smithi106 ceph-mon[35899]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T00:43:25.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:24 smithi106 ceph-mon[35899]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T00:43:25.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:24 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:25.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:24 smithi106 ceph-mon[35899]: from='osd.2 [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:43:25.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:24 smithi106 ceph-mon[35899]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:43:25.561 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T00:43:25.561 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T00:43:25.561 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-03-18T00:43:25.561 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-03-18T00:43:25.561 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00843315 s, 1.2 GB/s 2024-03-18T00:43:25.561 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-03-18T00:43:25.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:25 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:43:25.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:25 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:43:25.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:25 smithi049 ceph-mon[32130]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:43:25.919 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:25 smithi049 ceph-mon[32130]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T00:43:25.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:25.920 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:26.075 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:25 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:43:26.075 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:25 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:43:26.076 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:25 smithi107 ceph-mon[36270]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:43:26.076 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:25 smithi107 ceph-mon[36270]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T00:43:26.076 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:26.076 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:26.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:25 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:43:26.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:25 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:43:26.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:25 smithi106 ceph-mon[35899]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:43:26.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:25 smithi106 ceph-mon[35899]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T00:43:26.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:26.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:26.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi049:vg_nvme/lv_3 2024-03-18T00:43:26.468 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:43:26.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: osd.2 [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] boot 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi107 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi107 to 17080M 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:26.858 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:26.859 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:27.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:27.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: osd.2 [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] boot 2024-03-18T00:43:27.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T00:43:27.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:27.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi107 2024-03-18T00:43:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi107 to 17080M 2024-03-18T00:43:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:27.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:27.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-03-18T00:43:27.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: osd.2 [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] boot 2024-03-18T00:43:27.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T00:43:27.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T00:43:27.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi107 2024-03-18T00:43:27.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:27.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:27.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:27.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi107 to 17080M 2024-03-18T00:43:27.197 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:27.197 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:27.197 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:27.197 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:28.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.095+0000 7fe879cf1700 1 -- 172.21.15.49:0/1082323496 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874102320 msgr2=0x7fe874102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:28.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.095+0000 7fe879cf1700 1 --2- 172.21.15.49:0/1082323496 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874102320 0x7fe874102740 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe8640099c0 tx=0x7fe864009cd0 comp rx=0 tx=0).stop 2024-03-18T00:43:28.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.096+0000 7fe879cf1700 1 -- 172.21.15.49:0/1082323496 shutdown_connections 2024-03-18T00:43:28.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.096+0000 7fe879cf1700 1 --2- 172.21.15.49:0/1082323496 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe874103ec0 0x7fe87410a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:28.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.096+0000 7fe879cf1700 1 --2- 172.21.15.49:0/1082323496 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874102320 0x7fe874102740 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:28.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.096+0000 7fe879cf1700 1 --2- 172.21.15.49:0/1082323496 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe874103520 0x7fe874103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:28.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.096+0000 7fe879cf1700 1 -- 172.21.15.49:0/1082323496 >> 172.21.15.49:0/1082323496 conn(0x7fe8740fd8e0 msgr2=0x7fe8740ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:43:28.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.096+0000 7fe879cf1700 1 -- 172.21.15.49:0/1082323496 shutdown_connections 2024-03-18T00:43:28.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.096+0000 7fe879cf1700 1 -- 172.21.15.49:0/1082323496 wait complete. 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.096+0000 7fe879cf1700 1 Processor -- start 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.096+0000 7fe879cf1700 1 -- start start 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe879cf1700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe874102320 0x7fe874198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe879cf1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe874103520 0x7fe874198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe879cf1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874103ec0 0x7fe87419fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe879cf1700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe87410def0 con 0x7fe874103ec0 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe879cf1700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe87410dd70 con 0x7fe874102320 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe879cf1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe87410e070 con 0x7fe874103520 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe873fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874103ec0 0x7fe87419fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe872ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe874103520 0x7fe874198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe873fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874103ec0 0x7fe87419fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41220/0 (socket says 172.21.15.49:41220) 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe873fff700 1 -- 172.21.15.49:0/769403746 learned_addr learned my addr 172.21.15.49:0/769403746 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe8737fe700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe874102320 0x7fe874198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:28.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe873fff700 1 -- 172.21.15.49:0/769403746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe874103520 msgr2=0x7fe874198a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe873fff700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe874103520 0x7fe874198a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe873fff700 1 -- 172.21.15.49:0/769403746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe874102320 msgr2=0x7fe874198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe873fff700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe874102320 0x7fe874198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe873fff700 1 -- 172.21.15.49:0/769403746 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8741a0160 con 0x7fe874103ec0 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.097+0000 7fe873fff700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874103ec0 0x7fe87419fac0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe86c007f10 tx=0x7fe86c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.098+0000 7fe870ff9700 1 -- 172.21.15.49:0/769403746 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c00dd20 con 0x7fe874103ec0 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.098+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8741a0450 con 0x7fe874103ec0 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.098+0000 7fe870ff9700 1 -- 172.21.15.49:0/769403746 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe86c004020 con 0x7fe874103ec0 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.098+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8741a0990 con 0x7fe874103ec0 2024-03-18T00:43:28.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.098+0000 7fe870ff9700 1 -- 172.21.15.49:0/769403746 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c015600 con 0x7fe874103ec0 2024-03-18T00:43:28.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.099+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe874068f90 con 0x7fe874103ec0 2024-03-18T00:43:28.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.100+0000 7fe870ff9700 1 -- 172.21.15.49:0/769403746 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7fe86c0041d0 con 0x7fe874103ec0 2024-03-18T00:43:28.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.101+0000 7fe870ff9700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe85c0a2800 0x7fe85c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:28.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.101+0000 7fe8737fe700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe85c0a2800 0x7fe85c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:28.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.101+0000 7fe870ff9700 1 -- 172.21.15.49:0/769403746 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2851+0+0 (secure 0 0 0) 0x7fe86c0cd000 con 0x7fe874103ec0 2024-03-18T00:43:28.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.102+0000 7fe8737fe700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe85c0a2800 0x7fe85c0a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe86400b580 tx=0x7fe864005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:43:28.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.105+0000 7fe870ff9700 1 -- 172.21.15.49:0/769403746 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe86c010070 con 0x7fe874103ec0 2024-03-18T00:43:28.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:28.321+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fe874062710 con 0x7fe85c0a2800 2024-03-18T00:43:28.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:28 smithi049 ceph-mon[32130]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T00:43:28.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:43:28.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:28 smithi107 ceph-mon[36270]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T00:43:28.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:43:28.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:28 smithi106 ceph-mon[35899]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T00:43:28.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:43:29.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:29 smithi049 ceph-mon[32130]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:29.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T00:43:29.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:29 smithi049 ceph-mon[32130]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T00:43:29.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:43:29.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:43:29.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:43:29.243 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:29.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:29 smithi106 ceph-mon[35899]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:29.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T00:43:29.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:29 smithi106 ceph-mon[35899]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T00:43:29.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:43:29.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:43:29.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:43:29.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:29 smithi107 ceph-mon[36270]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T00:43:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:29 smithi107 ceph-mon[36270]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T00:43:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T00:43:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:43:29.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:43:29.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:30.057 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:43:29 smithi049 sudo[51204]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T00:43:30.057 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:43:29 smithi049 sudo[51204]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:30.057 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:43:29 smithi049 sudo[51204]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:30.057 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:43:29 smithi049 sudo[51204]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:30.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:30 smithi049 ceph-mon[32130]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:43:30.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:30 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T00:43:30.317 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:30 smithi049 ceph-mon[32130]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T00:43:30.317 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:43:30 smithi049 sudo[51228]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T00:43:30.317 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:43:30 smithi049 sudo[51228]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:30.317 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:43:30 smithi049 sudo[51228]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:30.317 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 00:43:30 smithi049 sudo[51228]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:30.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:30 smithi106 ceph-mon[35899]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:43:30.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:30 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T00:43:30.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:30 smithi106 ceph-mon[35899]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T00:43:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:30 smithi107 ceph-mon[36270]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:43:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:30 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T00:43:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:30 smithi107 ceph-mon[36270]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T00:43:30.703 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:30 smithi106 sudo[44652]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T00:43:30.703 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:30 smithi106 sudo[44652]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:30.703 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:30 smithi106 sudo[44652]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:30.703 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:30 smithi106 sudo[44652]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:31.189 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:30 smithi106 sudo[44655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T00:43:31.189 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:30 smithi106 sudo[44655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:31.189 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:30 smithi106 sudo[44655]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:31.189 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 00:43:30 smithi106 sudo[44655]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:31.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:31.213+0000 7fe870ff9700 1 -- 172.21.15.49:0/769403746 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe86c0fb740 con 0x7fe874103ec0 2024-03-18T00:43:31.230 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:30 smithi107 sudo[45099]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T00:43:31.230 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:30 smithi107 sudo[45099]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:31.230 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:30 smithi107 sudo[45099]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:31.230 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:30 smithi107 sudo[45099]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:31.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:31 smithi107 ceph-mon[36270]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:31.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:31 smithi107 ceph-mon[36270]: osdmap e22: 3 total, 3 up, 3 in 2024-03-18T00:43:31.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:31.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:31.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:31 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2485435501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0adbb9e1-881b-4d18-8933-6ac789b3d595"}]: dispatch 2024-03-18T00:43:31.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:31 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2485435501' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0adbb9e1-881b-4d18-8933-6ac789b3d595"}]': finished 2024-03-18T00:43:31.231 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:31 smithi107 ceph-mon[36270]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T00:43:31.231 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:31.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:31 smithi106 ceph-mon[35899]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:31.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:31 smithi106 ceph-mon[35899]: osdmap e22: 3 total, 3 up, 3 in 2024-03-18T00:43:31.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:31.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:31.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:31 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2485435501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0adbb9e1-881b-4d18-8933-6ac789b3d595"}]: dispatch 2024-03-18T00:43:31.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:31 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2485435501' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0adbb9e1-881b-4d18-8933-6ac789b3d595"}]': finished 2024-03-18T00:43:31.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:31 smithi106 ceph-mon[35899]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T00:43:31.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 ceph-mon[32130]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 ceph-mon[32130]: osdmap e22: 3 total, 3 up, 3 in 2024-03-18T00:43:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:31.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2485435501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0adbb9e1-881b-4d18-8933-6ac789b3d595"}]: dispatch 2024-03-18T00:43:31.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2485435501' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0adbb9e1-881b-4d18-8933-6ac789b3d595"}]': finished 2024-03-18T00:43:31.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 ceph-mon[32130]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T00:43:31.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:31.680 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:31 smithi107 sudo[45102]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T00:43:31.680 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:31 smithi107 sudo[45102]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:31.680 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:31 smithi107 sudo[45102]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:31.680 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 00:43:31 smithi107 sudo[45102]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:32.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 sudo[51313]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T00:43:32.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 sudo[51313]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:32.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 sudo[51313]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:32.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:31 smithi049 sudo[51313]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:32.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 sudo[51316]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T00:43:32.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 sudo[51316]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:32.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 sudo[51316]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:32.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 sudo[51316]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:32.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 sudo[45105]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T00:43:32.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 sudo[45105]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:32.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 sudo[45105]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:32.656 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 sudo[45105]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:32.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 sudo[45108]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T00:43:32.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 sudo[45108]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:32.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 sudo[45108]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:32.657 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 sudo[45108]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:32.916 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 sudo[44658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 sudo[44658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 sudo[44658]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:32.917 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:32 smithi106 sudo[44658]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T00:43:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T00:43:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T00:43:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:43:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:43:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:43:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:43:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:43:32.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:43:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T00:43:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T00:43:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T00:43:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:43:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:43:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:43:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:43:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:43:33.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:43:33.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 sudo[44661]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T00:43:33.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 sudo[44661]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T00:43:33.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 sudo[44661]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T00:43:33.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 sudo[44661]: pam_unix(sudo:session): session closed for user root 2024-03-18T00:43:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:33 smithi107 ceph-mon[36270]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:33 smithi107 ceph-mon[36270]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T00:43:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:33 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2660719245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:43:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:33 smithi107 ceph-mon[36270]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T00:43:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:33 smithi107 ceph-mon[36270]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T00:43:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:43:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:43:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:43:33.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:33 smithi107 ceph-mon[36270]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T00:43:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 ceph-mon[35899]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 ceph-mon[35899]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T00:43:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2660719245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:43:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 ceph-mon[35899]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T00:43:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 ceph-mon[35899]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T00:43:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:43:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:43:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:43:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:33 smithi106 ceph-mon[35899]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T00:43:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:33 smithi049 ceph-mon[32130]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:33 smithi049 ceph-mon[32130]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T00:43:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:33 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2660719245' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:43:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:33 smithi049 ceph-mon[32130]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T00:43:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:33 smithi049 ceph-mon[32130]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T00:43:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T00:43:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T00:43:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T00:43:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:33 smithi049 ceph-mon[32130]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T00:43:34.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:34 smithi107 ceph-mon[36270]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:34.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:34 smithi106 ceph-mon[35899]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:35.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:34 smithi049 ceph-mon[32130]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:36 smithi107 ceph-mon[36270]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:36.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:36 smithi106 ceph-mon[35899]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:36.976 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:36 smithi049 ceph-mon[32130]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:38.869 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:38 smithi049 ceph-mon[32130]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:38 smithi107 ceph-mon[36270]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:38.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:38 smithi106 ceph-mon[35899]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:39.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T00:43:39.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:39.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:39 smithi106 ceph-mon[35899]: Deploying daemon osd.3 on smithi049 2024-03-18T00:43:39.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T00:43:39.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:39.956 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:39 smithi049 ceph-mon[32130]: Deploying daemon osd.3 on smithi049 2024-03-18T00:43:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T00:43:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:39 smithi107 ceph-mon[36270]: Deploying daemon osd.3 on smithi049 2024-03-18T00:43:40.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:40 smithi106 ceph-mon[35899]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-18T00:43:40.992 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:40 smithi049 ceph-mon[32130]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-18T00:43:41.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:40 smithi107 ceph-mon[36270]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-18T00:43:42.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:42 smithi049 ceph-mon[32130]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-18T00:43:42.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:42 smithi106 ceph-mon[35899]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-18T00:43:43.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:42 smithi107 ceph-mon[36270]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-18T00:43:44.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:44 smithi106 ceph-mon[35899]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-18T00:43:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:44 smithi049 ceph-mon[32130]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-18T00:43:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:44 smithi107 ceph-mon[36270]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-18T00:43:46.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:46.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:46.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:46.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:46 smithi106 ceph-mon[35899]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T00:43:47.210 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:46 smithi049 ceph-mon[32130]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T00:43:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:46 smithi107 ceph-mon[36270]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T00:43:48.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.330+0000 7fe870ff9700 1 -- 172.21.15.49:0/769403746 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe874062710 con 0x7fe85c0a2800 2024-03-18T00:43:48.331 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 3 on host 'smithi049' 2024-03-18T00:43:48.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe85c0a2800 msgr2=0x7fe85c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:48.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe85c0a2800 0x7fe85c0a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe86400b580 tx=0x7fe864005fb0 comp rx=0 tx=0).stop 2024-03-18T00:43:48.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874103ec0 msgr2=0x7fe87419fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:48.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874103ec0 0x7fe87419fac0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe86c007f10 tx=0x7fe86c00d3b0 comp rx=0 tx=0).stop 2024-03-18T00:43:48.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 shutdown_connections 2024-03-18T00:43:48.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe874103ec0 0x7fe87419fac0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:48.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe874102320 0x7fe874198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:48.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe85c0a2800 0x7fe85c0a4cc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:48.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 --2- 172.21.15.49:0/769403746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe874103520 0x7fe874198a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:48.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 >> 172.21.15.49:0/769403746 conn(0x7fe8740fd8e0 msgr2=0x7fe8741068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:43:48.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.332+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 shutdown_connections 2024-03-18T00:43:48.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:43:48.333+0000 7fe879cf1700 1 -- 172.21.15.49:0/769403746 wait complete. 2024-03-18T00:43:48.907 DEBUG:teuthology.orchestra.run.smithi049:osd.3> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.3.service 2024-03-18T00:43:48.909 INFO:tasks.cephadm:Deploying osd.4 on smithi106 with /dev/vg_nvme/lv_3... 2024-03-18T00:43:48.909 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T00:43:49.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:48 smithi049 ceph-mon[32130]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T00:43:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:48 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:48 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:48 smithi049 ceph-mon[32130]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T00:43:49.154 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:43:49.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:48 smithi106 ceph-mon[35899]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T00:43:49.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:48 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:49.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:48 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:49.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:48 smithi106 ceph-mon[35899]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T00:43:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:48 smithi107 ceph-mon[36270]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T00:43:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:48 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:48 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:48 smithi107 ceph-mon[36270]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T00:43:50.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:50.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:50.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:50 smithi049 ceph-mon[32130]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T00:43:50.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:50 smithi049 ceph-mon[32130]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T00:43:50.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:50 smithi049 ceph-mon[32130]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:43:50.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:50.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:50.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:50.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:50 smithi107 ceph-mon[36270]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T00:43:50.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:50 smithi107 ceph-mon[36270]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T00:43:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:50 smithi107 ceph-mon[36270]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:43:50.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:50.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:50.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:50.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:50 smithi106 ceph-mon[35899]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T00:43:50.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:50 smithi106 ceph-mon[35899]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T00:43:50.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:50 smithi106 ceph-mon[35899]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:43:50.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:50.625 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 00:43:50 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3[53315]: 2024-03-18T00:43:50.336+0000 7fd544d8b700 -1 osd.3 0 waiting for initial osdmap 2024-03-18T00:43:50.625 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 00:43:50 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3[53315]: 2024-03-18T00:43:50.341+0000 7fd540b7e700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:43:51.193 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T00:43:51.194 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T00:43:51.194 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-18T00:43:51.194 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-18T00:43:51.194 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122902 s, 853 MB/s 2024-03-18T00:43:51.194 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-18T00:43:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:51 smithi049 ceph-mon[32130]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:51 smithi049 ceph-mon[32130]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:43:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:51 smithi049 ceph-mon[32130]: osdmap e25: 4 total, 3 up, 4 in 2024-03-18T00:43:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:51.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:51 smithi107 ceph-mon[36270]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:51.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:51 smithi107 ceph-mon[36270]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:43:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:51 smithi107 ceph-mon[36270]: osdmap e25: 4 total, 3 up, 4 in 2024-03-18T00:43:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:51.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:51.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:51 smithi106 ceph-mon[35899]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-18T00:43:51.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:51 smithi106 ceph-mon[35899]: from='osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:43:51.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:51 smithi106 ceph-mon[35899]: osdmap e25: 4 total, 3 up, 4 in 2024-03-18T00:43:51.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:51.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:52.012 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_3 2024-03-18T00:43:52.250 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:43:52.520 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:52 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:43:52.520 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:52 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:43:52.520 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:52 smithi049 ceph-mon[32130]: osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] boot 2024-03-18T00:43:52.520 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:52 smithi049 ceph-mon[32130]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T00:43:52.520 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:52.520 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:52 smithi049 ceph-mon[32130]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:52.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:52 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:43:52.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:52 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:43:52.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:52 smithi106 ceph-mon[35899]: osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] boot 2024-03-18T00:43:52.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:52 smithi106 ceph-mon[35899]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T00:43:52.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:52.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:52 smithi106 ceph-mon[35899]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:52.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:52 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:43:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:52 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:43:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:52 smithi107 ceph-mon[36270]: osd.3 [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] boot 2024-03-18T00:43:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:52 smithi107 ceph-mon[36270]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T00:43:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T00:43:52.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:52 smithi107 ceph-mon[36270]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:53.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T00:43:53.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi049 2024-03-18T00:43:53.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:53.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:53.507 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi049 to 8544M 2024-03-18T00:43:53.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:53.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:53.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:53.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:53.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:53.508 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T00:43:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi049 2024-03-18T00:43:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi049 to 8544M 2024-03-18T00:43:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:53.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:53.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:53.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:53.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:53.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T00:43:53.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi049 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi049 to 8544M 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:43:53.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:43:53.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 -- 172.21.15.106:0/2457811621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50102dc0 msgr2=0x7fcb501051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:53.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 --2- 172.21.15.106:0/2457811621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50102dc0 0x7fcb501051b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009990 tx=0x7fcb40009ca0 comp rx=0 tx=0).stop 2024-03-18T00:43:53.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 -- 172.21.15.106:0/2457811621 shutdown_connections 2024-03-18T00:43:53.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 --2- 172.21.15.106:0/2457811621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb50108020 0x7fcb5010a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:53.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 --2- 172.21.15.106:0/2457811621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50102dc0 0x7fcb501051b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:53.727 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 --2- 172.21.15.106:0/2457811621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb501056f0 0x7fcb50107ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 -- 172.21.15.106:0/2457811621 >> 172.21.15.106:0/2457811621 conn(0x7fcb500fc7f0 msgr2=0x7fcb500fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 -- 172.21.15.106:0/2457811621 shutdown_connections 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 -- 172.21.15.106:0/2457811621 wait complete. 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 Processor -- start 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.725+0000 7fcb56589700 1 -- start start 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb56589700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb50102dc0 0x7fcb5019a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb56589700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb501056f0 0x7fcb5019ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb56589700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50108020 0x7fcb501a1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb56589700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5010dfd0 con 0x7fcb50102dc0 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb56589700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5010de50 con 0x7fcb50108020 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb56589700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5010e150 con 0x7fcb501056f0 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb54b26700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50108020 0x7fcb501a1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:53.728 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb4ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb50102dc0 0x7fcb5019a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb54b26700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50108020 0x7fcb501a1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:33068/0 (socket says 172.21.15.106:33068) 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb54b26700 1 -- 172.21.15.106:0/2725567464 learned_addr learned my addr 172.21.15.106:0/2725567464 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb4f7fe700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb501056f0 0x7fcb5019ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb4ffff700 1 -- 172.21.15.106:0/2725567464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb501056f0 msgr2=0x7fcb5019ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb4ffff700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb501056f0 0x7fcb5019ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb4ffff700 1 -- 172.21.15.106:0/2725567464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50108020 msgr2=0x7fcb501a1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb4ffff700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50108020 0x7fcb501a1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb4ffff700 1 -- 172.21.15.106:0/2725567464 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb501a2430 con 0x7fcb50102dc0 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb54b26700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50108020 0x7fcb501a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb4ffff700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb50102dc0 0x7fcb5019a7d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000b7b0 tx=0x7fcb4000bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.726+0000 7fcb4d7fa700 1 -- 172.21.15.106:0/2725567464 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40026070 con 0x7fcb50102dc0 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.727+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb501a26c0 con 0x7fcb50102dc0 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.727+0000 7fcb4d7fa700 1 -- 172.21.15.106:0/2725567464 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb40003d70 con 0x7fcb50102dc0 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.727+0000 7fcb4d7fa700 1 -- 172.21.15.106:0/2725567464 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40021410 con 0x7fcb50102dc0 2024-03-18T00:43:53.729 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.727+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb501a2b70 con 0x7fcb50102dc0 2024-03-18T00:43:53.730 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.727+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb50068f90 con 0x7fcb50102dc0 2024-03-18T00:43:53.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.729+0000 7fcb4d7fa700 1 -- 172.21.15.106:0/2725567464 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcb40029070 con 0x7fcb50102dc0 2024-03-18T00:43:53.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.729+0000 7fcb4d7fa700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcb380a2890 0x7fcb380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:43:53.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.729+0000 7fcb4d7fa700 1 -- 172.21.15.106:0/2725567464 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3698+0+0 (secure 0 0 0) 0x7fcb400cec50 con 0x7fcb50102dc0 2024-03-18T00:43:53.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.729+0000 7fcb4f7fe700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcb380a2890 0x7fcb380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:43:53.731 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.730+0000 7fcb4f7fe700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcb380a2890 0x7fcb380a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009c60 tx=0x7fcb44009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:43:53.733 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.733+0000 7fcb4d7fa700 1 -- 172.21.15.106:0/2725567464 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcb40030020 con 0x7fcb50102dc0 2024-03-18T00:43:53.933 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:43:53.932+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fcb50062710 con 0x7fcb380a2890 2024-03-18T00:43:54.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:54 smithi049 ceph-mon[32130]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T00:43:54.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:54 smithi049 ceph-mon[32130]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:54.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:54 smithi049 ceph-mon[32130]: from='client.14325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:43:54.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:43:54.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:43:54.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:54.646 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:54 smithi106 ceph-mon[35899]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T00:43:54.646 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:54 smithi106 ceph-mon[35899]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:54.646 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:54 smithi106 ceph-mon[35899]: from='client.14325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:43:54.646 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:43:54.646 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:43:54.646 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:54 smithi107 ceph-mon[36270]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T00:43:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:54 smithi107 ceph-mon[36270]: pgmap v104: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:54 smithi107 ceph-mon[36270]: from='client.14325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:43:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:43:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:43:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:43:55.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:55 smithi049 ceph-mon[32130]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T00:43:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:55 smithi107 ceph-mon[36270]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T00:43:55.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:55 smithi106 ceph-mon[35899]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T00:43:56.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:56 smithi049 ceph-mon[32130]: osdmap e30: 4 total, 4 up, 4 in 2024-03-18T00:43:56.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:56 smithi049 ceph-mon[32130]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:56 smithi107 ceph-mon[36270]: osdmap e30: 4 total, 4 up, 4 in 2024-03-18T00:43:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:56 smithi107 ceph-mon[36270]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:56 smithi106 ceph-mon[35899]: osdmap e30: 4 total, 4 up, 4 in 2024-03-18T00:43:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:56 smithi106 ceph-mon[35899]: pgmap v107: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:57.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:57 smithi049 ceph-mon[32130]: from='client.? 172.21.15.106:0/4000813910' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95b4131-8533-4e43-b25c-c081adff6639"}]: dispatch 2024-03-18T00:43:57.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:57 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95b4131-8533-4e43-b25c-c081adff6639"}]: dispatch 2024-03-18T00:43:57.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:57 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d95b4131-8533-4e43-b25c-c081adff6639"}]': finished 2024-03-18T00:43:57.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:57 smithi049 ceph-mon[32130]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T00:43:57.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:57 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:43:57.648 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:57 smithi049 ceph-mon[32130]: from='client.? 172.21.15.106:0/1566462918' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:43:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:57 smithi107 ceph-mon[36270]: from='client.? 172.21.15.106:0/4000813910' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95b4131-8533-4e43-b25c-c081adff6639"}]: dispatch 2024-03-18T00:43:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:57 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95b4131-8533-4e43-b25c-c081adff6639"}]: dispatch 2024-03-18T00:43:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:57 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d95b4131-8533-4e43-b25c-c081adff6639"}]': finished 2024-03-18T00:43:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:57 smithi107 ceph-mon[36270]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T00:43:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:57 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:43:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:57 smithi107 ceph-mon[36270]: from='client.? 172.21.15.106:0/1566462918' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:43:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:57 smithi106 ceph-mon[35899]: from='client.? 172.21.15.106:0/4000813910' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95b4131-8533-4e43-b25c-c081adff6639"}]: dispatch 2024-03-18T00:43:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:57 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d95b4131-8533-4e43-b25c-c081adff6639"}]: dispatch 2024-03-18T00:43:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:57 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d95b4131-8533-4e43-b25c-c081adff6639"}]': finished 2024-03-18T00:43:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:57 smithi106 ceph-mon[35899]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T00:43:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:57 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:43:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:57 smithi106 ceph-mon[35899]: from='client.? 172.21.15.106:0/1566462918' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:43:58.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:43:58 smithi049 ceph-mon[32130]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:43:58 smithi107 ceph-mon[36270]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:43:58.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:43:58 smithi106 ceph-mon[35899]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:00.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:00 smithi106 ceph-mon[35899]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:44:00.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:00 smithi049 ceph-mon[32130]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:44:00.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:00 smithi107 ceph-mon[36270]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:44:02.865 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:02 smithi106 ceph-mon[35899]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-18T00:44:02.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:02 smithi049 ceph-mon[32130]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-18T00:44:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:02 smithi107 ceph-mon[36270]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-18T00:44:04.759 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:04 smithi106 ceph-mon[35899]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T00:44:04.759 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T00:44:04.759 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:04.759 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:04 smithi106 ceph-mon[35899]: Deploying daemon osd.4 on smithi106 2024-03-18T00:44:04.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:04 smithi049 ceph-mon[32130]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T00:44:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T00:44:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:04 smithi049 ceph-mon[32130]: Deploying daemon osd.4 on smithi106 2024-03-18T00:44:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:04 smithi107 ceph-mon[36270]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T00:44:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T00:44:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:04 smithi107 ceph-mon[36270]: Deploying daemon osd.4 on smithi106 2024-03-18T00:44:06.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:06 smithi049 ceph-mon[32130]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T00:44:06.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:06 smithi106 ceph-mon[35899]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T00:44:06.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:06 smithi107 ceph-mon[36270]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T00:44:08.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:08 smithi106 ceph-mon[35899]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-18T00:44:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:08 smithi049 ceph-mon[32130]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-18T00:44:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:08 smithi107 ceph-mon[36270]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-03-18T00:44:10.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:10 smithi049 ceph-mon[32130]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-18T00:44:10.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:10.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:10.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:10.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:10 smithi107 ceph-mon[36270]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-18T00:44:10.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:10.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:10.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:10.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:10 smithi106 ceph-mon[35899]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-18T00:44:10.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:10.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:10.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:12.732 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:12 smithi107 ceph-mon[36270]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:12.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:12 smithi049 ceph-mon[32130]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:12.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:12 smithi106 ceph-mon[35899]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:13.302 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.302+0000 7fcb4d7fa700 1 -- 172.21.15.106:0/2725567464 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcb50062710 con 0x7fcb380a2890 2024-03-18T00:44:13.303 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 4 on host 'smithi106' 2024-03-18T00:44:13.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcb380a2890 msgr2=0x7fcb380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:13.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcb380a2890 0x7fcb380a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009c60 tx=0x7fcb44009260 comp rx=0 tx=0).stop 2024-03-18T00:44:13.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb50102dc0 msgr2=0x7fcb5019a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:13.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb50102dc0 0x7fcb5019a7d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000b7b0 tx=0x7fcb4000bbb0 comp rx=0 tx=0).stop 2024-03-18T00:44:13.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 shutdown_connections 2024-03-18T00:44:13.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcb50108020 0x7fcb501a1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:13.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcb50102dc0 0x7fcb5019a7d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:13.305 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcb380a2890 0x7fcb380a4d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:13.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 --2- 172.21.15.106:0/2725567464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcb501056f0 0x7fcb5019ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:13.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 >> 172.21.15.106:0/2725567464 conn(0x7fcb500fc7f0 msgr2=0x7fcb50102450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:44:13.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 shutdown_connections 2024-03-18T00:44:13.306 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:44:13.304+0000 7fcb56589700 1 -- 172.21.15.106:0/2725567464 wait complete. 2024-03-18T00:44:14.015 DEBUG:teuthology.orchestra.run.smithi106:osd.4> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.4.service 2024-03-18T00:44:14.017 INFO:tasks.cephadm:Deploying osd.5 on smithi107 with /dev/vg_nvme/lv_3... 2024-03-18T00:44:14.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T00:44:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:44:14.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:14 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:14 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:14 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.382 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:14 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.383 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:14 smithi107 ceph-mon[36270]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:14.383 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:14 smithi107 ceph-mon[36270]: from='osd.4 [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T00:44:14.383 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:14 smithi107 ceph-mon[36270]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T00:44:14.551 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:14 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.551 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:14 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.551 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:14 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.552 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:14 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.552 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:14 smithi106 ceph-mon[35899]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:14.552 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:14 smithi106 ceph-mon[35899]: from='osd.4 [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T00:44:14.552 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:14 smithi106 ceph-mon[35899]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T00:44:14.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:14 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:14 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:14 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:14 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:14.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:14 smithi049 ceph-mon[32130]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:14.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:14 smithi049 ceph-mon[32130]: from='osd.4 [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T00:44:14.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:14 smithi049 ceph-mon[32130]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T00:44:15.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:15 smithi107 ceph-mon[36270]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T00:44:15.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:15 smithi107 ceph-mon[36270]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T00:44:15.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:15.588 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:15 smithi107 ceph-mon[36270]: from='osd.4 [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:44:15.588 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:15 smithi107 ceph-mon[36270]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:44:15.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:15 smithi106 ceph-mon[35899]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T00:44:15.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:15 smithi106 ceph-mon[35899]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T00:44:15.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:15.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:15 smithi106 ceph-mon[35899]: from='osd.4 [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:44:15.593 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:15 smithi106 ceph-mon[35899]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:44:15.593 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 00:44:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4[47489]: 2024-03-18T00:44:15.395+0000 7fd3c8381700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T00:44:15.594 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 00:44:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4[47489]: 2024-03-18T00:44:15.402+0000 7fd3c4174700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:44:15.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:15 smithi049 ceph-mon[32130]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T00:44:15.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:15 smithi049 ceph-mon[32130]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T00:44:15.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:15.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:15 smithi049 ceph-mon[32130]: from='osd.4 [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:44:15.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:15 smithi049 ceph-mon[32130]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:44:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T00:44:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T00:44:16.133 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-03-18T00:44:16.133 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-03-18T00:44:16.134 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011244 s, 933 MB/s 2024-03-18T00:44:16.134 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-03-18T00:44:16.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:16 smithi107 ceph-mon[36270]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:44:16.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:16 smithi107 ceph-mon[36270]: osdmap e33: 5 total, 4 up, 5 in 2024-03-18T00:44:16.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:16 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:16.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:16 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:16.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:16 smithi107 ceph-mon[36270]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:16.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:16 smithi106 ceph-mon[35899]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:44:16.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:16 smithi106 ceph-mon[35899]: osdmap e33: 5 total, 4 up, 5 in 2024-03-18T00:44:16.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:16 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:16.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:16 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:16.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:16 smithi106 ceph-mon[35899]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:16.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_3 2024-03-18T00:44:16.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:16 smithi049 ceph-mon[32130]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:44:16.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:16 smithi049 ceph-mon[32130]: osdmap e33: 5 total, 4 up, 5 in 2024-03-18T00:44:16.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:16 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:16.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:16 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:16.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:16 smithi049 ceph-mon[32130]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-18T00:44:16.944 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:44:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:44:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:44:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: osd.4 [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] boot 2024-03-18T00:44:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi106 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi106 to 8540M 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:17.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:17.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:17.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: osd.4 [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] boot 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi106 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi106 to 8540M 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:17.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:17.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:17.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:17.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:17.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:44:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:44:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: osd.4 [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] boot 2024-03-18T00:44:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T00:44:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T00:44:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi106 2024-03-18T00:44:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi106 to 8540M 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:18.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.493+0000 7f893c734700 1 -- 172.21.15.107:0/3831044535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934102de0 msgr2=0x7f89341051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:18.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.493+0000 7f893c734700 1 --2- 172.21.15.107:0/3831044535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934102de0 0x7f89341051d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8930009990 tx=0x7f8930009ca0 comp rx=0 tx=0).stop 2024-03-18T00:44:18.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.493+0000 7f893c734700 1 -- 172.21.15.107:0/3831044535 shutdown_connections 2024-03-18T00:44:18.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.493+0000 7f893c734700 1 --2- 172.21.15.107:0/3831044535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934102de0 0x7f89341051d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:18.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.493+0000 7f893c734700 1 --2- 172.21.15.107:0/3831044535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8934108040 0x7f893410a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:18.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.493+0000 7f893c734700 1 --2- 172.21.15.107:0/3831044535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8934105710 0x7f8934107b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:18.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.493+0000 7f893c734700 1 -- 172.21.15.107:0/3831044535 >> 172.21.15.107:0/3831044535 conn(0x7f89340fc790 msgr2=0x7f89340febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:44:18.496 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.493+0000 7f893c734700 1 -- 172.21.15.107:0/3831044535 shutdown_connections 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.493+0000 7f893c734700 1 -- 172.21.15.107:0/3831044535 wait complete. 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893c734700 1 Processor -- start 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893c734700 1 -- start start 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893c734700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8934102de0 0x7f8934102740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893c734700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8934105710 0x7f8934100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893c734700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934108040 0x7f89341012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893c734700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f893410df00 con 0x7f8934105710 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893c734700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f893410dd80 con 0x7f8934102de0 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893c734700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f893410e080 con 0x7f8934108040 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893acd1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934108040 0x7f89341012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893acd1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934108040 0x7f89341012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:60970/0 (socket says 172.21.15.107:60970) 2024-03-18T00:44:18.497 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893acd1700 1 -- 172.21.15.107:0/2842044027 learned_addr learned my addr 172.21.15.107:0/2842044027 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893a4d0700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8934102de0 0x7f8934102740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893acd1700 1 -- 172.21.15.107:0/2842044027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8934102de0 msgr2=0x7f8934102740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893acd1700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8934102de0 0x7f8934102740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893acd1700 1 -- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8934105710 msgr2=0x7f8934100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f8939ccf700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8934105710 0x7f8934100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893acd1700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8934105710 0x7f8934100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893acd1700 1 -- 172.21.15.107:0/2842044027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8934101b80 con 0x7f8934108040 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893a4d0700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8934102de0 0x7f8934102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f8939ccf700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8934105710 0x7f8934100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.494+0000 7f893acd1700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934108040 0x7f89341012d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f892c007d40 tx=0x7f892c00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:44:18.498 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.495+0000 7f892b7fe700 1 -- 172.21.15.107:0/2842044027 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c016020 con 0x7f8934108040 2024-03-18T00:44:18.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.495+0000 7f892b7fe700 1 -- 172.21.15.107:0/2842044027 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f892c00ad40 con 0x7f8934108040 2024-03-18T00:44:18.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.495+0000 7f892b7fe700 1 -- 172.21.15.107:0/2842044027 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c011740 con 0x7f8934108040 2024-03-18T00:44:18.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.495+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89341a25d0 con 0x7f8934108040 2024-03-18T00:44:18.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.495+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89341a2a90 con 0x7f8934108040 2024-03-18T00:44:18.499 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.496+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8934068f90 con 0x7f8934108040 2024-03-18T00:44:18.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.497+0000 7f892b7fe700 1 -- 172.21.15.107:0/2842044027 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f892c019020 con 0x7f8934108040 2024-03-18T00:44:18.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.497+0000 7f892b7fe700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89200a2890 0x7f89200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:44:18.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.498+0000 7f892b7fe700 1 -- 172.21.15.107:0/2842044027 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f892c0cde10 con 0x7f8934108040 2024-03-18T00:44:18.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.498+0000 7f893a4d0700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89200a2890 0x7f89200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:44:18.500 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.498+0000 7f893a4d0700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89200a2890 0x7f89200a4d50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f89300039c0 tx=0x7f8930005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:44:18.501 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.501+0000 7f892b7fe700 1 -- 172.21.15.107:0/2842044027 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f892c020020 con 0x7f8934108040 2024-03-18T00:44:18.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:18 smithi107 ceph-mon[36270]: osdmap e35: 5 total, 5 up, 5 in 2024-03-18T00:44:18.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:18 smithi107 ceph-mon[36270]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:18.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:18 smithi106 ceph-mon[35899]: osdmap e35: 5 total, 5 up, 5 in 2024-03-18T00:44:18.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:18 smithi106 ceph-mon[35899]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:18.706 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:18.706+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8934062710 con 0x7f89200a2890 2024-03-18T00:44:18.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:18 smithi049 ceph-mon[32130]: osdmap e35: 5 total, 5 up, 5 in 2024-03-18T00:44:18.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:18 smithi049 ceph-mon[32130]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:19.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:19 smithi107 ceph-mon[36270]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:44:19.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:19 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:44:19.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:19 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:44:19.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:19 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:19.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:19 smithi106 ceph-mon[35899]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:44:19.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:19 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:44:19.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:19 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:44:19.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:19 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:19.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:19 smithi049 ceph-mon[32130]: from='client.24238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:44:19.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:19 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:44:19.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:19 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:44:19.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:19 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:20.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:20 smithi107 ceph-mon[36270]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:20.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:20 smithi106 ceph-mon[35899]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:20.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:20 smithi049 ceph-mon[32130]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:22 smithi107 ceph-mon[36270]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:22 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/272463752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41055562-3e13-4785-9ba2-de16fcd415f1"}]: dispatch 2024-03-18T00:44:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:22 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41055562-3e13-4785-9ba2-de16fcd415f1"}]: dispatch 2024-03-18T00:44:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:22 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41055562-3e13-4785-9ba2-de16fcd415f1"}]': finished 2024-03-18T00:44:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:22 smithi107 ceph-mon[36270]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T00:44:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:22.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:22 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/2486493584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:44:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:22 smithi106 ceph-mon[35899]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:22 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/272463752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41055562-3e13-4785-9ba2-de16fcd415f1"}]: dispatch 2024-03-18T00:44:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:22 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41055562-3e13-4785-9ba2-de16fcd415f1"}]: dispatch 2024-03-18T00:44:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:22 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41055562-3e13-4785-9ba2-de16fcd415f1"}]': finished 2024-03-18T00:44:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:22 smithi106 ceph-mon[35899]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T00:44:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:22.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:22 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/2486493584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:44:22.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:22 smithi049 ceph-mon[32130]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:22.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:22 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/272463752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41055562-3e13-4785-9ba2-de16fcd415f1"}]: dispatch 2024-03-18T00:44:22.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:22 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41055562-3e13-4785-9ba2-de16fcd415f1"}]: dispatch 2024-03-18T00:44:22.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:22 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41055562-3e13-4785-9ba2-de16fcd415f1"}]': finished 2024-03-18T00:44:22.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:22 smithi049 ceph-mon[32130]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T00:44:22.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:22.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:22 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/2486493584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:44:24.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:24 smithi107 ceph-mon[36270]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:24.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:24 smithi106 ceph-mon[35899]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:24.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:24 smithi049 ceph-mon[32130]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:26.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:26 smithi106 ceph-mon[35899]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:26.677 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:26 smithi049 ceph-mon[32130]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:26 smithi107 ceph-mon[36270]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:28.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:28 smithi107 ceph-mon[36270]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:28.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:28 smithi106 ceph-mon[35899]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:28.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:28 smithi049 ceph-mon[32130]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T00:44:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:29 smithi107 ceph-mon[36270]: Deploying daemon osd.5 on smithi107 2024-03-18T00:44:29.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T00:44:29.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:29.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:29 smithi106 ceph-mon[35899]: Deploying daemon osd.5 on smithi107 2024-03-18T00:44:29.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T00:44:29.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:29.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:29 smithi049 ceph-mon[32130]: Deploying daemon osd.5 on smithi107 2024-03-18T00:44:30.623 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:30 smithi107 ceph-mon[36270]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:30.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:30 smithi106 ceph-mon[35899]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:30.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:30 smithi049 ceph-mon[32130]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:32.605 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:32 smithi107 ceph-mon[36270]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:32.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:32 smithi049 ceph-mon[32130]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:32.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:32 smithi106 ceph-mon[35899]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:35.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:35 smithi049 ceph-mon[32130]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:35.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:35.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:35.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:35 smithi107 ceph-mon[36270]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:35 smithi106 ceph-mon[35899]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:37 smithi049 ceph-mon[32130]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:37.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:37 smithi107 ceph-mon[36270]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:37.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:37.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:37.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:37.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:37.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:37.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:37 smithi106 ceph-mon[35899]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:37.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:37.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:37.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:37.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:37.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:37.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.775+0000 7f892b7fe700 1 -- 172.21.15.107:0/2842044027 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8934062710 con 0x7f89200a2890 2024-03-18T00:44:37.775 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 5 on host 'smithi107' 2024-03-18T00:44:37.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89200a2890 msgr2=0x7f89200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:37.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89200a2890 0x7f89200a4d50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f89300039c0 tx=0x7f8930005fb0 comp rx=0 tx=0).stop 2024-03-18T00:44:37.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934108040 msgr2=0x7f89341012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:37.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934108040 0x7f89341012d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f892c007d40 tx=0x7f892c00a340 comp rx=0 tx=0).stop 2024-03-18T00:44:37.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 shutdown_connections 2024-03-18T00:44:37.777 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89200a2890 0x7f89200a4d50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:37.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8934108040 0x7f89341012d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:37.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8934102de0 0x7f8934102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:37.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 --2- 172.21.15.107:0/2842044027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8934105710 0x7f8934100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:37.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 >> 172.21.15.107:0/2842044027 conn(0x7f89340fc790 msgr2=0x7f89341036e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:44:37.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 shutdown_connections 2024-03-18T00:44:37.778 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:44:37.778+0000 7f893c734700 1 -- 172.21.15.107:0/2842044027 wait complete. 2024-03-18T00:44:38.395 DEBUG:teuthology.orchestra.run.smithi107:osd.5> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.5.service 2024-03-18T00:44:38.398 INFO:tasks.cephadm:Deploying osd.6 on smithi049 with /dev/vg_nvme/lv_2... 2024-03-18T00:44:38.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T00:44:38.661 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:44:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:38 smithi107 ceph-mon[36270]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:38 smithi107 ceph-mon[36270]: from='osd.5 [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T00:44:38.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:38 smithi107 ceph-mon[36270]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T00:44:39.079 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:38 smithi049 ceph-mon[32130]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:39.079 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:39.079 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:39.079 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:39.079 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:38 smithi049 ceph-mon[32130]: from='osd.5 [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T00:44:39.079 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:38 smithi049 ceph-mon[32130]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T00:44:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:38 smithi106 ceph-mon[35899]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:44:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:38 smithi106 ceph-mon[35899]: from='osd.5 [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T00:44:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:38 smithi106 ceph-mon[35899]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T00:44:40.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:39 smithi049 ceph-mon[32130]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T00:44:40.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:39 smithi049 ceph-mon[32130]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T00:44:40.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:40.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:39 smithi049 ceph-mon[32130]: from='osd.5 [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:44:40.113 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:39 smithi049 ceph-mon[32130]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:44:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:39 smithi107 ceph-mon[36270]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T00:44:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:39 smithi107 ceph-mon[36270]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T00:44:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:39 smithi107 ceph-mon[36270]: from='osd.5 [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:44:40.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:39 smithi107 ceph-mon[36270]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:44:40.181 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 00:44:39 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T00:44:39.781+0000 7f2c08a8a700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T00:44:40.181 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 00:44:39 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T00:44:39.790+0000 7f2c0487d700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:44:40.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:39 smithi106 ceph-mon[35899]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T00:44:40.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:39 smithi106 ceph-mon[35899]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T00:44:40.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:40.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:39 smithi106 ceph-mon[35899]: from='osd.5 [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:44:40.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:39 smithi106 ceph-mon[35899]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:44:40.722 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T00:44:40.723 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T00:44:40.723 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-03-18T00:44:40.723 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-03-18T00:44:40.723 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0084613 s, 1.2 GB/s 2024-03-18T00:44:40.723 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-03-18T00:44:40.929 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:40 smithi107 ceph-mon[36270]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:40.929 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:40 smithi107 ceph-mon[36270]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:44:40.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:40 smithi107 ceph-mon[36270]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T00:44:40.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:40 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:40.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:40 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:40.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:40 smithi107 ceph-mon[36270]: osd.5 [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] boot 2024-03-18T00:44:40.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:40 smithi107 ceph-mon[36270]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T00:44:40.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:40 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:41.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:40 smithi049 ceph-mon[32130]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:41.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:40 smithi049 ceph-mon[32130]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:44:41.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:40 smithi049 ceph-mon[32130]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T00:44:41.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:41.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:41.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:40 smithi049 ceph-mon[32130]: osd.5 [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] boot 2024-03-18T00:44:41.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:40 smithi049 ceph-mon[32130]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T00:44:41.041 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:41.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:40 smithi106 ceph-mon[35899]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-18T00:44:41.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:40 smithi106 ceph-mon[35899]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:44:41.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:40 smithi106 ceph-mon[35899]: osdmap e38: 6 total, 5 up, 6 in 2024-03-18T00:44:41.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:40 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:41.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:40 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:41.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:40 smithi106 ceph-mon[35899]: osd.5 [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] boot 2024-03-18T00:44:41.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:40 smithi106 ceph-mon[35899]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T00:44:41.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:40 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T00:44:41.327 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi049:vg_nvme/lv_2 2024-03-18T00:44:41.587 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:44:41.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:44:41.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:44:41.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:41.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:41.953 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:41.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:41.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:41.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:41.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:41.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:41.954 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:41 smithi049 ceph-mon[32130]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T00:44:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:44:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:44:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:42.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:42.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:41 smithi107 ceph-mon[36270]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T00:44:42.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:44:42.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:44:42.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:42.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:42.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:42.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:44:42.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:42.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:42.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:44:42.197 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:44:42.197 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:41 smithi106 ceph-mon[35899]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T00:44:43.111 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:42 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi107 2024-03-18T00:44:43.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:42 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi107 to 8540M 2024-03-18T00:44:43.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:42 smithi049 ceph-mon[32130]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:44:43.112 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:42 smithi049 ceph-mon[32130]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T00:44:43.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.127+0000 7ff855419700 1 -- 172.21.15.49:0/1537365895 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850103ea0 msgr2=0x7ff85010a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:43.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.127+0000 7ff855419700 1 --2- 172.21.15.49:0/1537365895 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850103ea0 0x7ff85010a750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff84800b0d0 tx=0x7ff84800b3e0 comp rx=0 tx=0).stop 2024-03-18T00:44:43.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.127+0000 7ff855419700 1 -- 172.21.15.49:0/1537365895 shutdown_connections 2024-03-18T00:44:43.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.127+0000 7ff855419700 1 --2- 172.21.15.49:0/1537365895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff850102300 0x7ff850102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:43.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.127+0000 7ff855419700 1 --2- 172.21.15.49:0/1537365895 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850103ea0 0x7ff85010a750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.127+0000 7ff855419700 1 --2- 172.21.15.49:0/1537365895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff850103500 0x7ff850103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.127+0000 7ff855419700 1 -- 172.21.15.49:0/1537365895 >> 172.21.15.49:0/1537365895 conn(0x7ff8500fd8e0 msgr2=0x7ff8500ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.127+0000 7ff855419700 1 -- 172.21.15.49:0/1537365895 shutdown_connections 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.127+0000 7ff855419700 1 -- 172.21.15.49:0/1537365895 wait complete. 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff855419700 1 Processor -- start 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff855419700 1 -- start start 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff855419700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850102300 0x7ff8501984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff855419700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff850103500 0x7ff8501989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff855419700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff850103ea0 0x7ff85019fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff855419700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff85010def0 con 0x7ff850102300 2024-03-18T00:44:43.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff855419700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff85010dd70 con 0x7ff850103500 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff855419700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff85010e070 con 0x7ff850103ea0 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff84effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850102300 0x7ff8501984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff84effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850102300 0x7ff8501984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47254/0 (socket says 172.21.15.49:47254) 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff84effd700 1 -- 172.21.15.49:0/3961368840 learned_addr learned my addr 172.21.15.49:0/3961368840 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.128+0000 7ff84f7fe700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff850103ea0 0x7ff85019fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff84e7fc700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff850103500 0x7ff8501989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff84effd700 1 -- 172.21.15.49:0/3961368840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff850103ea0 msgr2=0x7ff85019fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff84effd700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff850103ea0 0x7ff85019fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff84effd700 1 -- 172.21.15.49:0/3961368840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff850103500 msgr2=0x7ff8501989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff84effd700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff850103500 0x7ff8501989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff84effd700 1 -- 172.21.15.49:0/3961368840 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8501a0100 con 0x7ff850102300 2024-03-18T00:44:43.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff84e7fc700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff850103500 0x7ff8501989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:44:43.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff84effd700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850102300 0x7ff8501984a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff84000d9b0 tx=0x7ff84000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:44:43.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff837fff700 1 -- 172.21.15.49:0/3961368840 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840017070 con 0x7ff850102300 2024-03-18T00:44:43.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff837fff700 1 -- 172.21.15.49:0/3961368840 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff840009e10 con 0x7ff850102300 2024-03-18T00:44:43.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8501a03f0 con 0x7ff850102300 2024-03-18T00:44:43.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff837fff700 1 -- 172.21.15.49:0/3961368840 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84000f620 con 0x7ff850102300 2024-03-18T00:44:43.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.129+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8501a08e0 con 0x7ff850102300 2024-03-18T00:44:43.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.130+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff850068f90 con 0x7ff850102300 2024-03-18T00:44:43.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.131+0000 7ff837fff700 1 -- 172.21.15.49:0/3961368840 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff840009930 con 0x7ff850102300 2024-03-18T00:44:43.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.132+0000 7ff837fff700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8380a2890 0x7ff8380a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:44:43.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.132+0000 7ff84e7fc700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8380a2890 0x7ff8380a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:44:43.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.132+0000 7ff837fff700 1 -- 172.21.15.49:0/3961368840 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4562+0+0 (secure 0 0 0) 0x7ff8400cd760 con 0x7ff850102300 2024-03-18T00:44:43.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.133+0000 7ff84e7fc700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8380a2890 0x7ff8380a4d50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff844005950 tx=0x7ff844005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:44:43.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.137+0000 7ff837fff700 1 -- 172.21.15.49:0/3961368840 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff84009b6a0 con 0x7ff850102300 2024-03-18T00:44:43.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:42 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi107 2024-03-18T00:44:43.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:42 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi107 to 8540M 2024-03-18T00:44:43.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:42 smithi107 ceph-mon[36270]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:44:43.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:42 smithi107 ceph-mon[36270]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T00:44:43.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:42 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi107 2024-03-18T00:44:43.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:42 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi107 to 8540M 2024-03-18T00:44:43.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:42 smithi106 ceph-mon[35899]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:44:43.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:42 smithi106 ceph-mon[35899]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T00:44:43.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:44:43.352+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ff850062710 con 0x7ff8380a2890 2024-03-18T00:44:44.068 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:44:44.068 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:44:44.068 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:44.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:44:44.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:44:44.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:44.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:44:44.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:44:44.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:44.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:44 smithi049 ceph-mon[32130]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:44:44.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:44 smithi049 ceph-mon[32130]: pgmap v142: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-18T00:44:44.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:44 smithi049 ceph-mon[32130]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-18T00:44:44.952 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:44 smithi049 ceph-mon[32130]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T00:44:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:44 smithi107 ceph-mon[36270]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:44:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:44 smithi107 ceph-mon[36270]: pgmap v142: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-18T00:44:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:44 smithi107 ceph-mon[36270]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-18T00:44:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:44 smithi107 ceph-mon[36270]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T00:44:45.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:44 smithi106 ceph-mon[35899]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:44:45.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:44 smithi106 ceph-mon[35899]: pgmap v142: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-18T00:44:45.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:44 smithi106 ceph-mon[35899]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-18T00:44:45.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:44 smithi106 ceph-mon[35899]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T00:44:46.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:45 smithi049 ceph-mon[32130]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T00:44:46.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:45 smithi107 ceph-mon[36270]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T00:44:46.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:45 smithi106 ceph-mon[35899]: osdmap e43: 6 total, 6 up, 6 in 2024-03-18T00:44:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:46 smithi049 ceph-mon[32130]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-18T00:44:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:46 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1411458457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8945dfe3-e4b2-49bc-81bd-ed8ef849eab6"}]: dispatch 2024-03-18T00:44:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:46 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1411458457' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8945dfe3-e4b2-49bc-81bd-ed8ef849eab6"}]': finished 2024-03-18T00:44:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:46 smithi049 ceph-mon[32130]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T00:44:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:44:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:46 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2924594496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:44:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:46 smithi107 ceph-mon[36270]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-18T00:44:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:46 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1411458457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8945dfe3-e4b2-49bc-81bd-ed8ef849eab6"}]: dispatch 2024-03-18T00:44:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:46 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1411458457' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8945dfe3-e4b2-49bc-81bd-ed8ef849eab6"}]': finished 2024-03-18T00:44:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:46 smithi107 ceph-mon[36270]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T00:44:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:44:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:46 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2924594496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:44:47.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:46 smithi106 ceph-mon[35899]: pgmap v145: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-18T00:44:47.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:46 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1411458457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8945dfe3-e4b2-49bc-81bd-ed8ef849eab6"}]: dispatch 2024-03-18T00:44:47.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:46 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1411458457' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8945dfe3-e4b2-49bc-81bd-ed8ef849eab6"}]': finished 2024-03-18T00:44:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:46 smithi106 ceph-mon[35899]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T00:44:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:44:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:46 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2924594496' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:44:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:48 smithi049 ceph-mon[32130]: pgmap v147: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-18T00:44:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:48 smithi107 ceph-mon[36270]: pgmap v147: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-18T00:44:49.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:48 smithi106 ceph-mon[35899]: pgmap v147: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-03-18T00:44:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:49 smithi049 ceph-mon[32130]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-18T00:44:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:49 smithi049 ceph-mon[32130]: Cluster is now healthy 2024-03-18T00:44:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:49 smithi107 ceph-mon[36270]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-18T00:44:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:49 smithi107 ceph-mon[36270]: Cluster is now healthy 2024-03-18T00:44:50.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:49 smithi106 ceph-mon[35899]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-18T00:44:50.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:49 smithi106 ceph-mon[35899]: Cluster is now healthy 2024-03-18T00:44:51.109 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:50 smithi049 ceph-mon[32130]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:44:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:50 smithi107 ceph-mon[36270]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:44:51.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:50 smithi106 ceph-mon[35899]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:44:53.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:52 smithi049 ceph-mon[32130]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-18T00:44:53.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:52 smithi107 ceph-mon[36270]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-18T00:44:53.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:52 smithi106 ceph-mon[35899]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-18T00:44:54.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T00:44:54.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:54.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T00:44:54.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:54.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T00:44:54.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:44:55.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:54 smithi049 ceph-mon[32130]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-18T00:44:55.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:54 smithi049 ceph-mon[32130]: Deploying daemon osd.6 on smithi049 2024-03-18T00:44:55.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:54 smithi107 ceph-mon[36270]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-18T00:44:55.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:54 smithi107 ceph-mon[36270]: Deploying daemon osd.6 on smithi049 2024-03-18T00:44:55.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:54 smithi106 ceph-mon[35899]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-18T00:44:55.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:54 smithi106 ceph-mon[35899]: Deploying daemon osd.6 on smithi049 2024-03-18T00:44:57.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:56 smithi049 ceph-mon[32130]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T00:44:57.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:56 smithi107 ceph-mon[36270]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T00:44:57.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:56 smithi106 ceph-mon[35899]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T00:44:59.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:44:58 smithi049 ceph-mon[32130]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-18T00:44:59.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:44:58 smithi107 ceph-mon[36270]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-18T00:44:59.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:44:58 smithi106 ceph-mon[35899]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-18T00:45:00.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:00 smithi049 ceph-mon[32130]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-18T00:45:00.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:00.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:00.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:00.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:00 smithi107 ceph-mon[36270]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-18T00:45:00.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:00.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:00.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:00.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:00 smithi106 ceph-mon[35899]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-18T00:45:00.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:00.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:00.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:02.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:02 smithi049 ceph-mon[32130]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:45:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:02 smithi107 ceph-mon[36270]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:45:02.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:02 smithi106 ceph-mon[35899]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:45:03.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.213+0000 7ff837fff700 1 -- 172.21.15.49:0/3961368840 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff850062710 con 0x7ff8380a2890 2024-03-18T00:45:03.214 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 6 on host 'smithi049' 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8380a2890 msgr2=0x7ff8380a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8380a2890 0x7ff8380a4d50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff844005950 tx=0x7ff844005f50 comp rx=0 tx=0).stop 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850102300 msgr2=0x7ff8501984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850102300 0x7ff8501984a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff84000d9b0 tx=0x7ff84000dcc0 comp rx=0 tx=0).stop 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 shutdown_connections 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8380a2890 0x7ff8380a4d50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff850103ea0 0x7ff85019fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff850102300 0x7ff8501984a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 --2- 172.21.15.49:0/3961368840 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff850103500 0x7ff8501989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 >> 172.21.15.49:0/3961368840 conn(0x7ff8500fd8e0 msgr2=0x7ff850106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 shutdown_connections 2024-03-18T00:45:03.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:45:03.216+0000 7ff855419700 1 -- 172.21.15.49:0/3961368840 wait complete. 2024-03-18T00:45:03.970 DEBUG:teuthology.orchestra.run.smithi049:osd.6> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.6.service 2024-03-18T00:45:03.973 INFO:tasks.cephadm:Deploying osd.7 on smithi106 with /dev/vg_nvme/lv_2... 2024-03-18T00:45:03.973 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T00:45:04.219 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:45:04.341 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.341 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.341 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:04 smithi106 ceph-mon[35899]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T00:45:04.342 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.342 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:04 smithi049 ceph-mon[32130]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T00:45:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:04 smithi107 ceph-mon[36270]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T00:45:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:05.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:05 smithi049 ceph-mon[32130]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:45:05.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:05 smithi049 ceph-mon[32130]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T00:45:05.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:05 smithi049 ceph-mon[32130]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T00:45:05.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:05 smithi049 ceph-mon[32130]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:45:05.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:05 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:05.375 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 00:45:05 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T00:45:05.190+0000 7fa0ca6a1700 -1 osd.6 0 waiting for initial osdmap 2024-03-18T00:45:05.375 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 00:45:05 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T00:45:05.202+0000 7fa0c6494700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:45:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:05 smithi106 ceph-mon[35899]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:45:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:05 smithi106 ceph-mon[35899]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T00:45:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:05 smithi106 ceph-mon[35899]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T00:45:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:05 smithi106 ceph-mon[35899]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:45:05.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:05 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:05 smithi107 ceph-mon[36270]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:45:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:05 smithi107 ceph-mon[36270]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T00:45:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:05 smithi107 ceph-mon[36270]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T00:45:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:05 smithi107 ceph-mon[36270]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:45:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:05 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:06.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:06 smithi049 ceph-mon[32130]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:45:06.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:06 smithi049 ceph-mon[32130]: osdmap e46: 7 total, 6 up, 7 in 2024-03-18T00:45:06.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:06 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:06.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:06 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:06.420 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:06 smithi106 ceph-mon[35899]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:45:06.420 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:06 smithi106 ceph-mon[35899]: osdmap e46: 7 total, 6 up, 7 in 2024-03-18T00:45:06.420 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:06 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:06.420 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:06 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:06.420 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T00:45:06.420 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T00:45:06.420 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-18T00:45:06.420 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-18T00:45:06.420 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0141164 s, 743 MB/s 2024-03-18T00:45:06.421 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 2024-03-18T00:45:06.421 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-18T00:45:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:06 smithi107 ceph-mon[36270]: from='osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:45:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:06 smithi107 ceph-mon[36270]: osdmap e46: 7 total, 6 up, 7 in 2024-03-18T00:45:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:06 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:06 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:07.139 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_2 2024-03-18T00:45:07.378 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:45:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:07 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:45:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:07 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:45:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:07 smithi106 ceph-mon[35899]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:45:07.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:07.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:07 smithi106 ceph-mon[35899]: osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] boot 2024-03-18T00:45:07.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:07 smithi106 ceph-mon[35899]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T00:45:07.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:07.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:07 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:45:07.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:07 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:45:07.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:07 smithi049 ceph-mon[32130]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:45:07.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:07.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:07 smithi049 ceph-mon[32130]: osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] boot 2024-03-18T00:45:07.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:07 smithi049 ceph-mon[32130]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T00:45:07.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:07.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:07 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:45:07.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:07 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:45:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:07 smithi107 ceph-mon[36270]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-18T00:45:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:07 smithi107 ceph-mon[36270]: osd.6 [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] boot 2024-03-18T00:45:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:07 smithi107 ceph-mon[36270]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T00:45:07.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T00:45:08.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: osdmap e48: 7 total, 7 up, 7 in 2024-03-18T00:45:08.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:08.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:08.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:08.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:08.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:08.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:08.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:08.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:08.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:08 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: osdmap e48: 7 total, 7 up, 7 in 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:08.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:08.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:08.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:08 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: osdmap e48: 7 total, 7 up, 7 in 2024-03-18T00:45:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:08.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:08 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:09.002 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.000+0000 7fc59b489700 1 -- 172.21.15.106:0/3382701399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102400 msgr2=0x7fc594102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.000+0000 7fc59b489700 1 --2- 172.21.15.106:0/3382701399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102400 0x7fc594102860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc5900099c0 tx=0x7fc590009cd0 comp rx=0 tx=0).stop 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.000+0000 7fc59b489700 1 -- 172.21.15.106:0/3382701399 shutdown_connections 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.000+0000 7fc59b489700 1 --2- 172.21.15.106:0/3382701399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc594102da0 0x7fc594109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.000+0000 7fc59b489700 1 --2- 172.21.15.106:0/3382701399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc594101200 0x7fc594101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.000+0000 7fc59b489700 1 --2- 172.21.15.106:0/3382701399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102400 0x7fc594102860 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.000+0000 7fc59b489700 1 -- 172.21.15.106:0/3382701399 >> 172.21.15.106:0/3382701399 conn(0x7fc5940fc7a0 msgr2=0x7fc5940febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.000+0000 7fc59b489700 1 -- 172.21.15.106:0/3382701399 shutdown_connections 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.000+0000 7fc59b489700 1 -- 172.21.15.106:0/3382701399 wait complete. 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc59b489700 1 Processor -- start 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc59b489700 1 -- start start 2024-03-18T00:45:09.003 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc59b489700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc594101200 0x7fc594105190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc59b489700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc594102400 0x7fc5941037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc59b489700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102da0 0x7fc594103d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc59b489700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc59410ce00 con 0x7fc594102400 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc59b489700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc59410cc80 con 0x7fc594102da0 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc59b489700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc59410cf80 con 0x7fc594101200 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc599a26700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102da0 0x7fc594103d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc599a26700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102da0 0x7fc594103d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:55526/0 (socket says 172.21.15.106:55526) 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc599a26700 1 -- 172.21.15.106:0/4181908119 learned_addr learned my addr 172.21.15.106:0/4181908119 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc598a24700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc594102400 0x7fc5941037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc599225700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc594101200 0x7fc594105190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc599a26700 1 -- 172.21.15.106:0/4181908119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc594101200 msgr2=0x7fc594105190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:09.004 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc599a26700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc594101200 0x7fc594105190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:09.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc599a26700 1 -- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc594102400 msgr2=0x7fc5941037e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:09.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc599a26700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc594102400 0x7fc5941037e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:09.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.001+0000 7fc599a26700 1 -- 172.21.15.106:0/4181908119 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5941045d0 con 0x7fc594102da0 2024-03-18T00:45:09.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.002+0000 7fc599a26700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102da0 0x7fc594103d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc58c007ee0 tx=0x7fc58c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:45:09.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.002+0000 7fc58a7fc700 1 -- 172.21.15.106:0/4181908119 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c004bb0 con 0x7fc594102da0 2024-03-18T00:45:09.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.002+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5941a26f0 con 0x7fc594102da0 2024-03-18T00:45:09.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.002+0000 7fc58a7fc700 1 -- 172.21.15.106:0/4181908119 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc58c004d50 con 0x7fc594102da0 2024-03-18T00:45:09.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.002+0000 7fc58a7fc700 1 -- 172.21.15.106:0/4181908119 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc58c005960 con 0x7fc594102da0 2024-03-18T00:45:09.005 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.002+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5941a2bb0 con 0x7fc594102da0 2024-03-18T00:45:09.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.003+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc594068f90 con 0x7fc594102da0 2024-03-18T00:45:09.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.004+0000 7fc58a7fc700 1 -- 172.21.15.106:0/4181908119 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc58c01a020 con 0x7fc594102da0 2024-03-18T00:45:09.006 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.005+0000 7fc58a7fc700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5800a2890 0x7fc5800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:45:09.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.005+0000 7fc58a7fc700 1 -- 172.21.15.106:0/4181908119 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 4965+0+0 (secure 0 0 0) 0x7fc58c0cdbc0 con 0x7fc594102da0 2024-03-18T00:45:09.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.005+0000 7fc599225700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5800a2890 0x7fc5800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:45:09.007 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.006+0000 7fc599225700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5800a2890 0x7fc5800a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc594102260 tx=0x7fc58400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:45:09.009 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.009+0000 7fc58a7fc700 1 -- 172.21.15.106:0/4181908119 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc58c010070 con 0x7fc594102da0 2024-03-18T00:45:09.212 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:09.212+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fc5940fe970 con 0x7fc5800a2890 2024-03-18T00:45:09.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:09 smithi049 ceph-mon[32130]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:09.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:09 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi049 2024-03-18T00:45:09.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:09 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi049 to 5696M 2024-03-18T00:45:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:09 smithi107 ceph-mon[36270]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:09 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi049 2024-03-18T00:45:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:09 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi049 to 5696M 2024-03-18T00:45:09.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:09 smithi106 ceph-mon[35899]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:09.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:09 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi049 2024-03-18T00:45:09.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:09 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi049 to 5696M 2024-03-18T00:45:10.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:10 smithi106 ceph-mon[35899]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:45:10.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:45:10.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:45:10.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:10.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:10 smithi049 ceph-mon[32130]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:45:10.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:45:10.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:45:10.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:10 smithi107 ceph-mon[36270]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:45:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:45:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:45:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:11 smithi106 ceph-mon[35899]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:11.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:11 smithi049 ceph-mon[32130]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:11 smithi107 ceph-mon[36270]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:12.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:12 smithi107 ceph-mon[36270]: from='client.? 172.21.15.106:0/1456609311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc773d75-ff12-45ce-ac11-27de406961c7"}]: dispatch 2024-03-18T00:45:12.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:12 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc773d75-ff12-45ce-ac11-27de406961c7"}]: dispatch 2024-03-18T00:45:12.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:12 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc773d75-ff12-45ce-ac11-27de406961c7"}]': finished 2024-03-18T00:45:12.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:12 smithi107 ceph-mon[36270]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T00:45:12.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:12 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:12.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:12 smithi049 ceph-mon[32130]: from='client.? 172.21.15.106:0/1456609311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc773d75-ff12-45ce-ac11-27de406961c7"}]: dispatch 2024-03-18T00:45:12.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:12 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc773d75-ff12-45ce-ac11-27de406961c7"}]: dispatch 2024-03-18T00:45:12.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:12 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc773d75-ff12-45ce-ac11-27de406961c7"}]': finished 2024-03-18T00:45:12.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:12 smithi049 ceph-mon[32130]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T00:45:12.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:12 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:12.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:12 smithi106 ceph-mon[35899]: from='client.? 172.21.15.106:0/1456609311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc773d75-ff12-45ce-ac11-27de406961c7"}]: dispatch 2024-03-18T00:45:12.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:12 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc773d75-ff12-45ce-ac11-27de406961c7"}]: dispatch 2024-03-18T00:45:12.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:12 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc773d75-ff12-45ce-ac11-27de406961c7"}]': finished 2024-03-18T00:45:12.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:12 smithi106 ceph-mon[35899]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T00:45:12.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:12 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:13.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:13 smithi049 ceph-mon[32130]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:13.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:13 smithi049 ceph-mon[32130]: from='client.? 172.21.15.106:0/4156058237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:45:13.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:13 smithi107 ceph-mon[36270]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:13.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:13 smithi107 ceph-mon[36270]: from='client.? 172.21.15.106:0/4156058237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:45:13.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:13 smithi106 ceph-mon[35899]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:13.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:13 smithi106 ceph-mon[35899]: from='client.? 172.21.15.106:0/4156058237' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:45:15.497 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:15 smithi106 ceph-mon[35899]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:15.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:15 smithi049 ceph-mon[32130]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:15.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:15 smithi107 ceph-mon[36270]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:17.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:17 smithi106 ceph-mon[35899]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:17.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:17 smithi049 ceph-mon[32130]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:17 smithi107 ceph-mon[36270]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:19.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:19 smithi049 ceph-mon[32130]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:19 smithi107 ceph-mon[36270]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:19.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:19 smithi106 ceph-mon[35899]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:20.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T00:45:20.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:20.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T00:45:20.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:20.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T00:45:20.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:21.533 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:21 smithi106 ceph-mon[35899]: Deploying daemon osd.7 on smithi106 2024-03-18T00:45:21.534 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:21 smithi106 ceph-mon[35899]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:21.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:21 smithi049 ceph-mon[32130]: Deploying daemon osd.7 on smithi106 2024-03-18T00:45:21.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:21 smithi049 ceph-mon[32130]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:21 smithi107 ceph-mon[36270]: Deploying daemon osd.7 on smithi106 2024-03-18T00:45:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:21 smithi107 ceph-mon[36270]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:23.440 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:23 smithi106 ceph-mon[35899]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:23 smithi049 ceph-mon[32130]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:23 smithi107 ceph-mon[36270]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:25.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:25 smithi106 ceph-mon[35899]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:25 smithi049 ceph-mon[32130]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:25 smithi107 ceph-mon[36270]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:26.546 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:26 smithi049 ceph-mon[32130]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:26 smithi107 ceph-mon[36270]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:26.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:26.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:26.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:26.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:26 smithi106 ceph-mon[35899]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:28.619 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:28 smithi106 ceph-mon[35899]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:28.679 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.679+0000 7fc58a7fc700 1 -- 172.21.15.106:0/4181908119 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc5940fe970 con 0x7fc5800a2890 2024-03-18T00:45:28.680 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 7 on host 'smithi106' 2024-03-18T00:45:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5800a2890 msgr2=0x7fc5800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5800a2890 0x7fc5800a4d50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc594102260 tx=0x7fc58400b410 comp rx=0 tx=0).stop 2024-03-18T00:45:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102da0 msgr2=0x7fc594103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102da0 0x7fc594103d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc58c007ee0 tx=0x7fc58c00d3b0 comp rx=0 tx=0).stop 2024-03-18T00:45:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 shutdown_connections 2024-03-18T00:45:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5800a2890 0x7fc5800a4d50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc594101200 0x7fc594105190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:28.682 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc594102da0 0x7fc594103d20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:28.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 --2- 172.21.15.106:0/4181908119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc594102400 0x7fc5941037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:28.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 >> 172.21.15.106:0/4181908119 conn(0x7fc5940fc7a0 msgr2=0x7fc5940fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:45:28.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 shutdown_connections 2024-03-18T00:45:28.683 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:45:28.681+0000 7fc59b489700 1 -- 172.21.15.106:0/4181908119 wait complete. 2024-03-18T00:45:28.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:28 smithi049 ceph-mon[32130]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:28 smithi107 ceph-mon[36270]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:29.652 DEBUG:teuthology.orchestra.run.smithi106:osd.7> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.7.service 2024-03-18T00:45:29.654 INFO:tasks.cephadm:Deploying osd.8 on smithi107 with /dev/vg_nvme/lv_2... 2024-03-18T00:45:29.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T00:45:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:45:29.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:29.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:29.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:29 smithi107 ceph-mon[36270]: from='osd.7 [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T00:45:29.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:29 smithi107 ceph-mon[36270]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T00:45:29.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:29.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:29.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:29 smithi106 ceph-mon[35899]: from='osd.7 [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T00:45:29.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:29 smithi106 ceph-mon[35899]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T00:45:30.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:30.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:30.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:29 smithi049 ceph-mon[32130]: from='osd.7 [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T00:45:30.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:29 smithi049 ceph-mon[32130]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T00:45:30.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:30 smithi106 ceph-mon[35899]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:30.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:30 smithi106 ceph-mon[35899]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T00:45:30.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:30 smithi106 ceph-mon[35899]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T00:45:30.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:30 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:30.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:30 smithi106 ceph-mon[35899]: from='osd.7 [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:45:30.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:30 smithi106 ceph-mon[35899]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:45:30.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:30 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:30.679 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:30 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:30 smithi107 ceph-mon[36270]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:30 smithi107 ceph-mon[36270]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T00:45:30.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:30 smithi107 ceph-mon[36270]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T00:45:30.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:30 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:30.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:30 smithi107 ceph-mon[36270]: from='osd.7 [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:45:30.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:30 smithi107 ceph-mon[36270]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:45:30.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:30 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:30.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:30 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:30 smithi049 ceph-mon[32130]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:30 smithi049 ceph-mon[32130]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T00:45:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:30 smithi049 ceph-mon[32130]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T00:45:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:30 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:30 smithi049 ceph-mon[32130]: from='osd.7 [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:45:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:30 smithi049 ceph-mon[32130]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:45:31.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:30 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:31.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:30 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:31.417 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 00:45:31 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T00:45:31.077+0000 7f3b6f070700 -1 osd.7 0 waiting for initial osdmap 2024-03-18T00:45:31.417 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 00:45:31 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T00:45:31.086+0000 7f3b66e5b700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:45:31.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:31 smithi107 ceph-mon[36270]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:45:31.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:31 smithi107 ceph-mon[36270]: osdmap e51: 8 total, 7 up, 8 in 2024-03-18T00:45:31.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:31.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:31.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:31 smithi107 ceph-mon[36270]: from='osd.7 ' entity='osd.7' 2024-03-18T00:45:31.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:31 smithi106 ceph-mon[35899]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:45:31.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:31 smithi106 ceph-mon[35899]: osdmap e51: 8 total, 7 up, 8 in 2024-03-18T00:45:31.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:31.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:31.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:31 smithi106 ceph-mon[35899]: from='osd.7 ' entity='osd.7' 2024-03-18T00:45:32.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:31 smithi049 ceph-mon[32130]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:45:32.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:31 smithi049 ceph-mon[32130]: osdmap e51: 8 total, 7 up, 8 in 2024-03-18T00:45:32.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:32.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:32.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:31 smithi049 ceph-mon[32130]: from='osd.7 ' entity='osd.7' 2024-03-18T00:45:32.196 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T00:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T00:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-03-18T00:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-03-18T00:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113913 s, 921 MB/s 2024-03-18T00:45:32.197 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-03-18T00:45:32.740 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:32 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:45:32.740 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:32 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:45:32.740 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:32 smithi107 ceph-mon[36270]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:32.740 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:32.740 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:32 smithi107 ceph-mon[36270]: osd.7 [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] boot 2024-03-18T00:45:32.740 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:32 smithi107 ceph-mon[36270]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T00:45:32.740 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:32.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_2 2024-03-18T00:45:32.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:32 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:45:32.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:32 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:45:32.882 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:32 smithi106 ceph-mon[35899]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:32.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:32.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:32 smithi106 ceph-mon[35899]: osd.7 [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] boot 2024-03-18T00:45:32.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:32 smithi106 ceph-mon[35899]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T00:45:32.883 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:32.977 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:45:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:32 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:45:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:32 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:45:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:32 smithi049 ceph-mon[32130]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-03-18T00:45:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:32 smithi049 ceph-mon[32130]: osd.7 [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] boot 2024-03-18T00:45:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:32 smithi049 ceph-mon[32130]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T00:45:33.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T00:45:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T00:45:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:33.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:33.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:33.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:33.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:33.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:33.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:33.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:33.947 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:33.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:33.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:33.948 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.086 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T00:45:34.086 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.086 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.086 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:34.087 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:34.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:34.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:34.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:34.580 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.578+0000 7f373fcd2700 1 -- 172.21.15.107:0/3972102007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 msgr2=0x7f3738102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.578+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3972102007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 0x7f3738102740 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3728009990 tx=0x7f3728009ca0 comp rx=0 tx=0).stop 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.578+0000 7f373fcd2700 1 -- 172.21.15.107:0/3972102007 shutdown_connections 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.578+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3972102007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 0x7f3738102740 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.578+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3972102007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3738103ec0 0x7f373810a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.578+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3972102007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3738103520 0x7f3738103980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.578+0000 7f373fcd2700 1 -- 172.21.15.107:0/3972102007 >> 172.21.15.107:0/3972102007 conn(0x7f37380fd8e0 msgr2=0x7f37380ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.578+0000 7f373fcd2700 1 -- 172.21.15.107:0/3972102007 shutdown_connections 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.578+0000 7f373fcd2700 1 -- 172.21.15.107:0/3972102007 wait complete. 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373fcd2700 1 Processor -- start 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373fcd2700 1 -- start start 2024-03-18T00:45:34.581 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373fcd2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 0x7f3738076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373fcd2700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3738103520 0x7f37380746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373fcd2700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3738103ec0 0x7f3738074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373fcd2700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f373810dfe0 con 0x7f3738103520 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373fcd2700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f373810de60 con 0x7f3738103ec0 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373fcd2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f373810e160 con 0x7f3738102320 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373da6e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 0x7f3738076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373da6e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 0x7f3738076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:39674/0 (socket says 172.21.15.107:39674) 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373da6e700 1 -- 172.21.15.107:0/3176374386 learned_addr learned my addr 172.21.15.107:0/3176374386 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373d26d700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3738103520 0x7f37380746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373e26f700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3738103ec0 0x7f3738074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373da6e700 1 -- 172.21.15.107:0/3176374386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3738103ec0 msgr2=0x7f3738074c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373da6e700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3738103ec0 0x7f3738074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373da6e700 1 -- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3738103520 msgr2=0x7f37380746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373da6e700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3738103520 0x7f37380746c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373da6e700 1 -- 172.21.15.107:0/3176374386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37380754b0 con 0x7f3738102320 2024-03-18T00:45:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.579+0000 7f373d26d700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3738103520 0x7f37380746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:45:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.580+0000 7f373da6e700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 0x7f3738076070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3728000c00 tx=0x7f3728009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:45:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.580+0000 7f372effd700 1 -- 172.21.15.107:0/3176374386 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3728027070 con 0x7f3738102320 2024-03-18T00:45:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.580+0000 7f372effd700 1 -- 172.21.15.107:0/3176374386 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3728022470 con 0x7f3738102320 2024-03-18T00:45:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.580+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37381a27b0 con 0x7f3738102320 2024-03-18T00:45:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.580+0000 7f372effd700 1 -- 172.21.15.107:0/3176374386 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372800f6c0 con 0x7f3738102320 2024-03-18T00:45:34.583 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.580+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37381a2c80 con 0x7f3738102320 2024-03-18T00:45:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.581+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3738068f90 con 0x7f3738102320 2024-03-18T00:45:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.583+0000 7f372effd700 1 -- 172.21.15.107:0/3176374386 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f372802a070 con 0x7f3738102320 2024-03-18T00:45:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.583+0000 7f372effd700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37240a2890 0x7f37240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:45:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.583+0000 7f372effd700 1 -- 172.21.15.107:0/3176374386 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5426+0+0 (secure 0 0 0) 0x7f37280cf390 con 0x7f3738102320 2024-03-18T00:45:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.583+0000 7f373d26d700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37240a2890 0x7f37240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:45:34.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.584+0000 7f373d26d700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37240a2890 0x7f37240a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f37340099d0 tx=0x7f3734008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:45:34.587 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.587+0000 7f372effd700 1 -- 172.21.15.107:0/3176374386 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3728013070 con 0x7f3738102320 2024-03-18T00:45:34.793 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:34.792+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3738062710 con 0x7f37240a2890 2024-03-18T00:45:34.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:34 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi106 2024-03-18T00:45:34.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:34 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi106 to 5693M 2024-03-18T00:45:34.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:34 smithi106 ceph-mon[35899]: pgmap v179: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:34.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:34 smithi106 ceph-mon[35899]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T00:45:35.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:34 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi106 2024-03-18T00:45:35.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:34 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi106 to 5693M 2024-03-18T00:45:35.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:34 smithi049 ceph-mon[32130]: pgmap v179: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:35.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:34 smithi049 ceph-mon[32130]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T00:45:35.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:34 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi106 2024-03-18T00:45:35.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:34 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi106 to 5693M 2024-03-18T00:45:35.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:34 smithi107 ceph-mon[36270]: pgmap v179: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:35.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:34 smithi107 ceph-mon[36270]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T00:45:36.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:35 smithi049 ceph-mon[32130]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T00:45:36.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:35 smithi049 ceph-mon[32130]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:45:36.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:45:36.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:45:36.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:35 smithi107 ceph-mon[36270]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T00:45:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:35 smithi107 ceph-mon[36270]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:45:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:45:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:45:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:36.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:35 smithi106 ceph-mon[35899]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T00:45:36.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:35 smithi106 ceph-mon[35899]: from='client.24301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:45:36.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:45:36.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:45:36.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:36 smithi107 ceph-mon[36270]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:36 smithi107 ceph-mon[36270]: osdmap e56: 8 total, 8 up, 8 in 2024-03-18T00:45:37.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:36 smithi049 ceph-mon[32130]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:37.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:36 smithi049 ceph-mon[32130]: osdmap e56: 8 total, 8 up, 8 in 2024-03-18T00:45:37.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:36 smithi106 ceph-mon[35899]: pgmap v182: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:37.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:36 smithi106 ceph-mon[35899]: osdmap e56: 8 total, 8 up, 8 in 2024-03-18T00:45:37.973 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:37 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/616286212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e583e9c-fd87-4cdf-9b71-ced791a6419e"}]: dispatch 2024-03-18T00:45:37.973 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:37 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e583e9c-fd87-4cdf-9b71-ced791a6419e"}]: dispatch 2024-03-18T00:45:37.973 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:37 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e583e9c-fd87-4cdf-9b71-ced791a6419e"}]': finished 2024-03-18T00:45:37.973 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:37 smithi107 ceph-mon[36270]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T00:45:37.974 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:38.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:37 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/616286212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e583e9c-fd87-4cdf-9b71-ced791a6419e"}]: dispatch 2024-03-18T00:45:38.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:37 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e583e9c-fd87-4cdf-9b71-ced791a6419e"}]: dispatch 2024-03-18T00:45:38.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:37 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e583e9c-fd87-4cdf-9b71-ced791a6419e"}]': finished 2024-03-18T00:45:38.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:37 smithi049 ceph-mon[32130]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T00:45:38.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:38.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:37 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/616286212' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e583e9c-fd87-4cdf-9b71-ced791a6419e"}]: dispatch 2024-03-18T00:45:38.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:37 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e583e9c-fd87-4cdf-9b71-ced791a6419e"}]: dispatch 2024-03-18T00:45:38.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:37 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e583e9c-fd87-4cdf-9b71-ced791a6419e"}]': finished 2024-03-18T00:45:38.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:37 smithi106 ceph-mon[35899]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T00:45:38.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:39.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:38 smithi049 ceph-mon[32130]: pgmap v184: 1 pgs: 1 unknown; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:39.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:38 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/762777664' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:45:39.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:38 smithi107 ceph-mon[36270]: pgmap v184: 1 pgs: 1 unknown; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:39.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:38 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/762777664' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:45:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:38 smithi106 ceph-mon[35899]: pgmap v184: 1 pgs: 1 unknown; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:39.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:38 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/762777664' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:45:41.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:40 smithi049 ceph-mon[32130]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:41.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:40 smithi107 ceph-mon[36270]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:41.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:40 smithi106 ceph-mon[35899]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:43.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:42 smithi049 ceph-mon[32130]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:43.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:42 smithi107 ceph-mon[36270]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:43.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:42 smithi106 ceph-mon[35899]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:44.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:44 smithi107 ceph-mon[36270]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:44 smithi049 ceph-mon[32130]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:45.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:44 smithi106 ceph-mon[35899]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:45.760 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T00:45:45.760 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:45.760 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:45 smithi107 ceph-mon[36270]: Deploying daemon osd.8 on smithi107 2024-03-18T00:45:46.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T00:45:46.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:46.097 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:45 smithi106 ceph-mon[35899]: Deploying daemon osd.8 on smithi107 2024-03-18T00:45:46.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T00:45:46.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:46.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:45 smithi049 ceph-mon[32130]: Deploying daemon osd.8 on smithi107 2024-03-18T00:45:46.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:46 smithi107 ceph-mon[36270]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:46 smithi049 ceph-mon[32130]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:47.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:46 smithi106 ceph-mon[35899]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:48.926 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:48 smithi107 ceph-mon[36270]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:48 smithi049 ceph-mon[32130]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:49.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:48 smithi106 ceph-mon[35899]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:51.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:50 smithi049 ceph-mon[32130]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:51.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:51.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:51.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:50 smithi107 ceph-mon[36270]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:51.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:50 smithi106 ceph-mon[35899]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:51.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:51.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:51.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:52 smithi107 ceph-mon[36270]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:53.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:52 smithi049 ceph-mon[32130]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:53.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:52 smithi106 ceph-mon[35899]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:54.362 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.362+0000 7f372effd700 1 -- 172.21.15.107:0/3176374386 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3738062710 con 0x7f37240a2890 2024-03-18T00:45:54.363 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 8 on host 'smithi107' 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37240a2890 msgr2=0x7f37240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37240a2890 0x7f37240a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f37340099d0 tx=0x7f3734008040 comp rx=0 tx=0).stop 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 msgr2=0x7f3738076070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 0x7f3738076070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3728000c00 tx=0x7f3728009f90 comp rx=0 tx=0).stop 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 shutdown_connections 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37240a2890 0x7f37240a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3738102320 0x7f3738076070 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3738103ec0 0x7f3738074c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 --2- 172.21.15.107:0/3176374386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3738103520 0x7f37380746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.364+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 >> 172.21.15.107:0/3176374386 conn(0x7f37380fd8e0 msgr2=0x7f3738106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.365+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 shutdown_connections 2024-03-18T00:45:54.365 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:45:54.365+0000 7f373fcd2700 1 -- 172.21.15.107:0/3176374386 wait complete. 2024-03-18T00:45:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='osd.8 [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T00:45:54.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:54 smithi107 ceph-mon[36270]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='osd.8 [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T00:45:54.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:54 smithi049 ceph-mon[32130]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='osd.8 [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T00:45:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:54 smithi106 ceph-mon[35899]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T00:45:55.062 DEBUG:teuthology.orchestra.run.smithi107:osd.8> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.8.service 2024-03-18T00:45:55.064 INFO:tasks.cephadm:Deploying osd.9 on smithi049 with /dev/vg_nvme/lv_1... 2024-03-18T00:45:55.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T00:45:55.326 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:45:56.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:56 smithi049 ceph-mon[32130]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T00:45:56.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:56 smithi049 ceph-mon[32130]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T00:45:56.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:56 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:56.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:56 smithi049 ceph-mon[32130]: from='osd.8 [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:45:56.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:56 smithi049 ceph-mon[32130]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:45:56.467 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:56 smithi049 ceph-mon[32130]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:56.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:56 smithi107 ceph-mon[36270]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T00:45:56.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:56 smithi107 ceph-mon[36270]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T00:45:56.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:56 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:56.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:56 smithi107 ceph-mon[36270]: from='osd.8 [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:45:56.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:56 smithi107 ceph-mon[36270]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:45:56.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:56 smithi107 ceph-mon[36270]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:56 smithi106 ceph-mon[35899]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T00:45:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:56 smithi106 ceph-mon[35899]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T00:45:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:56 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:56 smithi106 ceph-mon[35899]: from='osd.8 [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:45:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:56 smithi106 ceph-mon[35899]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:45:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:56 smithi106 ceph-mon[35899]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:56.930 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 00:45:56 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T00:45:56.765+0000 7fcc17bfa700 -1 osd.8 0 waiting for initial osdmap 2024-03-18T00:45:56.930 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 00:45:56 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T00:45:56.777+0000 7fcc0f9e5700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:45:57.543 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T00:45:57.543 INFO:teuthology.orchestra.run.smithi049.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T00:45:57.543 INFO:teuthology.orchestra.run.smithi049.stderr: stderr: 10+0 records in 2024-03-18T00:45:57.543 INFO:teuthology.orchestra.run.smithi049.stderr:10+0 records out 2024-03-18T00:45:57.543 INFO:teuthology.orchestra.run.smithi049.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00830631 s, 1.3 GB/s 2024-03-18T00:45:57.543 INFO:teuthology.orchestra.run.smithi049.stderr:--> Zapping successful for: 2024-03-18T00:45:57.544 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:57 smithi049 ceph-mon[32130]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:45:57.544 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:57 smithi049 ceph-mon[32130]: osdmap e59: 9 total, 8 up, 9 in 2024-03-18T00:45:57.544 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:57 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:57.544 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:57 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:57.544 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:57 smithi049 ceph-mon[32130]: from='osd.8 ' entity='osd.8' 2024-03-18T00:45:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:57 smithi107 ceph-mon[36270]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:45:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:57 smithi107 ceph-mon[36270]: osdmap e59: 9 total, 8 up, 9 in 2024-03-18T00:45:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:57 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:57 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:57 smithi107 ceph-mon[36270]: from='osd.8 ' entity='osd.8' 2024-03-18T00:45:57.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:57 smithi106 ceph-mon[35899]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:45:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:57 smithi106 ceph-mon[35899]: osdmap e59: 9 total, 8 up, 9 in 2024-03-18T00:45:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:57 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:57 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:57.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:57 smithi106 ceph-mon[35899]: from='osd.8 ' entity='osd.8' 2024-03-18T00:45:58.206 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi049:vg_nvme/lv_1 2024-03-18T00:45:58.472 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:45:58.594 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:45:58.594 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:45:58.594 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:58.594 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: osd.8 [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] boot 2024-03-18T00:45:58.594 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi107 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi107 to 5693M 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:58.595 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:58.596 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: osd.8 [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] boot 2024-03-18T00:45:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi107 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi107 to 5693M 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:58.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:58.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: osd.8 [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] boot 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi107 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:58.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:58.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:45:58.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi107 to 5693M 2024-03-18T00:45:58.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:45:58.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:45:58.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:45:58.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:00.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.154+0000 7fc98fa3f700 1 -- 172.21.15.49:0/3957045861 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988102300 msgr2=0x7fc988102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:00.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.154+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/3957045861 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988102300 0x7fc988102720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0099c0 tx=0x7fc97c009cd0 comp rx=0 tx=0).stop 2024-03-18T00:46:00.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.154+0000 7fc98fa3f700 1 -- 172.21.15.49:0/3957045861 shutdown_connections 2024-03-18T00:46:00.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.154+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/3957045861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc988103ea0 0x7fc98810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:00.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.154+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/3957045861 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988102300 0x7fc988102720 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:00.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.154+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/3957045861 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc988103500 0x7fc988103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:00.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.154+0000 7fc98fa3f700 1 -- 172.21.15.49:0/3957045861 >> 172.21.15.49:0/3957045861 conn(0x7fc9880fd8e0 msgr2=0x7fc9880ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:46:00.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.154+0000 7fc98fa3f700 1 -- 172.21.15.49:0/3957045861 shutdown_connections 2024-03-18T00:46:00.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.154+0000 7fc98fa3f700 1 -- 172.21.15.49:0/3957045861 wait complete. 2024-03-18T00:46:00.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.155+0000 7fc98fa3f700 1 Processor -- start 2024-03-18T00:46:00.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.155+0000 7fc98fa3f700 1 -- start start 2024-03-18T00:46:00.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.155+0000 7fc98fa3f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc988102300 0x7fc9881984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:00.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.155+0000 7fc98fa3f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc988103500 0x7fc988198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:00.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.155+0000 7fc98fa3f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988103ea0 0x7fc98819fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:00.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.155+0000 7fc98fa3f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc98810dee0 con 0x7fc988103ea0 2024-03-18T00:46:00.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.155+0000 7fc98fa3f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc98810dd60 con 0x7fc988102300 2024-03-18T00:46:00.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.155+0000 7fc98fa3f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc98810e060 con 0x7fc988103500 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.155+0000 7fc98dfdc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988103ea0 0x7fc98819fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98cfda700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc988103500 0x7fc988198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98dfdc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988103ea0 0x7fc98819fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60864/0 (socket says 172.21.15.49:60864) 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98dfdc700 1 -- 172.21.15.49:0/2545572237 learned_addr learned my addr 172.21.15.49:0/2545572237 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98cfda700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc988103500 0x7fc988198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:50044/0 (socket says 172.21.15.49:50044) 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98d7db700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc988102300 0x7fc9881984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98dfdc700 1 -- 172.21.15.49:0/2545572237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc988103500 msgr2=0x7fc988198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98dfdc700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc988103500 0x7fc988198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98dfdc700 1 -- 172.21.15.49:0/2545572237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc988102300 msgr2=0x7fc9881984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98dfdc700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc988102300 0x7fc9881984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:00.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98dfdc700 1 -- 172.21.15.49:0/2545572237 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9881a0120 con 0x7fc988103ea0 2024-03-18T00:46:00.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98dfdc700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988103ea0 0x7fc98819fa80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc98000ef60 tx=0x7fc98000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:46:00.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc97a7fc700 1 -- 172.21.15.49:0/2545572237 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9800092e0 con 0x7fc988103ea0 2024-03-18T00:46:00.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9881a03b0 con 0x7fc988103ea0 2024-03-18T00:46:00.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc97a7fc700 1 -- 172.21.15.49:0/2545572237 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc980004bb0 con 0x7fc988103ea0 2024-03-18T00:46:00.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc97a7fc700 1 -- 172.21.15.49:0/2545572237 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980004280 con 0x7fc988103ea0 2024-03-18T00:46:00.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.156+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9881a0940 con 0x7fc988103ea0 2024-03-18T00:46:00.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.157+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc988068f90 con 0x7fc988103ea0 2024-03-18T00:46:00.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.158+0000 7fc97a7fc700 1 -- 172.21.15.49:0/2545572237 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc98001b030 con 0x7fc988103ea0 2024-03-18T00:46:00.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.159+0000 7fc97a7fc700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9740a2890 0x7fc9740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:00.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.159+0000 7fc98d7db700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9740a2890 0x7fc9740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:00.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.159+0000 7fc97a7fc700 1 -- 172.21.15.49:0/2545572237 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7fc9800cdaa0 con 0x7fc988103ea0 2024-03-18T00:46:00.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.160+0000 7fc98d7db700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9740a2890 0x7fc9740a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00b580 tx=0x7fc97c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:46:00.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.164+0000 7fc97a7fc700 1 -- 172.21.15.49:0/2545572237 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc980022020 con 0x7fc988103ea0 2024-03-18T00:46:00.164 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:45:59 smithi049 ceph-mon[32130]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T00:46:00.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:00.380+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fc988062710 con 0x7fc9740a2890 2024-03-18T00:46:00.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:45:59 smithi107 ceph-mon[36270]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T00:46:00.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:45:59 smithi106 ceph-mon[35899]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T00:46:01.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:00 smithi049 ceph-mon[32130]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:01.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:00 smithi049 ceph-mon[32130]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T00:46:01.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:46:01.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:46:01.226 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:00 smithi107 ceph-mon[36270]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:00 smithi107 ceph-mon[36270]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T00:46:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:46:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:46:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:01.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:00 smithi106 ceph-mon[35899]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:01.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:00 smithi106 ceph-mon[35899]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T00:46:01.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:46:01.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:46:01.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:02.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:01 smithi049 ceph-mon[32130]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:46:02.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:01 smithi049 ceph-mon[32130]: osdmap e64: 9 total, 9 up, 9 in 2024-03-18T00:46:02.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:01 smithi107 ceph-mon[36270]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:46:02.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:01 smithi107 ceph-mon[36270]: osdmap e64: 9 total, 9 up, 9 in 2024-03-18T00:46:02.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:01 smithi106 ceph-mon[35899]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi049:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:46:02.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:01 smithi106 ceph-mon[35899]: osdmap e64: 9 total, 9 up, 9 in 2024-03-18T00:46:03.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:02 smithi049 ceph-mon[32130]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:03.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:02 smithi107 ceph-mon[36270]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:03.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:02 smithi106 ceph-mon[35899]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:03 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3717978711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a435579c-62d2-409c-83ce-6c44d620a592"}]: dispatch 2024-03-18T00:46:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:03 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3717978711' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a435579c-62d2-409c-83ce-6c44d620a592"}]': finished 2024-03-18T00:46:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:03 smithi049 ceph-mon[32130]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T00:46:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:04.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:03 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/128645255' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:46:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:03 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3717978711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a435579c-62d2-409c-83ce-6c44d620a592"}]: dispatch 2024-03-18T00:46:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:03 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3717978711' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a435579c-62d2-409c-83ce-6c44d620a592"}]': finished 2024-03-18T00:46:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:03 smithi107 ceph-mon[36270]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T00:46:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:04.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:03 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/128645255' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:46:04.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:03 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3717978711' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a435579c-62d2-409c-83ce-6c44d620a592"}]: dispatch 2024-03-18T00:46:04.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:03 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3717978711' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a435579c-62d2-409c-83ce-6c44d620a592"}]': finished 2024-03-18T00:46:04.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:03 smithi106 ceph-mon[35899]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T00:46:04.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:04.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:03 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/128645255' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:46:05.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:05 smithi107 ceph-mon[36270]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-18T00:46:05.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:05 smithi106 ceph-mon[35899]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-18T00:46:05.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:05 smithi049 ceph-mon[32130]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-18T00:46:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:07 smithi107 ceph-mon[36270]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:46:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:07 smithi106 ceph-mon[35899]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:46:07.599 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:07 smithi049 ceph-mon[32130]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:46:09.287 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:09 smithi049 ceph-mon[32130]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T00:46:09.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:09 smithi107 ceph-mon[36270]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T00:46:09.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:09 smithi106 ceph-mon[35899]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T00:46:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:11 smithi107 ceph-mon[36270]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T00:46:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:11 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T00:46:11.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:11 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:11 smithi106 ceph-mon[35899]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T00:46:11.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:11 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T00:46:11.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:11 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:11.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:11 smithi049 ceph-mon[32130]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T00:46:11.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:11 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T00:46:11.567 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:11 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:12.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:12 smithi107 ceph-mon[36270]: Deploying daemon osd.9 on smithi049 2024-03-18T00:46:12.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:12 smithi106 ceph-mon[35899]: Deploying daemon osd.9 on smithi049 2024-03-18T00:46:12.588 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:12 smithi049 ceph-mon[32130]: Deploying daemon osd.9 on smithi049 2024-03-18T00:46:13.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:13 smithi049 ceph-mon[32130]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T00:46:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:13 smithi107 ceph-mon[36270]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T00:46:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:13 smithi106 ceph-mon[35899]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T00:46:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:15 smithi049 ceph-mon[32130]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:15 smithi107 ceph-mon[36270]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:15 smithi106 ceph-mon[35899]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:17 smithi107 ceph-mon[36270]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:17.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:17 smithi106 ceph-mon[35899]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:17.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:17.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:17.524 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:17 smithi049 ceph-mon[32130]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:17.524 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:17.524 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:17.524 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:19.264 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:19 smithi049 ceph-mon[32130]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:19 smithi107 ceph-mon[36270]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:19 smithi106 ceph-mon[35899]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:20.892 INFO:teuthology.orchestra.run.smithi049.stdout:Created osd(s) 9 on host 'smithi049' 2024-03-18T00:46:20.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.892+0000 7fc97a7fc700 1 -- 172.21.15.49:0/2545572237 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc988062710 con 0x7fc9740a2890 2024-03-18T00:46:20.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9740a2890 msgr2=0x7fc9740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:20.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9740a2890 0x7fc9740a4d50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc97c00b580 tx=0x7fc97c005fb0 comp rx=0 tx=0).stop 2024-03-18T00:46:20.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988103ea0 msgr2=0x7fc98819fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:20.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988103ea0 0x7fc98819fa80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc98000ef60 tx=0x7fc98000ef90 comp rx=0 tx=0).stop 2024-03-18T00:46:20.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 shutdown_connections 2024-03-18T00:46:20.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc988103ea0 0x7fc98819fa80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:20.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc988102300 0x7fc9881984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:20.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9740a2890 0x7fc9740a4d50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:20.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 --2- 172.21.15.49:0/2545572237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc988103500 0x7fc988198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:20.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 >> 172.21.15.49:0/2545572237 conn(0x7fc9880fd8e0 msgr2=0x7fc988106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:46:20.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 shutdown_connections 2024-03-18T00:46:20.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:46:20.894+0000 7fc98fa3f700 1 -- 172.21.15.49:0/2545572237 wait complete. 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:20 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:21.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:21.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:21.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:21.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T00:46:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:20 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:21.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:21.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:21.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:21.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T00:46:21.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:21.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:20 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:21.597 DEBUG:teuthology.orchestra.run.smithi049:osd.9> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.9.service 2024-03-18T00:46:21.599 INFO:tasks.cephadm:Deploying osd.10 on smithi106 with /dev/vg_nvme/lv_1... 2024-03-18T00:46:21.600 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T00:46:21.845 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:46:22.219 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:22 smithi106 ceph-mon[35899]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T00:46:22.219 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:22 smithi106 ceph-mon[35899]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T00:46:22.219 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:22 smithi106 ceph-mon[35899]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:46:22.219 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:22 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:22.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:22 smithi049 ceph-mon[32130]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T00:46:22.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:22 smithi049 ceph-mon[32130]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T00:46:22.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:22 smithi049 ceph-mon[32130]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:46:22.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:22 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:22.375 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 00:46:22 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T00:46:22.182+0000 7f9af0aec700 -1 osd.9 0 waiting for initial osdmap 2024-03-18T00:46:22.375 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 00:46:22 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T00:46:22.196+0000 7f9aec8df700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:46:22.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:22 smithi107 ceph-mon[36270]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T00:46:22.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:22 smithi107 ceph-mon[36270]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T00:46:22.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:22 smithi107 ceph-mon[36270]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]: dispatch 2024-03-18T00:46:22.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:22 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:23.321 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:23 smithi106 ceph-mon[35899]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:23.321 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:23 smithi106 ceph-mon[35899]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:46:23.321 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:23 smithi106 ceph-mon[35899]: osdmap e67: 10 total, 9 up, 10 in 2024-03-18T00:46:23.321 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:23.321 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:23.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:23 smithi049 ceph-mon[32130]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:23.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:23 smithi049 ceph-mon[32130]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:46:23.434 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:23 smithi049 ceph-mon[32130]: osdmap e67: 10 total, 9 up, 10 in 2024-03-18T00:46:23.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:23.435 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:23 smithi107 ceph-mon[36270]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:23 smithi107 ceph-mon[36270]: from='osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi049", "root=default"]}]': finished 2024-03-18T00:46:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:23 smithi107 ceph-mon[36270]: osdmap e67: 10 total, 9 up, 10 in 2024-03-18T00:46:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:23.884 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T00:46:23.884 INFO:teuthology.orchestra.run.smithi106.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T00:46:23.884 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 10+0 records in 2024-03-18T00:46:23.884 INFO:teuthology.orchestra.run.smithi106.stderr:10+0 records out 2024-03-18T00:46:23.884 INFO:teuthology.orchestra.run.smithi106.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0148391 s, 707 MB/s 2024-03-18T00:46:23.884 INFO:teuthology.orchestra.run.smithi106.stderr: stderr: 2024-03-18T00:46:23.885 INFO:teuthology.orchestra.run.smithi106.stderr:--> Zapping successful for: 2024-03-18T00:46:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:24 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:46:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:24 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:46:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:24 smithi049 ceph-mon[32130]: osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] boot 2024-03-18T00:46:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:24 smithi049 ceph-mon[32130]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T00:46:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:24 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:24.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:24 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:46:24.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:24 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:46:24.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:24 smithi107 ceph-mon[36270]: osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] boot 2024-03-18T00:46:24.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:24 smithi107 ceph-mon[36270]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T00:46:24.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:24 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:24.632 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:24 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:46:24.632 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:24 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:46:24.632 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:24 smithi106 ceph-mon[35899]: osd.9 [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] boot 2024-03-18T00:46:24.632 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:24 smithi106 ceph-mon[35899]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T00:46:24.632 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:24 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T00:46:24.658 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi106:vg_nvme/lv_1 2024-03-18T00:46:24.892 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T00:46:25.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:25.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: osdmap e69: 10 total, 10 up, 10 in 2024-03-18T00:46:25.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.266 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:25.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:25.267 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: osdmap e69: 10 total, 10 up, 10 in 2024-03-18T00:46:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:25.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:25.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-18T00:46:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: osdmap e69: 10 total, 10 up, 10 in 2024-03-18T00:46:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:25.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:26.351 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:26 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi049 2024-03-18T00:46:26.351 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:26 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi049 to 4272M 2024-03-18T00:46:26.527 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:26 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi049 2024-03-18T00:46:26.527 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:26 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi049 to 4272M 2024-03-18T00:46:26.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.608+0000 7f2427bbc700 1 -- 172.21.15.106:0/875271917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102400 msgr2=0x7f2420102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:26.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.608+0000 7f2427bbc700 1 --2- 172.21.15.106:0/875271917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102400 0x7f2420102860 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f241c0099c0 tx=0x7f241c009cd0 comp rx=0 tx=0).stop 2024-03-18T00:46:26.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.608+0000 7f2427bbc700 1 -- 172.21.15.106:0/875271917 shutdown_connections 2024-03-18T00:46:26.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.608+0000 7f2427bbc700 1 --2- 172.21.15.106:0/875271917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2420101200 0x7f2420101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:26.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.608+0000 7f2427bbc700 1 --2- 172.21.15.106:0/875271917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2420102da0 0x7f2420109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:26.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.608+0000 7f2427bbc700 1 --2- 172.21.15.106:0/875271917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102400 0x7f2420102860 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:26.611 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.608+0000 7f2427bbc700 1 -- 172.21.15.106:0/875271917 >> 172.21.15.106:0/875271917 conn(0x7f24200fc7a0 msgr2=0x7f24200febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.608+0000 7f2427bbc700 1 -- 172.21.15.106:0/875271917 shutdown_connections 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.608+0000 7f2427bbc700 1 -- 172.21.15.106:0/875271917 wait complete. 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2427bbc700 1 Processor -- start 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2427bbc700 1 -- start start 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2427bbc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2420101200 0x7f242006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2427bbc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2420102400 0x7f242019d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2427bbc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102da0 0x7f242019f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2427bbc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f242010cde0 con 0x7f2420101200 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2427bbc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f242010cc60 con 0x7f2420102da0 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2427bbc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f242010cf60 con 0x7f2420102400 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2426159700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102da0 0x7f242019f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2426159700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102da0 0x7f242019f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.106:49000/0 (socket says 172.21.15.106:49000) 2024-03-18T00:46:26.612 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2426159700 1 -- 172.21.15.106:0/589645044 learned_addr learned my addr 172.21.15.106:0/589645044 (peer_addr_for_me v2:172.21.15.106:0/0) 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.609+0000 7f2425958700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2420101200 0x7f242006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2425157700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2420102400 0x7f242019d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2426159700 1 -- 172.21.15.106:0/589645044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2420102400 msgr2=0x7f242019d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2426159700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2420102400 0x7f242019d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2426159700 1 -- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2420101200 msgr2=0x7f242006b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2426159700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2420101200 0x7f242006b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2426159700 1 -- 172.21.15.106:0/589645044 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24201a0120 con 0x7f2420102da0 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2425958700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2420101200 0x7f242006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2425157700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2420102400 0x7f242019d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:46:26.613 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2426159700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102da0 0x7f242019f850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2418007ee0 tx=0x7f241800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:46:26.614 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2416ffd700 1 -- 172.21.15.106:0/589645044 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241800dbd0 con 0x7f2420102da0 2024-03-18T00:46:26.614 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24201a0410 con 0x7f2420102da0 2024-03-18T00:46:26.614 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2416ffd700 1 -- 172.21.15.106:0/589645044 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f241800dd70 con 0x7f2420102da0 2024-03-18T00:46:26.614 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2416ffd700 1 -- 172.21.15.106:0/589645044 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24180133f0 con 0x7f2420102da0 2024-03-18T00:46:26.614 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.610+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24201a0950 con 0x7f2420102da0 2024-03-18T00:46:26.614 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.611+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2420068f90 con 0x7f2420102da0 2024-03-18T00:46:26.615 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.612+0000 7f2416ffd700 1 -- 172.21.15.106:0/589645044 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f241801a020 con 0x7f2420102da0 2024-03-18T00:46:26.615 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.613+0000 7f2416ffd700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f240c0a2890 0x7f240c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:26.615 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.613+0000 7f2416ffd700 1 -- 172.21.15.106:0/589645044 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f24180cda70 con 0x7f2420102da0 2024-03-18T00:46:26.615 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.613+0000 7f2425958700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f240c0a2890 0x7f240c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:26.615 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.614+0000 7f2425958700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f240c0a2890 0x7f240c0a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2410005950 tx=0x7f24100058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:46:26.617 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.618+0000 7f2416ffd700 1 -- 172.21.15.106:0/589645044 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2418010070 con 0x7f2420102da0 2024-03-18T00:46:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:26 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi049 2024-03-18T00:46:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:26 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi049 to 4272M 2024-03-18T00:46:26.821 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:26.820+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2420062710 con 0x7f240c0a2890 2024-03-18T00:46:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:27 smithi049 ceph-mon[32130]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:27.543 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:27 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:46:27.544 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:27 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:46:27.544 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:27 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:27.549 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:27 smithi106 ceph-mon[35899]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:27.550 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:27 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:46:27.550 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:27 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:46:27.550 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:27 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:27.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:27 smithi107 ceph-mon[36270]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:27.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:27 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:46:27.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:27 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:46:27.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:27 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:28.485 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:28 smithi106 ceph-mon[35899]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:46:28.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:28 smithi049 ceph-mon[32130]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:46:28.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:28 smithi107 ceph-mon[36270]: from='client.24317 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi106:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:46:29.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:29 smithi106 ceph-mon[35899]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:29.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:29 smithi049 ceph-mon[32130]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:29 smithi107 ceph-mon[36270]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:30.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:30 smithi106 ceph-mon[35899]: from='client.? 172.21.15.106:0/575539723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f13f5e88-b234-4c16-a11b-b65a9100334b"}]: dispatch 2024-03-18T00:46:30.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:30 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f13f5e88-b234-4c16-a11b-b65a9100334b"}]: dispatch 2024-03-18T00:46:30.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:30 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f13f5e88-b234-4c16-a11b-b65a9100334b"}]': finished 2024-03-18T00:46:30.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:30 smithi106 ceph-mon[35899]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T00:46:30.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:30 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:30.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:30 smithi049 ceph-mon[32130]: from='client.? 172.21.15.106:0/575539723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f13f5e88-b234-4c16-a11b-b65a9100334b"}]: dispatch 2024-03-18T00:46:30.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:30 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f13f5e88-b234-4c16-a11b-b65a9100334b"}]: dispatch 2024-03-18T00:46:30.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:30 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f13f5e88-b234-4c16-a11b-b65a9100334b"}]': finished 2024-03-18T00:46:30.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:30 smithi049 ceph-mon[32130]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T00:46:30.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:30 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:30 smithi107 ceph-mon[36270]: from='client.? 172.21.15.106:0/575539723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f13f5e88-b234-4c16-a11b-b65a9100334b"}]: dispatch 2024-03-18T00:46:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:30 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f13f5e88-b234-4c16-a11b-b65a9100334b"}]: dispatch 2024-03-18T00:46:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:30 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f13f5e88-b234-4c16-a11b-b65a9100334b"}]': finished 2024-03-18T00:46:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:30 smithi107 ceph-mon[36270]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T00:46:30.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:30 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:31 smithi049 ceph-mon[32130]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:31 smithi049 ceph-mon[32130]: from='client.? 172.21.15.106:0/3981742033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:46:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:31 smithi107 ceph-mon[36270]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:31 smithi107 ceph-mon[36270]: from='client.? 172.21.15.106:0/3981742033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:46:31.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:31 smithi106 ceph-mon[35899]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:31.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:31 smithi106 ceph-mon[35899]: from='client.? 172.21.15.106:0/3981742033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:46:33.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:33 smithi049 ceph-mon[32130]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:33.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:33 smithi107 ceph-mon[36270]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:33.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:33 smithi106 ceph-mon[35899]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:35.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:35 smithi106 ceph-mon[35899]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:35.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:35 smithi049 ceph-mon[32130]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:35.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:35 smithi107 ceph-mon[36270]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:37.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:37 smithi106 ceph-mon[35899]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:37.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T00:46:37.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:37.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:37 smithi049 ceph-mon[32130]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:37.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T00:46:37.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:37 smithi107 ceph-mon[36270]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T00:46:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:38.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:38 smithi106 ceph-mon[35899]: Deploying daemon osd.10 on smithi106 2024-03-18T00:46:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:38 smithi049 ceph-mon[32130]: Deploying daemon osd.10 on smithi106 2024-03-18T00:46:38.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:38 smithi107 ceph-mon[36270]: Deploying daemon osd.10 on smithi106 2024-03-18T00:46:39.325 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:39 smithi106 ceph-mon[35899]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:39.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:39 smithi049 ceph-mon[32130]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:39 smithi107 ceph-mon[36270]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:41.278 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:41 smithi106 ceph-mon[35899]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:41.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:41 smithi049 ceph-mon[32130]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:41.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:41 smithi107 ceph-mon[36270]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:43.591 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:43 smithi106 ceph-mon[35899]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:43.592 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:43.592 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:43.592 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:43.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:43 smithi049 ceph-mon[32130]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:43.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:43.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:43.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:43 smithi107 ceph-mon[36270]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:43.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:45.602 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:45 smithi106 ceph-mon[35899]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:45.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:45 smithi049 ceph-mon[32130]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:45 smithi107 ceph-mon[36270]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:46.900 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.899+0000 7f2416ffd700 1 -- 172.21.15.106:0/589645044 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f2420062710 con 0x7f240c0a2890 2024-03-18T00:46:46.900 INFO:teuthology.orchestra.run.smithi106.stdout:Created osd(s) 10 on host 'smithi106' 2024-03-18T00:46:46.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f240c0a2890 msgr2=0x7f240c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:46.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f240c0a2890 0x7f240c0a4d50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2410005950 tx=0x7f24100058e0 comp rx=0 tx=0).stop 2024-03-18T00:46:46.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102da0 msgr2=0x7f242019f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:46.901 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102da0 0x7f242019f850 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2418007ee0 tx=0x7f241800d3b0 comp rx=0 tx=0).stop 2024-03-18T00:46:46.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 shutdown_connections 2024-03-18T00:46:46.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2420102da0 0x7f242019f850 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:46.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2420101200 0x7f242006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:46.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f240c0a2890 0x7f240c0a4d50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:46.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 --2- 172.21.15.106:0/589645044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2420102400 0x7f242019d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:46.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 >> 172.21.15.106:0/589645044 conn(0x7f24200fc7a0 msgr2=0x7f2420105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:46:46.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 shutdown_connections 2024-03-18T00:46:46.902 INFO:teuthology.orchestra.run.smithi106.stderr:2024-03-18T00:46:46.901+0000 7f2427bbc700 1 -- 172.21.15.106:0/589645044 wait complete. 2024-03-18T00:46:47.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='osd.10 [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='osd.10 [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T00:46:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T00:46:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='osd.10 [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:47.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:46:47.612 DEBUG:teuthology.orchestra.run.smithi106:osd.10> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.10.service 2024-03-18T00:46:47.615 INFO:tasks.cephadm:Deploying osd.11 on smithi107 with /dev/vg_nvme/lv_1... 2024-03-18T00:46:47.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T00:46:47.861 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:46:48.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:48 smithi106 ceph-mon[35899]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T00:46:48.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:48 smithi106 ceph-mon[35899]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T00:46:48.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:48 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:48.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:48 smithi106 ceph-mon[35899]: from='osd.10 [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:46:48.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:48 smithi106 ceph-mon[35899]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:46:48.447 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 00:46:48 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T00:46:48.252+0000 7f3d837be700 -1 osd.10 0 waiting for initial osdmap 2024-03-18T00:46:48.447 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 00:46:48 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T00:46:48.264+0000 7f3d7f5b1700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:46:48.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:48 smithi049 ceph-mon[32130]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T00:46:48.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:48 smithi049 ceph-mon[32130]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T00:46:48.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:48 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:48.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:48 smithi049 ceph-mon[32130]: from='osd.10 [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:46:48.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:48 smithi049 ceph-mon[32130]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:46:48.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:48 smithi107 ceph-mon[36270]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T00:46:48.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:48 smithi107 ceph-mon[36270]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T00:46:48.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:48 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:48.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:48 smithi107 ceph-mon[36270]: from='osd.10 [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:46:48.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:48 smithi107 ceph-mon[36270]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]: dispatch 2024-03-18T00:46:49.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:49 smithi107 ceph-mon[36270]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:49.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:49 smithi107 ceph-mon[36270]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:46:49.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:49 smithi107 ceph-mon[36270]: osdmap e72: 11 total, 10 up, 11 in 2024-03-18T00:46:49.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:49.389 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:49.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:49 smithi106 ceph-mon[35899]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:49.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:49 smithi106 ceph-mon[35899]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:46:49.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:49 smithi106 ceph-mon[35899]: osdmap e72: 11 total, 10 up, 11 in 2024-03-18T00:46:49.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:49.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:49.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:49 smithi049 ceph-mon[32130]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:49.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:49 smithi049 ceph-mon[32130]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi106", "root=default"]}]': finished 2024-03-18T00:46:49.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:49 smithi049 ceph-mon[32130]: osdmap e72: 11 total, 10 up, 11 in 2024-03-18T00:46:49.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:49.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:50.045 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T00:46:50.045 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T00:46:50.045 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-03-18T00:46:50.046 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-03-18T00:46:50.046 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113637 s, 923 MB/s 2024-03-18T00:46:50.046 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-03-18T00:46:50.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:50 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:46:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:50 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:46:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:50 smithi107 ceph-mon[36270]: osd.10 [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] boot 2024-03-18T00:46:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:50 smithi107 ceph-mon[36270]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T00:46:50.344 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:50 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:50.406 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:50 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:46:50.407 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:50 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:46:50.407 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:50.407 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:50 smithi106 ceph-mon[35899]: osd.10 [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] boot 2024-03-18T00:46:50.407 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:50 smithi106 ceph-mon[35899]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T00:46:50.407 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:50 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:50.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:50 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:46:50.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:50 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:46:50.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:50.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:50 smithi049 ceph-mon[32130]: osd.10 [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] boot 2024-03-18T00:46:50.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:50 smithi049 ceph-mon[32130]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T00:46:50.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:50 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T00:46:50.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi107:vg_nvme/lv_1 2024-03-18T00:46:50.989 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:46:51.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:51.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T00:46:51.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.386 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:51.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:51.387 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T00:46:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:51.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:51.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 291 MiB used, 894 GiB / 894 GiB avail 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:46:51.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:46:52.413 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:52 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi106 2024-03-18T00:46:52.413 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:52 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi106 to 4270M 2024-03-18T00:46:52.413 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:52 smithi107 ceph-mon[36270]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T00:46:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:52 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi106 2024-03-18T00:46:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:52 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi106 to 4270M 2024-03-18T00:46:52.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:52 smithi049 ceph-mon[32130]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.646+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/655576149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 msgr2=0x7fd49c1096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.646+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/655576149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 0x7fd49c1096c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd49400b0a0 tx=0x7fd49400b3b0 comp rx=0 tx=0).stop 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.646+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/655576149 shutdown_connections 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.646+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/655576149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 0x7fd49c1096c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.646+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/655576149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd49c101270 0x7fd49c101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.646+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/655576149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c102470 0x7fd49c1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.646+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/655576149 >> 172.21.15.107:0/655576149 conn(0x7fd49c0fc830 msgr2=0x7fd49c0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.646+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/655576149 shutdown_connections 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.646+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/655576149 wait complete. 2024-03-18T00:46:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd4a1bf3700 1 Processor -- start 2024-03-18T00:46:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd4a1bf3700 1 -- start start 2024-03-18T00:46:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd4a1bf3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd49c101270 0x7fd49c105200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd4a1bf3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c102470 0x7fd49c103850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd4a1bf3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 0x7fd49c103d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd4a1bf3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c10ce70 con 0x7fd49c102470 2024-03-18T00:46:52.649 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd4a1bf3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c10ccf0 con 0x7fd49c101270 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd4a1bf3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd49c10cff0 con 0x7fd49c102e10 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd49affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c102470 0x7fd49c103850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd49bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 0x7fd49c103d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.647+0000 7fd49affd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c102470 0x7fd49c103850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.107:51680/0 (socket says 172.21.15.107:51680) 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd49bfff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 0x7fd49c103d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:56442/0 (socket says 172.21.15.107:56442) 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd49bfff700 1 -- 172.21.15.107:0/4199616902 learned_addr learned my addr 172.21.15.107:0/4199616902 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd49b7fe700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd49c101270 0x7fd49c105200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd49bfff700 1 -- 172.21.15.107:0/4199616902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd49c101270 msgr2=0x7fd49c105200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd49bfff700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd49c101270 0x7fd49c105200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd49bfff700 1 -- 172.21.15.107:0/4199616902 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c102470 msgr2=0x7fd49c103850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd49bfff700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c102470 0x7fd49c103850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:46:52.650 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd49bfff700 1 -- 172.21.15.107:0/4199616902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd49c104640 con 0x7fd49c102e10 2024-03-18T00:46:52.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd49bfff700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 0x7fd49c103d90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd49400b070 tx=0x7fd49400be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:46:52.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd498ff9700 1 -- 172.21.15.107:0/4199616902 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd494021020 con 0x7fd49c102e10 2024-03-18T00:46:52.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd49c1a25a0 con 0x7fd49c102e10 2024-03-18T00:46:52.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd498ff9700 1 -- 172.21.15.107:0/4199616902 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd494005520 con 0x7fd49c102e10 2024-03-18T00:46:52.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd498ff9700 1 -- 172.21.15.107:0/4199616902 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd49401b900 con 0x7fd49c102e10 2024-03-18T00:46:52.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.648+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd49c1a2ad0 con 0x7fd49c102e10 2024-03-18T00:46:52.651 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.649+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd49c068f90 con 0x7fd49c102e10 2024-03-18T00:46:52.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.650+0000 7fd498ff9700 1 -- 172.21.15.107:0/4199616902 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd494024020 con 0x7fd49c102e10 2024-03-18T00:46:52.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.651+0000 7fd498ff9700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd4840a2890 0x7fd4840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:46:52.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.651+0000 7fd498ff9700 1 -- 172.21.15.107:0/4199616902 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7fd49400f020 con 0x7fd49c102e10 2024-03-18T00:46:52.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.651+0000 7fd49b7fe700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd4840a2890 0x7fd4840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:46:52.652 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.652+0000 7fd49b7fe700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd4840a2890 0x7fd4840a4d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd49c1022d0 tx=0x7fd48c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:46:52.654 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.654+0000 7fd498ff9700 1 -- 172.21.15.107:0/4199616902 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd49409d180 con 0x7fd49c102e10 2024-03-18T00:46:52.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:52 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi106 2024-03-18T00:46:52.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:52 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi106 to 4270M 2024-03-18T00:46:52.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:52 smithi106 ceph-mon[35899]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T00:46:52.860 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:46:52.860+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd49c0fe890 con 0x7fd4840a2890 2024-03-18T00:46:53.576 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:53 smithi107 ceph-mon[36270]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:46:53.576 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:53 smithi107 ceph-mon[36270]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T00:46:53.576 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:46:53.576 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:46:53.576 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:53 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:53 smithi049 ceph-mon[32130]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:46:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:53 smithi049 ceph-mon[32130]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T00:46:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:46:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:46:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:53 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:53.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:53 smithi106 ceph-mon[35899]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:46:53.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:53 smithi106 ceph-mon[35899]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T00:46:53.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T00:46:53.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T00:46:53.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:53 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:46:54.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:54 smithi107 ceph-mon[36270]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:46:54.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:54 smithi107 ceph-mon[36270]: osdmap e77: 11 total, 11 up, 11 in 2024-03-18T00:46:54.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:54 smithi049 ceph-mon[32130]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:46:54.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:54 smithi049 ceph-mon[32130]: osdmap e77: 11 total, 11 up, 11 in 2024-03-18T00:46:54.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:54 smithi106 ceph-mon[35899]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:46:54.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:54 smithi106 ceph-mon[35899]: osdmap e77: 11 total, 11 up, 11 in 2024-03-18T00:46:55.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:55 smithi049 ceph-mon[32130]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T00:46:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:55 smithi107 ceph-mon[36270]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T00:46:55.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:55 smithi106 ceph-mon[35899]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-18T00:46:56.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:56 smithi107 ceph-mon[36270]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T00:46:56.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:56 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/523538539' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95663d7f-7dd2-4eb8-bcf1-e7f5923105d4"}]: dispatch 2024-03-18T00:46:56.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:56 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95663d7f-7dd2-4eb8-bcf1-e7f5923105d4"}]: dispatch 2024-03-18T00:46:56.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:56 smithi107 ceph-mon[36270]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95663d7f-7dd2-4eb8-bcf1-e7f5923105d4"}]': finished 2024-03-18T00:46:56.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:56 smithi107 ceph-mon[36270]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T00:46:56.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:56 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:46:56.526 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:56 smithi049 ceph-mon[32130]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T00:46:56.526 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:56 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/523538539' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95663d7f-7dd2-4eb8-bcf1-e7f5923105d4"}]: dispatch 2024-03-18T00:46:56.526 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:56 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95663d7f-7dd2-4eb8-bcf1-e7f5923105d4"}]: dispatch 2024-03-18T00:46:56.527 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:56 smithi049 ceph-mon[32130]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95663d7f-7dd2-4eb8-bcf1-e7f5923105d4"}]': finished 2024-03-18T00:46:56.527 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:56 smithi049 ceph-mon[32130]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T00:46:56.527 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:56 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:46:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:56 smithi106 ceph-mon[35899]: pgmap v244: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-18T00:46:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:56 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/523538539' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95663d7f-7dd2-4eb8-bcf1-e7f5923105d4"}]: dispatch 2024-03-18T00:46:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:56 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95663d7f-7dd2-4eb8-bcf1-e7f5923105d4"}]: dispatch 2024-03-18T00:46:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:56 smithi106 ceph-mon[35899]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95663d7f-7dd2-4eb8-bcf1-e7f5923105d4"}]': finished 2024-03-18T00:46:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:56 smithi106 ceph-mon[35899]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T00:46:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:56 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:46:57.542 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:57 smithi049 ceph-mon[32130]: from='client.? 172.21.15.107:0/1777976615' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:46:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:57 smithi107 ceph-mon[36270]: from='client.? 172.21.15.107:0/1777976615' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:46:57.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:57 smithi106 ceph-mon[35899]: from='client.? 172.21.15.107:0/1777976615' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T00:46:58.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:46:58 smithi049 ceph-mon[32130]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:46:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:46:58 smithi107 ceph-mon[36270]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:46:58.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:46:58 smithi106 ceph-mon[35899]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T00:47:00.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:00 smithi107 ceph-mon[36270]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T00:47:00.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:00 smithi049 ceph-mon[32130]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T00:47:00.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:00 smithi106 ceph-mon[35899]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-18T00:47:02.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:02 smithi049 ceph-mon[32130]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:02 smithi107 ceph-mon[36270]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:02.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:02 smithi106 ceph-mon[35899]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T00:47:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:03.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T00:47:03.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:03.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T00:47:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:04.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:04 smithi107 ceph-mon[36270]: Deploying daemon osd.11 on smithi107 2024-03-18T00:47:04.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:04 smithi107 ceph-mon[36270]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:04.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:04 smithi049 ceph-mon[32130]: Deploying daemon osd.11 on smithi107 2024-03-18T00:47:04.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:04 smithi049 ceph-mon[32130]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:04.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:04 smithi106 ceph-mon[35899]: Deploying daemon osd.11 on smithi107 2024-03-18T00:47:04.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:04 smithi106 ceph-mon[35899]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:06.815 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:06 smithi107 ceph-mon[36270]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:06.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:06 smithi049 ceph-mon[32130]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:06.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:06 smithi106 ceph-mon[35899]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:08 smithi049 ceph-mon[32130]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:08 smithi107 ceph-mon[36270]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:08.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:08 smithi106 ceph-mon[35899]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:10.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:10.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:10.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:10 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:47:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:10 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:47:10.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:10.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:10.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:10 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:47:11.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:11 smithi049 ceph-mon[32130]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:11.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:11 smithi107 ceph-mon[36270]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:11 smithi106 ceph-mon[35899]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:12.582 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.582+0000 7fd498ff9700 1 -- 172.21.15.107:0/4199616902 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fd49c0fe890 con 0x7fd4840a2890 2024-03-18T00:47:12.582 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 11 on host 'smithi107' 2024-03-18T00:47:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd4840a2890 msgr2=0x7fd4840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd4840a2890 0x7fd4840a4d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd49c1022d0 tx=0x7fd48c009190 comp rx=0 tx=0).stop 2024-03-18T00:47:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 msgr2=0x7fd49c103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 0x7fd49c103d90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd49400b070 tx=0x7fd49400be60 comp rx=0 tx=0).stop 2024-03-18T00:47:12.584 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 shutdown_connections 2024-03-18T00:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd4840a2890 0x7fd4840a4d50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd49c102e10 0x7fd49c103d90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd49c101270 0x7fd49c105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 --2- 172.21.15.107:0/4199616902 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd49c102470 0x7fd49c103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 >> 172.21.15.107:0/4199616902 conn(0x7fd49c0fc830 msgr2=0x7fd49c0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 shutdown_connections 2024-03-18T00:47:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:12.584+0000 7fd4a1bf3700 1 -- 172.21.15.107:0/4199616902 wait complete. 2024-03-18T00:47:13.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:13 smithi107 ceph-mon[36270]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:13.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:13.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:13 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:13.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:13 smithi107 ceph-mon[36270]: from='osd.11 [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T00:47:13.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:13 smithi107 ceph-mon[36270]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T00:47:13.242 DEBUG:teuthology.orchestra.run.smithi107:osd.11> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.11.service 2024-03-18T00:47:13.244 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-18T00:47:13.245 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T00:47:13.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:13 smithi049 ceph-mon[32130]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:13.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:13.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:13 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:13.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:13 smithi049 ceph-mon[32130]: from='osd.11 [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T00:47:13.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:13 smithi049 ceph-mon[32130]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T00:47:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:13 smithi106 ceph-mon[35899]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:13 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:13 smithi106 ceph-mon[35899]: from='osd.11 [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T00:47:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:13 smithi106 ceph-mon[35899]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T00:47:13.520 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:47:14.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:14.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:14.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-mon[36270]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:14.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-mon[36270]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T00:47:14.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-mon[36270]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T00:47:14.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:14.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-mon[36270]: from='osd.11 [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:47:14.543 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-mon[36270]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:47:14.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:14 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:14.666 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:14 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:14.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:14 smithi049 ceph-mon[32130]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:14.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:14 smithi049 ceph-mon[32130]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T00:47:14.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:14 smithi049 ceph-mon[32130]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T00:47:14.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:14 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:14.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:14 smithi049 ceph-mon[32130]: from='osd.11 [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:47:14.667 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:14 smithi049 ceph-mon[32130]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:47:14.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:14 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:14.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:14 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:14.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:14 smithi106 ceph-mon[35899]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:14.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:14 smithi106 ceph-mon[35899]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T00:47:14.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:14 smithi106 ceph-mon[35899]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T00:47:14.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:14 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:14.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:14 smithi106 ceph-mon[35899]: from='osd.11 [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:47:14.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:14 smithi106 ceph-mon[35899]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-18T00:47:14.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T00:47:14.557+0000 7f659f4de700 -1 osd.11 0 waiting for initial osdmap 2024-03-18T00:47:14.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 00:47:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T00:47:14.572+0000 7f659b2d1700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T00:47:15.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.435+0000 7f437583d700 1 -- 172.21.15.49:0/1559182503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 msgr2=0x7f4370105280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:15.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 --2- 172.21.15.49:0/1559182503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 0x7f4370105280 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4360009990 tx=0x7f4360009ca0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 -- 172.21.15.49:0/1559182503 shutdown_connections 2024-03-18T00:47:15.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 --2- 172.21.15.49:0/1559182503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43701080f0 0x7f437010a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 --2- 172.21.15.49:0/1559182503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 0x7f4370105280 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 --2- 172.21.15.49:0/1559182503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43701057c0 0x7f4370107bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 -- 172.21.15.49:0/1559182503 >> 172.21.15.49:0/1559182503 conn(0x7f43700fc860 msgr2=0x7f43700fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 -- 172.21.15.49:0/1559182503 shutdown_connections 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 -- 172.21.15.49:0/1559182503 wait complete. 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 Processor -- start 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.436+0000 7f437583d700 1 -- start start 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f437583d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 0x7f4370198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f437583d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43701057c0 0x7f4370198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:15.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f437583d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43701080f0 0x7f437019fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f437583d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f437010df40 con 0x7f4370102e90 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f437583d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f437010ddc0 con 0x7f43701080f0 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f437583d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f437010e0c0 con 0x7f43701057c0 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 0x7f4370198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 0x7f4370198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54546/0 (socket says 172.21.15.49:54546) 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436effd700 1 -- 172.21.15.49:0/2209680623 learned_addr learned my addr 172.21.15.49:0/2209680623 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436e7fc700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43701057c0 0x7f4370198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436f7fe700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43701080f0 0x7f437019fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436e7fc700 1 -- 172.21.15.49:0/2209680623 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43701080f0 msgr2=0x7f437019fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436e7fc700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43701080f0 0x7f437019fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436e7fc700 1 -- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 msgr2=0x7f4370198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436e7fc700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 0x7f4370198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436e7fc700 1 -- 172.21.15.49:0/2209680623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43701a0160 con 0x7f43701057c0 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436effd700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 0x7f4370198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436f7fe700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43701080f0 0x7f437019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f436e7fc700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43701057c0 0x7f4370198a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f436400d980 tx=0x7f436400dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.437+0000 7f437483b700 1 -- 172.21.15.49:0/2209680623 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4364017070 con 0x7f43701057c0 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.438+0000 7f437483b700 1 -- 172.21.15.49:0/2209680623 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4364009e10 con 0x7f43701057c0 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.438+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43701a0450 con 0x7f43701057c0 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.438+0000 7f437483b700 1 -- 172.21.15.49:0/2209680623 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436400f3c0 con 0x7f43701057c0 2024-03-18T00:47:15.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.438+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43701a0990 con 0x7f43701057c0 2024-03-18T00:47:15.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.438+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4370068f90 con 0x7f43701057c0 2024-03-18T00:47:15.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.440+0000 7f437483b700 1 -- 172.21.15.49:0/2209680623 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f436401a070 con 0x7f43701057c0 2024-03-18T00:47:15.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.440+0000 7f437483b700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43580a2890 0x7f43580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:15.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.440+0000 7f436effd700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43580a2890 0x7f43580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:15.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.440+0000 7f437483b700 1 -- 172.21.15.49:0/2209680623 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f43640cdf20 con 0x7f43701057c0 2024-03-18T00:47:15.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.441+0000 7f436effd700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43580a2890 0x7f43580a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f436000b2e0 tx=0x7f436000b880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:15.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.444+0000 7f437483b700 1 -- 172.21.15.49:0/2209680623 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f436409b580 con 0x7f43701057c0 2024-03-18T00:47:15.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:15 smithi049 ceph-mon[32130]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:47:15.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:15 smithi049 ceph-mon[32130]: osdmap e80: 12 total, 11 up, 12 in 2024-03-18T00:47:15.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:15.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:15 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:15.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.652+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f4370062ee0 con 0x7f43701057c0 2024-03-18T00:47:15.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.653+0000 7f437483b700 1 -- 172.21.15.49:0/2209680623 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f436409ae80 con 0x7f43701057c0 2024-03-18T00:47:15.653 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:15.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43580a2890 msgr2=0x7f43580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:15.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43580a2890 0x7f43580a4d50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f436000b2e0 tx=0x7f436000b880 comp rx=0 tx=0).stop 2024-03-18T00:47:15.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43701057c0 msgr2=0x7f4370198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:15.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43701057c0 0x7f4370198a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f436400d980 tx=0x7f436400dc90 comp rx=0 tx=0).stop 2024-03-18T00:47:15.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 shutdown_connections 2024-03-18T00:47:15.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43701080f0 0x7f437019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4370102e90 0x7f4370198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43580a2890 0x7f43580a4d50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 --2- 172.21.15.49:0/2209680623 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43701057c0 0x7f4370198a40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:15.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 >> 172.21.15.49:0/2209680623 conn(0x7f43700fc860 msgr2=0x7f43701038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:15.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 shutdown_connections 2024-03-18T00:47:15.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:15.655+0000 7f437583d700 1 -- 172.21.15.49:0/2209680623 wait complete. 2024-03-18T00:47:15.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:15 smithi107 ceph-mon[36270]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:47:15.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:15 smithi107 ceph-mon[36270]: osdmap e80: 12 total, 11 up, 12 in 2024-03-18T00:47:15.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:15.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:15 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:15.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:15 smithi106 ceph-mon[35899]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-18T00:47:15.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:15 smithi106 ceph-mon[35899]: osdmap e80: 12 total, 11 up, 12 in 2024-03-18T00:47:15.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:15.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:15 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:16.480 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1710722835,"num_in_osds":12,"osd_in_since":1710722815,"num_remapped_pgs":0} 2024-03-18T00:47:16.480 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T00:47:16.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:16 smithi107 ceph-mon[36270]: purged_snaps scrub starts 2024-03-18T00:47:16.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:16 smithi107 ceph-mon[36270]: purged_snaps scrub ok 2024-03-18T00:47:16.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:16 smithi107 ceph-mon[36270]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:16 smithi107 ceph-mon[36270]: osd.11 [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] boot 2024-03-18T00:47:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:16 smithi107 ceph-mon[36270]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T00:47:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:16 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:16.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:16 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2209680623' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T00:47:16.742 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:47:16.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:16 smithi049 ceph-mon[32130]: purged_snaps scrub starts 2024-03-18T00:47:16.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:16 smithi049 ceph-mon[32130]: purged_snaps scrub ok 2024-03-18T00:47:16.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:16 smithi049 ceph-mon[32130]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:16.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:16 smithi049 ceph-mon[32130]: osd.11 [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] boot 2024-03-18T00:47:16.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:16 smithi049 ceph-mon[32130]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T00:47:16.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:16 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:16.778 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:16 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2209680623' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T00:47:16.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:16 smithi106 ceph-mon[35899]: purged_snaps scrub starts 2024-03-18T00:47:16.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:16 smithi106 ceph-mon[35899]: purged_snaps scrub ok 2024-03-18T00:47:16.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:16 smithi106 ceph-mon[35899]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-03-18T00:47:16.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:16 smithi106 ceph-mon[35899]: osd.11 [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] boot 2024-03-18T00:47:16.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:16 smithi106 ceph-mon[35899]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T00:47:16.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:16 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T00:47:16.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:16 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2209680623' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T00:47:17.847 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T00:47:17.847 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: Detected new or changed devices on smithi107 2024-03-18T00:47:17.847 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.847 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi107 to 4270M 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:47:17.848 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:17 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: Detected new or changed devices on smithi107 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi107 to 4270M 2024-03-18T00:47:17.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:47:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:47:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:47:17.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:17 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T00:47:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: Detected new or changed devices on smithi107 2024-03-18T00:47:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi107 to 4270M 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:47:17.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:17.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:47:17.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:17 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:18.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.556+0000 7fc778bf3700 1 -- 172.21.15.49:0/2975131219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102ec0 msgr2=0x7fc774106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.556+0000 7fc778bf3700 1 --2- 172.21.15.49:0/2975131219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102ec0 0x7fc774106f10 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0099c0 tx=0x7fc75c009cd0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.556+0000 7fc778bf3700 1 -- 172.21.15.49:0/2975131219 shutdown_connections 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.556+0000 7fc778bf3700 1 --2- 172.21.15.49:0/2975131219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc774102510 0x7fc7741028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.556+0000 7fc778bf3700 1 --2- 172.21.15.49:0/2975131219 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7741075f0 0x7fc77410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.556+0000 7fc778bf3700 1 --2- 172.21.15.49:0/2975131219 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102ec0 0x7fc774106f10 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.556+0000 7fc778bf3700 1 -- 172.21.15.49:0/2975131219 >> 172.21.15.49:0/2975131219 conn(0x7fc7740fdd80 msgr2=0x7fc7741001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.556+0000 7fc778bf3700 1 -- 172.21.15.49:0/2975131219 shutdown_connections 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.556+0000 7fc778bf3700 1 -- 172.21.15.49:0/2975131219 wait complete. 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc778bf3700 1 Processor -- start 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc778bf3700 1 -- start start 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc778bf3700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102510 0x7fc7740745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:18.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc778bf3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc774102ec0 0x7fc774074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc778bf3700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7741075f0 0x7fc774075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc778bf3700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc774112630 con 0x7fc774102510 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc778bf3700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc7741124b0 con 0x7fc7741075f0 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc778bf3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc7741127b0 con 0x7fc774102ec0 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc77259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102510 0x7fc7740745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc77259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102510 0x7fc7740745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56860/0 (socket says 172.21.15.49:56860) 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.557+0000 7fc77259c700 1 -- 172.21.15.49:0/4011715268 learned_addr learned my addr 172.21.15.49:0/4011715268 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc772d9d700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7741075f0 0x7fc774075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc771d9b700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc774102ec0 0x7fc774074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc77259c700 1 -- 172.21.15.49:0/4011715268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc774102ec0 msgr2=0x7fc774074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc77259c700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc774102ec0 0x7fc774074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc77259c700 1 -- 172.21.15.49:0/4011715268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7741075f0 msgr2=0x7fc774075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:18.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc77259c700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7741075f0 0x7fc774075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc77259c700 1 -- 172.21.15.49:0/4011715268 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7741a7000 con 0x7fc774102510 2024-03-18T00:47:18.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc772d9d700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7741075f0 0x7fc774075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:47:18.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc77259c700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102510 0x7fc7740745d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00ba50 tx=0x7fc76c00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:18.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc76b7fe700 1 -- 172.21.15.49:0/4011715268 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c019070 con 0x7fc774102510 2024-03-18T00:47:18.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc76b7fe700 1 -- 172.21.15.49:0/4011715268 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc76c014440 con 0x7fc774102510 2024-03-18T00:47:18.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7741a72f0 con 0x7fc774102510 2024-03-18T00:47:18.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc76b7fe700 1 -- 172.21.15.49:0/4011715268 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0075d0 con 0x7fc774102510 2024-03-18T00:47:18.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.558+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7741a7850 con 0x7fc774102510 2024-03-18T00:47:18.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.559+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc774068f90 con 0x7fc774102510 2024-03-18T00:47:18.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.560+0000 7fc76b7fe700 1 -- 172.21.15.49:0/4011715268 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc76c01c070 con 0x7fc774102510 2024-03-18T00:47:18.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.561+0000 7fc76b7fe700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc7600a2910 0x7fc7600a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:18.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.561+0000 7fc771d9b700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc7600a2910 0x7fc7600a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:18.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.561+0000 7fc76b7fe700 1 -- 172.21.15.49:0/4011715268 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc76c0cd900 con 0x7fc774102510 2024-03-18T00:47:18.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.561+0000 7fc771d9b700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc7600a2910 0x7fc7600a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc75c000c00 tx=0x7fc75c005940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:18.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.565+0000 7fc76b7fe700 1 -- 172.21.15.49:0/4011715268 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc76c09ae40 con 0x7fc774102510 2024-03-18T00:47:18.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.771+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc774075ab0 con 0x7fc774102510 2024-03-18T00:47:18.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.772+0000 7fc76b7fe700 1 -- 172.21.15.49:0/4011715268 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18443 (secure 0 0 0) 0x7fc76c09a740 con 0x7fc774102510 2024-03-18T00:47:18.773 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:18.773 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":82,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:47:16.556495+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc7600a2910 msgr2=0x7fc7600a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc7600a2910 0x7fc7600a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc75c000c00 tx=0x7fc75c005940 comp rx=0 tx=0).stop 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102510 msgr2=0x7fc7740745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102510 0x7fc7740745d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc76c00ba50 tx=0x7fc76c00bd60 comp rx=0 tx=0).stop 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 shutdown_connections 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc7741075f0 0x7fc774075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc774102510 0x7fc7740745d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc7600a2910 0x7fc7600a4dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 --2- 172.21.15.49:0/4011715268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc774102ec0 0x7fc774074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 >> 172.21.15.49:0/4011715268 conn(0x7fc7740fdd80 msgr2=0x7fc774100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 shutdown_connections 2024-03-18T00:47:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:18.775+0000 7fc778bf3700 1 -- 172.21.15.49:0/4011715268 wait complete. 2024-03-18T00:47:19.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:19 smithi049 ceph-mon[32130]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:19.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:19 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/4011715268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:47:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:19 smithi107 ceph-mon[36270]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:19 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/4011715268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:47:19.437 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:47:19.437 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-18T00:47:19.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:19 smithi106 ceph-mon[35899]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:19.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:19 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/4011715268' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:47:19.692 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:47:21.280 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:21 smithi049 ceph-mon[32130]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:21 smithi107 ceph-mon[36270]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:21.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:21 smithi106 ceph-mon[35899]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:21.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.678+0000 7f0018bfa700 1 -- 172.21.15.49:0/3779388420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0014107680 msgr2=0x7f001410f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:21.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.678+0000 7f0018bfa700 1 --2- 172.21.15.49:0/3779388420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0014107680 0x7f001410f200 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f00040099c0 tx=0x7f0004009cd0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.678+0000 7f0018bfa700 1 -- 172.21.15.49:0/3779388420 shutdown_connections 2024-03-18T00:47:21.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.678+0000 7f0018bfa700 1 --2- 172.21.15.49:0/3779388420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0014107680 0x7f001410f200 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.678+0000 7f0018bfa700 1 --2- 172.21.15.49:0/3779388420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f001406ac30 0x7f001406b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.678+0000 7f0018bfa700 1 --2- 172.21.15.49:0/3779388420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001406b5e0 0x7f0014106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.678+0000 7f0018bfa700 1 -- 172.21.15.49:0/3779388420 >> 172.21.15.49:0/3779388420 conn(0x7f0014077220 msgr2=0x7f0014077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:21.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.678+0000 7f0018bfa700 1 -- 172.21.15.49:0/3779388420 shutdown_connections 2024-03-18T00:47:21.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.678+0000 7f0018bfa700 1 -- 172.21.15.49:0/3779388420 wait complete. 2024-03-18T00:47:21.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0018bfa700 1 Processor -- start 2024-03-18T00:47:21.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0018bfa700 1 -- start start 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0018bfa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001406ac30 0x7f001419e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0018bfa700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f001406b5e0 0x7f001419ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0018bfa700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0014107680 0x7f00141988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0018bfa700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0014112670 con 0x7f001406b5e0 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0018bfa700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f00141124f0 con 0x7f0014107680 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0018bfa700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f00141127f0 con 0x7f001406ac30 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0011d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f001406b5e0 0x7f001419ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0011d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f001406b5e0 0x7f001419ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56868/0 (socket says 172.21.15.49:56868) 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.679+0000 7f0011d9b700 1 -- 172.21.15.49:0/1084565803 learned_addr learned my addr 172.21.15.49:0/1084565803 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f0012d9d700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0014107680 0x7f00141988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f001259c700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001406ac30 0x7f001419e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f0011d9b700 1 -- 172.21.15.49:0/1084565803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001406ac30 msgr2=0x7f001419e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:21.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f0011d9b700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001406ac30 0x7f001419e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f0011d9b700 1 -- 172.21.15.49:0/1084565803 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0014107680 msgr2=0x7f00141988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f0011d9b700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0014107680 0x7f00141988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f0011d9b700 1 -- 172.21.15.49:0/1084565803 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0014199050 con 0x7f001406b5e0 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f001259c700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001406ac30 0x7f001419e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f0011d9b700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f001406b5e0 0x7f001419ec70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0008007ee0 tx=0x7f000800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f00037fe700 1 -- 172.21.15.49:0/1084565803 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000800dbd0 con 0x7f001406b5e0 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0014199340 con 0x7f001406b5e0 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f00037fe700 1 -- 172.21.15.49:0/1084565803 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f000800dd70 con 0x7f001406b5e0 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f00037fe700 1 -- 172.21.15.49:0/1084565803 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00080133f0 con 0x7f001406b5e0 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.680+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00141a5600 con 0x7f001406b5e0 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.681+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0014103a30 con 0x7f001406b5e0 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.682+0000 7f00037fe700 1 -- 172.21.15.49:0/1084565803 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0008005000 con 0x7f001406b5e0 2024-03-18T00:47:21.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.683+0000 7f00037fe700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efffc0a2890 0x7efffc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:21.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.683+0000 7f001259c700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efffc0a2890 0x7efffc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:21.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.683+0000 7f00037fe700 1 -- 172.21.15.49:0/1084565803 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f00080cdc20 con 0x7f001406b5e0 2024-03-18T00:47:21.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.683+0000 7f001259c700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efffc0a2890 0x7efffc0a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f000c005d90 tx=0x7f000c005d00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:21.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.686+0000 7f00037fe700 1 -- 172.21.15.49:0/1084565803 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0008021080 con 0x7f001406b5e0 2024-03-18T00:47:21.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.890+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0014199e20 con 0x7f001406b5e0 2024-03-18T00:47:21.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.890+0000 7f00037fe700 1 -- 172.21.15.49:0/1084565803 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f0008010070 con 0x7f001406b5e0 2024-03-18T00:47:21.891 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:47:21.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efffc0a2890 msgr2=0x7efffc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:21.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efffc0a2890 0x7efffc0a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f000c005d90 tx=0x7f000c005d00 comp rx=0 tx=0).stop 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f001406b5e0 msgr2=0x7f001419ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f001406b5e0 0x7f001419ec70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0008007ee0 tx=0x7f000800d3b0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 shutdown_connections 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efffc0a2890 0x7efffc0a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f001406ac30 0x7f001419e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0014107680 0x7f00141988f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 --2- 172.21.15.49:0/1084565803 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f001406b5e0 0x7f001419ec70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 >> 172.21.15.49:0/1084565803 conn(0x7f0014077220 msgr2=0x7f00140ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 shutdown_connections 2024-03-18T00:47:21.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:21.892+0000 7f0018bfa700 1 -- 172.21.15.49:0/1084565803 wait complete. 2024-03-18T00:47:22.296 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:22 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1084565803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:47:22.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:22 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1084565803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:47:22.457 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:22 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1084565803' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:47:22.666 INFO:tasks.cephadm:Adding mds.a on smithi049 2024-03-18T00:47:22.666 INFO:tasks.cephadm:Adding mds.d on smithi049 2024-03-18T00:47:22.666 INFO:tasks.cephadm:Adding mds.g on smithi049 2024-03-18T00:47:22.666 INFO:tasks.cephadm:Adding mds.j on smithi049 2024-03-18T00:47:22.666 INFO:tasks.cephadm:Adding mds.b on smithi106 2024-03-18T00:47:22.667 INFO:tasks.cephadm:Adding mds.e on smithi106 2024-03-18T00:47:22.667 INFO:tasks.cephadm:Adding mds.h on smithi106 2024-03-18T00:47:22.667 INFO:tasks.cephadm:Adding mds.k on smithi106 2024-03-18T00:47:22.667 INFO:tasks.cephadm:Adding mds.c on smithi107 2024-03-18T00:47:22.667 INFO:tasks.cephadm:Adding mds.f on smithi107 2024-03-18T00:47:22.667 INFO:tasks.cephadm:Adding mds.i on smithi107 2024-03-18T00:47:22.667 INFO:tasks.cephadm:Adding mds.l on smithi107 2024-03-18T00:47:22.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l' 2024-03-18T00:47:22.919 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T00:47:23.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:23 smithi107 ceph-mon[36270]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:23.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:23 smithi049 ceph-mon[32130]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:23.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:23 smithi106 ceph-mon[35899]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:24.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.547+0000 7f97bb835700 1 -- 172.21.15.107:0/722021208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 msgr2=0x7f97b4109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:24.550 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.547+0000 7f97bb835700 1 --2- 172.21.15.107:0/722021208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 0x7f97b4109730 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00b0d0 tx=0x7f97ac00b3e0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.547+0000 7f97bb835700 1 -- 172.21.15.107:0/722021208 shutdown_connections 2024-03-18T00:47:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.547+0000 7f97bb835700 1 --2- 172.21.15.107:0/722021208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 0x7f97b4109730 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.547+0000 7f97bb835700 1 --2- 172.21.15.107:0/722021208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97b4102790 0x7f97b4102bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.547+0000 7f97bb835700 1 --2- 172.21.15.107:0/722021208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97b41014d0 0x7f97b4101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.547+0000 7f97bb835700 1 -- 172.21.15.107:0/722021208 >> 172.21.15.107:0/722021208 conn(0x7f97b40fbf40 msgr2=0x7f97b40fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.547+0000 7f97bb835700 1 -- 172.21.15.107:0/722021208 shutdown_connections 2024-03-18T00:47:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.547+0000 7f97bb835700 1 -- 172.21.15.107:0/722021208 wait complete. 2024-03-18T00:47:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97bb835700 1 Processor -- start 2024-03-18T00:47:24.551 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97bb835700 1 -- start start 2024-03-18T00:47:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97bb835700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97b41014d0 0x7f97b4105270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97bb835700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 0x7f97b41038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97bb835700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97b4102790 0x7f97b4103e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97bb835700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f97b410cee0 con 0x7f97b4102790 2024-03-18T00:47:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97bb835700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f97b410cd60 con 0x7f97b41014d0 2024-03-18T00:47:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97bb835700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f97b410d060 con 0x7f97b4101e70 2024-03-18T00:47:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97b8dd0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 0x7f97b41038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97b8dd0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 0x7f97b41038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:36688/0 (socket says 172.21.15.107:36688) 2024-03-18T00:47:24.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.548+0000 7f97b8dd0700 1 -- 172.21.15.107:0/2032451288 learned_addr learned my addr 172.21.15.107:0/2032451288 (peer_addr_for_me v2:172.21.15.107:0/0) 2024-03-18T00:47:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97b9dd2700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97b4102790 0x7f97b4103e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97b95d1700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97b41014d0 0x7f97b4105270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97b8dd0700 1 -- 172.21.15.107:0/2032451288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97b41014d0 msgr2=0x7f97b4105270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97b8dd0700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97b41014d0 0x7f97b4105270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97b8dd0700 1 -- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97b4102790 msgr2=0x7f97b4103e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97b8dd0700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97b4102790 0x7f97b4103e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97b8dd0700 1 -- 172.21.15.107:0/2032451288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b41046b0 con 0x7f97b4101e70 2024-03-18T00:47:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97b95d1700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97b41014d0 0x7f97b4105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:47:24.553 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97b8dd0700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 0x7f97b41038c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97b000b530 tx=0x7f97b000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97aa7fc700 1 -- 172.21.15.107:0/2032451288 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0015020 con 0x7f97b4101e70 2024-03-18T00:47:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b41a2850 con 0x7f97b4101e70 2024-03-18T00:47:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97aa7fc700 1 -- 172.21.15.107:0/2032451288 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b0011d10 con 0x7f97b4101e70 2024-03-18T00:47:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97aa7fc700 1 -- 172.21.15.107:0/2032451288 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b0010e50 con 0x7f97b4101e70 2024-03-18T00:47:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.549+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b41a2dc0 con 0x7f97b4101e70 2024-03-18T00:47:24.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.550+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b4068f90 con 0x7f97b4101e70 2024-03-18T00:47:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.551+0000 7f97aa7fc700 1 -- 172.21.15.107:0/2032451288 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f97b0018020 con 0x7f97b4101e70 2024-03-18T00:47:24.555 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.552+0000 7f97aa7fc700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f97a00a2a30 0x7f97a00a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.552+0000 7f97aa7fc700 1 -- 172.21.15.107:0/2032451288 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f97b00cde00 con 0x7f97b4101e70 2024-03-18T00:47:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.552+0000 7f97b95d1700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f97a00a2a30 0x7f97a00a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.552+0000 7f97b95d1700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f97a00a2a30 0x7f97a00a4ef0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f97b4069d00 tx=0x7f97a4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:24.556 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.556+0000 7f97aa7fc700 1 -- 172.21.15.107:0/2032451288 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f97b001f020 con 0x7f97b4101e70 2024-03-18T00:47:24.763 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.763+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l", "target": ["mon-mgr", ""]}) -- 0x7f97b40fe1e0 con 0x7f97a00a2a30 2024-03-18T00:47:24.771 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.771+0000 7f97aa7fc700 1 -- 172.21.15.107:0/2032451288 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f97b40fe1e0 con 0x7f97a00a2a30 2024-03-18T00:47:24.771 INFO:teuthology.orchestra.run.smithi107.stdout:Scheduled mds.all update... 2024-03-18T00:47:24.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f97a00a2a30 msgr2=0x7f97a00a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:24.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f97a00a2a30 0x7f97a00a4ef0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f97b4069d00 tx=0x7f97a4008040 comp rx=0 tx=0).stop 2024-03-18T00:47:24.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 msgr2=0x7f97b41038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:24.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 0x7f97b41038c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f97b000b530 tx=0x7f97b000b8f0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 shutdown_connections 2024-03-18T00:47:24.773 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f97b4102790 0x7f97b4103e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f97b41014d0 0x7f97b4105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f97a00a2a30 0x7f97a00a4ef0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 --2- 172.21.15.107:0/2032451288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f97b4101e70 0x7f97b41038c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 >> 172.21.15.107:0/2032451288 conn(0x7f97b40fbf40 msgr2=0x7f97b40fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 shutdown_connections 2024-03-18T00:47:24.774 INFO:teuthology.orchestra.run.smithi107.stderr:2024-03-18T00:47:24.773+0000 7f97bb835700 1 -- 172.21.15.107:0/2032451288 wait complete. 2024-03-18T00:47:25.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:25 smithi049 ceph-mon[32130]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:25.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:25.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:25 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:47:25.391 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:25 smithi107 ceph-mon[36270]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:25.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:25.392 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:25 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:47:25.428 DEBUG:teuthology.orchestra.run.smithi049:mds.a> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.a.service 2024-03-18T00:47:25.431 DEBUG:teuthology.orchestra.run.smithi049:mds.d> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.d.service 2024-03-18T00:47:25.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:25 smithi106 ceph-mon[35899]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:25.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:25.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:25 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:47:25.474 DEBUG:teuthology.orchestra.run.smithi049:mds.g> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.g.service 2024-03-18T00:47:25.516 DEBUG:teuthology.orchestra.run.smithi049:mds.j> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.j.service 2024-03-18T00:47:25.559 DEBUG:teuthology.orchestra.run.smithi106:mds.b> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.b.service 2024-03-18T00:47:25.562 DEBUG:teuthology.orchestra.run.smithi106:mds.e> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.e.service 2024-03-18T00:47:25.605 DEBUG:teuthology.orchestra.run.smithi106:mds.h> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.h.service 2024-03-18T00:47:25.647 DEBUG:teuthology.orchestra.run.smithi106:mds.k> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.k.service 2024-03-18T00:47:25.691 DEBUG:teuthology.orchestra.run.smithi107:mds.c> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.c.service 2024-03-18T00:47:25.693 DEBUG:teuthology.orchestra.run.smithi107:mds.f> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.f.service 2024-03-18T00:47:25.737 DEBUG:teuthology.orchestra.run.smithi107:mds.i> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.i.service 2024-03-18T00:47:25.782 DEBUG:teuthology.orchestra.run.smithi107:mds.l> sudo journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.l.service 2024-03-18T00:47:25.825 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-18T00:47:25.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:47:25.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.942+0000 7fe57f976640 1 -- 172.21.15.49:0/796185097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101a00 msgr2=0x7fe578108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:25.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.942+0000 7fe57f976640 1 --2- 172.21.15.49:0/796185097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101a00 0x7fe578108320 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe570009ca0 tx=0x7fe57002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:47:25.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 -- 172.21.15.49:0/796185097 shutdown_connections 2024-03-18T00:47:25.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 --2- 172.21.15.49:0/796185097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101a00 0x7fe578108320 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:25.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 --2- 172.21.15.49:0/796185097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe578101040 0x7fe5781014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 --2- 172.21.15.49:0/796185097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5780ffe60 0x7fe578100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 -- 172.21.15.49:0/796185097 >> 172.21.15.49:0/796185097 conn(0x7fe5780fb420 msgr2=0x7fe5780fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 -- 172.21.15.49:0/796185097 shutdown_connections 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 -- 172.21.15.49:0/796185097 wait complete. 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 Processor -- start 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 -- start start 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5780ffe60 0x7fe5781969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101040 0x7fe578196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe578101a00 0x7fe57819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe57810c3d0 con 0x7fe578101040 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe57810c250 con 0x7fe5780ffe60 2024-03-18T00:47:25.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57f976640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe57810c550 con 0x7fe578101a00 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57cf11640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101040 0x7fe578196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57cf11640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101040 0x7fe578196f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56886/0 (socket says 172.21.15.49:56886) 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.943+0000 7fe57cf11640 1 -- 172.21.15.49:0/2602614580 learned_addr learned my addr 172.21.15.49:0/2602614580 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57d712640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5780ffe60 0x7fe5781969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57df13640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe578101a00 0x7fe57819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57cf11640 1 -- 172.21.15.49:0/2602614580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe578101a00 msgr2=0x7fe57819df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57cf11640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe578101a00 0x7fe57819df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57cf11640 1 -- 172.21.15.49:0/2602614580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5780ffe60 msgr2=0x7fe5781969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57cf11640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5780ffe60 0x7fe5781969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57cf11640 1 -- 172.21.15.49:0/2602614580 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe57819e600 con 0x7fe578101040 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57df13640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe578101a00 0x7fe57819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57d712640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5780ffe60 0x7fe5781969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57cf11640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101040 0x7fe578196f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe57400e9f0 tx=0x7fe57400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:25.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe56e7fc640 1 -- 172.21.15.49:0/2602614580 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57400cd80 con 0x7fe578101040 2024-03-18T00:47:25.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe56e7fc640 1 -- 172.21.15.49:0/2602614580 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe574005500 con 0x7fe578101040 2024-03-18T00:47:25.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe57819e8f0 con 0x7fe578101040 2024-03-18T00:47:25.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe56e7fc640 1 -- 172.21.15.49:0/2602614580 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe574010690 con 0x7fe578101040 2024-03-18T00:47:25.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.944+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57819edf0 con 0x7fe578101040 2024-03-18T00:47:25.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.945+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5781071a0 con 0x7fe578101040 2024-03-18T00:47:25.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.946+0000 7fe56e7fc640 1 -- 172.21.15.49:0/2602614580 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe5740046c0 con 0x7fe578101040 2024-03-18T00:47:25.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.947+0000 7fe56e7fc640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe5640a2800 0x7fe5640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:25.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.947+0000 7fe57d712640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe5640a2800 0x7fe5640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:25.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.947+0000 7fe56e7fc640 1 -- 172.21.15.49:0/2602614580 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe5740cec40 con 0x7fe578101040 2024-03-18T00:47:25.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.947+0000 7fe57d712640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe5640a2800 0x7fe5640a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe578100bb0 tx=0x7fe568005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:25.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:25.950+0000 7fe56e7fc640 1 -- 172.21.15.49:0/2602614580 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe574053740 con 0x7fe578101040 2024-03-18T00:47:26.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:26 smithi049 ceph-mon[32130]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:47:26.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:26 smithi049 ceph-mon[32130]: Saving service mds.all spec with placement smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l;count:12 2024-03-18T00:47:26.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:26.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:47:26.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:26.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:26.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:26.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:26.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.140+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe57802d1b0 con 0x7fe578101040 2024-03-18T00:47:26.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.141+0000 7fe56e7fc640 1 -- 172.21.15.49:0/2602614580 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18443 (secure 0 0 0) 0x7fe57409c180 con 0x7fe578101040 2024-03-18T00:47:26.141 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:26.142 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":82,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:47:16.556495+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe5640a2800 msgr2=0x7fe5640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe5640a2800 0x7fe5640a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe578100bb0 tx=0x7fe568005f90 comp rx=0 tx=0).stop 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101040 msgr2=0x7fe578196f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101040 0x7fe578196f20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe57400e9f0 tx=0x7fe57400edf0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 shutdown_connections 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe5640a2800 0x7fe5640a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe578101a00 0x7fe57819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe578101040 0x7fe578196f20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 --2- 172.21.15.49:0/2602614580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5780ffe60 0x7fe5781969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 >> 172.21.15.49:0/2602614580 conn(0x7fe5780fb420 msgr2=0x7fe578104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 shutdown_connections 2024-03-18T00:47:26.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.144+0000 7fe57f976640 1 -- 172.21.15.49:0/2602614580 wait complete. 2024-03-18T00:47:26.155 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:47:26.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:47:26.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:26 smithi107 ceph-mon[36270]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:47:26.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:26 smithi107 ceph-mon[36270]: Saving service mds.all spec with placement smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l;count:12 2024-03-18T00:47:26.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:26.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:47:26.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:26.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:26.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:26.246 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:26.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.267+0000 7fdf9f574640 1 -- 172.21.15.49:0/340692395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981017f0 msgr2=0x7fdf98101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.267+0000 7fdf9f574640 1 --2- 172.21.15.49:0/340692395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981017f0 0x7fdf98101c10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009900 tx=0x7fdf9402bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.267+0000 7fdf9f574640 1 -- 172.21.15.49:0/340692395 shutdown_connections 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.267+0000 7fdf9f574640 1 --2- 172.21.15.49:0/340692395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf981033a0 0x7fdf98109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.267+0000 7fdf9f574640 1 --2- 172.21.15.49:0/340692395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf981029e0 0x7fdf98102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.267+0000 7fdf9f574640 1 --2- 172.21.15.49:0/340692395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981017f0 0x7fdf98101c10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.267+0000 7fdf9f574640 1 -- 172.21.15.49:0/340692395 >> 172.21.15.49:0/340692395 conn(0x7fdf980fcd60 msgr2=0x7fdf980ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.267+0000 7fdf9f574640 1 -- 172.21.15.49:0/340692395 shutdown_connections 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.267+0000 7fdf9f574640 1 -- 172.21.15.49:0/340692395 wait complete. 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9f574640 1 Processor -- start 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9f574640 1 -- start start 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9f574640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf981017f0 0x7fdf98197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9f574640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981029e0 0x7fdf981980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:26.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9f574640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf981033a0 0x7fdf9819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9f574640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9810d460 con 0x7fdf981029e0 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9f574640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9810d2e0 con 0x7fdf981017f0 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9f574640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9810d5e0 con 0x7fdf981033a0 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9cb0f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981029e0 0x7fdf981980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9cb0f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981029e0 0x7fdf981980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56906/0 (socket says 172.21.15.49:56906) 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9cb0f640 1 -- 172.21.15.49:0/2920020698 learned_addr learned my addr 172.21.15.49:0/2920020698 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9db11640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf981033a0 0x7fdf9819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9d310640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf981017f0 0x7fdf98197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9cb0f640 1 -- 172.21.15.49:0/2920020698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf981033a0 msgr2=0x7fdf9819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9cb0f640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf981033a0 0x7fdf9819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9cb0f640 1 -- 172.21.15.49:0/2920020698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf981017f0 msgr2=0x7fdf98197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9cb0f640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf981017f0 0x7fdf98197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9cb0f640 1 -- 172.21.15.49:0/2920020698 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9819f780 con 0x7fdf981029e0 2024-03-18T00:47:26.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.268+0000 7fdf9d310640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf981017f0 0x7fdf98197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:47:26.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.269+0000 7fdf9cb0f640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981029e0 0x7fdf981980a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800e9f0 tx=0x7fdf8800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:26.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.269+0000 7fdf8e7fc640 1 -- 172.21.15.49:0/2920020698 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8800cd80 con 0x7fdf981029e0 2024-03-18T00:47:26.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.269+0000 7fdf8e7fc640 1 -- 172.21.15.49:0/2920020698 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf88005500 con 0x7fdf981029e0 2024-03-18T00:47:26.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.269+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9819fa70 con 0x7fdf981029e0 2024-03-18T00:47:26.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.269+0000 7fdf8e7fc640 1 -- 172.21.15.49:0/2920020698 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf88010690 con 0x7fdf981029e0 2024-03-18T00:47:26.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.269+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9819ff70 con 0x7fdf981029e0 2024-03-18T00:47:26.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.270+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf98108b80 con 0x7fdf981029e0 2024-03-18T00:47:26.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.271+0000 7fdf8e7fc640 1 -- 172.21.15.49:0/2920020698 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdf880046c0 con 0x7fdf981029e0 2024-03-18T00:47:26.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.272+0000 7fdf8e7fc640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdf840a2800 0x7fdf840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:26.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.272+0000 7fdf9d310640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdf840a2800 0x7fdf840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:26.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.272+0000 7fdf8e7fc640 1 -- 172.21.15.49:0/2920020698 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fdf880cea60 con 0x7fdf981029e0 2024-03-18T00:47:26.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.272+0000 7fdf9d310640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdf840a2800 0x7fdf840a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdf9402c7b0 tx=0x7fdf94005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:26.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.276+0000 7fdf8e7fc640 1 -- 172.21.15.49:0/2920020698 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf8809bfa0 con 0x7fdf981029e0 2024-03-18T00:47:26.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:26 smithi106 ceph-mon[35899]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:47:26.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:26 smithi106 ceph-mon[35899]: Saving service mds.all spec with placement smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l;count:12 2024-03-18T00:47:26.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:26.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:47:26.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:26.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:26.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:26.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:26.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.459+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdf9802d3c0 con 0x7fdf981029e0 2024-03-18T00:47:26.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.460+0000 7fdf8e7fc640 1 -- 172.21.15.49:0/2920020698 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fdf8809b8a0 con 0x7fdf981029e0 2024-03-18T00:47:26.460 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdf840a2800 msgr2=0x7fdf840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdf840a2800 0x7fdf840a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdf9402c7b0 tx=0x7fdf94005fd0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981029e0 msgr2=0x7fdf981980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981029e0 0x7fdf981980a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800e9f0 tx=0x7fdf8800edf0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 shutdown_connections 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdf840a2800 0x7fdf840a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdf981033a0 0x7fdf9819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdf981029e0 0x7fdf981980a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 --2- 172.21.15.49:0/2920020698 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdf981017f0 0x7fdf98197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 >> 172.21.15.49:0/2920020698 conn(0x7fdf980fcd60 msgr2=0x7fdf98105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 shutdown_connections 2024-03-18T00:47:26.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.462+0000 7fdf9f574640 1 -- 172.21.15.49:0/2920020698 wait complete. 2024-03-18T00:47:26.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:26.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.581+0000 7f829843d640 1 -- 172.21.15.49:0/2359828218 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8290101810 msgr2=0x7f8290101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.581+0000 7f829843d640 1 --2- 172.21.15.49:0/2359828218 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8290101810 0x7f8290101c30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f828c009900 tx=0x7f828c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.581+0000 7f829843d640 1 -- 172.21.15.49:0/2359828218 shutdown_connections 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.581+0000 7f829843d640 1 --2- 172.21.15.49:0/2359828218 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8290102a00 0x7f8290102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.581+0000 7f829843d640 1 --2- 172.21.15.49:0/2359828218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82901033c0 0x7f8290109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.581+0000 7f829843d640 1 --2- 172.21.15.49:0/2359828218 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8290101810 0x7f8290101c30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.581+0000 7f829843d640 1 -- 172.21.15.49:0/2359828218 >> 172.21.15.49:0/2359828218 conn(0x7f82900fcdc0 msgr2=0x7f82900ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.581+0000 7f829843d640 1 -- 172.21.15.49:0/2359828218 shutdown_connections 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.581+0000 7f829843d640 1 -- 172.21.15.49:0/2359828218 wait complete. 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f829843d640 1 Processor -- start 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f829843d640 1 -- start start 2024-03-18T00:47:26.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f829843d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8290101810 0x7f8290197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f829843d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8290102a00 0x7f8290198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f829843d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82901033c0 0x7f829019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f829843d640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f829010d460 con 0x7f82901033c0 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f829843d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f829010d2e0 con 0x7f8290101810 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f829843d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f829010d5e0 con 0x7f8290102a00 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f82969da640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82901033c0 0x7f829019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f82969da640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82901033c0 0x7f829019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56910/0 (socket says 172.21.15.49:56910) 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f82969da640 1 -- 172.21.15.49:0/1449900570 learned_addr learned my addr 172.21.15.49:0/1449900570 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.582+0000 7f82961d9640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8290101810 0x7f8290197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82959d8640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8290102a00 0x7f8290198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82969da640 1 -- 172.21.15.49:0/1449900570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8290102a00 msgr2=0x7f8290198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82969da640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8290102a00 0x7f8290198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82969da640 1 -- 172.21.15.49:0/1449900570 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8290101810 msgr2=0x7f8290197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82969da640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8290101810 0x7f8290197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82969da640 1 -- 172.21.15.49:0/1449900570 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829019f720 con 0x7f82901033c0 2024-03-18T00:47:26.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82969da640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82901033c0 0x7f829019f0b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f828800eaa0 tx=0x7f828800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:26.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82877fe640 1 -- 172.21.15.49:0/1449900570 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828800cdb0 con 0x7f82901033c0 2024-03-18T00:47:26.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829019fa10 con 0x7f82901033c0 2024-03-18T00:47:26.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82877fe640 1 -- 172.21.15.49:0/1449900570 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8288005020 con 0x7f82901033c0 2024-03-18T00:47:26.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f82877fe640 1 -- 172.21.15.49:0/1449900570 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82880116b0 con 0x7f82901033c0 2024-03-18T00:47:26.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.583+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f829019ff10 con 0x7f82901033c0 2024-03-18T00:47:26.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.584+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8290108bb0 con 0x7f82901033c0 2024-03-18T00:47:26.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.585+0000 7f82877fe640 1 -- 172.21.15.49:0/1449900570 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8288005560 con 0x7f82901033c0 2024-03-18T00:47:26.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.586+0000 7f82877fe640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f827c0a2800 0x7f827c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:26.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.586+0000 7f82961d9640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f827c0a2800 0x7f827c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:26.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.586+0000 7f82877fe640 1 -- 172.21.15.49:0/1449900570 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f82880ce880 con 0x7f82901033c0 2024-03-18T00:47:26.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.586+0000 7f82961d9640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f827c0a2800 0x7f827c0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f828c02c7b0 tx=0x7f828c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:26.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.590+0000 7f82877fe640 1 -- 172.21.15.49:0/1449900570 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8288016030 con 0x7f82901033c0 2024-03-18T00:47:26.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.812+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f829002d1b0 con 0x7f82901033c0 2024-03-18T00:47:26.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.812+0000 7f82877fe640 1 -- 172.21.15.49:0/1449900570 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f828809bdc0 con 0x7f82901033c0 2024-03-18T00:47:26.812 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:26.812 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T00:47:26.812 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 1 2024-03-18T00:47:26.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f827c0a2800 msgr2=0x7f827c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f827c0a2800 0x7f827c0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f828c02c7b0 tx=0x7f828c005fd0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82901033c0 msgr2=0x7f829019f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:26.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82901033c0 0x7f829019f0b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f828800eaa0 tx=0x7f828800ef60 comp rx=0 tx=0).stop 2024-03-18T00:47:26.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 shutdown_connections 2024-03-18T00:47:26.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f827c0a2800 0x7f827c0a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82901033c0 0x7f829019f0b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8290102a00 0x7f8290198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 --2- 172.21.15.49:0/1449900570 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8290101810 0x7f8290197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:26.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 >> 172.21.15.49:0/1449900570 conn(0x7f82900fcdc0 msgr2=0x7f8290105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:26.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 shutdown_connections 2024-03-18T00:47:26.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:26.814+0000 7f829843d640 1 -- 172.21.15.49:0/1449900570 wait complete. 2024-03-18T00:47:27.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:27 smithi049 ceph-mon[32130]: Deploying daemon mds.l on smithi107 2024-03-18T00:47:27.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:27 smithi049 ceph-mon[32130]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:27.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:27 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2602614580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:47:27.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:27 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2920020698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:47:27.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:27 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1449900570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:27.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:27 smithi107 ceph-mon[36270]: Deploying daemon mds.l on smithi107 2024-03-18T00:47:27.257 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:27 smithi107 ceph-mon[36270]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:27.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:27 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2602614580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:47:27.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:27 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2920020698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:47:27.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:27 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1449900570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:27.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:27 smithi106 ceph-mon[35899]: Deploying daemon mds.l on smithi107 2024-03-18T00:47:27.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:27 smithi106 ceph-mon[35899]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:27.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:27 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2602614580' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:47:27.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:27 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2920020698' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:47:27.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:27 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1449900570' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:27.655 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 00:47:27 smithi107 systemd[1]: Starting Ceph mds.l for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:27.656 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 00:47:27 smithi107 podman[66461]: 2024-03-18 00:47:27.558249218 +0000 UTC m=+0.017830063 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:27.913 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 00:47:27 smithi107 podman[66461]: 2024-03-18 00:47:27.67803606 +0000 UTC m=+0.137616889 container create 88fa35199912406a4d8442a03c45e49967cac5650384b31bb71e3b5dd84ac945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-18T00:47:27.913 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 00:47:27 smithi107 podman[66461]: 2024-03-18 00:47:27.914163313 +0000 UTC m=+0.373744156 container init 88fa35199912406a4d8442a03c45e49967cac5650384b31bb71e3b5dd84ac945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-18T00:47:28.180 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 00:47:27 smithi107 podman[66461]: 2024-03-18 00:47:27.916754368 +0000 UTC m=+0.376335190 container start 88fa35199912406a4d8442a03c45e49967cac5650384b31bb71e3b5dd84ac945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T00:47:28.180 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 00:47:27 smithi107 ceph-mds[66480]: -- 172.21.15.107:0/3246091638 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558d258d9a40 con 0x558d258ad000 2024-03-18T00:47:28.180 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 00:47:27 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l[66476]: starting mds.l at 2024-03-18T00:47:28.180 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 00:47:28 smithi107 bash[66461]: 88fa35199912406a4d8442a03c45e49967cac5650384b31bb71e3b5dd84ac945 2024-03-18T00:47:28.180 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 00:47:28 smithi107 systemd[1]: Started Ceph mds.l for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:28.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:28.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.946+0000 7f186f8ea640 1 -- 172.21.15.49:0/3693066649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 msgr2=0x7f1868100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:28.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.946+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3693066649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 0x7f1868100b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1858009900 tx=0x7f185802bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:28.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.946+0000 7f186f8ea640 1 -- 172.21.15.49:0/3693066649 shutdown_connections 2024-03-18T00:47:28.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.946+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3693066649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18681022b0 0x7f1868108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:28.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.946+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3693066649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18681018f0 0x7f1868101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:28.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.946+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3693066649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 0x7f1868100b20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.946+0000 7f186f8ea640 1 -- 172.21.15.49:0/3693066649 >> 172.21.15.49:0/3693066649 conn(0x7f18680fbc70 msgr2=0x7f18680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.946+0000 7f186f8ea640 1 -- 172.21.15.49:0/3693066649 shutdown_connections 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.946+0000 7f186f8ea640 1 -- 172.21.15.49:0/3693066649 wait complete. 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186f8ea640 1 Processor -- start 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186f8ea640 1 -- start start 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186f8ea640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 0x7f1868197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186f8ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18681018f0 0x7f1868198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186f8ea640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18681022b0 0x7f186819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186f8ea640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f186810d4a0 con 0x7f1868100700 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186f8ea640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f186810d320 con 0x7f18681022b0 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186f8ea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f186810d620 con 0x7f18681018f0 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186d686640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 0x7f1868197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186d686640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 0x7f1868197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57314/0 (socket says 172.21.15.49:57314) 2024-03-18T00:47:28.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186d686640 1 -- 172.21.15.49:0/3823307771 learned_addr learned my addr 172.21.15.49:0/3823307771 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186de87640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18681022b0 0x7f186819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186ce85640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18681018f0 0x7f1868198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186d686640 1 -- 172.21.15.49:0/3823307771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18681018f0 msgr2=0x7f1868198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186d686640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18681018f0 0x7f1868198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186d686640 1 -- 172.21.15.49:0/3823307771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18681022b0 msgr2=0x7f186819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186d686640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18681022b0 0x7f186819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186d686640 1 -- 172.21.15.49:0/3823307771 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f186819f750 con 0x7f1868100700 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186de87640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18681022b0 0x7f186819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.947+0000 7f186d686640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 0x7f1868197b30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1858000c00 tx=0x7f1858005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.948+0000 7f185e7fc640 1 -- 172.21.15.49:0/3823307771 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858041070 con 0x7f1868100700 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.948+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186819f9e0 con 0x7f1868100700 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.948+0000 7f185e7fc640 1 -- 172.21.15.49:0/3823307771 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18580044d0 con 0x7f1868100700 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.948+0000 7f185e7fc640 1 -- 172.21.15.49:0/3823307771 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858034440 con 0x7f1868100700 2024-03-18T00:47:28.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.948+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f186819fed0 con 0x7f1868100700 2024-03-18T00:47:28.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.949+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1868107aa0 con 0x7f1868100700 2024-03-18T00:47:28.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.950+0000 7f185e7fc640 1 -- 172.21.15.49:0/3823307771 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1858044070 con 0x7f1868100700 2024-03-18T00:47:28.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.951+0000 7f185e7fc640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a2800 0x7f18540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:28.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.951+0000 7f186ce85640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a2800 0x7f18540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:28.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.951+0000 7f185e7fc640 1 -- 172.21.15.49:0/3823307771 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f18580f0210 con 0x7f1868100700 2024-03-18T00:47:28.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.951+0000 7f186ce85640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a2800 0x7f18540a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1864005fd0 tx=0x7f1864005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:28.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:28.955+0000 7f185e7fc640 1 -- 172.21.15.49:0/3823307771 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f18580bd750 con 0x7f1868100700 2024-03-18T00:47:29.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:29.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6832/1442063907,v1:172.21.15.107:6833/1442063907] up:boot 2024-03-18T00:47:29.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: fsmap 1 up:standby 2024-03-18T00:47:29.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T00:47:29.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:29.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:29.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:29.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:29.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:29.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:29.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:28 smithi049 ceph-mon[32130]: Deploying daemon mds.k on smithi106 2024-03-18T00:47:29.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.177+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f186802d1b0 con 0x7f1868100700 2024-03-18T00:47:29.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.177+0000 7f185e7fc640 1 -- 172.21.15.49:0/3823307771 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f18580bd050 con 0x7f1868100700 2024-03-18T00:47:29.177 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:29.177 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T00:47:29.178 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 2 2024-03-18T00:47:29.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a2800 msgr2=0x7f18540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:29.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a2800 0x7f18540a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1864005fd0 tx=0x7f1864005850 comp rx=0 tx=0).stop 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 msgr2=0x7f1868197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 0x7f1868197b30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1858000c00 tx=0x7f1858005d50 comp rx=0 tx=0).stop 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 shutdown_connections 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a2800 0x7f18540a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18681022b0 0x7f186819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18681018f0 0x7f1868198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 --2- 172.21.15.49:0/3823307771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868100700 0x7f1868197b30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 >> 172.21.15.49:0/3823307771 conn(0x7f18680fbc70 msgr2=0x7f1868104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 shutdown_connections 2024-03-18T00:47:29.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:29.179+0000 7f186f8ea640 1 -- 172.21.15.49:0/3823307771 wait complete. 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6832/1442063907,v1:172.21.15.107:6833/1442063907] up:boot 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: fsmap 1 up:standby 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:29.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:28 smithi106 ceph-mon[35899]: Deploying daemon mds.k on smithi106 2024-03-18T00:47:29.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:29.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6832/1442063907,v1:172.21.15.107:6833/1442063907] up:boot 2024-03-18T00:47:29.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: fsmap 1 up:standby 2024-03-18T00:47:29.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T00:47:29.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:29.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:29.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:29.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:29.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:29.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:29.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:28 smithi107 ceph-mon[36270]: Deploying daemon mds.k on smithi106 2024-03-18T00:47:30.170 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:29 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3823307771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:30.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:29 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3823307771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:30.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:29 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3823307771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:30.445 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 00:47:30 smithi106 systemd[1]: Starting Ceph mds.k for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:30.852 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 00:47:30 smithi106 podman[65891]: 2024-03-18 00:47:30.469101394 +0000 UTC m=+0.017654114 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:30.852 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 00:47:30 smithi106 podman[65891]: 2024-03-18 00:47:30.591221125 +0000 UTC m=+0.139773879 container create 6a9c5dec50936d490c57ac3a4897fee359f753998521d0bc11577d98f2e069d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:47:30.852 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 00:47:30 smithi106 podman[65891]: 2024-03-18 00:47:30.8524137 +0000 UTC m=+0.400966428 container init 6a9c5dec50936d490c57ac3a4897fee359f753998521d0bc11577d98f2e069d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-18T00:47:31.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:31.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:30 smithi106 ceph-mon[35899]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:31.196 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 00:47:30 smithi106 podman[65891]: 2024-03-18 00:47:30.855123543 +0000 UTC m=+0.403676261 container start 6a9c5dec50936d490c57ac3a4897fee359f753998521d0bc11577d98f2e069d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-18T00:47:31.196 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 00:47:30 smithi106 ceph-mds[65911]: -- 172.21.15.106:0/1520107875 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555787883a40 con 0x555787856c00 2024-03-18T00:47:31.196 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 00:47:30 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k[65907]: starting mds.k at 2024-03-18T00:47:31.196 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 00:47:30 smithi106 bash[65891]: 6a9c5dec50936d490c57ac3a4897fee359f753998521d0bc11577d98f2e069d7 2024-03-18T00:47:31.196 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 00:47:30 smithi106 systemd[1]: Started Ceph mds.k for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:31.219 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:30 smithi049 ceph-mon[32130]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:31.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.302+0000 7f0a8bd89640 1 -- 172.21.15.49:0/1816987269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841022b0 msgr2=0x7f0a84108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:31.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.302+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/1816987269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841022b0 0x7f0a84108b40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c009ca0 tx=0x7f0a7c02c0f0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.302+0000 7f0a8bd89640 1 -- 172.21.15.49:0/1816987269 shutdown_connections 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.302+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/1816987269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a841018f0 0x7f0a84101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.302+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/1816987269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841022b0 0x7f0a84108b40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.302+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/1816987269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a84100700 0x7f0a84100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.302+0000 7f0a8bd89640 1 -- 172.21.15.49:0/1816987269 >> 172.21.15.49:0/1816987269 conn(0x7f0a840fbc70 msgr2=0x7f0a840fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.302+0000 7f0a8bd89640 1 -- 172.21.15.49:0/1816987269 shutdown_connections 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.302+0000 7f0a8bd89640 1 -- 172.21.15.49:0/1816987269 wait complete. 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a8bd89640 1 Processor -- start 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a8bd89640 1 -- start start 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a8bd89640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a84100700 0x7f0a840694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a8bd89640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841018f0 0x7f0a8419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:31.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a8bd89640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a841022b0 0x7f0a8419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a8bd89640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8410c380 con 0x7f0a841018f0 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a8bd89640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8410c200 con 0x7f0a84100700 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a8bd89640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8410c500 con 0x7f0a841022b0 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a89324640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841018f0 0x7f0a8419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a89324640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841018f0 0x7f0a8419c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57322/0 (socket says 172.21.15.49:57322) 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a89324640 1 -- 172.21.15.49:0/512269345 learned_addr learned my addr 172.21.15.49:0/512269345 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a8a326640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a841022b0 0x7f0a8419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a89b25640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a84100700 0x7f0a840694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a89324640 1 -- 172.21.15.49:0/512269345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a841022b0 msgr2=0x7f0a8419ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a89324640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a841022b0 0x7f0a8419ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a89324640 1 -- 172.21.15.49:0/512269345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a84100700 msgr2=0x7f0a840694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a89324640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a84100700 0x7f0a840694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.303+0000 7f0a89324640 1 -- 172.21.15.49:0/512269345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8419f770 con 0x7f0a841018f0 2024-03-18T00:47:31.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.304+0000 7f0a8a326640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a841022b0 0x7f0a8419ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:47:31.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.304+0000 7f0a89b25640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a84100700 0x7f0a840694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:47:31.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.304+0000 7f0a89324640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841018f0 0x7f0a8419c840 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000e9f0 tx=0x7f0a8000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:31.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.304+0000 7f0a7affd640 1 -- 172.21.15.49:0/512269345 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8000cd80 con 0x7f0a841018f0 2024-03-18T00:47:31.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.304+0000 7f0a7affd640 1 -- 172.21.15.49:0/512269345 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a80010910 con 0x7f0a841018f0 2024-03-18T00:47:31.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.304+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8419fa00 con 0x7f0a841018f0 2024-03-18T00:47:31.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.304+0000 7f0a7affd640 1 -- 172.21.15.49:0/512269345 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a80018990 con 0x7f0a841018f0 2024-03-18T00:47:31.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.304+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8419ffa0 con 0x7f0a841018f0 2024-03-18T00:47:31.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.305+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a84107a20 con 0x7f0a841018f0 2024-03-18T00:47:31.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.306+0000 7f0a7affd640 1 -- 172.21.15.49:0/512269345 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0a80010ac0 con 0x7f0a841018f0 2024-03-18T00:47:31.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.307+0000 7f0a7affd640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a700a2880 0x7f0a700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:31.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.307+0000 7f0a89b25640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a700a2880 0x7f0a700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:31.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.307+0000 7f0a7affd640 1 -- 172.21.15.49:0/512269345 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0a800ce9d0 con 0x7f0a841018f0 2024-03-18T00:47:31.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.307+0000 7f0a89b25640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a700a2880 0x7f0a700a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0a84101750 tx=0x7f0a74005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:31.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.311+0000 7f0a7affd640 1 -- 172.21.15.49:0/512269345 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a8009bf10 con 0x7f0a841018f0 2024-03-18T00:47:31.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:30 smithi107 ceph-mon[36270]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:31.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.540+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a840623e0 con 0x7f0a841018f0 2024-03-18T00:47:31.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.541+0000 7f0a7affd640 1 -- 172.21.15.49:0/512269345 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f0a8009b810 con 0x7f0a841018f0 2024-03-18T00:47:31.541 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:31.541 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T00:47:31.541 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 3 2024-03-18T00:47:31.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a700a2880 msgr2=0x7f0a700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:31.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a700a2880 0x7f0a700a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0a84101750 tx=0x7f0a74005f90 comp rx=0 tx=0).stop 2024-03-18T00:47:31.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841018f0 msgr2=0x7f0a8419c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:31.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841018f0 0x7f0a8419c840 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000e9f0 tx=0x7f0a8000eeb0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 shutdown_connections 2024-03-18T00:47:31.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a700a2880 0x7f0a700a4d40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a841022b0 0x7f0a8419ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a841018f0 0x7f0a8419c840 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 --2- 172.21.15.49:0/512269345 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a84100700 0x7f0a840694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:31.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 >> 172.21.15.49:0/512269345 conn(0x7f0a840fbc70 msgr2=0x7f0a84104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:31.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 shutdown_connections 2024-03-18T00:47:31.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:31.543+0000 7f0a8bd89640 1 -- 172.21.15.49:0/512269345 wait complete. 2024-03-18T00:47:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6832/701158707,v1:172.21.15.106:6833/701158707] up:boot 2024-03-18T00:47:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: fsmap 2 up:standby 2024-03-18T00:47:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T00:47:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:32.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: Deploying daemon mds.j on smithi049 2024-03-18T00:47:32.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:31 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/512269345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:32.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6832/701158707,v1:172.21.15.106:6833/701158707] up:boot 2024-03-18T00:47:32.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: fsmap 2 up:standby 2024-03-18T00:47:32.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T00:47:32.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: Deploying daemon mds.j on smithi049 2024-03-18T00:47:32.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:31 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/512269345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6832/701158707,v1:172.21.15.106:6833/701158707] up:boot 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: fsmap 2 up:standby 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:32.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: Deploying daemon mds.j on smithi049 2024-03-18T00:47:32.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:31 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/512269345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:33.176 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:33 smithi049 ceph-mon[32130]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:33.268 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:33 smithi106 ceph-mon[35899]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:33.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:33 smithi107 ceph-mon[36270]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:33.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:33.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.669+0000 7fbb58ecc640 1 -- 172.21.15.49:0/464758414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb54104c30 msgr2=0x7fbb54107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:33.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.669+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/464758414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb54104c30 0x7fbb54107020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbb48009900 tx=0x7fbb4802bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:33.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.669+0000 7fbb58ecc640 1 -- 172.21.15.49:0/464758414 shutdown_connections 2024-03-18T00:47:33.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.669+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/464758414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb54104c30 0x7fbb54107020 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:33.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.669+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/464758414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb54107560 0x7fbb541099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.669+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/464758414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb54102300 0x7fbb541046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.669+0000 7fbb58ecc640 1 -- 172.21.15.49:0/464758414 >> 172.21.15.49:0/464758414 conn(0x7fbb540fbd10 msgr2=0x7fbb540fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.669+0000 7fbb58ecc640 1 -- 172.21.15.49:0/464758414 shutdown_connections 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.669+0000 7fbb58ecc640 1 -- 172.21.15.49:0/464758414 wait complete. 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb58ecc640 1 Processor -- start 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb58ecc640 1 -- start start 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb58ecc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb54102300 0x7fbb54197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb58ecc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb54104c30 0x7fbb54198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb58ecc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb54107560 0x7fbb5419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb58ecc640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5410d490 con 0x7fbb54107560 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb58ecc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5410d310 con 0x7fbb54104c30 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb58ecc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5410d610 con 0x7fbb54102300 2024-03-18T00:47:33.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb52d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb54107560 0x7fbb5419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb52d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb54107560 0x7fbb5419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57340/0 (socket says 172.21.15.49:57340) 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb52d9d640 1 -- 172.21.15.49:0/2905276390 learned_addr learned my addr 172.21.15.49:0/2905276390 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb51d9b640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb54104c30 0x7fbb54198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.670+0000 7fbb5259c640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb54102300 0x7fbb54197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb52d9d640 1 -- 172.21.15.49:0/2905276390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb54102300 msgr2=0x7fbb54197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb52d9d640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb54102300 0x7fbb54197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb52d9d640 1 -- 172.21.15.49:0/2905276390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb54104c30 msgr2=0x7fbb54198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb52d9d640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb54104c30 0x7fbb54198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb52d9d640 1 -- 172.21.15.49:0/2905276390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5419f720 con 0x7fbb54107560 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb5259c640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb54102300 0x7fbb54197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb52d9d640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb54107560 0x7fbb5419f0b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00d750 tx=0x7fbb4c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb437fe640 1 -- 172.21.15.49:0/2905276390 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c017070 con 0x7fbb54107560 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5419fa10 con 0x7fbb54107560 2024-03-18T00:47:33.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb5419ff80 con 0x7fbb54107560 2024-03-18T00:47:33.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb437fe640 1 -- 172.21.15.49:0/2905276390 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb4c005500 con 0x7fbb54107560 2024-03-18T00:47:33.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.671+0000 7fbb437fe640 1 -- 172.21.15.49:0/2905276390 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c020060 con 0x7fbb54107560 2024-03-18T00:47:33.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.672+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb54068a20 con 0x7fbb54107560 2024-03-18T00:47:33.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.673+0000 7fbb437fe640 1 -- 172.21.15.49:0/2905276390 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fbb4c009c60 con 0x7fbb54107560 2024-03-18T00:47:33.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.674+0000 7fbb437fe640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:33.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.674+0000 7fbb5259c640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:33.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.674+0000 7fbb437fe640 1 -- 172.21.15.49:0/2905276390 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbb4c0cdfe0 con 0x7fbb54107560 2024-03-18T00:47:33.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.674+0000 7fbb5259c640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbb44005fd0 tx=0x7fbb44009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:33.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.678+0000 7fbb437fe640 1 -- 172.21.15.49:0/2905276390 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbb4c09b520 con 0x7fbb54107560 2024-03-18T00:47:33.689 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 00:47:33 smithi049 podman[72170]: 2024-03-18 00:47:33.689752716 +0000 UTC m=+0.151409950 container create ae2cab6fbc2187453bcf57976df56d53555b8f0aa2ea996c86c0a6d2fd5c9efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-j, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:47:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.903+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb5402d3e0 con 0x7fbb54107560 2024-03-18T00:47:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.903+0000 7fbb437fe640 1 -- 172.21.15.49:0/2905276390 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7fbb4c09ae20 con 0x7fbb54107560 2024-03-18T00:47:33.904 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:33.904 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T00:47:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 3 2024-03-18T00:47:33.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbb3c0a2800 msgr2=0x7fbb3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:33.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbb44005fd0 tx=0x7fbb44009500 comp rx=0 tx=0).stop 2024-03-18T00:47:33.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb54107560 msgr2=0x7fbb5419f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:33.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb54107560 0x7fbb5419f0b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00d750 tx=0x7fbb4c00dc10 comp rx=0 tx=0).stop 2024-03-18T00:47:33.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 shutdown_connections 2024-03-18T00:47:33.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbb3c0a2800 0x7fbb3c0a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:33.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbb54104c30 0x7fbb54198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:33.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbb54107560 0x7fbb5419f0b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:33.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 --2- 172.21.15.49:0/2905276390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbb54102300 0x7fbb54197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:33.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 >> 172.21.15.49:0/2905276390 conn(0x7fbb540fbd10 msgr2=0x7fbb54102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:33.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 shutdown_connections 2024-03-18T00:47:33.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:33.905+0000 7fbb58ecc640 1 -- 172.21.15.49:0/2905276390 wait complete. 2024-03-18T00:47:33.959 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 00:47:33 smithi049 podman[72170]: 2024-03-18 00:47:33.934242356 +0000 UTC m=+0.395899598 container init ae2cab6fbc2187453bcf57976df56d53555b8f0aa2ea996c86c0a6d2fd5c9efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-18T00:47:33.959 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 00:47:33 smithi049 podman[72170]: 2024-03-18 00:47:33.93693616 +0000 UTC m=+0.398593398 container start ae2cab6fbc2187453bcf57976df56d53555b8f0aa2ea996c86c0a6d2fd5c9efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-18T00:47:34.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:34 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2905276390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:34.375 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 00:47:33 smithi049 ceph-mds[72230]: -- 172.21.15.49:0/2442378289 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563289a25a40 con 0x5632899f8800 2024-03-18T00:47:34.375 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 00:47:33 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-j[72224]: starting mds.j at 2024-03-18T00:47:34.375 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 00:47:34 smithi049 bash[72170]: ae2cab6fbc2187453bcf57976df56d53555b8f0aa2ea996c86c0a6d2fd5c9efb 2024-03-18T00:47:34.375 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 00:47:34 smithi049 systemd[1]: Started Ceph mds.j for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:34.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:34 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2905276390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:34.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:34 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2905276390' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:35.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:35.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.49:6834/1901706381,v1:172.21.15.49:6835/1901706381] up:boot 2024-03-18T00:47:35.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: fsmap 3 up:standby 2024-03-18T00:47:35.121 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T00:47:35.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:35.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:35.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:35.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:35.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:35.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:35.122 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:35 smithi049 ceph-mon[32130]: Deploying daemon mds.d on smithi049 2024-03-18T00:47:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.49:6834/1901706381,v1:172.21.15.49:6835/1901706381] up:boot 2024-03-18T00:47:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: fsmap 3 up:standby 2024-03-18T00:47:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T00:47:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:35.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:35 smithi107 ceph-mon[36270]: Deploying daemon mds.d on smithi049 2024-03-18T00:47:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.49:6834/1901706381,v1:172.21.15.49:6835/1901706381] up:boot 2024-03-18T00:47:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: fsmap 3 up:standby 2024-03-18T00:47:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T00:47:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:35.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:35.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:35.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:35.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:35 smithi106 ceph-mon[35899]: Deploying daemon mds.d on smithi049 2024-03-18T00:47:35.917 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.043+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/1302195812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc101720 msgr2=0x7fe7bc101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.043+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/1302195812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc101720 0x7fe7bc101ba0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009900 tx=0x7fe7a402bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.043+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/1302195812 shutdown_connections 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.043+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/1302195812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7bc1020e0 0x7fe7bc108970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.043+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/1302195812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc101720 0x7fe7bc101ba0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.043+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/1302195812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7bc100530 0x7fe7bc100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.043+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/1302195812 >> 172.21.15.49:0/1302195812 conn(0x7fe7bc0fbac0 msgr2=0x7fe7bc0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.044+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/1302195812 shutdown_connections 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.044+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/1302195812 wait complete. 2024-03-18T00:47:36.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.044+0000 7fe7c0b9d640 1 Processor -- start 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.044+0000 7fe7c0b9d640 1 -- start start 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7c0b9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7bc100530 0x7fe7bc1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7c0b9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7bc101720 0x7fe7bc197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7c0b9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc1020e0 0x7fe7bc19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7c0b9d640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc10c220 con 0x7fe7bc1020e0 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7c0b9d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc10c0a0 con 0x7fe7bc100530 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7c0b9d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe7bc10c3a0 con 0x7fe7bc101720 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc1020e0 0x7fe7bc19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc1020e0 0x7fe7bc19ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57378/0 (socket says 172.21.15.49:57378) 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bbfff640 1 -- 172.21.15.49:0/2714649 learned_addr learned my addr 172.21.15.49:0/2714649 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bb7fe640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7bc100530 0x7fe7bc1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7baffd640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7bc101720 0x7fe7bc197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:36.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bbfff640 1 -- 172.21.15.49:0/2714649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7bc101720 msgr2=0x7fe7bc197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bbfff640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7bc101720 0x7fe7bc197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bbfff640 1 -- 172.21.15.49:0/2714649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7bc100530 msgr2=0x7fe7bc1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bbfff640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7bc100530 0x7fe7bc1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bbfff640 1 -- 172.21.15.49:0/2714649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7bc19f600 con 0x7fe7bc1020e0 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bb7fe640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7bc100530 0x7fe7bc1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7baffd640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7bc101720 0x7fe7bc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7bbfff640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc1020e0 0x7fe7bc19ef90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400cad0 tx=0x7fe7b400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7b8ff9640 1 -- 172.21.15.49:0/2714649 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b400ae20 con 0x7fe7bc1020e0 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7b8ff9640 1 -- 172.21.15.49:0/2714649 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b4005020 con 0x7fe7bc1020e0 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7bc19f8f0 con 0x7fe7bc1020e0 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7b8ff9640 1 -- 172.21.15.49:0/2714649 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b400f6f0 con 0x7fe7bc1020e0 2024-03-18T00:47:36.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.045+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7bc19fe40 con 0x7fe7bc1020e0 2024-03-18T00:47:36.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.046+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7bc1a00d0 con 0x7fe7bc1020e0 2024-03-18T00:47:36.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.047+0000 7fe7b8ff9640 1 -- 172.21.15.49:0/2714649 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe7b40051d0 con 0x7fe7bc1020e0 2024-03-18T00:47:36.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.048+0000 7fe7b8ff9640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2880 0x7fe7a80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:36.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.048+0000 7fe7bb7fe640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2880 0x7fe7a80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:36.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.048+0000 7fe7b8ff9640 1 -- 172.21.15.49:0/2714649 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe7b40ce660 con 0x7fe7bc1020e0 2024-03-18T00:47:36.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.049+0000 7fe7bb7fe640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2880 0x7fe7a80a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac006fd0 tx=0x7fe7ac008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:36.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.053+0000 7fe7b8ff9640 1 -- 172.21.15.49:0/2714649 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe7b4014030 con 0x7fe7bc1020e0 2024-03-18T00:47:36.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.323+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7bc1a02a0 con 0x7fe7bc1020e0 2024-03-18T00:47:36.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.324+0000 7fe7b8ff9640 1 -- 172.21.15.49:0/2714649 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fe7b409bba0 con 0x7fe7bc1020e0 2024-03-18T00:47:36.325 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:36.325 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T00:47:36.325 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 4 2024-03-18T00:47:36.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2880 msgr2=0x7fe7a80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:36.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2880 0x7fe7a80a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac006fd0 tx=0x7fe7ac008040 comp rx=0 tx=0).stop 2024-03-18T00:47:36.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc1020e0 msgr2=0x7fe7bc19ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:36.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc1020e0 0x7fe7bc19ef90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400cad0 tx=0x7fe7b400cf90 comp rx=0 tx=0).stop 2024-03-18T00:47:36.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 shutdown_connections 2024-03-18T00:47:36.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2880 0x7fe7a80a4d40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:36.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7bc1020e0 0x7fe7bc19ef90 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:36.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7bc101720 0x7fe7bc197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:36.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 --2- 172.21.15.49:0/2714649 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7bc100530 0x7fe7bc1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:36.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 >> 172.21.15.49:0/2714649 conn(0x7fe7bc0fbac0 msgr2=0x7fe7bc0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:36.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 shutdown_connections 2024-03-18T00:47:36.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:36.326+0000 7fe7c0b9d640 1 -- 172.21.15.49:0/2714649 wait complete. 2024-03-18T00:47:36.625 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 00:47:36 smithi049 systemd[1]: Starting Ceph mds.d for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:37.034 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 00:47:36 smithi049 podman[72564]: 2024-03-18 00:47:36.57391953 +0000 UTC m=+0.018128953 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:37.034 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 00:47:36 smithi049 podman[72564]: 2024-03-18 00:47:36.68787756 +0000 UTC m=+0.132087021 container create 60f6373da18496ef6fa041bf969cc8005ec08a603062be6b29dd59f573463980 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux ) 2024-03-18T00:47:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:37 smithi049 ceph-mon[32130]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:37 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2714649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:37.375 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 00:47:37 smithi049 podman[72564]: 2024-03-18 00:47:37.057450567 +0000 UTC m=+0.501659990 container init 60f6373da18496ef6fa041bf969cc8005ec08a603062be6b29dd59f573463980 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:47:37.375 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 00:47:37 smithi049 podman[72564]: 2024-03-18 00:47:37.06032079 +0000 UTC m=+0.504530223 container start 60f6373da18496ef6fa041bf969cc8005ec08a603062be6b29dd59f573463980 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T00:47:37.375 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 00:47:37 smithi049 ceph-mds[72583]: -- 172.21.15.49:0/796918165 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559ec9adba40 con 0x559ec9aae800 2024-03-18T00:47:37.375 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 00:47:37 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-d[72579]: starting mds.d at 2024-03-18T00:47:37.375 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 00:47:37 smithi049 bash[72564]: 60f6373da18496ef6fa041bf969cc8005ec08a603062be6b29dd59f573463980 2024-03-18T00:47:37.375 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 00:47:37 smithi049 systemd[1]: Started Ceph mds.d for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:37.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:37 smithi107 ceph-mon[36270]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:37.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:37 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2714649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:37.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:37 smithi106 ceph-mon[35899]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:37.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:37 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2714649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:38.339 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:38.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:38.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:38.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:38.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:38.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:38.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:38.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: Deploying daemon mds.e on smithi106 2024-03-18T00:47:38.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.49:6836/535636420,v1:172.21.15.49:6837/535636420] up:boot 2024-03-18T00:47:38.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: fsmap 4 up:standby 2024-03-18T00:47:38.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:38 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.462+0000 7fe1b6374640 1 -- 172.21.15.49:0/2179218460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 msgr2=0x7fe1b01099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.462+0000 7fe1b6374640 1 --2- 172.21.15.49:0/2179218460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 0x7fe1b01099b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8009ca0 tx=0x7fe1a802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.462+0000 7fe1b6374640 1 -- 172.21.15.49:0/2179218460 shutdown_connections 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.462+0000 7fe1b6374640 1 --2- 172.21.15.49:0/2179218460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 0x7fe1b01099b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.462+0000 7fe1b6374640 1 --2- 172.21.15.49:0/2179218460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b0104c30 0x7fe1b0107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.462+0000 7fe1b6374640 1 --2- 172.21.15.49:0/2179218460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1b0102300 0x7fe1b01046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.462+0000 7fe1b6374640 1 -- 172.21.15.49:0/2179218460 >> 172.21.15.49:0/2179218460 conn(0x7fe1b00fbd10 msgr2=0x7fe1b00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.463+0000 7fe1b6374640 1 -- 172.21.15.49:0/2179218460 shutdown_connections 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.463+0000 7fe1b6374640 1 -- 172.21.15.49:0/2179218460 wait complete. 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.463+0000 7fe1b6374640 1 Processor -- start 2024-03-18T00:47:38.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.463+0000 7fe1b6374640 1 -- start start 2024-03-18T00:47:38.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.463+0000 7fe1b6374640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1b0102300 0x7fe1b0197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.463+0000 7fe1b6374640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b0104c30 0x7fe1b0198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b6374640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 0x7fe1b019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b6374640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b010d4b0 con 0x7fe1b0107560 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b6374640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b010d330 con 0x7fe1b0102300 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b6374640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b010d630 con 0x7fe1b0104c30 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b4911640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 0x7fe1b019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1affff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1b0102300 0x7fe1b0197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b4911640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 0x7fe1b019f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57128/0 (socket says 172.21.15.49:57128) 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b4911640 1 -- 172.21.15.49:0/194247385 learned_addr learned my addr 172.21.15.49:0/194247385 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1af7fe640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b0104c30 0x7fe1b0198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b4911640 1 -- 172.21.15.49:0/194247385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b0104c30 msgr2=0x7fe1b0198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b4911640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b0104c30 0x7fe1b0198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b4911640 1 -- 172.21.15.49:0/194247385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1b0102300 msgr2=0x7fe1b0197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b4911640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1b0102300 0x7fe1b0197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b4911640 1 -- 172.21.15.49:0/194247385 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b019f700 con 0x7fe1b0107560 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1affff640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1b0102300 0x7fe1b0197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b4911640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 0x7fe1b019f090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8037040 tx=0x7fe1a8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1ad7fa640 1 -- 172.21.15.49:0/194247385 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8041020 con 0x7fe1b0107560 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1ad7fa640 1 -- 172.21.15.49:0/194247385 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1a8034d90 con 0x7fe1b0107560 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b019f990 con 0x7fe1b0107560 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1ad7fa640 1 -- 172.21.15.49:0/194247385 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8039040 con 0x7fe1b0107560 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.464+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b019fe80 con 0x7fe1b0107560 2024-03-18T00:47:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.465+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b00fd800 con 0x7fe1b0107560 2024-03-18T00:47:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.466+0000 7fe1ad7fa640 1 -- 172.21.15.49:0/194247385 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe1a8044020 con 0x7fe1b0107560 2024-03-18T00:47:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.467+0000 7fe1ad7fa640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1980a2800 0x7fe1980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.467+0000 7fe1affff640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1980a2800 0x7fe1980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.467+0000 7fe1ad7fa640 1 -- 172.21.15.49:0/194247385 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fe1a80f08d0 con 0x7fe1b0107560 2024-03-18T00:47:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.468+0000 7fe1affff640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1980a2800 0x7fe1980a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009fc0 tx=0x7fe1a4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:38.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.471+0000 7fe1ad7fa640 1 -- 172.21.15.49:0/194247385 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1a804b020 con 0x7fe1b0107560 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: Deploying daemon mds.e on smithi106 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.49:6836/535636420,v1:172.21.15.49:6837/535636420] up:boot 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: fsmap 4 up:standby 2024-03-18T00:47:38.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:38 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T00:47:38.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:38.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:38.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:38.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:38.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:38.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:38.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: Deploying daemon mds.e on smithi106 2024-03-18T00:47:38.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.49:6836/535636420,v1:172.21.15.49:6837/535636420] up:boot 2024-03-18T00:47:38.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: fsmap 4 up:standby 2024-03-18T00:47:38.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:38 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T00:47:38.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.693+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1b002d1b0 con 0x7fe1b0107560 2024-03-18T00:47:38.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.694+0000 7fe1ad7fa640 1 -- 172.21.15.49:0/194247385 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7fe1a80bde10 con 0x7fe1b0107560 2024-03-18T00:47:38.695 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:38.695 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T00:47:38.695 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 5 2024-03-18T00:47:38.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1980a2800 msgr2=0x7fe1980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:38.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1980a2800 0x7fe1980a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009fc0 tx=0x7fe1a4005f90 comp rx=0 tx=0).stop 2024-03-18T00:47:38.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 msgr2=0x7fe1b019f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:38.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 0x7fe1b019f090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8037040 tx=0x7fe1a8032f20 comp rx=0 tx=0).stop 2024-03-18T00:47:38.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 shutdown_connections 2024-03-18T00:47:38.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1980a2800 0x7fe1980a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe1b0107560 0x7fe1b019f090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe1b0104c30 0x7fe1b0198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 --2- 172.21.15.49:0/194247385 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe1b0102300 0x7fe1b0197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:38.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 >> 172.21.15.49:0/194247385 conn(0x7fe1b00fbd10 msgr2=0x7fe1b0102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:38.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 shutdown_connections 2024-03-18T00:47:38.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:38.696+0000 7fe1b6374640 1 -- 172.21.15.49:0/194247385 wait complete. 2024-03-18T00:47:39.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:39 smithi049 ceph-mon[32130]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:39.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:39 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/194247385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:39.677 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:39 smithi106 ceph-mon[35899]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:39.677 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:39 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/194247385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:39.677 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 00:47:39 smithi106 systemd[1]: Starting Ceph mds.e for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:39 smithi107 ceph-mon[36270]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:39 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/194247385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:39.945 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 00:47:39 smithi106 podman[66218]: 2024-03-18 00:47:39.580205893 +0000 UTC m=+0.018282370 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:39.946 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 00:47:39 smithi106 podman[66218]: 2024-03-18 00:47:39.680228874 +0000 UTC m=+0.118305337 container create ebdb9d656f990f1472ad02baf492a5e5037af5efee922ece87fb0b484ac02a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-03-18T00:47:40.446 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 00:47:39 smithi106 podman[66218]: 2024-03-18 00:47:39.98373423 +0000 UTC m=+0.421810716 container init ebdb9d656f990f1472ad02baf492a5e5037af5efee922ece87fb0b484ac02a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T00:47:40.446 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 00:47:39 smithi106 podman[66218]: 2024-03-18 00:47:39.986387008 +0000 UTC m=+0.424463486 container start ebdb9d656f990f1472ad02baf492a5e5037af5efee922ece87fb0b484ac02a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:47:40.446 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 00:47:40 smithi106 ceph-mds[66237]: -- 172.21.15.106:0/180474045 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e9f1941a40 con 0x55e9f1914c00 2024-03-18T00:47:40.446 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 00:47:40 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e[66233]: starting mds.e at 2024-03-18T00:47:40.446 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 00:47:40 smithi106 bash[66218]: ebdb9d656f990f1472ad02baf492a5e5037af5efee922ece87fb0b484ac02a7d 2024-03-18T00:47:40.446 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 00:47:40 smithi106 systemd[1]: Started Ceph mds.e for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:40.708 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:40.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.828+0000 7f3a09d77640 1 -- 172.21.15.49:0/1098004962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04100780 msgr2=0x7f3a04100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:40.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.828+0000 7f3a09d77640 1 --2- 172.21.15.49:0/1098004962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04100780 0x7f3a04100ba0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f39f8009900 tx=0x7f39f802bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:40.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.829+0000 7f3a09d77640 1 -- 172.21.15.49:0/1098004962 shutdown_connections 2024-03-18T00:47:40.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.829+0000 7f3a09d77640 1 --2- 172.21.15.49:0/1098004962 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a04102330 0x7f3a04108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:40.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.829+0000 7f3a09d77640 1 --2- 172.21.15.49:0/1098004962 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a04101970 0x7f3a04101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:40.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.829+0000 7f3a09d77640 1 --2- 172.21.15.49:0/1098004962 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04100780 0x7f3a04100ba0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:40.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.829+0000 7f3a09d77640 1 -- 172.21.15.49:0/1098004962 >> 172.21.15.49:0/1098004962 conn(0x7f3a040fbd10 msgr2=0x7f3a040fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:40.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.829+0000 7f3a09d77640 1 -- 172.21.15.49:0/1098004962 shutdown_connections 2024-03-18T00:47:40.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.829+0000 7f3a09d77640 1 -- 172.21.15.49:0/1098004962 wait complete. 2024-03-18T00:47:40.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.829+0000 7f3a09d77640 1 Processor -- start 2024-03-18T00:47:40.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.829+0000 7f3a09d77640 1 -- start start 2024-03-18T00:47:40.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a09d77640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a04100780 0x7f3a041046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:40.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a09d77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04101970 0x7f3a04102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:40.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a09d77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a04102330 0x7f3a04103290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:40.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a09d77640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0410c360 con 0x7f3a04101970 2024-03-18T00:47:40.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a09d77640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0410c1e0 con 0x7f3a04100780 2024-03-18T00:47:40.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a09d77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0410c4e0 con 0x7f3a04102330 2024-03-18T00:47:40.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a02ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04101970 0x7f3a04102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:40.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a02ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04101970 0x7f3a04102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57136/0 (socket says 172.21.15.49:57136) 2024-03-18T00:47:40.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a02ffd640 1 -- 172.21.15.49:0/608397830 learned_addr learned my addr 172.21.15.49:0/608397830 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a037fe640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a04100780 0x7f3a041046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a03fff640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a04102330 0x7f3a04103290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a02ffd640 1 -- 172.21.15.49:0/608397830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a04102330 msgr2=0x7f3a04103290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a02ffd640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a04102330 0x7f3a04103290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a02ffd640 1 -- 172.21.15.49:0/608397830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a04100780 msgr2=0x7f3a041046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a02ffd640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a04100780 0x7f3a041046a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a02ffd640 1 -- 172.21.15.49:0/608397830 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a04103ab0 con 0x7f3a04101970 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a02ffd640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04101970 0x7f3a04102d50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f39f400d950 tx=0x7f39f400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a00ff9640 1 -- 172.21.15.49:0/608397830 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f4017030 con 0x7f3a04101970 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a041a1c70 con 0x7f3a04101970 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.830+0000 7f3a00ff9640 1 -- 172.21.15.49:0/608397830 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39f4004020 con 0x7f3a04101970 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.831+0000 7f3a00ff9640 1 -- 172.21.15.49:0/608397830 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f40047a0 con 0x7f3a04101970 2024-03-18T00:47:40.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.831+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a041a21c0 con 0x7f3a04101970 2024-03-18T00:47:40.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.832+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a041a23f0 con 0x7f3a04101970 2024-03-18T00:47:40.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.833+0000 7f3a00ff9640 1 -- 172.21.15.49:0/608397830 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f39f40041d0 con 0x7f3a04101970 2024-03-18T00:47:40.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.833+0000 7f3a00ff9640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f39ec0a2800 0x7f39ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:40.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.833+0000 7f3a037fe640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f39ec0a2800 0x7f39ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:40.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.833+0000 7f3a00ff9640 1 -- 172.21.15.49:0/608397830 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f39f40cdf60 con 0x7f3a04101970 2024-03-18T00:47:40.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.834+0000 7f3a037fe640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f39ec0a2800 0x7f39ec0a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f39f8000c00 tx=0x7f39f8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:40.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:40.839+0000 7f3a00ff9640 1 -- 172.21.15.49:0/608397830 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f39f409b4a0 con 0x7f3a04101970 2024-03-18T00:47:41.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.061+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a041a26e0 con 0x7f3a04101970 2024-03-18T00:47:41.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.061+0000 7f3a00ff9640 1 -- 172.21.15.49:0/608397830 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7f39f409ada0 con 0x7f3a04101970 2024-03-18T00:47:41.061 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:41.061 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-18T00:47:41.062 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 6 2024-03-18T00:47:41.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.063+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f39ec0a2800 msgr2=0x7f39ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.063+0000 7f3a09d77640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f39ec0a2800 0x7f39ec0a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f39f8000c00 tx=0x7f39f8005fd0 comp rx=0 tx=0).stop 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.063+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04101970 msgr2=0x7f3a04102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.063+0000 7f3a09d77640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04101970 0x7f3a04102d50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f39f400d950 tx=0x7f39f400de10 comp rx=0 tx=0).stop 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.064+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 shutdown_connections 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.064+0000 7f3a09d77640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f39ec0a2800 0x7f39ec0a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.064+0000 7f3a09d77640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a04102330 0x7f3a04103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.064+0000 7f3a09d77640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a04101970 0x7f3a04102d50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.064+0000 7f3a09d77640 1 --2- 172.21.15.49:0/608397830 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a04100780 0x7f3a041046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.064+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 >> 172.21.15.49:0/608397830 conn(0x7f3a040fbd10 msgr2=0x7f3a040fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.064+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 shutdown_connections 2024-03-18T00:47:41.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:41.064+0000 7f3a09d77640 1 -- 172.21.15.49:0/608397830 wait complete. 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: Deploying daemon mds.f on smithi107 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6834/353176461,v1:172.21.15.106:6835/353176461] up:boot 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: fsmap 5 up:standby 2024-03-18T00:47:41.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T00:47:41.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:41 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/608397830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:41.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:41.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:41.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:41.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: Deploying daemon mds.f on smithi107 2024-03-18T00:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6834/353176461,v1:172.21.15.106:6835/353176461] up:boot 2024-03-18T00:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: fsmap 5 up:standby 2024-03-18T00:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T00:47:41.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:41 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/608397830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:41.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: Deploying daemon mds.f on smithi107 2024-03-18T00:47:41.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6834/353176461,v1:172.21.15.106:6835/353176461] up:boot 2024-03-18T00:47:41.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: fsmap 5 up:standby 2024-03-18T00:47:41.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T00:47:41.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:41 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/608397830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:42.431 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 00:47:42 smithi107 systemd[1]: Starting Ceph mds.f for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:42.775 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 00:47:42 smithi107 podman[66774]: 2024-03-18 00:47:42.506880683 +0000 UTC m=+0.110424073 container create 72008fe08c4a5764b32c0df63a3b8f0f5099f1cb9c5371a0fa9ba90c22145ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:47:42.775 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 00:47:42 smithi107 podman[66774]: 2024-03-18 00:47:42.41424332 +0000 UTC m=+0.017786714 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:42.775 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 00:47:42 smithi107 podman[66774]: 2024-03-18 00:47:42.751145797 +0000 UTC m=+0.354689188 container init 72008fe08c4a5764b32c0df63a3b8f0f5099f1cb9c5371a0fa9ba90c22145ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-18T00:47:42.776 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 00:47:42 smithi107 podman[66774]: 2024-03-18 00:47:42.753673051 +0000 UTC m=+0.357216435 container start 72008fe08c4a5764b32c0df63a3b8f0f5099f1cb9c5371a0fa9ba90c22145ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-18T00:47:42.776 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 00:47:42 smithi107 ceph-mds[66792]: -- 172.21.15.107:0/179561213 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557f17b6da40 con 0x557f17b40c00 2024-03-18T00:47:43.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:43.139 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 00:47:42 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f[66788]: starting mds.f at 2024-03-18T00:47:43.139 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 00:47:42 smithi107 bash[66774]: 72008fe08c4a5764b32c0df63a3b8f0f5099f1cb9c5371a0fa9ba90c22145ce5 2024-03-18T00:47:43.139 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 00:47:42 smithi107 systemd[1]: Started Ceph mds.f for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:43.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.198+0000 7f5f468ff640 1 -- 172.21.15.49:0/3684909370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40102330 msgr2=0x7f5f40108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:43.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.198+0000 7f5f468ff640 1 --2- 172.21.15.49:0/3684909370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40102330 0x7f5f40108bc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009ca0 tx=0x7f5f3802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.198+0000 7f5f468ff640 1 -- 172.21.15.49:0/3684909370 shutdown_connections 2024-03-18T00:47:43.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.198+0000 7f5f468ff640 1 --2- 172.21.15.49:0/3684909370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f40101970 0x7f5f40101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.198+0000 7f5f468ff640 1 --2- 172.21.15.49:0/3684909370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40102330 0x7f5f40108bc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.198+0000 7f5f468ff640 1 --2- 172.21.15.49:0/3684909370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f40100780 0x7f5f40100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.198+0000 7f5f468ff640 1 -- 172.21.15.49:0/3684909370 >> 172.21.15.49:0/3684909370 conn(0x7f5f400fbd10 msgr2=0x7f5f400fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:43.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.198+0000 7f5f468ff640 1 -- 172.21.15.49:0/3684909370 shutdown_connections 2024-03-18T00:47:43.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.198+0000 7f5f468ff640 1 -- 172.21.15.49:0/3684909370 wait complete. 2024-03-18T00:47:43.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f468ff640 1 Processor -- start 2024-03-18T00:47:43.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f468ff640 1 -- start start 2024-03-18T00:47:43.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f468ff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f40100780 0x7f5f40197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:43.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f468ff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40101970 0x7f5f40197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f468ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f40102330 0x7f5f4019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f468ff640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4010c370 con 0x7f5f40101970 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f468ff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4010c1f0 con 0x7f5f40100780 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f468ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4010c4f0 con 0x7f5f40102330 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f3f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40101970 0x7f5f40197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f3f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40101970 0x7f5f40197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57158/0 (socket says 172.21.15.49:57158) 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f3f7fe640 1 -- 172.21.15.49:0/1527039215 learned_addr learned my addr 172.21.15.49:0/1527039215 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f44e9c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f40102330 0x7f5f4019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.199+0000 7f5f3ffff640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f40100780 0x7f5f40197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f44e9c640 1 -- 172.21.15.49:0/1527039215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f40100780 msgr2=0x7f5f40197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:43.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f44e9c640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f40100780 0x7f5f40197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f44e9c640 1 -- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40101970 msgr2=0x7f5f40197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:43.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f44e9c640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40101970 0x7f5f40197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f44e9c640 1 -- 172.21.15.49:0/1527039215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4019f6c0 con 0x7f5f40102330 2024-03-18T00:47:43.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f3f7fe640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40101970 0x7f5f40197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:47:43.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f44e9c640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f40102330 0x7f5f4019f050 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5f3802c630 tx=0x7f5f38032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:43.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f3d7fa640 1 -- 172.21.15.49:0/1527039215 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f38039040 con 0x7f5f40102330 2024-03-18T00:47:43.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4019f950 con 0x7f5f40102330 2024-03-18T00:47:43.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f3d7fa640 1 -- 172.21.15.49:0/1527039215 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f38044070 con 0x7f5f40102330 2024-03-18T00:47:43.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f3d7fa640 1 -- 172.21.15.49:0/1527039215 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f380042a0 con 0x7f5f40102330 2024-03-18T00:47:43.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.200+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f4019fe60 con 0x7f5f40102330 2024-03-18T00:47:43.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.201+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f401a00f0 con 0x7f5f40102330 2024-03-18T00:47:43.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.202+0000 7f5f3d7fa640 1 -- 172.21.15.49:0/1527039215 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5f38071b40 con 0x7f5f40102330 2024-03-18T00:47:43.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.203+0000 7f5f3d7fa640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5f280a2800 0x7f5f280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:43.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.203+0000 7f5f3ffff640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5f280a2800 0x7f5f280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:43.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.203+0000 7f5f3d7fa640 1 -- 172.21.15.49:0/1527039215 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5f380f0a80 con 0x7f5f40102330 2024-03-18T00:47:43.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.203+0000 7f5f3ffff640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5f280a2800 0x7f5f280a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5f401017d0 tx=0x7f5f30005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:43.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.206+0000 7f5f3d7fa640 1 -- 172.21.15.49:0/1527039215 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5f380bdff0 con 0x7f5f40102330 2024-03-18T00:47:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:43 smithi049 ceph-mon[32130]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:43 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:43.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.429+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f401a0480 con 0x7f5f40102330 2024-03-18T00:47:43.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.430+0000 7f5f3d7fa640 1 -- 172.21.15.49:0/1527039215 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7f5f380bd8f0 con 0x7f5f40102330 2024-03-18T00:47:43.430 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:43.430 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-18T00:47:43.430 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 7 2024-03-18T00:47:43.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:43 smithi107 ceph-mon[36270]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:43.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:43 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5f280a2800 msgr2=0x7f5f280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5f280a2800 0x7f5f280a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5f401017d0 tx=0x7f5f30005f90 comp rx=0 tx=0).stop 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f40102330 msgr2=0x7f5f4019f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f40102330 0x7f5f4019f050 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5f3802c630 tx=0x7f5f38032f20 comp rx=0 tx=0).stop 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 shutdown_connections 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5f280a2800 0x7f5f280a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5f40102330 0x7f5f4019f050 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5f40101970 0x7f5f40197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 --2- 172.21.15.49:0/1527039215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5f40100780 0x7f5f40197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 >> 172.21.15.49:0/1527039215 conn(0x7f5f400fbd10 msgr2=0x7f5f400fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:43.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 shutdown_connections 2024-03-18T00:47:43.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:43.432+0000 7f5f468ff640 1 -- 172.21.15.49:0/1527039215 wait complete. 2024-03-18T00:47:43.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:43 smithi106 ceph-mon[35899]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:43.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:43.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:43 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:44.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:44 smithi107 ceph-mon[36270]: Deploying daemon mds.c on smithi107 2024-03-18T00:47:44.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:44 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] up:boot 2024-03-18T00:47:44.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:44 smithi107 ceph-mon[36270]: fsmap 6 up:standby 2024-03-18T00:47:44.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:44 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T00:47:44.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:44 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1527039215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:44.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:44 smithi106 ceph-mon[35899]: Deploying daemon mds.c on smithi107 2024-03-18T00:47:44.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:44 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] up:boot 2024-03-18T00:47:44.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:44 smithi106 ceph-mon[35899]: fsmap 6 up:standby 2024-03-18T00:47:44.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:44 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T00:47:44.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:44 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1527039215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:44.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:44 smithi049 ceph-mon[32130]: Deploying daemon mds.c on smithi107 2024-03-18T00:47:44.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:44 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] up:boot 2024-03-18T00:47:44.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:44 smithi049 ceph-mon[32130]: fsmap 6 up:standby 2024-03-18T00:47:44.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:44 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T00:47:44.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:44 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1527039215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:45.126 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 00:47:45 smithi107 systemd[1]: Starting Ceph mds.c for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:45.413 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:45 smithi107 ceph-mon[36270]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:45.414 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 00:47:45 smithi107 podman[67081]: 2024-03-18 00:47:45.188990696 +0000 UTC m=+0.017900664 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:45.414 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 00:47:45 smithi107 podman[67081]: 2024-03-18 00:47:45.295075556 +0000 UTC m=+0.123985526 container create b6e77d53936fa32630a9731807ce3d70d9cfed233643fdc192947fc50639e8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-18T00:47:45.444 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:45.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:45 smithi106 ceph-mon[35899]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:45.476 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:45 smithi049 ceph-mon[32130]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 -- 172.21.15.49:0/2701318782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 msgr2=0x7ff8ac100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 --2- 172.21.15.49:0/2701318782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 0x7ff8ac100eb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009900 tx=0x7ff89c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 -- 172.21.15.49:0/2701318782 shutdown_connections 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 --2- 172.21.15.49:0/2701318782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ac1013f0 0x7ff8ac101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 --2- 172.21.15.49:0/2701318782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 0x7ff8ac100eb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 --2- 172.21.15.49:0/2701318782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8ac1000f0 0x7ff8ac100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 -- 172.21.15.49:0/2701318782 >> 172.21.15.49:0/2701318782 conn(0x7ff8ac0fbd10 msgr2=0x7ff8ac0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 -- 172.21.15.49:0/2701318782 shutdown_connections 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 -- 172.21.15.49:0/2701318782 wait complete. 2024-03-18T00:47:45.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 Processor -- start 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 -- start start 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ac1000f0 0x7ff8ac199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 0x7ff8ac19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.565+0000 7ff8b3697640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8ac1013f0 0x7ff8ac1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b3697640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac10f650 con 0x7ff8ac100a50 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b3697640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac10f4d0 con 0x7ff8ac1013f0 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b3697640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ac10f7d0 con 0x7ff8ac1000f0 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b0c32640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 0x7ff8ac19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b1433640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ac1000f0 0x7ff8ac199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b0c32640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 0x7ff8ac19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57176/0 (socket says 172.21.15.49:57176) 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b0c32640 1 -- 172.21.15.49:0/99063891 learned_addr learned my addr 172.21.15.49:0/99063891 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b1c34640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8ac1013f0 0x7ff8ac1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:45.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b1433640 1 -- 172.21.15.49:0/99063891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8ac1013f0 msgr2=0x7ff8ac1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b1433640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8ac1013f0 0x7ff8ac1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b1433640 1 -- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 msgr2=0x7ff8ac19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b1433640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 0x7ff8ac19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b1433640 1 -- 172.21.15.49:0/99063891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ac1a1900 con 0x7ff8ac1000f0 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b0c32640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 0x7ff8ac19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b1433640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ac1000f0 0x7ff8ac199cc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005fd0 tx=0x7ff8a800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8a27fc640 1 -- 172.21.15.49:0/99063891 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8009c30 con 0x7ff8ac1000f0 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ac1a1bf0 con 0x7ff8ac1000f0 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8a27fc640 1 -- 172.21.15.49:0/99063891 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8a8009dd0 con 0x7ff8ac1000f0 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8a27fc640 1 -- 172.21.15.49:0/99063891 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8a8010460 con 0x7ff8ac1000f0 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.566+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ac1a2110 con 0x7ff8ac1000f0 2024-03-18T00:47:45.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.567+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8ac10ae40 con 0x7ff8ac1000f0 2024-03-18T00:47:45.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.568+0000 7ff8a27fc640 1 -- 172.21.15.49:0/99063891 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff8a8003ef0 con 0x7ff8ac1000f0 2024-03-18T00:47:45.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.569+0000 7ff8a27fc640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8980a2800 0x7ff8980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:45.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.569+0000 7ff8b0c32640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8980a2800 0x7ff8980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:45.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.569+0000 7ff8a27fc640 1 -- 172.21.15.49:0/99063891 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff8a80ce120 con 0x7ff8ac1000f0 2024-03-18T00:47:45.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.570+0000 7ff8b0c32640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8980a2800 0x7ff8980a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff89c000c00 tx=0x7ff89c02c880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:45.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.572+0000 7ff8a27fc640 1 -- 172.21.15.49:0/99063891 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff8a8017020 con 0x7ff8ac1000f0 2024-03-18T00:47:45.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.797+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8ac02d1b0 con 0x7ff8ac1000f0 2024-03-18T00:47:45.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.798+0000 7ff8a27fc640 1 -- 172.21.15.49:0/99063891 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7ff8a809aff0 con 0x7ff8ac1000f0 2024-03-18T00:47:45.797 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:45.797 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-18T00:47:45.798 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 7 2024-03-18T00:47:45.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.799+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8980a2800 msgr2=0x7ff8980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.799+0000 7ff8b3697640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8980a2800 0x7ff8980a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff89c000c00 tx=0x7ff89c02c880 comp rx=0 tx=0).stop 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.799+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ac1000f0 msgr2=0x7ff8ac199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.799+0000 7ff8b3697640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ac1000f0 0x7ff8ac199cc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff8a8005fd0 tx=0x7ff8a800dde0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.800+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 shutdown_connections 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.800+0000 7ff8b3697640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff8980a2800 0x7ff8980a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.800+0000 7ff8b3697640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff8ac1013f0 0x7ff8ac1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.800+0000 7ff8b3697640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff8ac100a50 0x7ff8ac19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.800+0000 7ff8b3697640 1 --2- 172.21.15.49:0/99063891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ac1000f0 0x7ff8ac199cc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.800+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 >> 172.21.15.49:0/99063891 conn(0x7ff8ac0fbd10 msgr2=0x7ff8ac0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.800+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 shutdown_connections 2024-03-18T00:47:45.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:45.800+0000 7ff8b3697640 1 -- 172.21.15.49:0/99063891 wait complete. 2024-03-18T00:47:45.930 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 00:47:45 smithi107 podman[67081]: 2024-03-18 00:47:45.548025871 +0000 UTC m=+0.376935844 container init b6e77d53936fa32630a9731807ce3d70d9cfed233643fdc192947fc50639e8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-18T00:47:45.930 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 00:47:45 smithi107 podman[67081]: 2024-03-18 00:47:45.551089952 +0000 UTC m=+0.379999920 container start b6e77d53936fa32630a9731807ce3d70d9cfed233643fdc192947fc50639e8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:47:45.930 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 00:47:45 smithi107 ceph-mds[67110]: -- 172.21.15.107:0/841789611 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56101b20fa40 con 0x56101b1e2800 2024-03-18T00:47:45.930 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 00:47:45 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c[67104]: starting mds.c at 2024-03-18T00:47:45.930 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 00:47:45 smithi107 bash[67081]: b6e77d53936fa32630a9731807ce3d70d9cfed233643fdc192947fc50639e8be 2024-03-18T00:47:45.931 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 00:47:45 smithi107 systemd[1]: Started Ceph mds.c for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/99063891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:47.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:47.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:47.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: Deploying daemon mds.h on smithi106 2024-03-18T00:47:47.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] up:boot 2024-03-18T00:47:47.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: fsmap 7 up:standby 2024-03-18T00:47:47.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T00:47:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/99063891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: Deploying daemon mds.h on smithi106 2024-03-18T00:47:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] up:boot 2024-03-18T00:47:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: fsmap 7 up:standby 2024-03-18T00:47:47.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/99063891' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: Deploying daemon mds.h on smithi106 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] up:boot 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: fsmap 7 up:standby 2024-03-18T00:47:47.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T00:47:47.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.932+0000 7fee0b59e640 1 -- 172.21.15.49:0/446307882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 msgr2=0x7fee0c101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.932+0000 7fee0b59e640 1 --2- 172.21.15.49:0/446307882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 0x7fee0c101ba0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fee00009900 tx=0x7fee0002bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.933+0000 7fee0b59e640 1 -- 172.21.15.49:0/446307882 shutdown_connections 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.933+0000 7fee0b59e640 1 --2- 172.21.15.49:0/446307882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c1020e0 0x7fee0c108970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.933+0000 7fee0b59e640 1 --2- 172.21.15.49:0/446307882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 0x7fee0c101ba0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.933+0000 7fee0b59e640 1 --2- 172.21.15.49:0/446307882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee0c100530 0x7fee0c100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.933+0000 7fee0b59e640 1 -- 172.21.15.49:0/446307882 >> 172.21.15.49:0/446307882 conn(0x7fee0c0fbac0 msgr2=0x7fee0c0fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.933+0000 7fee0b59e640 1 -- 172.21.15.49:0/446307882 shutdown_connections 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.933+0000 7fee0b59e640 1 -- 172.21.15.49:0/446307882 wait complete. 2024-03-18T00:47:47.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.933+0000 7fee0b59e640 1 Processor -- start 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.933+0000 7fee0b59e640 1 -- start start 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee0b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee0c100530 0x7fee0c197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee0b59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 0x7fee0c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee0b59e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c1020e0 0x7fee0c19eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee0b59e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c10c1c0 con 0x7fee0c101720 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee0b59e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c10c040 con 0x7fee0c1020e0 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee0b59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c10c340 con 0x7fee0c100530 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee09d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 0x7fee0c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee09d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 0x7fee0c197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41882/0 (socket says 172.21.15.49:41882) 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee09d9b640 1 -- 172.21.15.49:0/3051015177 learned_addr learned my addr 172.21.15.49:0/3051015177 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee0ad9d640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c1020e0 0x7fee0c19eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee0a59c640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee0c100530 0x7fee0c197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:47.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee09d9b640 1 -- 172.21.15.49:0/3051015177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee0c100530 msgr2=0x7fee0c197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee09d9b640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee0c100530 0x7fee0c197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee09d9b640 1 -- 172.21.15.49:0/3051015177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c1020e0 msgr2=0x7fee0c19eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee09d9b640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c1020e0 0x7fee0c19eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee09d9b640 1 -- 172.21.15.49:0/3051015177 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c19f530 con 0x7fee0c101720 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fee09d9b640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 0x7fee0c197e50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee00009230 tx=0x7fee0002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fedfb7fe640 1 -- 172.21.15.49:0/3051015177 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00041070 con 0x7fee0c101720 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.934+0000 7fedfb7fe640 1 -- 172.21.15.49:0/3051015177 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee00004450 con 0x7fee0c101720 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.935+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c19f7c0 con 0x7fee0c101720 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.935+0000 7fedfb7fe640 1 -- 172.21.15.49:0/3051015177 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00003e20 con 0x7fee0c101720 2024-03-18T00:47:47.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.935+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c19fd00 con 0x7fee0c101720 2024-03-18T00:47:47.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.936+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee0c19ff60 con 0x7fee0c101720 2024-03-18T00:47:47.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.937+0000 7fedfb7fe640 1 -- 172.21.15.49:0/3051015177 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fee00044070 con 0x7fee0c101720 2024-03-18T00:47:47.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.937+0000 7fedfb7fe640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedf40a2880 0x7fedf40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:47.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.938+0000 7fedfb7fe640 1 -- 172.21.15.49:0/3051015177 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fee000f04f0 con 0x7fee0c101720 2024-03-18T00:47:47.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.938+0000 7fee0a59c640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedf40a2880 0x7fedf40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:47.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.938+0000 7fee0a59c640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedf40a2880 0x7fedf40a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fee0c101580 tx=0x7fee04005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:47.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:47.941+0000 7fedfb7fe640 1 -- 172.21.15.49:0/3051015177 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fee000bda30 con 0x7fee0c101720 2024-03-18T00:47:48.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.168+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee0c1a0130 con 0x7fee0c101720 2024-03-18T00:47:48.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.168+0000 7fedfb7fe640 1 -- 172.21.15.49:0/3051015177 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7fee000bd330 con 0x7fee0c101720 2024-03-18T00:47:48.168 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:48.168 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T00:47:48.169 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 8 2024-03-18T00:47:48.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.170+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedf40a2880 msgr2=0x7fedf40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.170+0000 7fee0b59e640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedf40a2880 0x7fedf40a4d40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fee0c101580 tx=0x7fee04005f90 comp rx=0 tx=0).stop 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 msgr2=0x7fee0c197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 0x7fee0c197e50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fee00009230 tx=0x7fee0002c650 comp rx=0 tx=0).stop 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 shutdown_connections 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedf40a2880 0x7fedf40a4d40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee0c1020e0 0x7fee0c19eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee0c101720 0x7fee0c197e50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 --2- 172.21.15.49:0/3051015177 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee0c100530 0x7fee0c197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 >> 172.21.15.49:0/3051015177 conn(0x7fee0c0fbac0 msgr2=0x7fee0c0fd550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 shutdown_connections 2024-03-18T00:47:48.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:48.171+0000 7fee0b59e640 1 -- 172.21.15.49:0/3051015177 wait complete. 2024-03-18T00:47:48.196 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 00:47:48 smithi106 systemd[1]: Starting Ceph mds.h for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:48.598 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 00:47:48 smithi106 podman[66524]: 2024-03-18 00:47:48.18776813 +0000 UTC m=+0.017783317 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:48.598 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 00:47:48 smithi106 podman[66524]: 2024-03-18 00:47:48.320740745 +0000 UTC m=+0.150755922 container create c065f4492ff73440b01601298facb29169b9eeeefba6bede87ac2f97f4a463cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T00:47:48.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:48 smithi106 ceph-mon[35899]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:48.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:48 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3051015177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:48.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:48 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:48.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:48 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:48.946 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 00:47:48 smithi106 podman[66524]: 2024-03-18 00:47:48.598611824 +0000 UTC m=+0.428627013 container init c065f4492ff73440b01601298facb29169b9eeeefba6bede87ac2f97f4a463cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T00:47:48.946 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 00:47:48 smithi106 podman[66524]: 2024-03-18 00:47:48.601731575 +0000 UTC m=+0.431746752 container start c065f4492ff73440b01601298facb29169b9eeeefba6bede87ac2f97f4a463cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-18T00:47:48.946 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 00:47:48 smithi106 ceph-mds[66551]: -- 172.21.15.106:0/4019549537 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562890691a40 con 0x562890665000 2024-03-18T00:47:48.946 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 00:47:48 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h[66542]: starting mds.h at 2024-03-18T00:47:48.946 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 00:47:48 smithi106 bash[66524]: c065f4492ff73440b01601298facb29169b9eeeefba6bede87ac2f97f4a463cc 2024-03-18T00:47:48.947 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 00:47:48 smithi106 systemd[1]: Started Ceph mds.h for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:48 smithi049 ceph-mon[32130]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:48 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3051015177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:48 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:49.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:48 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:48 smithi107 ceph-mon[36270]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:48 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3051015177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:48 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:48 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:49 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] up:boot 2024-03-18T00:47:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:49 smithi049 ceph-mon[32130]: fsmap 8 up:standby 2024-03-18T00:47:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T00:47:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:49 smithi049 ceph-mon[32130]: Deploying daemon mds.i on smithi107 2024-03-18T00:47:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:49 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] up:boot 2024-03-18T00:47:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:49 smithi107 ceph-mon[36270]: fsmap 8 up:standby 2024-03-18T00:47:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T00:47:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:49 smithi107 ceph-mon[36270]: Deploying daemon mds.i on smithi107 2024-03-18T00:47:50.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:50.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:49 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] up:boot 2024-03-18T00:47:50.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:49 smithi106 ceph-mon[35899]: fsmap 8 up:standby 2024-03-18T00:47:50.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T00:47:50.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:50.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:50.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:50.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:50.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:49 smithi106 ceph-mon[35899]: Deploying daemon mds.i on smithi107 2024-03-18T00:47:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 -- 172.21.15.49:0/3329579990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c80100700 msgr2=0x7f5c80100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 --2- 172.21.15.49:0/3329579990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c80100700 0x7f5c80100b20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5c70009900 tx=0x7f5c7002bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 -- 172.21.15.49:0/3329579990 shutdown_connections 2024-03-18T00:47:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 --2- 172.21.15.49:0/3329579990 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c801018f0 0x7f5c80101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 --2- 172.21.15.49:0/3329579990 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c801022b0 0x7f5c80108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 --2- 172.21.15.49:0/3329579990 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c80100700 0x7f5c80100b20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 -- 172.21.15.49:0/3329579990 >> 172.21.15.49:0/3329579990 conn(0x7f5c800fbc70 msgr2=0x7f5c800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:50.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 -- 172.21.15.49:0/3329579990 shutdown_connections 2024-03-18T00:47:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 -- 172.21.15.49:0/3329579990 wait complete. 2024-03-18T00:47:50.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.304+0000 7f5c857d3640 1 Processor -- start 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c857d3640 1 -- start start 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c857d3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c80100700 0x7f5c80197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c857d3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c801018f0 0x7f5c80198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c857d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c801022b0 0x7f5c8019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c857d3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8010d460 con 0x7f5c801018f0 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c857d3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8010d2e0 con 0x7f5c80100700 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c857d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8010d5e0 con 0x7f5c801022b0 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c801018f0 0x7f5c80198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c801018f0 0x7f5c80198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41894/0 (socket says 172.21.15.49:41894) 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7e7fc640 1 -- 172.21.15.49:0/1779938077 learned_addr learned my addr 172.21.15.49:0/1779938077 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:50.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7effd640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c80100700 0x7f5c80197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7f7fe640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c801022b0 0x7f5c8019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7e7fc640 1 -- 172.21.15.49:0/1779938077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c801022b0 msgr2=0x7f5c8019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7e7fc640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c801022b0 0x7f5c8019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7e7fc640 1 -- 172.21.15.49:0/1779938077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c80100700 msgr2=0x7f5c80197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7e7fc640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c80100700 0x7f5c80197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7e7fc640 1 -- 172.21.15.49:0/1779938077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8019f750 con 0x7f5c801018f0 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7f7fe640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c801022b0 0x7f5c8019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.305+0000 7f5c7e7fc640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c801018f0 0x7f5c80198070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400c890 tx=0x7f5c7400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.306+0000 7f5c67fff640 1 -- 172.21.15.49:0/1779938077 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74004220 con 0x7f5c801018f0 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.306+0000 7f5c67fff640 1 -- 172.21.15.49:0/1779938077 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c740043c0 con 0x7f5c801018f0 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.306+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c8019fa40 con 0x7f5c801018f0 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.306+0000 7f5c67fff640 1 -- 172.21.15.49:0/1779938077 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c740049e0 con 0x7f5c801018f0 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.306+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c8019ff90 con 0x7f5c801018f0 2024-03-18T00:47:50.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.307+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c80107aa0 con 0x7f5c801018f0 2024-03-18T00:47:50.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.309+0000 7f5c67fff640 1 -- 172.21.15.49:0/1779938077 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5c740079b0 con 0x7f5c801018f0 2024-03-18T00:47:50.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.309+0000 7f5c67fff640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5c680a2800 0x7f5c680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.309+0000 7f5c7effd640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5c680a2800 0x7f5c680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.309+0000 7f5c67fff640 1 -- 172.21.15.49:0/1779938077 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5c740ce620 con 0x7f5c801018f0 2024-03-18T00:47:50.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.310+0000 7f5c7effd640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5c680a2800 0x7f5c680a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c70000c00 tx=0x7f5c70004f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:50.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.313+0000 7f5c67fff640 1 -- 172.21.15.49:0/1779938077 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5c74014030 con 0x7f5c801018f0 2024-03-18T00:47:50.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.537+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c8002d1b0 con 0x7f5c801018f0 2024-03-18T00:47:50.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.538+0000 7f5c67fff640 1 -- 172.21.15.49:0/1779938077 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f5c7409bb60 con 0x7f5c801018f0 2024-03-18T00:47:50.538 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:50.538 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T00:47:50.538 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 9 2024-03-18T00:47:50.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5c680a2800 msgr2=0x7f5c680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:50.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5c680a2800 0x7f5c680a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c70000c00 tx=0x7f5c70004f10 comp rx=0 tx=0).stop 2024-03-18T00:47:50.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c801018f0 msgr2=0x7f5c80198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:50.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c801018f0 0x7f5c80198070 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400c890 tx=0x7f5c7400cd50 comp rx=0 tx=0).stop 2024-03-18T00:47:50.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 shutdown_connections 2024-03-18T00:47:50.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5c680a2800 0x7f5c680a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:50.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c801022b0 0x7f5c8019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:50.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c801018f0 0x7f5c80198070 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:50.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 --2- 172.21.15.49:0/1779938077 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c80100700 0x7f5c80197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:50.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 >> 172.21.15.49:0/1779938077 conn(0x7f5c800fbc70 msgr2=0x7f5c80104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:50.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 shutdown_connections 2024-03-18T00:47:50.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:50.540+0000 7f5c857d3640 1 -- 172.21.15.49:0/1779938077 wait complete. 2024-03-18T00:47:50.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:50 smithi049 ceph-mon[32130]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:50.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:50 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1779938077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:50 smithi107 ceph-mon[36270]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:50 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1779938077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:51.181 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 00:47:50 smithi107 systemd[1]: Starting Ceph mds.i for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:51.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:50 smithi106 ceph-mon[35899]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:51.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:50 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1779938077' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:51.459 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 00:47:51 smithi107 podman[67399]: 2024-03-18 00:47:51.084960444 +0000 UTC m=+0.017827160 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:51.460 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 00:47:51 smithi107 podman[67399]: 2024-03-18 00:47:51.21621701 +0000 UTC m=+0.149083721 container create eea5ffc66de3490d39701764637c8d68956368162d6f33148babe15743b1bfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T00:47:51.460 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 00:47:51 smithi107 podman[67399]: 2024-03-18 00:47:51.460634806 +0000 UTC m=+0.393501524 container init eea5ffc66de3490d39701764637c8d68956368162d6f33148babe15743b1bfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-18T00:47:51.930 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 00:47:51 smithi107 podman[67399]: 2024-03-18 00:47:51.463318565 +0000 UTC m=+0.396185284 container start eea5ffc66de3490d39701764637c8d68956368162d6f33148babe15743b1bfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-03-18T00:47:51.930 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 00:47:51 smithi107 ceph-mds[67419]: -- 172.21.15.107:0/1386857191 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56132ef25a40 con 0x56132eef8800 2024-03-18T00:47:51.930 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 00:47:51 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i[67415]: starting mds.i at 2024-03-18T00:47:51.930 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 00:47:51 smithi107 bash[67399]: eea5ffc66de3490d39701764637c8d68956368162d6f33148babe15743b1bfda 2024-03-18T00:47:51.931 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 00:47:51 smithi107 systemd[1]: Started Ceph mds.i for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:52.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:52.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.664+0000 7fee0a405640 1 -- 172.21.15.49:0/1338228684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee041022b0 msgr2=0x7fee04108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.664+0000 7fee0a405640 1 --2- 172.21.15.49:0/1338228684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee041022b0 0x7fee04108b40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fedfc009ca0 tx=0x7fedfc02c0f0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.664+0000 7fee0a405640 1 -- 172.21.15.49:0/1338228684 shutdown_connections 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.664+0000 7fee0a405640 1 --2- 172.21.15.49:0/1338228684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee041018f0 0x7fee04101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.664+0000 7fee0a405640 1 --2- 172.21.15.49:0/1338228684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee041022b0 0x7fee04108b40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.664+0000 7fee0a405640 1 --2- 172.21.15.49:0/1338228684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee04100700 0x7fee04100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.664+0000 7fee0a405640 1 -- 172.21.15.49:0/1338228684 >> 172.21.15.49:0/1338228684 conn(0x7fee040fbc70 msgr2=0x7fee040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.664+0000 7fee0a405640 1 -- 172.21.15.49:0/1338228684 shutdown_connections 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.664+0000 7fee0a405640 1 -- 172.21.15.49:0/1338228684 wait complete. 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee0a405640 1 Processor -- start 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee0a405640 1 -- start start 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee0a405640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee04100700 0x7fee04197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:52.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee0a405640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee041018f0 0x7fee04198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee0a405640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee041022b0 0x7fee0419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee0a405640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fee0410d4a0 con 0x7fee04100700 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee0a405640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fee0410d320 con 0x7fee041022b0 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee0a405640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fee0410d620 con 0x7fee041018f0 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee03fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee04100700 0x7fee04197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee03fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee04100700 0x7fee04197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41900/0 (socket says 172.21.15.49:41900) 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.665+0000 7fee03fff640 1 -- 172.21.15.49:0/1407070065 learned_addr learned my addr 172.21.15.49:0/1407070065 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee089a2640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee041022b0 0x7fee0419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee037fe640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee041018f0 0x7fee04198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee03fff640 1 -- 172.21.15.49:0/1407070065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee041018f0 msgr2=0x7fee04198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee03fff640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee041018f0 0x7fee04198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee03fff640 1 -- 172.21.15.49:0/1407070065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee041022b0 msgr2=0x7fee0419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee03fff640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee041022b0 0x7fee0419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee03fff640 1 -- 172.21.15.49:0/1407070065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0419f6f0 con 0x7fee04100700 2024-03-18T00:47:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee03fff640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee04100700 0x7fee04197ad0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fedf400e9c0 tx=0x7fedf400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee017fa640 1 -- 172.21.15.49:0/1407070065 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf400cd80 con 0x7fee04100700 2024-03-18T00:47:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee017fa640 1 -- 172.21.15.49:0/1407070065 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedf40035f0 con 0x7fee04100700 2024-03-18T00:47:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee017fa640 1 -- 172.21.15.49:0/1407070065 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4010690 con 0x7fee04100700 2024-03-18T00:47:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0419f9e0 con 0x7fee04100700 2024-03-18T00:47:52.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.666+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0419fee0 con 0x7fee04100700 2024-03-18T00:47:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.667+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee04107a20 con 0x7fee04100700 2024-03-18T00:47:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.668+0000 7fee017fa640 1 -- 172.21.15.49:0/1407070065 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fedf40037a0 con 0x7fee04100700 2024-03-18T00:47:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.669+0000 7fee017fa640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedec0a2800 0x7fedec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.669+0000 7fee037fe640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedec0a2800 0x7fedec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.669+0000 7fee017fa640 1 -- 172.21.15.49:0/1407070065 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fedf40ce9d0 con 0x7fee04100700 2024-03-18T00:47:52.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.670+0000 7fee037fe640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedec0a2800 0x7fedec0a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fedf8005fd0 tx=0x7fedf8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:52.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.674+0000 7fee017fa640 1 -- 172.21.15.49:0/1407070065 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fedf409bf10 con 0x7fee04100700 2024-03-18T00:47:52.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:52.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:52.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:52.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:52.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:52.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:52.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:52.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: Deploying daemon mds.g on smithi049 2024-03-18T00:47:52.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] up:boot 2024-03-18T00:47:52.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: fsmap 9 up:standby 2024-03-18T00:47:52.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T00:47:52.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.899+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee0402d430 con 0x7fee04100700 2024-03-18T00:47:52.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.900+0000 7fee017fa640 1 -- 172.21.15.49:0/1407070065 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7fedf409b810 con 0x7fee04100700 2024-03-18T00:47:52.901 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:52.901 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T00:47:52.902 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 10 2024-03-18T00:47:52.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedec0a2800 msgr2=0x7fedec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:52.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedec0a2800 0x7fedec0a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fedf8005fd0 tx=0x7fedf8005850 comp rx=0 tx=0).stop 2024-03-18T00:47:52.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee04100700 msgr2=0x7fee04197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:52.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee04100700 0x7fee04197ad0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fedf400e9c0 tx=0x7fedf400ee80 comp rx=0 tx=0).stop 2024-03-18T00:47:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 shutdown_connections 2024-03-18T00:47:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fedec0a2800 0x7fedec0a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee041022b0 0x7fee0419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee041018f0 0x7fee04198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 --2- 172.21.15.49:0/1407070065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee04100700 0x7fee04197ad0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 >> 172.21.15.49:0/1407070065 conn(0x7fee040fbc70 msgr2=0x7fee04104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 shutdown_connections 2024-03-18T00:47:52.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:52.902+0000 7fee0a405640 1 -- 172.21.15.49:0/1407070065 wait complete. 2024-03-18T00:47:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:52.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: Deploying daemon mds.g on smithi049 2024-03-18T00:47:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] up:boot 2024-03-18T00:47:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: fsmap 9 up:standby 2024-03-18T00:47:52.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: Deploying daemon mds.g on smithi049 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] up:boot 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: fsmap 9 up:standby 2024-03-18T00:47:52.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T00:47:53.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:53 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1407070065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:53.875 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 00:47:53 smithi049 systemd[1]: Starting Ceph mds.g for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:53.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:53 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1407070065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:53.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:53 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1407070065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:54.290 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 00:47:54 smithi049 podman[73160]: 2024-03-18 00:47:53.942609011 +0000 UTC m=+0.018619299 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:54.290 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 00:47:54 smithi049 podman[73160]: 2024-03-18 00:47:54.046433189 +0000 UTC m=+0.122443500 container create 9ebcde9fa222f9a71bdf241662530e2254ab3e6238d4c2318056d3f134b18e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:47:54.290 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 00:47:54 smithi049 podman[73160]: 2024-03-18 00:47:54.29102921 +0000 UTC m=+0.367039496 container init 9ebcde9fa222f9a71bdf241662530e2254ab3e6238d4c2318056d3f134b18e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-18T00:47:54.618 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 00:47:54 smithi049 podman[73160]: 2024-03-18 00:47:54.293878687 +0000 UTC m=+0.369888964 container start 9ebcde9fa222f9a71bdf241662530e2254ab3e6238d4c2318056d3f134b18e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T00:47:54.618 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 00:47:54 smithi049 ceph-mds[73179]: -- 172.21.15.49:0/4235441855 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562065d0da40 con 0x562065ce0800 2024-03-18T00:47:54.619 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 00:47:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g[73175]: starting mds.g at 2024-03-18T00:47:54.619 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 00:47:54 smithi049 bash[73160]: 9ebcde9fa222f9a71bdf241662530e2254ab3e6238d4c2318056d3f134b18e5f 2024-03-18T00:47:54.619 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 00:47:54 smithi049 systemd[1]: Started Ceph mds.g for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:54.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:54 smithi049 ceph-mon[32130]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:54.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:54.914 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:54.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:54 smithi107 ceph-mon[36270]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:54.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:54.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:54.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:54.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:54.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:54.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:54.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:54 smithi106 ceph-mon[35899]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:54.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:55.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.032+0000 7efd26d9d640 1 -- 172.21.15.49:0/1254471462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd201018f0 msgr2=0x7efd20101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:55.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.032+0000 7efd26d9d640 1 --2- 172.21.15.49:0/1254471462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd201018f0 0x7efd20101d70 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efd0c009900 tx=0x7efd0c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.032+0000 7efd26d9d640 1 -- 172.21.15.49:0/1254471462 shutdown_connections 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.032+0000 7efd26d9d640 1 --2- 172.21.15.49:0/1254471462 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd201022b0 0x7efd20108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.032+0000 7efd26d9d640 1 --2- 172.21.15.49:0/1254471462 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd201018f0 0x7efd20101d70 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.032+0000 7efd26d9d640 1 --2- 172.21.15.49:0/1254471462 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd20100700 0x7efd20100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.032+0000 7efd26d9d640 1 -- 172.21.15.49:0/1254471462 >> 172.21.15.49:0/1254471462 conn(0x7efd200fbc70 msgr2=0x7efd200fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 -- 172.21.15.49:0/1254471462 shutdown_connections 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 -- 172.21.15.49:0/1254471462 wait complete. 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 Processor -- start 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 -- start start 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd20100700 0x7efd20197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd201018f0 0x7efd20198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd201022b0 0x7efd2019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:55.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010d4a0 con 0x7efd20100700 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010d320 con 0x7efd201022b0 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd26d9d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efd2010d620 con 0x7efd201018f0 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd24b39640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd20100700 0x7efd20197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd24b39640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd20100700 0x7efd20197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41932/0 (socket says 172.21.15.49:41932) 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.033+0000 7efd24b39640 1 -- 172.21.15.49:0/3164057451 learned_addr learned my addr 172.21.15.49:0/3164057451 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd1ffff640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd201018f0 0x7efd20198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd24b39640 1 -- 172.21.15.49:0/3164057451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd201018f0 msgr2=0x7efd20198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd24b39640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd201018f0 0x7efd20198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd24b39640 1 -- 172.21.15.49:0/3164057451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd201022b0 msgr2=0x7efd2019f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd24b39640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd201022b0 0x7efd2019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd24b39640 1 -- 172.21.15.49:0/3164057451 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2019f740 con 0x7efd20100700 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd24b39640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd20100700 0x7efd20197b20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efd1400d950 tx=0x7efd1400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd1dffb640 1 -- 172.21.15.49:0/3164057451 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd14017070 con 0x7efd20100700 2024-03-18T00:47:55.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd1dffb640 1 -- 172.21.15.49:0/3164057451 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd14009c30 con 0x7efd20100700 2024-03-18T00:47:55.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2019fa30 con 0x7efd20100700 2024-03-18T00:47:55.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd1dffb640 1 -- 172.21.15.49:0/3164057451 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1400f600 con 0x7efd20100700 2024-03-18T00:47:55.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.034+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd2019ff50 con 0x7efd20100700 2024-03-18T00:47:55.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.035+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd20107a20 con 0x7efd20100700 2024-03-18T00:47:55.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.036+0000 7efd1dffb640 1 -- 172.21.15.49:0/3164057451 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efd1401a070 con 0x7efd20100700 2024-03-18T00:47:55.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.037+0000 7efd1dffb640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd100a2880 0x7efd100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:55.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.037+0000 7efd1ffff640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd100a2880 0x7efd100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:55.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.037+0000 7efd1dffb640 1 -- 172.21.15.49:0/3164057451 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7efd140ce030 con 0x7efd20100700 2024-03-18T00:47:55.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.038+0000 7efd1ffff640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd100a2880 0x7efd100a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efd0c006010 tx=0x7efd0c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:55.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.041+0000 7efd1dffb640 1 -- 172.21.15.49:0/3164057451 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd14009de0 con 0x7efd20100700 2024-03-18T00:47:55.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.268+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd2002d1b0 con 0x7efd20100700 2024-03-18T00:47:55.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.268+0000 7efd1dffb640 1 -- 172.21.15.49:0/3164057451 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7efd2002d1b0 con 0x7efd20100700 2024-03-18T00:47:55.269 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:55.269 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T00:47:55.269 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 11 2024-03-18T00:47:55.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd100a2880 msgr2=0x7efd100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:55.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd100a2880 0x7efd100a4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efd0c006010 tx=0x7efd0c000bc0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd20100700 msgr2=0x7efd20197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd20100700 0x7efd20197b20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7efd1400d950 tx=0x7efd1400de10 comp rx=0 tx=0).stop 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 shutdown_connections 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd100a2880 0x7efd100a4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd201022b0 0x7efd2019f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd201018f0 0x7efd20198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 --2- 172.21.15.49:0/3164057451 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd20100700 0x7efd20197b20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.270+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 >> 172.21.15.49:0/3164057451 conn(0x7efd200fbc70 msgr2=0x7efd20104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.271+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 shutdown_connections 2024-03-18T00:47:55.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:55.271+0000 7efd26d9d640 1 -- 172.21.15.49:0/3164057451 wait complete. 2024-03-18T00:47:55.791 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:55 smithi049 ceph-mon[32130]: Deploying daemon mds.a on smithi049 2024-03-18T00:47:55.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:55 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.49:6838/1964983089,v1:172.21.15.49:6839/1964983089] up:boot 2024-03-18T00:47:55.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:55 smithi049 ceph-mon[32130]: fsmap 10 up:standby 2024-03-18T00:47:55.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:55 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T00:47:55.792 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:55 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3164057451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:55.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:55 smithi107 ceph-mon[36270]: Deploying daemon mds.a on smithi049 2024-03-18T00:47:55.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:55 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.49:6838/1964983089,v1:172.21.15.49:6839/1964983089] up:boot 2024-03-18T00:47:55.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:55 smithi107 ceph-mon[36270]: fsmap 10 up:standby 2024-03-18T00:47:55.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:55 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T00:47:55.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:55 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3164057451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:55.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:55 smithi106 ceph-mon[35899]: Deploying daemon mds.a on smithi049 2024-03-18T00:47:55.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:55 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.49:6838/1964983089,v1:172.21.15.49:6839/1964983089] up:boot 2024-03-18T00:47:55.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:55 smithi106 ceph-mon[35899]: fsmap 10 up:standby 2024-03-18T00:47:55.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:55 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T00:47:55.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:55 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3164057451' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:56.625 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 00:47:56 smithi049 systemd[1]: Starting Ceph mds.a for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T00:47:56.925 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:56 smithi049 ceph-mon[32130]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:56.925 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 00:47:56 smithi049 podman[73514]: 2024-03-18 00:47:56.684207079 +0000 UTC m=+0.018796547 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T00:47:56.925 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 00:47:56 smithi049 podman[73514]: 2024-03-18 00:47:56.811133355 +0000 UTC m=+0.145722850 container create 240276f0c2668c71d6efce021b307f213b95d1c1dc0d312566d8de1ec9839b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T00:47:56.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:56 smithi107 ceph-mon[36270]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:56.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:56 smithi106 ceph-mon[35899]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:57.283 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:57.313 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 00:47:57 smithi049 podman[73514]: 2024-03-18 00:47:57.03886603 +0000 UTC m=+0.373455495 container init 240276f0c2668c71d6efce021b307f213b95d1c1dc0d312566d8de1ec9839b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-03-18T00:47:57.313 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 00:47:57 smithi049 podman[73514]: 2024-03-18 00:47:57.04143735 +0000 UTC m=+0.376026808 container start 240276f0c2668c71d6efce021b307f213b95d1c1dc0d312566d8de1ec9839b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T00:47:57.313 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 00:47:57 smithi049 ceph-mds[73533]: -- 172.21.15.49:0/2596893017 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cdcb047a40 con 0x55cdcb01ac00 2024-03-18T00:47:57.313 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 00:47:57 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a[73529]: starting mds.a at 2024-03-18T00:47:57.313 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 00:47:57 smithi049 bash[73514]: 240276f0c2668c71d6efce021b307f213b95d1c1dc0d312566d8de1ec9839b0c 2024-03-18T00:47:57.314 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 00:47:57 smithi049 systemd[1]: Started Ceph mds.a for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:47:57.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 -- 172.21.15.49:0/2474523714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 msgr2=0x7f86441099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:57.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 --2- 172.21.15.49:0/2474523714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 0x7f86441099b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8634009ca0 tx=0x7f863402c0f0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 -- 172.21.15.49:0/2474523714 shutdown_connections 2024-03-18T00:47:57.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 --2- 172.21.15.49:0/2474523714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 0x7f86441099b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 --2- 172.21.15.49:0/2474523714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8644104c30 0x7f8644107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 --2- 172.21.15.49:0/2474523714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644102300 0x7f86441046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 -- 172.21.15.49:0/2474523714 >> 172.21.15.49:0/2474523714 conn(0x7f86440fbd10 msgr2=0x7f86440fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:57.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 -- 172.21.15.49:0/2474523714 shutdown_connections 2024-03-18T00:47:57.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 -- 172.21.15.49:0/2474523714 wait complete. 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.396+0000 7f864891b640 1 Processor -- start 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f864891b640 1 -- start start 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f864891b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644102300 0x7f8644101c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f864891b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8644104c30 0x7f86441002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f864891b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 0x7f86441007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f864891b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f864410d500 con 0x7f8644107560 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f864891b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f864410d380 con 0x7f8644102300 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f864891b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f864410d680 con 0x7f8644104c30 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8642d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 0x7f86441007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8642d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 0x7f86441007f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41980/0 (socket says 172.21.15.49:41980) 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8642d9d640 1 -- 172.21.15.49:0/1947731159 learned_addr learned my addr 172.21.15.49:0/1947731159 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f864259c640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644102300 0x7f8644101c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8641d9b640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8644104c30 0x7f86441002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8642d9d640 1 -- 172.21.15.49:0/1947731159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8644104c30 msgr2=0x7f86441002b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8642d9d640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8644104c30 0x7f86441002b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8642d9d640 1 -- 172.21.15.49:0/1947731159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644102300 msgr2=0x7f8644101c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8642d9d640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644102300 0x7f8644101c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8642d9d640 1 -- 172.21.15.49:0/1947731159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8644101010 con 0x7f8644107560 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f8642d9d640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 0x7f86441007f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f863402c700 tx=0x7f8634032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f86337fe640 1 -- 172.21.15.49:0/1947731159 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634039040 con 0x7f8644107560 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86441a1c00 con 0x7f8644107560 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f86337fe640 1 -- 172.21.15.49:0/1947731159 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8634044070 con 0x7f8644107560 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.397+0000 7f86337fe640 1 -- 172.21.15.49:0/1947731159 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86340042a0 con 0x7f8644107560 2024-03-18T00:47:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.398+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86441a2140 con 0x7f8644107560 2024-03-18T00:47:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.398+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86440fd800 con 0x7f8644107560 2024-03-18T00:47:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.400+0000 7f86337fe640 1 -- 172.21.15.49:0/1947731159 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8634003970 con 0x7f8644107560 2024-03-18T00:47:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.400+0000 7f86337fe640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f862c0a2800 0x7f862c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.400+0000 7f864259c640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f862c0a2800 0x7f862c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.400+0000 7f86337fe640 1 -- 172.21.15.49:0/1947731159 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f86340f1100 con 0x7f8644107560 2024-03-18T00:47:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.401+0000 7f864259c640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f862c0a2800 0x7f862c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f863c00bbe0 tx=0x7f863c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:57.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.404+0000 7f86337fe640 1 -- 172.21.15.49:0/1947731159 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f86340be6d0 con 0x7f8644107560 2024-03-18T00:47:57.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.626+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f864402d360 con 0x7f8644107560 2024-03-18T00:47:57.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.630+0000 7f86337fe640 1 -- 172.21.15.49:0/1947731159 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f86340bdfd0 con 0x7f8644107560 2024-03-18T00:47:57.630 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:47:57.630 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T00:47:57.630 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 12 2024-03-18T00:47:57.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f862c0a2800 msgr2=0x7f862c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:57.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f862c0a2800 0x7f862c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f863c00bbe0 tx=0x7f863c00b390 comp rx=0 tx=0).stop 2024-03-18T00:47:57.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 msgr2=0x7f86441007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 0x7f86441007f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f863402c700 tx=0x7f8634032f20 comp rx=0 tx=0).stop 2024-03-18T00:47:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 shutdown_connections 2024-03-18T00:47:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f862c0a2800 0x7f862c0a4cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8644107560 0x7f86441007f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8644104c30 0x7f86441002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 --2- 172.21.15.49:0/1947731159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8644102300 0x7f8644101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.632+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 >> 172.21.15.49:0/1947731159 conn(0x7f86440fbd10 msgr2=0x7f8644102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.633+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 shutdown_connections 2024-03-18T00:47:57.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:57.633+0000 7f864891b640 1 -- 172.21.15.49:0/1947731159 wait complete. 2024-03-18T00:47:58.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:58.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: Deploying daemon mds.b on smithi106 2024-03-18T00:47:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.49:6840/3327527918,v1:172.21.15.49:6841/3327527918] up:boot 2024-03-18T00:47:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: fsmap 11 up:standby 2024-03-18T00:47:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T00:47:58.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:58 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1947731159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:58.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: Deploying daemon mds.b on smithi106 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.49:6840/3327527918,v1:172.21.15.49:6841/3327527918] up:boot 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: fsmap 11 up:standby 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T00:47:58.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:58 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1947731159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:58.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:58.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:58.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:47:58.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T00:47:58.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T00:47:58.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:47:58.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: Deploying daemon mds.b on smithi106 2024-03-18T00:47:58.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.49:6840/3327527918,v1:172.21.15.49:6841/3327527918] up:boot 2024-03-18T00:47:58.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: fsmap 11 up:standby 2024-03-18T00:47:58.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T00:47:58.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:58 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1947731159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:47:59.312 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:47:59 smithi106 ceph-mon[35899]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:59.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:47:59 smithi049 ceph-mon[32130]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:59.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:47:59.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:47:59 smithi107 ceph-mon[36270]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:47:59.695 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:47:59 smithi106 podman[66841]: 2024-03-18 00:47:59.677186148 +0000 UTC m=+0.138304633 container create e291b8ea36c847a09c42cab7d772a0075c9564652e7c681de8e6f5afc1b4ca2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T00:47:59.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.766+0000 7f9c02234640 1 -- 172.21.15.49:0/1591426641 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 msgr2=0x7f9bfc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:59.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.766+0000 7f9c02234640 1 --2- 172.21.15.49:0/1591426641 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 0x7f9bfc108b40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4009ca0 tx=0x7f9bf402c0f0 comp rx=0 tx=0).stop 2024-03-18T00:47:59.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.766+0000 7f9c02234640 1 -- 172.21.15.49:0/1591426641 shutdown_connections 2024-03-18T00:47:59.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.766+0000 7f9c02234640 1 --2- 172.21.15.49:0/1591426641 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bfc1018f0 0x7f9bfc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:59.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.766+0000 7f9c02234640 1 --2- 172.21.15.49:0/1591426641 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 0x7f9bfc108b40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:59.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.766+0000 7f9c02234640 1 --2- 172.21.15.49:0/1591426641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bfc100700 0x7f9bfc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.766+0000 7f9c02234640 1 -- 172.21.15.49:0/1591426641 >> 172.21.15.49:0/1591426641 conn(0x7f9bfc0fbc70 msgr2=0x7f9bfc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.766+0000 7f9c02234640 1 -- 172.21.15.49:0/1591426641 shutdown_connections 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.766+0000 7f9c02234640 1 -- 172.21.15.49:0/1591426641 wait complete. 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9c02234640 1 Processor -- start 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9c02234640 1 -- start start 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9c02234640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bfc100700 0x7f9bfc197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9c02234640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bfc1018f0 0x7f9bfc197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9c02234640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 0x7f9bfc19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9c02234640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9bfc10d4a0 con 0x7f9bfc1022b0 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9c02234640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9bfc10d320 con 0x7f9bfc100700 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9c02234640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bfc10d620 con 0x7f9bfc1018f0 2024-03-18T00:47:59.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9bfbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 0x7f9bfc19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9bfbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 0x7f9bfc19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34606/0 (socket says 172.21.15.49:34606) 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9bfbfff640 1 -- 172.21.15.49:0/2146829295 learned_addr learned my addr 172.21.15.49:0/2146829295 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9bfb7fe640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bfc100700 0x7f9bfc197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9bfaffd640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bfc1018f0 0x7f9bfc197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9bfbfff640 1 -- 172.21.15.49:0/2146829295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bfc1018f0 msgr2=0x7f9bfc197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.767+0000 7f9bfbfff640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bfc1018f0 0x7f9bfc197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9bfbfff640 1 -- 172.21.15.49:0/2146829295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bfc100700 msgr2=0x7f9bfc197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9bfbfff640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bfc100700 0x7f9bfc197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9bfbfff640 1 -- 172.21.15.49:0/2146829295 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bfc19f6b0 con 0x7f9bfc1022b0 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9bfb7fe640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bfc100700 0x7f9bfc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9bfbfff640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 0x7f9bfc19f040 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9bf402c700 tx=0x7f9bf4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9bf8ff9640 1 -- 172.21.15.49:0/2146829295 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf4039040 con 0x7f9bfc1022b0 2024-03-18T00:47:59.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9bf8ff9640 1 -- 172.21.15.49:0/2146829295 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bf4044070 con 0x7f9bfc1022b0 2024-03-18T00:47:59.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bfc19f940 con 0x7f9bfc1022b0 2024-03-18T00:47:59.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9bf8ff9640 1 -- 172.21.15.49:0/2146829295 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bf40042a0 con 0x7f9bfc1022b0 2024-03-18T00:47:59.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.768+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bfc19fe80 con 0x7f9bfc1022b0 2024-03-18T00:47:59.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.769+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bfc107a20 con 0x7f9bfc1022b0 2024-03-18T00:47:59.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.770+0000 7f9bf8ff9640 1 -- 172.21.15.49:0/2146829295 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9bf4003970 con 0x7f9bfc1022b0 2024-03-18T00:47:59.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.771+0000 7f9bf8ff9640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9be40a2880 0x7f9be40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:47:59.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.771+0000 7f9bfb7fe640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9be40a2880 0x7f9be40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:47:59.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.771+0000 7f9bf8ff9640 1 -- 172.21.15.49:0/2146829295 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f9bf40f0b50 con 0x7f9bfc1022b0 2024-03-18T00:47:59.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.771+0000 7f9bfb7fe640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9be40a2880 0x7f9be40a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc101750 tx=0x7f9bf0005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:47:59.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:47:59.776+0000 7f9bf8ff9640 1 -- 172.21.15.49:0/2146829295 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9bf40be120 con 0x7f9bfc1022b0 2024-03-18T00:48:00.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.000+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bfc02d380 con 0x7f9bfc1022b0 2024-03-18T00:48:00.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.001+0000 7f9bf8ff9640 1 -- 172.21.15.49:0/2146829295 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f9bf40bda20 con 0x7f9bfc1022b0 2024-03-18T00:48:00.000 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:00.001 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T00:48:00.001 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 12 2024-03-18T00:48:00.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9be40a2880 msgr2=0x7f9be40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9be40a2880 0x7f9be40a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc101750 tx=0x7f9bf0005e70 comp rx=0 tx=0).stop 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 msgr2=0x7f9bfc19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 0x7f9bfc19f040 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9bf402c700 tx=0x7f9bf4032f20 comp rx=0 tx=0).stop 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 shutdown_connections 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9be40a2880 0x7f9be40a4d40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bfc1022b0 0x7f9bfc19f040 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bfc1018f0 0x7f9bfc197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 --2- 172.21.15.49:0/2146829295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bfc100700 0x7f9bfc197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.002+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 >> 172.21.15.49:0/2146829295 conn(0x7f9bfc0fbc70 msgr2=0x7f9bfc104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.003+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 shutdown_connections 2024-03-18T00:48:00.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:00.003+0000 7f9c02234640 1 -- 172.21.15.49:0/2146829295 wait complete. 2024-03-18T00:48:00.196 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:47:59 smithi106 podman[66841]: 2024-03-18 00:47:59.930096443 +0000 UTC m=+0.391214893 container init e291b8ea36c847a09c42cab7d772a0075c9564652e7c681de8e6f5afc1b4ca2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:48:00.196 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:47:59 smithi106 podman[66841]: 2024-03-18 00:47:59.932736499 +0000 UTC m=+0.393854940 container start e291b8ea36c847a09c42cab7d772a0075c9564652e7c681de8e6f5afc1b4ca2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T00:48:00.196 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:47:59 smithi106 ceph-mds[66860]: -- 172.21.15.106:0/4216500973 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556d2a323a40 con 0x556d2a2f7000 2024-03-18T00:48:00.196 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:47:59 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b[66856]: starting mds.b at 2024-03-18T00:48:00.196 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:48:00 smithi106 bash[66841]: e291b8ea36c847a09c42cab7d772a0075c9564652e7c681de8e6f5afc1b4ca2f 2024-03-18T00:48:00.196 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:48:00 smithi106 systemd[1]: Started Ceph mds.b for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T00:48:00.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:00 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2146829295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:48:00.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:00 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:00 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2146829295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:48:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:00 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:00 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2146829295' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:48:00.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:00.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:00 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:01.313 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:01 smithi106 ceph-mon[35899]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:01.313 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:01 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:boot 2024-03-18T00:48:01.313 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:01 smithi106 ceph-mon[35899]: fsmap 12 up:standby 2024-03-18T00:48:01.314 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:01 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T00:48:01.314 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:01 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:48:01.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:01 smithi049 ceph-mon[32130]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:01.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:01 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:boot 2024-03-18T00:48:01.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:01 smithi049 ceph-mon[32130]: fsmap 12 up:standby 2024-03-18T00:48:01.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T00:48:01.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:01 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:48:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:01 smithi107 ceph-mon[36270]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:01 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:boot 2024-03-18T00:48:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:01 smithi107 ceph-mon[36270]: fsmap 12 up:standby 2024-03-18T00:48:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T00:48:01.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:01 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:48:02.014 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:48:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 -- 172.21.15.49:0/927144521 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11401023c0 msgr2=0x7f11401047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 --2- 172.21.15.49:0/927144521 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11401023c0 0x7f11401047b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1134009900 tx=0x7f113402bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 -- 172.21.15.49:0/927144521 shutdown_connections 2024-03-18T00:48:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 --2- 172.21.15.49:0/927144521 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1140104cf0 0x7f11401070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 --2- 172.21.15.49:0/927144521 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1140107620 0x7f1140109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 --2- 172.21.15.49:0/927144521 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f11401023c0 0x7f11401047b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 -- 172.21.15.49:0/927144521 >> 172.21.15.49:0/927144521 conn(0x7f11400fbd10 msgr2=0x7f11400fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 -- 172.21.15.49:0/927144521 shutdown_connections 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 -- 172.21.15.49:0/927144521 wait complete. 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 Processor -- start 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 -- start start 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11401023c0 0x7f1140197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1140104cf0 0x7f1140198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1140107620 0x7f114019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f114010d510 con 0x7f1140107620 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f114010d390 con 0x7f1140104cf0 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.131+0000 7f1146b53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f114010d690 con 0x7f11401023c0 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11450f0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1140107620 0x7f114019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11450f0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1140107620 0x7f114019f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34624/0 (socket says 172.21.15.49:34624) 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11448ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11401023c0 0x7f1140197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11450f0640 1 -- 172.21.15.49:0/1575261172 learned_addr learned my addr 172.21.15.49:0/1575261172 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f113ffff640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1140104cf0 0x7f1140198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11450f0640 1 -- 172.21.15.49:0/1575261172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11401023c0 msgr2=0x7f1140197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11450f0640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11401023c0 0x7f1140197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11450f0640 1 -- 172.21.15.49:0/1575261172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1140104cf0 msgr2=0x7f1140198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11450f0640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1140104cf0 0x7f1140198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11450f0640 1 -- 172.21.15.49:0/1575261172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114019f770 con 0x7f1140107620 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f11450f0640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1140107620 0x7f114019f100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f113800eaa0 tx=0x7f113800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f113dffb640 1 -- 172.21.15.49:0/1575261172 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113800cdb0 con 0x7f1140107620 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f113dffb640 1 -- 172.21.15.49:0/1575261172 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1138005020 con 0x7f1140107620 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f114019fa60 con 0x7f1140107620 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f113dffb640 1 -- 172.21.15.49:0/1575261172 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11380116b0 con 0x7f1140107620 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.132+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11401a0000 con 0x7f1140107620 2024-03-18T00:48:02.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.133+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11400fd800 con 0x7f1140107620 2024-03-18T00:48:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.134+0000 7f113dffb640 1 -- 172.21.15.49:0/1575261172 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1138005560 con 0x7f1140107620 2024-03-18T00:48:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.135+0000 7f113dffb640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11300a2880 0x7f11300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.135+0000 7f11448ef640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11300a2880 0x7f11300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.135+0000 7f113dffb640 1 -- 172.21.15.49:0/1575261172 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f11380ce650 con 0x7f1140107620 2024-03-18T00:48:02.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.135+0000 7f11448ef640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11300a2880 0x7f11300a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f113402c240 tx=0x7f113402df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:02.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.139+0000 7f113dffb640 1 -- 172.21.15.49:0/1575261172 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1138016030 con 0x7f1140107620 2024-03-18T00:48:02.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.362+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f114002d360 con 0x7f1140107620 2024-03-18T00:48:02.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.362+0000 7f113dffb640 1 -- 172.21.15.49:0/1575261172 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10485 (secure 0 0 0) 0x7f113809b540 con 0x7f1140107620 2024-03-18T00:48:02.362 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:02.362 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14625,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24404,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24463,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24470,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24499,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-18T00:48:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 13 2024-03-18T00:48:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11300a2880 msgr2=0x7f11300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11300a2880 0x7f11300a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f113402c240 tx=0x7f113402df90 comp rx=0 tx=0).stop 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1140107620 msgr2=0x7f114019f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1140107620 0x7f114019f100 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f113800eaa0 tx=0x7f113800ef60 comp rx=0 tx=0).stop 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 shutdown_connections 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11300a2880 0x7f11300a4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1140104cf0 0x7f1140198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1140107620 0x7f114019f100 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 --2- 172.21.15.49:0/1575261172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11401023c0 0x7f1140197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.364+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 >> 172.21.15.49:0/1575261172 conn(0x7f11400fbd10 msgr2=0x7f1140102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.365+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 shutdown_connections 2024-03-18T00:48:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.365+0000 7f1146b53640 1 -- 172.21.15.49:0/1575261172 wait complete. 2024-03-18T00:48:02.376 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:48:02.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.486+0000 7f896d27e640 1 -- 172.21.15.49:0/2178865008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89681022b0 msgr2=0x7f8968108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.486+0000 7f896d27e640 1 --2- 172.21.15.49:0/2178865008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89681022b0 0x7f8968108b40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8960009ca0 tx=0x7f896002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.486+0000 7f896d27e640 1 -- 172.21.15.49:0/2178865008 shutdown_connections 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.486+0000 7f896d27e640 1 --2- 172.21.15.49:0/2178865008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f89681022b0 0x7f8968108b40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.486+0000 7f896d27e640 1 --2- 172.21.15.49:0/2178865008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89681018f0 0x7f8968101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.486+0000 7f896d27e640 1 --2- 172.21.15.49:0/2178865008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8968100700 0x7f8968100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.486+0000 7f896d27e640 1 -- 172.21.15.49:0/2178865008 >> 172.21.15.49:0/2178865008 conn(0x7f89680fbc70 msgr2=0x7f89680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 -- 172.21.15.49:0/2178865008 shutdown_connections 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 -- 172.21.15.49:0/2178865008 wait complete. 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 Processor -- start 2024-03-18T00:48:02.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 -- start start 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8968100700 0x7f896810b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89681018f0 0x7f89681096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89681022b0 0x7f8968109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f896810d460 con 0x7f8968100700 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f896810d2e0 con 0x7f89681022b0 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.487+0000 7f896d27e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f896810d5e0 con 0x7f89681018f0 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f8966ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8968100700 0x7f896810b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f8966ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8968100700 0x7f896810b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34630/0 (socket says 172.21.15.49:34630) 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f8966ffd640 1 -- 172.21.15.49:0/3985067619 learned_addr learned my addr 172.21.15.49:0/3985067619 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f89677fe640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89681022b0 0x7f8968109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f89667fc640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89681018f0 0x7f89681096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f8966ffd640 1 -- 172.21.15.49:0/3985067619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89681018f0 msgr2=0x7f89681096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f8966ffd640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89681018f0 0x7f89681096c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f8966ffd640 1 -- 172.21.15.49:0/3985067619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89681022b0 msgr2=0x7f8968109c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f8966ffd640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89681022b0 0x7f8968109c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f8966ffd640 1 -- 172.21.15.49:0/3985067619 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896810a490 con 0x7f8968100700 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f8966ffd640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8968100700 0x7f896810b080 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f895800e9c0 tx=0x7f895800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f894ffff640 1 -- 172.21.15.49:0/3985067619 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895800cd80 con 0x7f8968100700 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f894ffff640 1 -- 172.21.15.49:0/3985067619 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89580035f0 con 0x7f8968100700 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89681a1bc0 con 0x7f8968100700 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f894ffff640 1 -- 172.21.15.49:0/3985067619 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8958010690 con 0x7f8968100700 2024-03-18T00:48:02.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.488+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89681a2110 con 0x7f8968100700 2024-03-18T00:48:02.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.489+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89681079c0 con 0x7f8968100700 2024-03-18T00:48:02.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.490+0000 7f894ffff640 1 -- 172.21.15.49:0/3985067619 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f89580037a0 con 0x7f8968100700 2024-03-18T00:48:02.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.491+0000 7f894ffff640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89500a2800 0x7f89500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.491+0000 7f89667fc640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89500a2800 0x7f89500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.491+0000 7f894ffff640 1 -- 172.21.15.49:0/3985067619 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f89580cea80 con 0x7f8968100700 2024-03-18T00:48:02.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.492+0000 7f89667fc640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89500a2800 0x7f89500a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f896810adf0 tx=0x7f895c005c60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:02.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.496+0000 7f894ffff640 1 -- 172.21.15.49:0/3985067619 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f895809bfc0 con 0x7f8968100700 2024-03-18T00:48:02.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:02 smithi049 ceph-mon[32130]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:02.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:02 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1575261172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:48:02.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.720+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f896802d460 con 0x7f8968100700 2024-03-18T00:48:02.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.721+0000 7f894ffff640 1 -- 172.21.15.49:0/3985067619 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18443 (secure 0 0 0) 0x7f895809b8c0 con 0x7f8968100700 2024-03-18T00:48:02.721 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:02.721 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":82,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:47:16.556495+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89500a2800 msgr2=0x7f89500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89500a2800 0x7f89500a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f896810adf0 tx=0x7f895c005c60 comp rx=0 tx=0).stop 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8968100700 msgr2=0x7f896810b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8968100700 0x7f896810b080 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f895800e9c0 tx=0x7f895800ee80 comp rx=0 tx=0).stop 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 shutdown_connections 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f89500a2800 0x7f89500a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f89681022b0 0x7f8968109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89681018f0 0x7f89681096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 --2- 172.21.15.49:0/3985067619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8968100700 0x7f896810b080 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 >> 172.21.15.49:0/3985067619 conn(0x7f89680fbc70 msgr2=0x7f8968104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:02.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 shutdown_connections 2024-03-18T00:48:02.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.723+0000 7f896d27e640 1 -- 172.21.15.49:0/3985067619 wait complete. 2024-03-18T00:48:02.735 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:48:02.735 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:48:02.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 -- 172.21.15.49:0/1260098441 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 msgr2=0x7fc5b4100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 --2- 172.21.15.49:0/1260098441 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 0x7fc5b4100b00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009900 tx=0x7fc5b002bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 -- 172.21.15.49:0/1260098441 shutdown_connections 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 --2- 172.21.15.49:0/1260098441 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b41018d0 0x7fc5b4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 --2- 172.21.15.49:0/1260098441 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b4102290 0x7fc5b4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 --2- 172.21.15.49:0/1260098441 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 0x7fc5b4100b00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 -- 172.21.15.49:0/1260098441 >> 172.21.15.49:0/1260098441 conn(0x7fc5b40fbc70 msgr2=0x7fc5b40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 -- 172.21.15.49:0/1260098441 shutdown_connections 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 -- 172.21.15.49:0/1260098441 wait complete. 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 Processor -- start 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.848+0000 7fc5bc748640 1 -- start start 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5bc748640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 0x7fc5b4199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5bc748640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b41018d0 0x7fc5b419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5bc748640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b4102290 0x7fc5b41a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5bc748640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b410c360 con 0x7fc5b41006e0 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5bc748640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b410c1e0 con 0x7fc5b4102290 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5bc748640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b410c4e0 con 0x7fc5b41018d0 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ba4e4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 0x7fc5b4199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ba4e4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 0x7fc5b4199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34650/0 (socket says 172.21.15.49:34650) 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ba4e4640 1 -- 172.21.15.49:0/617278611 learned_addr learned my addr 172.21.15.49:0/617278611 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5bace5640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b4102290 0x7fc5b41a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5b9ce3640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b41018d0 0x7fc5b419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ba4e4640 1 -- 172.21.15.49:0/617278611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b41018d0 msgr2=0x7fc5b419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ba4e4640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b41018d0 0x7fc5b419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ba4e4640 1 -- 172.21.15.49:0/617278611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b4102290 msgr2=0x7fc5b41a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ba4e4640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b4102290 0x7fc5b41a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:02.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ba4e4640 1 -- 172.21.15.49:0/617278611 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b41a1950 con 0x7fc5b41006e0 2024-03-18T00:48:02.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ba4e4640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 0x7fc5b4199d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0000c00 tx=0x7fc5b0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:02.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ab7fe640 1 -- 172.21.15.49:0/617278611 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0041070 con 0x7fc5b41006e0 2024-03-18T00:48:02.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b41a1be0 con 0x7fc5b41006e0 2024-03-18T00:48:02.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ab7fe640 1 -- 172.21.15.49:0/617278611 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b00044d0 con 0x7fc5b41006e0 2024-03-18T00:48:02.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5ab7fe640 1 -- 172.21.15.49:0/617278611 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0034440 con 0x7fc5b41006e0 2024-03-18T00:48:02.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.849+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b41a2120 con 0x7fc5b41006e0 2024-03-18T00:48:02.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.851+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b41a22f0 con 0x7fc5b41006e0 2024-03-18T00:48:02.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.851+0000 7fc5ab7fe640 1 -- 172.21.15.49:0/617278611 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc5b0044070 con 0x7fc5b41006e0 2024-03-18T00:48:02.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.852+0000 7fc5ab7fe640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5a00a2800 0x7fc5a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:02.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.852+0000 7fc5ab7fe640 1 -- 172.21.15.49:0/617278611 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fc5b00f0140 con 0x7fc5b41006e0 2024-03-18T00:48:02.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.852+0000 7fc5b9ce3640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5a00a2800 0x7fc5a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:02.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.853+0000 7fc5b9ce3640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5a00a2800 0x7fc5a00a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4005d00 tx=0x7fc5a400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:02.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:02.857+0000 7fc5ab7fe640 1 -- 172.21.15.49:0/617278611 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc5b00bd030 con 0x7fc5b41006e0 2024-03-18T00:48:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:02 smithi107 ceph-mon[36270]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:02 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1575261172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:48:02.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:02 smithi106 ceph-mon[35899]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:02.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:02.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:02 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1575261172' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:48:03.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.041+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc5b41a24c0 con 0x7fc5b41006e0 2024-03-18T00:48:03.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.041+0000 7fc5ab7fe640 1 -- 172.21.15.49:0/617278611 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fc5b00c10d0 con 0x7fc5b41006e0 2024-03-18T00:48:03.041 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:48:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5a00a2800 msgr2=0x7fc5a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5a00a2800 0x7fc5a00a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc5a4005d00 tx=0x7fc5a400b410 comp rx=0 tx=0).stop 2024-03-18T00:48:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 msgr2=0x7fc5b4199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 0x7fc5b4199d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0000c00 tx=0x7fc5b0005d50 comp rx=0 tx=0).stop 2024-03-18T00:48:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 shutdown_connections 2024-03-18T00:48:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc5a00a2800 0x7fc5a00a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc5b4102290 0x7fc5b41a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5b41018d0 0x7fc5b419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 --2- 172.21.15.49:0/617278611 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc5b41006e0 0x7fc5b4199d40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 >> 172.21.15.49:0/617278611 conn(0x7fc5b40fbc70 msgr2=0x7fc5b40fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 shutdown_connections 2024-03-18T00:48:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.043+0000 7fc5bc748640 1 -- 172.21.15.49:0/617278611 wait complete. 2024-03-18T00:48:03.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T00:48:03.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.172+0000 7f9c17c0b640 1 -- 172.21.15.49:0/2753936704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101018f0 msgr2=0x7f9c10101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.172+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/2753936704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101018f0 0x7f9c10101d70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009900 tx=0x7f9c0002bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.172+0000 7f9c17c0b640 1 -- 172.21.15.49:0/2753936704 shutdown_connections 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.172+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/2753936704 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c101022b0 0x7f9c10108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.172+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/2753936704 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101018f0 0x7f9c10101d70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.172+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/2753936704 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c10100700 0x7f9c10100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.172+0000 7f9c17c0b640 1 -- 172.21.15.49:0/2753936704 >> 172.21.15.49:0/2753936704 conn(0x7f9c100fbc70 msgr2=0x7f9c100fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 -- 172.21.15.49:0/2753936704 shutdown_connections 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 -- 172.21.15.49:0/2753936704 wait complete. 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 Processor -- start 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 -- start start 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c10100700 0x7f9c10104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:03.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c101018f0 0x7f9c10102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101022b0 0x7f9c10103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1010c380 con 0x7f9c101022b0 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1010c200 con 0x7f9c10100700 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c17c0b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9c1010c500 con 0x7f9c101018f0 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.173+0000 7f9c161a8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101022b0 0x7f9c10103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c161a8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101022b0 0x7f9c10103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34666/0 (socket says 172.21.15.49:34666) 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c161a8640 1 -- 172.21.15.49:0/4192260331 learned_addr learned my addr 172.21.15.49:0/4192260331 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c151a6640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c101018f0 0x7f9c10102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c159a7640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c10100700 0x7f9c10104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c161a8640 1 -- 172.21.15.49:0/4192260331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c101018f0 msgr2=0x7f9c10102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c161a8640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c101018f0 0x7f9c10102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c161a8640 1 -- 172.21.15.49:0/4192260331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c10100700 msgr2=0x7f9c10104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c161a8640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c10100700 0x7f9c10104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c161a8640 1 -- 172.21.15.49:0/4192260331 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c10103aa0 con 0x7f9c101022b0 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c151a6640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c101018f0 0x7f9c10102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:03.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c161a8640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101022b0 0x7f9c10103210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800eb10 tx=0x7f9c0800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:03.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c06ffd640 1 -- 172.21.15.49:0/4192260331 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08004b20 con 0x7f9c101022b0 2024-03-18T00:48:03.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c101a1c40 con 0x7f9c101022b0 2024-03-18T00:48:03.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c06ffd640 1 -- 172.21.15.49:0/4192260331 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c08004cc0 con 0x7f9c101022b0 2024-03-18T00:48:03.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c06ffd640 1 -- 172.21.15.49:0/4192260331 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08010760 con 0x7f9c101022b0 2024-03-18T00:48:03.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.174+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c101a2160 con 0x7f9c101022b0 2024-03-18T00:48:03.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.175+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c100685d0 con 0x7f9c101022b0 2024-03-18T00:48:03.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.176+0000 7f9c06ffd640 1 -- 172.21.15.49:0/4192260331 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9c08010900 con 0x7f9c101022b0 2024-03-18T00:48:03.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.177+0000 7f9c06ffd640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bfc0a2800 0x7f9bfc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:03.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.177+0000 7f9c06ffd640 1 -- 172.21.15.49:0/4192260331 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f9c080ced50 con 0x7f9c101022b0 2024-03-18T00:48:03.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.177+0000 7f9c159a7640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bfc0a2800 0x7f9bfc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:03.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.178+0000 7f9c159a7640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bfc0a2800 0x7f9bfc0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9c10101750 tx=0x7f9c0c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:03.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.181+0000 7f9c06ffd640 1 -- 172.21.15.49:0/4192260331 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9c08053740 con 0x7f9c101022b0 2024-03-18T00:48:03.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.399+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9c101a25a0 con 0x7f9c101022b0 2024-03-18T00:48:03.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.399+0000 7f9c06ffd640 1 -- 172.21.15.49:0/4192260331 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7f9c0809c290 con 0x7f9c101022b0 2024-03-18T00:48:03.547 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:03.547 INFO:teuthology.orchestra.run.smithi049.stdout:[ 2024-03-18T00:48:03.547 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T00:48:03.547 INFO:teuthology.orchestra.run.smithi049.stdout: "poolnum": 1, 2024-03-18T00:48:03.547 INFO:teuthology.orchestra.run.smithi049.stdout: "poolname": ".mgr" 2024-03-18T00:48:03.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:48:03.547 INFO:teuthology.orchestra.run.smithi049.stdout:] 2024-03-18T00:48:03.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bfc0a2800 msgr2=0x7f9bfc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bfc0a2800 0x7f9bfc0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9c10101750 tx=0x7f9c0c005f90 comp rx=0 tx=0).stop 2024-03-18T00:48:03.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101022b0 msgr2=0x7f9c10103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101022b0 0x7f9c10103210 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9c0800eb10 tx=0x7f9c0800c2d0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 shutdown_connections 2024-03-18T00:48:03.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bfc0a2800 0x7f9bfc0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9c101022b0 0x7f9c10103210 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9c101018f0 0x7f9c10102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 --2- 172.21.15.49:0/4192260331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9c10100700 0x7f9c10104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 >> 172.21.15.49:0/4192260331 conn(0x7f9c100fbc70 msgr2=0x7f9c100fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:03.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 shutdown_connections 2024-03-18T00:48:03.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.401+0000 7f9c17c0b640 1 -- 172.21.15.49:0/4192260331 wait complete. 2024-03-18T00:48:03.549 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T00:48:03.549 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T00:48:03.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 -- 172.21.15.49:0/3734289547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01033a0 msgr2=0x7f12e0109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 --2- 172.21.15.49:0/3734289547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01033a0 0x7f12e0109c30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f12d8009ca0 tx=0x7f12d802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 -- 172.21.15.49:0/3734289547 shutdown_connections 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 --2- 172.21.15.49:0/3734289547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12e01029e0 0x7f12e0102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 --2- 172.21.15.49:0/3734289547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01033a0 0x7f12e0109c30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 --2- 172.21.15.49:0/3734289547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12e01017f0 0x7f12e0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 -- 172.21.15.49:0/3734289547 >> 172.21.15.49:0/3734289547 conn(0x7f12e00fcd80 msgr2=0x7f12e00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 -- 172.21.15.49:0/3734289547 shutdown_connections 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 -- 172.21.15.49:0/3734289547 wait complete. 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 Processor -- start 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.665+0000 7f12e7aff640 1 -- start start 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e7aff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12e01017f0 0x7f12e0075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e7aff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01029e0 0x7f12e0073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e7aff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12e01033a0 0x7f12e0074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:03.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e7aff640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f12e010d450 con 0x7f12e01029e0 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e7aff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f12e010d2d0 con 0x7f12e01033a0 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e7aff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f12e010d5d0 con 0x7f12e01017f0 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e509a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01029e0 0x7f12e0073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e509a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01029e0 0x7f12e0073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34678/0 (socket says 172.21.15.49:34678) 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e509a640 1 -- 172.21.15.49:0/2284200435 learned_addr learned my addr 172.21.15.49:0/2284200435 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e589b640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12e01017f0 0x7f12e0075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e609c640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12e01033a0 0x7f12e0074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e509a640 1 -- 172.21.15.49:0/2284200435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12e01017f0 msgr2=0x7f12e0075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e509a640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12e01017f0 0x7f12e0075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e509a640 1 -- 172.21.15.49:0/2284200435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12e01033a0 msgr2=0x7f12e0074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e509a640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12e01033a0 0x7f12e0074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e509a640 1 -- 172.21.15.49:0/2284200435 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e00749a0 con 0x7f12e01029e0 2024-03-18T00:48:03.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e509a640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01029e0 0x7f12e0073bd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f12d000c920 tx=0x7f12d000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:03.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12d6ffd640 1 -- 172.21.15.49:0/2284200435 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d0007c70 con 0x7f12e01029e0 2024-03-18T00:48:03.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12d6ffd640 1 -- 172.21.15.49:0/2284200435 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12d0007e10 con 0x7f12e01029e0 2024-03-18T00:48:03.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12e01a1b50 con 0x7f12e01029e0 2024-03-18T00:48:03.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12d6ffd640 1 -- 172.21.15.49:0/2284200435 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12d0018890 con 0x7f12e01029e0 2024-03-18T00:48:03.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.666+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12e01a20a0 con 0x7f12e01029e0 2024-03-18T00:48:03.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.667+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12e0108ab0 con 0x7f12e01029e0 2024-03-18T00:48:03.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.669+0000 7f12d6ffd640 1 -- 172.21.15.49:0/2284200435 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f12d0018a50 con 0x7f12e01029e0 2024-03-18T00:48:03.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.670+0000 7f12d6ffd640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:03.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.670+0000 7f12e589b640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:03.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.670+0000 7f12d6ffd640 1 -- 172.21.15.49:0/2284200435 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f12d00ce8e0 con 0x7f12e01029e0 2024-03-18T00:48:03.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.670+0000 7f12e589b640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12dc005fd0 tx=0x7f12dc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:03.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.673+0000 7f12d6ffd640 1 -- 172.21.15.49:0/2284200435 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f12d0014030 con 0x7f12e01029e0 2024-03-18T00:48:03.716 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:48:03.716 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:48:03.716 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:03.717 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:48:03.717 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:03 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3985067619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:03.717 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:03 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/617278611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:48:03.717 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:03 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/4192260331' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T00:48:03.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:48:03.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:48:03.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:03.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:48:03.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:03 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3985067619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:03.722 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:03 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/617278611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:48:03.723 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:03 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/4192260331' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T00:48:03.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:03.858+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f12e002d250 con 0x7f12e01029e0 2024-03-18T00:48:03.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:48:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:48:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:48:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:03 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3985067619' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:03 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/617278611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:48:03.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:03 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/4192260331' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T00:48:04.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.505+0000 7f12d6ffd640 1 -- 172.21.15.49:0/2284200435 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f12d009beb0 con 0x7f12e01029e0 2024-03-18T00:48:04.506 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'cephfs_metadata' created 2024-03-18T00:48:04.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12cc0a2800 msgr2=0x7f12cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:04.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12dc005fd0 tx=0x7f12dc00b410 comp rx=0 tx=0).stop 2024-03-18T00:48:04.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01029e0 msgr2=0x7f12e0073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:04.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01029e0 0x7f12e0073bd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f12d000c920 tx=0x7f12d000cde0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 shutdown_connections 2024-03-18T00:48:04.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12cc0a2800 0x7f12cc0a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12e01033a0 0x7f12e0074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12e01029e0 0x7f12e0073bd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 --2- 172.21.15.49:0/2284200435 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12e01017f0 0x7f12e0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 >> 172.21.15.49:0/2284200435 conn(0x7f12e00fcd80 msgr2=0x7f12e0105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:04.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 shutdown_connections 2024-03-18T00:48:04.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.509+0000 7f12e7aff640 1 -- 172.21.15.49:0/2284200435 wait complete. 2024-03-18T00:48:04.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T00:48:04.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 -- 172.21.15.49:0/3209569543 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101d10 msgr2=0x7f99c8102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:04.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3209569543 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101d10 0x7f99c8102130 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f99c4009900 tx=0x7f99c402bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:04.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 -- 172.21.15.49:0/3209569543 shutdown_connections 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3209569543 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99c8100960 0x7f99c8100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3209569543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c8101320 0x7f99c8108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3209569543 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101d10 0x7f99c8102130 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 -- 172.21.15.49:0/3209569543 >> 172.21.15.49:0/3209569543 conn(0x7f99c80fb410 msgr2=0x7f99c80fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 -- 172.21.15.49:0/3209569543 shutdown_connections 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 -- 172.21.15.49:0/3209569543 wait complete. 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 Processor -- start 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 -- start start 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99c8100960 0x7f99c81003c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:04.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.638+0000 7f99cfb52640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101320 0x7f99c819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cfb52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c8101d10 0x7f99c819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cfb52640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f99c810c460 con 0x7f99c8101320 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cfb52640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f99c810c2e0 con 0x7f99c8100960 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cfb52640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f99c810c5e0 con 0x7f99c8101d10 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd0ed640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101320 0x7f99c819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd0ed640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101320 0x7f99c819c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34696/0 (socket says 172.21.15.49:34696) 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd0ed640 1 -- 172.21.15.49:0/3438202879 learned_addr learned my addr 172.21.15.49:0/3438202879 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd8ee640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99c8100960 0x7f99c81003c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99ce0ef640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c8101d10 0x7f99c819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd0ed640 1 -- 172.21.15.49:0/3438202879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c8101d10 msgr2=0x7f99c819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:04.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd0ed640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c8101d10 0x7f99c819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd0ed640 1 -- 172.21.15.49:0/3438202879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99c8100960 msgr2=0x7f99c81003c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd0ed640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99c8100960 0x7f99c81003c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd0ed640 1 -- 172.21.15.49:0/3438202879 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c819f850 con 0x7f99c8101320 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cd0ed640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101320 0x7f99c819c920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f99b800b450 tx=0x7f99b800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99beffd640 1 -- 172.21.15.49:0/3438202879 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8015020 con 0x7f99c8101320 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99beffd640 1 -- 172.21.15.49:0/3438202879 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99b8011d70 con 0x7f99c8101320 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c819fae0 con 0x7f99c8101320 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99beffd640 1 -- 172.21.15.49:0/3438202879 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b800f340 con 0x7f99c8101320 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.639+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c81a0030 con 0x7f99c8101320 2024-03-18T00:48:04.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.640+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c8107bc0 con 0x7f99c8101320 2024-03-18T00:48:04.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.641+0000 7f99beffd640 1 -- 172.21.15.49:0/3438202879 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f99b8018020 con 0x7f99c8101320 2024-03-18T00:48:04.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.642+0000 7f99beffd640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f99b40a2800 0x7f99b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:04.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.642+0000 7f99cd8ee640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f99b40a2800 0x7f99b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:04.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.642+0000 7f99beffd640 1 -- 172.21.15.49:0/3438202879 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7f99b80ce190 con 0x7f99c8101320 2024-03-18T00:48:04.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.643+0000 7f99cd8ee640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f99b40a2800 0x7f99b40a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f99c402c170 tx=0x7f99c402df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:04.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.647+0000 7f99beffd640 1 -- 172.21.15.49:0/3438202879 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f99b801f020 con 0x7f99c8101320 2024-03-18T00:48:04.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:04.833+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f99c802d1b0 con 0x7f99c8101320 2024-03-18T00:48:04.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2284200435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:48:04.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:04 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:04.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2284200435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T00:48:04.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:48:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:48:04.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:04 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2284200435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:48:04.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:04 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:48:05.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.511+0000 7f99beffd640 1 -- 172.21.15.49:0/3438202879 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7f99b800d040 con 0x7f99c8101320 2024-03-18T00:48:05.511 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'cephfs_data' created 2024-03-18T00:48:05.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.514+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f99b40a2800 msgr2=0x7f99b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:05.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.514+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f99b40a2800 0x7f99b40a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f99c402c170 tx=0x7f99c402df90 comp rx=0 tx=0).stop 2024-03-18T00:48:05.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.514+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101320 msgr2=0x7f99c819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:05.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.514+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101320 0x7f99c819c920 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f99b800b450 tx=0x7f99b800b910 comp rx=0 tx=0).stop 2024-03-18T00:48:05.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.514+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 shutdown_connections 2024-03-18T00:48:05.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.514+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f99b40a2800 0x7f99b40a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:05.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.514+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f99c8101d10 0x7f99c819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:05.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.514+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f99c8101320 0x7f99c819c920 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:05.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.514+0000 7f99cfb52640 1 --2- 172.21.15.49:0/3438202879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f99c8100960 0x7f99c81003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:05.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.515+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 >> 172.21.15.49:0/3438202879 conn(0x7f99c80fb410 msgr2=0x7f99c8104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:05.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.515+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 shutdown_connections 2024-03-18T00:48:05.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.515+0000 7f99cfb52640 1 -- 172.21.15.49:0/3438202879 wait complete. 2024-03-18T00:48:05.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T00:48:05.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.647+0000 7f729cd74640 1 -- 172.21.15.49:0/1892625313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 msgr2=0x7f7298100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:05.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.647+0000 7f729cd74640 1 --2- 172.21.15.49:0/1892625313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 0x7f7298100b20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7288009900 tx=0x7f728802bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:05.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.647+0000 7f729cd74640 1 -- 172.21.15.49:0/1892625313 shutdown_connections 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.647+0000 7f729cd74640 1 --2- 172.21.15.49:0/1892625313 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72981018f0 0x7f7298101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.647+0000 7f729cd74640 1 --2- 172.21.15.49:0/1892625313 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72981022b0 0x7f7298108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.647+0000 7f729cd74640 1 --2- 172.21.15.49:0/1892625313 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 0x7f7298100b20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.647+0000 7f729cd74640 1 -- 172.21.15.49:0/1892625313 >> 172.21.15.49:0/1892625313 conn(0x7f72980fbc90 msgr2=0x7f72980fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.647+0000 7f729cd74640 1 -- 172.21.15.49:0/1892625313 shutdown_connections 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.647+0000 7f729cd74640 1 -- 172.21.15.49:0/1892625313 wait complete. 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729cd74640 1 Processor -- start 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729cd74640 1 -- start start 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729cd74640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 0x7f7298197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729cd74640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72981018f0 0x7f72981980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729cd74640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72981022b0 0x7f729819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:05.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729cd74640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f729810c350 con 0x7f7298100700 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729cd74640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f729810c1d0 con 0x7f72981018f0 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729cd74640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f729810c4d0 con 0x7f72981022b0 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 0x7f7298197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 0x7f7298197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34700/0 (socket says 172.21.15.49:34700) 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.648+0000 7f729659c640 1 -- 172.21.15.49:0/2388392811 learned_addr learned my addr 172.21.15.49:0/2388392811 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f7295d9b640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72981018f0 0x7f72981980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f7296d9d640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72981022b0 0x7f729819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f729659c640 1 -- 172.21.15.49:0/2388392811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72981022b0 msgr2=0x7f729819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f729659c640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72981022b0 0x7f729819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f729659c640 1 -- 172.21.15.49:0/2388392811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72981018f0 msgr2=0x7f72981980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f729659c640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72981018f0 0x7f72981980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f729659c640 1 -- 172.21.15.49:0/2388392811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729819f770 con 0x7f7298100700 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f729659c640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 0x7f7298197ba0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7288000c00 tx=0x7f7288005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:05.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f72877fe640 1 -- 172.21.15.49:0/2388392811 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7288041070 con 0x7f7298100700 2024-03-18T00:48:05.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f72877fe640 1 -- 172.21.15.49:0/2388392811 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72880044d0 con 0x7f7298100700 2024-03-18T00:48:05.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729819fa00 con 0x7f7298100700 2024-03-18T00:48:05.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f72877fe640 1 -- 172.21.15.49:0/2388392811 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7288034440 con 0x7f7298100700 2024-03-18T00:48:05.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.649+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729819fef0 con 0x7f7298100700 2024-03-18T00:48:05.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.650+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72981a00c0 con 0x7f7298100700 2024-03-18T00:48:05.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.651+0000 7f72877fe640 1 -- 172.21.15.49:0/2388392811 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7288044070 con 0x7f7298100700 2024-03-18T00:48:05.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.652+0000 7f72877fe640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72800a2800 0x7f72800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:05.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.652+0000 7f7295d9b640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72800a2800 0x7f72800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:05.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.652+0000 7f72877fe640 1 -- 172.21.15.49:0/2388392811 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f72880f0450 con 0x7f7298100700 2024-03-18T00:48:05.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.652+0000 7f7295d9b640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72800a2800 0x7f72800a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7298199060 tx=0x7f728c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:05.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.657+0000 7f72877fe640 1 -- 172.21.15.49:0/2388392811 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f72880bd6a0 con 0x7f7298100700 2024-03-18T00:48:05.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:05 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2284200435' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T00:48:05.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:05 smithi049 ceph-mon[32130]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T00:48:05.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:05 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3438202879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T00:48:05.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:05.880+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f72981a0290 con 0x7f7298100700 2024-03-18T00:48:05.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:05 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2284200435' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T00:48:05.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:05 smithi107 ceph-mon[36270]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T00:48:05.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:05 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3438202879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T00:48:05.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:05 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2284200435' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T00:48:05.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:05 smithi106 ceph-mon[35899]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T00:48:05.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:05 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3438202879' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T00:48:06.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.513+0000 7f72877fe640 1 -- 172.21.15.49:0/2388392811 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f72880bcfa0 con 0x7f7298100700 2024-03-18T00:48:06.513 INFO:teuthology.orchestra.run.smithi049.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T00:48:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.516+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72800a2800 msgr2=0x7f72800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.516+0000 7f729cd74640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72800a2800 0x7f72800a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7298199060 tx=0x7f728c005f90 comp rx=0 tx=0).stop 2024-03-18T00:48:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.516+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 msgr2=0x7f7298197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.516+0000 7f729cd74640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 0x7f7298197ba0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7288000c00 tx=0x7f7288005d50 comp rx=0 tx=0).stop 2024-03-18T00:48:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.517+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 shutdown_connections 2024-03-18T00:48:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.517+0000 7f729cd74640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72800a2800 0x7f72800a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.517+0000 7f729cd74640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72981018f0 0x7f72981980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.517+0000 7f729cd74640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72981022b0 0x7f729819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.517+0000 7f729cd74640 1 --2- 172.21.15.49:0/2388392811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7298100700 0x7f7298197ba0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.517+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 >> 172.21.15.49:0/2388392811 conn(0x7f72980fbc90 msgr2=0x7f72980fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:06.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.517+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 shutdown_connections 2024-03-18T00:48:06.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.517+0000 7f729cd74640 1 -- 172.21.15.49:0/2388392811 wait complete. 2024-03-18T00:48:06.538 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:48:06.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.654+0000 7f37b32ac640 1 -- 172.21.15.49:0/748750431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 msgr2=0x7f37ac102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.654+0000 7f37b32ac640 1 --2- 172.21.15.49:0/748750431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 0x7f37ac102e80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f379c009900 tx=0x7f379c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:06.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.654+0000 7f37b32ac640 1 -- 172.21.15.49:0/748750431 shutdown_connections 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.654+0000 7f37b32ac640 1 --2- 172.21.15.49:0/748750431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37ac1033c0 0x7f37ac109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.654+0000 7f37b32ac640 1 --2- 172.21.15.49:0/748750431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 0x7f37ac102e80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.654+0000 7f37b32ac640 1 --2- 172.21.15.49:0/748750431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37ac101810 0x7f37ac101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.654+0000 7f37b32ac640 1 -- 172.21.15.49:0/748750431 >> 172.21.15.49:0/748750431 conn(0x7f37ac0fcdc0 msgr2=0x7f37ac0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.654+0000 7f37b32ac640 1 -- 172.21.15.49:0/748750431 shutdown_connections 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.654+0000 7f37b32ac640 1 -- 172.21.15.49:0/748750431 wait complete. 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b32ac640 1 Processor -- start 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b32ac640 1 -- start start 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b32ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37ac101810 0x7f37ac197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b32ac640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 0x7f37ac198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b32ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37ac1033c0 0x7f37ac19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b32ac640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac10d4f0 con 0x7f37ac102a00 2024-03-18T00:48:06.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b32ac640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac10d370 con 0x7f37ac101810 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b32ac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f37ac10d670 con 0x7f37ac1033c0 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b0847640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 0x7f37ac198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b0847640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 0x7f37ac198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34718/0 (socket says 172.21.15.49:34718) 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b0847640 1 -- 172.21.15.49:0/219631308 learned_addr learned my addr 172.21.15.49:0/219631308 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b1048640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37ac101810 0x7f37ac197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b0847640 1 -- 172.21.15.49:0/219631308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37ac1033c0 msgr2=0x7f37ac19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b0847640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37ac1033c0 0x7f37ac19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b0847640 1 -- 172.21.15.49:0/219631308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37ac101810 msgr2=0x7f37ac197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b0847640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37ac101810 0x7f37ac197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b0847640 1 -- 172.21.15.49:0/219631308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37ac19f710 con 0x7f37ac102a00 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.655+0000 7f37b1048640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37ac101810 0x7f37ac197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.656+0000 7f37b0847640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 0x7f37ac198080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f379c009230 tx=0x7f379c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:06.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.656+0000 7f37a27fc640 1 -- 172.21.15.49:0/219631308 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c041070 con 0x7f37ac102a00 2024-03-18T00:48:06.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.656+0000 7f37a27fc640 1 -- 172.21.15.49:0/219631308 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f379c0054e0 con 0x7f37ac102a00 2024-03-18T00:48:06.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.656+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37ac19f9a0 con 0x7f37ac102a00 2024-03-18T00:48:06.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.656+0000 7f37a27fc640 1 -- 172.21.15.49:0/219631308 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379c0346a0 con 0x7f37ac102a00 2024-03-18T00:48:06.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.656+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37ac19fee0 con 0x7f37ac102a00 2024-03-18T00:48:06.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.657+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37ac108a60 con 0x7f37ac102a00 2024-03-18T00:48:06.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.658+0000 7f37a27fc640 1 -- 172.21.15.49:0/219631308 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f379c044070 con 0x7f37ac102a00 2024-03-18T00:48:06.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.659+0000 7f37a27fc640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37980a2880 0x7f37980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:06.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.659+0000 7f37b1048640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37980a2880 0x7f37980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:06.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.659+0000 7f37a27fc640 1 -- 172.21.15.49:0/219631308 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f379c0f0aa0 con 0x7f37ac102a00 2024-03-18T00:48:06.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.659+0000 7f37b1048640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37980a2880 0x7f37980a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f37ac102860 tx=0x7f37a8005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:06.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.662+0000 7f37a27fc640 1 -- 172.21.15.49:0/219631308 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f379c0bdc90 con 0x7f37ac102a00 2024-03-18T00:48:06.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.847+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37ac02d3c0 con 0x7f37ac102a00 2024-03-18T00:48:06.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.849+0000 7f37a27fc640 1 -- 172.21.15.49:0/219631308 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22219 (secure 0 0 0) 0x7f379c0bd590 con 0x7f37ac102a00 2024-03-18T00:48:06.849 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:06.849 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":85,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:48:05.881439+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37980a2880 msgr2=0x7f37980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37980a2880 0x7f37980a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f37ac102860 tx=0x7f37a8005db0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 msgr2=0x7f37ac198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 0x7f37ac198080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f379c009230 tx=0x7f379c004040 comp rx=0 tx=0).stop 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 shutdown_connections 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f37980a2880 0x7f37980a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37ac1033c0 0x7f37ac19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f37ac102a00 0x7f37ac198080 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 --2- 172.21.15.49:0/219631308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f37ac101810 0x7f37ac197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 >> 172.21.15.49:0/219631308 conn(0x7f37ac0fcdc0 msgr2=0x7f37ac105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:06.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 shutdown_connections 2024-03-18T00:48:06.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.851+0000 7f37b32ac640 1 -- 172.21.15.49:0/219631308 wait complete. 2024-03-18T00:48:06.860 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:06 smithi049 ceph-mon[32130]: pgmap v285: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:06.860 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:06 smithi049 ceph-mon[32130]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T00:48:06.860 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:06 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3438202879' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T00:48:06.860 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:06 smithi049 ceph-mon[32130]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T00:48:06.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:06 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2388392811' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T00:48:06.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:06 smithi049 ceph-mon[32130]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T00:48:06.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:48:06.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:06 smithi107 ceph-mon[36270]: pgmap v285: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:06.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:06 smithi107 ceph-mon[36270]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T00:48:06.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:06 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3438202879' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T00:48:06.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:06 smithi107 ceph-mon[36270]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T00:48:06.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:06 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2388392811' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T00:48:06.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:06 smithi107 ceph-mon[36270]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T00:48:06.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:06 smithi106 ceph-mon[35899]: pgmap v285: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T00:48:06.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:06 smithi106 ceph-mon[35899]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T00:48:06.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:06 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3438202879' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T00:48:06.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:06 smithi106 ceph-mon[35899]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T00:48:06.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:06 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2388392811' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T00:48:06.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:06 smithi106 ceph-mon[35899]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T00:48:06.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 -- 172.21.15.49:0/1044725164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 msgr2=0x7fcd04108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/1044725164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 0x7fcd04108b40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009ca0 tx=0x7fccfc02c0f0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 -- 172.21.15.49:0/1044725164 shutdown_connections 2024-03-18T00:48:06.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/1044725164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 0x7fcd04108b40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/1044725164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd041018f0 0x7fcd04101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/1044725164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd04100700 0x7fcd04100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 -- 172.21.15.49:0/1044725164 >> 172.21.15.49:0/1044725164 conn(0x7fcd040fbc70 msgr2=0x7fcd040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:06.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 -- 172.21.15.49:0/1044725164 shutdown_connections 2024-03-18T00:48:06.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 -- 172.21.15.49:0/1044725164 wait complete. 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 Processor -- start 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 -- start start 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd04100700 0x7fcd04104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd041018f0 0x7fcd04102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 0x7fcd04103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.970+0000 7fcd0c27e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0410c360 con 0x7fcd041022b0 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0c27e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0410c1e0 con 0x7fcd041018f0 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0c27e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0410c4e0 con 0x7fcd04100700 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a81b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 0x7fcd04103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a81b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 0x7fcd04103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34734/0 (socket says 172.21.15.49:34734) 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a81b640 1 -- 172.21.15.49:0/3518154359 learned_addr learned my addr 172.21.15.49:0/3518154359 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd09819640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd041018f0 0x7fcd04102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a01a640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd04100700 0x7fcd04104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:06.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a81b640 1 -- 172.21.15.49:0/3518154359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd04100700 msgr2=0x7fcd04104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a81b640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd04100700 0x7fcd04104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a81b640 1 -- 172.21.15.49:0/3518154359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd041018f0 msgr2=0x7fcd04102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a81b640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd041018f0 0x7fcd04102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a81b640 1 -- 172.21.15.49:0/3518154359 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd04103aa0 con 0x7fcd041022b0 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a01a640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd04100700 0x7fcd04104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0a81b640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 0x7fcd04103210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fccfc02c700 tx=0x7fccfc032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fccfb7fe640 1 -- 172.21.15.49:0/3518154359 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc039040 con 0x7fcd041022b0 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fccfb7fe640 1 -- 172.21.15.49:0/3518154359 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccfc044070 con 0x7fcd041022b0 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fccfb7fe640 1 -- 172.21.15.49:0/3518154359 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccfc0042a0 con 0x7fcd041022b0 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd041a1b80 con 0x7fcd041022b0 2024-03-18T00:48:06.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.971+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd041a2120 con 0x7fcd041022b0 2024-03-18T00:48:06.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.972+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd041a22f0 con 0x7fcd041022b0 2024-03-18T00:48:06.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.973+0000 7fccfb7fe640 1 -- 172.21.15.49:0/3518154359 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fccfc003970 con 0x7fcd041022b0 2024-03-18T00:48:06.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.974+0000 7fccfb7fe640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccf00a2880 0x7fccf00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:06.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.974+0000 7fcd0a01a640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccf00a2880 0x7fccf00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:06.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.974+0000 7fccfb7fe640 1 -- 172.21.15.49:0/3518154359 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fccfc0f0e80 con 0x7fcd041022b0 2024-03-18T00:48:06.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.974+0000 7fcd0a01a640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccf00a2880 0x7fccf00a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd04101750 tx=0x7fcd00005d30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:06.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:06.978+0000 7fccfb7fe640 1 -- 172.21.15.49:0/3518154359 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fccfc0333a0 con 0x7fcd041022b0 2024-03-18T00:48:07.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.163+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd041a2680 con 0x7fcd041022b0 2024-03-18T00:48:07.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.165+0000 7fccfb7fe640 1 -- 172.21.15.49:0/3518154359 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22219 (secure 0 0 0) 0x7fccfc0be100 con 0x7fcd041022b0 2024-03-18T00:48:07.166 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:07.166 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":85,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:48:05.881439+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:48:07.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccf00a2880 msgr2=0x7fccf00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccf00a2880 0x7fccf00a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fcd04101750 tx=0x7fcd00005d30 comp rx=0 tx=0).stop 2024-03-18T00:48:07.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 msgr2=0x7fcd04103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 0x7fcd04103210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fccfc02c700 tx=0x7fccfc032f20 comp rx=0 tx=0).stop 2024-03-18T00:48:07.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 shutdown_connections 2024-03-18T00:48:07.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccf00a2880 0x7fccf00a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcd041018f0 0x7fcd04102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcd041022b0 0x7fcd04103210 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 --2- 172.21.15.49:0/3518154359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcd04100700 0x7fcd04104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 >> 172.21.15.49:0/3518154359 conn(0x7fcd040fbc70 msgr2=0x7fcd040fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:07.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 shutdown_connections 2024-03-18T00:48:07.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.167+0000 7fcd0c27e640 1 -- 172.21.15.49:0/3518154359 wait complete. 2024-03-18T00:48:07.178 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T00:48:07.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.289+0000 7fb869af8640 1 -- 172.21.15.49:0/3120538938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8641006e0 msgr2=0x7fb864100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.289+0000 7fb869af8640 1 --2- 172.21.15.49:0/3120538938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8641006e0 0x7fb864100b00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb854009ca0 tx=0x7fb85402c0f0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 -- 172.21.15.49:0/3120538938 shutdown_connections 2024-03-18T00:48:07.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 --2- 172.21.15.49:0/3120538938 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb864102290 0x7fb864108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 --2- 172.21.15.49:0/3120538938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8641018d0 0x7fb864101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 --2- 172.21.15.49:0/3120538938 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8641006e0 0x7fb864100b00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 -- 172.21.15.49:0/3120538938 >> 172.21.15.49:0/3120538938 conn(0x7fb8640fbc70 msgr2=0x7fb8640fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:07.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 -- 172.21.15.49:0/3120538938 shutdown_connections 2024-03-18T00:48:07.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 -- 172.21.15.49:0/3120538938 wait complete. 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 Processor -- start 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 -- start start 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8641006e0 0x7fb864069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8641018d0 0x7fb86419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb864102290 0x7fb86419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb86410c330 con 0x7fb864102290 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb86410c1b0 con 0x7fb8641018d0 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.290+0000 7fb869af8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb86410c4b0 con 0x7fb8641006e0 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb863fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb864102290 0x7fb86419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb863fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb864102290 0x7fb86419ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34748/0 (socket says 172.21.15.49:34748) 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb863fff640 1 -- 172.21.15.49:0/865051188 learned_addr learned my addr 172.21.15.49:0/865051188 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb8637fe640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8641006e0 0x7fb864069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:07.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb862ffd640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8641018d0 0x7fb86419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb863fff640 1 -- 172.21.15.49:0/865051188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8641006e0 msgr2=0x7fb864069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb863fff640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8641006e0 0x7fb864069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb863fff640 1 -- 172.21.15.49:0/865051188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8641018d0 msgr2=0x7fb86419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb863fff640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8641018d0 0x7fb86419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb863fff640 1 -- 172.21.15.49:0/865051188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86419f6f0 con 0x7fb864102290 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb863fff640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb864102290 0x7fb86419ee30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00c970 tx=0x7fb85c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb860ff9640 1 -- 172.21.15.49:0/865051188 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85c007bb0 con 0x7fb864102290 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb860ff9640 1 -- 172.21.15.49:0/865051188 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb85c007d50 con 0x7fb864102290 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb86419f9e0 con 0x7fb864102290 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb860ff9640 1 -- 172.21.15.49:0/865051188 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb85c018800 con 0x7fb864102290 2024-03-18T00:48:07.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.291+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb86419fee0 con 0x7fb864102290 2024-03-18T00:48:07.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.292+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb864107a20 con 0x7fb864102290 2024-03-18T00:48:07.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.293+0000 7fb860ff9640 1 -- 172.21.15.49:0/865051188 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb85c00fa50 con 0x7fb864102290 2024-03-18T00:48:07.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.294+0000 7fb860ff9640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8500a2800 0x7fb8500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:07.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.294+0000 7fb860ff9640 1 -- 172.21.15.49:0/865051188 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb85c0ce8f0 con 0x7fb864102290 2024-03-18T00:48:07.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.294+0000 7fb8637fe640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8500a2800 0x7fb8500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:07.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.295+0000 7fb8637fe640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8500a2800 0x7fb8500a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb85402c700 tx=0x7fb854032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:07.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.298+0000 7fb860ff9640 1 -- 172.21.15.49:0/865051188 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb85c014030 con 0x7fb864102290 2024-03-18T00:48:07.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.520+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb86402d1b0 con 0x7fb864102290 2024-03-18T00:48:07.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.532+0000 7fb860ff9640 1 -- 172.21.15.49:0/865051188 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7fb85c09bae0 con 0x7fb864102290 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8500a2800 msgr2=0x7fb8500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8500a2800 0x7fb8500a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb85402c700 tx=0x7fb854032590 comp rx=0 tx=0).stop 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb864102290 msgr2=0x7fb86419ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb864102290 0x7fb86419ee30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00c970 tx=0x7fb85c00ce30 comp rx=0 tx=0).stop 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 shutdown_connections 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8500a2800 0x7fb8500a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8641018d0 0x7fb86419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb864102290 0x7fb86419ee30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 --2- 172.21.15.49:0/865051188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8641006e0 0x7fb864069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 >> 172.21.15.49:0/865051188 conn(0x7fb8640fbc70 msgr2=0x7fb864104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:07.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 shutdown_connections 2024-03-18T00:48:07.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.535+0000 7fb869af8640 1 -- 172.21.15.49:0/865051188 wait complete. 2024-03-18T00:48:07.546 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-18T00:48:07.547 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-18T00:48:07.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.658+0000 7f495c444640 1 -- 172.21.15.49:0/3822109886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 msgr2=0x7f4954101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.658+0000 7f495c444640 1 --2- 172.21.15.49:0/3822109886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 0x7f4954101c20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4950009900 tx=0x7f495002bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:07.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.658+0000 7f495c444640 1 -- 172.21.15.49:0/3822109886 shutdown_connections 2024-03-18T00:48:07.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.658+0000 7f495c444640 1 --2- 172.21.15.49:0/3822109886 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49541033b0 0x7f4954109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.658+0000 7f495c444640 1 --2- 172.21.15.49:0/3822109886 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49541029f0 0x7f4954102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.658+0000 7f495c444640 1 --2- 172.21.15.49:0/3822109886 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 0x7f4954101c20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.658+0000 7f495c444640 1 -- 172.21.15.49:0/3822109886 >> 172.21.15.49:0/3822109886 conn(0x7f49540fcd70 msgr2=0x7f49540ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:07.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.658+0000 7f495c444640 1 -- 172.21.15.49:0/3822109886 shutdown_connections 2024-03-18T00:48:07.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.658+0000 7f495c444640 1 -- 172.21.15.49:0/3822109886 wait complete. 2024-03-18T00:48:07.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495c444640 1 Processor -- start 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495c444640 1 -- start start 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495c444640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 0x7f4954075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495c444640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49541029f0 0x7f4954073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495c444640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49541033b0 0x7f4954074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495c444640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f495410d450 con 0x7f4954101800 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495c444640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f495410d2d0 con 0x7f49541033b0 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495c444640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f495410d5d0 con 0x7f49541029f0 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495a1e0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 0x7f4954075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495a1e0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 0x7f4954075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60604/0 (socket says 172.21.15.49:60604) 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.659+0000 7f495a1e0640 1 -- 172.21.15.49:0/1211595620 learned_addr learned my addr 172.21.15.49:0/1211595620 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495a9e1640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49541033b0 0x7f4954074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495a1e0640 1 -- 172.21.15.49:0/1211595620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49541029f0 msgr2=0x7f4954073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495a1e0640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49541029f0 0x7f4954073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495a1e0640 1 -- 172.21.15.49:0/1211595620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49541033b0 msgr2=0x7f4954074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495a1e0640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49541033b0 0x7f4954074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495a1e0640 1 -- 172.21.15.49:0/1211595620 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49540749a0 con 0x7f4954101800 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495a9e1640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49541033b0 0x7f4954074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495a1e0640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 0x7f4954075590 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4950000c00 tx=0x7f4950005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f494b7fe640 1 -- 172.21.15.49:0/1211595620 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950041070 con 0x7f4954101800 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f494b7fe640 1 -- 172.21.15.49:0/1211595620 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49500044d0 con 0x7f4954101800 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49541a1bb0 con 0x7f4954101800 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f494b7fe640 1 -- 172.21.15.49:0/1211595620 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4950034440 con 0x7f4954101800 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.660+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49541a2100 con 0x7f4954101800 2024-03-18T00:48:07.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.661+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4954108bf0 con 0x7f4954101800 2024-03-18T00:48:07.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.662+0000 7f494b7fe640 1 -- 172.21.15.49:0/1211595620 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4950044070 con 0x7f4954101800 2024-03-18T00:48:07.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.663+0000 7f494b7fe640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49400a2800 0x7f49400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:07.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.663+0000 7f49599df640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49400a2800 0x7f49400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:07.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.663+0000 7f494b7fe640 1 -- 172.21.15.49:0/1211595620 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f49500f0590 con 0x7f4954101800 2024-03-18T00:48:07.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.663+0000 7f49599df640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49400a2800 0x7f49400a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4954075300 tx=0x7f4944005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:07.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.667+0000 7f494b7fe640 1 -- 172.21.15.49:0/1211595620 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f49500bd780 con 0x7f4954101800 2024-03-18T00:48:07.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2388392811' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T00:48:07.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:07 smithi049 ceph-mon[32130]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T00:48:07.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:07 smithi049 ceph-mon[32130]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T00:48:07.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:07 smithi049 ceph-mon[32130]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T00:48:07.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/219631308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:07.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3518154359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:07.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/865051188' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T00:48:07.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.894+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f495402d3f0 con 0x7f4954101800 2024-03-18T00:48:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:07 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2388392811' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T00:48:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:07 smithi107 ceph-mon[36270]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T00:48:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:07 smithi107 ceph-mon[36270]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T00:48:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:07 smithi107 ceph-mon[36270]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T00:48:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:07 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/219631308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:07.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:07 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3518154359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:07.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:07 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/865051188' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T00:48:07.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:07 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2388392811' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T00:48:07.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:07 smithi106 ceph-mon[35899]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T00:48:07.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:07 smithi106 ceph-mon[35899]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T00:48:07.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:07 smithi106 ceph-mon[35899]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T00:48:07.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:07 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/219631308' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:07.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:07 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3518154359' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:07.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:07 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/865051188' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T00:48:07.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.972+0000 7f494b7fe640 1 -- 172.21.15.49:0/1211595620 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7f49500bd080 con 0x7f4954101800 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49400a2800 msgr2=0x7f49400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49400a2800 0x7f49400a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4954075300 tx=0x7f4944005ed0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 msgr2=0x7f4954075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 0x7f4954075590 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4950000c00 tx=0x7f4950005d50 comp rx=0 tx=0).stop 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 shutdown_connections 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49400a2800 0x7f49400a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f49541033b0 0x7f4954074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f49541029f0 0x7f4954073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 --2- 172.21.15.49:0/1211595620 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4954101800 0x7f4954075590 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:07.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 >> 172.21.15.49:0/1211595620 conn(0x7f49540fcd70 msgr2=0x7f4954105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:07.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 shutdown_connections 2024-03-18T00:48:07.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:07.975+0000 7f495c444640 1 -- 172.21.15.49:0/1211595620 wait complete. 2024-03-18T00:48:07.990 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-18T00:48:08.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.101+0000 7efc2495b640 1 -- 172.21.15.49:0/2207356869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc20101800 msgr2=0x7efc20101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:08.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.101+0000 7efc2495b640 1 --2- 172.21.15.49:0/2207356869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc20101800 0x7efc20101c20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efc18009900 tx=0x7efc1802bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:08.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.101+0000 7efc2495b640 1 -- 172.21.15.49:0/2207356869 shutdown_connections 2024-03-18T00:48:08.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.101+0000 7efc2495b640 1 --2- 172.21.15.49:0/2207356869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc201033b0 0x7efc20109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:08.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.101+0000 7efc2495b640 1 --2- 172.21.15.49:0/2207356869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc201029f0 0x7efc20102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:08.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.101+0000 7efc2495b640 1 --2- 172.21.15.49:0/2207356869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc20101800 0x7efc20101c20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:08.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.101+0000 7efc2495b640 1 -- 172.21.15.49:0/2207356869 >> 172.21.15.49:0/2207356869 conn(0x7efc200fcd70 msgr2=0x7efc200ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:08.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.101+0000 7efc2495b640 1 -- 172.21.15.49:0/2207356869 shutdown_connections 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.101+0000 7efc2495b640 1 -- 172.21.15.49:0/2207356869 wait complete. 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc2495b640 1 Processor -- start 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc2495b640 1 -- start start 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc2495b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc20101800 0x7efc20197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc2495b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc201029f0 0x7efc20198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc2495b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc201033b0 0x7efc2019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc2495b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efc2010d4a0 con 0x7efc201029f0 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc2495b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efc2010d320 con 0x7efc20101800 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc2495b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efc2010d620 con 0x7efc201033b0 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc1dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc201029f0 0x7efc20198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc1dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc201029f0 0x7efc20198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60614/0 (socket says 172.21.15.49:60614) 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc1dd9b640 1 -- 172.21.15.49:0/1383218983 learned_addr learned my addr 172.21.15.49:0/1383218983 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.102+0000 7efc1e59c640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc20101800 0x7efc20197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:08.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc1ed9d640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc201033b0 0x7efc2019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc1dd9b640 1 -- 172.21.15.49:0/1383218983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc201033b0 msgr2=0x7efc2019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc1dd9b640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc201033b0 0x7efc2019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc1dd9b640 1 -- 172.21.15.49:0/1383218983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc20101800 msgr2=0x7efc20197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc1dd9b640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc20101800 0x7efc20197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc1dd9b640 1 -- 172.21.15.49:0/1383218983 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc2019f700 con 0x7efc201029f0 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc1e59c640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc20101800 0x7efc20197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc1dd9b640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc201029f0 0x7efc20198020 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7efc0800d950 tx=0x7efc0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc177fe640 1 -- 172.21.15.49:0/1383218983 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08017070 con 0x7efc201029f0 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc177fe640 1 -- 172.21.15.49:0/1383218983 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc08009c30 con 0x7efc201029f0 2024-03-18T00:48:08.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc2019f9f0 con 0x7efc201029f0 2024-03-18T00:48:08.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc177fe640 1 -- 172.21.15.49:0/1383218983 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0800f6b0 con 0x7efc201029f0 2024-03-18T00:48:08.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.103+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc2019ff90 con 0x7efc201029f0 2024-03-18T00:48:08.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.104+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc20108bf0 con 0x7efc201029f0 2024-03-18T00:48:08.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.105+0000 7efc177fe640 1 -- 172.21.15.49:0/1383218983 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7efc080109d0 con 0x7efc201029f0 2024-03-18T00:48:08.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.106+0000 7efc177fe640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc0c0a2880 0x7efc0c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:08.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.106+0000 7efc1e59c640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc0c0a2880 0x7efc0c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:08.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.106+0000 7efc177fe640 1 -- 172.21.15.49:0/1383218983 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7efc080ce420 con 0x7efc201029f0 2024-03-18T00:48:08.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.106+0000 7efc1e59c640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc0c0a2880 0x7efc0c0a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efc1802c170 tx=0x7efc1802df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:08.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.109+0000 7efc177fe640 1 -- 172.21.15.49:0/1383218983 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efc08009de0 con 0x7efc201029f0 2024-03-18T00:48:08.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.331+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7efc2002d400 con 0x7efc201029f0 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: pgmap v288: 97 pgs: 18 creating+peering, 34 active+clean, 45 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 0 op/s 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: Cluster is now healthy 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/865051188' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:active 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1211595620' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T00:48:08.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1211595620' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T00:48:08.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T00:48:08.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T00:48:08.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:08 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1383218983' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T00:48:08.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: pgmap v288: 97 pgs: 18 creating+peering, 34 active+clean, 45 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 0 op/s 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: Cluster is now healthy 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/865051188' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:active 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1211595620' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1211595620' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T00:48:08.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:08 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1383218983' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T00:48:08.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: pgmap v288: 97 pgs: 18 creating+peering, 34 active+clean, 45 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 0 op/s 2024-03-18T00:48:08.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: Cluster is now healthy 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/865051188' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:active 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1211595620' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1211595620' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T00:48:08.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:08 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1383218983' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T00:48:08.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.979+0000 7efc177fe640 1 -- 172.21.15.49:0/1383218983 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7efc0809b040 con 0x7efc201029f0 2024-03-18T00:48:08.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc0c0a2880 msgr2=0x7efc0c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:08.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc0c0a2880 0x7efc0c0a4d40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efc1802c170 tx=0x7efc1802df90 comp rx=0 tx=0).stop 2024-03-18T00:48:08.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc201029f0 msgr2=0x7efc20198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:08.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc201029f0 0x7efc20198020 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7efc0800d950 tx=0x7efc0800de10 comp rx=0 tx=0).stop 2024-03-18T00:48:08.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 shutdown_connections 2024-03-18T00:48:08.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efc0c0a2880 0x7efc0c0a4d40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:08.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efc201033b0 0x7efc2019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:08.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efc201029f0 0x7efc20198020 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:08.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 --2- 172.21.15.49:0/1383218983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efc20101800 0x7efc20197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:08.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 >> 172.21.15.49:0/1383218983 conn(0x7efc200fcd70 msgr2=0x7efc20105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:08.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 shutdown_connections 2024-03-18T00:48:08.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:08.982+0000 7efc2495b640 1 -- 172.21.15.49:0/1383218983 wait complete. 2024-03-18T00:48:08.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-18T00:48:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.109+0000 7f8a991d4640 1 -- 172.21.15.49:0/2904257526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 msgr2=0x7f8a94102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.109+0000 7f8a991d4640 1 --2- 172.21.15.49:0/2904257526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 0x7f8a94102e70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009900 tx=0x7f8a7c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.109+0000 7f8a991d4640 1 -- 172.21.15.49:0/2904257526 shutdown_connections 2024-03-18T00:48:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.109+0000 7f8a991d4640 1 --2- 172.21.15.49:0/2904257526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a941033b0 0x7f8a94109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.109+0000 7f8a991d4640 1 --2- 172.21.15.49:0/2904257526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 0x7f8a94102e70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.109+0000 7f8a991d4640 1 --2- 172.21.15.49:0/2904257526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a94101800 0x7f8a94101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.109+0000 7f8a991d4640 1 -- 172.21.15.49:0/2904257526 >> 172.21.15.49:0/2904257526 conn(0x7f8a940fcd70 msgr2=0x7f8a940ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.109+0000 7f8a991d4640 1 -- 172.21.15.49:0/2904257526 shutdown_connections 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.109+0000 7f8a991d4640 1 -- 172.21.15.49:0/2904257526 wait complete. 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a991d4640 1 Processor -- start 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a991d4640 1 -- start start 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a991d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a94101800 0x7f8a94197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a991d4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 0x7f8a94198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a991d4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a941033b0 0x7f8a9419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a991d4640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9410d4d0 con 0x7f8a941029f0 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a991d4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9410d350 con 0x7f8a941033b0 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a991d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9410d650 con 0x7f8a94101800 2024-03-18T00:48:09.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a8a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 0x7f8a94198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a8a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 0x7f8a94198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60622/0 (socket says 172.21.15.49:60622) 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a8a59c640 1 -- 172.21.15.49:0/3014560501 learned_addr learned my addr 172.21.15.49:0/3014560501 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a9359e640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a941033b0 0x7f8a9419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a92d9d640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a94101800 0x7f8a94197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a8a59c640 1 -- 172.21.15.49:0/3014560501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a94101800 msgr2=0x7f8a94197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a8a59c640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a94101800 0x7f8a94197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a8a59c640 1 -- 172.21.15.49:0/3014560501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a941033b0 msgr2=0x7f8a9419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a8a59c640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a941033b0 0x7f8a9419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.110+0000 7f8a8a59c640 1 -- 172.21.15.49:0/3014560501 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9419f700 con 0x7f8a941029f0 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.111+0000 7f8a8a59c640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 0x7f8a94198020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c005ab0 tx=0x7f8a7c02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.111+0000 7f8a90d99640 1 -- 172.21.15.49:0/3014560501 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c043020 con 0x7f8a941029f0 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.111+0000 7f8a90d99640 1 -- 172.21.15.49:0/3014560501 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a7c0054e0 con 0x7f8a941029f0 2024-03-18T00:48:09.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.111+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9419f990 con 0x7f8a941029f0 2024-03-18T00:48:09.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.111+0000 7f8a90d99640 1 -- 172.21.15.49:0/3014560501 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c003ce0 con 0x7f8a941029f0 2024-03-18T00:48:09.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.111+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a9419ff30 con 0x7f8a941029f0 2024-03-18T00:48:09.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.112+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a94108ac0 con 0x7f8a941029f0 2024-03-18T00:48:09.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.113+0000 7f8a90d99640 1 -- 172.21.15.49:0/3014560501 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8a7c046020 con 0x7f8a941029f0 2024-03-18T00:48:09.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.114+0000 7f8a90d99640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a800a2880 0x7f8a800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:09.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.114+0000 7f8a92d9d640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a800a2880 0x7f8a800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:09.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.114+0000 7f8a90d99640 1 -- 172.21.15.49:0/3014560501 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f8a7c0f0cf0 con 0x7f8a941029f0 2024-03-18T00:48:09.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.114+0000 7f8a92d9d640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a800a2880 0x7f8a800a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005fd0 tx=0x7f8a8400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:09.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.117+0000 7f8a90d99640 1 -- 172.21.15.49:0/3014560501 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a7c04d020 con 0x7f8a941029f0 2024-03-18T00:48:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.343+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f8a9402d400 con 0x7f8a941029f0 2024-03-18T00:48:09.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.989+0000 7f8a90d99640 1 -- 172.21.15.49:0/3014560501 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7f8a7c0bdee0 con 0x7f8a941029f0 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a800a2880 msgr2=0x7f8a800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a800a2880 0x7f8a800a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8a84005fd0 tx=0x7f8a8400b410 comp rx=0 tx=0).stop 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 msgr2=0x7f8a94198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 0x7f8a94198020 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c005ab0 tx=0x7f8a7c02c6c0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 shutdown_connections 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a800a2880 0x7f8a800a4d40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a941033b0 0x7f8a9419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a941029f0 0x7f8a94198020 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 --2- 172.21.15.49:0/3014560501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a94101800 0x7f8a94197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 >> 172.21.15.49:0/3014560501 conn(0x7f8a940fcd70 msgr2=0x7f8a94105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:09.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 shutdown_connections 2024-03-18T00:48:09.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:09.992+0000 7f8a991d4640 1 -- 172.21.15.49:0/3014560501 wait complete. 2024-03-18T00:48:10.009 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-03-18T00:48:10.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-18T00:48:10.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.124+0000 7f602aa97640 1 -- 172.21.15.49:0/2155497466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 msgr2=0x7f6024108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.124+0000 7f602aa97640 1 --2- 172.21.15.49:0/2155497466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 0x7f6024108b40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f601c009ca0 tx=0x7f601c02c0f0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.124+0000 7f602aa97640 1 -- 172.21.15.49:0/2155497466 shutdown_connections 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.124+0000 7f602aa97640 1 --2- 172.21.15.49:0/2155497466 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 0x7f6024108b40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.124+0000 7f602aa97640 1 --2- 172.21.15.49:0/2155497466 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60241018f0 0x7f6024101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.124+0000 7f602aa97640 1 --2- 172.21.15.49:0/2155497466 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6024100700 0x7f6024100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.124+0000 7f602aa97640 1 -- 172.21.15.49:0/2155497466 >> 172.21.15.49:0/2155497466 conn(0x7f60240fbc90 msgr2=0x7f60240fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.124+0000 7f602aa97640 1 -- 172.21.15.49:0/2155497466 shutdown_connections 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.124+0000 7f602aa97640 1 -- 172.21.15.49:0/2155497466 wait complete. 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f602aa97640 1 Processor -- start 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f602aa97640 1 -- start start 2024-03-18T00:48:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f602aa97640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6024100700 0x7f6024197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f602aa97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60241018f0 0x7f6024198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f602aa97640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 0x7f602419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f602aa97640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f602410c360 con 0x7f60241022b0 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f602aa97640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f602410c1e0 con 0x7f6024100700 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f602aa97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f602410c4e0 con 0x7f60241018f0 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6029034640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 0x7f602419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6029034640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 0x7f602419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60650/0 (socket says 172.21.15.49:60650) 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6023fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60241018f0 0x7f6024198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6029034640 1 -- 172.21.15.49:0/1570786488 learned_addr learned my addr 172.21.15.49:0/1570786488 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6028833640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6024100700 0x7f6024197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6029034640 1 -- 172.21.15.49:0/1570786488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60241018f0 msgr2=0x7f6024198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6029034640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60241018f0 0x7f6024198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6029034640 1 -- 172.21.15.49:0/1570786488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6024100700 msgr2=0x7f6024197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6029034640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6024100700 0x7f6024197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.125+0000 7f6029034640 1 -- 172.21.15.49:0/1570786488 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602419f6e0 con 0x7f60241022b0 2024-03-18T00:48:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.126+0000 7f6029034640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 0x7f602419f070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f601c037040 tx=0x7f601c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.126+0000 7f6021ffb640 1 -- 172.21.15.49:0/1570786488 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c041020 con 0x7f60241022b0 2024-03-18T00:48:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.126+0000 7f6021ffb640 1 -- 172.21.15.49:0/1570786488 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f601c034cd0 con 0x7f60241022b0 2024-03-18T00:48:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.126+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f602419f970 con 0x7f60241022b0 2024-03-18T00:48:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.126+0000 7f6021ffb640 1 -- 172.21.15.49:0/1570786488 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c039040 con 0x7f60241022b0 2024-03-18T00:48:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.126+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f602419fe60 con 0x7f60241022b0 2024-03-18T00:48:10.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.127+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60241a0030 con 0x7f60241022b0 2024-03-18T00:48:10.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.128+0000 7f6021ffb640 1 -- 172.21.15.49:0/1570786488 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f601c044020 con 0x7f60241022b0 2024-03-18T00:48:10.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.129+0000 7f6021ffb640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60140a2800 0x7f60140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.129+0000 7f6021ffb640 1 -- 172.21.15.49:0/1570786488 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f601c0f0800 con 0x7f60241022b0 2024-03-18T00:48:10.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.129+0000 7f6028833640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60140a2800 0x7f60140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.129+0000 7f6028833640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60140a2800 0x7f60140a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f60180095f0 tx=0x7f6018005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:10.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.133+0000 7f6021ffb640 1 -- 172.21.15.49:0/1570786488 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f601c04b020 con 0x7f60241022b0 2024-03-18T00:48:10.244 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:09 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1383218983' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T00:48:10.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:09 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] up:active 2024-03-18T00:48:10.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:09 smithi106 ceph-mon[35899]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T00:48:10.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:09 smithi106 ceph-mon[35899]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T00:48:10.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:09 smithi106 ceph-mon[35899]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T00:48:10.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:09 smithi106 ceph-mon[35899]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T00:48:10.245 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:09 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3014560501' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T00:48:10.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.343+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f60240fddd0 con 0x7f60140a2800 2024-03-18T00:48:10.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:09 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1383218983' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T00:48:10.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:09 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] up:active 2024-03-18T00:48:10.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:09 smithi049 ceph-mon[32130]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T00:48:10.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:09 smithi049 ceph-mon[32130]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T00:48:10.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:09 smithi049 ceph-mon[32130]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T00:48:10.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:09 smithi049 ceph-mon[32130]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T00:48:10.350 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:09 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3014560501' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T00:48:10.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.364+0000 7f6021ffb640 1 -- 172.21.15.49:0/1570786488 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f60240fddd0 con 0x7f60140a2800 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.365+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60140a2800 msgr2=0x7f60140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.365+0000 7f602aa97640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60140a2800 0x7f60140a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f60180095f0 tx=0x7f6018005f90 comp rx=0 tx=0).stop 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.365+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 msgr2=0x7f602419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.365+0000 7f602aa97640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 0x7f602419f070 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f601c037040 tx=0x7f601c032f40 comp rx=0 tx=0).stop 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.366+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 shutdown_connections 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.366+0000 7f602aa97640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60140a2800 0x7f60140a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.366+0000 7f602aa97640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f60241022b0 0x7f602419f070 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.366+0000 7f602aa97640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f60241018f0 0x7f6024198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.366+0000 7f602aa97640 1 --2- 172.21.15.49:0/1570786488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6024100700 0x7f6024197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.366+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 >> 172.21.15.49:0/1570786488 conn(0x7f60240fbc90 msgr2=0x7f60240fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.366+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 shutdown_connections 2024-03-18T00:48:10.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.366+0000 7f602aa97640 1 -- 172.21.15.49:0/1570786488 wait complete. 2024-03-18T00:48:10.377 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-18T00:48:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:09 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1383218983' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T00:48:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:09 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] up:active 2024-03-18T00:48:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:09 smithi107 ceph-mon[36270]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T00:48:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:09 smithi107 ceph-mon[36270]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T00:48:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:09 smithi107 ceph-mon[36270]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T00:48:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:09 smithi107 ceph-mon[36270]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T00:48:10.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:09 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3014560501' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T00:48:10.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.489+0000 7f573f316640 1 -- 172.21.15.49:0/3488315235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57381022a0 msgr2=0x7f5738104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.489+0000 7f573f316640 1 --2- 172.21.15.49:0/3488315235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57381022a0 0x7f5738104690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5730009ca0 tx=0x7f573002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 -- 172.21.15.49:0/3488315235 shutdown_connections 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 --2- 172.21.15.49:0/3488315235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5738107500 0x7f5738109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 --2- 172.21.15.49:0/3488315235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5738104bd0 0x7f5738106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 --2- 172.21.15.49:0/3488315235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f57381022a0 0x7f5738104690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 -- 172.21.15.49:0/3488315235 >> 172.21.15.49:0/3488315235 conn(0x7f57380fbc70 msgr2=0x7f57380fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 -- 172.21.15.49:0/3488315235 shutdown_connections 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 -- 172.21.15.49:0/3488315235 wait complete. 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 Processor -- start 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 -- start start 2024-03-18T00:48:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57381022a0 0x7f5738101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5738104bd0 0x7f5738100250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5738107500 0x7f5738100790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f573810d450 con 0x7f5738107500 2024-03-18T00:48:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f573810d2d0 con 0x7f57381022a0 2024-03-18T00:48:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573f316640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f573810d5d0 con 0x7f5738104bd0 2024-03-18T00:48:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573d8b3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5738107500 0x7f5738100790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573d8b3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5738107500 0x7f5738100790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60678/0 (socket says 172.21.15.49:60678) 2024-03-18T00:48:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.490+0000 7f573d8b3640 1 -- 172.21.15.49:0/3802722810 learned_addr learned my addr 172.21.15.49:0/3802722810 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573c8b1640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5738104bd0 0x7f5738100250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573d0b2640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57381022a0 0x7f5738101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573d8b3640 1 -- 172.21.15.49:0/3802722810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5738104bd0 msgr2=0x7f5738100250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573d8b3640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5738104bd0 0x7f5738100250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573d8b3640 1 -- 172.21.15.49:0/3802722810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57381022a0 msgr2=0x7f5738101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573d8b3640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57381022a0 0x7f5738101c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573d8b3640 1 -- 172.21.15.49:0/3802722810 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5738101020 con 0x7f5738107500 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573d8b3640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5738107500 0x7f5738100790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f573400e970 tx=0x7f573400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f572e7fc640 1 -- 172.21.15.49:0/3802722810 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573400ccc0 con 0x7f5738107500 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57381a1ca0 con 0x7f5738107500 2024-03-18T00:48:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f572e7fc640 1 -- 172.21.15.49:0/3802722810 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f573400ce60 con 0x7f5738107500 2024-03-18T00:48:10.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f572e7fc640 1 -- 172.21.15.49:0/3802722810 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5734010490 con 0x7f5738107500 2024-03-18T00:48:10.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.491+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57381a21a0 con 0x7f5738107500 2024-03-18T00:48:10.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.492+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57380fd760 con 0x7f5738107500 2024-03-18T00:48:10.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.493+0000 7f572e7fc640 1 -- 172.21.15.49:0/3802722810 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5734005000 con 0x7f5738107500 2024-03-18T00:48:10.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.494+0000 7f572e7fc640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57240a2800 0x7f57240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.494+0000 7f573d0b2640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57240a2800 0x7f57240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.494+0000 7f572e7fc640 1 -- 172.21.15.49:0/3802722810 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f57340cefb0 con 0x7f5738107500 2024-03-18T00:48:10.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.494+0000 7f573d0b2640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57240a2800 0x7f57240a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5730006010 tx=0x7f5730032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:10.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.497+0000 7f572e7fc640 1 -- 172.21.15.49:0/3802722810 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5734053740 con 0x7f5738107500 2024-03-18T00:48:10.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:48:10.348+0000 7fc75dac3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T00:48:10.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:48:10.348+0000 7fc75dac3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T00:48:10.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:48:10.348+0000 7fc75dac3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T00:48:10.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:48:10.348+0000 7fc75dac3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T00:48:10.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T00:48:10.348+0000 7fc75dac3700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T00:48:10.695 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b[66856]: 2024-03-18T00:48:10.243+0000 7fb66248e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T00:48:10.696 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b[66856]: 2024-03-18T00:48:10.243+0000 7fb66248e700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-18T00:48:10.696 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b[66856]: 2024-03-18T00:48:10.243+0000 7fb66248e700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-18T00:48:10.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.715+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f5738061cb0 con 0x7f57240a2800 2024-03-18T00:48:10.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.734+0000 7f572e7fc640 1 -- 172.21.15.49:0/3802722810 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5738061cb0 con 0x7f57240a2800 2024-03-18T00:48:10.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57240a2800 msgr2=0x7f57240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57240a2800 0x7f57240a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5730006010 tx=0x7f5730032590 comp rx=0 tx=0).stop 2024-03-18T00:48:10.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5738107500 msgr2=0x7f5738100790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5738107500 0x7f5738100790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f573400e970 tx=0x7f573400ed70 comp rx=0 tx=0).stop 2024-03-18T00:48:10.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 shutdown_connections 2024-03-18T00:48:10.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57240a2800 0x7f57240a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5738107500 0x7f5738100790 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5738104bd0 0x7f5738100250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 --2- 172.21.15.49:0/3802722810 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57381022a0 0x7f5738101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 >> 172.21.15.49:0/3802722810 conn(0x7f57380fbc70 msgr2=0x7f5738102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:10.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 shutdown_connections 2024-03-18T00:48:10.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.737+0000 7f573f316640 1 -- 172.21.15.49:0/3802722810 wait complete. 2024-03-18T00:48:10.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-18T00:48:10.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.862+0000 7f312cedc640 1 -- 172.21.15.49:0/2531628445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3128107540 msgr2=0x7f3128109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.862+0000 7f312cedc640 1 --2- 172.21.15.49:0/2531628445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3128107540 0x7f3128109930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3120009ca0 tx=0x7f312002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.862+0000 7f312cedc640 1 -- 172.21.15.49:0/2531628445 shutdown_connections 2024-03-18T00:48:10.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.862+0000 7f312cedc640 1 --2- 172.21.15.49:0/2531628445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3128104c10 0x7f3128107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.862+0000 7f312cedc640 1 --2- 172.21.15.49:0/2531628445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3128107540 0x7f3128109930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.862+0000 7f312cedc640 1 --2- 172.21.15.49:0/2531628445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f31281022e0 0x7f31281046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.862+0000 7f312cedc640 1 -- 172.21.15.49:0/2531628445 >> 172.21.15.49:0/2531628445 conn(0x7f31280fbc90 msgr2=0x7f31280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.862+0000 7f312cedc640 1 -- 172.21.15.49:0/2531628445 shutdown_connections 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.862+0000 7f312cedc640 1 -- 172.21.15.49:0/2531628445 wait complete. 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312cedc640 1 Processor -- start 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312cedc640 1 -- start start 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312cedc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31281022e0 0x7f3128197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312cedc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3128104c10 0x7f3128198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312cedc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3128107540 0x7f312819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312cedc640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f312810d4a0 con 0x7f31281022e0 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312cedc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f312810d320 con 0x7f3128107540 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312cedc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f312810d620 con 0x7f3128104c10 2024-03-18T00:48:10.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31281022e0 0x7f3128197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31281022e0 0x7f3128197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60696/0 (socket says 172.21.15.49:60696) 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312659c640 1 -- 172.21.15.49:0/657737172 learned_addr learned my addr 172.21.15.49:0/657737172 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f3125d9b640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3128104c10 0x7f3128198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f3126d9d640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3128107540 0x7f312819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312659c640 1 -- 172.21.15.49:0/657737172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3128104c10 msgr2=0x7f3128198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312659c640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3128104c10 0x7f3128198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312659c640 1 -- 172.21.15.49:0/657737172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3128107540 msgr2=0x7f312819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312659c640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3128107540 0x7f312819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.863+0000 7f312659c640 1 -- 172.21.15.49:0/657737172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312819f730 con 0x7f31281022e0 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.864+0000 7f312659c640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31281022e0 0x7f3128197b10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f311800e9f0 tx=0x7f311800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.864+0000 7f31177fe640 1 -- 172.21.15.49:0/657737172 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311800cd80 con 0x7f31281022e0 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.864+0000 7f31177fe640 1 -- 172.21.15.49:0/657737172 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3118005500 con 0x7f31281022e0 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.864+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f312819fa20 con 0x7f31281022e0 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.864+0000 7f31177fe640 1 -- 172.21.15.49:0/657737172 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3118010690 con 0x7f31281022e0 2024-03-18T00:48:10.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.864+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f312819ff20 con 0x7f31281022e0 2024-03-18T00:48:10.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.865+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31280fd780 con 0x7f31281022e0 2024-03-18T00:48:10.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.866+0000 7f31177fe640 1 -- 172.21.15.49:0/657737172 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f31180046c0 con 0x7f31281022e0 2024-03-18T00:48:10.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.866+0000 7f31177fe640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f31100a2800 0x7f31100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:10.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.867+0000 7f31177fe640 1 -- 172.21.15.49:0/657737172 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f31180ceee0 con 0x7f31281022e0 2024-03-18T00:48:10.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.867+0000 7f3125d9b640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f31100a2800 0x7f31100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:10.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.867+0000 7f3125d9b640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f31100a2800 0x7f31100a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3128199020 tx=0x7f311c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:10.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:10.870+0000 7f31177fe640 1 -- 172.21.15.49:0/657737172 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3118053740 con 0x7f31281022e0 2024-03-18T00:48:11.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.091+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f3128061c20 con 0x7f31100a2800 2024-03-18T00:48:11.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.095+0000 7f31177fe640 1 -- 172.21.15.49:0/657737172 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3128061c20 con 0x7f31100a2800 2024-03-18T00:48:11.095 INFO:teuthology.orchestra.run.smithi049.stdout:/volumes/qa/sv_0/24595a07-0013-40f3-94d8-c6c58dbced08 2024-03-18T00:48:11.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f31100a2800 msgr2=0x7f31100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f31100a2800 0x7f31100a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3128199020 tx=0x7f311c005f90 comp rx=0 tx=0).stop 2024-03-18T00:48:11.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31281022e0 msgr2=0x7f3128197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31281022e0 0x7f3128197b10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f311800e9f0 tx=0x7f311800edf0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 shutdown_connections 2024-03-18T00:48:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f31100a2800 0x7f31100a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3128107540 0x7f312819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3128104c10 0x7f3128198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 --2- 172.21.15.49:0/657737172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f31281022e0 0x7f3128197b10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 >> 172.21.15.49:0/657737172 conn(0x7f31280fbc90 msgr2=0x7f3128102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 shutdown_connections 2024-03-18T00:48:11.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.097+0000 7f312cedc640 1 -- 172.21.15.49:0/657737172 wait complete. 2024-03-18T00:48:11.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-mon[32130]: pgmap v290: 97 pgs: 18 creating+peering, 75 active+clean, 4 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 9 op/s 2024-03-18T00:48:11.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3014560501' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T00:48:11.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] up:active 2024-03-18T00:48:11.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-mon[32130]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T00:48:11.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-mon[32130]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T00:48:11.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-mon[32130]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-18T00:48:11.108 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:10 smithi049 ceph-mon[32130]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T00:48:11.109 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-18T00:48:11.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.221+0000 7f0fee753640 1 -- 172.21.15.49:0/636024203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 msgr2=0x7f0fe8102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.221+0000 7f0fee753640 1 --2- 172.21.15.49:0/636024203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 0x7f0fe8102e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc009900 tx=0x7f0fdc02bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.221+0000 7f0fee753640 1 -- 172.21.15.49:0/636024203 shutdown_connections 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.221+0000 7f0fee753640 1 --2- 172.21.15.49:0/636024203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fe81033a0 0x7f0fe8109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.221+0000 7f0fee753640 1 --2- 172.21.15.49:0/636024203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 0x7f0fe8102e60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.221+0000 7f0fee753640 1 --2- 172.21.15.49:0/636024203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0fe81017f0 0x7f0fe8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.221+0000 7f0fee753640 1 -- 172.21.15.49:0/636024203 >> 172.21.15.49:0/636024203 conn(0x7f0fe80fcd80 msgr2=0x7f0fe80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.221+0000 7f0fee753640 1 -- 172.21.15.49:0/636024203 shutdown_connections 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.221+0000 7f0fee753640 1 -- 172.21.15.49:0/636024203 wait complete. 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fee753640 1 Processor -- start 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fee753640 1 -- start start 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fee753640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fe81017f0 0x7f0fe8197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fee753640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 0x7f0fe8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fee753640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0fe81033a0 0x7f0fe819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fee753640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe810d4a0 con 0x7f0fe81029e0 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fee753640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe810d320 con 0x7f0fe81033a0 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fee753640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe810d620 con 0x7f0fe81017f0 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 0x7f0fe8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 0x7f0fe8198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60716/0 (socket says 172.21.15.49:60716) 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe77fe640 1 -- 172.21.15.49:0/2645515895 learned_addr learned my addr 172.21.15.49:0/2645515895 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe7fff640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fe81017f0 0x7f0fe8197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0feccf0640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0fe81033a0 0x7f0fe819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe77fe640 1 -- 172.21.15.49:0/2645515895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fe81017f0 msgr2=0x7f0fe8197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe77fe640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fe81017f0 0x7f0fe8197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe77fe640 1 -- 172.21.15.49:0/2645515895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0fe81033a0 msgr2=0x7f0fe819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe77fe640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0fe81033a0 0x7f0fe819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe77fe640 1 -- 172.21.15.49:0/2645515895 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe819f760 con 0x7f0fe81029e0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.222+0000 7f0fe77fe640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 0x7f0fe8198080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc000c00 tx=0x7f0fdc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.223+0000 7f0fe57fa640 1 -- 172.21.15.49:0/2645515895 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc043020 con 0x7f0fe81029e0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.223+0000 7f0fe57fa640 1 -- 172.21.15.49:0/2645515895 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fdc0054e0 con 0x7f0fe81029e0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.223+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe819f9f0 con 0x7f0fe81029e0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.223+0000 7f0fe57fa640 1 -- 172.21.15.49:0/2645515895 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fdc003bd0 con 0x7f0fe81029e0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.223+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe819ff90 con 0x7f0fe81029e0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.224+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe8108b10 con 0x7f0fe81029e0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.225+0000 7f0fe57fa640 1 -- 172.21.15.49:0/2645515895 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0fdc046020 con 0x7f0fe81029e0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.225+0000 7f0fe57fa640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0fd00a2880 0x7f0fd00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.226+0000 7f0fe7fff640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0fd00a2880 0x7f0fd00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.226+0000 7f0fe57fa640 1 -- 172.21.15.49:0/2645515895 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f0fdc0f08e0 con 0x7f0fe81029e0 2024-03-18T00:48:11.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.226+0000 7f0fe7fff640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0fd00a2880 0x7f0fd00a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80095f0 tx=0x7f0fd8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:11.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.230+0000 7f0fe57fa640 1 -- 172.21.15.49:0/2645515895 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0fdc04d020 con 0x7f0fe81029e0 2024-03-18T00:48:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:10 smithi107 ceph-mon[36270]: pgmap v290: 97 pgs: 18 creating+peering, 75 active+clean, 4 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 9 op/s 2024-03-18T00:48:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:10 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3014560501' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T00:48:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:10 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] up:active 2024-03-18T00:48:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:10 smithi107 ceph-mon[36270]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T00:48:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:10 smithi107 ceph-mon[36270]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T00:48:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:10 smithi107 ceph-mon[36270]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-18T00:48:11.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:10 smithi107 ceph-mon[36270]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T00:48:11.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.443+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f0fe8061cb0 con 0x7f0fd00a2880 2024-03-18T00:48:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-mon[35899]: pgmap v290: 97 pgs: 18 creating+peering, 75 active+clean, 4 unknown; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 9 op/s 2024-03-18T00:48:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3014560501' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T00:48:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] up:active 2024-03-18T00:48:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-mon[35899]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T00:48:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-mon[35899]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T00:48:11.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-mon[35899]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-18T00:48:11.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:10 smithi106 ceph-mon[35899]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T00:48:11.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.469+0000 7f0fe57fa640 1 -- 172.21.15.49:0/2645515895 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f0fe8061cb0 con 0x7f0fd00a2880 2024-03-18T00:48:11.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0fd00a2880 msgr2=0x7f0fd00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0fd00a2880 0x7f0fd00a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0fd80095f0 tx=0x7f0fd8005f90 comp rx=0 tx=0).stop 2024-03-18T00:48:11.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 msgr2=0x7f0fe8198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 0x7f0fe8198080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc000c00 tx=0x7f0fdc02c650 comp rx=0 tx=0).stop 2024-03-18T00:48:11.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 shutdown_connections 2024-03-18T00:48:11.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0fd00a2880 0x7f0fd00a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0fe81033a0 0x7f0fe819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0fe81029e0 0x7f0fe8198080 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 --2- 172.21.15.49:0/2645515895 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0fe81017f0 0x7f0fe8197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 >> 172.21.15.49:0/2645515895 conn(0x7f0fe80fcd80 msgr2=0x7f0fe8105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:11.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 shutdown_connections 2024-03-18T00:48:11.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.471+0000 7f0fee753640 1 -- 172.21.15.49:0/2645515895 wait complete. 2024-03-18T00:48:11.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-18T00:48:11.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.596+0000 7f4be5243640 1 -- 172.21.15.49:0/1854265899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01018d0 msgr2=0x7f4be0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.596+0000 7f4be5243640 1 --2- 172.21.15.49:0/1854265899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01018d0 0x7f4be0101d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009900 tx=0x7f4bd402bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.596+0000 7f4be5243640 1 -- 172.21.15.49:0/1854265899 shutdown_connections 2024-03-18T00:48:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.596+0000 7f4be5243640 1 --2- 172.21.15.49:0/1854265899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0102290 0x7f4be0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.596+0000 7f4be5243640 1 --2- 172.21.15.49:0/1854265899 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01018d0 0x7f4be0101d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.596+0000 7f4be5243640 1 --2- 172.21.15.49:0/1854265899 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be01006e0 0x7f4be0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.596+0000 7f4be5243640 1 -- 172.21.15.49:0/1854265899 >> 172.21.15.49:0/1854265899 conn(0x7f4be00fbc70 msgr2=0x7f4be00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.596+0000 7f4be5243640 1 -- 172.21.15.49:0/1854265899 shutdown_connections 2024-03-18T00:48:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.596+0000 7f4be5243640 1 -- 172.21.15.49:0/1854265899 wait complete. 2024-03-18T00:48:11.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4be5243640 1 Processor -- start 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4be5243640 1 -- start start 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4be5243640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01006e0 0x7f4be0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4be5243640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be01018d0 0x7f4be0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4be5243640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0102290 0x7f4be019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4be5243640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4be010c340 con 0x7f4be01006e0 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4be5243640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4be010c1c0 con 0x7f4be01018d0 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4be5243640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4be010c4c0 con 0x7f4be0102290 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4bded9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01006e0 0x7f4be0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4bded9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01006e0 0x7f4be0197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60734/0 (socket says 172.21.15.49:60734) 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.597+0000 7f4bded9d640 1 -- 172.21.15.49:0/1107083684 learned_addr learned my addr 172.21.15.49:0/1107083684 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bdf59e640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0102290 0x7f4be019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bde59c640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be01018d0 0x7f4be0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bded9d640 1 -- 172.21.15.49:0/1107083684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0102290 msgr2=0x7f4be019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bded9d640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0102290 0x7f4be019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bded9d640 1 -- 172.21.15.49:0/1107083684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be01018d0 msgr2=0x7f4be0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bded9d640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be01018d0 0x7f4be0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bded9d640 1 -- 172.21.15.49:0/1107083684 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be019f760 con 0x7f4be01006e0 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bded9d640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01006e0 0x7f4be0197b40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800e9c0 tx=0x7f4bd800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bc7fff640 1 -- 172.21.15.49:0/1107083684 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd800ccc0 con 0x7f4be01006e0 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be019fa50 con 0x7f4be01006e0 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bc7fff640 1 -- 172.21.15.49:0/1107083684 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd80035f0 con 0x7f4be01006e0 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4bc7fff640 1 -- 172.21.15.49:0/1107083684 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd800ce60 con 0x7f4be01006e0 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.598+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be019ffc0 con 0x7f4be01006e0 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.599+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be01a0250 con 0x7f4be01006e0 2024-03-18T00:48:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.600+0000 7f4bc7fff640 1 -- 172.21.15.49:0/1107083684 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4bd80037a0 con 0x7f4be01006e0 2024-03-18T00:48:11.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.601+0000 7f4bc7fff640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2800 0x7f4bc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.601+0000 7f4bde59c640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2800 0x7f4bc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.601+0000 7f4bc7fff640 1 -- 172.21.15.49:0/1107083684 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f4bd80cecc0 con 0x7f4be01006e0 2024-03-18T00:48:11.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.601+0000 7f4bde59c640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2800 0x7f4bc80a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4bd402c7b0 tx=0x7f4bd4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:11.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.605+0000 7f4bc7fff640 1 -- 172.21.15.49:0/1107083684 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4bd809beb0 con 0x7f4be01006e0 2024-03-18T00:48:11.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.818+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f4be0069050 con 0x7f4bc80a2800 2024-03-18T00:48:11.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.821+0000 7f4bc7fff640 1 -- 172.21.15.49:0/1107083684 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f4be0069050 con 0x7f4bc80a2800 2024-03-18T00:48:11.822 INFO:teuthology.orchestra.run.smithi049.stdout:/volumes/qa/sv_1/d7393f6b-0735-4414-ac84-f375918aae72 2024-03-18T00:48:11.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2800 msgr2=0x7f4bc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2800 0x7f4bc80a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4bd402c7b0 tx=0x7f4bd4005fd0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01006e0 msgr2=0x7f4be0197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01006e0 0x7f4be0197b40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800e9c0 tx=0x7f4bd800edc0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 shutdown_connections 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2800 0x7f4bc80a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be01018d0 0x7f4be0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0102290 0x7f4be019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 --2- 172.21.15.49:0/1107083684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be01006e0 0x7f4be0197b40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.823+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 >> 172.21.15.49:0/1107083684 conn(0x7f4be00fbc70 msgr2=0x7f4be0068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.824+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 shutdown_connections 2024-03-18T00:48:11.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.824+0000 7f4be5243640 1 -- 172.21.15.49:0/1107083684 wait complete. 2024-03-18T00:48:11.836 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:48:11.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.943+0000 7f26152ac640 1 -- 172.21.15.49:0/1153018866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26101022d0 msgr2=0x7f2610108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.943+0000 7f26152ac640 1 --2- 172.21.15.49:0/1153018866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26101022d0 0x7f2610108b60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2600009ca0 tx=0x7f260002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.943+0000 7f26152ac640 1 -- 172.21.15.49:0/1153018866 shutdown_connections 2024-03-18T00:48:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.943+0000 7f26152ac640 1 --2- 172.21.15.49:0/1153018866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26101022d0 0x7f2610108b60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.943+0000 7f26152ac640 1 --2- 172.21.15.49:0/1153018866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2610101910 0x7f2610101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.943+0000 7f26152ac640 1 --2- 172.21.15.49:0/1153018866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2610100720 0x7f2610100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.943+0000 7f26152ac640 1 -- 172.21.15.49:0/1153018866 >> 172.21.15.49:0/1153018866 conn(0x7f26100fbc90 msgr2=0x7f26100fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.943+0000 7f26152ac640 1 -- 172.21.15.49:0/1153018866 shutdown_connections 2024-03-18T00:48:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.943+0000 7f26152ac640 1 -- 172.21.15.49:0/1153018866 wait complete. 2024-03-18T00:48:11.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f26152ac640 1 Processor -- start 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f26152ac640 1 -- start start 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f26152ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2610100720 0x7f261006a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f26152ac640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2610101910 0x7f261019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f26152ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26101022d0 0x7f261019ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f26152ac640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f261010c380 con 0x7f2610101910 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f26152ac640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f261010c200 con 0x7f26101022d0 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f26152ac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f261010c500 con 0x7f2610100720 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f260e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2610101910 0x7f261019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f260effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2610100720 0x7f261006a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f260e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2610101910 0x7f261019c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60762/0 (socket says 172.21.15.49:60762) 2024-03-18T00:48:11.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f260effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2610100720 0x7f261006a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:43682/0 (socket says 172.21.15.49:43682) 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f260e7fc640 1 -- 172.21.15.49:0/2592549453 learned_addr learned my addr 172.21.15.49:0/2592549453 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.944+0000 7f260f7fe640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26101022d0 0x7f261019ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f260e7fc640 1 -- 172.21.15.49:0/2592549453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2610100720 msgr2=0x7f261006a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f260e7fc640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2610100720 0x7f261006a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f260e7fc640 1 -- 172.21.15.49:0/2592549453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26101022d0 msgr2=0x7f261019ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f260e7fc640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26101022d0 0x7f261019ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f260e7fc640 1 -- 172.21.15.49:0/2592549453 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261019f6c0 con 0x7f2610101910 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f260effd640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2610100720 0x7f261006a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f260f7fe640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26101022d0 0x7f261019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f260e7fc640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2610101910 0x7f261019c7c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f260400e9f0 tx=0x7f260400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f25f7fff640 1 -- 172.21.15.49:0/2592549453 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260400cd80 con 0x7f2610101910 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f25f7fff640 1 -- 172.21.15.49:0/2592549453 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2604010910 con 0x7f2610101910 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f261019f9b0 con 0x7f2610101910 2024-03-18T00:48:11.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f25f7fff640 1 -- 172.21.15.49:0/2592549453 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604018990 con 0x7f2610101910 2024-03-18T00:48:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.945+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f261019ff00 con 0x7f2610101910 2024-03-18T00:48:11.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.946+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2610107a40 con 0x7f2610101910 2024-03-18T00:48:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.947+0000 7f25f7fff640 1 -- 172.21.15.49:0/2592549453 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2604010ac0 con 0x7f2610101910 2024-03-18T00:48:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.948+0000 7f25f7fff640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25f80a2800 0x7f25f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.948+0000 7f260effd640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25f80a2800 0x7f25f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.948+0000 7f25f7fff640 1 -- 172.21.15.49:0/2592549453 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f26040cec40 con 0x7f2610101910 2024-03-18T00:48:11.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.948+0000 7f260effd640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25f80a2800 0x7f25f80a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2610101770 tx=0x7f2608005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:11.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:11.953+0000 7f25f7fff640 1 -- 172.21.15.49:0/2592549453 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f260409b7e0 con 0x7f2610101910 2024-03-18T00:48:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.021+0000 7f25f7fff640 1 -- 172.21.15.49:0/2592549453 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2604097e50 con 0x7f2610101910 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] up:active 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: Cluster is now healthy 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-18T00:48:12.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:11 smithi049 ceph-mon[32130]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:12.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.187+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f261002d250 con 0x7f2610101910 2024-03-18T00:48:12.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.187+0000 7f25f7fff640 1 -- 172.21.15.49:0/2592549453 <== mon.0 v2:172.21.15.49:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11932 (secure 0 0 0) 0x7f26040c5c40 con 0x7f2610101910 2024-03-18T00:48:12.188 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:12.188 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:48:12.008019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24499,24506,24481,24470,24463]},"id":1}]} 2024-03-18T00:48:12.189 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 24 2024-03-18T00:48:12.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25f80a2800 msgr2=0x7f25f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25f80a2800 0x7f25f80a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2610101770 tx=0x7f2608005f90 comp rx=0 tx=0).stop 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2610101910 msgr2=0x7f261019c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2610101910 0x7f261019c7c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f260400e9f0 tx=0x7f260400eeb0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 shutdown_connections 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25f80a2800 0x7f25f80a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f26101022d0 0x7f261019ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2610101910 0x7f261019c7c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 --2- 172.21.15.49:0/2592549453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2610100720 0x7f261006a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.189+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 >> 172.21.15.49:0/2592549453 conn(0x7f26100fbc90 msgr2=0x7f2610104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.190+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 shutdown_connections 2024-03-18T00:48:12.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.190+0000 7f26152ac640 1 -- 172.21.15.49:0/2592549453 wait complete. 2024-03-18T00:48:12.201 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:48:12.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.308+0000 7fd66e69a640 1 -- 172.21.15.49:0/4235662382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6680ffaf0 msgr2=0x7fd6680fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.308+0000 7fd66e69a640 1 --2- 172.21.15.49:0/4235662382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6680ffaf0 0x7fd6680fff10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd658009930 tx=0x7fd65802bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:12.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.308+0000 7fd66e69a640 1 -- 172.21.15.49:0/4235662382 shutdown_connections 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.308+0000 7fd66e69a640 1 --2- 172.21.15.49:0/4235662382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd668102b60 0x7fd668102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.308+0000 7fd66e69a640 1 --2- 172.21.15.49:0/4235662382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd668103520 0x7fd668109db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.308+0000 7fd66e69a640 1 --2- 172.21.15.49:0/4235662382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd6680ffaf0 0x7fd6680fff10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.308+0000 7fd66e69a640 1 -- 172.21.15.49:0/4235662382 >> 172.21.15.49:0/4235662382 conn(0x7fd6680fb400 msgr2=0x7fd6680fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.308+0000 7fd66e69a640 1 -- 172.21.15.49:0/4235662382 shutdown_connections 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.308+0000 7fd66e69a640 1 -- 172.21.15.49:0/4235662382 wait complete. 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66e69a640 1 Processor -- start 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66e69a640 1 -- start start 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66e69a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6680ffaf0 0x7fd668197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66e69a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd668102b60 0x7fd668198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66e69a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd668103520 0x7fd66819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66e69a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd66810d4a0 con 0x7fd668103520 2024-03-18T00:48:12.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66e69a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd66810d320 con 0x7fd6680ffaf0 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66e69a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd66810d620 con 0x7fd668102b60 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66cc37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd668103520 0x7fd66819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd667fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6680ffaf0 0x7fd668197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd6677fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd668102b60 0x7fd668198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66cc37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd668103520 0x7fd66819f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60766/0 (socket says 172.21.15.49:60766) 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.309+0000 7fd66cc37640 1 -- 172.21.15.49:0/3326094334 learned_addr learned my addr 172.21.15.49:0/3326094334 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd66cc37640 1 -- 172.21.15.49:0/3326094334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd668102b60 msgr2=0x7fd668198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd66cc37640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd668102b60 0x7fd668198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd66cc37640 1 -- 172.21.15.49:0/3326094334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6680ffaf0 msgr2=0x7fd668197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd66cc37640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6680ffaf0 0x7fd668197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd66cc37640 1 -- 172.21.15.49:0/3326094334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66819f740 con 0x7fd668103520 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd66cc37640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd668103520 0x7fd66819f0d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd66000eb10 tx=0x7fd66000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd6657fa640 1 -- 172.21.15.49:0/3326094334 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660004b20 con 0x7fd668103520 2024-03-18T00:48:12.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd6657fa640 1 -- 172.21.15.49:0/3326094334 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd660004cc0 con 0x7fd668103520 2024-03-18T00:48:12.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66819fa30 con 0x7fd668103520 2024-03-18T00:48:12.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd6657fa640 1 -- 172.21.15.49:0/3326094334 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660010760 con 0x7fd668103520 2024-03-18T00:48:12.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.310+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66819ffd0 con 0x7fd668103520 2024-03-18T00:48:12.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.311+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd668108c90 con 0x7fd668103520 2024-03-18T00:48:12.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.312+0000 7fd6657fa640 1 -- 172.21.15.49:0/3326094334 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd660005020 con 0x7fd668103520 2024-03-18T00:48:12.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.313+0000 7fd6657fa640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd6500a2910 0x7fd6500a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:12.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.313+0000 7fd667fff640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd6500a2910 0x7fd6500a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:12.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.313+0000 7fd6657fa640 1 -- 172.21.15.49:0/3326094334 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd6600cf090 con 0x7fd668103520 2024-03-18T00:48:12.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.313+0000 7fd667fff640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd6500a2910 0x7fd6500a4dd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd65802c7b0 tx=0x7fd658005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:12.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.317+0000 7fd6657fa640 1 -- 172.21.15.49:0/3326094334 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd66009c180 con 0x7fd668103520 2024-03-18T00:48:12.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:12.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:12.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] up:active 2024-03-18T00:48:12.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T00:48:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T00:48:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: Cluster is now healthy 2024-03-18T00:48:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-18T00:48:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-18T00:48:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-18T00:48:12.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:11 smithi107 ceph-mon[36270]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:12.449 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:12.449 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:12.449 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] up:active 2024-03-18T00:48:12.450 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T00:48:12.450 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T00:48:12.450 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: Cluster is now healthy 2024-03-18T00:48:12.450 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 8 up:standby 2024-03-18T00:48:12.450 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-18T00:48:12.450 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-03-18T00:48:12.450 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:11 smithi106 ceph-mon[35899]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:12.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.502+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd66802d400 con 0x7fd668103520 2024-03-18T00:48:12.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.503+0000 7fd6657fa640 1 -- 172.21.15.49:0/3326094334 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22206 (secure 0 0 0) 0x7fd66009ba80 con 0x7fd668103520 2024-03-18T00:48:12.504 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:12.505 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":86,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:48:07.527912+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd6500a2910 msgr2=0x7fd6500a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd6500a2910 0x7fd6500a4dd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd65802c7b0 tx=0x7fd658005fd0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd668103520 msgr2=0x7fd66819f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd668103520 0x7fd66819f0d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd66000eb10 tx=0x7fd66000c2d0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 shutdown_connections 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd6500a2910 0x7fd6500a4dd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd668103520 0x7fd66819f0d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd668102b60 0x7fd668198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 --2- 172.21.15.49:0/3326094334 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd6680ffaf0 0x7fd668197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 >> 172.21.15.49:0/3326094334 conn(0x7fd6680fb400 msgr2=0x7fd668105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 shutdown_connections 2024-03-18T00:48:12.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.506+0000 7fd66e69a640 1 -- 172.21.15.49:0/3326094334 wait complete. 2024-03-18T00:48:12.519 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T00:48:12.519 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T00:48:12.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.626+0000 7f595b9ab640 1 -- 172.21.15.49:0/91743029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 msgr2=0x7f5954109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.626+0000 7f595b9ab640 1 --2- 172.21.15.49:0/91743029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 0x7f5954109c60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5950009ca0 tx=0x7f595002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.626+0000 7f595b9ab640 1 -- 172.21.15.49:0/91743029 shutdown_connections 2024-03-18T00:48:12.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.626+0000 7f595b9ab640 1 --2- 172.21.15.49:0/91743029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5954102a10 0x7f5954102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.626+0000 7f595b9ab640 1 --2- 172.21.15.49:0/91743029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 0x7f5954109c60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.626+0000 7f595b9ab640 1 --2- 172.21.15.49:0/91743029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5954101820 0x7f5954101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.626+0000 7f595b9ab640 1 -- 172.21.15.49:0/91743029 >> 172.21.15.49:0/91743029 conn(0x7f59540fcd90 msgr2=0x7f59540ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:12.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 -- 172.21.15.49:0/91743029 shutdown_connections 2024-03-18T00:48:12.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 -- 172.21.15.49:0/91743029 wait complete. 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 Processor -- start 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 -- start start 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5954101820 0x7f5954197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5954102a10 0x7f5954197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 0x7f595419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f595410d480 con 0x7f59541033d0 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f595410d300 con 0x7f5954102a10 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f595b9ab640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f595410d600 con 0x7f5954101820 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f5959f48640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 0x7f595419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f5959747640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5954101820 0x7f5954197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f5959f48640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 0x7f595419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60782/0 (socket says 172.21.15.49:60782) 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f5959747640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5954101820 0x7f5954197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:43710/0 (socket says 172.21.15.49:43710) 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f5959f48640 1 -- 172.21.15.49:0/399958911 learned_addr learned my addr 172.21.15.49:0/399958911 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:12.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.627+0000 7f5958f46640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5954102a10 0x7f5954197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f5959747640 1 -- 172.21.15.49:0/399958911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5954102a10 msgr2=0x7f5954197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f5959747640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5954102a10 0x7f5954197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f5959747640 1 -- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 msgr2=0x7f595419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f5959747640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 0x7f595419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f5959747640 1 -- 172.21.15.49:0/399958911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f595419f6c0 con 0x7f5954101820 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f5959f48640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 0x7f595419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f5959747640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5954101820 0x7f5954197aa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f594c00e9c0 tx=0x7f594c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f594a7fc640 1 -- 172.21.15.49:0/399958911 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c00cd80 con 0x7f5954101820 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f595419f9b0 con 0x7f5954101820 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f594a7fc640 1 -- 172.21.15.49:0/399958911 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f594c010910 con 0x7f5954101820 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f594a7fc640 1 -- 172.21.15.49:0/399958911 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c018980 con 0x7f5954101820 2024-03-18T00:48:12.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.628+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f595419ff50 con 0x7f5954101820 2024-03-18T00:48:12.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.629+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59540685d0 con 0x7f5954101820 2024-03-18T00:48:12.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.630+0000 7f594a7fc640 1 -- 172.21.15.49:0/399958911 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f594c01f020 con 0x7f5954101820 2024-03-18T00:48:12.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.631+0000 7f594a7fc640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f59400a2890 0x7f59400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:12.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.631+0000 7f5958f46640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f59400a2890 0x7f59400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:12.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.631+0000 7f594a7fc640 1 -- 172.21.15.49:0/399958911 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f594c0cebf0 con 0x7f5954101820 2024-03-18T00:48:12.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.631+0000 7f5958f46640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f59400a2890 0x7f59400a4d50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5954198fb0 tx=0x7f5944005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:12.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.634+0000 7f594a7fc640 1 -- 172.21.15.49:0/399958911 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f594c09bc60 con 0x7f5954101820 2024-03-18T00:48:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.829+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5954061ce0 con 0x7f59400a2890 2024-03-18T00:48:12.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.837+0000 7f594a7fc640 1 -- 172.21.15.49:0/399958911 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258895 (secure 0 0 0) 0x7f5954061ce0 con 0x7f59400a2890 2024-03-18T00:48:12.843 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:12.845 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-03-18T00:48:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f59400a2890 msgr2=0x7f59400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f59400a2890 0x7f59400a4d50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5954198fb0 tx=0x7f5944005f90 comp rx=0 tx=0).stop 2024-03-18T00:48:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5954101820 msgr2=0x7f5954197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5954101820 0x7f5954197aa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f594c00e9c0 tx=0x7f594c00ee80 comp rx=0 tx=0).stop 2024-03-18T00:48:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 shutdown_connections 2024-03-18T00:48:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f59400a2890 0x7f59400a4d50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5954102a10 0x7f5954197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f59541033d0 0x7f595419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 --2- 172.21.15.49:0/399958911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5954101820 0x7f5954197aa0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 >> 172.21.15.49:0/399958911 conn(0x7f59540fcd90 msgr2=0x7f5954105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 shutdown_connections 2024-03-18T00:48:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:12.840+0000 7f595b9ab640 1 -- 172.21.15.49:0/399958911 wait complete. 2024-03-18T00:48:12.852 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-03-18T00:48:11.460422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596015,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":300,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356060,"kb_used_data":11004,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585764,"statfs":{"total":1151940427776,"available":1151575822336,"internally_reserved":0,"allocated":11268096,"data_stored":7001998,"data_compressed":108276,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5647,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001388"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.537623+0000","last_change":"2024-03-18T00:48:06.536034+0000","last_active":"2024-03-18T00:48:06.537623+0000","last_peered":"2024-03-18T00:48:06.537623+0000","last_clean":"2024-03-18T00:48:06.537623+0000","last_became_active":"2024-03-18T00:48:06.535694+0000","last_became_peered":"2024-03-18T00:48:06.535694+0000","last_unstale":"2024-03-18T00:48:06.537623+0000","last_undegraded":"2024-03-18T00:48:06.537623+0000","last_fullsized":"2024-03-18T00:48:06.537623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:56.816104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984930+0000","last_change":"2024-03-18T00:48:06.536741+0000","last_active":"2024-03-18T00:48:07.984930+0000","last_peered":"2024-03-18T00:48:07.984930+0000","last_clean":"2024-03-18T00:48:07.984930+0000","last_became_active":"2024-03-18T00:48:06.535865+0000","last_became_peered":"2024-03-18T00:48:06.535865+0000","last_unstale":"2024-03-18T00:48:07.984930+0000","last_undegraded":"2024-03-18T00:48:07.984930+0000","last_fullsized":"2024-03-18T00:48:07.984930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:58.104983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.985112+0000","last_change":"2024-03-18T00:48:06.537162+0000","last_active":"2024-03-18T00:48:07.985112+0000","last_peered":"2024-03-18T00:48:07.985112+0000","last_clean":"2024-03-18T00:48:07.985112+0000","last_became_active":"2024-03-18T00:48:06.536835+0000","last_became_peered":"2024-03-18T00:48:06.536835+0000","last_unstale":"2024-03-18T00:48:07.985112+0000","last_undegraded":"2024-03-18T00:48:07.985112+0000","last_fullsized":"2024-03-18T00:48:07.985112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:15:08.519180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536822+0000","last_change":"2024-03-18T00:48:06.540084+0000","last_active":"2024-03-18T00:48:07.536822+0000","last_peered":"2024-03-18T00:48:07.536822+0000","last_clean":"2024-03-18T00:48:07.536822+0000","last_became_active":"2024-03-18T00:48:06.539736+0000","last_became_peered":"2024-03-18T00:48:06.539736+0000","last_unstale":"2024-03-18T00:48:07.536822+0000","last_undegraded":"2024-03-18T00:48:07.536822+0000","last_fullsized":"2024-03-18T00:48:07.536822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:37:51.748603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989527+0000","last_change":"2024-03-18T00:48:06.540022+0000","last_active":"2024-03-18T00:48:07.989527+0000","last_peered":"2024-03-18T00:48:07.989527+0000","last_clean":"2024-03-18T00:48:07.989527+0000","last_became_active":"2024-03-18T00:48:06.539665+0000","last_became_peered":"2024-03-18T00:48:06.539665+0000","last_unstale":"2024-03-18T00:48:07.989527+0000","last_undegraded":"2024-03-18T00:48:07.989527+0000","last_fullsized":"2024-03-18T00:48:07.989527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:55:52.448279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.653245+0000","last_change":"2024-03-18T00:48:06.538672+0000","last_active":"2024-03-18T00:48:07.653245+0000","last_peered":"2024-03-18T00:48:07.653245+0000","last_clean":"2024-03-18T00:48:07.653245+0000","last_became_active":"2024-03-18T00:48:06.538189+0000","last_became_peered":"2024-03-18T00:48:06.538189+0000","last_unstale":"2024-03-18T00:48:07.653245+0000","last_undegraded":"2024-03-18T00:48:07.653245+0000","last_fullsized":"2024-03-18T00:48:07.653245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:09:04.672255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.547139+0000","last_change":"2024-03-18T00:48:06.546417+0000","last_active":"2024-03-18T00:48:06.547139+0000","last_peered":"2024-03-18T00:48:06.547139+0000","last_clean":"2024-03-18T00:48:06.547139+0000","last_became_active":"2024-03-18T00:48:06.545759+0000","last_became_peered":"2024-03-18T00:48:06.545759+0000","last_unstale":"2024-03-18T00:48:06.547139+0000","last_undegraded":"2024-03-18T00:48:06.547139+0000","last_fullsized":"2024-03-18T00:48:06.547139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:37.878207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535247+0000","last_change":"2024-03-18T00:48:06.545875+0000","last_active":"2024-03-18T00:48:07.535247+0000","last_peered":"2024-03-18T00:48:07.535247+0000","last_clean":"2024-03-18T00:48:07.535247+0000","last_became_active":"2024-03-18T00:48:06.545316+0000","last_became_peered":"2024-03-18T00:48:06.545316+0000","last_unstale":"2024-03-18T00:48:07.535247+0000","last_undegraded":"2024-03-18T00:48:07.535247+0000","last_fullsized":"2024-03-18T00:48:07.535247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:22:40.998671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.534852+0000","last_change":"2024-03-18T00:48:06.533947+0000","last_active":"2024-03-18T00:48:06.534852+0000","last_peered":"2024-03-18T00:48:06.534852+0000","last_clean":"2024-03-18T00:48:06.534852+0000","last_became_active":"2024-03-18T00:48:06.533615+0000","last_became_peered":"2024-03-18T00:48:06.533615+0000","last_unstale":"2024-03-18T00:48:06.534852+0000","last_undegraded":"2024-03-18T00:48:06.534852+0000","last_fullsized":"2024-03-18T00:48:06.534852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:41:17.501153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.653533+0000","last_change":"2024-03-18T00:48:06.537614+0000","last_active":"2024-03-18T00:48:07.653533+0000","last_peered":"2024-03-18T00:48:07.653533+0000","last_clean":"2024-03-18T00:48:07.653533+0000","last_became_active":"2024-03-18T00:48:06.536617+0000","last_became_peered":"2024-03-18T00:48:06.536617+0000","last_unstale":"2024-03-18T00:48:07.653533+0000","last_undegraded":"2024-03-18T00:48:07.653533+0000","last_fullsized":"2024-03-18T00:48:07.653533+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:52.222603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546599+0000","last_change":"2024-03-18T00:48:06.540363+0000","last_active":"2024-03-18T00:48:06.546599+0000","last_peered":"2024-03-18T00:48:06.546599+0000","last_clean":"2024-03-18T00:48:06.546599+0000","last_became_active":"2024-03-18T00:48:06.540011+0000","last_became_peered":"2024-03-18T00:48:06.540011+0000","last_unstale":"2024-03-18T00:48:06.546599+0000","last_undegraded":"2024-03-18T00:48:06.546599+0000","last_fullsized":"2024-03-18T00:48:06.546599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:52:47.287604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536647+0000","last_change":"2024-03-18T00:48:06.532611+0000","last_active":"2024-03-18T00:48:07.536647+0000","last_peered":"2024-03-18T00:48:07.536647+0000","last_clean":"2024-03-18T00:48:07.536647+0000","last_became_active":"2024-03-18T00:48:06.532301+0000","last_became_peered":"2024-03-18T00:48:06.532301+0000","last_unstale":"2024-03-18T00:48:07.536647+0000","last_undegraded":"2024-03-18T00:48:07.536647+0000","last_fullsized":"2024-03-18T00:48:07.536647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:40.795105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.532473+0000","last_change":"2024-03-18T00:48:06.531993+0000","last_active":"2024-03-18T00:48:06.532473+0000","last_peered":"2024-03-18T00:48:06.532473+0000","last_clean":"2024-03-18T00:48:06.532473+0000","last_became_active":"2024-03-18T00:48:06.531677+0000","last_became_peered":"2024-03-18T00:48:06.531677+0000","last_unstale":"2024-03-18T00:48:06.532473+0000","last_undegraded":"2024-03-18T00:48:06.532473+0000","last_fullsized":"2024-03-18T00:48:06.532473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:36:02.027315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984887+0000","last_change":"2024-03-18T00:48:06.536830+0000","last_active":"2024-03-18T00:48:07.984887+0000","last_peered":"2024-03-18T00:48:07.984887+0000","last_clean":"2024-03-18T00:48:07.984887+0000","last_became_active":"2024-03-18T00:48:06.535908+0000","last_became_peered":"2024-03-18T00:48:06.535908+0000","last_unstale":"2024-03-18T00:48:07.984887+0000","last_undegraded":"2024-03-18T00:48:07.984887+0000","last_fullsized":"2024-03-18T00:48:07.984887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:23:26.155130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984501+0000","last_change":"2024-03-18T00:48:06.543641+0000","last_active":"2024-03-18T00:48:07.984501+0000","last_peered":"2024-03-18T00:48:07.984501+0000","last_clean":"2024-03-18T00:48:07.984501+0000","last_became_active":"2024-03-18T00:48:06.543293+0000","last_became_peered":"2024-03-18T00:48:06.543293+0000","last_unstale":"2024-03-18T00:48:07.984501+0000","last_undegraded":"2024-03-18T00:48:07.984501+0000","last_fullsized":"2024-03-18T00:48:07.984501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:52.406255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546754+0000","last_change":"2024-03-18T00:48:06.544732+0000","last_active":"2024-03-18T00:48:06.546754+0000","last_peered":"2024-03-18T00:48:06.546754+0000","last_clean":"2024-03-18T00:48:06.546754+0000","last_became_active":"2024-03-18T00:48:06.544400+0000","last_became_peered":"2024-03-18T00:48:06.544400+0000","last_unstale":"2024-03-18T00:48:06.546754+0000","last_undegraded":"2024-03-18T00:48:06.546754+0000","last_fullsized":"2024-03-18T00:48:06.546754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:51:54.744799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.537522+0000","last_change":"2024-03-18T00:48:06.536595+0000","last_active":"2024-03-18T00:48:06.537522+0000","last_peered":"2024-03-18T00:48:06.537522+0000","last_clean":"2024-03-18T00:48:06.537522+0000","last_became_active":"2024-03-18T00:48:06.535874+0000","last_became_peered":"2024-03-18T00:48:06.535874+0000","last_unstale":"2024-03-18T00:48:06.537522+0000","last_undegraded":"2024-03-18T00:48:06.537522+0000","last_fullsized":"2024-03-18T00:48:06.537522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:38:10.958438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.982813+0000","last_change":"2024-03-18T00:48:06.546676+0000","last_active":"2024-03-18T00:48:07.982813+0000","last_peered":"2024-03-18T00:48:07.982813+0000","last_clean":"2024-03-18T00:48:07.982813+0000","last_became_active":"2024-03-18T00:48:06.546356+0000","last_became_peered":"2024-03-18T00:48:06.546356+0000","last_unstale":"2024-03-18T00:48:07.982813+0000","last_undegraded":"2024-03-18T00:48:07.982813+0000","last_fullsized":"2024-03-18T00:48:07.982813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:04:35.467415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.532812+0000","last_change":"2024-03-18T00:48:06.532184+0000","last_active":"2024-03-18T00:48:06.532812+0000","last_peered":"2024-03-18T00:48:06.532812+0000","last_clean":"2024-03-18T00:48:06.532812+0000","last_became_active":"2024-03-18T00:48:06.531816+0000","last_became_peered":"2024-03-18T00:48:06.531816+0000","last_unstale":"2024-03-18T00:48:06.532812+0000","last_undegraded":"2024-03-18T00:48:06.532812+0000","last_fullsized":"2024-03-18T00:48:06.532812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:58:47.981121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:08.998127+0000","last_change":"2024-03-18T00:48:05.530213+0000","last_active":"2024-03-18T00:48:08.998127+0000","last_peered":"2024-03-18T00:48:08.998127+0000","last_clean":"2024-03-18T00:48:08.998127+0000","last_became_active":"2024-03-18T00:48:05.529897+0000","last_became_peered":"2024-03-18T00:48:05.529897+0000","last_unstale":"2024-03-18T00:48:08.998127+0000","last_undegraded":"2024-03-18T00:48:08.998127+0000","last_fullsized":"2024-03-18T00:48:08.998127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:49.003362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535869+0000","last_change":"2024-03-18T00:48:06.544437+0000","last_active":"2024-03-18T00:48:07.535869+0000","last_peered":"2024-03-18T00:48:07.535869+0000","last_clean":"2024-03-18T00:48:07.535869+0000","last_became_active":"2024-03-18T00:48:06.544125+0000","last_became_peered":"2024-03-18T00:48:06.544125+0000","last_unstale":"2024-03-18T00:48:07.535869+0000","last_undegraded":"2024-03-18T00:48:07.535869+0000","last_fullsized":"2024-03-18T00:48:07.535869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:56:42.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535680+0000","last_change":"2024-03-18T00:48:05.536249+0000","last_active":"2024-03-18T00:48:07.535680+0000","last_peered":"2024-03-18T00:48:07.535680+0000","last_clean":"2024-03-18T00:48:07.535680+0000","last_became_active":"2024-03-18T00:48:05.535954+0000","last_became_peered":"2024-03-18T00:48:05.535954+0000","last_unstale":"2024-03-18T00:48:07.535680+0000","last_undegraded":"2024-03-18T00:48:07.535680+0000","last_fullsized":"2024-03-18T00:48:07.535680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:18:52.341453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.985140+0000","last_change":"2024-03-18T00:48:06.541377+0000","last_active":"2024-03-18T00:48:07.985140+0000","last_peered":"2024-03-18T00:48:07.985140+0000","last_clean":"2024-03-18T00:48:07.985140+0000","last_became_active":"2024-03-18T00:48:06.540937+0000","last_became_peered":"2024-03-18T00:48:06.540937+0000","last_unstale":"2024-03-18T00:48:07.985140+0000","last_undegraded":"2024-03-18T00:48:07.985140+0000","last_fullsized":"2024-03-18T00:48:07.985140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:51:55.268517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989866+0000","last_change":"2024-03-18T00:48:05.534372+0000","last_active":"2024-03-18T00:48:07.989866+0000","last_peered":"2024-03-18T00:48:07.989866+0000","last_clean":"2024-03-18T00:48:07.989866+0000","last_became_active":"2024-03-18T00:48:05.534034+0000","last_became_peered":"2024-03-18T00:48:05.534034+0000","last_unstale":"2024-03-18T00:48:07.989866+0000","last_undegraded":"2024-03-18T00:48:07.989866+0000","last_fullsized":"2024-03-18T00:48:07.989866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:58:55.013167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.529837+0000","last_change":"2024-03-18T00:48:06.527276+0000","last_active":"2024-03-18T00:48:06.529837+0000","last_peered":"2024-03-18T00:48:06.529837+0000","last_clean":"2024-03-18T00:48:06.529837+0000","last_became_active":"2024-03-18T00:48:06.526904+0000","last_became_peered":"2024-03-18T00:48:06.526904+0000","last_unstale":"2024-03-18T00:48:06.529837+0000","last_undegraded":"2024-03-18T00:48:06.529837+0000","last_fullsized":"2024-03-18T00:48:06.529837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:29:23.222222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989512+0000","last_change":"2024-03-18T00:48:05.528188+0000","last_active":"2024-03-18T00:48:07.989512+0000","last_peered":"2024-03-18T00:48:07.989512+0000","last_clean":"2024-03-18T00:48:07.989512+0000","last_became_active":"2024-03-18T00:48:05.527788+0000","last_became_peered":"2024-03-18T00:48:05.527788+0000","last_unstale":"2024-03-18T00:48:07.989512+0000","last_undegraded":"2024-03-18T00:48:07.989512+0000","last_fullsized":"2024-03-18T00:48:07.989512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:47.939771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536355+0000","last_change":"2024-03-18T00:48:06.546937+0000","last_active":"2024-03-18T00:48:07.536355+0000","last_peered":"2024-03-18T00:48:07.536355+0000","last_clean":"2024-03-18T00:48:07.536355+0000","last_became_active":"2024-03-18T00:48:06.546624+0000","last_became_peered":"2024-03-18T00:48:06.546624+0000","last_unstale":"2024-03-18T00:48:07.536355+0000","last_undegraded":"2024-03-18T00:48:07.536355+0000","last_fullsized":"2024-03-18T00:48:07.536355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:52:30.830967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.990797+0000","last_change":"2024-03-18T00:48:05.534490+0000","last_active":"2024-03-18T00:48:07.990797+0000","last_peered":"2024-03-18T00:48:07.990797+0000","last_clean":"2024-03-18T00:48:07.990797+0000","last_became_active":"2024-03-18T00:48:05.534141+0000","last_became_peered":"2024-03-18T00:48:05.534141+0000","last_unstale":"2024-03-18T00:48:07.990797+0000","last_undegraded":"2024-03-18T00:48:07.990797+0000","last_fullsized":"2024-03-18T00:48:07.990797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:04:07.477814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989060+0000","last_change":"2024-03-18T00:48:06.537405+0000","last_active":"2024-03-18T00:48:07.989060+0000","last_peered":"2024-03-18T00:48:07.989060+0000","last_clean":"2024-03-18T00:48:07.989060+0000","last_became_active":"2024-03-18T00:48:06.536919+0000","last_became_peered":"2024-03-18T00:48:06.536919+0000","last_unstale":"2024-03-18T00:48:07.989060+0000","last_undegraded":"2024-03-18T00:48:07.989060+0000","last_fullsized":"2024-03-18T00:48:07.989060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:31:59.863571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:10.005453+0000","last_change":"2024-03-18T00:48:05.528588+0000","last_active":"2024-03-18T00:48:10.005453+0000","last_peered":"2024-03-18T00:48:10.005453+0000","last_clean":"2024-03-18T00:48:10.005453+0000","last_became_active":"2024-03-18T00:48:05.528263+0000","last_became_peered":"2024-03-18T00:48:05.528263+0000","last_unstale":"2024-03-18T00:48:10.005453+0000","last_undegraded":"2024-03-18T00:48:10.005453+0000","last_fullsized":"2024-03-18T00:48:10.005453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:18:10.283278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536199+0000","last_change":"2024-03-18T00:48:06.546323+0000","last_active":"2024-03-18T00:48:07.536199+0000","last_peered":"2024-03-18T00:48:07.536199+0000","last_clean":"2024-03-18T00:48:07.536199+0000","last_became_active":"2024-03-18T00:48:06.545956+0000","last_became_peered":"2024-03-18T00:48:06.545956+0000","last_unstale":"2024-03-18T00:48:07.536199+0000","last_undegraded":"2024-03-18T00:48:07.536199+0000","last_fullsized":"2024-03-18T00:48:07.536199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:32.618724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.516831+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:48:06.516831+0000","last_peered":"2024-03-18T00:48:06.516831+0000","last_clean":"2024-03-18T00:48:06.516831+0000","last_became_active":"2024-03-18T00:48:05.525590+0000","last_became_peered":"2024-03-18T00:48:05.525590+0000","last_unstale":"2024-03-18T00:48:06.516831+0000","last_undegraded":"2024-03-18T00:48:06.516831+0000","last_fullsized":"2024-03-18T00:48:06.516831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:19.357331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535616+0000","last_change":"2024-03-18T00:48:06.538798+0000","last_active":"2024-03-18T00:48:07.535616+0000","last_peered":"2024-03-18T00:48:07.535616+0000","last_clean":"2024-03-18T00:48:07.535616+0000","last_became_active":"2024-03-18T00:48:06.538457+0000","last_became_peered":"2024-03-18T00:48:06.538457+0000","last_unstale":"2024-03-18T00:48:07.535616+0000","last_undegraded":"2024-03-18T00:48:07.535616+0000","last_fullsized":"2024-03-18T00:48:07.535616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:10.738683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.987554+0000","last_change":"2024-03-18T00:48:05.528094+0000","last_active":"2024-03-18T00:48:07.987554+0000","last_peered":"2024-03-18T00:48:07.987554+0000","last_clean":"2024-03-18T00:48:07.987554+0000","last_became_active":"2024-03-18T00:48:05.527730+0000","last_became_peered":"2024-03-18T00:48:05.527730+0000","last_unstale":"2024-03-18T00:48:07.987554+0000","last_undegraded":"2024-03-18T00:48:07.987554+0000","last_fullsized":"2024-03-18T00:48:07.987554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:48:27.707505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984234+0000","last_change":"2024-03-18T00:48:06.548676+0000","last_active":"2024-03-18T00:48:07.984234+0000","last_peered":"2024-03-18T00:48:07.984234+0000","last_clean":"2024-03-18T00:48:07.984234+0000","last_became_active":"2024-03-18T00:48:06.547693+0000","last_became_peered":"2024-03-18T00:48:06.547693+0000","last_unstale":"2024-03-18T00:48:07.984234+0000","last_undegraded":"2024-03-18T00:48:07.984234+0000","last_fullsized":"2024-03-18T00:48:07.984234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:13.849175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.986256+0000","last_change":"2024-03-18T00:48:05.523639+0000","last_active":"2024-03-18T00:48:07.986256+0000","last_peered":"2024-03-18T00:48:07.986256+0000","last_clean":"2024-03-18T00:48:07.986256+0000","last_became_active":"2024-03-18T00:48:05.523269+0000","last_became_peered":"2024-03-18T00:48:05.523269+0000","last_unstale":"2024-03-18T00:48:07.986256+0000","last_undegraded":"2024-03-18T00:48:07.986256+0000","last_fullsized":"2024-03-18T00:48:07.986256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:39.961247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.543930+0000","last_change":"2024-03-18T00:48:06.537786+0000","last_active":"2024-03-18T00:48:06.543930+0000","last_peered":"2024-03-18T00:48:06.543930+0000","last_clean":"2024-03-18T00:48:06.543930+0000","last_became_active":"2024-03-18T00:48:06.537488+0000","last_became_peered":"2024-03-18T00:48:06.537488+0000","last_unstale":"2024-03-18T00:48:06.543930+0000","last_undegraded":"2024-03-18T00:48:06.543930+0000","last_fullsized":"2024-03-18T00:48:06.543930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:25:50.608244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.524735+0000","last_change":"2024-03-18T00:48:05.535421+0000","last_active":"2024-03-18T00:48:06.524735+0000","last_peered":"2024-03-18T00:48:06.524735+0000","last_clean":"2024-03-18T00:48:06.524735+0000","last_became_active":"2024-03-18T00:48:05.534902+0000","last_became_peered":"2024-03-18T00:48:05.534902+0000","last_unstale":"2024-03-18T00:48:06.524735+0000","last_undegraded":"2024-03-18T00:48:06.524735+0000","last_fullsized":"2024-03-18T00:48:06.524735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:53:14.516189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.985097+0000","last_change":"2024-03-18T00:48:06.545416+0000","last_active":"2024-03-18T00:48:07.985097+0000","last_peered":"2024-03-18T00:48:07.985097+0000","last_clean":"2024-03-18T00:48:07.985097+0000","last_became_active":"2024-03-18T00:48:06.545087+0000","last_became_peered":"2024-03-18T00:48:06.545087+0000","last_unstale":"2024-03-18T00:48:07.985097+0000","last_undegraded":"2024-03-18T00:48:07.985097+0000","last_fullsized":"2024-03-18T00:48:07.985097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:02:25.309990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984082+0000","last_change":"2024-03-18T00:48:05.533088+0000","last_active":"2024-03-18T00:48:07.984082+0000","last_peered":"2024-03-18T00:48:07.984082+0000","last_clean":"2024-03-18T00:48:07.984082+0000","last_became_active":"2024-03-18T00:48:05.532308+0000","last_became_peered":"2024-03-18T00:48:05.532308+0000","last_unstale":"2024-03-18T00:48:07.984082+0000","last_undegraded":"2024-03-18T00:48:07.984082+0000","last_fullsized":"2024-03-18T00:48:07.984082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:35.071283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546898+0000","last_change":"2024-03-18T00:48:06.545011+0000","last_active":"2024-03-18T00:48:06.546898+0000","last_peered":"2024-03-18T00:48:06.546898+0000","last_clean":"2024-03-18T00:48:06.546898+0000","last_became_active":"2024-03-18T00:48:06.544519+0000","last_became_peered":"2024-03-18T00:48:06.544519+0000","last_unstale":"2024-03-18T00:48:06.546898+0000","last_undegraded":"2024-03-18T00:48:06.546898+0000","last_fullsized":"2024-03-18T00:48:06.546898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:08:36.618883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984228+0000","last_change":"2024-03-18T00:48:05.527557+0000","last_active":"2024-03-18T00:48:07.984228+0000","last_peered":"2024-03-18T00:48:07.984228+0000","last_clean":"2024-03-18T00:48:07.984228+0000","last_became_active":"2024-03-18T00:48:05.527167+0000","last_became_peered":"2024-03-18T00:48:05.527167+0000","last_unstale":"2024-03-18T00:48:07.984228+0000","last_undegraded":"2024-03-18T00:48:07.984228+0000","last_fullsized":"2024-03-18T00:48:07.984228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:02:04.105091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989666+0000","last_change":"2024-03-18T00:48:06.543153+0000","last_active":"2024-03-18T00:48:07.989666+0000","last_peered":"2024-03-18T00:48:07.989666+0000","last_clean":"2024-03-18T00:48:07.989666+0000","last_became_active":"2024-03-18T00:48:06.541353+0000","last_became_peered":"2024-03-18T00:48:06.541353+0000","last_unstale":"2024-03-18T00:48:07.989666+0000","last_undegraded":"2024-03-18T00:48:07.989666+0000","last_fullsized":"2024-03-18T00:48:07.989666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:35.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.548724+0000","last_change":"2024-03-18T00:48:05.530560+0000","last_active":"2024-03-18T00:48:06.548724+0000","last_peered":"2024-03-18T00:48:06.548724+0000","last_clean":"2024-03-18T00:48:06.548724+0000","last_became_active":"2024-03-18T00:48:05.528858+0000","last_became_peered":"2024-03-18T00:48:05.528858+0000","last_unstale":"2024-03-18T00:48:06.548724+0000","last_undegraded":"2024-03-18T00:48:06.548724+0000","last_fullsized":"2024-03-18T00:48:06.548724+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:03:01.496919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.545375+0000","last_change":"2024-03-18T00:48:06.544748+0000","last_active":"2024-03-18T00:48:06.545375+0000","last_peered":"2024-03-18T00:48:06.545375+0000","last_clean":"2024-03-18T00:48:06.545375+0000","last_became_active":"2024-03-18T00:48:06.541951+0000","last_became_peered":"2024-03-18T00:48:06.541951+0000","last_unstale":"2024-03-18T00:48:06.545375+0000","last_undegraded":"2024-03-18T00:48:06.545375+0000","last_fullsized":"2024-03-18T00:48:06.545375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:38:38.757028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.987918+0000","last_change":"2024-03-18T00:48:05.527902+0000","last_active":"2024-03-18T00:48:07.987918+0000","last_peered":"2024-03-18T00:48:07.987918+0000","last_clean":"2024-03-18T00:48:07.987918+0000","last_became_active":"2024-03-18T00:48:05.527495+0000","last_became_peered":"2024-03-18T00:48:05.527495+0000","last_unstale":"2024-03-18T00:48:07.987918+0000","last_undegraded":"2024-03-18T00:48:07.987918+0000","last_fullsized":"2024-03-18T00:48:07.987918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:25.504652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.551415+0000","last_change":"2024-03-18T00:48:06.550413+0000","last_active":"2024-03-18T00:48:06.551415+0000","last_peered":"2024-03-18T00:48:06.551415+0000","last_clean":"2024-03-18T00:48:06.551415+0000","last_became_active":"2024-03-18T00:48:06.547727+0000","last_became_peered":"2024-03-18T00:48:06.547727+0000","last_unstale":"2024-03-18T00:48:06.551415+0000","last_undegraded":"2024-03-18T00:48:06.551415+0000","last_fullsized":"2024-03-18T00:48:06.551415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:45:03.068840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:10.004320+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:48:10.004320+0000","last_peered":"2024-03-18T00:48:10.004320+0000","last_clean":"2024-03-18T00:48:10.004320+0000","last_became_active":"2024-03-18T00:48:05.525607+0000","last_became_peered":"2024-03-18T00:48:05.525607+0000","last_unstale":"2024-03-18T00:48:10.004320+0000","last_undegraded":"2024-03-18T00:48:10.004320+0000","last_fullsized":"2024-03-18T00:48:10.004320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:32.457447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546975+0000","last_change":"2024-03-18T00:48:06.545838+0000","last_active":"2024-03-18T00:48:06.546975+0000","last_peered":"2024-03-18T00:48:06.546975+0000","last_clean":"2024-03-18T00:48:06.546975+0000","last_became_active":"2024-03-18T00:48:06.543258+0000","last_became_peered":"2024-03-18T00:48:06.543258+0000","last_unstale":"2024-03-18T00:48:06.546975+0000","last_undegraded":"2024-03-18T00:48:06.546975+0000","last_fullsized":"2024-03-18T00:48:06.546975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:17:06.569478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.991530+0000","last_change":"2024-03-18T00:48:05.536296+0000","last_active":"2024-03-18T00:48:07.991530+0000","last_peered":"2024-03-18T00:48:07.991530+0000","last_clean":"2024-03-18T00:48:07.991530+0000","last_became_active":"2024-03-18T00:48:05.535945+0000","last_became_peered":"2024-03-18T00:48:05.535945+0000","last_unstale":"2024-03-18T00:48:07.991530+0000","last_undegraded":"2024-03-18T00:48:07.991530+0000","last_fullsized":"2024-03-18T00:48:07.991530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:48:20.978213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535895+0000","last_change":"2024-03-18T00:48:06.544084+0000","last_active":"2024-03-18T00:48:07.535895+0000","last_peered":"2024-03-18T00:48:07.535895+0000","last_clean":"2024-03-18T00:48:07.535895+0000","last_became_active":"2024-03-18T00:48:06.543419+0000","last_became_peered":"2024-03-18T00:48:06.543419+0000","last_unstale":"2024-03-18T00:48:07.535895+0000","last_undegraded":"2024-03-18T00:48:07.535895+0000","last_fullsized":"2024-03-18T00:48:07.535895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:15:28.527473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989321+0000","last_change":"2024-03-18T00:48:05.528347+0000","last_active":"2024-03-18T00:48:07.989321+0000","last_peered":"2024-03-18T00:48:07.989321+0000","last_clean":"2024-03-18T00:48:07.989321+0000","last_became_active":"2024-03-18T00:48:05.527761+0000","last_became_peered":"2024-03-18T00:48:05.527761+0000","last_unstale":"2024-03-18T00:48:07.989321+0000","last_undegraded":"2024-03-18T00:48:07.989321+0000","last_fullsized":"2024-03-18T00:48:07.989321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:36:58.562495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.542369+0000","last_change":"2024-03-18T00:48:06.541862+0000","last_active":"2024-03-18T00:48:06.542369+0000","last_peered":"2024-03-18T00:48:06.542369+0000","last_clean":"2024-03-18T00:48:06.542369+0000","last_became_active":"2024-03-18T00:48:06.541488+0000","last_became_peered":"2024-03-18T00:48:06.541488+0000","last_unstale":"2024-03-18T00:48:06.542369+0000","last_undegraded":"2024-03-18T00:48:06.542369+0000","last_fullsized":"2024-03-18T00:48:06.542369+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:54:42.065399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.982380+0000","last_change":"2024-03-18T00:48:05.535498+0000","last_active":"2024-03-18T00:48:07.982380+0000","last_peered":"2024-03-18T00:48:07.982380+0000","last_clean":"2024-03-18T00:48:07.982380+0000","last_became_active":"2024-03-18T00:48:05.534907+0000","last_became_peered":"2024-03-18T00:48:05.534907+0000","last_unstale":"2024-03-18T00:48:07.982380+0000","last_undegraded":"2024-03-18T00:48:07.982380+0000","last_fullsized":"2024-03-18T00:48:07.982380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:47:50.196929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.653538+0000","last_change":"2024-03-18T00:48:06.534423+0000","last_active":"2024-03-18T00:48:07.653538+0000","last_peered":"2024-03-18T00:48:07.653538+0000","last_clean":"2024-03-18T00:48:07.653538+0000","last_became_active":"2024-03-18T00:48:06.534091+0000","last_became_peered":"2024-03-18T00:48:06.534091+0000","last_unstale":"2024-03-18T00:48:07.653538+0000","last_undegraded":"2024-03-18T00:48:07.653538+0000","last_fullsized":"2024-03-18T00:48:07.653538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:39:08.755634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.530685+0000","last_change":"2024-03-18T00:48:05.532891+0000","last_active":"2024-03-18T00:48:06.530685+0000","last_peered":"2024-03-18T00:48:06.530685+0000","last_clean":"2024-03-18T00:48:06.530685+0000","last_became_active":"2024-03-18T00:48:05.532515+0000","last_became_peered":"2024-03-18T00:48:05.532515+0000","last_unstale":"2024-03-18T00:48:06.530685+0000","last_undegraded":"2024-03-18T00:48:06.530685+0000","last_fullsized":"2024-03-18T00:48:06.530685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:46:52.056837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.653438+0000","last_change":"2024-03-18T00:48:06.542992+0000","last_active":"2024-03-18T00:48:07.653438+0000","last_peered":"2024-03-18T00:48:07.653438+0000","last_clean":"2024-03-18T00:48:07.653438+0000","last_became_active":"2024-03-18T00:48:06.542617+0000","last_became_peered":"2024-03-18T00:48:06.542617+0000","last_unstale":"2024-03-18T00:48:07.653438+0000","last_undegraded":"2024-03-18T00:48:07.653438+0000","last_fullsized":"2024-03-18T00:48:07.653438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:22.736118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536275+0000","last_change":"2024-03-18T00:48:06.536018+0000","last_active":"2024-03-18T00:48:07.536275+0000","last_peered":"2024-03-18T00:48:07.536275+0000","last_clean":"2024-03-18T00:48:07.536275+0000","last_became_active":"2024-03-18T00:48:06.535656+0000","last_became_peered":"2024-03-18T00:48:06.535656+0000","last_unstale":"2024-03-18T00:48:07.536275+0000","last_undegraded":"2024-03-18T00:48:07.536275+0000","last_fullsized":"2024-03-18T00:48:07.536275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:33:00.492887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.516602+0000","last_change":"2024-03-18T00:48:05.536635+0000","last_active":"2024-03-18T00:48:06.516602+0000","last_peered":"2024-03-18T00:48:06.516602+0000","last_clean":"2024-03-18T00:48:06.516602+0000","last_became_active":"2024-03-18T00:48:05.535941+0000","last_became_peered":"2024-03-18T00:48:05.535941+0000","last_unstale":"2024-03-18T00:48:06.516602+0000","last_undegraded":"2024-03-18T00:48:06.516602+0000","last_fullsized":"2024-03-18T00:48:06.516602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:20:52.167217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535650+0000","last_change":"2024-03-18T00:48:06.546266+0000","last_active":"2024-03-18T00:48:07.535650+0000","last_peered":"2024-03-18T00:48:07.535650+0000","last_clean":"2024-03-18T00:48:07.535650+0000","last_became_active":"2024-03-18T00:48:06.543565+0000","last_became_peered":"2024-03-18T00:48:06.543565+0000","last_unstale":"2024-03-18T00:48:07.535650+0000","last_undegraded":"2024-03-18T00:48:07.535650+0000","last_fullsized":"2024-03-18T00:48:07.535650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:49:09.673212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:10.005272+0000","last_change":"2024-03-18T00:48:05.526184+0000","last_active":"2024-03-18T00:48:10.005272+0000","last_peered":"2024-03-18T00:48:10.005272+0000","last_clean":"2024-03-18T00:48:10.005272+0000","last_became_active":"2024-03-18T00:48:05.525011+0000","last_became_peered":"2024-03-18T00:48:05.525011+0000","last_unstale":"2024-03-18T00:48:10.005272+0000","last_undegraded":"2024-03-18T00:48:10.005272+0000","last_fullsized":"2024-03-18T00:48:10.005272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:12.690966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":338,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.526335+0000","last_change":"2024-03-18T00:46:53.288768+0000","last_active":"2024-03-18T00:48:06.526335+0000","last_peered":"2024-03-18T00:48:06.526335+0000","last_clean":"2024-03-18T00:48:06.526335+0000","last_became_active":"2024-03-18T00:46:53.288394+0000","last_became_peered":"2024-03-18T00:46:53.288394+0000","last_unstale":"2024-03-18T00:48:06.526335+0000","last_undegraded":"2024-03-18T00:48:06.526335+0000","last_fullsized":"2024-03-18T00:48:06.526335+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_clean_scrub_stamp":"2024-03-18T00:43:28.173150+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:49.949940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.534926+0000","last_change":"2024-03-18T00:48:06.534383+0000","last_active":"2024-03-18T00:48:06.534926+0000","last_peered":"2024-03-18T00:48:06.534926+0000","last_clean":"2024-03-18T00:48:06.534926+0000","last_became_active":"2024-03-18T00:48:06.534085+0000","last_became_peered":"2024-03-18T00:48:06.534085+0000","last_unstale":"2024-03-18T00:48:06.534926+0000","last_undegraded":"2024-03-18T00:48:06.534926+0000","last_fullsized":"2024-03-18T00:48:06.534926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:24:52.854412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.526580+0000","last_change":"2024-03-18T00:48:05.534298+0000","last_active":"2024-03-18T00:48:06.526580+0000","last_peered":"2024-03-18T00:48:06.526580+0000","last_clean":"2024-03-18T00:48:06.526580+0000","last_became_active":"2024-03-18T00:48:05.533970+0000","last_became_peered":"2024-03-18T00:48:05.533970+0000","last_unstale":"2024-03-18T00:48:06.526580+0000","last_undegraded":"2024-03-18T00:48:06.526580+0000","last_fullsized":"2024-03-18T00:48:06.526580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:02:13.757387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.991564+0000","last_change":"2024-03-18T00:48:05.532472+0000","last_active":"2024-03-18T00:48:07.991564+0000","last_peered":"2024-03-18T00:48:07.991564+0000","last_clean":"2024-03-18T00:48:07.991564+0000","last_became_active":"2024-03-18T00:48:05.532122+0000","last_became_peered":"2024-03-18T00:48:05.532122+0000","last_unstale":"2024-03-18T00:48:07.991564+0000","last_undegraded":"2024-03-18T00:48:07.991564+0000","last_fullsized":"2024-03-18T00:48:07.991564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:49:07.545179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.530531+0000","last_change":"2024-03-18T00:48:06.530108+0000","last_active":"2024-03-18T00:48:06.530531+0000","last_peered":"2024-03-18T00:48:06.530531+0000","last_clean":"2024-03-18T00:48:06.530531+0000","last_became_active":"2024-03-18T00:48:06.529768+0000","last_became_peered":"2024-03-18T00:48:06.529768+0000","last_unstale":"2024-03-18T00:48:06.530531+0000","last_undegraded":"2024-03-18T00:48:06.530531+0000","last_fullsized":"2024-03-18T00:48:06.530531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:17:18.062729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:08.997525+0000","last_change":"2024-03-18T00:48:05.535703+0000","last_active":"2024-03-18T00:48:08.997525+0000","last_peered":"2024-03-18T00:48:08.997525+0000","last_clean":"2024-03-18T00:48:08.997525+0000","last_became_active":"2024-03-18T00:48:05.535086+0000","last_became_peered":"2024-03-18T00:48:05.535086+0000","last_unstale":"2024-03-18T00:48:08.997525+0000","last_undegraded":"2024-03-18T00:48:08.997525+0000","last_fullsized":"2024-03-18T00:48:08.997525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:10:24.172353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535941+0000","last_change":"2024-03-18T00:48:06.541281+0000","last_active":"2024-03-18T00:48:07.535941+0000","last_peered":"2024-03-18T00:48:07.535941+0000","last_clean":"2024-03-18T00:48:07.535941+0000","last_became_active":"2024-03-18T00:48:06.540946+0000","last_became_peered":"2024-03-18T00:48:06.540946+0000","last_unstale":"2024-03-18T00:48:07.535941+0000","last_undegraded":"2024-03-18T00:48:07.535941+0000","last_fullsized":"2024-03-18T00:48:07.535941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:02.987661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:11.012766+0000","last_change":"2024-03-18T00:48:05.531685+0000","last_active":"2024-03-18T00:48:11.012766+0000","last_peered":"2024-03-18T00:48:11.012766+0000","last_clean":"2024-03-18T00:48:11.012766+0000","last_became_active":"2024-03-18T00:48:05.531239+0000","last_became_peered":"2024-03-18T00:48:05.531239+0000","last_unstale":"2024-03-18T00:48:11.012766+0000","last_undegraded":"2024-03-18T00:48:11.012766+0000","last_fullsized":"2024-03-18T00:48:11.012766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:26:35.034842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984746+0000","last_change":"2024-03-18T00:48:06.546783+0000","last_active":"2024-03-18T00:48:07.984746+0000","last_peered":"2024-03-18T00:48:07.984746+0000","last_clean":"2024-03-18T00:48:07.984746+0000","last_became_active":"2024-03-18T00:48:06.546411+0000","last_became_peered":"2024-03-18T00:48:06.546411+0000","last_unstale":"2024-03-18T00:48:07.984746+0000","last_undegraded":"2024-03-18T00:48:07.984746+0000","last_fullsized":"2024-03-18T00:48:07.984746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:39:34.432104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.518189+0000","last_change":"2024-03-18T00:48:05.534701+0000","last_active":"2024-03-18T00:48:06.518189+0000","last_peered":"2024-03-18T00:48:06.518189+0000","last_clean":"2024-03-18T00:48:06.518189+0000","last_became_active":"2024-03-18T00:48:05.534300+0000","last_became_peered":"2024-03-18T00:48:05.534300+0000","last_unstale":"2024-03-18T00:48:06.518189+0000","last_undegraded":"2024-03-18T00:48:06.518189+0000","last_fullsized":"2024-03-18T00:48:06.518189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:10.580001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.988954+0000","last_change":"2024-03-18T00:48:06.528286+0000","last_active":"2024-03-18T00:48:07.988954+0000","last_peered":"2024-03-18T00:48:07.988954+0000","last_clean":"2024-03-18T00:48:07.988954+0000","last_became_active":"2024-03-18T00:48:06.527947+0000","last_became_peered":"2024-03-18T00:48:06.527947+0000","last_unstale":"2024-03-18T00:48:07.988954+0000","last_undegraded":"2024-03-18T00:48:07.988954+0000","last_fullsized":"2024-03-18T00:48:07.988954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:34.953923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:11.012781+0000","last_change":"2024-03-18T00:48:05.526530+0000","last_active":"2024-03-18T00:48:11.012781+0000","last_peered":"2024-03-18T00:48:11.012781+0000","last_clean":"2024-03-18T00:48:11.012781+0000","last_became_active":"2024-03-18T00:48:05.526210+0000","last_became_peered":"2024-03-18T00:48:05.526210+0000","last_unstale":"2024-03-18T00:48:11.012781+0000","last_undegraded":"2024-03-18T00:48:11.012781+0000","last_fullsized":"2024-03-18T00:48:11.012781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:11:11.094469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.537490+0000","last_change":"2024-03-18T00:48:06.535316+0000","last_active":"2024-03-18T00:48:06.537490+0000","last_peered":"2024-03-18T00:48:06.537490+0000","last_clean":"2024-03-18T00:48:06.537490+0000","last_became_active":"2024-03-18T00:48:06.534945+0000","last_became_peered":"2024-03-18T00:48:06.534945+0000","last_unstale":"2024-03-18T00:48:06.537490+0000","last_undegraded":"2024-03-18T00:48:06.537490+0000","last_fullsized":"2024-03-18T00:48:06.537490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:30:49.085979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.518194+0000","last_change":"2024-03-18T00:48:05.529300+0000","last_active":"2024-03-18T00:48:06.518194+0000","last_peered":"2024-03-18T00:48:06.518194+0000","last_clean":"2024-03-18T00:48:06.518194+0000","last_became_active":"2024-03-18T00:48:05.528935+0000","last_became_peered":"2024-03-18T00:48:05.528935+0000","last_unstale":"2024-03-18T00:48:06.518194+0000","last_undegraded":"2024-03-18T00:48:06.518194+0000","last_fullsized":"2024-03-18T00:48:06.518194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:29:45.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535866+0000","last_change":"2024-03-18T00:48:06.536426+0000","last_active":"2024-03-18T00:48:07.535866+0000","last_peered":"2024-03-18T00:48:07.535866+0000","last_clean":"2024-03-18T00:48:07.535866+0000","last_became_active":"2024-03-18T00:48:06.536005+0000","last_became_peered":"2024-03-18T00:48:06.536005+0000","last_unstale":"2024-03-18T00:48:07.535866+0000","last_undegraded":"2024-03-18T00:48:07.535866+0000","last_fullsized":"2024-03-18T00:48:07.535866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:40:52.028794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.991868+0000","last_change":"2024-03-18T00:48:05.536495+0000","last_active":"2024-03-18T00:48:07.991868+0000","last_peered":"2024-03-18T00:48:07.991868+0000","last_clean":"2024-03-18T00:48:07.991868+0000","last_became_active":"2024-03-18T00:48:05.536072+0000","last_became_peered":"2024-03-18T00:48:05.536072+0000","last_unstale":"2024-03-18T00:48:07.991868+0000","last_undegraded":"2024-03-18T00:48:07.991868+0000","last_fullsized":"2024-03-18T00:48:07.991868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:25.528261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.553497+0000","last_change":"2024-03-18T00:48:06.552900+0000","last_active":"2024-03-18T00:48:06.553497+0000","last_peered":"2024-03-18T00:48:06.553497+0000","last_clean":"2024-03-18T00:48:06.553497+0000","last_became_active":"2024-03-18T00:48:06.552568+0000","last_became_peered":"2024-03-18T00:48:06.552568+0000","last_unstale":"2024-03-18T00:48:06.553497+0000","last_undegraded":"2024-03-18T00:48:06.553497+0000","last_fullsized":"2024-03-18T00:48:06.553497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:15:53.518654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.518802+0000","last_change":"2024-03-18T00:48:05.537327+0000","last_active":"2024-03-18T00:48:06.518802+0000","last_peered":"2024-03-18T00:48:06.518802+0000","last_clean":"2024-03-18T00:48:06.518802+0000","last_became_active":"2024-03-18T00:48:05.536825+0000","last_became_peered":"2024-03-18T00:48:05.536825+0000","last_unstale":"2024-03-18T00:48:06.518802+0000","last_undegraded":"2024-03-18T00:48:06.518802+0000","last_fullsized":"2024-03-18T00:48:06.518802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:03.141142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546685+0000","last_change":"2024-03-18T00:48:06.542593+0000","last_active":"2024-03-18T00:48:06.546685+0000","last_peered":"2024-03-18T00:48:06.546685+0000","last_clean":"2024-03-18T00:48:06.546685+0000","last_became_active":"2024-03-18T00:48:06.541624+0000","last_became_peered":"2024-03-18T00:48:06.541624+0000","last_unstale":"2024-03-18T00:48:06.546685+0000","last_undegraded":"2024-03-18T00:48:06.546685+0000","last_fullsized":"2024-03-18T00:48:06.546685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:19:16.021707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536201+0000","last_change":"2024-03-18T00:48:05.529140+0000","last_active":"2024-03-18T00:48:07.536201+0000","last_peered":"2024-03-18T00:48:07.536201+0000","last_clean":"2024-03-18T00:48:07.536201+0000","last_became_active":"2024-03-18T00:48:05.528682+0000","last_became_peered":"2024-03-18T00:48:05.528682+0000","last_unstale":"2024-03-18T00:48:07.536201+0000","last_undegraded":"2024-03-18T00:48:07.536201+0000","last_fullsized":"2024-03-18T00:48:07.536201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:57:52.663150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546947+0000","last_change":"2024-03-18T00:48:06.546027+0000","last_active":"2024-03-18T00:48:06.546947+0000","last_peered":"2024-03-18T00:48:06.546947+0000","last_clean":"2024-03-18T00:48:06.546947+0000","last_became_active":"2024-03-18T00:48:06.545670+0000","last_became_peered":"2024-03-18T00:48:06.545670+0000","last_unstale":"2024-03-18T00:48:06.546947+0000","last_undegraded":"2024-03-18T00:48:06.546947+0000","last_fullsized":"2024-03-18T00:48:06.546947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:43:40.816065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.551728+0000","last_change":"2024-03-18T00:48:05.526929+0000","last_active":"2024-03-18T00:48:06.551728+0000","last_peered":"2024-03-18T00:48:06.551728+0000","last_clean":"2024-03-18T00:48:06.551728+0000","last_became_active":"2024-03-18T00:48:05.526530+0000","last_became_peered":"2024-03-18T00:48:05.526530+0000","last_unstale":"2024-03-18T00:48:06.551728+0000","last_undegraded":"2024-03-18T00:48:06.551728+0000","last_fullsized":"2024-03-18T00:48:06.551728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:51:25.110895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546546+0000","last_change":"2024-03-18T00:48:06.541223+0000","last_active":"2024-03-18T00:48:06.546546+0000","last_peered":"2024-03-18T00:48:06.546546+0000","last_clean":"2024-03-18T00:48:06.546546+0000","last_became_active":"2024-03-18T00:48:06.540898+0000","last_became_peered":"2024-03-18T00:48:06.540898+0000","last_unstale":"2024-03-18T00:48:06.546546+0000","last_undegraded":"2024-03-18T00:48:06.546546+0000","last_fullsized":"2024-03-18T00:48:06.546546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:31:23.356867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.534267+0000","last_change":"2024-03-18T00:48:06.532906+0000","last_active":"2024-03-18T00:48:06.534267+0000","last_peered":"2024-03-18T00:48:06.534267+0000","last_clean":"2024-03-18T00:48:06.534267+0000","last_became_active":"2024-03-18T00:48:06.532583+0000","last_became_peered":"2024-03-18T00:48:06.532583+0000","last_unstale":"2024-03-18T00:48:06.534267+0000","last_undegraded":"2024-03-18T00:48:06.534267+0000","last_fullsized":"2024-03-18T00:48:06.534267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:31:16.565495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.531717+0000","last_change":"2024-03-18T00:48:06.530020+0000","last_active":"2024-03-18T00:48:06.531717+0000","last_peered":"2024-03-18T00:48:06.531717+0000","last_clean":"2024-03-18T00:48:06.531717+0000","last_became_active":"2024-03-18T00:48:06.529709+0000","last_became_peered":"2024-03-18T00:48:06.529709+0000","last_unstale":"2024-03-18T00:48:06.531717+0000","last_undegraded":"2024-03-18T00:48:06.531717+0000","last_fullsized":"2024-03-18T00:48:06.531717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:26.325479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.538966+0000","last_change":"2024-03-18T00:48:06.537658+0000","last_active":"2024-03-18T00:48:06.538966+0000","last_peered":"2024-03-18T00:48:06.538966+0000","last_clean":"2024-03-18T00:48:06.538966+0000","last_became_active":"2024-03-18T00:48:06.537339+0000","last_became_peered":"2024-03-18T00:48:06.537339+0000","last_unstale":"2024-03-18T00:48:06.538966+0000","last_undegraded":"2024-03-18T00:48:06.538966+0000","last_fullsized":"2024-03-18T00:48:06.538966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:00:50.867863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.985132+0000","last_change":"2024-03-18T00:48:06.544514+0000","last_active":"2024-03-18T00:48:07.985132+0000","last_peered":"2024-03-18T00:48:07.985132+0000","last_clean":"2024-03-18T00:48:07.985132+0000","last_became_active":"2024-03-18T00:48:06.544165+0000","last_became_peered":"2024-03-18T00:48:06.544165+0000","last_unstale":"2024-03-18T00:48:07.985132+0000","last_undegraded":"2024-03-18T00:48:07.985132+0000","last_fullsized":"2024-03-18T00:48:07.985132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:48:02.498940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.536842+0000","last_change":"2024-03-18T00:48:06.536208+0000","last_active":"2024-03-18T00:48:06.536842+0000","last_peered":"2024-03-18T00:48:06.536842+0000","last_clean":"2024-03-18T00:48:06.536842+0000","last_became_active":"2024-03-18T00:48:06.535920+0000","last_became_peered":"2024-03-18T00:48:06.535920+0000","last_unstale":"2024-03-18T00:48:06.536842+0000","last_undegraded":"2024-03-18T00:48:06.536842+0000","last_fullsized":"2024-03-18T00:48:06.536842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:28:49.720721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.536110+0000","last_change":"2024-03-18T00:48:06.533837+0000","last_active":"2024-03-18T00:48:06.536110+0000","last_peered":"2024-03-18T00:48:06.536110+0000","last_clean":"2024-03-18T00:48:06.536110+0000","last_became_active":"2024-03-18T00:48:06.533418+0000","last_became_peered":"2024-03-18T00:48:06.533418+0000","last_unstale":"2024-03-18T00:48:06.536110+0000","last_undegraded":"2024-03-18T00:48:06.536110+0000","last_fullsized":"2024-03-18T00:48:06.536110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:04:15.025748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.529224+0000","last_change":"2024-03-18T00:48:06.527394+0000","last_active":"2024-03-18T00:48:06.529224+0000","last_peered":"2024-03-18T00:48:06.529224+0000","last_clean":"2024-03-18T00:48:06.529224+0000","last_became_active":"2024-03-18T00:48:06.526879+0000","last_became_peered":"2024-03-18T00:48:06.526879+0000","last_unstale":"2024-03-18T00:48:06.529224+0000","last_undegraded":"2024-03-18T00:48:06.529224+0000","last_fullsized":"2024-03-18T00:48:06.529224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:30:30.074091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535869+0000","last_change":"2024-03-18T00:48:06.542645+0000","last_active":"2024-03-18T00:48:07.535869+0000","last_peered":"2024-03-18T00:48:07.535869+0000","last_clean":"2024-03-18T00:48:07.535869+0000","last_became_active":"2024-03-18T00:48:06.542073+0000","last_became_peered":"2024-03-18T00:48:06.542073+0000","last_unstale":"2024-03-18T00:48:07.535869+0000","last_undegraded":"2024-03-18T00:48:07.535869+0000","last_fullsized":"2024-03-18T00:48:07.535869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:08:44.479318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536173+0000","last_change":"2024-03-18T00:48:06.541698+0000","last_active":"2024-03-18T00:48:07.536173+0000","last_peered":"2024-03-18T00:48:07.536173+0000","last_clean":"2024-03-18T00:48:07.536173+0000","last_became_active":"2024-03-18T00:48:06.541378+0000","last_became_peered":"2024-03-18T00:48:06.541378+0000","last_unstale":"2024-03-18T00:48:07.536173+0000","last_undegraded":"2024-03-18T00:48:07.536173+0000","last_fullsized":"2024-03-18T00:48:07.536173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:42:23.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535289+0000","last_change":"2024-03-18T00:48:06.533184+0000","last_active":"2024-03-18T00:48:07.535289+0000","last_peered":"2024-03-18T00:48:07.535289+0000","last_clean":"2024-03-18T00:48:07.535289+0000","last_became_active":"2024-03-18T00:48:06.532034+0000","last_became_peered":"2024-03-18T00:48:06.532034+0000","last_unstale":"2024-03-18T00:48:07.535289+0000","last_undegraded":"2024-03-18T00:48:07.535289+0000","last_fullsized":"2024-03-18T00:48:07.535289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:45.265799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546822+0000","last_change":"2024-03-18T00:48:06.545941+0000","last_active":"2024-03-18T00:48:06.546822+0000","last_peered":"2024-03-18T00:48:06.546822+0000","last_clean":"2024-03-18T00:48:06.546822+0000","last_became_active":"2024-03-18T00:48:06.545402+0000","last_became_peered":"2024-03-18T00:48:06.545402+0000","last_unstale":"2024-03-18T00:48:06.546822+0000","last_undegraded":"2024-03-18T00:48:06.546822+0000","last_fullsized":"2024-03-18T00:48:06.546822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:54:13.711332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.544058+0000","last_change":"2024-03-18T00:48:06.541513+0000","last_active":"2024-03-18T00:48:06.544058+0000","last_peered":"2024-03-18T00:48:06.544058+0000","last_clean":"2024-03-18T00:48:06.544058+0000","last_became_active":"2024-03-18T00:48:06.541128+0000","last_became_peered":"2024-03-18T00:48:06.541128+0000","last_unstale":"2024-03-18T00:48:06.544058+0000","last_undegraded":"2024-03-18T00:48:06.544058+0000","last_fullsized":"2024-03-18T00:48:06.544058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:03:24.725850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.543745+0000","last_change":"2024-03-18T00:48:06.542316+0000","last_active":"2024-03-18T00:48:06.543745+0000","last_peered":"2024-03-18T00:48:06.543745+0000","last_clean":"2024-03-18T00:48:06.543745+0000","last_became_active":"2024-03-18T00:48:06.541914+0000","last_became_peered":"2024-03-18T00:48:06.541914+0000","last_unstale":"2024-03-18T00:48:06.543745+0000","last_undegraded":"2024-03-18T00:48:06.543745+0000","last_fullsized":"2024-03-18T00:48:06.543745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:16:18.226087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249372,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5647,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":13051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":897024,"data_stored":423925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1035130,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":433606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":1023952,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462951,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":909312,"data_stored":433002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":921600,"data_stored":434614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":913408,"data_stored":433092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":897024,"data_stored":423925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":441980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":442238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":1034178,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":3314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T00:48:12.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T00:48:13.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.001+0000 7f27a5740640 1 -- 172.21.15.49:0/4246843603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a0100700 msgr2=0x7f27a0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:13.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.001+0000 7f27a5740640 1 --2- 172.21.15.49:0/4246843603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a0100700 0x7f27a0100b20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2790009900 tx=0x7f279002bc30 comp rx=0 tx=0).stop 2024-03-18T00:48:13.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.001+0000 7f27a5740640 1 -- 172.21.15.49:0/4246843603 shutdown_connections 2024-03-18T00:48:13.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.001+0000 7f27a5740640 1 --2- 172.21.15.49:0/4246843603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27a01018f0 0x7f27a0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.001+0000 7f27a5740640 1 --2- 172.21.15.49:0/4246843603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01022b0 0x7f27a0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.001+0000 7f27a5740640 1 --2- 172.21.15.49:0/4246843603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a0100700 0x7f27a0100b20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.001+0000 7f27a5740640 1 -- 172.21.15.49:0/4246843603 >> 172.21.15.49:0/4246843603 conn(0x7f27a00fbc70 msgr2=0x7f27a00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:13.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 -- 172.21.15.49:0/4246843603 shutdown_connections 2024-03-18T00:48:13.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 -- 172.21.15.49:0/4246843603 wait complete. 2024-03-18T00:48:13.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 Processor -- start 2024-03-18T00:48:13.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 -- start start 2024-03-18T00:48:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27a0100700 0x7f27a0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a01018f0 0x7f27a0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01022b0 0x7f27a019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f27a010d4a0 con 0x7f27a01018f0 2024-03-18T00:48:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f27a010d320 con 0x7f27a0100700 2024-03-18T00:48:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f27a5740640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27a010d620 con 0x7f27a01022b0 2024-03-18T00:48:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.002+0000 7f2797fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a01018f0 0x7f27a0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f2797fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a01018f0 0x7f27a0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60798/0 (socket says 172.21.15.49:60798) 2024-03-18T00:48:13.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f2797fff640 1 -- 172.21.15.49:0/812299158 learned_addr learned my addr 172.21.15.49:0/812299158 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:13.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f279f7fe640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01022b0 0x7f27a019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:13.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f279effd640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27a0100700 0x7f27a0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:13.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f2797fff640 1 -- 172.21.15.49:0/812299158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01022b0 msgr2=0x7f27a019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:13.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f2797fff640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01022b0 0x7f27a019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f2797fff640 1 -- 172.21.15.49:0/812299158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27a0100700 msgr2=0x7f27a0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:13.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f2797fff640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27a0100700 0x7f27a0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f2797fff640 1 -- 172.21.15.49:0/812299158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27a019f710 con 0x7f27a01018f0 2024-03-18T00:48:13.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f2797fff640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a01018f0 0x7f27a0198030 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f278800e9f0 tx=0x7f278800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:13.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f279cff9640 1 -- 172.21.15.49:0/812299158 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f278800cd80 con 0x7f27a01018f0 2024-03-18T00:48:13.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f279cff9640 1 -- 172.21.15.49:0/812299158 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2788010910 con 0x7f27a01018f0 2024-03-18T00:48:13.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27a019fa00 con 0x7f27a01018f0 2024-03-18T00:48:13.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f279cff9640 1 -- 172.21.15.49:0/812299158 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2788018990 con 0x7f27a01018f0 2024-03-18T00:48:13.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.003+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27a019ffa0 con 0x7f27a01018f0 2024-03-18T00:48:13.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.004+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27a0107a20 con 0x7f27a01018f0 2024-03-18T00:48:13.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.008+0000 7f279cff9640 1 -- 172.21.15.49:0/812299158 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2788010ac0 con 0x7f27a01018f0 2024-03-18T00:48:13.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.009+0000 7f279cff9640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f278c0a2910 0x7f278c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:13.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.009+0000 7f279effd640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f278c0a2910 0x7f278c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:13.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.009+0000 7f279cff9640 1 -- 172.21.15.49:0/812299158 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f27880cf1d0 con 0x7f27a01018f0 2024-03-18T00:48:13.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.010+0000 7f279effd640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f278c0a2910 0x7f278c0a4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2790000c00 tx=0x7f2790005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:13.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.013+0000 7f279cff9640 1 -- 172.21.15.49:0/812299158 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f278809bb70 con 0x7f27a01018f0 2024-03-18T00:48:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:13 smithi049 ceph-mon[32130]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:13 smithi049 ceph-mon[32130]: pgmap v291: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s wr, 19 op/s 2024-03-18T00:48:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:13 smithi049 ceph-mon[32130]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:13 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] up:active 2024-03-18T00:48:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:13 smithi049 ceph-mon[32130]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-18T00:48:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:13 smithi049 ceph-mon[32130]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T00:48:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:13 smithi049 ceph-mon[32130]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:48:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:13 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2592549453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:48:13.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:13 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3326094334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:13.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.196+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f27a0061c20 con 0x7f278c0a2910 2024-03-18T00:48:13.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.204+0000 7f279cff9640 1 -- 172.21.15.49:0/812299158 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258895 (secure 0 0 0) 0x7f27a0061c20 con 0x7f278c0a2910 2024-03-18T00:48:13.207 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:13.208 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.206+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f278c0a2910 msgr2=0x7f278c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f278c0a2910 0x7f278c0a4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2790000c00 tx=0x7f2790005fd0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a01018f0 msgr2=0x7f27a0198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a01018f0 0x7f27a0198030 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f278800e9f0 tx=0x7f278800eeb0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 shutdown_connections 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f278c0a2910 0x7f278c0a4dd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27a01022b0 0x7f27a019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27a01018f0 0x7f27a0198030 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 --2- 172.21.15.49:0/812299158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27a0100700 0x7f27a0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:13.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 >> 172.21.15.49:0/812299158 conn(0x7f27a00fbc70 msgr2=0x7f27a0104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:13.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 shutdown_connections 2024-03-18T00:48:13.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:13.207+0000 7f27a5740640 1 -- 172.21.15.49:0/812299158 wait complete. 2024-03-18T00:48:13.218 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-03-18T00:48:11.460422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596015,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":300,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":238,"ondisk_log_size":238,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356060,"kb_used_data":11004,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585764,"statfs":{"total":1151940427776,"available":1151575822336,"internally_reserved":0,"allocated":11268096,"data_stored":7001998,"data_compressed":108276,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5647,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001388"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.537623+0000","last_change":"2024-03-18T00:48:06.536034+0000","last_active":"2024-03-18T00:48:06.537623+0000","last_peered":"2024-03-18T00:48:06.537623+0000","last_clean":"2024-03-18T00:48:06.537623+0000","last_became_active":"2024-03-18T00:48:06.535694+0000","last_became_peered":"2024-03-18T00:48:06.535694+0000","last_unstale":"2024-03-18T00:48:06.537623+0000","last_undegraded":"2024-03-18T00:48:06.537623+0000","last_fullsized":"2024-03-18T00:48:06.537623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:44:56.816104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984930+0000","last_change":"2024-03-18T00:48:06.536741+0000","last_active":"2024-03-18T00:48:07.984930+0000","last_peered":"2024-03-18T00:48:07.984930+0000","last_clean":"2024-03-18T00:48:07.984930+0000","last_became_active":"2024-03-18T00:48:06.535865+0000","last_became_peered":"2024-03-18T00:48:06.535865+0000","last_unstale":"2024-03-18T00:48:07.984930+0000","last_undegraded":"2024-03-18T00:48:07.984930+0000","last_fullsized":"2024-03-18T00:48:07.984930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:58.104983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.985112+0000","last_change":"2024-03-18T00:48:06.537162+0000","last_active":"2024-03-18T00:48:07.985112+0000","last_peered":"2024-03-18T00:48:07.985112+0000","last_clean":"2024-03-18T00:48:07.985112+0000","last_became_active":"2024-03-18T00:48:06.536835+0000","last_became_peered":"2024-03-18T00:48:06.536835+0000","last_unstale":"2024-03-18T00:48:07.985112+0000","last_undegraded":"2024-03-18T00:48:07.985112+0000","last_fullsized":"2024-03-18T00:48:07.985112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:15:08.519180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536822+0000","last_change":"2024-03-18T00:48:06.540084+0000","last_active":"2024-03-18T00:48:07.536822+0000","last_peered":"2024-03-18T00:48:07.536822+0000","last_clean":"2024-03-18T00:48:07.536822+0000","last_became_active":"2024-03-18T00:48:06.539736+0000","last_became_peered":"2024-03-18T00:48:06.539736+0000","last_unstale":"2024-03-18T00:48:07.536822+0000","last_undegraded":"2024-03-18T00:48:07.536822+0000","last_fullsized":"2024-03-18T00:48:07.536822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:37:51.748603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989527+0000","last_change":"2024-03-18T00:48:06.540022+0000","last_active":"2024-03-18T00:48:07.989527+0000","last_peered":"2024-03-18T00:48:07.989527+0000","last_clean":"2024-03-18T00:48:07.989527+0000","last_became_active":"2024-03-18T00:48:06.539665+0000","last_became_peered":"2024-03-18T00:48:06.539665+0000","last_unstale":"2024-03-18T00:48:07.989527+0000","last_undegraded":"2024-03-18T00:48:07.989527+0000","last_fullsized":"2024-03-18T00:48:07.989527+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:55:52.448279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.653245+0000","last_change":"2024-03-18T00:48:06.538672+0000","last_active":"2024-03-18T00:48:07.653245+0000","last_peered":"2024-03-18T00:48:07.653245+0000","last_clean":"2024-03-18T00:48:07.653245+0000","last_became_active":"2024-03-18T00:48:06.538189+0000","last_became_peered":"2024-03-18T00:48:06.538189+0000","last_unstale":"2024-03-18T00:48:07.653245+0000","last_undegraded":"2024-03-18T00:48:07.653245+0000","last_fullsized":"2024-03-18T00:48:07.653245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:09:04.672255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.547139+0000","last_change":"2024-03-18T00:48:06.546417+0000","last_active":"2024-03-18T00:48:06.547139+0000","last_peered":"2024-03-18T00:48:06.547139+0000","last_clean":"2024-03-18T00:48:06.547139+0000","last_became_active":"2024-03-18T00:48:06.545759+0000","last_became_peered":"2024-03-18T00:48:06.545759+0000","last_unstale":"2024-03-18T00:48:06.547139+0000","last_undegraded":"2024-03-18T00:48:06.547139+0000","last_fullsized":"2024-03-18T00:48:06.547139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:35:37.878207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535247+0000","last_change":"2024-03-18T00:48:06.545875+0000","last_active":"2024-03-18T00:48:07.535247+0000","last_peered":"2024-03-18T00:48:07.535247+0000","last_clean":"2024-03-18T00:48:07.535247+0000","last_became_active":"2024-03-18T00:48:06.545316+0000","last_became_peered":"2024-03-18T00:48:06.545316+0000","last_unstale":"2024-03-18T00:48:07.535247+0000","last_undegraded":"2024-03-18T00:48:07.535247+0000","last_fullsized":"2024-03-18T00:48:07.535247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:22:40.998671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.534852+0000","last_change":"2024-03-18T00:48:06.533947+0000","last_active":"2024-03-18T00:48:06.534852+0000","last_peered":"2024-03-18T00:48:06.534852+0000","last_clean":"2024-03-18T00:48:06.534852+0000","last_became_active":"2024-03-18T00:48:06.533615+0000","last_became_peered":"2024-03-18T00:48:06.533615+0000","last_unstale":"2024-03-18T00:48:06.534852+0000","last_undegraded":"2024-03-18T00:48:06.534852+0000","last_fullsized":"2024-03-18T00:48:06.534852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:41:17.501153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.653533+0000","last_change":"2024-03-18T00:48:06.537614+0000","last_active":"2024-03-18T00:48:07.653533+0000","last_peered":"2024-03-18T00:48:07.653533+0000","last_clean":"2024-03-18T00:48:07.653533+0000","last_became_active":"2024-03-18T00:48:06.536617+0000","last_became_peered":"2024-03-18T00:48:06.536617+0000","last_unstale":"2024-03-18T00:48:07.653533+0000","last_undegraded":"2024-03-18T00:48:07.653533+0000","last_fullsized":"2024-03-18T00:48:07.653533+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:52.222603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546599+0000","last_change":"2024-03-18T00:48:06.540363+0000","last_active":"2024-03-18T00:48:06.546599+0000","last_peered":"2024-03-18T00:48:06.546599+0000","last_clean":"2024-03-18T00:48:06.546599+0000","last_became_active":"2024-03-18T00:48:06.540011+0000","last_became_peered":"2024-03-18T00:48:06.540011+0000","last_unstale":"2024-03-18T00:48:06.546599+0000","last_undegraded":"2024-03-18T00:48:06.546599+0000","last_fullsized":"2024-03-18T00:48:06.546599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:52:47.287604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536647+0000","last_change":"2024-03-18T00:48:06.532611+0000","last_active":"2024-03-18T00:48:07.536647+0000","last_peered":"2024-03-18T00:48:07.536647+0000","last_clean":"2024-03-18T00:48:07.536647+0000","last_became_active":"2024-03-18T00:48:06.532301+0000","last_became_peered":"2024-03-18T00:48:06.532301+0000","last_unstale":"2024-03-18T00:48:07.536647+0000","last_undegraded":"2024-03-18T00:48:07.536647+0000","last_fullsized":"2024-03-18T00:48:07.536647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:40.795105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.532473+0000","last_change":"2024-03-18T00:48:06.531993+0000","last_active":"2024-03-18T00:48:06.532473+0000","last_peered":"2024-03-18T00:48:06.532473+0000","last_clean":"2024-03-18T00:48:06.532473+0000","last_became_active":"2024-03-18T00:48:06.531677+0000","last_became_peered":"2024-03-18T00:48:06.531677+0000","last_unstale":"2024-03-18T00:48:06.532473+0000","last_undegraded":"2024-03-18T00:48:06.532473+0000","last_fullsized":"2024-03-18T00:48:06.532473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:36:02.027315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984887+0000","last_change":"2024-03-18T00:48:06.536830+0000","last_active":"2024-03-18T00:48:07.984887+0000","last_peered":"2024-03-18T00:48:07.984887+0000","last_clean":"2024-03-18T00:48:07.984887+0000","last_became_active":"2024-03-18T00:48:06.535908+0000","last_became_peered":"2024-03-18T00:48:06.535908+0000","last_unstale":"2024-03-18T00:48:07.984887+0000","last_undegraded":"2024-03-18T00:48:07.984887+0000","last_fullsized":"2024-03-18T00:48:07.984887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:23:26.155130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984501+0000","last_change":"2024-03-18T00:48:06.543641+0000","last_active":"2024-03-18T00:48:07.984501+0000","last_peered":"2024-03-18T00:48:07.984501+0000","last_clean":"2024-03-18T00:48:07.984501+0000","last_became_active":"2024-03-18T00:48:06.543293+0000","last_became_peered":"2024-03-18T00:48:06.543293+0000","last_unstale":"2024-03-18T00:48:07.984501+0000","last_undegraded":"2024-03-18T00:48:07.984501+0000","last_fullsized":"2024-03-18T00:48:07.984501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:52.406255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546754+0000","last_change":"2024-03-18T00:48:06.544732+0000","last_active":"2024-03-18T00:48:06.546754+0000","last_peered":"2024-03-18T00:48:06.546754+0000","last_clean":"2024-03-18T00:48:06.546754+0000","last_became_active":"2024-03-18T00:48:06.544400+0000","last_became_peered":"2024-03-18T00:48:06.544400+0000","last_unstale":"2024-03-18T00:48:06.546754+0000","last_undegraded":"2024-03-18T00:48:06.546754+0000","last_fullsized":"2024-03-18T00:48:06.546754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:51:54.744799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.537522+0000","last_change":"2024-03-18T00:48:06.536595+0000","last_active":"2024-03-18T00:48:06.537522+0000","last_peered":"2024-03-18T00:48:06.537522+0000","last_clean":"2024-03-18T00:48:06.537522+0000","last_became_active":"2024-03-18T00:48:06.535874+0000","last_became_peered":"2024-03-18T00:48:06.535874+0000","last_unstale":"2024-03-18T00:48:06.537522+0000","last_undegraded":"2024-03-18T00:48:06.537522+0000","last_fullsized":"2024-03-18T00:48:06.537522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:38:10.958438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.982813+0000","last_change":"2024-03-18T00:48:06.546676+0000","last_active":"2024-03-18T00:48:07.982813+0000","last_peered":"2024-03-18T00:48:07.982813+0000","last_clean":"2024-03-18T00:48:07.982813+0000","last_became_active":"2024-03-18T00:48:06.546356+0000","last_became_peered":"2024-03-18T00:48:06.546356+0000","last_unstale":"2024-03-18T00:48:07.982813+0000","last_undegraded":"2024-03-18T00:48:07.982813+0000","last_fullsized":"2024-03-18T00:48:07.982813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:04:35.467415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.532812+0000","last_change":"2024-03-18T00:48:06.532184+0000","last_active":"2024-03-18T00:48:06.532812+0000","last_peered":"2024-03-18T00:48:06.532812+0000","last_clean":"2024-03-18T00:48:06.532812+0000","last_became_active":"2024-03-18T00:48:06.531816+0000","last_became_peered":"2024-03-18T00:48:06.531816+0000","last_unstale":"2024-03-18T00:48:06.532812+0000","last_undegraded":"2024-03-18T00:48:06.532812+0000","last_fullsized":"2024-03-18T00:48:06.532812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:58:47.981121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:08.998127+0000","last_change":"2024-03-18T00:48:05.530213+0000","last_active":"2024-03-18T00:48:08.998127+0000","last_peered":"2024-03-18T00:48:08.998127+0000","last_clean":"2024-03-18T00:48:08.998127+0000","last_became_active":"2024-03-18T00:48:05.529897+0000","last_became_peered":"2024-03-18T00:48:05.529897+0000","last_unstale":"2024-03-18T00:48:08.998127+0000","last_undegraded":"2024-03-18T00:48:08.998127+0000","last_fullsized":"2024-03-18T00:48:08.998127+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:49.003362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535869+0000","last_change":"2024-03-18T00:48:06.544437+0000","last_active":"2024-03-18T00:48:07.535869+0000","last_peered":"2024-03-18T00:48:07.535869+0000","last_clean":"2024-03-18T00:48:07.535869+0000","last_became_active":"2024-03-18T00:48:06.544125+0000","last_became_peered":"2024-03-18T00:48:06.544125+0000","last_unstale":"2024-03-18T00:48:07.535869+0000","last_undegraded":"2024-03-18T00:48:07.535869+0000","last_fullsized":"2024-03-18T00:48:07.535869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:56:42.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535680+0000","last_change":"2024-03-18T00:48:05.536249+0000","last_active":"2024-03-18T00:48:07.535680+0000","last_peered":"2024-03-18T00:48:07.535680+0000","last_clean":"2024-03-18T00:48:07.535680+0000","last_became_active":"2024-03-18T00:48:05.535954+0000","last_became_peered":"2024-03-18T00:48:05.535954+0000","last_unstale":"2024-03-18T00:48:07.535680+0000","last_undegraded":"2024-03-18T00:48:07.535680+0000","last_fullsized":"2024-03-18T00:48:07.535680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:18:52.341453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.985140+0000","last_change":"2024-03-18T00:48:06.541377+0000","last_active":"2024-03-18T00:48:07.985140+0000","last_peered":"2024-03-18T00:48:07.985140+0000","last_clean":"2024-03-18T00:48:07.985140+0000","last_became_active":"2024-03-18T00:48:06.540937+0000","last_became_peered":"2024-03-18T00:48:06.540937+0000","last_unstale":"2024-03-18T00:48:07.985140+0000","last_undegraded":"2024-03-18T00:48:07.985140+0000","last_fullsized":"2024-03-18T00:48:07.985140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:51:55.268517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989866+0000","last_change":"2024-03-18T00:48:05.534372+0000","last_active":"2024-03-18T00:48:07.989866+0000","last_peered":"2024-03-18T00:48:07.989866+0000","last_clean":"2024-03-18T00:48:07.989866+0000","last_became_active":"2024-03-18T00:48:05.534034+0000","last_became_peered":"2024-03-18T00:48:05.534034+0000","last_unstale":"2024-03-18T00:48:07.989866+0000","last_undegraded":"2024-03-18T00:48:07.989866+0000","last_fullsized":"2024-03-18T00:48:07.989866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:58:55.013167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.529837+0000","last_change":"2024-03-18T00:48:06.527276+0000","last_active":"2024-03-18T00:48:06.529837+0000","last_peered":"2024-03-18T00:48:06.529837+0000","last_clean":"2024-03-18T00:48:06.529837+0000","last_became_active":"2024-03-18T00:48:06.526904+0000","last_became_peered":"2024-03-18T00:48:06.526904+0000","last_unstale":"2024-03-18T00:48:06.529837+0000","last_undegraded":"2024-03-18T00:48:06.529837+0000","last_fullsized":"2024-03-18T00:48:06.529837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:29:23.222222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989512+0000","last_change":"2024-03-18T00:48:05.528188+0000","last_active":"2024-03-18T00:48:07.989512+0000","last_peered":"2024-03-18T00:48:07.989512+0000","last_clean":"2024-03-18T00:48:07.989512+0000","last_became_active":"2024-03-18T00:48:05.527788+0000","last_became_peered":"2024-03-18T00:48:05.527788+0000","last_unstale":"2024-03-18T00:48:07.989512+0000","last_undegraded":"2024-03-18T00:48:07.989512+0000","last_fullsized":"2024-03-18T00:48:07.989512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:47.939771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536355+0000","last_change":"2024-03-18T00:48:06.546937+0000","last_active":"2024-03-18T00:48:07.536355+0000","last_peered":"2024-03-18T00:48:07.536355+0000","last_clean":"2024-03-18T00:48:07.536355+0000","last_became_active":"2024-03-18T00:48:06.546624+0000","last_became_peered":"2024-03-18T00:48:06.546624+0000","last_unstale":"2024-03-18T00:48:07.536355+0000","last_undegraded":"2024-03-18T00:48:07.536355+0000","last_fullsized":"2024-03-18T00:48:07.536355+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:52:30.830967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.990797+0000","last_change":"2024-03-18T00:48:05.534490+0000","last_active":"2024-03-18T00:48:07.990797+0000","last_peered":"2024-03-18T00:48:07.990797+0000","last_clean":"2024-03-18T00:48:07.990797+0000","last_became_active":"2024-03-18T00:48:05.534141+0000","last_became_peered":"2024-03-18T00:48:05.534141+0000","last_unstale":"2024-03-18T00:48:07.990797+0000","last_undegraded":"2024-03-18T00:48:07.990797+0000","last_fullsized":"2024-03-18T00:48:07.990797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:04:07.477814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989060+0000","last_change":"2024-03-18T00:48:06.537405+0000","last_active":"2024-03-18T00:48:07.989060+0000","last_peered":"2024-03-18T00:48:07.989060+0000","last_clean":"2024-03-18T00:48:07.989060+0000","last_became_active":"2024-03-18T00:48:06.536919+0000","last_became_peered":"2024-03-18T00:48:06.536919+0000","last_unstale":"2024-03-18T00:48:07.989060+0000","last_undegraded":"2024-03-18T00:48:07.989060+0000","last_fullsized":"2024-03-18T00:48:07.989060+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:31:59.863571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:10.005453+0000","last_change":"2024-03-18T00:48:05.528588+0000","last_active":"2024-03-18T00:48:10.005453+0000","last_peered":"2024-03-18T00:48:10.005453+0000","last_clean":"2024-03-18T00:48:10.005453+0000","last_became_active":"2024-03-18T00:48:05.528263+0000","last_became_peered":"2024-03-18T00:48:05.528263+0000","last_unstale":"2024-03-18T00:48:10.005453+0000","last_undegraded":"2024-03-18T00:48:10.005453+0000","last_fullsized":"2024-03-18T00:48:10.005453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:18:10.283278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536199+0000","last_change":"2024-03-18T00:48:06.546323+0000","last_active":"2024-03-18T00:48:07.536199+0000","last_peered":"2024-03-18T00:48:07.536199+0000","last_clean":"2024-03-18T00:48:07.536199+0000","last_became_active":"2024-03-18T00:48:06.545956+0000","last_became_peered":"2024-03-18T00:48:06.545956+0000","last_unstale":"2024-03-18T00:48:07.536199+0000","last_undegraded":"2024-03-18T00:48:07.536199+0000","last_fullsized":"2024-03-18T00:48:07.536199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:32.618724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.516831+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:48:06.516831+0000","last_peered":"2024-03-18T00:48:06.516831+0000","last_clean":"2024-03-18T00:48:06.516831+0000","last_became_active":"2024-03-18T00:48:05.525590+0000","last_became_peered":"2024-03-18T00:48:05.525590+0000","last_unstale":"2024-03-18T00:48:06.516831+0000","last_undegraded":"2024-03-18T00:48:06.516831+0000","last_fullsized":"2024-03-18T00:48:06.516831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:19.357331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535616+0000","last_change":"2024-03-18T00:48:06.538798+0000","last_active":"2024-03-18T00:48:07.535616+0000","last_peered":"2024-03-18T00:48:07.535616+0000","last_clean":"2024-03-18T00:48:07.535616+0000","last_became_active":"2024-03-18T00:48:06.538457+0000","last_became_peered":"2024-03-18T00:48:06.538457+0000","last_unstale":"2024-03-18T00:48:07.535616+0000","last_undegraded":"2024-03-18T00:48:07.535616+0000","last_fullsized":"2024-03-18T00:48:07.535616+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:10.738683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.987554+0000","last_change":"2024-03-18T00:48:05.528094+0000","last_active":"2024-03-18T00:48:07.987554+0000","last_peered":"2024-03-18T00:48:07.987554+0000","last_clean":"2024-03-18T00:48:07.987554+0000","last_became_active":"2024-03-18T00:48:05.527730+0000","last_became_peered":"2024-03-18T00:48:05.527730+0000","last_unstale":"2024-03-18T00:48:07.987554+0000","last_undegraded":"2024-03-18T00:48:07.987554+0000","last_fullsized":"2024-03-18T00:48:07.987554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:48:27.707505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984234+0000","last_change":"2024-03-18T00:48:06.548676+0000","last_active":"2024-03-18T00:48:07.984234+0000","last_peered":"2024-03-18T00:48:07.984234+0000","last_clean":"2024-03-18T00:48:07.984234+0000","last_became_active":"2024-03-18T00:48:06.547693+0000","last_became_peered":"2024-03-18T00:48:06.547693+0000","last_unstale":"2024-03-18T00:48:07.984234+0000","last_undegraded":"2024-03-18T00:48:07.984234+0000","last_fullsized":"2024-03-18T00:48:07.984234+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:13.849175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.986256+0000","last_change":"2024-03-18T00:48:05.523639+0000","last_active":"2024-03-18T00:48:07.986256+0000","last_peered":"2024-03-18T00:48:07.986256+0000","last_clean":"2024-03-18T00:48:07.986256+0000","last_became_active":"2024-03-18T00:48:05.523269+0000","last_became_peered":"2024-03-18T00:48:05.523269+0000","last_unstale":"2024-03-18T00:48:07.986256+0000","last_undegraded":"2024-03-18T00:48:07.986256+0000","last_fullsized":"2024-03-18T00:48:07.986256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:39.961247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.543930+0000","last_change":"2024-03-18T00:48:06.537786+0000","last_active":"2024-03-18T00:48:06.543930+0000","last_peered":"2024-03-18T00:48:06.543930+0000","last_clean":"2024-03-18T00:48:06.543930+0000","last_became_active":"2024-03-18T00:48:06.537488+0000","last_became_peered":"2024-03-18T00:48:06.537488+0000","last_unstale":"2024-03-18T00:48:06.543930+0000","last_undegraded":"2024-03-18T00:48:06.543930+0000","last_fullsized":"2024-03-18T00:48:06.543930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:25:50.608244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.524735+0000","last_change":"2024-03-18T00:48:05.535421+0000","last_active":"2024-03-18T00:48:06.524735+0000","last_peered":"2024-03-18T00:48:06.524735+0000","last_clean":"2024-03-18T00:48:06.524735+0000","last_became_active":"2024-03-18T00:48:05.534902+0000","last_became_peered":"2024-03-18T00:48:05.534902+0000","last_unstale":"2024-03-18T00:48:06.524735+0000","last_undegraded":"2024-03-18T00:48:06.524735+0000","last_fullsized":"2024-03-18T00:48:06.524735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:53:14.516189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.985097+0000","last_change":"2024-03-18T00:48:06.545416+0000","last_active":"2024-03-18T00:48:07.985097+0000","last_peered":"2024-03-18T00:48:07.985097+0000","last_clean":"2024-03-18T00:48:07.985097+0000","last_became_active":"2024-03-18T00:48:06.545087+0000","last_became_peered":"2024-03-18T00:48:06.545087+0000","last_unstale":"2024-03-18T00:48:07.985097+0000","last_undegraded":"2024-03-18T00:48:07.985097+0000","last_fullsized":"2024-03-18T00:48:07.985097+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:02:25.309990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984082+0000","last_change":"2024-03-18T00:48:05.533088+0000","last_active":"2024-03-18T00:48:07.984082+0000","last_peered":"2024-03-18T00:48:07.984082+0000","last_clean":"2024-03-18T00:48:07.984082+0000","last_became_active":"2024-03-18T00:48:05.532308+0000","last_became_peered":"2024-03-18T00:48:05.532308+0000","last_unstale":"2024-03-18T00:48:07.984082+0000","last_undegraded":"2024-03-18T00:48:07.984082+0000","last_fullsized":"2024-03-18T00:48:07.984082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:35.071283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546898+0000","last_change":"2024-03-18T00:48:06.545011+0000","last_active":"2024-03-18T00:48:06.546898+0000","last_peered":"2024-03-18T00:48:06.546898+0000","last_clean":"2024-03-18T00:48:06.546898+0000","last_became_active":"2024-03-18T00:48:06.544519+0000","last_became_peered":"2024-03-18T00:48:06.544519+0000","last_unstale":"2024-03-18T00:48:06.546898+0000","last_undegraded":"2024-03-18T00:48:06.546898+0000","last_fullsized":"2024-03-18T00:48:06.546898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:08:36.618883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984228+0000","last_change":"2024-03-18T00:48:05.527557+0000","last_active":"2024-03-18T00:48:07.984228+0000","last_peered":"2024-03-18T00:48:07.984228+0000","last_clean":"2024-03-18T00:48:07.984228+0000","last_became_active":"2024-03-18T00:48:05.527167+0000","last_became_peered":"2024-03-18T00:48:05.527167+0000","last_unstale":"2024-03-18T00:48:07.984228+0000","last_undegraded":"2024-03-18T00:48:07.984228+0000","last_fullsized":"2024-03-18T00:48:07.984228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:02:04.105091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989666+0000","last_change":"2024-03-18T00:48:06.543153+0000","last_active":"2024-03-18T00:48:07.989666+0000","last_peered":"2024-03-18T00:48:07.989666+0000","last_clean":"2024-03-18T00:48:07.989666+0000","last_became_active":"2024-03-18T00:48:06.541353+0000","last_became_peered":"2024-03-18T00:48:06.541353+0000","last_unstale":"2024-03-18T00:48:07.989666+0000","last_undegraded":"2024-03-18T00:48:07.989666+0000","last_fullsized":"2024-03-18T00:48:07.989666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:35.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.548724+0000","last_change":"2024-03-18T00:48:05.530560+0000","last_active":"2024-03-18T00:48:06.548724+0000","last_peered":"2024-03-18T00:48:06.548724+0000","last_clean":"2024-03-18T00:48:06.548724+0000","last_became_active":"2024-03-18T00:48:05.528858+0000","last_became_peered":"2024-03-18T00:48:05.528858+0000","last_unstale":"2024-03-18T00:48:06.548724+0000","last_undegraded":"2024-03-18T00:48:06.548724+0000","last_fullsized":"2024-03-18T00:48:06.548724+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:03:01.496919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.545375+0000","last_change":"2024-03-18T00:48:06.544748+0000","last_active":"2024-03-18T00:48:06.545375+0000","last_peered":"2024-03-18T00:48:06.545375+0000","last_clean":"2024-03-18T00:48:06.545375+0000","last_became_active":"2024-03-18T00:48:06.541951+0000","last_became_peered":"2024-03-18T00:48:06.541951+0000","last_unstale":"2024-03-18T00:48:06.545375+0000","last_undegraded":"2024-03-18T00:48:06.545375+0000","last_fullsized":"2024-03-18T00:48:06.545375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:38:38.757028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.987918+0000","last_change":"2024-03-18T00:48:05.527902+0000","last_active":"2024-03-18T00:48:07.987918+0000","last_peered":"2024-03-18T00:48:07.987918+0000","last_clean":"2024-03-18T00:48:07.987918+0000","last_became_active":"2024-03-18T00:48:05.527495+0000","last_became_peered":"2024-03-18T00:48:05.527495+0000","last_unstale":"2024-03-18T00:48:07.987918+0000","last_undegraded":"2024-03-18T00:48:07.987918+0000","last_fullsized":"2024-03-18T00:48:07.987918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:25.504652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.551415+0000","last_change":"2024-03-18T00:48:06.550413+0000","last_active":"2024-03-18T00:48:06.551415+0000","last_peered":"2024-03-18T00:48:06.551415+0000","last_clean":"2024-03-18T00:48:06.551415+0000","last_became_active":"2024-03-18T00:48:06.547727+0000","last_became_peered":"2024-03-18T00:48:06.547727+0000","last_unstale":"2024-03-18T00:48:06.551415+0000","last_undegraded":"2024-03-18T00:48:06.551415+0000","last_fullsized":"2024-03-18T00:48:06.551415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:45:03.068840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:10.004320+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:48:10.004320+0000","last_peered":"2024-03-18T00:48:10.004320+0000","last_clean":"2024-03-18T00:48:10.004320+0000","last_became_active":"2024-03-18T00:48:05.525607+0000","last_became_peered":"2024-03-18T00:48:05.525607+0000","last_unstale":"2024-03-18T00:48:10.004320+0000","last_undegraded":"2024-03-18T00:48:10.004320+0000","last_fullsized":"2024-03-18T00:48:10.004320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:32.457447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546975+0000","last_change":"2024-03-18T00:48:06.545838+0000","last_active":"2024-03-18T00:48:06.546975+0000","last_peered":"2024-03-18T00:48:06.546975+0000","last_clean":"2024-03-18T00:48:06.546975+0000","last_became_active":"2024-03-18T00:48:06.543258+0000","last_became_peered":"2024-03-18T00:48:06.543258+0000","last_unstale":"2024-03-18T00:48:06.546975+0000","last_undegraded":"2024-03-18T00:48:06.546975+0000","last_fullsized":"2024-03-18T00:48:06.546975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:17:06.569478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.991530+0000","last_change":"2024-03-18T00:48:05.536296+0000","last_active":"2024-03-18T00:48:07.991530+0000","last_peered":"2024-03-18T00:48:07.991530+0000","last_clean":"2024-03-18T00:48:07.991530+0000","last_became_active":"2024-03-18T00:48:05.535945+0000","last_became_peered":"2024-03-18T00:48:05.535945+0000","last_unstale":"2024-03-18T00:48:07.991530+0000","last_undegraded":"2024-03-18T00:48:07.991530+0000","last_fullsized":"2024-03-18T00:48:07.991530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:48:20.978213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535895+0000","last_change":"2024-03-18T00:48:06.544084+0000","last_active":"2024-03-18T00:48:07.535895+0000","last_peered":"2024-03-18T00:48:07.535895+0000","last_clean":"2024-03-18T00:48:07.535895+0000","last_became_active":"2024-03-18T00:48:06.543419+0000","last_became_peered":"2024-03-18T00:48:06.543419+0000","last_unstale":"2024-03-18T00:48:07.535895+0000","last_undegraded":"2024-03-18T00:48:07.535895+0000","last_fullsized":"2024-03-18T00:48:07.535895+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:15:28.527473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.989321+0000","last_change":"2024-03-18T00:48:05.528347+0000","last_active":"2024-03-18T00:48:07.989321+0000","last_peered":"2024-03-18T00:48:07.989321+0000","last_clean":"2024-03-18T00:48:07.989321+0000","last_became_active":"2024-03-18T00:48:05.527761+0000","last_became_peered":"2024-03-18T00:48:05.527761+0000","last_unstale":"2024-03-18T00:48:07.989321+0000","last_undegraded":"2024-03-18T00:48:07.989321+0000","last_fullsized":"2024-03-18T00:48:07.989321+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:36:58.562495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.542369+0000","last_change":"2024-03-18T00:48:06.541862+0000","last_active":"2024-03-18T00:48:06.542369+0000","last_peered":"2024-03-18T00:48:06.542369+0000","last_clean":"2024-03-18T00:48:06.542369+0000","last_became_active":"2024-03-18T00:48:06.541488+0000","last_became_peered":"2024-03-18T00:48:06.541488+0000","last_unstale":"2024-03-18T00:48:06.542369+0000","last_undegraded":"2024-03-18T00:48:06.542369+0000","last_fullsized":"2024-03-18T00:48:06.542369+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:54:42.065399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.982380+0000","last_change":"2024-03-18T00:48:05.535498+0000","last_active":"2024-03-18T00:48:07.982380+0000","last_peered":"2024-03-18T00:48:07.982380+0000","last_clean":"2024-03-18T00:48:07.982380+0000","last_became_active":"2024-03-18T00:48:05.534907+0000","last_became_peered":"2024-03-18T00:48:05.534907+0000","last_unstale":"2024-03-18T00:48:07.982380+0000","last_undegraded":"2024-03-18T00:48:07.982380+0000","last_fullsized":"2024-03-18T00:48:07.982380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:47:50.196929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.653538+0000","last_change":"2024-03-18T00:48:06.534423+0000","last_active":"2024-03-18T00:48:07.653538+0000","last_peered":"2024-03-18T00:48:07.653538+0000","last_clean":"2024-03-18T00:48:07.653538+0000","last_became_active":"2024-03-18T00:48:06.534091+0000","last_became_peered":"2024-03-18T00:48:06.534091+0000","last_unstale":"2024-03-18T00:48:07.653538+0000","last_undegraded":"2024-03-18T00:48:07.653538+0000","last_fullsized":"2024-03-18T00:48:07.653538+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:39:08.755634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.530685+0000","last_change":"2024-03-18T00:48:05.532891+0000","last_active":"2024-03-18T00:48:06.530685+0000","last_peered":"2024-03-18T00:48:06.530685+0000","last_clean":"2024-03-18T00:48:06.530685+0000","last_became_active":"2024-03-18T00:48:05.532515+0000","last_became_peered":"2024-03-18T00:48:05.532515+0000","last_unstale":"2024-03-18T00:48:06.530685+0000","last_undegraded":"2024-03-18T00:48:06.530685+0000","last_fullsized":"2024-03-18T00:48:06.530685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:46:52.056837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.653438+0000","last_change":"2024-03-18T00:48:06.542992+0000","last_active":"2024-03-18T00:48:07.653438+0000","last_peered":"2024-03-18T00:48:07.653438+0000","last_clean":"2024-03-18T00:48:07.653438+0000","last_became_active":"2024-03-18T00:48:06.542617+0000","last_became_peered":"2024-03-18T00:48:06.542617+0000","last_unstale":"2024-03-18T00:48:07.653438+0000","last_undegraded":"2024-03-18T00:48:07.653438+0000","last_fullsized":"2024-03-18T00:48:07.653438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:22.736118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536275+0000","last_change":"2024-03-18T00:48:06.536018+0000","last_active":"2024-03-18T00:48:07.536275+0000","last_peered":"2024-03-18T00:48:07.536275+0000","last_clean":"2024-03-18T00:48:07.536275+0000","last_became_active":"2024-03-18T00:48:06.535656+0000","last_became_peered":"2024-03-18T00:48:06.535656+0000","last_unstale":"2024-03-18T00:48:07.536275+0000","last_undegraded":"2024-03-18T00:48:07.536275+0000","last_fullsized":"2024-03-18T00:48:07.536275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:33:00.492887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.516602+0000","last_change":"2024-03-18T00:48:05.536635+0000","last_active":"2024-03-18T00:48:06.516602+0000","last_peered":"2024-03-18T00:48:06.516602+0000","last_clean":"2024-03-18T00:48:06.516602+0000","last_became_active":"2024-03-18T00:48:05.535941+0000","last_became_peered":"2024-03-18T00:48:05.535941+0000","last_unstale":"2024-03-18T00:48:06.516602+0000","last_undegraded":"2024-03-18T00:48:06.516602+0000","last_fullsized":"2024-03-18T00:48:06.516602+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:20:52.167217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535650+0000","last_change":"2024-03-18T00:48:06.546266+0000","last_active":"2024-03-18T00:48:07.535650+0000","last_peered":"2024-03-18T00:48:07.535650+0000","last_clean":"2024-03-18T00:48:07.535650+0000","last_became_active":"2024-03-18T00:48:06.543565+0000","last_became_peered":"2024-03-18T00:48:06.543565+0000","last_unstale":"2024-03-18T00:48:07.535650+0000","last_undegraded":"2024-03-18T00:48:07.535650+0000","last_fullsized":"2024-03-18T00:48:07.535650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:49:09.673212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:10.005272+0000","last_change":"2024-03-18T00:48:05.526184+0000","last_active":"2024-03-18T00:48:10.005272+0000","last_peered":"2024-03-18T00:48:10.005272+0000","last_clean":"2024-03-18T00:48:10.005272+0000","last_became_active":"2024-03-18T00:48:05.525011+0000","last_became_peered":"2024-03-18T00:48:05.525011+0000","last_unstale":"2024-03-18T00:48:10.005272+0000","last_undegraded":"2024-03-18T00:48:10.005272+0000","last_fullsized":"2024-03-18T00:48:10.005272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:12.690966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":338,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.526335+0000","last_change":"2024-03-18T00:46:53.288768+0000","last_active":"2024-03-18T00:48:06.526335+0000","last_peered":"2024-03-18T00:48:06.526335+0000","last_clean":"2024-03-18T00:48:06.526335+0000","last_became_active":"2024-03-18T00:46:53.288394+0000","last_became_peered":"2024-03-18T00:46:53.288394+0000","last_unstale":"2024-03-18T00:48:06.526335+0000","last_undegraded":"2024-03-18T00:48:06.526335+0000","last_fullsized":"2024-03-18T00:48:06.526335+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_clean_scrub_stamp":"2024-03-18T00:43:28.173150+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:49.949940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.534926+0000","last_change":"2024-03-18T00:48:06.534383+0000","last_active":"2024-03-18T00:48:06.534926+0000","last_peered":"2024-03-18T00:48:06.534926+0000","last_clean":"2024-03-18T00:48:06.534926+0000","last_became_active":"2024-03-18T00:48:06.534085+0000","last_became_peered":"2024-03-18T00:48:06.534085+0000","last_unstale":"2024-03-18T00:48:06.534926+0000","last_undegraded":"2024-03-18T00:48:06.534926+0000","last_fullsized":"2024-03-18T00:48:06.534926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:24:52.854412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.526580+0000","last_change":"2024-03-18T00:48:05.534298+0000","last_active":"2024-03-18T00:48:06.526580+0000","last_peered":"2024-03-18T00:48:06.526580+0000","last_clean":"2024-03-18T00:48:06.526580+0000","last_became_active":"2024-03-18T00:48:05.533970+0000","last_became_peered":"2024-03-18T00:48:05.533970+0000","last_unstale":"2024-03-18T00:48:06.526580+0000","last_undegraded":"2024-03-18T00:48:06.526580+0000","last_fullsized":"2024-03-18T00:48:06.526580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:02:13.757387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.991564+0000","last_change":"2024-03-18T00:48:05.532472+0000","last_active":"2024-03-18T00:48:07.991564+0000","last_peered":"2024-03-18T00:48:07.991564+0000","last_clean":"2024-03-18T00:48:07.991564+0000","last_became_active":"2024-03-18T00:48:05.532122+0000","last_became_peered":"2024-03-18T00:48:05.532122+0000","last_unstale":"2024-03-18T00:48:07.991564+0000","last_undegraded":"2024-03-18T00:48:07.991564+0000","last_fullsized":"2024-03-18T00:48:07.991564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:49:07.545179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.530531+0000","last_change":"2024-03-18T00:48:06.530108+0000","last_active":"2024-03-18T00:48:06.530531+0000","last_peered":"2024-03-18T00:48:06.530531+0000","last_clean":"2024-03-18T00:48:06.530531+0000","last_became_active":"2024-03-18T00:48:06.529768+0000","last_became_peered":"2024-03-18T00:48:06.529768+0000","last_unstale":"2024-03-18T00:48:06.530531+0000","last_undegraded":"2024-03-18T00:48:06.530531+0000","last_fullsized":"2024-03-18T00:48:06.530531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:17:18.062729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:08.997525+0000","last_change":"2024-03-18T00:48:05.535703+0000","last_active":"2024-03-18T00:48:08.997525+0000","last_peered":"2024-03-18T00:48:08.997525+0000","last_clean":"2024-03-18T00:48:08.997525+0000","last_became_active":"2024-03-18T00:48:05.535086+0000","last_became_peered":"2024-03-18T00:48:05.535086+0000","last_unstale":"2024-03-18T00:48:08.997525+0000","last_undegraded":"2024-03-18T00:48:08.997525+0000","last_fullsized":"2024-03-18T00:48:08.997525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:10:24.172353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535941+0000","last_change":"2024-03-18T00:48:06.541281+0000","last_active":"2024-03-18T00:48:07.535941+0000","last_peered":"2024-03-18T00:48:07.535941+0000","last_clean":"2024-03-18T00:48:07.535941+0000","last_became_active":"2024-03-18T00:48:06.540946+0000","last_became_peered":"2024-03-18T00:48:06.540946+0000","last_unstale":"2024-03-18T00:48:07.535941+0000","last_undegraded":"2024-03-18T00:48:07.535941+0000","last_fullsized":"2024-03-18T00:48:07.535941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:02.987661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:11.012766+0000","last_change":"2024-03-18T00:48:05.531685+0000","last_active":"2024-03-18T00:48:11.012766+0000","last_peered":"2024-03-18T00:48:11.012766+0000","last_clean":"2024-03-18T00:48:11.012766+0000","last_became_active":"2024-03-18T00:48:05.531239+0000","last_became_peered":"2024-03-18T00:48:05.531239+0000","last_unstale":"2024-03-18T00:48:11.012766+0000","last_undegraded":"2024-03-18T00:48:11.012766+0000","last_fullsized":"2024-03-18T00:48:11.012766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:26:35.034842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.984746+0000","last_change":"2024-03-18T00:48:06.546783+0000","last_active":"2024-03-18T00:48:07.984746+0000","last_peered":"2024-03-18T00:48:07.984746+0000","last_clean":"2024-03-18T00:48:07.984746+0000","last_became_active":"2024-03-18T00:48:06.546411+0000","last_became_peered":"2024-03-18T00:48:06.546411+0000","last_unstale":"2024-03-18T00:48:07.984746+0000","last_undegraded":"2024-03-18T00:48:07.984746+0000","last_fullsized":"2024-03-18T00:48:07.984746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:39:34.432104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.518189+0000","last_change":"2024-03-18T00:48:05.534701+0000","last_active":"2024-03-18T00:48:06.518189+0000","last_peered":"2024-03-18T00:48:06.518189+0000","last_clean":"2024-03-18T00:48:06.518189+0000","last_became_active":"2024-03-18T00:48:05.534300+0000","last_became_peered":"2024-03-18T00:48:05.534300+0000","last_unstale":"2024-03-18T00:48:06.518189+0000","last_undegraded":"2024-03-18T00:48:06.518189+0000","last_fullsized":"2024-03-18T00:48:06.518189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:10.580001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.988954+0000","last_change":"2024-03-18T00:48:06.528286+0000","last_active":"2024-03-18T00:48:07.988954+0000","last_peered":"2024-03-18T00:48:07.988954+0000","last_clean":"2024-03-18T00:48:07.988954+0000","last_became_active":"2024-03-18T00:48:06.527947+0000","last_became_peered":"2024-03-18T00:48:06.527947+0000","last_unstale":"2024-03-18T00:48:07.988954+0000","last_undegraded":"2024-03-18T00:48:07.988954+0000","last_fullsized":"2024-03-18T00:48:07.988954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:34.953923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:11.012781+0000","last_change":"2024-03-18T00:48:05.526530+0000","last_active":"2024-03-18T00:48:11.012781+0000","last_peered":"2024-03-18T00:48:11.012781+0000","last_clean":"2024-03-18T00:48:11.012781+0000","last_became_active":"2024-03-18T00:48:05.526210+0000","last_became_peered":"2024-03-18T00:48:05.526210+0000","last_unstale":"2024-03-18T00:48:11.012781+0000","last_undegraded":"2024-03-18T00:48:11.012781+0000","last_fullsized":"2024-03-18T00:48:11.012781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:11:11.094469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.537490+0000","last_change":"2024-03-18T00:48:06.535316+0000","last_active":"2024-03-18T00:48:06.537490+0000","last_peered":"2024-03-18T00:48:06.537490+0000","last_clean":"2024-03-18T00:48:06.537490+0000","last_became_active":"2024-03-18T00:48:06.534945+0000","last_became_peered":"2024-03-18T00:48:06.534945+0000","last_unstale":"2024-03-18T00:48:06.537490+0000","last_undegraded":"2024-03-18T00:48:06.537490+0000","last_fullsized":"2024-03-18T00:48:06.537490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:30:49.085979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.518194+0000","last_change":"2024-03-18T00:48:05.529300+0000","last_active":"2024-03-18T00:48:06.518194+0000","last_peered":"2024-03-18T00:48:06.518194+0000","last_clean":"2024-03-18T00:48:06.518194+0000","last_became_active":"2024-03-18T00:48:05.528935+0000","last_became_peered":"2024-03-18T00:48:05.528935+0000","last_unstale":"2024-03-18T00:48:06.518194+0000","last_undegraded":"2024-03-18T00:48:06.518194+0000","last_fullsized":"2024-03-18T00:48:06.518194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:29:45.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535866+0000","last_change":"2024-03-18T00:48:06.536426+0000","last_active":"2024-03-18T00:48:07.535866+0000","last_peered":"2024-03-18T00:48:07.535866+0000","last_clean":"2024-03-18T00:48:07.535866+0000","last_became_active":"2024-03-18T00:48:06.536005+0000","last_became_peered":"2024-03-18T00:48:06.536005+0000","last_unstale":"2024-03-18T00:48:07.535866+0000","last_undegraded":"2024-03-18T00:48:07.535866+0000","last_fullsized":"2024-03-18T00:48:07.535866+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:40:52.028794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.991868+0000","last_change":"2024-03-18T00:48:05.536495+0000","last_active":"2024-03-18T00:48:07.991868+0000","last_peered":"2024-03-18T00:48:07.991868+0000","last_clean":"2024-03-18T00:48:07.991868+0000","last_became_active":"2024-03-18T00:48:05.536072+0000","last_became_peered":"2024-03-18T00:48:05.536072+0000","last_unstale":"2024-03-18T00:48:07.991868+0000","last_undegraded":"2024-03-18T00:48:07.991868+0000","last_fullsized":"2024-03-18T00:48:07.991868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:25.528261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.553497+0000","last_change":"2024-03-18T00:48:06.552900+0000","last_active":"2024-03-18T00:48:06.553497+0000","last_peered":"2024-03-18T00:48:06.553497+0000","last_clean":"2024-03-18T00:48:06.553497+0000","last_became_active":"2024-03-18T00:48:06.552568+0000","last_became_peered":"2024-03-18T00:48:06.552568+0000","last_unstale":"2024-03-18T00:48:06.553497+0000","last_undegraded":"2024-03-18T00:48:06.553497+0000","last_fullsized":"2024-03-18T00:48:06.553497+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:15:53.518654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.518802+0000","last_change":"2024-03-18T00:48:05.537327+0000","last_active":"2024-03-18T00:48:06.518802+0000","last_peered":"2024-03-18T00:48:06.518802+0000","last_clean":"2024-03-18T00:48:06.518802+0000","last_became_active":"2024-03-18T00:48:05.536825+0000","last_became_peered":"2024-03-18T00:48:05.536825+0000","last_unstale":"2024-03-18T00:48:06.518802+0000","last_undegraded":"2024-03-18T00:48:06.518802+0000","last_fullsized":"2024-03-18T00:48:06.518802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:03.141142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546685+0000","last_change":"2024-03-18T00:48:06.542593+0000","last_active":"2024-03-18T00:48:06.546685+0000","last_peered":"2024-03-18T00:48:06.546685+0000","last_clean":"2024-03-18T00:48:06.546685+0000","last_became_active":"2024-03-18T00:48:06.541624+0000","last_became_peered":"2024-03-18T00:48:06.541624+0000","last_unstale":"2024-03-18T00:48:06.546685+0000","last_undegraded":"2024-03-18T00:48:06.546685+0000","last_fullsized":"2024-03-18T00:48:06.546685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:19:16.021707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536201+0000","last_change":"2024-03-18T00:48:05.529140+0000","last_active":"2024-03-18T00:48:07.536201+0000","last_peered":"2024-03-18T00:48:07.536201+0000","last_clean":"2024-03-18T00:48:07.536201+0000","last_became_active":"2024-03-18T00:48:05.528682+0000","last_became_peered":"2024-03-18T00:48:05.528682+0000","last_unstale":"2024-03-18T00:48:07.536201+0000","last_undegraded":"2024-03-18T00:48:07.536201+0000","last_fullsized":"2024-03-18T00:48:07.536201+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:57:52.663150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546947+0000","last_change":"2024-03-18T00:48:06.546027+0000","last_active":"2024-03-18T00:48:06.546947+0000","last_peered":"2024-03-18T00:48:06.546947+0000","last_clean":"2024-03-18T00:48:06.546947+0000","last_became_active":"2024-03-18T00:48:06.545670+0000","last_became_peered":"2024-03-18T00:48:06.545670+0000","last_unstale":"2024-03-18T00:48:06.546947+0000","last_undegraded":"2024-03-18T00:48:06.546947+0000","last_fullsized":"2024-03-18T00:48:06.546947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:43:40.816065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.551728+0000","last_change":"2024-03-18T00:48:05.526929+0000","last_active":"2024-03-18T00:48:06.551728+0000","last_peered":"2024-03-18T00:48:06.551728+0000","last_clean":"2024-03-18T00:48:06.551728+0000","last_became_active":"2024-03-18T00:48:05.526530+0000","last_became_peered":"2024-03-18T00:48:05.526530+0000","last_unstale":"2024-03-18T00:48:06.551728+0000","last_undegraded":"2024-03-18T00:48:06.551728+0000","last_fullsized":"2024-03-18T00:48:06.551728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:51:25.110895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546546+0000","last_change":"2024-03-18T00:48:06.541223+0000","last_active":"2024-03-18T00:48:06.546546+0000","last_peered":"2024-03-18T00:48:06.546546+0000","last_clean":"2024-03-18T00:48:06.546546+0000","last_became_active":"2024-03-18T00:48:06.540898+0000","last_became_peered":"2024-03-18T00:48:06.540898+0000","last_unstale":"2024-03-18T00:48:06.546546+0000","last_undegraded":"2024-03-18T00:48:06.546546+0000","last_fullsized":"2024-03-18T00:48:06.546546+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:31:23.356867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.534267+0000","last_change":"2024-03-18T00:48:06.532906+0000","last_active":"2024-03-18T00:48:06.534267+0000","last_peered":"2024-03-18T00:48:06.534267+0000","last_clean":"2024-03-18T00:48:06.534267+0000","last_became_active":"2024-03-18T00:48:06.532583+0000","last_became_peered":"2024-03-18T00:48:06.532583+0000","last_unstale":"2024-03-18T00:48:06.534267+0000","last_undegraded":"2024-03-18T00:48:06.534267+0000","last_fullsized":"2024-03-18T00:48:06.534267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:31:16.565495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.531717+0000","last_change":"2024-03-18T00:48:06.530020+0000","last_active":"2024-03-18T00:48:06.531717+0000","last_peered":"2024-03-18T00:48:06.531717+0000","last_clean":"2024-03-18T00:48:06.531717+0000","last_became_active":"2024-03-18T00:48:06.529709+0000","last_became_peered":"2024-03-18T00:48:06.529709+0000","last_unstale":"2024-03-18T00:48:06.531717+0000","last_undegraded":"2024-03-18T00:48:06.531717+0000","last_fullsized":"2024-03-18T00:48:06.531717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:26.325479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.538966+0000","last_change":"2024-03-18T00:48:06.537658+0000","last_active":"2024-03-18T00:48:06.538966+0000","last_peered":"2024-03-18T00:48:06.538966+0000","last_clean":"2024-03-18T00:48:06.538966+0000","last_became_active":"2024-03-18T00:48:06.537339+0000","last_became_peered":"2024-03-18T00:48:06.537339+0000","last_unstale":"2024-03-18T00:48:06.538966+0000","last_undegraded":"2024-03-18T00:48:06.538966+0000","last_fullsized":"2024-03-18T00:48:06.538966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:00:50.867863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.985132+0000","last_change":"2024-03-18T00:48:06.544514+0000","last_active":"2024-03-18T00:48:07.985132+0000","last_peered":"2024-03-18T00:48:07.985132+0000","last_clean":"2024-03-18T00:48:07.985132+0000","last_became_active":"2024-03-18T00:48:06.544165+0000","last_became_peered":"2024-03-18T00:48:06.544165+0000","last_unstale":"2024-03-18T00:48:07.985132+0000","last_undegraded":"2024-03-18T00:48:07.985132+0000","last_fullsized":"2024-03-18T00:48:07.985132+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:48:02.498940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.536842+0000","last_change":"2024-03-18T00:48:06.536208+0000","last_active":"2024-03-18T00:48:06.536842+0000","last_peered":"2024-03-18T00:48:06.536842+0000","last_clean":"2024-03-18T00:48:06.536842+0000","last_became_active":"2024-03-18T00:48:06.535920+0000","last_became_peered":"2024-03-18T00:48:06.535920+0000","last_unstale":"2024-03-18T00:48:06.536842+0000","last_undegraded":"2024-03-18T00:48:06.536842+0000","last_fullsized":"2024-03-18T00:48:06.536842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:28:49.720721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.536110+0000","last_change":"2024-03-18T00:48:06.533837+0000","last_active":"2024-03-18T00:48:06.536110+0000","last_peered":"2024-03-18T00:48:06.536110+0000","last_clean":"2024-03-18T00:48:06.536110+0000","last_became_active":"2024-03-18T00:48:06.533418+0000","last_became_peered":"2024-03-18T00:48:06.533418+0000","last_unstale":"2024-03-18T00:48:06.536110+0000","last_undegraded":"2024-03-18T00:48:06.536110+0000","last_fullsized":"2024-03-18T00:48:06.536110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:04:15.025748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.529224+0000","last_change":"2024-03-18T00:48:06.527394+0000","last_active":"2024-03-18T00:48:06.529224+0000","last_peered":"2024-03-18T00:48:06.529224+0000","last_clean":"2024-03-18T00:48:06.529224+0000","last_became_active":"2024-03-18T00:48:06.526879+0000","last_became_peered":"2024-03-18T00:48:06.526879+0000","last_unstale":"2024-03-18T00:48:06.529224+0000","last_undegraded":"2024-03-18T00:48:06.529224+0000","last_fullsized":"2024-03-18T00:48:06.529224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:30:30.074091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535869+0000","last_change":"2024-03-18T00:48:06.542645+0000","last_active":"2024-03-18T00:48:07.535869+0000","last_peered":"2024-03-18T00:48:07.535869+0000","last_clean":"2024-03-18T00:48:07.535869+0000","last_became_active":"2024-03-18T00:48:06.542073+0000","last_became_peered":"2024-03-18T00:48:06.542073+0000","last_unstale":"2024-03-18T00:48:07.535869+0000","last_undegraded":"2024-03-18T00:48:07.535869+0000","last_fullsized":"2024-03-18T00:48:07.535869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:08:44.479318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.536173+0000","last_change":"2024-03-18T00:48:06.541698+0000","last_active":"2024-03-18T00:48:07.536173+0000","last_peered":"2024-03-18T00:48:07.536173+0000","last_clean":"2024-03-18T00:48:07.536173+0000","last_became_active":"2024-03-18T00:48:06.541378+0000","last_became_peered":"2024-03-18T00:48:06.541378+0000","last_unstale":"2024-03-18T00:48:07.536173+0000","last_undegraded":"2024-03-18T00:48:07.536173+0000","last_fullsized":"2024-03-18T00:48:07.536173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:42:23.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-18T00:48:07.535289+0000","last_change":"2024-03-18T00:48:06.533184+0000","last_active":"2024-03-18T00:48:07.535289+0000","last_peered":"2024-03-18T00:48:07.535289+0000","last_clean":"2024-03-18T00:48:07.535289+0000","last_became_active":"2024-03-18T00:48:06.532034+0000","last_became_peered":"2024-03-18T00:48:06.532034+0000","last_unstale":"2024-03-18T00:48:07.535289+0000","last_undegraded":"2024-03-18T00:48:07.535289+0000","last_fullsized":"2024-03-18T00:48:07.535289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:45.265799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.546822+0000","last_change":"2024-03-18T00:48:06.545941+0000","last_active":"2024-03-18T00:48:06.546822+0000","last_peered":"2024-03-18T00:48:06.546822+0000","last_clean":"2024-03-18T00:48:06.546822+0000","last_became_active":"2024-03-18T00:48:06.545402+0000","last_became_peered":"2024-03-18T00:48:06.545402+0000","last_unstale":"2024-03-18T00:48:06.546822+0000","last_undegraded":"2024-03-18T00:48:06.546822+0000","last_fullsized":"2024-03-18T00:48:06.546822+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:54:13.711332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.544058+0000","last_change":"2024-03-18T00:48:06.541513+0000","last_active":"2024-03-18T00:48:06.544058+0000","last_peered":"2024-03-18T00:48:06.544058+0000","last_clean":"2024-03-18T00:48:06.544058+0000","last_became_active":"2024-03-18T00:48:06.541128+0000","last_became_peered":"2024-03-18T00:48:06.541128+0000","last_unstale":"2024-03-18T00:48:06.544058+0000","last_undegraded":"2024-03-18T00:48:06.544058+0000","last_fullsized":"2024-03-18T00:48:06.544058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:03:24.725850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-18T00:48:06.543745+0000","last_change":"2024-03-18T00:48:06.542316+0000","last_active":"2024-03-18T00:48:06.543745+0000","last_peered":"2024-03-18T00:48:06.543745+0000","last_clean":"2024-03-18T00:48:06.543745+0000","last_became_active":"2024-03-18T00:48:06.541914+0000","last_became_peered":"2024-03-18T00:48:06.541914+0000","last_unstale":"2024-03-18T00:48:06.543745+0000","last_undegraded":"2024-03-18T00:48:06.543745+0000","last_fullsized":"2024-03-18T00:48:06.543745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:16:18.226087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249372,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5647,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":119,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":13051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":897024,"data_stored":423925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":1035130,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037789,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":917504,"data_stored":433606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":1023952,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462951,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":909312,"data_stored":433002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":921600,"data_stored":434614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36240,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708912,"statfs":{"total":95995035648,"available":95957925888,"internally_reserved":0,"allocated":913408,"data_stored":433092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":897024,"data_stored":423925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":441980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":442238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":1034178,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":3314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T00:48:13.225 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T00:48:13.225 INFO:tasks.cephadm:Setting up client nodes... 2024-03-18T00:48:13.226 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T00:48:13.331 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:13 smithi107 ceph-mon[36270]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:13.331 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:13 smithi107 ceph-mon[36270]: pgmap v291: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s wr, 19 op/s 2024-03-18T00:48:13.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:13 smithi107 ceph-mon[36270]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:13.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:13 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] up:active 2024-03-18T00:48:13.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:13 smithi107 ceph-mon[36270]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-18T00:48:13.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:13 smithi107 ceph-mon[36270]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T00:48:13.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:13 smithi107 ceph-mon[36270]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:48:13.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:13 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2592549453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:48:13.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:13 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3326094334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:13 smithi106 ceph-mon[35899]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:13 smithi106 ceph-mon[35899]: pgmap v291: 97 pgs: 97 active+clean; 582 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s wr, 19 op/s 2024-03-18T00:48:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:13 smithi106 ceph-mon[35899]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:48:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:13 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] up:active 2024-03-18T00:48:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:13 smithi106 ceph-mon[35899]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-18T00:48:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:13 smithi106 ceph-mon[35899]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-18T00:48:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:13 smithi106 ceph-mon[35899]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:48:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:13 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2592549453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:48:13.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:13 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3326094334' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:13.508 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:14 smithi107 ceph-mon[36270]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:48:14.343 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:14 smithi107 ceph-mon[36270]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:48:14.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:14 smithi049 ceph-mon[32130]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:48:14.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:14 smithi049 ceph-mon[32130]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:48:14.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:14 smithi106 ceph-mon[35899]: from='client.24643 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:48:14.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:14 smithi106 ceph-mon[35899]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:48:15.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.166+0000 7f4980a74700 1 -- 172.21.15.49:0/2468323583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c102e10 msgr2=0x7f497c1096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:15.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.166+0000 7f4980a74700 1 --2- 172.21.15.49:0/2468323583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c102e10 0x7f497c1096a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f496c00b0a0 tx=0x7f496c00b3b0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.166+0000 7f4980a74700 1 -- 172.21.15.49:0/2468323583 shutdown_connections 2024-03-18T00:48:15.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.166+0000 7f4980a74700 1 --2- 172.21.15.49:0/2468323583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f497c101730 0x7f497c101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.166+0000 7f4980a74700 1 --2- 172.21.15.49:0/2468323583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c102e10 0x7f497c1096a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.166+0000 7f4980a74700 1 --2- 172.21.15.49:0/2468323583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f497c102450 0x7f497c1028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.166+0000 7f4980a74700 1 -- 172.21.15.49:0/2468323583 >> 172.21.15.49:0/2468323583 conn(0x7f497c0fbf70 msgr2=0x7f497c0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:15.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.166+0000 7f4980a74700 1 -- 172.21.15.49:0/2468323583 shutdown_connections 2024-03-18T00:48:15.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.166+0000 7f4980a74700 1 -- 172.21.15.49:0/2468323583 wait complete. 2024-03-18T00:48:15.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f4980a74700 1 Processor -- start 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f4980a74700 1 -- start start 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f4980a74700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c101730 0x7f497c198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f4980a74700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f497c102450 0x7f497c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f4980a74700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f497c102e10 0x7f497c19f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f4980a74700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f497c10ce20 con 0x7f497c101730 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f4980a74700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f497c10cca0 con 0x7f497c102e10 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f4980a74700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f497c10cfa0 con 0x7f497c102450 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f497a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c101730 0x7f497c198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f497a59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c101730 0x7f497c198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60824/0 (socket says 172.21.15.49:60824) 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f497a59c700 1 -- 172.21.15.49:0/363936770 learned_addr learned my addr 172.21.15.49:0/363936770 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f4979d9b700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f497c102450 0x7f497c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:15.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f497ad9d700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f497c102e10 0x7f497c19f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f497a59c700 1 -- 172.21.15.49:0/363936770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f497c102450 msgr2=0x7f497c198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.167+0000 7f497a59c700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f497c102450 0x7f497c198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f497a59c700 1 -- 172.21.15.49:0/363936770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f497c102e10 msgr2=0x7f497c19f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f497a59c700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f497c102e10 0x7f497c19f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f497a59c700 1 -- 172.21.15.49:0/363936770 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c1a0050 con 0x7f497c101730 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f4979d9b700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f497c102450 0x7f497c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f497ad9d700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f497c102e10 0x7f497c19f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f497a59c700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c101730 0x7f497c198440 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f497400e980 tx=0x7f497400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f49737fe700 1 -- 172.21.15.49:0/363936770 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974004230 con 0x7f497c101730 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f49737fe700 1 -- 172.21.15.49:0/363936770 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49740043d0 con 0x7f497c101730 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497c1a0340 con 0x7f497c101730 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f49737fe700 1 -- 172.21.15.49:0/363936770 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974003dc0 con 0x7f497c101730 2024-03-18T00:48:15.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.168+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497c1a0880 con 0x7f497c101730 2024-03-18T00:48:15.172 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:15 smithi049 ceph-mon[32130]: pgmap v292: 97 pgs: 97 active+clean; 585 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 7.9 KiB/s wr, 25 op/s 2024-03-18T00:48:15.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.169+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497c0507e0 con 0x7f497c101730 2024-03-18T00:48:15.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.170+0000 7f49737fe700 1 -- 172.21.15.49:0/363936770 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f497400c810 con 0x7f497c101730 2024-03-18T00:48:15.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.171+0000 7f49737fe700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49680a2a50 0x7f49680a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:15.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.171+0000 7f4979d9b700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49680a2a50 0x7f49680a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:15.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.171+0000 7f49737fe700 1 -- 172.21.15.49:0/363936770 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f49740ced50 con 0x7f497c101730 2024-03-18T00:48:15.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.171+0000 7f4979d9b700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49680a2a50 0x7f49680a4f10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f497c101460 tx=0x7f4964009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:15.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.176+0000 7f49737fe700 1 -- 172.21.15.49:0/363936770 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f497409be40 con 0x7f497c101730 2024-03-18T00:48:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:15 smithi107 ceph-mon[36270]: pgmap v292: 97 pgs: 97 active+clean; 585 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 7.9 KiB/s wr, 25 op/s 2024-03-18T00:48:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:15 smithi106 ceph-mon[35899]: pgmap v292: 97 pgs: 97 active+clean; 585 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 7.9 KiB/s wr, 25 op/s 2024-03-18T00:48:15.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.465+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f497c1a0b40 con 0x7f497c101730 2024-03-18T00:48:15.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.467+0000 7f49737fe700 1 -- 172.21.15.49:0/363936770 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f497409b740 con 0x7f497c101730 2024-03-18T00:48:15.467 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2024-03-18T00:48:15.468 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQBPj/dlEV3aGxAAbvurlRbNOmV14cRD3i6YDg== 2024-03-18T00:48:15.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49680a2a50 msgr2=0x7f49680a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:15.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49680a2a50 0x7f49680a4f10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f497c101460 tx=0x7f4964009260 comp rx=0 tx=0).stop 2024-03-18T00:48:15.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c101730 msgr2=0x7f497c198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:15.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c101730 0x7f497c198440 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f497400e980 tx=0x7f497400ec90 comp rx=0 tx=0).stop 2024-03-18T00:48:15.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 shutdown_connections 2024-03-18T00:48:15.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f497c102e10 0x7f497c19f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f497c101730 0x7f497c198440 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49680a2a50 0x7f49680a4f10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 --2- 172.21.15.49:0/363936770 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f497c102450 0x7f497c198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:15.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 >> 172.21.15.49:0/363936770 conn(0x7f497c0fbf70 msgr2=0x7f497c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:15.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 shutdown_connections 2024-03-18T00:48:15.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:15.469+0000 7f4980a74700 1 -- 172.21.15.49:0/363936770 wait complete. 2024-03-18T00:48:16.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:16 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/363936770' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T00:48:16.058 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:16 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/363936770' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T00:48:16.086 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:48:16.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-18T00:48:16.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T00:48:16.137 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T00:48:16.137 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-18T00:48:16.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-18T00:48:16.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:16 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/363936770' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T00:48:16.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:16 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/363936770' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T00:48:16.429 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:16.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:16 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/363936770' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T00:48:16.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:16 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/363936770' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T00:48:17.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:17 smithi049 ceph-mon[32130]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 12 KiB/s wr, 28 op/s 2024-03-18T00:48:17.346 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:17 smithi106 ceph-mon[35899]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 12 KiB/s wr, 28 op/s 2024-03-18T00:48:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:17 smithi107 ceph-mon[36270]: pgmap v293: 97 pgs: 97 active+clean; 626 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 12 KiB/s wr, 28 op/s 2024-03-18T00:48:18.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 -- 172.21.15.49:0/4223005517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc701027a0 msgr2=0x7fbc7010e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 --2- 172.21.15.49:0/4223005517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc701027a0 0x7fbc7010e100 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800b0a0 tx=0x7fbc6800b3b0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 -- 172.21.15.49:0/4223005517 shutdown_connections 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 --2- 172.21.15.49:0/4223005517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc701027a0 0x7fbc7010e100 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 --2- 172.21.15.49:0/4223005517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc70106260 0x7fbc70106640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 --2- 172.21.15.49:0/4223005517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc70101de0 0x7fbc70102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 -- 172.21.15.49:0/4223005517 >> 172.21.15.49:0/4223005517 conn(0x7fbc70076520 msgr2=0x7fbc70076930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 -- 172.21.15.49:0/4223005517 shutdown_connections 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 -- 172.21.15.49:0/4223005517 wait complete. 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 Processor -- start 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.053+0000 7fbc78494700 1 -- start start 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc78494700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc70101de0 0x7fbc70198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:18.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc78494700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc701027a0 0x7fbc70199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc78494700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc70106260 0x7fbc7019d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc78494700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbc70111540 con 0x7fbc70101de0 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc78494700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701113c0 con 0x7fbc70106260 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc78494700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc701116c0 con 0x7fbc701027a0 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76230700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc70101de0 0x7fbc70198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76230700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc70101de0 0x7fbc70198ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55946/0 (socket says 172.21.15.49:55946) 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76230700 1 -- 172.21.15.49:0/2083291149 learned_addr learned my addr 172.21.15.49:0/2083291149 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc75a2f700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc701027a0 0x7fbc70199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76a31700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc70106260 0x7fbc7019d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76230700 1 -- 172.21.15.49:0/2083291149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc701027a0 msgr2=0x7fbc70199000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76230700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc701027a0 0x7fbc70199000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76230700 1 -- 172.21.15.49:0/2083291149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc70106260 msgr2=0x7fbc7019d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76230700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc70106260 0x7fbc7019d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76230700 1 -- 172.21.15.49:0/2083291149 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7019dac0 con 0x7fbc70101de0 2024-03-18T00:48:18.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc76230700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc70101de0 0x7fbc70198ac0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d980 tx=0x7fbc6c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:18.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc677fe700 1 -- 172.21.15.49:0/2083291149 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c016070 con 0x7fbc70101de0 2024-03-18T00:48:18.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7019ddb0 con 0x7fbc70101de0 2024-03-18T00:48:18.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc677fe700 1 -- 172.21.15.49:0/2083291149 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc6c004d10 con 0x7fbc70101de0 2024-03-18T00:48:18.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.054+0000 7fbc677fe700 1 -- 172.21.15.49:0/2083291149 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c005670 con 0x7fbc70101de0 2024-03-18T00:48:18.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.055+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc701a5670 con 0x7fbc70101de0 2024-03-18T00:48:18.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.055+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc70068f90 con 0x7fbc70101de0 2024-03-18T00:48:18.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.057+0000 7fbc677fe700 1 -- 172.21.15.49:0/2083291149 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbc6c004750 con 0x7fbc70101de0 2024-03-18T00:48:18.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.057+0000 7fbc677fe700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbc5c0a2920 0x7fbc5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:18.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.057+0000 7fbc75a2f700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbc5c0a2920 0x7fbc5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:18.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.057+0000 7fbc677fe700 1 -- 172.21.15.49:0/2083291149 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fbc6c0ce770 con 0x7fbc70101de0 2024-03-18T00:48:18.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.058+0000 7fbc75a2f700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbc5c0a2920 0x7fbc5c0a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbc70076a20 tx=0x7fbc60009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:18.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.062+0000 7fbc677fe700 1 -- 172.21.15.49:0/2083291149 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbc6c01f020 con 0x7fbc70101de0 2024-03-18T00:48:18.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.326+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fbc701a5840 con 0x7fbc70101de0 2024-03-18T00:48:18.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.331+0000 7fbc677fe700 1 -- 172.21.15.49:0/2083291149 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262659 (secure 0 0 0) 0x7fbc6c022350 con 0x7fbc70101de0 2024-03-18T00:48:18.334 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:18.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbc5c0a2920 msgr2=0x7fbc5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:18.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbc5c0a2920 0x7fbc5c0a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbc70076a20 tx=0x7fbc60009260 comp rx=0 tx=0).stop 2024-03-18T00:48:18.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc70101de0 msgr2=0x7fbc70198ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:18.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc70101de0 0x7fbc70198ac0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00d980 tx=0x7fbc6c00dc90 comp rx=0 tx=0).stop 2024-03-18T00:48:18.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 shutdown_connections 2024-03-18T00:48:18.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbc70106260 0x7fbc7019d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbc70101de0 0x7fbc70198ac0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbc5c0a2920 0x7fbc5c0a4de0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 --2- 172.21.15.49:0/2083291149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc701027a0 0x7fbc70199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:18.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 >> 172.21.15.49:0/2083291149 conn(0x7fbc70076520 msgr2=0x7fbc70100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:18.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 shutdown_connections 2024-03-18T00:48:18.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:18.334+0000 7fbc78494700 1 -- 172.21.15.49:0/2083291149 wait complete. 2024-03-18T00:48:18.996 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":1246732627},{"type":"v1","addr":"172.21.15.49:6801","nonce":1246732627}]},"active_addr":"172.21.15.49:6801/1246732627","active_change":"2024-03-18T00:41:01.376865+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.49:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":270568828}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":900173211}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1950297592}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":3078209257}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1408743531}]}]} 2024-03-18T00:48:19.002 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-18T00:48:19.002 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-18T00:48:19.002 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T00:48:19.259 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:19.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:19 smithi049 ceph-mon[32130]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 10 KiB/s wr, 29 op/s 2024-03-18T00:48:19.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:19 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2083291149' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T00:48:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:19 smithi107 ceph-mon[36270]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 10 KiB/s wr, 29 op/s 2024-03-18T00:48:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:19 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2083291149' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T00:48:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:19 smithi106 ceph-mon[35899]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 10 KiB/s wr, 29 op/s 2024-03-18T00:48:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:19 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2083291149' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T00:48:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.025+0000 7fd14ef59700 1 -- 172.21.15.49:0/3126608605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd148107640 msgr2=0x7fd14810f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.025+0000 7fd14ef59700 1 --2- 172.21.15.49:0/3126608605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd148107640 0x7fd14810f1c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd14000b0a0 tx=0x7fd14000b3b0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 -- 172.21.15.49:0/3126608605 shutdown_connections 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 --2- 172.21.15.49:0/3126608605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd148107640 0x7fd14810f1c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 --2- 172.21.15.49:0/3126608605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd14806abd0 0x7fd14806afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 --2- 172.21.15.49:0/3126608605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd14806b580 0x7fd148106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 -- 172.21.15.49:0/3126608605 >> 172.21.15.49:0/3126608605 conn(0x7fd148078140 msgr2=0x7fd148078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 -- 172.21.15.49:0/3126608605 shutdown_connections 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 -- 172.21.15.49:0/3126608605 wait complete. 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 Processor -- start 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 -- start start 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd14806abd0 0x7fd1481a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd14806b580 0x7fd1481a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148107640 0x7fd14819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:21.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd148112630 con 0x7fd14806abd0 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd1481124b0 con 0x7fd14806b580 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.026+0000 7fd14ef59700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd1481127b0 con 0x7fd148107640 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd14ccf5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd14806abd0 0x7fd1481a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd14ccf5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd14806abd0 0x7fd1481a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55966/0 (socket says 172.21.15.49:55966) 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd14ccf5700 1 -- 172.21.15.49:0/2725378417 learned_addr learned my addr 172.21.15.49:0/2725378417 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd147fff700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd14806b580 0x7fd1481a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd14d4f6700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148107640 0x7fd14819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd147fff700 1 -- 172.21.15.49:0/2725378417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148107640 msgr2=0x7fd14819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd147fff700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148107640 0x7fd14819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd147fff700 1 -- 172.21.15.49:0/2725378417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd14806abd0 msgr2=0x7fd1481a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd147fff700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd14806abd0 0x7fd1481a0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd147fff700 1 -- 172.21.15.49:0/2725378417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd14819b400 con 0x7fd14806b580 2024-03-18T00:48:21.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd147fff700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd14806b580 0x7fd1481a0f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd148103740 tx=0x7fd13400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd145ffb700 1 -- 172.21.15.49:0/2725378417 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd134010900 con 0x7fd14806b580 2024-03-18T00:48:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd14819b6f0 con 0x7fd14806b580 2024-03-18T00:48:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd145ffb700 1 -- 172.21.15.49:0/2725378417 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd13400c430 con 0x7fd14806b580 2024-03-18T00:48:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.027+0000 7fd145ffb700 1 -- 172.21.15.49:0/2725378417 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd134018660 con 0x7fd14806b580 2024-03-18T00:48:21.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.028+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1481a79d0 con 0x7fd14806b580 2024-03-18T00:48:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.028+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd14806a220 con 0x7fd14806b580 2024-03-18T00:48:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.030+0000 7fd145ffb700 1 -- 172.21.15.49:0/2725378417 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd13400c5e0 con 0x7fd14806b580 2024-03-18T00:48:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.030+0000 7fd145ffb700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1380a2920 0x7fd1380a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.030+0000 7fd14ccf5700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1380a2920 0x7fd1380a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.030+0000 7fd145ffb700 1 -- 172.21.15.49:0/2725378417 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fd1340cf080 con 0x7fd14806b580 2024-03-18T00:48:21.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.031+0000 7fd14ccf5700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1380a2920 0x7fd1380a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009680 tx=0x7fd13c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:21.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.034+0000 7fd145ffb700 1 -- 172.21.15.49:0/2725378417 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd13409c070 con 0x7fd14806b580 2024-03-18T00:48:21.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:21 smithi049 ceph-mon[32130]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.8 KiB/s wr, 25 op/s 2024-03-18T00:48:21.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.242+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd14819bfe0 con 0x7fd14806b580 2024-03-18T00:48:21.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.244+0000 7fd145ffb700 1 -- 172.21.15.49:0/2725378417 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22206 (secure 0 0 0) 0x7fd13409b970 con 0x7fd14806b580 2024-03-18T00:48:21.244 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:21.244 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":86,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:48:07.527912+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:48:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1380a2920 msgr2=0x7fd1380a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1380a2920 0x7fd1380a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009680 tx=0x7fd13c009260 comp rx=0 tx=0).stop 2024-03-18T00:48:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd14806b580 msgr2=0x7fd1481a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:21.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd14806b580 0x7fd1481a0f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd148103740 tx=0x7fd13400eca0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 shutdown_connections 2024-03-18T00:48:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1380a2920 0x7fd1380a4de0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd148107640 0x7fd14819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd14806abd0 0x7fd1481a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 --2- 172.21.15.49:0/2725378417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd14806b580 0x7fd1481a0f90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 >> 172.21.15.49:0/2725378417 conn(0x7fd148078140 msgr2=0x7fd14810d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 shutdown_connections 2024-03-18T00:48:21.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:21.246+0000 7fd14ef59700 1 -- 172.21.15.49:0/2725378417 wait complete. 2024-03-18T00:48:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:21 smithi107 ceph-mon[36270]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.8 KiB/s wr, 25 op/s 2024-03-18T00:48:21.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:21 smithi106 ceph-mon[35899]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.8 KiB/s wr, 25 op/s 2024-03-18T00:48:21.940 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-18T00:48:21.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T00:48:22.198 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:22.320 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:22 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2725378417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:22.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:22 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2725378417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:22.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:22 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2725378417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:23.314 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:23 smithi049 ceph-mon[32130]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 25 op/s 2024-03-18T00:48:23.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:23 smithi107 ceph-mon[36270]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 25 op/s 2024-03-18T00:48:23.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:23 smithi106 ceph-mon[35899]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 25 op/s 2024-03-18T00:48:23.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.831+0000 7f642fb36700 1 -- 172.21.15.49:0/3550797559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101520 msgr2=0x7f64281019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:23.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.831+0000 7f642fb36700 1 --2- 172.21.15.49:0/3550797559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101520 0x7f64281019a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6418009990 tx=0x7f6418009ca0 comp rx=0 tx=0).stop 2024-03-18T00:48:23.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.831+0000 7f642fb36700 1 -- 172.21.15.49:0/3550797559 shutdown_connections 2024-03-18T00:48:23.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.831+0000 7f642fb36700 1 --2- 172.21.15.49:0/3550797559 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6428106160 0x7f6428106540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:23.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.831+0000 7f642fb36700 1 --2- 172.21.15.49:0/3550797559 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6428101ee0 0x7f642810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.831+0000 7f642fb36700 1 --2- 172.21.15.49:0/3550797559 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101520 0x7f64281019a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.831+0000 7f642fb36700 1 -- 172.21.15.49:0/3550797559 >> 172.21.15.49:0/3550797559 conn(0x7f6428077f80 msgr2=0x7f6428078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.831+0000 7f642fb36700 1 -- 172.21.15.49:0/3550797559 shutdown_connections 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.831+0000 7f642fb36700 1 -- 172.21.15.49:0/3550797559 wait complete. 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642fb36700 1 Processor -- start 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642fb36700 1 -- start start 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642fb36700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6428101520 0x7f6428102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642fb36700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101ee0 0x7f64281033c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642fb36700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6428106160 0x7f6428103900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642fb36700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f6428111560 con 0x7f6428101ee0 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642fb36700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f64281113e0 con 0x7f6428101520 2024-03-18T00:48:23.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642fb36700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f64281116e0 con 0x7f6428106160 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642d0d1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101ee0 0x7f64281033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642d0d1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101ee0 0x7f64281033c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55976/0 (socket says 172.21.15.49:55976) 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642d0d1700 1 -- 172.21.15.49:0/2756193628 learned_addr learned my addr 172.21.15.49:0/2756193628 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642d8d2700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6428101520 0x7f6428102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.832+0000 7f642e0d3700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6428106160 0x7f6428103900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f642d0d1700 1 -- 172.21.15.49:0/2756193628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6428106160 msgr2=0x7f6428103900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f642d0d1700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6428106160 0x7f6428103900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f642d0d1700 1 -- 172.21.15.49:0/2756193628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6428101520 msgr2=0x7f6428102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f642d0d1700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6428101520 0x7f6428102e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f642d0d1700 1 -- 172.21.15.49:0/2756193628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64281a6f80 con 0x7f6428101ee0 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f642d8d2700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6428101520 0x7f6428102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:23.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f642d0d1700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101ee0 0x7f64281033c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6418000c00 tx=0x7f641800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:23.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f641effd700 1 -- 172.21.15.49:0/2756193628 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418027070 con 0x7f6428101ee0 2024-03-18T00:48:23.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64281a7210 con 0x7f6428101ee0 2024-03-18T00:48:23.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f641effd700 1 -- 172.21.15.49:0/2756193628 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6418003d70 con 0x7f6428101ee0 2024-03-18T00:48:23.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64281a76c0 con 0x7f6428101ee0 2024-03-18T00:48:23.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.833+0000 7f641effd700 1 -- 172.21.15.49:0/2756193628 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641800f460 con 0x7f6428101ee0 2024-03-18T00:48:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.834+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6428068f90 con 0x7f6428101ee0 2024-03-18T00:48:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.835+0000 7f641effd700 1 -- 172.21.15.49:0/2756193628 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f641802a070 con 0x7f6428101ee0 2024-03-18T00:48:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.836+0000 7f641effd700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f64140a2920 0x7f64140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.836+0000 7f642d8d2700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f64140a2920 0x7f64140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.836+0000 7f641effd700 1 -- 172.21.15.49:0/2756193628 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f64180cfb40 con 0x7f6428101ee0 2024-03-18T00:48:23.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.836+0000 7f642d8d2700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f64140a2920 0x7f64140a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6424009fd0 tx=0x7f6424009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:23.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:23.839+0000 7f641effd700 1 -- 172.21.15.49:0/2756193628 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6418013070 con 0x7f6428101ee0 2024-03-18T00:48:24.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.046+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f642802db70 con 0x7f6428101ee0 2024-03-18T00:48:24.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.047+0000 7f641effd700 1 -- 172.21.15.49:0/2756193628 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22206 (secure 0 0 0) 0x7f641809ccc0 con 0x7f6428101ee0 2024-03-18T00:48:24.048 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:48:24.048 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":86,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:48:07.527912+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:48:24.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.049+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f64140a2920 msgr2=0x7f64140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.049+0000 7f642fb36700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f64140a2920 0x7f64140a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6424009fd0 tx=0x7f6424009260 comp rx=0 tx=0).stop 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.049+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101ee0 msgr2=0x7f64281033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.049+0000 7f642fb36700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101ee0 0x7f64281033c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6418000c00 tx=0x7f641800bab0 comp rx=0 tx=0).stop 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.050+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 shutdown_connections 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.050+0000 7f642fb36700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f64140a2920 0x7f64140a4de0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.050+0000 7f642fb36700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6428106160 0x7f6428103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.050+0000 7f642fb36700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6428101520 0x7f6428102e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.050+0000 7f642fb36700 1 --2- 172.21.15.49:0/2756193628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6428101ee0 0x7f64281033c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.050+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 >> 172.21.15.49:0/2756193628 conn(0x7f6428077f80 msgr2=0x7f642806a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.050+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 shutdown_connections 2024-03-18T00:48:24.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:24.050+0000 7f642fb36700 1 -- 172.21.15.49:0/2756193628 wait complete. 2024-03-18T00:48:24.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-18T00:48:24.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-18T00:48:24.711 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-18T00:48:24.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-18T00:48:24.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-18T00:48:24.712 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-18T00:48:24.713 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-18T00:48:24.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-18T00:48:24.714 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-18T00:48:24.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-18T00:48:24.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-18T00:48:24.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-18T00:48:25.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:25 smithi049 ceph-mon[32130]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 20 op/s 2024-03-18T00:48:25.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:25 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2756193628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:25.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:25 smithi107 ceph-mon[36270]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 20 op/s 2024-03-18T00:48:25.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:25 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2756193628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:25.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:25 smithi106 ceph-mon[35899]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 20 op/s 2024-03-18T00:48:25.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:25 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2756193628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:48:25.448 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:25.527 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:25.527 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:26.183 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:26.185 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:26.185 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:26.196 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:26.197 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:26.197 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:26.201 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:26.206 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:27.248 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:27 smithi049 ceph-mon[32130]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.3 KiB/s wr, 15 op/s 2024-03-18T00:48:27.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:27 smithi107 ceph-mon[36270]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.3 KiB/s wr, 15 op/s 2024-03-18T00:48:27.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:27 smithi106 ceph-mon[35899]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.3 KiB/s wr, 15 op/s 2024-03-18T00:48:29.289 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:29 smithi049 ceph-mon[32130]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-18T00:48:29.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:29 smithi107 ceph-mon[36270]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-18T00:48:29.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:29 smithi106 ceph-mon[35899]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-18T00:48:31.367 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:31 smithi049 ceph-mon[32130]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T00:48:31.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:31 smithi107 ceph-mon[36270]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T00:48:31.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:31 smithi106 ceph-mon[35899]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-03-18T00:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.036+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2942160693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba241083f0 msgr2=0x7fba241087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.036+0000 7fba2ad4e700 1 --2- 172.21.15.49:0/2942160693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba241083f0 0x7fba241087d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fba18009990 tx=0x7fba18009ca0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.036+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2942160693 shutdown_connections 2024-03-18T00:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.036+0000 7fba2ad4e700 1 --2- 172.21.15.49:0/2942160693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba24102d40 0x7fba2410f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.036+0000 7fba2ad4e700 1 --2- 172.21.15.49:0/2942160693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba241083f0 0x7fba241087d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.036+0000 7fba2ad4e700 1 --2- 172.21.15.49:0/2942160693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba24102380 0x7fba24102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.036+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2942160693 >> 172.21.15.49:0/2942160693 conn(0x7fba240793a0 msgr2=0x7fba240797b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.036+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2942160693 shutdown_connections 2024-03-18T00:48:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.036+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2942160693 wait complete. 2024-03-18T00:48:33.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.037+0000 7fba2ad4e700 1 Processor -- start 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.037+0000 7fba2ad4e700 1 -- start start 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.037+0000 7fba2ad4e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba24102380 0x7fba2419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.037+0000 7fba2ad4e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba24102d40 0x7fba2419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.037+0000 7fba2ad4e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba241083f0 0x7fba24198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.037+0000 7fba2ad4e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fba24112610 con 0x7fba24102d40 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.037+0000 7fba2ad4e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fba24112490 con 0x7fba24102380 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.037+0000 7fba2ad4e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fba24112790 con 0x7fba241083f0 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba28aea700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba24102380 0x7fba2419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba23fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba24102d40 0x7fba2419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba28aea700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba24102380 0x7fba2419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:35676/0 (socket says 172.21.15.49:35676) 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba28aea700 1 -- 172.21.15.49:0/2398763451 learned_addr learned my addr 172.21.15.49:0/2398763451 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba292eb700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba241083f0 0x7fba24198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba28aea700 1 -- 172.21.15.49:0/2398763451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba241083f0 msgr2=0x7fba24198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba28aea700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fba241083f0 0x7fba24198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba28aea700 1 -- 172.21.15.49:0/2398763451 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba24102d40 msgr2=0x7fba2419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba28aea700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba24102d40 0x7fba2419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba28aea700 1 -- 172.21.15.49:0/2398763451 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba24199030 con 0x7fba24102380 2024-03-18T00:48:33.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba23fff700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fba24102d40 0x7fba2419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:33.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.038+0000 7fba28aea700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba24102380 0x7fba2419e6d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba1800b2e0 tx=0x7fba1800b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:33.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.039+0000 7fba21ffb700 1 -- 172.21.15.49:0/2398763451 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18026070 con 0x7fba24102380 2024-03-18T00:48:33.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.039+0000 7fba21ffb700 1 -- 172.21.15.49:0/2398763451 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba18021410 con 0x7fba24102380 2024-03-18T00:48:33.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.039+0000 7fba21ffb700 1 -- 172.21.15.49:0/2398763451 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba18017650 con 0x7fba24102380 2024-03-18T00:48:33.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.039+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba241992c0 con 0x7fba24102380 2024-03-18T00:48:33.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.040+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba241a5530 con 0x7fba24102380 2024-03-18T00:48:33.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.040+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba240629e0 con 0x7fba24102380 2024-03-18T00:48:33.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.042+0000 7fba21ffb700 1 -- 172.21.15.49:0/2398763451 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fba18029070 con 0x7fba24102380 2024-03-18T00:48:33.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.042+0000 7fba21ffb700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba140a29a0 0x7fba140a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.042+0000 7fba23fff700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba140a29a0 0x7fba140a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.042+0000 7fba21ffb700 1 -- 172.21.15.49:0/2398763451 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fba180cf820 con 0x7fba24102380 2024-03-18T00:48:33.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.042+0000 7fba21ffb700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] conn(0x7fba140ac180 0x7fba140ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.042+0000 7fba21ffb700 1 -- 172.21.15.49:0/2398763451 --> [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba18001d30 con 0x7fba140ac180 2024-03-18T00:48:33.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.042+0000 7fba21ffb700 1 -- 172.21.15.49:0/2398763451 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fba180cfbd0 con 0x7fba24102380 2024-03-18T00:48:33.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.043+0000 7fba23fff700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba140a29a0 0x7fba140a4e60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fba240799a0 tx=0x7fba10009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:33.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.043+0000 7fba292eb700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] conn(0x7fba140ac180 0x7fba140ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.043+0000 7fba292eb700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] conn(0x7fba140ac180 0x7fba140ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:33.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.044+0000 7fba21ffb700 1 -- 172.21.15.49:0/2398763451 <== osd.1 v2:172.21.15.106:6800/3847617930 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba18001d30 con 0x7fba140ac180 2024-03-18T00:48:33.044 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:33 smithi049 ceph-mon[32130]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T00:48:33.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.061+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 --> [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fba24062de0 con 0x7fba140ac180 2024-03-18T00:48:33.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.061+0000 7fba21ffb700 1 -- 172.21.15.49:0/2398763451 <== osd.1 v2:172.21.15.106:6800/3847617930 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fba24062de0 con 0x7fba140ac180 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 >> [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] conn(0x7fba140ac180 msgr2=0x7fba140ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] conn(0x7fba140ac180 0x7fba140ae5a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba140a29a0 msgr2=0x7fba140a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fba140a29a0 0x7fba140a4e60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fba240799a0 tx=0x7fba10009260 comp rx=0 tx=0).stop 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba24102380 msgr2=0x7fba2419e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 --2- 172.21.15.49:0/2398763451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fba24102380 0x7fba2419e6d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fba1800b2e0 tx=0x7fba1800b610 comp rx=0 tx=0).stop 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba292eb700 1 -- 172.21.15.49:0/2398763451 reap_dead start 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 shutdown_connections 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 >> 172.21.15.49:0/2398763451 conn(0x7fba240793a0 msgr2=0x7fba24100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:33.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 shutdown_connections 2024-03-18T00:48:33.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.062+0000 7fba2ad4e700 1 -- 172.21.15.49:0/2398763451 wait complete. 2024-03-18T00:48:33.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:33 smithi107 ceph-mon[36270]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T00:48:33.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:33 smithi106 ceph-mon[35899]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.547+0000 7f0cac994700 1 -- 172.21.15.49:0/1764655689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ca81076a0 msgr2=0x7f0ca810f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.547+0000 7f0cac994700 1 --2- 172.21.15.49:0/1764655689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ca81076a0 0x7f0ca810f2b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0c980099c0 tx=0x7f0c98009cd0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.547+0000 7f0cac994700 1 -- 172.21.15.49:0/1764655689 shutdown_connections 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.547+0000 7f0cac994700 1 --2- 172.21.15.49:0/1764655689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca8100ed0 0x7f0ca81012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.547+0000 7f0cac994700 1 --2- 172.21.15.49:0/1764655689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ca81076a0 0x7f0ca810f2b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.547+0000 7f0cac994700 1 --2- 172.21.15.49:0/1764655689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ca8101880 0x7f0ca8106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.547+0000 7f0cac994700 1 -- 172.21.15.49:0/1764655689 >> 172.21.15.49:0/1764655689 conn(0x7f0ca8076550 msgr2=0x7f0ca8076960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.547+0000 7f0cac994700 1 -- 172.21.15.49:0/1764655689 shutdown_connections 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.547+0000 7f0cac994700 1 -- 172.21.15.49:0/1764655689 wait complete. 2024-03-18T00:48:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0cac994700 1 Processor -- start 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0cac994700 1 -- start start 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0cac994700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca8100ed0 0x7f0ca819e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0cac994700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ca8101880 0x7f0ca819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0cac994700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ca81076a0 0x7f0ca8198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0cac994700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca81126e0 con 0x7f0ca81076a0 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0cac994700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca8112560 con 0x7f0ca8101880 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0cac994700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca8112860 con 0x7f0ca8100ed0 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca8100ed0 0x7f0ca819e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca6d9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ca81076a0 0x7f0ca8198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca659c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca8100ed0 0x7f0ca819e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:41844/0 (socket says 172.21.15.49:41844) 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca659c700 1 -- 172.21.15.49:0/1920043939 learned_addr learned my addr 172.21.15.49:0/1920043939 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca5d9b700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ca8101880 0x7f0ca819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca6d9d700 1 -- 172.21.15.49:0/1920043939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca8100ed0 msgr2=0x7f0ca819e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca6d9d700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca8100ed0 0x7f0ca819e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca6d9d700 1 -- 172.21.15.49:0/1920043939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ca8101880 msgr2=0x7f0ca819ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca6d9d700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ca8101880 0x7f0ca819ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca6d9d700 1 -- 172.21.15.49:0/1920043939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca8199080 con 0x7f0ca81076a0 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca5d9b700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0ca8101880 0x7f0ca819ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.548+0000 7f0ca659c700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0ca8100ed0 0x7f0ca819e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.549+0000 7f0ca6d9d700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ca81076a0 0x7f0ca8198920 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0c980098e0 tx=0x7f0c9800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.549+0000 7f0c9f7fe700 1 -- 172.21.15.49:0/1920043939 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98027070 con 0x7f0ca81076a0 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.549+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca8199310 con 0x7f0ca81076a0 2024-03-18T00:48:33.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.549+0000 7f0c9f7fe700 1 -- 172.21.15.49:0/1920043939 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c98003680 con 0x7f0ca81076a0 2024-03-18T00:48:33.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.549+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca81a55e0 con 0x7f0ca81076a0 2024-03-18T00:48:33.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.549+0000 7f0c9f7fe700 1 -- 172.21.15.49:0/1920043939 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c98022e20 con 0x7f0ca81076a0 2024-03-18T00:48:33.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.550+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ca80629e0 con 0x7f0ca81076a0 2024-03-18T00:48:33.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.551+0000 7f0c9f7fe700 1 -- 172.21.15.49:0/1920043939 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0c9802a070 con 0x7f0ca81076a0 2024-03-18T00:48:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.551+0000 7f0c9f7fe700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0c940a2920 0x7f0c940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.552+0000 7f0ca659c700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0c940a2920 0x7f0c940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.552+0000 7f0c9f7fe700 1 -- 172.21.15.49:0/1920043939 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f0c980cfb50 con 0x7f0ca81076a0 2024-03-18T00:48:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.552+0000 7f0c9f7fe700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] conn(0x7f0c940ac100 0x7f0c940ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.552+0000 7f0c9f7fe700 1 -- 172.21.15.49:0/1920043939 --> [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c940aebd0 con 0x7f0c940ac100 2024-03-18T00:48:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.552+0000 7f0c9f7fe700 1 -- 172.21.15.49:0/1920043939 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f0c980d0030 con 0x7f0ca81076a0 2024-03-18T00:48:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.552+0000 7f0ca5d9b700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] conn(0x7f0c940ac100 0x7f0c940ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.552+0000 7f0ca659c700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0c940a2920 0x7f0c940a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009c10 tx=0x7f0ca000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.552+0000 7f0ca5d9b700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] conn(0x7f0c940ac100 0x7f0c940ae520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:33.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.553+0000 7f0c9f7fe700 1 -- 172.21.15.49:0/1920043939 <== osd.10 v2:172.21.15.106:6824/3868441447 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c940aebd0 con 0x7f0c940ac100 2024-03-18T00:48:33.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.569+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 --> [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ca8062de0 con 0x7f0c940ac100 2024-03-18T00:48:33.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0c9f7fe700 1 -- 172.21.15.49:0/1920043939 <== osd.10 v2:172.21.15.106:6824/3868441447 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0ca8062de0 con 0x7f0c940ac100 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 >> [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] conn(0x7f0c940ac100 msgr2=0x7f0c940ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] conn(0x7f0c940ac100 0x7f0c940ae520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0c940a2920 msgr2=0x7f0c940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0c940a2920 0x7f0c940a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009c10 tx=0x7f0ca000b040 comp rx=0 tx=0).stop 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ca81076a0 msgr2=0x7f0ca8198920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 --2- 172.21.15.49:0/1920043939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0ca81076a0 0x7f0ca8198920 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0c980098e0 tx=0x7f0c9800f690 comp rx=0 tx=0).stop 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0ca6d9d700 1 -- 172.21.15.49:0/1920043939 reap_dead start 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 shutdown_connections 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 >> 172.21.15.49:0/1920043939 conn(0x7f0ca8076550 msgr2=0x7f0ca80fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 shutdown_connections 2024-03-18T00:48:33.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:33.570+0000 7f0cac994700 1 -- 172.21.15.49:0/1920043939 wait complete. 2024-03-18T00:48:35.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:35 smithi049 ceph-mon[32130]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T00:48:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:35 smithi107 ceph-mon[36270]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T00:48:35.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:35 smithi106 ceph-mon[35899]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T00:48:35.677 INFO:teuthology.orchestra.run.smithi049.stdout:55834574918 2024-03-18T00:48:35.677 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-18T00:48:37.126 INFO:teuthology.orchestra.run.smithi049.stdout:313532612632 2024-03-18T00:48:37.126 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-18T00:48:37.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.489+0000 7f6487850700 1 -- 172.21.15.49:0/1044242684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6480102350 msgr2=0x7f64801027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:37.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.489+0000 7f6487850700 1 --2- 172.21.15.49:0/1044242684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6480102350 0x7f64801027d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f647c009990 tx=0x7f647c009ca0 comp rx=0 tx=0).stop 2024-03-18T00:48:37.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.489+0000 7f6487850700 1 -- 172.21.15.49:0/1044242684 shutdown_connections 2024-03-18T00:48:37.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.489+0000 7f6487850700 1 --2- 172.21.15.49:0/1044242684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6480102d10 0x7f648010f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.489+0000 7f6487850700 1 --2- 172.21.15.49:0/1044242684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f64801083a0 0x7f6480108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.489+0000 7f6487850700 1 --2- 172.21.15.49:0/1044242684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6480102350 0x7f64801027d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.489+0000 7f6487850700 1 -- 172.21.15.49:0/1044242684 >> 172.21.15.49:0/1044242684 conn(0x7f6480079390 msgr2=0x7f64800797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.489+0000 7f6487850700 1 -- 172.21.15.49:0/1044242684 shutdown_connections 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.489+0000 7f6487850700 1 -- 172.21.15.49:0/1044242684 wait complete. 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6487850700 1 Processor -- start 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6487850700 1 -- start start 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6487850700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6480102350 0x7f648019e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6487850700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6480102d10 0x7f648019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6487850700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64801083a0 0x7f6480198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:37.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6487850700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f64801125a0 con 0x7f6480102d10 2024-03-18T00:48:37.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6487850700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6480112420 con 0x7f6480102350 2024-03-18T00:48:37.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6487850700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6480112720 con 0x7f64801083a0 2024-03-18T00:48:37.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6484deb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6480102d10 0x7f648019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:37.319 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:37 smithi049 ceph-mon[32130]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f64855ec700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6480102350 0x7f648019e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6484deb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6480102d10 0x7f648019ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48070/0 (socket says 172.21.15.49:48070) 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6484deb700 1 -- 172.21.15.49:0/391734572 learned_addr learned my addr 172.21.15.49:0/391734572 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6485ded700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64801083a0 0x7f6480198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6484deb700 1 -- 172.21.15.49:0/391734572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64801083a0 msgr2=0x7f6480198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6484deb700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64801083a0 0x7f6480198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6484deb700 1 -- 172.21.15.49:0/391734572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6480102350 msgr2=0x7f648019e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6484deb700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6480102350 0x7f648019e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f6484deb700 1 -- 172.21.15.49:0/391734572 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6480199020 con 0x7f6480102d10 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.490+0000 7f64855ec700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6480102350 0x7f648019e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.491+0000 7f6484deb700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6480102d10 0x7f648019ec00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f647c000c00 tx=0x7f647c00bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:37.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.491+0000 7f64767fc700 1 -- 172.21.15.49:0/391734572 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c027070 con 0x7f6480102d10 2024-03-18T00:48:37.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.491+0000 7f64767fc700 1 -- 172.21.15.49:0/391734572 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f647c003d70 con 0x7f6480102d10 2024-03-18T00:48:37.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.491+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64801992b0 con 0x7f6480102d10 2024-03-18T00:48:37.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.491+0000 7f64767fc700 1 -- 172.21.15.49:0/391734572 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647c00f460 con 0x7f6480102d10 2024-03-18T00:48:37.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.491+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64801a5520 con 0x7f6480102d10 2024-03-18T00:48:37.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.492+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6480062a60 con 0x7f6480102d10 2024-03-18T00:48:37.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.493+0000 7f64767fc700 1 -- 172.21.15.49:0/391734572 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f647c02a070 con 0x7f6480102d10 2024-03-18T00:48:37.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.494+0000 7f64767fc700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f646c0a29a0 0x7f646c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:37.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.494+0000 7f64855ec700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f646c0a29a0 0x7f646c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:37.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.494+0000 7f64767fc700 1 -- 172.21.15.49:0/391734572 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f647c0cfdd0 con 0x7f6480102d10 2024-03-18T00:48:37.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.494+0000 7f64767fc700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] conn(0x7f646c0ac180 0x7f646c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:37.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.494+0000 7f64767fc700 1 -- 172.21.15.49:0/391734572 --> [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f647c0056d0 con 0x7f646c0ac180 2024-03-18T00:48:37.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.494+0000 7f64767fc700 1 -- 172.21.15.49:0/391734572 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f647c0d02b0 con 0x7f6480102d10 2024-03-18T00:48:37.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.494+0000 7f64855ec700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f646c0a29a0 0x7f646c0a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fd0 tx=0x7f6470009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:37.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.494+0000 7f6485ded700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] conn(0x7f646c0ac180 0x7f646c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:37.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.494+0000 7f6485ded700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] conn(0x7f646c0ac180 0x7f646c0ae5a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.495+0000 7f64767fc700 1 -- 172.21.15.49:0/391734572 <== osd.6 v2:172.21.15.49:6818/24691072 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f647c0056d0 con 0x7f646c0ac180 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.511+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 --> [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6480062e60 con 0x7f646c0ac180 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f64767fc700 1 -- 172.21.15.49:0/391734572 <== osd.6 v2:172.21.15.49:6818/24691072 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6480062e60 con 0x7f646c0ac180 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] conn(0x7f646c0ac180 msgr2=0x7f646c0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] conn(0x7f646c0ac180 0x7f646c0ae5a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f646c0a29a0 msgr2=0x7f646c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f646c0a29a0 0x7f646c0a4e60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fd0 tx=0x7f6470009260 comp rx=0 tx=0).stop 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6480102d10 msgr2=0x7f648019ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 --2- 172.21.15.49:0/391734572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6480102d10 0x7f648019ec00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f647c000c00 tx=0x7f647c00bab0 comp rx=0 tx=0).stop 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6485ded700 1 -- 172.21.15.49:0/391734572 reap_dead start 2024-03-18T00:48:37.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 shutdown_connections 2024-03-18T00:48:37.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 >> 172.21.15.49:0/391734572 conn(0x7f6480079390 msgr2=0x7f6480101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:37.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 shutdown_connections 2024-03-18T00:48:37.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:36.512+0000 7f6487850700 1 -- 172.21.15.49:0/391734572 wait complete. 2024-03-18T00:48:37.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:37 smithi107 ceph-mon[36270]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:48:37.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:37 smithi106 ceph-mon[35899]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:48:38.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.653+0000 7fe7c500e700 1 -- 172.21.15.49:0/4145394631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c01064e0 msgr2=0x7fe7c010e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:38.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.653+0000 7fe7c500e700 1 --2- 172.21.15.49:0/4145394631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c01064e0 0x7fe7c010e0b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800b0d0 tx=0x7fe7b800b3e0 comp rx=0 tx=0).stop 2024-03-18T00:48:38.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.653+0000 7fe7c500e700 1 -- 172.21.15.49:0/4145394631 shutdown_connections 2024-03-18T00:48:38.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.653+0000 7fe7c500e700 1 --2- 172.21.15.49:0/4145394631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c01064e0 0x7fe7c010e0b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:38.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.653+0000 7fe7c500e700 1 --2- 172.21.15.49:0/4145394631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7c0101490 0x7fe7c0101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:38.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.653+0000 7fe7c500e700 1 --2- 172.21.15.49:0/4145394631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c0101e40 0x7fe7c0105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:38.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.653+0000 7fe7c500e700 1 -- 172.21.15.49:0/4145394631 >> 172.21.15.49:0/4145394631 conn(0x7fe7c0077220 msgr2=0x7fe7c0077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:38.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.653+0000 7fe7c500e700 1 -- 172.21.15.49:0/4145394631 shutdown_connections 2024-03-18T00:48:38.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.653+0000 7fe7c500e700 1 -- 172.21.15.49:0/4145394631 wait complete. 2024-03-18T00:48:38.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7c500e700 1 Processor -- start 2024-03-18T00:48:38.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7c500e700 1 -- start start 2024-03-18T00:48:38.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7c500e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7c0101490 0x7fe7c019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:38.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7c500e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c0101e40 0x7fe7c019db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:38.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7c500e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c01064e0 0x7fe7c0197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:38.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7c500e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c01114f0 con 0x7fe7c0101e40 2024-03-18T00:48:38.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7c500e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c0111370 con 0x7fe7c0101490 2024-03-18T00:48:38.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7c500e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe7c0111670 con 0x7fe7c01064e0 2024-03-18T00:48:38.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7be59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c0101e40 0x7fe7c019db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:38.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7be59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c0101e40 0x7fe7c019db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40486/0 (socket says 172.21.15.49:40486) 2024-03-18T00:48:38.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.654+0000 7fe7be59c700 1 -- 172.21.15.49:0/712647790 learned_addr learned my addr 172.21.15.49:0/712647790 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:38.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7bf59e700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c01064e0 0x7fe7c0197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:38.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7bed9d700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7c0101490 0x7fe7c019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:38.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7bf59e700 1 -- 172.21.15.49:0/712647790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7c0101490 msgr2=0x7fe7c019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:38.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7bf59e700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe7c0101490 0x7fe7c019d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:38.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7bf59e700 1 -- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c0101e40 msgr2=0x7fe7c019db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:38.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7bf59e700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c0101e40 0x7fe7c019db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:38.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7bf59e700 1 -- 172.21.15.49:0/712647790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c0197ff0 con 0x7fe7c01064e0 2024-03-18T00:48:38.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7be59c700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe7c0101e40 0x7fe7c019db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7bf59e700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c01064e0 0x7fe7c0197710 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800b770 tx=0x7fe7b8007a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7a7fff700 1 -- 172.21.15.49:0/712647790 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b8021020 con 0x7fe7c01064e0 2024-03-18T00:48:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7c0198280 con 0x7fe7c01064e0 2024-03-18T00:48:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7a7fff700 1 -- 172.21.15.49:0/712647790 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7b8004c50 con 0x7fe7c01064e0 2024-03-18T00:48:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7c006aba0 con 0x7fe7c01064e0 2024-03-18T00:48:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.655+0000 7fe7a7fff700 1 -- 172.21.15.49:0/712647790 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80052b0 con 0x7fe7c01064e0 2024-03-18T00:48:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.656+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe7c006b210 con 0x7fe7c01064e0 2024-03-18T00:48:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.658+0000 7fe7a7fff700 1 -- 172.21.15.49:0/712647790 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe7b8024020 con 0x7fe7c01064e0 2024-03-18T00:48:38.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.658+0000 7fe7a7fff700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2920 0x7fe7a80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:38.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.658+0000 7fe7bed9d700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2920 0x7fe7a80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:38.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.658+0000 7fe7a7fff700 1 -- 172.21.15.49:0/712647790 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fe7b800f020 con 0x7fe7c01064e0 2024-03-18T00:48:38.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.658+0000 7fe7a7fff700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] conn(0x7fe7a80abfd0 0x7fe7a80ae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:38.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.658+0000 7fe7a7fff700 1 -- 172.21.15.49:0/712647790 --> [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7a80aeaa0 con 0x7fe7a80abfd0 2024-03-18T00:48:38.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.658+0000 7fe7a7fff700 1 -- 172.21.15.49:0/712647790 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fe7b80d0560 con 0x7fe7c01064e0 2024-03-18T00:48:38.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.659+0000 7fe7be59c700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] conn(0x7fe7a80abfd0 0x7fe7a80ae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:38.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.659+0000 7fe7bed9d700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2920 0x7fe7a80a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009c60 tx=0x7fe7b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:38.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.659+0000 7fe7be59c700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] conn(0x7fe7a80abfd0 0x7fe7a80ae3f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:38.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.660+0000 7fe7a7fff700 1 -- 172.21.15.49:0/712647790 <== osd.9 v2:172.21.15.49:6826/1286831410 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe7a80aeaa0 con 0x7fe7a80abfd0 2024-03-18T00:48:38.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 --> [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe7c006b610 con 0x7fe7a80abfd0 2024-03-18T00:48:38.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7a7fff700 1 -- 172.21.15.49:0/712647790 <== osd.9 v2:172.21.15.49:6826/1286831410 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe7c006b610 con 0x7fe7a80abfd0 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] conn(0x7fe7a80abfd0 msgr2=0x7fe7a80ae3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7c500e700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] conn(0x7fe7a80abfd0 0x7fe7a80ae3f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2920 msgr2=0x7fe7a80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7c500e700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe7a80a2920 0x7fe7a80a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009c60 tx=0x7fe7b0009260 comp rx=0 tx=0).stop 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c01064e0 msgr2=0x7fe7c0197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7c500e700 1 --2- 172.21.15.49:0/712647790 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe7c01064e0 0x7fe7c0197710 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe7b800b770 tx=0x7fe7b8007a20 comp rx=0 tx=0).stop 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7bf59e700 1 -- 172.21.15.49:0/712647790 reap_dead start 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 shutdown_connections 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.676+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 >> 172.21.15.49:0/712647790 conn(0x7fe7c0077220 msgr2=0x7fe7c00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.677+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 shutdown_connections 2024-03-18T00:48:38.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:38.677+0000 7fe7c500e700 1 -- 172.21.15.49:0/712647790 wait complete. 2024-03-18T00:48:39.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:39 smithi049 ceph-mon[32130]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:48:39.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:39 smithi107 ceph-mon[36270]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:48:39.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:39 smithi106 ceph-mon[35899]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:48:41.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 -- 172.21.15.49:0/1782778136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa384102f60 msgr2=0x7fa384106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 --2- 172.21.15.49:0/1782778136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa384102f60 0x7fa384106f30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa3700099c0 tx=0x7fa370009cd0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 -- 172.21.15.49:0/1782778136 shutdown_connections 2024-03-18T00:48:41.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 --2- 172.21.15.49:0/1782778136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3841025b0 0x7fa384102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 --2- 172.21.15.49:0/1782778136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa384107660 0x7fa38410f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 --2- 172.21.15.49:0/1782778136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa384102f60 0x7fa384106f30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 -- 172.21.15.49:0/1782778136 >> 172.21.15.49:0/1782778136 conn(0x7fa3840793f0 msgr2=0x7fa384079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 -- 172.21.15.49:0/1782778136 shutdown_connections 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 -- 172.21.15.49:0/1782778136 wait complete. 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 Processor -- start 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.011+0000 7fa38ad72700 1 -- start start 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa38ad72700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3841025b0 0x7fa38419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa38ad72700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa384102f60 0x7fa38419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa38ad72700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa384107660 0x7fa384198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa38ad72700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa3841125d0 con 0x7fa384102f60 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa38ad72700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa384112450 con 0x7fa3841025b0 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa38ad72700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa384112750 con 0x7fa384107660 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa383fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa384102f60 0x7fa38419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa388b0e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3841025b0 0x7fa38419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa383fff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa384102f60 0x7fa38419ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40506/0 (socket says 172.21.15.49:40506) 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa383fff700 1 -- 172.21.15.49:0/2740331424 learned_addr learned my addr 172.21.15.49:0/2740331424 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa38930f700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa384107660 0x7fa384198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa383fff700 1 -- 172.21.15.49:0/2740331424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa384107660 msgr2=0x7fa384198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa383fff700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa384107660 0x7fa384198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa383fff700 1 -- 172.21.15.49:0/2740331424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3841025b0 msgr2=0x7fa38419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa383fff700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3841025b0 0x7fa38419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa383fff700 1 -- 172.21.15.49:0/2740331424 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa384199070 con 0x7fa384102f60 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa38930f700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa384107660 0x7fa384198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa388b0e700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa3841025b0 0x7fa38419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa383fff700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa384102f60 0x7fa38419ec40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa3700098e0 tx=0x7fa370005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.012+0000 7fa381ffb700 1 -- 172.21.15.49:0/2740331424 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370027070 con 0x7fa384102f60 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.013+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa384199300 con 0x7fa384102f60 2024-03-18T00:48:41.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.013+0000 7fa381ffb700 1 -- 172.21.15.49:0/2740331424 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa370003680 con 0x7fa384102f60 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.013+0000 7fa381ffb700 1 -- 172.21.15.49:0/2740331424 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa370022e40 con 0x7fa384102f60 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.013+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3841a55d0 con 0x7fa384102f60 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.014+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa384062a60 con 0x7fa384102f60 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.015+0000 7fa381ffb700 1 -- 172.21.15.49:0/2740331424 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa37002a070 con 0x7fa384102f60 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.015+0000 7fa381ffb700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3740a2920 0x7fa3740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.015+0000 7fa388b0e700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3740a2920 0x7fa3740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.016+0000 7fa381ffb700 1 -- 172.21.15.49:0/2740331424 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa3700cfc50 con 0x7fa384102f60 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.016+0000 7fa381ffb700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] conn(0x7fa3740ac100 0x7fa3740ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.016+0000 7fa381ffb700 1 -- 172.21.15.49:0/2740331424 --> [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3740aebd0 con 0x7fa3740ac100 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.016+0000 7fa381ffb700 1 -- 172.21.15.49:0/2740331424 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fa3700d0130 con 0x7fa384102f60 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.016+0000 7fa388b0e700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3740a2920 0x7fa3740a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa384107100 tx=0x7fa378009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.016+0000 7fa38930f700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] conn(0x7fa3740ac100 0x7fa3740ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.016+0000 7fa38930f700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] conn(0x7fa3740ac100 0x7fa3740ae520 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:41.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.017+0000 7fa381ffb700 1 -- 172.21.15.49:0/2740331424 <== osd.8 v2:172.21.15.107:6816/3032692345 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3740aebd0 con 0x7fa3740ac100 2024-03-18T00:48:41.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.033+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 --> [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa384062ee0 con 0x7fa3740ac100 2024-03-18T00:48:41.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa381ffb700 1 -- 172.21.15.49:0/2740331424 <== osd.8 v2:172.21.15.107:6816/3032692345 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa384062ee0 con 0x7fa3740ac100 2024-03-18T00:48:41.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 >> [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] conn(0x7fa3740ac100 msgr2=0x7fa3740ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa38ad72700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] conn(0x7fa3740ac100 0x7fa3740ae520 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3740a2920 msgr2=0x7fa3740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa38ad72700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3740a2920 0x7fa3740a4de0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa384107100 tx=0x7fa378009260 comp rx=0 tx=0).stop 2024-03-18T00:48:41.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa384102f60 msgr2=0x7fa38419ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa38ad72700 1 --2- 172.21.15.49:0/2740331424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa384102f60 0x7fa38419ec40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa3700098e0 tx=0x7fa370005f10 comp rx=0 tx=0).stop 2024-03-18T00:48:41.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa38930f700 1 -- 172.21.15.49:0/2740331424 reap_dead start 2024-03-18T00:48:41.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 shutdown_connections 2024-03-18T00:48:41.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.034+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 >> 172.21.15.49:0/2740331424 conn(0x7fa3840793f0 msgr2=0x7fa384108790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:41.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.035+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 shutdown_connections 2024-03-18T00:48:41.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.035+0000 7fa38ad72700 1 -- 172.21.15.49:0/2740331424 wait complete. 2024-03-18T00:48:41.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:41 smithi049 ceph-mon[32130]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T00:48:41.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.230+0000 7f2afabea700 1 -- 172.21.15.49:0/2732129364 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af4107690 msgr2=0x7f2af410f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.230+0000 7f2afabea700 1 --2- 172.21.15.49:0/2732129364 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af4107690 0x7f2af410f210 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00b0d0 tx=0x7f2aec00b3e0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.230+0000 7f2afabea700 1 -- 172.21.15.49:0/2732129364 shutdown_connections 2024-03-18T00:48:41.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.230+0000 7f2afabea700 1 --2- 172.21.15.49:0/2732129364 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af4107690 0x7f2af410f210 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.230+0000 7f2afabea700 1 --2- 172.21.15.49:0/2732129364 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2af4100e70 0x7f2af4101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.230+0000 7f2afabea700 1 --2- 172.21.15.49:0/2732129364 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af4101820 0x7f2af4106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.230+0000 7f2afabea700 1 -- 172.21.15.49:0/2732129364 >> 172.21.15.49:0/2732129364 conn(0x7f2af4076510 msgr2=0x7f2af4076920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.230+0000 7f2afabea700 1 -- 172.21.15.49:0/2732129364 shutdown_connections 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.230+0000 7f2afabea700 1 -- 172.21.15.49:0/2732129364 wait complete. 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2afabea700 1 Processor -- start 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2afabea700 1 -- start start 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2afabea700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af4100e70 0x7f2af4101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2afabea700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2af4101820 0x7f2af4102330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2afabea700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af4107690 0x7f2af4102870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2afabea700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2af4112650 con 0x7f2af4100e70 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2afabea700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2af41124d0 con 0x7f2af4101820 2024-03-18T00:48:41.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2afabea700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2af41127d0 con 0x7f2af4107690 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2af8986700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af4100e70 0x7f2af4101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2af8986700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af4100e70 0x7f2af4101df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40518/0 (socket says 172.21.15.49:40518) 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2af8986700 1 -- 172.21.15.49:0/208123924 learned_addr learned my addr 172.21.15.49:0/208123924 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2af9187700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af4107690 0x7f2af4102870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.231+0000 7f2af3fff700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2af4101820 0x7f2af4102330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2af8986700 1 -- 172.21.15.49:0/208123924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af4107690 msgr2=0x7f2af4102870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2af8986700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2af4107690 0x7f2af4102870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2af8986700 1 -- 172.21.15.49:0/208123924 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2af4101820 msgr2=0x7f2af4102330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2af8986700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2af4101820 0x7f2af4102330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2af8986700 1 -- 172.21.15.49:0/208123924 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af41a6fa0 con 0x7f2af4100e70 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2af8986700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af4100e70 0x7f2af4101df0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e9a0 tx=0x7f2ae800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:41.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2af1ffb700 1 -- 172.21.15.49:0/208123924 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae800ca10 con 0x7f2af4100e70 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2af1ffb700 1 -- 172.21.15.49:0/208123924 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ae800cbb0 con 0x7f2af4100e70 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af41a7290 con 0x7f2af4100e70 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2af1ffb700 1 -- 172.21.15.49:0/208123924 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8018680 con 0x7f2af4100e70 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.232+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af41a77d0 con 0x7f2af4100e70 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.233+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2af40629e0 con 0x7f2af4100e70 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.234+0000 7f2af1ffb700 1 -- 172.21.15.49:0/208123924 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2ae8010430 con 0x7f2af4100e70 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.235+0000 7f2af1ffb700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ae40a2920 0x7f2ae40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.235+0000 7f2af1ffb700 1 -- 172.21.15.49:0/208123924 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f2ae80cf490 con 0x7f2af4100e70 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.235+0000 7f2af3fff700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ae40a2920 0x7f2ae40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.235+0000 7f2af1ffb700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] conn(0x7f2ae40ac100 0x7f2ae40ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.235+0000 7f2af1ffb700 1 -- 172.21.15.49:0/208123924 --> [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2ae801bec0 con 0x7f2ae40ac100 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.235+0000 7f2af1ffb700 1 -- 172.21.15.49:0/208123924 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f2ae80cf970 con 0x7f2af4100e70 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.235+0000 7f2af9187700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] conn(0x7f2ae40ac100 0x7f2ae40ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:41.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.235+0000 7f2af3fff700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ae40a2920 0x7f2ae40a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2af4108ac0 tx=0x7f2ae0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:41.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.236+0000 7f2af9187700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] conn(0x7f2ae40ac100 0x7f2ae40ae520 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:41.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.237+0000 7f2af1ffb700 1 -- 172.21.15.49:0/208123924 <== osd.4 v2:172.21.15.106:6808/2755514783 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2ae801bec0 con 0x7f2ae40ac100 2024-03-18T00:48:41.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.253+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 --> [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2af4062de0 con 0x7f2ae40ac100 2024-03-18T00:48:41.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2af1ffb700 1 -- 172.21.15.49:0/208123924 <== osd.4 v2:172.21.15.106:6808/2755514783 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2af4062de0 con 0x7f2ae40ac100 2024-03-18T00:48:41.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 >> [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] conn(0x7f2ae40ac100 msgr2=0x7f2ae40ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2afabea700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] conn(0x7f2ae40ac100 0x7f2ae40ae520 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ae40a2920 msgr2=0x7f2ae40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2afabea700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ae40a2920 0x7f2ae40a4de0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2af4108ac0 tx=0x7f2ae0009260 comp rx=0 tx=0).stop 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af4100e70 msgr2=0x7f2af4101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2afabea700 1 --2- 172.21.15.49:0/208123924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2af4100e70 0x7f2af4101df0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e9a0 tx=0x7f2ae800ecb0 comp rx=0 tx=0).stop 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2af9187700 1 -- 172.21.15.49:0/208123924 reap_dead start 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 shutdown_connections 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 >> 172.21.15.49:0/208123924 conn(0x7f2af4076510 msgr2=0x7f2af40fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.254+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 shutdown_connections 2024-03-18T00:48:41.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.255+0000 7f2afabea700 1 -- 172.21.15.49:0/208123924 wait complete. 2024-03-18T00:48:41.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:41 smithi107 ceph-mon[36270]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T00:48:41.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:41 smithi106 ceph-mon[35899]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T00:48:42.024 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:42.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.408+0000 7f21a9808700 1 -- 172.21.15.49:0/1102307231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21a4107660 msgr2=0x7f21a410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.408+0000 7f21a9808700 1 --2- 172.21.15.49:0/1102307231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21a4107660 0x7f21a410f1e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f219c0099c0 tx=0x7f219c009cd0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.408+0000 7f21a9808700 1 -- 172.21.15.49:0/1102307231 shutdown_connections 2024-03-18T00:48:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.408+0000 7f21a9808700 1 --2- 172.21.15.49:0/1102307231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21a4107660 0x7f21a410f1e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.408+0000 7f21a9808700 1 --2- 172.21.15.49:0/1102307231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21a41025b0 0x7f21a4102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.408+0000 7f21a9808700 1 --2- 172.21.15.49:0/1102307231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a4102f60 0x7f21a4106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.408+0000 7f21a9808700 1 -- 172.21.15.49:0/1102307231 >> 172.21.15.49:0/1102307231 conn(0x7f21a40793f0 msgr2=0x7f21a4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.408+0000 7f21a9808700 1 -- 172.21.15.49:0/1102307231 shutdown_connections 2024-03-18T00:48:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.408+0000 7f21a9808700 1 -- 172.21.15.49:0/1102307231 wait complete. 2024-03-18T00:48:42.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a9808700 1 Processor -- start 2024-03-18T00:48:42.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a9808700 1 -- start start 2024-03-18T00:48:42.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a9808700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a41025b0 0x7f21a419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a9808700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21a4102f60 0x7f21a419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a9808700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21a4107660 0x7f21a4198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a9808700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f21a4112620 con 0x7f21a4102f60 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a9808700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41124a0 con 0x7f21a4107660 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a9808700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f21a41127a0 con 0x7f21a41025b0 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a27fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21a4102f60 0x7f21a419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a2ffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a41025b0 0x7f21a419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a27fc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21a4102f60 0x7f21a419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40552/0 (socket says 172.21.15.49:40552) 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.409+0000 7f21a27fc700 1 -- 172.21.15.49:0/2369941669 learned_addr learned my addr 172.21.15.49:0/2369941669 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a37fe700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21a4107660 0x7f21a4198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a2ffd700 1 -- 172.21.15.49:0/2369941669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21a4107660 msgr2=0x7f21a4198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a2ffd700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21a4107660 0x7f21a4198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a2ffd700 1 -- 172.21.15.49:0/2369941669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21a4102f60 msgr2=0x7f21a419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a2ffd700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21a4102f60 0x7f21a419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a2ffd700 1 -- 172.21.15.49:0/2369941669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a4198fc0 con 0x7f21a41025b0 2024-03-18T00:48:42.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a37fe700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21a4107660 0x7f21a4198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:42.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a27fc700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21a4102f60 0x7f21a419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:42.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a2ffd700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a41025b0 0x7f21a419e6a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f219400ef10 tx=0x7f219400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:42.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a8806700 1 -- 172.21.15.49:0/2369941669 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219400c9c0 con 0x7f21a41025b0 2024-03-18T00:48:42.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a4199250 con 0x7f21a41025b0 2024-03-18T00:48:42.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a8806700 1 -- 172.21.15.49:0/2369941669 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2194019070 con 0x7f21a41025b0 2024-03-18T00:48:42.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a8806700 1 -- 172.21.15.49:0/2369941669 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21940146e0 con 0x7f21a41025b0 2024-03-18T00:48:42.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.410+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a41a5570 con 0x7f21a41025b0 2024-03-18T00:48:42.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.411+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f21a4062a60 con 0x7f21a41025b0 2024-03-18T00:48:42.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.413+0000 7f21a8806700 1 -- 172.21.15.49:0/2369941669 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2194012050 con 0x7f21a41025b0 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.413+0000 7f21a8806700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21900a2920 0x7f21900a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.413+0000 7f21a27fc700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21900a2920 0x7f21900a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.414+0000 7f21a8806700 1 -- 172.21.15.49:0/2369941669 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f21940cf4a0 con 0x7f21a41025b0 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.414+0000 7f21a8806700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] conn(0x7f21900ac100 0x7f21900ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.414+0000 7f21a8806700 1 -- 172.21.15.49:0/2369941669 --> [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f219400be50 con 0x7f21900ac100 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.414+0000 7f21a8806700 1 -- 172.21.15.49:0/2369941669 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f21940cf980 con 0x7f21a41025b0 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.414+0000 7f21a27fc700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21900a2920 0x7f21900a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f21a4108a60 tx=0x7f218c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.414+0000 7f21a37fe700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] conn(0x7f21900ac100 0x7f21900ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.414+0000 7f21a37fe700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] conn(0x7f21900ac100 0x7f21900ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:42.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.415+0000 7f21a8806700 1 -- 172.21.15.49:0/2369941669 <== osd.7 v2:172.21.15.106:6816/4213628543 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f219400be50 con 0x7f21900ac100 2024-03-18T00:48:42.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.431+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 --> [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f21a4062e60 con 0x7f21900ac100 2024-03-18T00:48:42.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a8806700 1 -- 172.21.15.49:0/2369941669 <== osd.7 v2:172.21.15.106:6816/4213628543 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f21a4062e60 con 0x7f21900ac100 2024-03-18T00:48:42.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 >> [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] conn(0x7f21900ac100 msgr2=0x7f21900ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] conn(0x7f21900ac100 0x7f21900ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21900a2920 msgr2=0x7f21900a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21900a2920 0x7f21900a4de0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f21a4108a60 tx=0x7f218c009260 comp rx=0 tx=0).stop 2024-03-18T00:48:42.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a41025b0 msgr2=0x7f21a419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 --2- 172.21.15.49:0/2369941669 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21a41025b0 0x7f21a419e6a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f219400ef10 tx=0x7f219400c3a0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a37fe700 1 -- 172.21.15.49:0/2369941669 reap_dead start 2024-03-18T00:48:42.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 shutdown_connections 2024-03-18T00:48:42.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 >> 172.21.15.49:0/2369941669 conn(0x7f21a40793f0 msgr2=0x7f21a41003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:42.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 shutdown_connections 2024-03-18T00:48:42.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:42.432+0000 7f21a9808700 1 -- 172.21.15.49:0/2369941669 wait complete. 2024-03-18T00:48:42.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 -- 172.21.15.49:0/2838738958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff120107210 msgr2=0x7ff1201075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 --2- 172.21.15.49:0/2838738958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff120107210 0x7ff1201075f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff1100098f0 tx=0x7ff110009c00 comp rx=0 tx=0).stop 2024-03-18T00:48:42.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 -- 172.21.15.49:0/2838738958 shutdown_connections 2024-03-18T00:48:42.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 --2- 172.21.15.49:0/2838738958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff120107210 0x7ff1201075f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 --2- 172.21.15.49:0/2838738958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff120101b70 0x7ff12010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 --2- 172.21.15.49:0/2838738958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1201011d0 0x7ff120101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 -- 172.21.15.49:0/2838738958 >> 172.21.15.49:0/2838738958 conn(0x7ff120078100 msgr2=0x7ff120078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 -- 172.21.15.49:0/2838738958 shutdown_connections 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 -- 172.21.15.49:0/2838738958 wait complete. 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.466+0000 7ff126267700 1 Processor -- start 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff126267700 1 -- start start 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff126267700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1201011d0 0x7ff120199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff126267700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff120101b70 0x7ff120199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff126267700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff120107210 0x7ff1201a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff126267700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff120111470 con 0x7ff1201011d0 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff126267700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff1201112f0 con 0x7ff120101b70 2024-03-18T00:48:42.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff126267700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1201115f0 con 0x7ff120107210 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff124804700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff120107210 0x7ff1201a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff11ffff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1201011d0 0x7ff120199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff124804700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff120107210 0x7ff1201a49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38968/0 (socket says 172.21.15.49:38968) 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff124804700 1 -- 172.21.15.49:0/1546976216 learned_addr learned my addr 172.21.15.49:0/1546976216 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff11f7fe700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff120101b70 0x7ff120199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff124804700 1 -- 172.21.15.49:0/1546976216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff120101b70 msgr2=0x7ff120199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff124804700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff120101b70 0x7ff120199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff124804700 1 -- 172.21.15.49:0/1546976216 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1201011d0 msgr2=0x7ff120199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff124804700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1201011d0 0x7ff120199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff124804700 1 -- 172.21.15.49:0/1546976216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1201a4f30 con 0x7ff120107210 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.467+0000 7ff11ffff700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1201011d0 0x7ff120199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.468+0000 7ff124804700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff120107210 0x7ff1201a49f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff118007ee0 tx=0x7ff11800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.468+0000 7ff11d7fa700 1 -- 172.21.15.49:0/1546976216 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118009e20 con 0x7ff120107210 2024-03-18T00:48:42.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.468+0000 7ff11d7fa700 1 -- 172.21.15.49:0/1546976216 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff11800dbd0 con 0x7ff120107210 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.468+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1201977c0 con 0x7ff120107210 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.468+0000 7ff11d7fa700 1 -- 172.21.15.49:0/1546976216 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118005990 con 0x7ff120107210 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.468+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff120197d00 con 0x7ff120107210 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.469+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff1200629a0 con 0x7ff120107210 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.470+0000 7ff11d7fa700 1 -- 172.21.15.49:0/1546976216 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff11801a020 con 0x7ff120107210 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.470+0000 7ff11d7fa700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1080a29a0 0x7ff1080a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.471+0000 7ff11ffff700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1080a29a0 0x7ff1080a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.471+0000 7ff11d7fa700 1 -- 172.21.15.49:0/1546976216 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff1180ce810 con 0x7ff120107210 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.471+0000 7ff11d7fa700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] conn(0x7ff1080ac180 0x7ff1080ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.471+0000 7ff11d7fa700 1 -- 172.21.15.49:0/1546976216 --> [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff11800be50 con 0x7ff1080ac180 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.471+0000 7ff11d7fa700 1 -- 172.21.15.49:0/1546976216 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7ff1180cec00 con 0x7ff120107210 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.471+0000 7ff11f7fe700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] conn(0x7ff1080ac180 0x7ff1080ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:42.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.471+0000 7ff11ffff700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1080a29a0 0x7ff1080a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff1100098c0 tx=0x7ff11001d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:42.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.471+0000 7ff11f7fe700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] conn(0x7ff1080ac180 0x7ff1080ae5a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:42.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.472+0000 7ff11d7fa700 1 -- 172.21.15.49:0/1546976216 <== osd.2 v2:172.21.15.107:6800/4213760131 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff11800be50 con 0x7ff1080ac180 2024-03-18T00:48:42.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.488+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 --> [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff120062da0 con 0x7ff1080ac180 2024-03-18T00:48:42.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.488+0000 7ff11d7fa700 1 -- 172.21.15.49:0/1546976216 <== osd.2 v2:172.21.15.107:6800/4213760131 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff120062da0 con 0x7ff1080ac180 2024-03-18T00:48:42.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 >> [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] conn(0x7ff1080ac180 msgr2=0x7ff1080ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] conn(0x7ff1080ac180 0x7ff1080ae5a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1080a29a0 msgr2=0x7ff1080a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1080a29a0 0x7ff1080a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff1100098c0 tx=0x7ff11001d040 comp rx=0 tx=0).stop 2024-03-18T00:48:42.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff120107210 msgr2=0x7ff1201a49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:42.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 --2- 172.21.15.49:0/1546976216 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff120107210 0x7ff1201a49f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff118007ee0 tx=0x7ff11800d3b0 comp rx=0 tx=0).stop 2024-03-18T00:48:42.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff124804700 1 -- 172.21.15.49:0/1546976216 reap_dead start 2024-03-18T00:48:42.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 shutdown_connections 2024-03-18T00:48:42.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 >> 172.21.15.49:0/1546976216 conn(0x7ff120078100 msgr2=0x7ff1200ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:42.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 shutdown_connections 2024-03-18T00:48:42.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:41.489+0000 7ff126267700 1 -- 172.21.15.49:0/1546976216 wait complete. 2024-03-18T00:48:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:43 smithi049 ceph-mon[32130]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-18T00:48:43.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:43 smithi107 ceph-mon[36270]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-18T00:48:43.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:43 smithi106 ceph-mon[35899]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-18T00:48:43.823 INFO:teuthology.orchestra.run.smithi049.stdout:201863462957 2024-03-18T00:48:43.824 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-18T00:48:43.879 INFO:teuthology.orchestra.run.smithi049.stdout:257698037796 2024-03-18T00:48:43.880 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-18T00:48:44.461 INFO:teuthology.orchestra.run.smithi049.stdout:146028888119 2024-03-18T00:48:44.461 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-18T00:48:45.371 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:45 smithi049 ceph-mon[32130]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T00:48:45.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:45 smithi107 ceph-mon[36270]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T00:48:45.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:45 smithi106 ceph-mon[35899]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T00:48:45.767 INFO:teuthology.orchestra.run.smithi049.stdout:223338299433 2024-03-18T00:48:45.768 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-18T00:48:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.716+0000 7f186ff9d700 1 -- 172.21.15.49:0/1151290713 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868106160 msgr2=0x7f1868106540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.716+0000 7f186ff9d700 1 --2- 172.21.15.49:0/1151290713 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868106160 0x7f1868106540 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1864009990 tx=0x7f1864009ca0 comp rx=0 tx=0).stop 2024-03-18T00:48:46.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.716+0000 7f186ff9d700 1 -- 172.21.15.49:0/1151290713 shutdown_connections 2024-03-18T00:48:46.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.716+0000 7f186ff9d700 1 --2- 172.21.15.49:0/1151290713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1868101ee0 0x7f186810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:46.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.716+0000 7f186ff9d700 1 --2- 172.21.15.49:0/1151290713 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868106160 0x7f1868106540 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:46.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.716+0000 7f186ff9d700 1 --2- 172.21.15.49:0/1151290713 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1868101520 0x7f18681019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:46.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.716+0000 7f186ff9d700 1 -- 172.21.15.49:0/1151290713 >> 172.21.15.49:0/1151290713 conn(0x7f1868077f80 msgr2=0x7f1868078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:46.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.716+0000 7f186ff9d700 1 -- 172.21.15.49:0/1151290713 shutdown_connections 2024-03-18T00:48:46.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.716+0000 7f186ff9d700 1 -- 172.21.15.49:0/1151290713 wait complete. 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186ff9d700 1 Processor -- start 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186ff9d700 1 -- start start 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186ff9d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1868101520 0x7f1868102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186ff9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1868101ee0 0x7f18681033c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186ff9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868106160 0x7f1868103900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186ff9d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f18681115b0 con 0x7f1868106160 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186ff9d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1868111430 con 0x7f1868101520 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186ff9d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1868111730 con 0x7f1868101ee0 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186e53a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868106160 0x7f1868103900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186e53a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868106160 0x7f1868103900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40576/0 (socket says 172.21.15.49:40576) 2024-03-18T00:48:46.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186e53a700 1 -- 172.21.15.49:0/1778957081 learned_addr learned my addr 172.21.15.49:0/1778957081 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186d538700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1868101ee0 0x7f18681033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.717+0000 7f186dd39700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1868101520 0x7f1868102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f186d538700 1 -- 172.21.15.49:0/1778957081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1868101520 msgr2=0x7f1868102e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f186d538700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1868101520 0x7f1868102e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f186d538700 1 -- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868106160 msgr2=0x7f1868103900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f186d538700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868106160 0x7f1868103900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f186d538700 1 -- 172.21.15.49:0/1778957081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18681a7060 con 0x7f1868101ee0 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f186e53a700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1868106160 0x7f1868103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f186d538700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1868101ee0 0x7f18681033c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f185c00e9f0 tx=0x7f185c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f185affd700 1 -- 172.21.15.49:0/1778957081 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c00c9c0 con 0x7f1868101ee0 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18681a7350 con 0x7f1868101ee0 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f185affd700 1 -- 172.21.15.49:0/1778957081 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f185c00cb60 con 0x7f1868101ee0 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f185affd700 1 -- 172.21.15.49:0/1778957081 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c018680 con 0x7f1868101ee0 2024-03-18T00:48:46.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.718+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18681a78e0 con 0x7f1868101ee0 2024-03-18T00:48:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.719+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f18680629e0 con 0x7f1868101ee0 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.721+0000 7f185affd700 1 -- 172.21.15.49:0/1778957081 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f185c010430 con 0x7f1868101ee0 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.721+0000 7f185affd700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a29a0 0x7f18540a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.721+0000 7f186dd39700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a29a0 0x7f18540a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.721+0000 7f185affd700 1 -- 172.21.15.49:0/1778957081 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f185c0cf100 con 0x7f1868101ee0 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.722+0000 7f185affd700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] conn(0x7f18540ac180 0x7f18540ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.722+0000 7f185affd700 1 -- 172.21.15.49:0/1778957081 --> [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f185c01bec0 con 0x7f18540ac180 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.722+0000 7f185affd700 1 -- 172.21.15.49:0/1778957081 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f185c0cf4f0 con 0x7f1868101ee0 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.722+0000 7f186e53a700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] conn(0x7f18540ac180 0x7f18540ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.722+0000 7f186dd39700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a29a0 0x7f18540a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f186400b2e0 tx=0x7f1864005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.722+0000 7f186e53a700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] conn(0x7f18540ac180 0x7f18540ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:46.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.723+0000 7f185affd700 1 -- 172.21.15.49:0/1778957081 <== osd.0 v2:172.21.15.49:6802/2159617645 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f185c01bec0 con 0x7f18540ac180 2024-03-18T00:48:46.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.740+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 --> [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1868062de0 con 0x7f18540ac180 2024-03-18T00:48:46.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f185affd700 1 -- 172.21.15.49:0/1778957081 <== osd.0 v2:172.21.15.49:6802/2159617645 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1868062de0 con 0x7f18540ac180 2024-03-18T00:48:46.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] conn(0x7f18540ac180 msgr2=0x7f18540ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:46.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] conn(0x7f18540ac180 0x7f18540ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:46.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a29a0 msgr2=0x7f18540a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:46.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18540a29a0 0x7f18540a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f186400b2e0 tx=0x7f1864005fd0 comp rx=0 tx=0).stop 2024-03-18T00:48:46.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1868101ee0 msgr2=0x7f18681033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:46.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 --2- 172.21.15.49:0/1778957081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1868101ee0 0x7f18681033c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f185c00e9f0 tx=0x7f185c00ed00 comp rx=0 tx=0).stop 2024-03-18T00:48:46.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186e53a700 1 -- 172.21.15.49:0/1778957081 reap_dead start 2024-03-18T00:48:46.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 shutdown_connections 2024-03-18T00:48:46.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 >> 172.21.15.49:0/1778957081 conn(0x7f1868077f80 msgr2=0x7f186806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:46.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 shutdown_connections 2024-03-18T00:48:46.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:46.741+0000 7f186ff9d700 1 -- 172.21.15.49:0/1778957081 wait complete. 2024-03-18T00:48:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:47 smithi049 ceph-mon[32130]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T00:48:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:47 smithi107 ceph-mon[36270]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T00:48:47.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:47 smithi106 ceph-mon[35899]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T00:48:48.111 INFO:teuthology.orchestra.run.smithi049.stdout:292057776157 2024-03-18T00:48:48.111 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-18T00:48:48.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.588+0000 7f477fc2f700 1 -- 172.21.15.49:0/2287206042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4778100990 msgr2=0x7f4778100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:48.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.588+0000 7f477fc2f700 1 --2- 172.21.15.49:0/2287206042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4778100990 0x7f4778100e10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f477400aaa0 tx=0x7f477400adb0 comp rx=0 tx=0).stop 2024-03-18T00:48:48.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.588+0000 7f477fc2f700 1 -- 172.21.15.49:0/2287206042 shutdown_connections 2024-03-18T00:48:48.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.588+0000 7f477fc2f700 1 --2- 172.21.15.49:0/2287206042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4778101350 0x7f477810d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:48.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.588+0000 7f477fc2f700 1 --2- 172.21.15.49:0/2287206042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4778106990 0x7f4778106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:48.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.588+0000 7f477fc2f700 1 --2- 172.21.15.49:0/2287206042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4778100990 0x7f4778100e10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:48.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.588+0000 7f477fc2f700 1 -- 172.21.15.49:0/2287206042 >> 172.21.15.49:0/2287206042 conn(0x7f47780777d0 msgr2=0x7f47780febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:48.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.588+0000 7f477fc2f700 1 -- 172.21.15.49:0/2287206042 shutdown_connections 2024-03-18T00:48:48.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.588+0000 7f477fc2f700 1 -- 172.21.15.49:0/2287206042 wait complete. 2024-03-18T00:48:48.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.589+0000 7f477fc2f700 1 Processor -- start 2024-03-18T00:48:48.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.589+0000 7f477fc2f700 1 -- start start 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.589+0000 7f477fc2f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4778100990 0x7f477819e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.589+0000 7f477fc2f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4778101350 0x7f477819ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477fc2f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4778106990 0x7f4778198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477fc2f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f47781104f0 con 0x7f4778101350 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477fc2f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4778110370 con 0x7f4778106990 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477fc2f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4778110670 con 0x7f4778100990 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d1ca700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4778101350 0x7f477819ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d9cb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4778100990 0x7f477819e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d9cb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4778100990 0x7f477819e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:39036/0 (socket says 172.21.15.49:39036) 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d9cb700 1 -- 172.21.15.49:0/5582277 learned_addr learned my addr 172.21.15.49:0/5582277 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d1ca700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4778101350 0x7f477819ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59232/0 (socket says 172.21.15.49:59232) 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477e1cc700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4778106990 0x7f4778198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:48.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d1ca700 1 -- 172.21.15.49:0/5582277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4778100990 msgr2=0x7f477819e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d1ca700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4778100990 0x7f477819e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d1ca700 1 -- 172.21.15.49:0/5582277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4778106990 msgr2=0x7f4778198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d1ca700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4778106990 0x7f4778198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d1ca700 1 -- 172.21.15.49:0/5582277 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47781990a0 con 0x7f4778101350 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d9cb700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4778100990 0x7f477819e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477d1ca700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4778101350 0x7f477819ec70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4774000c00 tx=0x7f4774004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f476effd700 1 -- 172.21.15.49:0/5582277 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4774023020 con 0x7f4778101350 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4778199330 con 0x7f4778101350 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f476effd700 1 -- 172.21.15.49:0/5582277 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4774003b60 con 0x7f4778101350 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f476effd700 1 -- 172.21.15.49:0/5582277 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f477400ed50 con 0x7f4778101350 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.590+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47781a5600 con 0x7f4778101350 2024-03-18T00:48:48.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.591+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f47780629e0 con 0x7f4778101350 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.593+0000 7f476effd700 1 -- 172.21.15.49:0/5582277 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4774026020 con 0x7f4778101350 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.593+0000 7f476effd700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f47640a29a0 0x7f47640a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.594+0000 7f477d9cb700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f47640a29a0 0x7f47640a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.594+0000 7f476effd700 1 -- 172.21.15.49:0/5582277 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f47740cfe60 con 0x7f4778101350 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.594+0000 7f476effd700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] conn(0x7f47640ac180 0x7f47640ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.594+0000 7f476effd700 1 -- 172.21.15.49:0/5582277 --> [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4774004510 con 0x7f47640ac180 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.594+0000 7f476effd700 1 -- 172.21.15.49:0/5582277 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f47740d2480 con 0x7f4778101350 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.594+0000 7f477d9cb700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f47640a29a0 0x7f47640a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f47781051e0 tx=0x7f4768009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.594+0000 7f477e1cc700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] conn(0x7f47640ac180 0x7f47640ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.594+0000 7f477e1cc700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] conn(0x7f47640ac180 0x7f47640ae5a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:48.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.595+0000 7f476effd700 1 -- 172.21.15.49:0/5582277 <== osd.5 v2:172.21.15.107:6808/391404983 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4774004510 con 0x7f47640ac180 2024-03-18T00:48:48.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.614+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 --> [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4778062de0 con 0x7f47640ac180 2024-03-18T00:48:48.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.614+0000 7f476effd700 1 -- 172.21.15.49:0/5582277 <== osd.5 v2:172.21.15.107:6808/391404983 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4778062de0 con 0x7f47640ac180 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 >> [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] conn(0x7f47640ac180 msgr2=0x7f47640ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] conn(0x7f47640ac180 0x7f47640ae5a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f47640a29a0 msgr2=0x7f47640a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f47640a29a0 0x7f47640a4e60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f47781051e0 tx=0x7f4768009260 comp rx=0 tx=0).stop 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4778101350 msgr2=0x7f477819ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 --2- 172.21.15.49:0/5582277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4778101350 0x7f477819ec70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4774000c00 tx=0x7f4774004080 comp rx=0 tx=0).stop 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477e1cc700 1 -- 172.21.15.49:0/5582277 reap_dead start 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 shutdown_connections 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 >> 172.21.15.49:0/5582277 conn(0x7f47780777d0 msgr2=0x7f4778103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 shutdown_connections 2024-03-18T00:48:48.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:48.615+0000 7f477fc2f700 1 -- 172.21.15.49:0/5582277 wait complete. 2024-03-18T00:48:49.192 INFO:teuthology.orchestra.run.smithi049.stdout:34359738444 2024-03-18T00:48:49.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-18T00:48:49.315 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:49 smithi049 ceph-mon[32130]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T00:48:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:49 smithi107 ceph-mon[36270]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T00:48:49.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:49 smithi106 ceph-mon[35899]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T00:48:50.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.111+0000 7fb612f3d700 1 -- 172.21.15.49:0/1658754622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb60c101b70 msgr2=0x7fb60c10e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.111+0000 7fb612f3d700 1 --2- 172.21.15.49:0/1658754622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb60c101b70 0x7fb60c10e050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb60400b0a0 tx=0x7fb60400b3b0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.111+0000 7fb612f3d700 1 -- 172.21.15.49:0/1658754622 shutdown_connections 2024-03-18T00:48:50.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.111+0000 7fb612f3d700 1 --2- 172.21.15.49:0/1658754622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb60c101b70 0x7fb60c10e050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.111+0000 7fb612f3d700 1 --2- 172.21.15.49:0/1658754622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c107210 0x7fb60c1075f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.111+0000 7fb612f3d700 1 --2- 172.21.15.49:0/1658754622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c1011d0 0x7fb60c101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.111+0000 7fb612f3d700 1 -- 172.21.15.49:0/1658754622 >> 172.21.15.49:0/1658754622 conn(0x7fb60c078100 msgr2=0x7fb60c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.111+0000 7fb612f3d700 1 -- 172.21.15.49:0/1658754622 shutdown_connections 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.111+0000 7fb612f3d700 1 -- 172.21.15.49:0/1658754622 wait complete. 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.112+0000 7fb612f3d700 1 Processor -- start 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.112+0000 7fb612f3d700 1 -- start start 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.112+0000 7fb612f3d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c1011d0 0x7fb60c19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.112+0000 7fb612f3d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb60c101b70 0x7fb60c19dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.112+0000 7fb612f3d700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c107210 0x7fb60c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.112+0000 7fb612f3d700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c1114e0 con 0x7fb60c101b70 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.112+0000 7fb612f3d700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c111360 con 0x7fb60c107210 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.112+0000 7fb612f3d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c111660 con 0x7fb60c1011d0 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb60bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb60c101b70 0x7fb60c19dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb60bfff700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb60c101b70 0x7fb60c19dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59250/0 (socket says 172.21.15.49:59250) 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb60bfff700 1 -- 172.21.15.49:0/2764137889 learned_addr learned my addr 172.21.15.49:0/2764137889 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:50.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb610cd9700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c1011d0 0x7fb60c19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb6114da700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c107210 0x7fb60c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb60bfff700 1 -- 172.21.15.49:0/2764137889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c1011d0 msgr2=0x7fb60c19d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb60bfff700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb60c1011d0 0x7fb60c19d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb60bfff700 1 -- 172.21.15.49:0/2764137889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c107210 msgr2=0x7fb60c1977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb60bfff700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb60c107210 0x7fb60c1977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb60bfff700 1 -- 172.21.15.49:0/2764137889 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60c198080 con 0x7fb60c101b70 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb60bfff700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb60c101b70 0x7fb60c19dbc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800e9c0 tx=0x7fb5f800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb609ffb700 1 -- 172.21.15.49:0/2764137889 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f800cbd0 con 0x7fb60c101b70 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb609ffb700 1 -- 172.21.15.49:0/2764137889 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5f800ee60 con 0x7fb60c101b70 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb60c198370 con 0x7fb60c101b70 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.113+0000 7fb609ffb700 1 -- 172.21.15.49:0/2764137889 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8005630 con 0x7fb60c101b70 2024-03-18T00:48:50.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.114+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb60c1a4600 con 0x7fb60c101b70 2024-03-18T00:48:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.114+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb60c062a20 con 0x7fb60c101b70 2024-03-18T00:48:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.116+0000 7fb609ffb700 1 -- 172.21.15.49:0/2764137889 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb5f8004750 con 0x7fb60c101b70 2024-03-18T00:48:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.116+0000 7fb609ffb700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5fc0a29a0 0x7fb5fc0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.116+0000 7fb610cd9700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5fc0a29a0 0x7fb5fc0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.116+0000 7fb609ffb700 1 -- 172.21.15.49:0/2764137889 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fb5f80cf070 con 0x7fb60c101b70 2024-03-18T00:48:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.116+0000 7fb609ffb700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] conn(0x7fb5fc0ac180 0x7fb5fc0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.116+0000 7fb609ffb700 1 -- 172.21.15.49:0/2764137889 --> [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5f801fe50 con 0x7fb5fc0ac180 2024-03-18T00:48:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.116+0000 7fb609ffb700 1 -- 172.21.15.49:0/2764137889 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fb5f80cf550 con 0x7fb60c101b70 2024-03-18T00:48:50.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.117+0000 7fb6114da700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] conn(0x7fb5fc0ac180 0x7fb5fc0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.117+0000 7fb610cd9700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5fc0a29a0 0x7fb5fc0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb600009be0 tx=0x7fb600009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:50.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.117+0000 7fb6114da700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] conn(0x7fb5fc0ac180 0x7fb5fc0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:50.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.118+0000 7fb609ffb700 1 -- 172.21.15.49:0/2764137889 <== osd.11 v2:172.21.15.107:6824/1232543995 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5f801fe50 con 0x7fb5fc0ac180 2024-03-18T00:48:50.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.134+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 --> [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb60c062ea0 con 0x7fb5fc0ac180 2024-03-18T00:48:50.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.134+0000 7fb609ffb700 1 -- 172.21.15.49:0/2764137889 <== osd.11 v2:172.21.15.107:6824/1232543995 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb60c062ea0 con 0x7fb5fc0ac180 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.134+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 >> [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] conn(0x7fb5fc0ac180 msgr2=0x7fb5fc0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.134+0000 7fb612f3d700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] conn(0x7fb5fc0ac180 0x7fb5fc0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.135+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5fc0a29a0 msgr2=0x7fb5fc0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.135+0000 7fb612f3d700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5fc0a29a0 0x7fb5fc0a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb600009be0 tx=0x7fb600009260 comp rx=0 tx=0).stop 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.135+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb60c101b70 msgr2=0x7fb60c19dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.135+0000 7fb612f3d700 1 --2- 172.21.15.49:0/2764137889 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb60c101b70 0x7fb60c19dbc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800e9c0 tx=0x7fb5f800ecd0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.135+0000 7fb6114da700 1 -- 172.21.15.49:0/2764137889 reap_dead start 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.135+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 shutdown_connections 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.135+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 >> 172.21.15.49:0/2764137889 conn(0x7fb60c078100 msgr2=0x7fb60c0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.135+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 shutdown_connections 2024-03-18T00:48:50.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.135+0000 7fb612f3d700 1 -- 172.21.15.49:0/2764137889 wait complete. 2024-03-18T00:48:50.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.646+0000 7f95dd885700 1 -- 172.21.15.49:0/1438680774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8102f60 msgr2=0x7f95d8106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.646+0000 7f95dd885700 1 --2- 172.21.15.49:0/1438680774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8102f60 0x7f95d8106f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009920 tx=0x7f95cc009c30 comp rx=0 tx=0).stop 2024-03-18T00:48:50.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.646+0000 7f95dd885700 1 -- 172.21.15.49:0/1438680774 shutdown_connections 2024-03-18T00:48:50.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.646+0000 7f95dd885700 1 --2- 172.21.15.49:0/1438680774 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95d8107660 0x7f95d810f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.646+0000 7f95dd885700 1 --2- 172.21.15.49:0/1438680774 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95d81025b0 0x7f95d8102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.646+0000 7f95dd885700 1 --2- 172.21.15.49:0/1438680774 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8102f60 0x7f95d8106f30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.646+0000 7f95dd885700 1 -- 172.21.15.49:0/1438680774 >> 172.21.15.49:0/1438680774 conn(0x7f95d80793f0 msgr2=0x7f95d8079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:50.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.646+0000 7f95dd885700 1 -- 172.21.15.49:0/1438680774 shutdown_connections 2024-03-18T00:48:50.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.647+0000 7f95dd885700 1 -- 172.21.15.49:0/1438680774 wait complete. 2024-03-18T00:48:50.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.647+0000 7f95dd885700 1 Processor -- start 2024-03-18T00:48:50.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.647+0000 7f95dd885700 1 -- start start 2024-03-18T00:48:50.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.647+0000 7f95dd885700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95d81025b0 0x7f95d819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.647+0000 7f95dd885700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95d8102f60 0x7f95d819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95dd885700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8107660 0x7f95d8198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95dd885700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f95d81125b0 con 0x7f95d81025b0 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95dd885700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f95d8112430 con 0x7f95d8102f60 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95dd885700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f95d8112730 con 0x7f95d8107660 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d6ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95d81025b0 0x7f95d819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d6ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95d81025b0 0x7f95d819e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59268/0 (socket says 172.21.15.49:59268) 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d6ffd700 1 -- 172.21.15.49:0/468187008 learned_addr learned my addr 172.21.15.49:0/468187008 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d77fe700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8107660 0x7f95d8198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d67fc700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95d8102f60 0x7f95d819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d6ffd700 1 -- 172.21.15.49:0/468187008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8107660 msgr2=0x7f95d8198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d6ffd700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95d8107660 0x7f95d8198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d6ffd700 1 -- 172.21.15.49:0/468187008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95d8102f60 msgr2=0x7f95d819eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d6ffd700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f95d8102f60 0x7f95d819eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d6ffd700 1 -- 172.21.15.49:0/468187008 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d81990d0 con 0x7f95d81025b0 2024-03-18T00:48:50.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95d6ffd700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95d81025b0 0x7f95d819e760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f95c800e9a0 tx=0x7f95c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:50.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95dc883700 1 -- 172.21.15.49:0/468187008 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c800ca10 con 0x7f95d81025b0 2024-03-18T00:48:50.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95d81993c0 con 0x7f95d81025b0 2024-03-18T00:48:50.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95d81a5630 con 0x7f95d81025b0 2024-03-18T00:48:50.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95dc883700 1 -- 172.21.15.49:0/468187008 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95c800cbb0 con 0x7f95d81025b0 2024-03-18T00:48:50.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.648+0000 7f95dc883700 1 -- 172.21.15.49:0/468187008 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8018680 con 0x7f95d81025b0 2024-03-18T00:48:50.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.649+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f95c4001030 con 0x7f95d81025b0 2024-03-18T00:48:50.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.650+0000 7f95dc883700 1 -- 172.21.15.49:0/468187008 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f95c8010430 con 0x7f95d81025b0 2024-03-18T00:48:50.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.651+0000 7f95dc883700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f95c00a2920 0x7f95c00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.651+0000 7f95d67fc700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f95c00a2920 0x7f95c00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.651+0000 7f95dc883700 1 -- 172.21.15.49:0/468187008 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f95c80cf680 con 0x7f95d81025b0 2024-03-18T00:48:50.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.651+0000 7f95dc883700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] conn(0x7f95c00ac100 0x7f95c00ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:50.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.651+0000 7f95dc883700 1 -- 172.21.15.49:0/468187008 --> [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95c800ae50 con 0x7f95c00ac100 2024-03-18T00:48:50.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.651+0000 7f95dc883700 1 -- 172.21.15.49:0/468187008 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f95c80cfb60 con 0x7f95d81025b0 2024-03-18T00:48:50.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.652+0000 7f95d77fe700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] conn(0x7f95c00ac100 0x7f95c00ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:50.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.652+0000 7f95d67fc700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f95c00a2920 0x7f95c00a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f95cc005e90 tx=0x7f95cc019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:50.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.652+0000 7f95d77fe700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] conn(0x7f95c00ac100 0x7f95c00ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:50.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.653+0000 7f95dc883700 1 -- 172.21.15.49:0/468187008 <== osd.3 v2:172.21.15.49:6810/725028995 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f95c800ae50 con 0x7f95c00ac100 2024-03-18T00:48:50.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.669+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 --> [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f95c4002e60 con 0x7f95c00ac100 2024-03-18T00:48:50.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.669+0000 7f95dc883700 1 -- 172.21.15.49:0/468187008 <== osd.3 v2:172.21.15.49:6810/725028995 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f95c4002e60 con 0x7f95c00ac100 2024-03-18T00:48:50.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.669+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] conn(0x7f95c00ac100 msgr2=0x7f95c00ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.669+0000 7f95dd885700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] conn(0x7f95c00ac100 0x7f95c00ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.669+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f95c00a2920 msgr2=0x7f95c00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.669+0000 7f95dd885700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f95c00a2920 0x7f95c00a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f95cc005e90 tx=0x7f95cc019040 comp rx=0 tx=0).stop 2024-03-18T00:48:50.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.669+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95d81025b0 msgr2=0x7f95d819e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:50.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.669+0000 7f95dd885700 1 --2- 172.21.15.49:0/468187008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f95d81025b0 0x7f95d819e760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f95c800e9a0 tx=0x7f95c800ecb0 comp rx=0 tx=0).stop 2024-03-18T00:48:50.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.670+0000 7f95d77fe700 1 -- 172.21.15.49:0/468187008 reap_dead start 2024-03-18T00:48:50.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.670+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 shutdown_connections 2024-03-18T00:48:50.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.670+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 >> 172.21.15.49:0/468187008 conn(0x7f95d80793f0 msgr2=0x7f95d81007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:50.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.670+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 shutdown_connections 2024-03-18T00:48:50.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:50.670+0000 7f95dd885700 1 -- 172.21.15.49:0/468187008 wait complete. 2024-03-18T00:48:51.319 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:51 smithi049 ceph-mon[32130]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T00:48:51.360 INFO:teuthology.orchestra.run.smithi049.stdout:167503724596 2024-03-18T00:48:51.360 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-18T00:48:51.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:51 smithi107 ceph-mon[36270]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T00:48:51.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:51 smithi106 ceph-mon[35899]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-18T00:48:51.858 INFO:teuthology.orchestra.run.smithi049.stdout:77309411394 2024-03-18T00:48:51.858 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-18T00:48:52.235 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:52.265 INFO:teuthology.orchestra.run.smithi049.stdout:111669149758 2024-03-18T00:48:52.265 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-18T00:48:52.672 INFO:teuthology.orchestra.run.smithi049.stdout:347892350997 2024-03-18T00:48:52.672 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-18T00:48:52.756 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:52.756 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:52.761 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:52.806 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:53.049 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:53.178 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:53 smithi049 ceph-mon[32130]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T00:48:53.294 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:53.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:53 smithi107 ceph-mon[36270]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T00:48:53.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:53 smithi106 ceph-mon[35899]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-03-18T00:48:54.427 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:55.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:55 smithi049 ceph-mon[32130]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:48:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:55 smithi107 ceph-mon[36270]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:48:55.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:55 smithi106 ceph-mon[35899]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:48:55.511 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:48:57.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:57 smithi049 ceph-mon[32130]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T00:48:57.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:57 smithi107 ceph-mon[36270]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T00:48:57.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:57 smithi106 ceph-mon[35899]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T00:48:59.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:48:59 smithi049 ceph-mon[32130]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:48:59.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:48:59 smithi107 ceph-mon[36270]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:48:59.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:48:59 smithi106 ceph-mon[35899]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:48:59.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 -- 172.21.15.49:0/1245489721 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406ac10 msgr2=0x7f424406aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:59.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1245489721 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406ac10 0x7f424406aff0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f423c00b0d0 tx=0x7f423c00b3e0 comp rx=0 tx=0).stop 2024-03-18T00:48:59.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 -- 172.21.15.49:0/1245489721 shutdown_connections 2024-03-18T00:48:59.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1245489721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4244107680 0x7f424410f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:59.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1245489721 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406ac10 0x7f424406aff0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:59.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1245489721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f424406b5c0 0x7f4244106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 -- 172.21.15.49:0/1245489721 >> 172.21.15.49:0/1245489721 conn(0x7f4244077220 msgr2=0x7f4244077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 -- 172.21.15.49:0/1245489721 shutdown_connections 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 -- 172.21.15.49:0/1245489721 wait complete. 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.833+0000 7f4249ecc700 1 Processor -- start 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4249ecc700 1 -- start start 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4249ecc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f424406ac10 0x7f4244101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4249ecc700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406b5c0 0x7f4244101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4249ecc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4244107680 0x7f4244101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4249ecc700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4244112680 con 0x7f424406b5c0 2024-03-18T00:48:59.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4249ecc700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4244112500 con 0x7f4244107680 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4249ecc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4244112800 con 0x7f424406ac10 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4242ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406b5c0 0x7f4244101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4242ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406b5c0 0x7f4244101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45162/0 (socket says 172.21.15.49:45162) 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f42437fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f424406ac10 0x7f4244101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4242ffd700 1 -- 172.21.15.49:0/1272872002 learned_addr learned my addr 172.21.15.49:0/1272872002 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4243fff700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4244107680 0x7f4244101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f42437fe700 1 -- 172.21.15.49:0/1272872002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4244107680 msgr2=0x7f4244101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f42437fe700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4244107680 0x7f4244101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f42437fe700 1 -- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406b5c0 msgr2=0x7f4244101840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f42437fe700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406b5c0 0x7f4244101840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f42437fe700 1 -- 172.21.15.49:0/1272872002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42441a7020 con 0x7f424406ac10 2024-03-18T00:48:59.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.834+0000 7f4242ffd700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406b5c0 0x7f4244101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:48:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.835+0000 7f42437fe700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f424406ac10 0x7f4244101300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f423c000f80 tx=0x7f423c007a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.835+0000 7f4240ff9700 1 -- 172.21.15.49:0/1272872002 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c003c00 con 0x7f424406ac10 2024-03-18T00:48:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.835+0000 7f4240ff9700 1 -- 172.21.15.49:0/1272872002 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f423c022070 con 0x7f424406ac10 2024-03-18T00:48:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.835+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42441a72b0 con 0x7f424406ac10 2024-03-18T00:48:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.835+0000 7f4240ff9700 1 -- 172.21.15.49:0/1272872002 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c01b6c0 con 0x7f424406ac10 2024-03-18T00:48:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.835+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42441a7700 con 0x7f424406ac10 2024-03-18T00:48:59.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.836+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f424410a830 con 0x7f424406ac10 2024-03-18T00:48:59.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.837+0000 7f4240ff9700 1 -- 172.21.15.49:0/1272872002 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f423c04eb70 con 0x7f424406ac10 2024-03-18T00:48:59.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.837+0000 7f4240ff9700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f42300a2920 0x7f42300a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:48:59.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.838+0000 7f4242ffd700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f42300a2920 0x7f42300a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:48:59.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.838+0000 7f4240ff9700 1 -- 172.21.15.49:0/1272872002 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f423c00f020 con 0x7f424406ac10 2024-03-18T00:48:59.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.838+0000 7f4242ffd700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f42300a2920 0x7f42300a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4244100df0 tx=0x7f422c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:48:59.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:48:59.841+0000 7f4240ff9700 1 -- 172.21.15.49:0/1272872002 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f423c09c860 con 0x7f424406ac10 2024-03-18T00:49:00.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.045+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f424410a210 con 0x7f424406ac10 2024-03-18T00:49:00.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.046+0000 7f4240ff9700 1 -- 172.21.15.49:0/1272872002 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f423c09c160 con 0x7f424406ac10 2024-03-18T00:49:00.046 INFO:teuthology.orchestra.run.smithi049.stdout:257698037799 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f42300a2920 msgr2=0x7f42300a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f42300a2920 0x7f42300a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4244100df0 tx=0x7f422c009260 comp rx=0 tx=0).stop 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f424406ac10 msgr2=0x7f4244101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f424406ac10 0x7f4244101300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f423c000f80 tx=0x7f423c007a20 comp rx=0 tx=0).stop 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 shutdown_connections 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f42300a2920 0x7f42300a4de0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f424406ac10 0x7f4244101300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4244107680 0x7f4244101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 --2- 172.21.15.49:0/1272872002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f424406b5c0 0x7f4244101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 >> 172.21.15.49:0/1272872002 conn(0x7f4244077220 msgr2=0x7f42440ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 shutdown_connections 2024-03-18T00:49:00.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:00.048+0000 7f4249ecc700 1 -- 172.21.15.49:0/1272872002 wait complete. 2024-03-18T00:49:00.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:00 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1272872002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T00:49:00.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:00 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1272872002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T00:49:00.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:00 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1272872002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T00:49:01.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:01 smithi049 ceph-mon[32130]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:01 smithi107 ceph-mon[36270]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:01.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:01 smithi106 ceph-mon[35899]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.140+0000 7f5098de5700 1 -- 172.21.15.49:0/2082342479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 msgr2=0x7f5094108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.140+0000 7f5098de5700 1 --2- 172.21.15.49:0/2082342479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 0x7f5094108800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f50840099c0 tx=0x7f5084009cd0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.140+0000 7f5098de5700 1 -- 172.21.15.49:0/2082342479 shutdown_connections 2024-03-18T00:49:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.140+0000 7f5098de5700 1 --2- 172.21.15.49:0/2082342479 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50941011e0 0x7f509410f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.140+0000 7f5098de5700 1 --2- 172.21.15.49:0/2082342479 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 0x7f5094108800 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.140+0000 7f5098de5700 1 --2- 172.21.15.49:0/2082342479 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5094100820 0x7f5094100ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.140+0000 7f5098de5700 1 -- 172.21.15.49:0/2082342479 >> 172.21.15.49:0/2082342479 conn(0x7f50940777a0 msgr2=0x7f50940feb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.140+0000 7f5098de5700 1 -- 172.21.15.49:0/2082342479 shutdown_connections 2024-03-18T00:49:02.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.140+0000 7f5098de5700 1 -- 172.21.15.49:0/2082342479 wait complete. 2024-03-18T00:49:02.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.141+0000 7f5098de5700 1 Processor -- start 2024-03-18T00:49:02.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.141+0000 7f5098de5700 1 -- start start 2024-03-18T00:49:02.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.141+0000 7f5098de5700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5094100820 0x7f509419e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:02.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.141+0000 7f5098de5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50941011e0 0x7f509419ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:02.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.141+0000 7f5098de5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 0x7f5094198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:02.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.141+0000 7f5091d9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50941011e0 0x7f509419ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:02.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.142+0000 7f5091d9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50941011e0 0x7f509419ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46446/0 (socket says 172.21.15.49:46446) 2024-03-18T00:49:02.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.142+0000 7f5092d9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 0x7f5094198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:02.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.142+0000 7f5092d9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 0x7f5094198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45180/0 (socket says 172.21.15.49:45180) 2024-03-18T00:49:02.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.142+0000 7f509259c700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5094100820 0x7f509419e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:02.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.142+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f50941126b0 con 0x7f5094108420 2024-03-18T00:49:02.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.142+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5094112530 con 0x7f5094100820 2024-03-18T00:49:02.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.142+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5094112830 con 0x7f50941011e0 2024-03-18T00:49:02.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.142+0000 7f5091d9b700 1 -- 172.21.15.49:0/973971169 learned_addr learned my addr 172.21.15.49:0/973971169 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:02.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f509259c700 1 -- 172.21.15.49:0/973971169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50941011e0 msgr2=0x7f509419ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:02.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f509259c700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50941011e0 0x7f509419ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f509259c700 1 -- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 msgr2=0x7f5094198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:02.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f509259c700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 0x7f5094198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f509259c700 1 -- 172.21.15.49:0/973971169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50941990d0 con 0x7f5094100820 2024-03-18T00:49:02.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f5091d9b700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50941011e0 0x7f509419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:49:02.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f5092d9d700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 0x7f5094198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:49:02.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f509259c700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5094100820 0x7f509419e7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f50840098e0 tx=0x7f5084005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:02.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f50837fe700 1 -- 172.21.15.49:0/973971169 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5084027070 con 0x7f5094100820 2024-03-18T00:49:02.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f50837fe700 1 -- 172.21.15.49:0/973971169 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5084003680 con 0x7f5094100820 2024-03-18T00:49:02.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5094199360 con 0x7f5094100820 2024-03-18T00:49:02.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.143+0000 7f50837fe700 1 -- 172.21.15.49:0/973971169 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50840173f0 con 0x7f5094100820 2024-03-18T00:49:02.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.144+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50941a5680 con 0x7f5094100820 2024-03-18T00:49:02.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.145+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509410a8b0 con 0x7f5094100820 2024-03-18T00:49:02.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.146+0000 7f50837fe700 1 -- 172.21.15.49:0/973971169 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f508402a070 con 0x7f5094100820 2024-03-18T00:49:02.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.146+0000 7f50837fe700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f507c0a29a0 0x7f507c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:02.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.147+0000 7f50837fe700 1 -- 172.21.15.49:0/973971169 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f50840cf7c0 con 0x7f5094100820 2024-03-18T00:49:02.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.147+0000 7f5091d9b700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f507c0a29a0 0x7f507c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:02.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.147+0000 7f5091d9b700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f507c0a29a0 0x7f507c0a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f50940ffe00 tx=0x7f5088009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:02.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.151+0000 7f50837fe700 1 -- 172.21.15.49:0/973971169 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5084013070 con 0x7f5094100820 2024-03-18T00:49:02.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.357+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5094068f90 con 0x7f5094100820 2024-03-18T00:49:02.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-03-18T00:49:02.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-18T00:49:02.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-18T00:49:02.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 6]}]: dispatch 2024-03-18T00:49:02.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]: dispatch 2024-03-18T00:49:02.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-18T00:49:02.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-03-18T00:49:02.357 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-18T00:49:02.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-18T00:49:02.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:02 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-18T00:49:02.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.357+0000 7f50837fe700 1 -- 172.21.15.49:0/973971169 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f508409c770 con 0x7f5094100820 2024-03-18T00:49:02.358 INFO:teuthology.orchestra.run.smithi049.stdout:55834574923 2024-03-18T00:49:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f507c0a29a0 msgr2=0x7f507c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f507c0a29a0 0x7f507c0a4e60 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f50940ffe00 tx=0x7f5088009260 comp rx=0 tx=0).stop 2024-03-18T00:49:02.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5094100820 msgr2=0x7f509419e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5094100820 0x7f509419e7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f50840098e0 tx=0x7f5084005f10 comp rx=0 tx=0).stop 2024-03-18T00:49:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 shutdown_connections 2024-03-18T00:49:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5094108420 0x7f5094198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5094100820 0x7f509419e7b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f507c0a29a0 0x7f507c0a4e60 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 --2- 172.21.15.49:0/973971169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50941011e0 0x7f509419ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 >> 172.21.15.49:0/973971169 conn(0x7f50940777a0 msgr2=0x7f5094105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 shutdown_connections 2024-03-18T00:49:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:02.359+0000 7f5098de5700 1 -- 172.21.15.49:0/973971169 wait complete. 2024-03-18T00:49:02.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-03-18T00:49:02.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-18T00:49:02.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-18T00:49:02.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 6]}]: dispatch 2024-03-18T00:49:02.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]: dispatch 2024-03-18T00:49:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-18T00:49:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-03-18T00:49:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-18T00:49:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-18T00:49:02.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:02 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-18T00:49:02.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]: dispatch 2024-03-18T00:49:02.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-18T00:49:02.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2024-03-18T00:49:02.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 6]}]: dispatch 2024-03-18T00:49:02.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]: dispatch 2024-03-18T00:49:02.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2024-03-18T00:49:02.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-03-18T00:49:02.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-18T00:49:02.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-03-18T00:49:02.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:02 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-18T00:49:03.358 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 6]}]': finished 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]': finished 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-18T00:49:03.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-18T00:49:03.360 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T00:49:03.360 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:03 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/973971169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T00:49:03.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:03.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-03-18T00:49:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-18T00:49:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-18T00:49:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 6]}]': finished 2024-03-18T00:49:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]': finished 2024-03-18T00:49:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-18T00:49:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-03-18T00:49:03.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-18T00:49:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-18T00:49:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-18T00:49:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T00:49:03.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:03 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/973971169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T00:49:03.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: pgmap v316: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:03.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [8, 4]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 6]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [5, 11]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T00:49:03.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:03 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/973971169' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T00:49:04.312 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:04 smithi049 ceph-mon[32130]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T00:49:04.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:04 smithi107 ceph-mon[36270]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T00:49:04.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:04 smithi106 ceph-mon[35899]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T00:49:05.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:05 smithi049 ceph-mon[32130]: pgmap v319: 97 pgs: 1 activating+degraded, 6 peering, 90 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s; 8/306 objects degraded (2.614%) 2024-03-18T00:49:05.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:05 smithi049 ceph-mon[32130]: Health check failed: Degraded data redundancy: 8/306 objects degraded (2.614%), 1 pg degraded (PG_DEGRADED) 2024-03-18T00:49:05.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:05 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/594812730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T00:49:05.307 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:05 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:49:05.411 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:05 smithi106 ceph-mon[35899]: pgmap v319: 97 pgs: 1 activating+degraded, 6 peering, 90 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s; 8/306 objects degraded (2.614%) 2024-03-18T00:49:05.411 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:05 smithi106 ceph-mon[35899]: Health check failed: Degraded data redundancy: 8/306 objects degraded (2.614%), 1 pg degraded (PG_DEGRADED) 2024-03-18T00:49:05.411 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:05 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/594812730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T00:49:05.411 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:05 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:49:05.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:05 smithi107 ceph-mon[36270]: pgmap v319: 97 pgs: 1 activating+degraded, 6 peering, 90 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s; 8/306 objects degraded (2.614%) 2024-03-18T00:49:05.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:05 smithi107 ceph-mon[36270]: Health check failed: Degraded data redundancy: 8/306 objects degraded (2.614%), 1 pg degraded (PG_DEGRADED) 2024-03-18T00:49:05.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:05 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/594812730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T00:49:05.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:05 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:49:05.698 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:05.699 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:05.700 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:06.123 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037796 got 257698037799 for osd.8 2024-03-18T00:49:06.123 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:06.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.319+0000 7fe4ce32a700 1 -- 172.21.15.49:0/3938760837 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8108300 msgr2=0x7fe4c81086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.319+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/3938760837 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8108300 0x7fe4c81086e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe4b80099c0 tx=0x7fe4b8009cd0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.319+0000 7fe4ce32a700 1 -- 172.21.15.49:0/3938760837 shutdown_connections 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.319+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/3938760837 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8102ce0 0x7fe4c810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.319+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/3938760837 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8108300 0x7fe4c81086e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.319+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/3938760837 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c8102320 0x7fe4c81027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.319+0000 7fe4ce32a700 1 -- 172.21.15.49:0/3938760837 >> 172.21.15.49:0/3938760837 conn(0x7fe4c8079400 msgr2=0x7fe4c8079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.319+0000 7fe4ce32a700 1 -- 172.21.15.49:0/3938760837 shutdown_connections 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.319+0000 7fe4ce32a700 1 -- 172.21.15.49:0/3938760837 wait complete. 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4ce32a700 1 Processor -- start 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4ce32a700 1 -- start start 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4ce32a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8102320 0x7fe4c80745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:06.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4ce32a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8102ce0 0x7fe4c8074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4ce32a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c8108300 0x7fe4c8075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4ce32a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c81125b0 con 0x7fe4c8102ce0 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4ce32a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c8112430 con 0x7fe4c8108300 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4ce32a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c8112730 con 0x7fe4c8102320 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4c77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8102ce0 0x7fe4c8074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.320+0000 7fe4c77fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8102ce0 0x7fe4c8074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45214/0 (socket says 172.21.15.49:45214) 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c77fe700 1 -- 172.21.15.49:0/4249666574 learned_addr learned my addr 172.21.15.49:0/4249666574 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c7fff700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8102320 0x7fe4c80745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4cc8c7700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c8108300 0x7fe4c8075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c77fe700 1 -- 172.21.15.49:0/4249666574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8102320 msgr2=0x7fe4c80745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c77fe700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8102320 0x7fe4c80745d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c77fe700 1 -- 172.21.15.49:0/4249666574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c8108300 msgr2=0x7fe4c8075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c77fe700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c8108300 0x7fe4c8075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c77fe700 1 -- 172.21.15.49:0/4249666574 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c81a6ed0 con 0x7fe4c8102ce0 2024-03-18T00:49:06.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c77fe700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8102ce0 0x7fe4c8074b10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00d980 tx=0x7fe4bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:06.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c57fa700 1 -- 172.21.15.49:0/4249666574 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc016070 con 0x7fe4c8102ce0 2024-03-18T00:49:06.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c81a71c0 con 0x7fe4c8102ce0 2024-03-18T00:49:06.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c57fa700 1 -- 172.21.15.49:0/4249666574 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4bc004d10 con 0x7fe4c8102ce0 2024-03-18T00:49:06.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4c57fa700 1 -- 172.21.15.49:0/4249666574 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc005670 con 0x7fe4c8102ce0 2024-03-18T00:49:06.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.321+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c81a7750 con 0x7fe4c8102ce0 2024-03-18T00:49:06.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.322+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c8068f90 con 0x7fe4c8102ce0 2024-03-18T00:49:06.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.324+0000 7fe4c57fa700 1 -- 172.21.15.49:0/4249666574 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe4bc004750 con 0x7fe4c8102ce0 2024-03-18T00:49:06.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.324+0000 7fe4c57fa700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe4b00a2920 0x7fe4b00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:06.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.324+0000 7fe4c7fff700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe4b00a2920 0x7fe4b00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:06.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.325+0000 7fe4c57fa700 1 -- 172.21.15.49:0/4249666574 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe4bc0ce9a0 con 0x7fe4c8102ce0 2024-03-18T00:49:06.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.325+0000 7fe4c7fff700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe4b00a2920 0x7fe4b00a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8000c00 tx=0x7fe4b8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:06.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.329+0000 7fe4c57fa700 1 -- 172.21.15.49:0/4249666574 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe4bc01f020 con 0x7fe4c8102ce0 2024-03-18T00:49:06.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.311+0000 7f8edc734700 1 -- 172.21.15.49:0/3720862715 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed406ac30 msgr2=0x7f8ed406b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.311+0000 7f8edc734700 1 --2- 172.21.15.49:0/3720862715 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed406ac30 0x7f8ed406b010 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80099c0 tx=0x7f8ec8009cd0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.311+0000 7f8edc734700 1 -- 172.21.15.49:0/3720862715 shutdown_connections 2024-03-18T00:49:06.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.311+0000 7f8edc734700 1 --2- 172.21.15.49:0/3720862715 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed4107680 0x7f8ed410f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.311+0000 7f8edc734700 1 --2- 172.21.15.49:0/3720862715 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed406ac30 0x7f8ed406b010 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.311+0000 7f8edc734700 1 --2- 172.21.15.49:0/3720862715 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed406b5e0 0x7f8ed4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.311+0000 7f8edc734700 1 -- 172.21.15.49:0/3720862715 >> 172.21.15.49:0/3720862715 conn(0x7f8ed4077220 msgr2=0x7f8ed4077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.311+0000 7f8edc734700 1 -- 172.21.15.49:0/3720862715 shutdown_connections 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.311+0000 7f8edc734700 1 -- 172.21.15.49:0/3720862715 wait complete. 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.312+0000 7f8edc734700 1 Processor -- start 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.312+0000 7f8edc734700 1 -- start start 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.312+0000 7f8edc734700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed406ac30 0x7f8ed419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.312+0000 7f8edc734700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed406b5e0 0x7f8ed419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.312+0000 7f8edc734700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed4107680 0x7f8ed41988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.312+0000 7f8edc734700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed4112630 con 0x7f8ed4107680 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.312+0000 7f8edc734700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed41124b0 con 0x7f8ed406ac30 2024-03-18T00:49:06.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.312+0000 7f8edc734700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed41127b0 con 0x7f8ed406b5e0 2024-03-18T00:49:06.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edacd1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed4107680 0x7f8ed41988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:06.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edacd1700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed4107680 0x7f8ed41988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45194/0 (socket says 172.21.15.49:45194) 2024-03-18T00:49:06.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8eda4d0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed406ac30 0x7f8ed419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:06.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edacd1700 1 -- 172.21.15.49:0/594812730 learned_addr learned my addr 172.21.15.49:0/594812730 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:06.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8ed9ccf700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed406b5e0 0x7f8ed419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:06.498 INFO:teuthology.orchestra.run.smithi049.stdout:201863462963 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edacd1700 1 -- 172.21.15.49:0/594812730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed406b5e0 msgr2=0x7f8ed419ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edacd1700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed406b5e0 0x7f8ed419ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edacd1700 1 -- 172.21.15.49:0/594812730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed406ac30 msgr2=0x7f8ed419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edacd1700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed406ac30 0x7f8ed419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edacd1700 1 -- 172.21.15.49:0/594812730 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed4199030 con 0x7f8ed4107680 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8eda4d0700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed406ac30 0x7f8ed419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edacd1700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed4107680 0x7f8ed41988d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00ef30 tx=0x7f8ecc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8ec77fe700 1 -- 172.21.15.49:0/594812730 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc0092e0 con 0x7f8ed4107680 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed41992c0 con 0x7f8ed4107680 2024-03-18T00:49:06.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8ec77fe700 1 -- 172.21.15.49:0/594812730 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ecc004bb0 con 0x7f8ed4107680 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8ec77fe700 1 -- 172.21.15.49:0/594812730 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc004280 con 0x7f8ed4107680 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.313+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed41a55e0 con 0x7f8ed4107680 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.314+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed410a850 con 0x7f8ed4107680 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.315+0000 7f8ec77fe700 1 -- 172.21.15.49:0/594812730 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8ecc01b030 con 0x7f8ed4107680 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.316+0000 7f8ec77fe700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ec00a2920 0x7f8ec00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.316+0000 7f8eda4d0700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ec00a2920 0x7f8ec00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.316+0000 7f8ec77fe700 1 -- 172.21.15.49:0/594812730 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8ecc0ce660 con 0x7f8ed4107680 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.317+0000 7f8eda4d0700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ec00a2920 0x7f8ec00a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8000c00 tx=0x7f8ec8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.321+0000 7f8ec77fe700 1 -- 172.21.15.49:0/594812730 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8ecc022020 con 0x7f8ed4107680 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.532+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f8ed4199e00 con 0x7f8ed4107680 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.532+0000 7f8ec77fe700 1 -- 172.21.15.49:0/594812730 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8ecc010040 con 0x7f8ed4107680 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ec00a2920 msgr2=0x7f8ec00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ec00a2920 0x7f8ec00a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8000c00 tx=0x7f8ec8005fb0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed4107680 msgr2=0x7f8ed41988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed4107680 0x7f8ed41988d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc00ef30 tx=0x7f8ecc00c3a0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 shutdown_connections 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ed4107680 0x7f8ed41988d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ed406ac30 0x7f8ed419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ec00a2920 0x7f8ec00a4de0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 --2- 172.21.15.49:0/594812730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ed406b5e0 0x7f8ed419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.534+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 >> 172.21.15.49:0/594812730 conn(0x7f8ed4077220 msgr2=0x7f8ed41087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.535+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 shutdown_connections 2024-03-18T00:49:06.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:04.535+0000 7f8edc734700 1 -- 172.21.15.49:0/594812730 wait complete. 2024-03-18T00:49:06.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.538+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fe4c802db70 con 0x7fe4c8102ce0 2024-03-18T00:49:06.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.538+0000 7fe4c57fa700 1 -- 172.21.15.49:0/4249666574 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe4bc09b8c0 con 0x7fe4c8102ce0 2024-03-18T00:49:06.538 INFO:teuthology.orchestra.run.smithi049.stdout:146028888124 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe4b00a2920 msgr2=0x7fe4b00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe4b00a2920 0x7fe4b00a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8000c00 tx=0x7fe4b8005fb0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8102ce0 msgr2=0x7fe4c8074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8102ce0 0x7fe4c8074b10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00d980 tx=0x7fe4bc00dc90 comp rx=0 tx=0).stop 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 shutdown_connections 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe4b00a2920 0x7fe4b00a4de0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4c8102320 0x7fe4c80745d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe4c8108300 0x7fe4c8075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 --2- 172.21.15.49:0/4249666574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe4c8102ce0 0x7fe4c8074b10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 >> 172.21.15.49:0/4249666574 conn(0x7fe4c8079400 msgr2=0x7fe4c8100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 shutdown_connections 2024-03-18T00:49:06.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:06.541+0000 7fe4ce32a700 1 -- 172.21.15.49:0/4249666574 wait complete. 2024-03-18T00:49:07.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.199+0000 7f6ad6e42700 1 -- 172.21.15.49:0/156610632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad01075f0 msgr2=0x7f6ad010f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.199+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/156610632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad01075f0 0x7f6ad010f1c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800b0d0 tx=0x7f6ac800b3e0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.199+0000 7f6ad6e42700 1 -- 172.21.15.49:0/156610632 shutdown_connections 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.199+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/156610632 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad0102510 0x7f6ad01028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.199+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/156610632 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad01075f0 0x7f6ad010f1c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.199+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/156610632 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ad0102ec0 0x7f6ad0106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.199+0000 7f6ad6e42700 1 -- 172.21.15.49:0/156610632 >> 172.21.15.49:0/156610632 conn(0x7f6ad00fdd80 msgr2=0x7f6ad01001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.199+0000 7f6ad6e42700 1 -- 172.21.15.49:0/156610632 shutdown_connections 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.199+0000 7f6ad6e42700 1 -- 172.21.15.49:0/156610632 wait complete. 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad6e42700 1 Processor -- start 2024-03-18T00:49:07.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad6e42700 1 -- start start 2024-03-18T00:49:07.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad6e42700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad0102510 0x7f6ad00745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad6e42700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad0102ec0 0x7f6ad0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad6e42700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ad01075f0 0x7f6ad0075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad6e42700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad01125e0 con 0x7f6ad0102510 2024-03-18T00:49:07.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad6e42700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad0112460 con 0x7f6ad01075f0 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad6e42700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad0112760 con 0x7f6ad0102ec0 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad4bde700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad0102510 0x7f6ad00745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad4bde700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad0102510 0x7f6ad00745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45250/0 (socket says 172.21.15.49:45250) 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad4bde700 1 -- 172.21.15.49:0/1303741541 learned_addr learned my addr 172.21.15.49:0/1303741541 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6ad53df700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ad01075f0 0x7f6ad0075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.200+0000 7f6acffff700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad0102ec0 0x7f6ad0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6ad4bde700 1 -- 172.21.15.49:0/1303741541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad0102ec0 msgr2=0x7f6ad0074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6ad4bde700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad0102ec0 0x7f6ad0074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6ad4bde700 1 -- 172.21.15.49:0/1303741541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ad01075f0 msgr2=0x7f6ad0075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6ad4bde700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ad01075f0 0x7f6ad0075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6ad4bde700 1 -- 172.21.15.49:0/1303741541 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad01a6f60 con 0x7f6ad0102510 2024-03-18T00:49:07.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6ad53df700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ad01075f0 0x7f6ad0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:07.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6ad4bde700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad0102510 0x7f6ad00745d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400e9a0 tx=0x7f6ac400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:07.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6acdffb700 1 -- 172.21.15.49:0/1303741541 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac400ca10 con 0x7f6ad0102510 2024-03-18T00:49:07.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad01a7250 con 0x7f6ad0102510 2024-03-18T00:49:07.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6acdffb700 1 -- 172.21.15.49:0/1303741541 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ac400cbb0 con 0x7f6ad0102510 2024-03-18T00:49:07.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6acdffb700 1 -- 172.21.15.49:0/1303741541 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4018680 con 0x7f6ad0102510 2024-03-18T00:49:07.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.201+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad01a77e0 con 0x7f6ad0102510 2024-03-18T00:49:07.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.202+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ad0068f90 con 0x7f6ad0102510 2024-03-18T00:49:07.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.204+0000 7f6acdffb700 1 -- 172.21.15.49:0/1303741541 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6ac4010430 con 0x7f6ad0102510 2024-03-18T00:49:07.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.204+0000 7f6acdffb700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6ac00a2920 0x7f6ac00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.204+0000 7f6acffff700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6ac00a2920 0x7f6ac00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.204+0000 7f6acdffb700 1 -- 172.21.15.49:0/1303741541 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6ac40cf2e0 con 0x7f6ad0102510 2024-03-18T00:49:07.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.205+0000 7f6acffff700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6ac00a2920 0x7f6ac00a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0108ad0 tx=0x7f6abc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:07.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.212+0000 7f6acdffb700 1 -- 172.21.15.49:0/1303741541 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ac409c200 con 0x7f6ad0102510 2024-03-18T00:49:07.296 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:07 smithi049 ceph-mon[32130]: pgmap v320: 97 pgs: 1 activating+degraded, 6 peering, 90 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s; 8/306 objects degraded (2.614%) 2024-03-18T00:49:07.296 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.296 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/4249666574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T00:49:07.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.297 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:07 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.404+0000 7f6ca1a29700 1 -- 172.21.15.49:0/2570386015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c9c06a9d0 msgr2=0x7f6c9c06ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.404+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/2570386015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c9c06a9d0 0x7f6c9c06ae30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6c900099c0 tx=0x7f6c90009cd0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.404+0000 7f6ca1a29700 1 -- 172.21.15.49:0/2570386015 shutdown_connections 2024-03-18T00:49:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.404+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/2570386015 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6c9c06b370 0x7f6c9c10f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.404+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/2570386015 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c9c108350 0x7f6c9c108730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.404+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/2570386015 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c9c06a9d0 0x7f6c9c06ae30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.404+0000 7f6ca1a29700 1 -- 172.21.15.49:0/2570386015 >> 172.21.15.49:0/2570386015 conn(0x7f6c9c0771f0 msgr2=0x7f6c9c077600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.405+0000 7f6ca1a29700 1 -- 172.21.15.49:0/2570386015 shutdown_connections 2024-03-18T00:49:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.405+0000 7f6ca1a29700 1 -- 172.21.15.49:0/2570386015 wait complete. 2024-03-18T00:49:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.405+0000 7f6ca1a29700 1 Processor -- start 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.405+0000 7f6ca1a29700 1 -- start start 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6ca1a29700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6c9c06a9d0 0x7f6c9c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6ca1a29700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c9c06b370 0x7f6c9c1a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6ca1a29700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c9c108350 0x7f6c9c19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6ca1a29700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c1125d0 con 0x7f6c9c06b370 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6ca1a29700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c112450 con 0x7f6c9c06a9d0 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6ca1a29700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9c112750 con 0x7f6c9c108350 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9affd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6c9c06a9d0 0x7f6c9c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9b7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c9c108350 0x7f6c9c19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9affd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6c9c06a9d0 0x7f6c9c1a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:50786/0 (socket says 172.21.15.49:50786) 2024-03-18T00:49:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9affd700 1 -- 172.21.15.49:0/4262411299 learned_addr learned my addr 172.21.15.49:0/4262411299 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9a7fc700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c9c06b370 0x7f6c9c1a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9b7fe700 1 -- 172.21.15.49:0/4262411299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6c9c06a9d0 msgr2=0x7f6c9c1a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9b7fe700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6c9c06a9d0 0x7f6c9c1a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9b7fe700 1 -- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c9c06b370 msgr2=0x7f6c9c1a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9b7fe700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c9c06b370 0x7f6c9c1a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9b7fe700 1 -- 172.21.15.49:0/4262411299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c9c19b1b0 con 0x7f6c9c108350 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9a7fc700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c9c06b370 0x7f6c9c1a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9affd700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6c9c06a9d0 0x7f6c9c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.406+0000 7f6c9b7fe700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c9c108350 0x7f6c9c19aa50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400efc0 tx=0x7f6c9400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.407+0000 7f6ca0a27700 1 -- 172.21.15.49:0/4262411299 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c940092e0 con 0x7f6c9c108350 2024-03-18T00:49:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.407+0000 7f6ca0a27700 1 -- 172.21.15.49:0/4262411299 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c94004bb0 con 0x7f6c9c108350 2024-03-18T00:49:07.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.407+0000 7f6ca0a27700 1 -- 172.21.15.49:0/4262411299 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c94004280 con 0x7f6c9c108350 2024-03-18T00:49:07.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.407+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c9c19b440 con 0x7f6c9c108350 2024-03-18T00:49:07.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.407+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c9c1a7760 con 0x7f6c9c108350 2024-03-18T00:49:07.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.408+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c9c068f90 con 0x7f6c9c108350 2024-03-18T00:49:07.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.409+0000 7f6ca0a27700 1 -- 172.21.15.49:0/4262411299 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f6c9401b030 con 0x7f6c9c108350 2024-03-18T00:49:07.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.409+0000 7f6ca0a27700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c840a2920 0x7f6c840a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.410+0000 7f6c9affd700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c840a2920 0x7f6c840a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.410+0000 7f6ca0a27700 1 -- 172.21.15.49:0/4262411299 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6c940ce8f0 con 0x7f6c9c108350 2024-03-18T00:49:07.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.410+0000 7f6c9affd700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c840a2920 0x7f6c840a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c19bb90 tx=0x7f6c8c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:07.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.413+0000 7f6ca0a27700 1 -- 172.21.15.49:0/4262411299 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6c94022020 con 0x7f6c9c108350 2024-03-18T00:49:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:07 smithi107 ceph-mon[36270]: pgmap v320: 97 pgs: 1 activating+degraded, 6 peering, 90 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s; 8/306 objects degraded (2.614%) 2024-03-18T00:49:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:07 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/4249666574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T00:49:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:07 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.443+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6ad002db70 con 0x7f6ad0102510 2024-03-18T00:49:07.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.444+0000 7f6acdffb700 1 -- 172.21.15.49:0/1303741541 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6ac409bb00 con 0x7f6ad0102510 2024-03-18T00:49:07.444 INFO:teuthology.orchestra.run.smithi049.stdout:167503724599 2024-03-18T00:49:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:07 smithi106 ceph-mon[35899]: pgmap v320: 97 pgs: 1 activating+degraded, 6 peering, 90 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s; 8/306 objects degraded (2.614%) 2024-03-18T00:49:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:07 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/4249666574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T00:49:07.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:07 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6ac00a2920 msgr2=0x7f6ac00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6ac00a2920 0x7f6ac00a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0108ad0 tx=0x7f6abc009260 comp rx=0 tx=0).stop 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad0102510 msgr2=0x7f6ad00745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad0102510 0x7f6ad00745d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400e9a0 tx=0x7f6ac400ecb0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 shutdown_connections 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ad01075f0 0x7f6ad0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ad0102510 0x7f6ad00745d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6ac00a2920 0x7f6ac00a4de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 --2- 172.21.15.49:0/1303741541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ad0102ec0 0x7f6ad0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 >> 172.21.15.49:0/1303741541 conn(0x7f6ad00fdd80 msgr2=0x7f6ad00fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:07.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 shutdown_connections 2024-03-18T00:49:07.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.446+0000 7f6ad6e42700 1 -- 172.21.15.49:0/1303741541 wait complete. 2024-03-18T00:49:07.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.632+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6c9c1a7930 con 0x7f6c9c108350 2024-03-18T00:49:07.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.632+0000 7f6ca0a27700 1 -- 172.21.15.49:0/4262411299 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6c94010040 con 0x7f6c9c108350 2024-03-18T00:49:07.632 INFO:teuthology.orchestra.run.smithi049.stdout:223338299438 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c840a2920 msgr2=0x7f6c840a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c840a2920 0x7f6c840a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c19bb90 tx=0x7f6c8c009260 comp rx=0 tx=0).stop 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c9c108350 msgr2=0x7f6c9c19aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c9c108350 0x7f6c9c19aa50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400efc0 tx=0x7f6c9400c3a0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 shutdown_connections 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c840a2920 0x7f6c840a4de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c9c108350 0x7f6c9c19aa50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6c9c06a9d0 0x7f6c9c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 --2- 172.21.15.49:0/4262411299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6c9c06b370 0x7f6c9c1a0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 >> 172.21.15.49:0/4262411299 conn(0x7f6c9c0771f0 msgr2=0x7f6c9c10d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 shutdown_connections 2024-03-18T00:49:07.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.635+0000 7f6ca1a29700 1 -- 172.21.15.49:0/4262411299 wait complete. 2024-03-18T00:49:07.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 -- 172.21.15.49:0/2871556231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101420 msgr2=0x7f9860101800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 --2- 172.21.15.49:0/2871556231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101420 0x7f9860101800 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9854009990 tx=0x7f9854009ca0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 -- 172.21.15.49:0/2871556231 shutdown_connections 2024-03-18T00:49:07.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 --2- 172.21.15.49:0/2871556231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98601064e0 0x7f986010e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 --2- 172.21.15.49:0/2871556231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101420 0x7f9860101800 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 --2- 172.21.15.49:0/2871556231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9860101dd0 0x7f9860105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 -- 172.21.15.49:0/2871556231 >> 172.21.15.49:0/2871556231 conn(0x7f9860078140 msgr2=0x7f9860078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:07.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 -- 172.21.15.49:0/2871556231 shutdown_connections 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 -- 172.21.15.49:0/2871556231 wait complete. 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.090+0000 7f9865af0700 1 Processor -- start 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f9865af0700 1 -- start start 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f9865af0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9860101420 0x7f98601998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f9865af0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101dd0 0x7f9860199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f9865af0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98601064e0 0x7f98601a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f9865af0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f98601114d0 con 0x7f9860101dd0 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f9865af0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9860111350 con 0x7f98601064e0 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f9865af0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9860111650 con 0x7f9860101420 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101dd0 0x7f9860199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985effd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101dd0 0x7f9860199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45232/0 (socket says 172.21.15.49:45232) 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985f7fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9860101420 0x7f98601998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985effd700 1 -- 172.21.15.49:0/1700442905 learned_addr learned my addr 172.21.15.49:0/1700442905 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:07.828 INFO:teuthology.orchestra.run.smithi049.stdout:313532612639 2024-03-18T00:49:07.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985ffff700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98601064e0 0x7f98601a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985effd700 1 -- 172.21.15.49:0/1700442905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9860101420 msgr2=0x7f98601998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985effd700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9860101420 0x7f98601998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985effd700 1 -- 172.21.15.49:0/1700442905 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98601064e0 msgr2=0x7f98601a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985effd700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98601064e0 0x7f98601a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.091+0000 7f985effd700 1 -- 172.21.15.49:0/1700442905 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98601a4e90 con 0x7f9860101dd0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.092+0000 7f985effd700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101dd0 0x7f9860199e10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f985000d950 tx=0x7f985000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.092+0000 7f985cff9700 1 -- 172.21.15.49:0/1700442905 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850016070 con 0x7f9860101dd0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.092+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98601a5060 con 0x7f9860101dd0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.092+0000 7f985cff9700 1 -- 172.21.15.49:0/1700442905 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9850004d10 con 0x7f9860101dd0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.092+0000 7f985cff9700 1 -- 172.21.15.49:0/1700442905 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850005600 con 0x7f9860101dd0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.092+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98601a55a0 con 0x7f9860101dd0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.093+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98601028e0 con 0x7f9860101dd0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.095+0000 7f985cff9700 1 -- 172.21.15.49:0/1700442905 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9850004220 con 0x7f9860101dd0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.096+0000 7f985cff9700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f98480a2920 0x7f98480a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.096+0000 7f985cff9700 1 -- 172.21.15.49:0/1700442905 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f98500ce920 con 0x7f9860101dd0 2024-03-18T00:49:07.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.096+0000 7f985f7fe700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f98480a2920 0x7f98480a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.097+0000 7f985f7fe700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f98480a2920 0x7f98480a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f985400b2e0 tx=0x7f9854005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.101+0000 7f985cff9700 1 -- 172.21.15.49:0/1700442905 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f985001f020 con 0x7f9860101dd0 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.315+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f986006af30 con 0x7f9860101dd0 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.316+0000 7f985cff9700 1 -- 172.21.15.49:0/1700442905 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f985009b840 con 0x7f9860101dd0 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f98480a2920 msgr2=0x7f98480a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f98480a2920 0x7f98480a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f985400b2e0 tx=0x7f9854005960 comp rx=0 tx=0).stop 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101dd0 msgr2=0x7f9860199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101dd0 0x7f9860199e10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f985000d950 tx=0x7f985000dc60 comp rx=0 tx=0).stop 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 shutdown_connections 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f98480a2920 0x7f98480a4de0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9860101420 0x7f98601998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f98601064e0 0x7f98601a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 --2- 172.21.15.49:0/1700442905 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9860101dd0 0x7f9860199e10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:07.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.318+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 >> 172.21.15.49:0/1700442905 conn(0x7f9860078140 msgr2=0x7f98600ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:07.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.319+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 shutdown_connections 2024-03-18T00:49:07.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:07.319+0000 7f9865af0700 1 -- 172.21.15.49:0/1700442905 wait complete. 2024-03-18T00:49:08.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:08 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1700442905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T00:49:08.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:08 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1303741541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T00:49:08.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:08 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/4262411299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T00:49:08.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:08 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1700442905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T00:49:08.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:08 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1303741541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T00:49:08.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:08 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/4262411299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T00:49:08.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:08 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1700442905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T00:49:08.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:08 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1303741541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T00:49:08.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:08 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/4262411299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T00:49:08.476 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612632 got 313532612639 for osd.10 2024-03-18T00:49:08.476 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:08.710 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462957 got 201863462963 for osd.6 2024-03-18T00:49:08.711 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:09.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:09 smithi049 ceph-mon[32130]: pgmap v321: 97 pgs: 1 activating+degraded, 4 peering, 92 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4 op/s; 8/306 objects degraded (2.614%); 140 B/s, 0 objects/s recovering 2024-03-18T00:49:09.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:09 smithi107 ceph-mon[36270]: pgmap v321: 97 pgs: 1 activating+degraded, 4 peering, 92 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4 op/s; 8/306 objects degraded (2.614%); 140 B/s, 0 objects/s recovering 2024-03-18T00:49:09.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:09 smithi106 ceph-mon[35899]: pgmap v321: 97 pgs: 1 activating+degraded, 4 peering, 92 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4 op/s; 8/306 objects degraded (2.614%); 140 B/s, 0 objects/s recovering 2024-03-18T00:49:09.672 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724596 got 167503724599 for osd.5 2024-03-18T00:49:09.672 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:10.061 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888119 got 146028888124 for osd.4 2024-03-18T00:49:10.061 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:10.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:10 smithi049 ceph-mon[32130]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/306 objects degraded (2.614%), 1 pg degraded) 2024-03-18T00:49:10.194 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:10 smithi049 ceph-mon[32130]: Cluster is now healthy 2024-03-18T00:49:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:10 smithi107 ceph-mon[36270]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/306 objects degraded (2.614%), 1 pg degraded) 2024-03-18T00:49:10.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:10 smithi107 ceph-mon[36270]: Cluster is now healthy 2024-03-18T00:49:10.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:10 smithi106 ceph-mon[35899]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/306 objects degraded (2.614%), 1 pg degraded) 2024-03-18T00:49:10.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:10 smithi106 ceph-mon[35899]: Cluster is now healthy 2024-03-18T00:49:11.085 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299433 got 223338299438 for osd.7 2024-03-18T00:49:11.085 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:11.274 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:11 smithi049 ceph-mon[32130]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2 op/s; 154 B/s, 1 objects/s recovering 2024-03-18T00:49:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:11 smithi107 ceph-mon[36270]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2 op/s; 154 B/s, 1 objects/s recovering 2024-03-18T00:49:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:11 smithi106 ceph-mon[35899]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2 op/s; 154 B/s, 1 objects/s recovering 2024-03-18T00:49:11.784 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574923 for osd.1 2024-03-18T00:49:11.784 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:13.220 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:13 smithi049 ceph-mon[32130]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 3 op/s; 131 B/s, 1 objects/s recovering 2024-03-18T00:49:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:13 smithi107 ceph-mon[36270]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 3 op/s; 131 B/s, 1 objects/s recovering 2024-03-18T00:49:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:13 smithi106 ceph-mon[35899]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 3 op/s; 131 B/s, 1 objects/s recovering 2024-03-18T00:49:13.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.881+0000 7f044242e700 1 -- 172.21.15.49:0/4024812972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c06a4c0 msgr2=0x7f043c06a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.881+0000 7f044242e700 1 --2- 172.21.15.49:0/4024812972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c06a4c0 0x7f043c06a920 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0430009990 tx=0x7f0430009ca0 comp rx=0 tx=0).stop 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.881+0000 7f044242e700 1 -- 172.21.15.49:0/4024812972 shutdown_connections 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.881+0000 7f044242e700 1 --2- 172.21.15.49:0/4024812972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f043c1071c0 0x7f043c069f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.881+0000 7f044242e700 1 --2- 172.21.15.49:0/4024812972 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f043c104cd0 0x7f043c105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.881+0000 7f044242e700 1 --2- 172.21.15.49:0/4024812972 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c06a4c0 0x7f043c06a920 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.881+0000 7f044242e700 1 -- 172.21.15.49:0/4024812972 >> 172.21.15.49:0/4024812972 conn(0x7f043c078100 msgr2=0x7f043c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.881+0000 7f044242e700 1 -- 172.21.15.49:0/4024812972 shutdown_connections 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.881+0000 7f044242e700 1 -- 172.21.15.49:0/4024812972 wait complete. 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f044242e700 1 Processor -- start 2024-03-18T00:49:13.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f044242e700 1 -- start start 2024-03-18T00:49:13.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f044242e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f043c06a4c0 0x7f043c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:13.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f044242e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f043c104cd0 0x7f043c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:13.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f044242e700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c1071c0 0x7f043c1a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:13.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f044242e700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f043c118b70 con 0x7f043c1071c0 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f044242e700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f043c1189f0 con 0x7f043c104cd0 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f044242e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f043c118cf0 con 0x7f043c06a4c0 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f04409cb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c1071c0 0x7f043c1a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f04409cb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c1071c0 0x7f043c1a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57268/0 (socket says 172.21.15.49:57268) 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f043b7fe700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f043c104cd0 0x7f043c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f043bfff700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f043c06a4c0 0x7f043c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.882+0000 7f04409cb700 1 -- 172.21.15.49:0/2515026998 learned_addr learned my addr 172.21.15.49:0/2515026998 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f04409cb700 1 -- 172.21.15.49:0/2515026998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f043c06a4c0 msgr2=0x7f043c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f04409cb700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f043c06a4c0 0x7f043c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f04409cb700 1 -- 172.21.15.49:0/2515026998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f043c104cd0 msgr2=0x7f043c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f04409cb700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f043c104cd0 0x7f043c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f04409cb700 1 -- 172.21.15.49:0/2515026998 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043c1a3ff0 con 0x7f043c1071c0 2024-03-18T00:49:13.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f043b7fe700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f043c104cd0 0x7f043c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:13.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f04409cb700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c1071c0 0x7f043c1a38b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f043400ef30 tx=0x7f043400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:13.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f04397fa700 1 -- 172.21.15.49:0/2515026998 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04340092e0 con 0x7f043c1071c0 2024-03-18T00:49:13.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f043c1a4280 con 0x7f043c1071c0 2024-03-18T00:49:13.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f04397fa700 1 -- 172.21.15.49:0/2515026998 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0434004bb0 con 0x7f043c1071c0 2024-03-18T00:49:13.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f04397fa700 1 -- 172.21.15.49:0/2515026998 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0434004280 con 0x7f043c1071c0 2024-03-18T00:49:13.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.883+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f043c1abb60 con 0x7f043c1071c0 2024-03-18T00:49:13.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.884+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f043c068f90 con 0x7f043c1071c0 2024-03-18T00:49:13.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.886+0000 7f04397fa700 1 -- 172.21.15.49:0/2515026998 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f043401b030 con 0x7f043c1071c0 2024-03-18T00:49:13.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.887+0000 7f04397fa700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f04240a29a0 0x7f04240a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:13.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.887+0000 7f043bfff700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f04240a29a0 0x7f04240a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:13.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.887+0000 7f04397fa700 1 -- 172.21.15.49:0/2515026998 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f04340cefe0 con 0x7f043c1071c0 2024-03-18T00:49:13.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.887+0000 7f043bfff700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f04240a29a0 0x7f04240a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f042c007b60 tx=0x7f042c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:13.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:13.890+0000 7f04397fa700 1 -- 172.21.15.49:0/2515026998 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0434022020 con 0x7f043c1071c0 2024-03-18T00:49:14.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.099+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f043c1abd30 con 0x7f043c1071c0 2024-03-18T00:49:14.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.099+0000 7f04397fa700 1 -- 172.21.15.49:0/2515026998 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0434010040 con 0x7f043c1071c0 2024-03-18T00:49:14.099 INFO:teuthology.orchestra.run.smithi049.stdout:77309411400 2024-03-18T00:49:14.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.101+0000 7f8aefece700 1 -- 172.21.15.49:0/315558854 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae8106160 msgr2=0x7f8ae8106540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:14.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.101+0000 7f8aefece700 1 --2- 172.21.15.49:0/315558854 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae8106160 0x7f8ae8106540 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ae40098f0 tx=0x7f8ae4009c00 comp rx=0 tx=0).stop 2024-03-18T00:49:14.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.101+0000 7f8aefece700 1 -- 172.21.15.49:0/315558854 shutdown_connections 2024-03-18T00:49:14.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.101+0000 7f8aefece700 1 --2- 172.21.15.49:0/315558854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ae8101ee0 0x7f8ae810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.101+0000 7f8aefece700 1 --2- 172.21.15.49:0/315558854 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae8106160 0x7f8ae8106540 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.101+0000 7f8aefece700 1 --2- 172.21.15.49:0/315558854 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae8101520 0x7f8ae81019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.101+0000 7f8aefece700 1 -- 172.21.15.49:0/315558854 >> 172.21.15.49:0/315558854 conn(0x7f8ae8077f80 msgr2=0x7f8ae8078390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:14.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.101+0000 7f8aefece700 1 -- 172.21.15.49:0/315558854 shutdown_connections 2024-03-18T00:49:14.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.101+0000 7f8aefece700 1 -- 172.21.15.49:0/315558854 wait complete. 2024-03-18T00:49:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f04240a29a0 msgr2=0x7f04240a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f04240a29a0 0x7f04240a4e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f042c007b60 tx=0x7f042c0073d0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c1071c0 msgr2=0x7f043c1a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c1071c0 0x7f043c1a38b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f043400ef30 tx=0x7f043400c3a0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 shutdown_connections 2024-03-18T00:49:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f04240a29a0 0x7f04240a4e60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f043c06a4c0 0x7f043c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f043c1071c0 0x7f043c1a38b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 --2- 172.21.15.49:0/2515026998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f043c104cd0 0x7f043c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 >> 172.21.15.49:0/2515026998 conn(0x7f043c078100 msgr2=0x7f043c103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:14.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 shutdown_connections 2024-03-18T00:49:14.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f8aefece700 1 Processor -- start 2024-03-18T00:49:14.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f044242e700 1 -- 172.21.15.49:0/2515026998 wait complete. 2024-03-18T00:49:14.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.102+0000 7f8aefece700 1 -- start start 2024-03-18T00:49:14.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aefece700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae8101520 0x7f8ae819d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:14.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aefece700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae8101ee0 0x7f8ae819db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aefece700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ae8106160 0x7f8ae81977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aedc6a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae8101520 0x7f8ae819d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aedc6a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae8101520 0x7f8ae819d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57286/0 (socket says 172.21.15.49:57286) 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aedc6a700 1 -- 172.21.15.49:0/918669293 learned_addr learned my addr 172.21.15.49:0/918669293 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae8111540 con 0x7f8ae8101520 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae81113c0 con 0x7f8ae8101ee0 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae81116c0 con 0x7f8ae8106160 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aee46b700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ae8106160 0x7f8ae81977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aed469700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae8101ee0 0x7f8ae819db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:14.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aee46b700 1 -- 172.21.15.49:0/918669293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae8101ee0 msgr2=0x7f8ae819db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aee46b700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae8101ee0 0x7f8ae819db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aee46b700 1 -- 172.21.15.49:0/918669293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae8101520 msgr2=0x7f8ae819d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aee46b700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae8101520 0x7f8ae819d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.103+0000 7f8aee46b700 1 -- 172.21.15.49:0/918669293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae8197fa0 con 0x7f8ae8106160 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.104+0000 7f8aee46b700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ae8106160 0x7f8ae81977c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ae000efc0 tx=0x7f8ae000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.104+0000 7f8adeffd700 1 -- 172.21.15.49:0/918669293 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae00092e0 con 0x7f8ae8106160 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.104+0000 7f8adeffd700 1 -- 172.21.15.49:0/918669293 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ae0004bb0 con 0x7f8ae8106160 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.104+0000 7f8adeffd700 1 -- 172.21.15.49:0/918669293 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0004280 con 0x7f8ae8106160 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.104+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae8198230 con 0x7f8ae8106160 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.104+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae8069f80 con 0x7f8ae8106160 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.105+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae81028c0 con 0x7f8ae8106160 2024-03-18T00:49:14.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.106+0000 7f8adeffd700 1 -- 172.21.15.49:0/918669293 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8ae001b030 con 0x7f8ae8106160 2024-03-18T00:49:14.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.107+0000 7f8adeffd700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ad40a2920 0x7f8ad40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:14.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.107+0000 7f8aedc6a700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ad40a2920 0x7f8ad40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:14.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.107+0000 7f8adeffd700 1 -- 172.21.15.49:0/918669293 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8ae00ce750 con 0x7f8ae8106160 2024-03-18T00:49:14.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.107+0000 7f8aedc6a700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ad40a2920 0x7f8ad40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4009680 tx=0x7f8ae4019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:14.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.116+0000 7f8adeffd700 1 -- 172.21.15.49:0/918669293 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8ae0022020 con 0x7f8ae8106160 2024-03-18T00:49:14.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.329+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f8ae806a9b0 con 0x7f8ae8106160 2024-03-18T00:49:14.329 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:14 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2515026998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T00:49:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.330+0000 7f8adeffd700 1 -- 172.21.15.49:0/918669293 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8ae0010040 con 0x7f8ae8106160 2024-03-18T00:49:14.330 INFO:teuthology.orchestra.run.smithi049.stdout:292057776164 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ad40a2920 msgr2=0x7f8ad40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ad40a2920 0x7f8ad40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4009680 tx=0x7f8ae4019040 comp rx=0 tx=0).stop 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ae8106160 msgr2=0x7f8ae81977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ae8106160 0x7f8ae81977c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ae000efc0 tx=0x7f8ae000c3a0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 shutdown_connections 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ad40a2920 0x7f8ad40a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8ae8106160 0x7f8ae81977c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8ae8101520 0x7f8ae819d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 --2- 172.21.15.49:0/918669293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8ae8101ee0 0x7f8ae819db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 >> 172.21.15.49:0/918669293 conn(0x7f8ae8077f80 msgr2=0x7f8ae80ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 shutdown_connections 2024-03-18T00:49:14.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:14.332+0000 7f8aefece700 1 -- 172.21.15.49:0/918669293 wait complete. 2024-03-18T00:49:14.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:14 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2515026998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T00:49:14.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:14 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2515026998' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T00:49:15.183 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411394 got 77309411400 for osd.2 2024-03-18T00:49:15.183 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:15 smithi049 ceph-mon[32130]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4 op/s; 119 B/s, 1 objects/s recovering 2024-03-18T00:49:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:15 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/918669293' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T00:49:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:15 smithi107 ceph-mon[36270]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4 op/s; 119 B/s, 1 objects/s recovering 2024-03-18T00:49:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:15 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/918669293' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T00:49:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:15 smithi106 ceph-mon[35899]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4 op/s; 119 B/s, 1 objects/s recovering 2024-03-18T00:49:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:15 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/918669293' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T00:49:15.804 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776157 got 292057776164 for osd.9 2024-03-18T00:49:15.804 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:16.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.259+0000 7f448baac700 1 -- 172.21.15.49:0/3365205142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4484107210 msgr2=0x7f44841075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:16.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.259+0000 7f448baac700 1 --2- 172.21.15.49:0/3365205142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4484107210 0x7f44841075f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4478009990 tx=0x7f4478009ca0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.259+0000 7f448baac700 1 -- 172.21.15.49:0/3365205142 shutdown_connections 2024-03-18T00:49:16.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.259+0000 7f448baac700 1 --2- 172.21.15.49:0/3365205142 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4484101b70 0x7f448410e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.259+0000 7f448baac700 1 --2- 172.21.15.49:0/3365205142 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4484107210 0x7f44841075f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.259+0000 7f448baac700 1 --2- 172.21.15.49:0/3365205142 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44841011d0 0x7f4484101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.259+0000 7f448baac700 1 -- 172.21.15.49:0/3365205142 >> 172.21.15.49:0/3365205142 conn(0x7f4484078100 msgr2=0x7f4484078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:16.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.259+0000 7f448baac700 1 -- 172.21.15.49:0/3365205142 shutdown_connections 2024-03-18T00:49:16.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.259+0000 7f448baac700 1 -- 172.21.15.49:0/3365205142 wait complete. 2024-03-18T00:49:16.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f448baac700 1 Processor -- start 2024-03-18T00:49:16.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f448baac700 1 -- start start 2024-03-18T00:49:16.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f448baac700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44841011d0 0x7f4484197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:16.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f448baac700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4484101b70 0x7f4484197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:16.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f448baac700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4484107210 0x7f44841980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:16.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f448baac700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4484111450 con 0x7f4484107210 2024-03-18T00:49:16.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f448baac700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f44841112d0 con 0x7f44841011d0 2024-03-18T00:49:16.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f448baac700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f44841115d0 con 0x7f4484101b70 2024-03-18T00:49:16.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f4489848700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44841011d0 0x7f4484197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f448a049700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4484107210 0x7f44841980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f4489047700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4484101b70 0x7f4484197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f4489848700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44841011d0 0x7f4484197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:37704/0 (socket says 172.21.15.49:37704) 2024-03-18T00:49:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.260+0000 7f4489848700 1 -- 172.21.15.49:0/133319864 learned_addr learned my addr 172.21.15.49:0/133319864 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f448a049700 1 -- 172.21.15.49:0/133319864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4484101b70 msgr2=0x7f4484197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f448a049700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4484101b70 0x7f4484197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f448a049700 1 -- 172.21.15.49:0/133319864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44841011d0 msgr2=0x7f4484197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:16.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f448a049700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44841011d0 0x7f4484197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f448a049700 1 -- 172.21.15.49:0/133319864 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44841a6eb0 con 0x7f4484107210 2024-03-18T00:49:16.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f4489047700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4484101b70 0x7f4484197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:16.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f4489848700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44841011d0 0x7f4484197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:16.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f448a049700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4484107210 0x7f44841980c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f447c00efc0 tx=0x7f447c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:16.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f4476ffd700 1 -- 172.21.15.49:0/133319864 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c0092e0 con 0x7f4484107210 2024-03-18T00:49:16.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f4476ffd700 1 -- 172.21.15.49:0/133319864 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f447c004bb0 con 0x7f4484107210 2024-03-18T00:49:16.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44841a7140 con 0x7f4484107210 2024-03-18T00:49:16.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f4476ffd700 1 -- 172.21.15.49:0/133319864 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c004280 con 0x7f4484107210 2024-03-18T00:49:16.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.261+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44841a7630 con 0x7f4484107210 2024-03-18T00:49:16.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.262+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4484068f90 con 0x7f4484107210 2024-03-18T00:49:16.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.263+0000 7f4476ffd700 1 -- 172.21.15.49:0/133319864 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f447c01b030 con 0x7f4484107210 2024-03-18T00:49:16.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.264+0000 7f4476ffd700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44700a2920 0x7f44700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:16.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.264+0000 7f4489848700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44700a2920 0x7f44700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:16.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.264+0000 7f4476ffd700 1 -- 172.21.15.49:0/133319864 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f447c0ceff0 con 0x7f4484107210 2024-03-18T00:49:16.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.264+0000 7f4489848700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44700a2920 0x7f44700a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f447800b2e0 tx=0x7f4478009590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:16.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.267+0000 7f4476ffd700 1 -- 172.21.15.49:0/133319864 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f447c022020 con 0x7f4484107210 2024-03-18T00:49:16.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.488+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f448402db70 con 0x7f4484107210 2024-03-18T00:49:16.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.489+0000 7f4476ffd700 1 -- 172.21.15.49:0/133319864 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f447c010040 con 0x7f4484107210 2024-03-18T00:49:16.489 INFO:teuthology.orchestra.run.smithi049.stdout:34359738450 2024-03-18T00:49:16.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44700a2920 msgr2=0x7f44700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:16.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44700a2920 0x7f44700a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f447800b2e0 tx=0x7f4478009590 comp rx=0 tx=0).stop 2024-03-18T00:49:16.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4484107210 msgr2=0x7f44841980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:16.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4484107210 0x7f44841980c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f447c00efc0 tx=0x7f447c00c3a0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 shutdown_connections 2024-03-18T00:49:16.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4484107210 0x7f44841980c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44841011d0 0x7f4484197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44700a2920 0x7f44700a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 --2- 172.21.15.49:0/133319864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4484101b70 0x7f4484197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:16.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 >> 172.21.15.49:0/133319864 conn(0x7f4484078100 msgr2=0x7f44840ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:16.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 shutdown_connections 2024-03-18T00:49:16.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:16.491+0000 7f448baac700 1 -- 172.21.15.49:0/133319864 wait complete. 2024-03-18T00:49:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:17 smithi107 ceph-mon[36270]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5 op/s; 103 B/s, 1 objects/s recovering 2024-03-18T00:49:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:17 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/133319864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T00:49:17.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:17 smithi106 ceph-mon[35899]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5 op/s; 103 B/s, 1 objects/s recovering 2024-03-18T00:49:17.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:17 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/133319864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T00:49:17.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:17 smithi049 ceph-mon[32130]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5 op/s; 103 B/s, 1 objects/s recovering 2024-03-18T00:49:17.495 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:17 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/133319864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T00:49:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 -- 172.21.15.49:0/2081670340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd750107210 msgr2=0x7fd7501075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 --2- 172.21.15.49:0/2081670340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd750107210 0x7fd7501075f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd740009990 tx=0x7fd740009ca0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 -- 172.21.15.49:0/2081670340 shutdown_connections 2024-03-18T00:49:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 --2- 172.21.15.49:0/2081670340 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd750101b70 0x7fd75010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 --2- 172.21.15.49:0/2081670340 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd750107210 0x7fd7501075f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 --2- 172.21.15.49:0/2081670340 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd7501011d0 0x7fd750101630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 -- 172.21.15.49:0/2081670340 >> 172.21.15.49:0/2081670340 conn(0x7fd750078100 msgr2=0x7fd750078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 -- 172.21.15.49:0/2081670340 shutdown_connections 2024-03-18T00:49:17.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 -- 172.21.15.49:0/2081670340 wait complete. 2024-03-18T00:49:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.500+0000 7fd754eeb700 1 Processor -- start 2024-03-18T00:49:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd754eeb700 1 -- start start 2024-03-18T00:49:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd754eeb700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7501011d0 0x7fd7501998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd754eeb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd750101b70 0x7fd750199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd754eeb700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd750107210 0x7fd7501a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd754eeb700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd750111470 con 0x7fd7501011d0 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd754eeb700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd7501112f0 con 0x7fd750107210 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd754eeb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd7501115f0 con 0x7fd750101b70 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74e59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7501011d0 0x7fd7501998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74e59c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7501011d0 0x7fd7501998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57326/0 (socket says 172.21.15.49:57326) 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74e59c700 1 -- 172.21.15.49:0/1343174867 learned_addr learned my addr 172.21.15.49:0/1343174867 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74dd9b700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd750101b70 0x7fd750199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74e59c700 1 -- 172.21.15.49:0/1343174867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd750101b70 msgr2=0x7fd750199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74e59c700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd750101b70 0x7fd750199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74e59c700 1 -- 172.21.15.49:0/1343174867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd750107210 msgr2=0x7fd7501a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74e59c700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd750107210 0x7fd7501a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74e59c700 1 -- 172.21.15.49:0/1343174867 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7501a4e70 con 0x7fd7501011d0 2024-03-18T00:49:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd74e59c700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7501011d0 0x7fd7501998b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd740000c00 tx=0x7fd74000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.501+0000 7fd73f7fe700 1 -- 172.21.15.49:0/1343174867 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd740027070 con 0x7fd7501011d0 2024-03-18T00:49:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.502+0000 7fd73f7fe700 1 -- 172.21.15.49:0/1343174867 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd740003d70 con 0x7fd7501011d0 2024-03-18T00:49:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.502+0000 7fd73f7fe700 1 -- 172.21.15.49:0/1343174867 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd74000f460 con 0x7fd7501011d0 2024-03-18T00:49:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.502+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7501a5040 con 0x7fd7501011d0 2024-03-18T00:49:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.502+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7501a54f0 con 0x7fd7501011d0 2024-03-18T00:49:17.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.503+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd750068f90 con 0x7fd7501011d0 2024-03-18T00:49:17.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.504+0000 7fd73f7fe700 1 -- 172.21.15.49:0/1343174867 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fd74002a070 con 0x7fd7501011d0 2024-03-18T00:49:17.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.504+0000 7fd73f7fe700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd7380a29a0 0x7fd7380a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:17.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.505+0000 7fd73f7fe700 1 -- 172.21.15.49:0/1343174867 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd7400cfbe0 con 0x7fd7501011d0 2024-03-18T00:49:17.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.505+0000 7fd74dd9b700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd7380a29a0 0x7fd7380a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:17.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.505+0000 7fd74dd9b700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd7380a29a0 0x7fd7380a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd7500787e0 tx=0x7fd744009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:17.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.508+0000 7fd73f7fe700 1 -- 172.21.15.49:0/1343174867 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd740013070 con 0x7fd7501011d0 2024-03-18T00:49:17.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.736+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd75002dd30 con 0x7fd7501011d0 2024-03-18T00:49:17.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.737+0000 7fd73f7fe700 1 -- 172.21.15.49:0/1343174867 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd74009cb90 con 0x7fd7501011d0 2024-03-18T00:49:17.737 INFO:teuthology.orchestra.run.smithi049.stdout:111669149763 2024-03-18T00:49:17.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd7380a29a0 msgr2=0x7fd7380a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:17.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd7380a29a0 0x7fd7380a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd7500787e0 tx=0x7fd744009260 comp rx=0 tx=0).stop 2024-03-18T00:49:17.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7501011d0 msgr2=0x7fd7501998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:17.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7501011d0 0x7fd7501998b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd740000c00 tx=0x7fd74000bab0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 shutdown_connections 2024-03-18T00:49:17.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd750107210 0x7fd7501a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd7501011d0 0x7fd7501998b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd7380a29a0 0x7fd7380a4e60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 --2- 172.21.15.49:0/1343174867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd750101b70 0x7fd750199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 >> 172.21.15.49:0/1343174867 conn(0x7fd750078100 msgr2=0x7fd7500fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 shutdown_connections 2024-03-18T00:49:17.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:17.739+0000 7fd754eeb700 1 -- 172.21.15.49:0/1343174867 wait complete. 2024-03-18T00:49:18.337 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738444 got 34359738450 for osd.0 2024-03-18T00:49:18.338 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:18.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:18 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1343174867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T00:49:18.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:18 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1343174867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T00:49:18.470 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:18 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1343174867' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T00:49:18.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.577+0000 7f33b0780700 1 -- 172.21.15.49:0/2436302445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 msgr2=0x7f33a8101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:18.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.577+0000 7f33b0780700 1 --2- 172.21.15.49:0/2436302445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 0x7f33a8101840 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f33a40099c0 tx=0x7f33a4009cd0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.577+0000 7f33b0780700 1 -- 172.21.15.49:0/2436302445 shutdown_connections 2024-03-18T00:49:18.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.577+0000 7f33b0780700 1 --2- 172.21.15.49:0/2436302445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33a8106480 0x7f33a810e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.577+0000 7f33b0780700 1 --2- 172.21.15.49:0/2436302445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 0x7f33a8101840 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.577+0000 7f33b0780700 1 --2- 172.21.15.49:0/2436302445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33a8101e10 0x7f33a8105da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.577+0000 7f33b0780700 1 -- 172.21.15.49:0/2436302445 >> 172.21.15.49:0/2436302445 conn(0x7f33a80772b0 msgr2=0x7f33a80776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:18.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.577+0000 7f33b0780700 1 -- 172.21.15.49:0/2436302445 shutdown_connections 2024-03-18T00:49:18.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.577+0000 7f33b0780700 1 -- 172.21.15.49:0/2436302445 wait complete. 2024-03-18T00:49:18.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33b0780700 1 Processor -- start 2024-03-18T00:49:18.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33b0780700 1 -- start start 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33b0780700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 0x7f33a8102e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33b0780700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33a8101e10 0x7f33a8103380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33b0780700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33a8106480 0x7f33a81038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33b0780700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f33a8111460 con 0x7f33a8101460 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33b0780700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81112e0 con 0x7f33a8101e10 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33b0780700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f33a81115e0 con 0x7f33a8106480 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33ae51c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 0x7f33a8102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33ae51c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 0x7f33a8102e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55178/0 (socket says 172.21.15.49:55178) 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33ae51c700 1 -- 172.21.15.49:0/2878267424 learned_addr learned my addr 172.21.15.49:0/2878267424 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33aed1d700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33a8106480 0x7f33a81038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:18.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.578+0000 7f33add1b700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33a8101e10 0x7f33a8103380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f33ae51c700 1 -- 172.21.15.49:0/2878267424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33a8106480 msgr2=0x7f33a81038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f33ae51c700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33a8106480 0x7f33a81038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f33ae51c700 1 -- 172.21.15.49:0/2878267424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33a8101e10 msgr2=0x7f33a8103380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f33ae51c700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33a8101e10 0x7f33a8103380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f33ae51c700 1 -- 172.21.15.49:0/2878267424 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a81a6f60 con 0x7f33a8101460 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f33add1b700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33a8101e10 0x7f33a8103380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f33ae51c700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 0x7f33a8102e40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f33a4006010 tx=0x7f33a4003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f339f7fe700 1 -- 172.21.15.49:0/2878267424 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4009e60 con 0x7f33a8101460 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f339f7fe700 1 -- 172.21.15.49:0/2878267424 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33a4003680 con 0x7f33a8101460 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f339f7fe700 1 -- 172.21.15.49:0/2878267424 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a40173f0 con 0x7f33a8101460 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a81a71f0 con 0x7f33a8101460 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.579+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a81a76a0 con 0x7f33a8101460 2024-03-18T00:49:18.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.580+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a8068f90 con 0x7f33a8101460 2024-03-18T00:49:18.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.581+0000 7f339f7fe700 1 -- 172.21.15.49:0/2878267424 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f33a400f5d0 con 0x7f33a8101460 2024-03-18T00:49:18.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.582+0000 7f339f7fe700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f33940a29a0 0x7f33940a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:18.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.582+0000 7f339f7fe700 1 -- 172.21.15.49:0/2878267424 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f33a40cf9e0 con 0x7f33a8101460 2024-03-18T00:49:18.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.582+0000 7f33add1b700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f33940a29a0 0x7f33940a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:18.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.583+0000 7f33add1b700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f33940a29a0 0x7f33940a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f33a8107a20 tx=0x7f3398009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:18.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.586+0000 7f339f7fe700 1 -- 172.21.15.49:0/2878267424 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f33a4013070 con 0x7f33a8101460 2024-03-18T00:49:18.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.792+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f33a802db70 con 0x7f33a8101460 2024-03-18T00:49:18.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.793+0000 7f339f7fe700 1 -- 172.21.15.49:0/2878267424 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f33a409c990 con 0x7f33a8101460 2024-03-18T00:49:18.793 INFO:teuthology.orchestra.run.smithi049.stdout:347892351003 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f33940a29a0 msgr2=0x7f33940a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f33940a29a0 0x7f33940a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f33a8107a20 tx=0x7f3398009260 comp rx=0 tx=0).stop 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 msgr2=0x7f33a8102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 0x7f33a8102e40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f33a4006010 tx=0x7f33a4003e00 comp rx=0 tx=0).stop 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 shutdown_connections 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f33940a29a0 0x7f33940a4e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33a8106480 0x7f33a81038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f33a8101460 0x7f33a8102e40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 --2- 172.21.15.49:0/2878267424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f33a8101e10 0x7f33a8103380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 >> 172.21.15.49:0/2878267424 conn(0x7f33a80772b0 msgr2=0x7f33a806ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 shutdown_connections 2024-03-18T00:49:18.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:18.795+0000 7f33b0780700 1 -- 172.21.15.49:0/2878267424 wait complete. 2024-03-18T00:49:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:19 smithi107 ceph-mon[36270]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 5 op/s; 103 B/s, 1 objects/s recovering 2024-03-18T00:49:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:19 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2878267424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T00:49:19.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:19 smithi049 ceph-mon[32130]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 5 op/s; 103 B/s, 1 objects/s recovering 2024-03-18T00:49:19.443 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:19 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2878267424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T00:49:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:19 smithi106 ceph-mon[35899]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 5 op/s; 103 B/s, 1 objects/s recovering 2024-03-18T00:49:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:19 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2878267424' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T00:49:19.472 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149758 got 111669149763 for osd.3 2024-03-18T00:49:19.472 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:19.843 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350997 got 347892351003 for osd.11 2024-03-18T00:49:19.843 DEBUG:teuthology.parallel:result is None 2024-03-18T00:49:19.843 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-18T00:49:19.843 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T00:49:20.133 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:21.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:21 smithi049 ceph-mon[32130]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-03-18T00:49:21.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:21 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:21.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:21 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:21.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:21 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:49:21.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:21 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:49:21.480 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:21 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:21 smithi107 ceph-mon[36270]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-03-18T00:49:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:21 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:21 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:21 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:49:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:21 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:49:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:21 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:21.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:21 smithi106 ceph-mon[35899]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-03-18T00:49:21.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:21 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:21.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:21 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:21.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:21 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:49:21.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:21 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:49:21.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:21 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:49:21.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.736+0000 7f26499c9700 1 -- 172.21.15.49:0/2507387337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2644107690 msgr2=0x7f264410f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:21.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.736+0000 7f26499c9700 1 --2- 172.21.15.49:0/2507387337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2644107690 0x7f264410f210 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f263c00b0d0 tx=0x7f263c00b3e0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.736+0000 7f26499c9700 1 -- 172.21.15.49:0/2507387337 shutdown_connections 2024-03-18T00:49:21.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.736+0000 7f26499c9700 1 --2- 172.21.15.49:0/2507387337 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2644107690 0x7f264410f210 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.736+0000 7f26499c9700 1 --2- 172.21.15.49:0/2507387337 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2644100e70 0x7f2644101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.736+0000 7f26499c9700 1 --2- 172.21.15.49:0/2507387337 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2644101820 0x7f2644106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.736+0000 7f26499c9700 1 -- 172.21.15.49:0/2507387337 >> 172.21.15.49:0/2507387337 conn(0x7f2644076510 msgr2=0x7f2644076920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:21.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.736+0000 7f26499c9700 1 -- 172.21.15.49:0/2507387337 shutdown_connections 2024-03-18T00:49:21.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.736+0000 7f26499c9700 1 -- 172.21.15.49:0/2507387337 wait complete. 2024-03-18T00:49:21.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.737+0000 7f26499c9700 1 Processor -- start 2024-03-18T00:49:21.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.737+0000 7f26499c9700 1 -- start start 2024-03-18T00:49:21.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.737+0000 7f26499c9700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2644100e70 0x7f264419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:21.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.737+0000 7f26499c9700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2644101820 0x7f264419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:21.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.737+0000 7f26499c9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2644107690 0x7f2644198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:21.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.737+0000 7f26499c9700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2644112650 con 0x7f2644100e70 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.737+0000 7f26499c9700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f26441124d0 con 0x7f2644101820 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.737+0000 7f26499c9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f26441127d0 con 0x7f2644107690 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.737+0000 7f2642ffd700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2644100e70 0x7f264419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26437fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2644107690 0x7f2644198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26427fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2644101820 0x7f264419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26437fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2644107690 0x7f2644198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58158/0 (socket says 172.21.15.49:58158) 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26437fe700 1 -- 172.21.15.49:0/123360153 learned_addr learned my addr 172.21.15.49:0/123360153 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26427fc700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2644101820 0x7f264419ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:49438/0 (socket says 172.21.15.49:49438) 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26427fc700 1 -- 172.21.15.49:0/123360153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2644107690 msgr2=0x7f2644198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26427fc700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2644107690 0x7f2644198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26427fc700 1 -- 172.21.15.49:0/123360153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2644100e70 msgr2=0x7f264419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26427fc700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2644100e70 0x7f264419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26427fc700 1 -- 172.21.15.49:0/123360153 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2644199010 con 0x7f2644101820 2024-03-18T00:49:21.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26437fe700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2644107690 0x7f2644198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26427fc700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2644101820 0x7f264419ebf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2644077a40 tx=0x7f263400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26489c7700 1 -- 172.21.15.49:0/123360153 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634004220 con 0x7f2644101820 2024-03-18T00:49:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26489c7700 1 -- 172.21.15.49:0/123360153 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2634004d10 con 0x7f2644101820 2024-03-18T00:49:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26489c7700 1 -- 172.21.15.49:0/123360153 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2634010700 con 0x7f2644101820 2024-03-18T00:49:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2644199300 con 0x7f2644101820 2024-03-18T00:49:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.738+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26441a5510 con 0x7f2644101820 2024-03-18T00:49:21.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.739+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f264406a4c0 con 0x7f2644101820 2024-03-18T00:49:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.740+0000 7f26489c7700 1 -- 172.21.15.49:0/123360153 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f2634004750 con 0x7f2644101820 2024-03-18T00:49:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.741+0000 7f26489c7700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26300a2920 0x7f26300a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.741+0000 7f2642ffd700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26300a2920 0x7f26300a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.741+0000 7f26489c7700 1 -- 172.21.15.49:0/123360153 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f26340ce9a0 con 0x7f2644101820 2024-03-18T00:49:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.741+0000 7f2642ffd700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26300a2920 0x7f26300a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f262c009c60 tx=0x7f262c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:21.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.744+0000 7f26489c7700 1 -- 172.21.15.49:0/123360153 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f263409b860 con 0x7f2644101820 2024-03-18T00:49:21.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.945+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f26440ff290 con 0x7f26300a2920 2024-03-18T00:49:21.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.952+0000 7f26489c7700 1 -- 172.21.15.49:0/123360153 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259044 (secure 0 0 0) 0x7f26440ff290 con 0x7f26300a2920 2024-03-18T00:49:21.957 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26300a2920 msgr2=0x7f26300a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26300a2920 0x7f26300a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f262c009c60 tx=0x7f262c009260 comp rx=0 tx=0).stop 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2644101820 msgr2=0x7f264419ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2644101820 0x7f264419ebf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2644077a40 tx=0x7f263400eca0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 shutdown_connections 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26300a2920 0x7f26300a4de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2644107690 0x7f2644198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2644100e70 0x7f264419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 --2- 172.21.15.49:0/123360153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2644101820 0x7f264419ebf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 >> 172.21.15.49:0/123360153 conn(0x7f2644076510 msgr2=0x7f26440feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 shutdown_connections 2024-03-18T00:49:21.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:21.955+0000 7f26499c9700 1 -- 172.21.15.49:0/123360153 wait complete. 2024-03-18T00:49:21.960 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-03-18T00:49:22.649 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-03-18T00:49:21.478896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658461,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":784,"num_read_kb":540,"num_write":419,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181973,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":344,"ondisk_log_size":344,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356868,"kb_used_data":11796,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584956,"statfs":{"total":1151940427776,"available":1151574994944,"internally_reserved":0,"allocated":12079104,"data_stored":7481559,"data_compressed":183621,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003286"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.085235+0000","last_change":"2024-03-18T00:48:06.536034+0000","last_active":"2024-03-18T00:49:03.085235+0000","last_peered":"2024-03-18T00:49:03.085235+0000","last_clean":"2024-03-18T00:49:03.085235+0000","last_became_active":"2024-03-18T00:48:06.535694+0000","last_became_peered":"2024-03-18T00:48:06.535694+0000","last_unstale":"2024-03-18T00:49:03.085235+0000","last_undegraded":"2024-03-18T00:49:03.085235+0000","last_fullsized":"2024-03-18T00:49:03.085235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:24:33.785669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086935+0000","last_change":"2024-03-18T00:48:06.536741+0000","last_active":"2024-03-18T00:49:03.086935+0000","last_peered":"2024-03-18T00:49:03.086935+0000","last_clean":"2024-03-18T00:49:03.086935+0000","last_became_active":"2024-03-18T00:48:06.535865+0000","last_became_peered":"2024-03-18T00:48:06.535865+0000","last_unstale":"2024-03-18T00:49:03.086935+0000","last_undegraded":"2024-03-18T00:49:03.086935+0000","last_fullsized":"2024-03-18T00:49:03.086935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:58.104983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086845+0000","last_change":"2024-03-18T00:48:06.537162+0000","last_active":"2024-03-18T00:49:03.086845+0000","last_peered":"2024-03-18T00:49:03.086845+0000","last_clean":"2024-03-18T00:49:03.086845+0000","last_became_active":"2024-03-18T00:48:06.536835+0000","last_became_peered":"2024-03-18T00:48:06.536835+0000","last_unstale":"2024-03-18T00:49:03.086845+0000","last_undegraded":"2024-03-18T00:49:03.086845+0000","last_fullsized":"2024-03-18T00:49:03.086845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:15:08.519180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.404142+0000","last_change":"2024-03-18T00:48:06.540084+0000","last_active":"2024-03-18T00:49:03.404142+0000","last_peered":"2024-03-18T00:49:03.404142+0000","last_clean":"2024-03-18T00:49:03.404142+0000","last_became_active":"2024-03-18T00:48:06.539736+0000","last_became_peered":"2024-03-18T00:48:06.539736+0000","last_unstale":"2024-03-18T00:49:03.404142+0000","last_undegraded":"2024-03-18T00:49:03.404142+0000","last_fullsized":"2024-03-18T00:49:03.404142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:37:51.748603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087335+0000","last_change":"2024-03-18T00:48:06.540022+0000","last_active":"2024-03-18T00:49:03.087335+0000","last_peered":"2024-03-18T00:49:03.087335+0000","last_clean":"2024-03-18T00:49:03.087335+0000","last_became_active":"2024-03-18T00:48:06.539665+0000","last_became_peered":"2024-03-18T00:48:06.539665+0000","last_unstale":"2024-03-18T00:49:03.087335+0000","last_undegraded":"2024-03-18T00:49:03.087335+0000","last_fullsized":"2024-03-18T00:49:03.087335+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:55:52.448279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088241+0000","last_change":"2024-03-18T00:48:06.538672+0000","last_active":"2024-03-18T00:49:03.088241+0000","last_peered":"2024-03-18T00:49:03.088241+0000","last_clean":"2024-03-18T00:49:03.088241+0000","last_became_active":"2024-03-18T00:48:06.538189+0000","last_became_peered":"2024-03-18T00:48:06.538189+0000","last_unstale":"2024-03-18T00:49:03.088241+0000","last_undegraded":"2024-03-18T00:49:03.088241+0000","last_fullsized":"2024-03-18T00:49:03.088241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:09:04.672255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086945+0000","last_change":"2024-03-18T00:48:06.546417+0000","last_active":"2024-03-18T00:49:03.086945+0000","last_peered":"2024-03-18T00:49:03.086945+0000","last_clean":"2024-03-18T00:49:03.086945+0000","last_became_active":"2024-03-18T00:48:06.545759+0000","last_became_peered":"2024-03-18T00:48:06.545759+0000","last_unstale":"2024-03-18T00:49:03.086945+0000","last_undegraded":"2024-03-18T00:49:03.086945+0000","last_fullsized":"2024-03-18T00:49:03.086945+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:56.759028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325436+0000","last_change":"2024-03-18T00:48:06.545875+0000","last_active":"2024-03-18T00:49:03.325436+0000","last_peered":"2024-03-18T00:49:03.325436+0000","last_clean":"2024-03-18T00:49:03.325436+0000","last_became_active":"2024-03-18T00:48:06.545316+0000","last_became_peered":"2024-03-18T00:48:06.545316+0000","last_unstale":"2024-03-18T00:49:03.325436+0000","last_undegraded":"2024-03-18T00:49:03.325436+0000","last_fullsized":"2024-03-18T00:49:03.325436+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:22:40.998671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089525+0000","last_change":"2024-03-18T00:48:06.533947+0000","last_active":"2024-03-18T00:49:03.089525+0000","last_peered":"2024-03-18T00:49:03.089525+0000","last_clean":"2024-03-18T00:49:03.089525+0000","last_became_active":"2024-03-18T00:48:06.533615+0000","last_became_peered":"2024-03-18T00:48:06.533615+0000","last_unstale":"2024-03-18T00:49:03.089525+0000","last_undegraded":"2024-03-18T00:49:03.089525+0000","last_fullsized":"2024-03-18T00:49:03.089525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:32:16.750269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088372+0000","last_change":"2024-03-18T00:48:06.537614+0000","last_active":"2024-03-18T00:49:03.088372+0000","last_peered":"2024-03-18T00:49:03.088372+0000","last_clean":"2024-03-18T00:49:03.088372+0000","last_became_active":"2024-03-18T00:48:06.536617+0000","last_became_peered":"2024-03-18T00:48:06.536617+0000","last_unstale":"2024-03-18T00:49:03.088372+0000","last_undegraded":"2024-03-18T00:49:03.088372+0000","last_fullsized":"2024-03-18T00:49:03.088372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:52.222603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087412+0000","last_change":"2024-03-18T00:48:06.540363+0000","last_active":"2024-03-18T00:49:03.087412+0000","last_peered":"2024-03-18T00:49:03.087412+0000","last_clean":"2024-03-18T00:49:03.087412+0000","last_became_active":"2024-03-18T00:48:06.540011+0000","last_became_peered":"2024-03-18T00:48:06.540011+0000","last_unstale":"2024-03-18T00:49:03.087412+0000","last_undegraded":"2024-03-18T00:49:03.087412+0000","last_fullsized":"2024-03-18T00:49:03.087412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:23:10.034444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.404019+0000","last_change":"2024-03-18T00:48:06.532611+0000","last_active":"2024-03-18T00:49:03.404019+0000","last_peered":"2024-03-18T00:49:03.404019+0000","last_clean":"2024-03-18T00:49:03.404019+0000","last_became_active":"2024-03-18T00:48:06.532301+0000","last_became_peered":"2024-03-18T00:48:06.532301+0000","last_unstale":"2024-03-18T00:49:03.404019+0000","last_undegraded":"2024-03-18T00:49:03.404019+0000","last_fullsized":"2024-03-18T00:49:03.404019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:40.795105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088078+0000","last_change":"2024-03-18T00:48:06.531993+0000","last_active":"2024-03-18T00:49:03.088078+0000","last_peered":"2024-03-18T00:49:03.088078+0000","last_clean":"2024-03-18T00:49:03.088078+0000","last_became_active":"2024-03-18T00:48:06.531677+0000","last_became_peered":"2024-03-18T00:48:06.531677+0000","last_unstale":"2024-03-18T00:49:03.088078+0000","last_undegraded":"2024-03-18T00:49:03.088078+0000","last_fullsized":"2024-03-18T00:49:03.088078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:48:24.865299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087046+0000","last_change":"2024-03-18T00:48:06.536830+0000","last_active":"2024-03-18T00:49:03.087046+0000","last_peered":"2024-03-18T00:49:03.087046+0000","last_clean":"2024-03-18T00:49:03.087046+0000","last_became_active":"2024-03-18T00:48:06.535908+0000","last_became_peered":"2024-03-18T00:48:06.535908+0000","last_unstale":"2024-03-18T00:49:03.087046+0000","last_undegraded":"2024-03-18T00:49:03.087046+0000","last_fullsized":"2024-03-18T00:49:03.087046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:23:26.155130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086601+0000","last_change":"2024-03-18T00:48:06.543641+0000","last_active":"2024-03-18T00:49:03.086601+0000","last_peered":"2024-03-18T00:49:03.086601+0000","last_clean":"2024-03-18T00:49:03.086601+0000","last_became_active":"2024-03-18T00:48:06.543293+0000","last_became_peered":"2024-03-18T00:48:06.543293+0000","last_unstale":"2024-03-18T00:49:03.086601+0000","last_undegraded":"2024-03-18T00:49:03.086601+0000","last_fullsized":"2024-03-18T00:49:03.086601+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:52.406255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089103+0000","last_change":"2024-03-18T00:48:06.544732+0000","last_active":"2024-03-18T00:49:03.089103+0000","last_peered":"2024-03-18T00:49:03.089103+0000","last_clean":"2024-03-18T00:49:03.089103+0000","last_became_active":"2024-03-18T00:48:06.544400+0000","last_became_peered":"2024-03-18T00:48:06.544400+0000","last_unstale":"2024-03-18T00:49:03.089103+0000","last_undegraded":"2024-03-18T00:49:03.089103+0000","last_fullsized":"2024-03-18T00:49:03.089103+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:23:39.655434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.085669+0000","last_change":"2024-03-18T00:48:06.536595+0000","last_active":"2024-03-18T00:49:03.085669+0000","last_peered":"2024-03-18T00:49:03.085669+0000","last_clean":"2024-03-18T00:49:03.085669+0000","last_became_active":"2024-03-18T00:48:06.535874+0000","last_became_peered":"2024-03-18T00:48:06.535874+0000","last_unstale":"2024-03-18T00:49:03.085669+0000","last_undegraded":"2024-03-18T00:49:03.085669+0000","last_fullsized":"2024-03-18T00:49:03.085669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:55.603614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084750+0000","last_change":"2024-03-18T00:48:06.546676+0000","last_active":"2024-03-18T00:49:03.084750+0000","last_peered":"2024-03-18T00:49:03.084750+0000","last_clean":"2024-03-18T00:49:03.084750+0000","last_became_active":"2024-03-18T00:48:06.546356+0000","last_became_peered":"2024-03-18T00:48:06.546356+0000","last_unstale":"2024-03-18T00:49:03.084750+0000","last_undegraded":"2024-03-18T00:49:03.084750+0000","last_fullsized":"2024-03-18T00:49:03.084750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:04:35.467415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086966+0000","last_change":"2024-03-18T00:48:06.532184+0000","last_active":"2024-03-18T00:49:03.086966+0000","last_peered":"2024-03-18T00:49:03.086966+0000","last_clean":"2024-03-18T00:49:03.086966+0000","last_became_active":"2024-03-18T00:48:06.531816+0000","last_became_peered":"2024-03-18T00:48:06.531816+0000","last_unstale":"2024-03-18T00:49:03.086966+0000","last_undegraded":"2024-03-18T00:49:03.086966+0000","last_fullsized":"2024-03-18T00:49:03.086966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:09:03.113024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":101,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:18.111759+0000","last_change":"2024-03-18T00:48:05.530213+0000","last_active":"2024-03-18T00:49:18.111759+0000","last_peered":"2024-03-18T00:49:18.111759+0000","last_clean":"2024-03-18T00:49:18.111759+0000","last_became_active":"2024-03-18T00:48:05.529897+0000","last_became_peered":"2024-03-18T00:48:05.529897+0000","last_unstale":"2024-03-18T00:49:18.111759+0000","last_undegraded":"2024-03-18T00:49:18.111759+0000","last_fullsized":"2024-03-18T00:49:18.111759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:49.003362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325786+0000","last_change":"2024-03-18T00:48:06.544437+0000","last_active":"2024-03-18T00:49:03.325786+0000","last_peered":"2024-03-18T00:49:03.325786+0000","last_clean":"2024-03-18T00:49:03.325786+0000","last_became_active":"2024-03-18T00:48:06.544125+0000","last_became_peered":"2024-03-18T00:48:06.544125+0000","last_unstale":"2024-03-18T00:49:03.325786+0000","last_undegraded":"2024-03-18T00:49:03.325786+0000","last_fullsized":"2024-03-18T00:49:03.325786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:56:42.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325937+0000","last_change":"2024-03-18T00:48:05.536249+0000","last_active":"2024-03-18T00:49:03.325937+0000","last_peered":"2024-03-18T00:49:03.325937+0000","last_clean":"2024-03-18T00:49:03.325937+0000","last_became_active":"2024-03-18T00:48:05.535954+0000","last_became_peered":"2024-03-18T00:48:05.535954+0000","last_unstale":"2024-03-18T00:49:03.325937+0000","last_undegraded":"2024-03-18T00:49:03.325937+0000","last_fullsized":"2024-03-18T00:49:03.325937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:18:52.341453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086984+0000","last_change":"2024-03-18T00:48:06.541377+0000","last_active":"2024-03-18T00:49:03.086984+0000","last_peered":"2024-03-18T00:49:03.086984+0000","last_clean":"2024-03-18T00:49:03.086984+0000","last_became_active":"2024-03-18T00:48:06.540937+0000","last_became_peered":"2024-03-18T00:48:06.540937+0000","last_unstale":"2024-03-18T00:49:03.086984+0000","last_undegraded":"2024-03-18T00:49:03.086984+0000","last_fullsized":"2024-03-18T00:49:03.086984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:51:55.268517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088568+0000","last_change":"2024-03-18T00:48:05.534372+0000","last_active":"2024-03-18T00:49:03.088568+0000","last_peered":"2024-03-18T00:49:03.088568+0000","last_clean":"2024-03-18T00:49:03.088568+0000","last_became_active":"2024-03-18T00:48:05.534034+0000","last_became_peered":"2024-03-18T00:48:05.534034+0000","last_unstale":"2024-03-18T00:49:03.088568+0000","last_undegraded":"2024-03-18T00:49:03.088568+0000","last_fullsized":"2024-03-18T00:49:03.088568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:58:55.013167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089575+0000","last_change":"2024-03-18T00:48:06.527276+0000","last_active":"2024-03-18T00:49:03.089575+0000","last_peered":"2024-03-18T00:49:03.089575+0000","last_clean":"2024-03-18T00:49:03.089575+0000","last_became_active":"2024-03-18T00:48:06.526904+0000","last_became_peered":"2024-03-18T00:48:06.526904+0000","last_unstale":"2024-03-18T00:49:03.089575+0000","last_undegraded":"2024-03-18T00:49:03.089575+0000","last_fullsized":"2024-03-18T00:49:03.089575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:16:10.731256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087607+0000","last_change":"2024-03-18T00:48:05.528188+0000","last_active":"2024-03-18T00:49:03.087607+0000","last_peered":"2024-03-18T00:49:03.087607+0000","last_clean":"2024-03-18T00:49:03.087607+0000","last_became_active":"2024-03-18T00:48:05.527788+0000","last_became_peered":"2024-03-18T00:48:05.527788+0000","last_unstale":"2024-03-18T00:49:03.087607+0000","last_undegraded":"2024-03-18T00:49:03.087607+0000","last_fullsized":"2024-03-18T00:49:03.087607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:47.939771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.092920+0000","last_change":"2024-03-18T00:49:03.092920+0000","last_active":"2024-03-18T00:49:03.092920+0000","last_peered":"2024-03-18T00:49:03.092920+0000","last_clean":"2024-03-18T00:49:03.092920+0000","last_became_active":"2024-03-18T00:49:03.092569+0000","last_became_peered":"2024-03-18T00:49:03.092569+0000","last_unstale":"2024-03-18T00:49:03.092920+0000","last_undegraded":"2024-03-18T00:49:03.092920+0000","last_fullsized":"2024-03-18T00:49:03.092920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:15:02.246688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:15.105842+0000","last_change":"2024-03-18T00:49:03.100514+0000","last_active":"2024-03-18T00:49:15.105842+0000","last_peered":"2024-03-18T00:49:15.105842+0000","last_clean":"2024-03-18T00:49:15.105842+0000","last_became_active":"2024-03-18T00:49:03.090898+0000","last_became_peered":"2024-03-18T00:49:03.090898+0000","last_unstale":"2024-03-18T00:49:15.105842+0000","last_undegraded":"2024-03-18T00:49:15.105842+0000","last_fullsized":"2024-03-18T00:49:15.105842+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:19:39.943819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087085+0000","last_change":"2024-03-18T00:48:06.537405+0000","last_active":"2024-03-18T00:49:03.087085+0000","last_peered":"2024-03-18T00:49:03.087085+0000","last_clean":"2024-03-18T00:49:03.087085+0000","last_became_active":"2024-03-18T00:48:06.536919+0000","last_became_peered":"2024-03-18T00:48:06.536919+0000","last_unstale":"2024-03-18T00:49:03.087085+0000","last_undegraded":"2024-03-18T00:49:03.087085+0000","last_fullsized":"2024-03-18T00:49:03.087085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:31:59.863571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088122+0000","last_change":"2024-03-18T00:48:05.528588+0000","last_active":"2024-03-18T00:49:03.088122+0000","last_peered":"2024-03-18T00:49:03.088122+0000","last_clean":"2024-03-18T00:49:03.088122+0000","last_became_active":"2024-03-18T00:48:05.528263+0000","last_became_peered":"2024-03-18T00:48:05.528263+0000","last_unstale":"2024-03-18T00:49:03.088122+0000","last_undegraded":"2024-03-18T00:49:03.088122+0000","last_fullsized":"2024-03-18T00:49:03.088122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:18:10.283278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088860+0000","last_change":"2024-03-18T00:48:06.546323+0000","last_active":"2024-03-18T00:49:03.088860+0000","last_peered":"2024-03-18T00:49:03.088860+0000","last_clean":"2024-03-18T00:49:03.088860+0000","last_became_active":"2024-03-18T00:48:06.545956+0000","last_became_peered":"2024-03-18T00:48:06.545956+0000","last_unstale":"2024-03-18T00:49:03.088860+0000","last_undegraded":"2024-03-18T00:49:03.088860+0000","last_fullsized":"2024-03-18T00:49:03.088860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:32.618724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087983+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:49:03.087983+0000","last_peered":"2024-03-18T00:49:03.087983+0000","last_clean":"2024-03-18T00:49:03.087983+0000","last_became_active":"2024-03-18T00:48:05.525590+0000","last_became_peered":"2024-03-18T00:48:05.525590+0000","last_unstale":"2024-03-18T00:49:03.087983+0000","last_undegraded":"2024-03-18T00:49:03.087983+0000","last_fullsized":"2024-03-18T00:49:03.087983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:19.357331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325522+0000","last_change":"2024-03-18T00:48:06.538798+0000","last_active":"2024-03-18T00:49:03.325522+0000","last_peered":"2024-03-18T00:49:03.325522+0000","last_clean":"2024-03-18T00:49:03.325522+0000","last_became_active":"2024-03-18T00:48:06.538457+0000","last_became_peered":"2024-03-18T00:48:06.538457+0000","last_unstale":"2024-03-18T00:49:03.325522+0000","last_undegraded":"2024-03-18T00:49:03.325522+0000","last_fullsized":"2024-03-18T00:49:03.325522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:10.738683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088047+0000","last_change":"2024-03-18T00:48:05.528094+0000","last_active":"2024-03-18T00:49:03.088047+0000","last_peered":"2024-03-18T00:49:03.088047+0000","last_clean":"2024-03-18T00:49:03.088047+0000","last_became_active":"2024-03-18T00:48:05.527730+0000","last_became_peered":"2024-03-18T00:48:05.527730+0000","last_unstale":"2024-03-18T00:49:03.088047+0000","last_undegraded":"2024-03-18T00:49:03.088047+0000","last_fullsized":"2024-03-18T00:49:03.088047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:48:27.707505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084567+0000","last_change":"2024-03-18T00:48:06.548676+0000","last_active":"2024-03-18T00:49:03.084567+0000","last_peered":"2024-03-18T00:49:03.084567+0000","last_clean":"2024-03-18T00:49:03.084567+0000","last_became_active":"2024-03-18T00:48:06.547693+0000","last_became_peered":"2024-03-18T00:48:06.547693+0000","last_unstale":"2024-03-18T00:49:03.084567+0000","last_undegraded":"2024-03-18T00:49:03.084567+0000","last_fullsized":"2024-03-18T00:49:03.084567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:13.849175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":161,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:20.102318+0000","last_change":"2024-03-18T00:48:05.523639+0000","last_active":"2024-03-18T00:49:20.102318+0000","last_peered":"2024-03-18T00:49:20.102318+0000","last_clean":"2024-03-18T00:49:20.102318+0000","last_became_active":"2024-03-18T00:48:05.523269+0000","last_became_peered":"2024-03-18T00:48:05.523269+0000","last_unstale":"2024-03-18T00:49:20.102318+0000","last_undegraded":"2024-03-18T00:49:20.102318+0000","last_fullsized":"2024-03-18T00:49:20.102318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:39.961247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089528+0000","last_change":"2024-03-18T00:48:06.537786+0000","last_active":"2024-03-18T00:49:03.089528+0000","last_peered":"2024-03-18T00:49:03.089528+0000","last_clean":"2024-03-18T00:49:03.089528+0000","last_became_active":"2024-03-18T00:48:06.537488+0000","last_became_peered":"2024-03-18T00:48:06.537488+0000","last_unstale":"2024-03-18T00:49:03.089528+0000","last_undegraded":"2024-03-18T00:49:03.089528+0000","last_fullsized":"2024-03-18T00:49:03.089528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:45:02.095610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.096001+0000","last_change":"2024-03-18T00:49:03.096001+0000","last_active":"2024-03-18T00:49:03.096001+0000","last_peered":"2024-03-18T00:49:03.096001+0000","last_clean":"2024-03-18T00:49:03.096001+0000","last_became_active":"2024-03-18T00:49:03.090868+0000","last_became_peered":"2024-03-18T00:49:03.090868+0000","last_unstale":"2024-03-18T00:49:03.096001+0000","last_undegraded":"2024-03-18T00:49:03.096001+0000","last_fullsized":"2024-03-18T00:49:03.096001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:03.930010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086987+0000","last_change":"2024-03-18T00:48:06.545416+0000","last_active":"2024-03-18T00:49:03.086987+0000","last_peered":"2024-03-18T00:49:03.086987+0000","last_clean":"2024-03-18T00:49:03.086987+0000","last_became_active":"2024-03-18T00:48:06.545087+0000","last_became_peered":"2024-03-18T00:48:06.545087+0000","last_unstale":"2024-03-18T00:49:03.086987+0000","last_undegraded":"2024-03-18T00:49:03.086987+0000","last_fullsized":"2024-03-18T00:49:03.086987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:02:25.309990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089159+0000","last_change":"2024-03-18T00:48:05.533088+0000","last_active":"2024-03-18T00:49:03.089159+0000","last_peered":"2024-03-18T00:49:03.089159+0000","last_clean":"2024-03-18T00:49:03.089159+0000","last_became_active":"2024-03-18T00:48:05.532308+0000","last_became_peered":"2024-03-18T00:48:05.532308+0000","last_unstale":"2024-03-18T00:49:03.089159+0000","last_undegraded":"2024-03-18T00:49:03.089159+0000","last_fullsized":"2024-03-18T00:49:03.089159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:35.071283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.092567+0000","last_change":"2024-03-18T00:49:03.092567+0000","last_active":"2024-03-18T00:49:03.092567+0000","last_peered":"2024-03-18T00:49:03.092567+0000","last_clean":"2024-03-18T00:49:03.092567+0000","last_became_active":"2024-03-18T00:49:03.092297+0000","last_became_peered":"2024-03-18T00:49:03.092297+0000","last_unstale":"2024-03-18T00:49:03.092567+0000","last_undegraded":"2024-03-18T00:49:03.092567+0000","last_fullsized":"2024-03-18T00:49:03.092567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:44.567584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'30","reported_seq":193,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:20.107887+0000","last_change":"2024-03-18T00:48:05.527557+0000","last_active":"2024-03-18T00:49:20.107887+0000","last_peered":"2024-03-18T00:49:20.107887+0000","last_clean":"2024-03-18T00:49:20.107887+0000","last_became_active":"2024-03-18T00:48:05.527167+0000","last_became_peered":"2024-03-18T00:48:05.527167+0000","last_unstale":"2024-03-18T00:49:20.107887+0000","last_undegraded":"2024-03-18T00:49:20.107887+0000","last_fullsized":"2024-03-18T00:49:20.107887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:02:04.105091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":144,"num_read_kb":132,"num_write":32,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088401+0000","last_change":"2024-03-18T00:48:06.543153+0000","last_active":"2024-03-18T00:49:03.088401+0000","last_peered":"2024-03-18T00:49:03.088401+0000","last_clean":"2024-03-18T00:49:03.088401+0000","last_became_active":"2024-03-18T00:48:06.541353+0000","last_became_peered":"2024-03-18T00:48:06.541353+0000","last_unstale":"2024-03-18T00:49:03.088401+0000","last_undegraded":"2024-03-18T00:49:03.088401+0000","last_fullsized":"2024-03-18T00:49:03.088401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:35.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086981+0000","last_change":"2024-03-18T00:48:05.530560+0000","last_active":"2024-03-18T00:49:03.086981+0000","last_peered":"2024-03-18T00:49:03.086981+0000","last_clean":"2024-03-18T00:49:03.086981+0000","last_became_active":"2024-03-18T00:48:05.528858+0000","last_became_peered":"2024-03-18T00:48:05.528858+0000","last_unstale":"2024-03-18T00:49:03.086981+0000","last_undegraded":"2024-03-18T00:49:03.086981+0000","last_fullsized":"2024-03-18T00:49:03.086981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:03:01.496919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.090951+0000","last_change":"2024-03-18T00:49:03.090951+0000","last_active":"2024-03-18T00:49:03.090951+0000","last_peered":"2024-03-18T00:49:03.090951+0000","last_clean":"2024-03-18T00:49:03.090951+0000","last_became_active":"2024-03-18T00:49:03.090578+0000","last_became_peered":"2024-03-18T00:49:03.090578+0000","last_unstale":"2024-03-18T00:49:03.090951+0000","last_undegraded":"2024-03-18T00:49:03.090951+0000","last_fullsized":"2024-03-18T00:49:03.090951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:04:12.703545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088756+0000","last_change":"2024-03-18T00:48:05.527902+0000","last_active":"2024-03-18T00:49:03.088756+0000","last_peered":"2024-03-18T00:49:03.088756+0000","last_clean":"2024-03-18T00:49:03.088756+0000","last_became_active":"2024-03-18T00:48:05.527495+0000","last_became_peered":"2024-03-18T00:48:05.527495+0000","last_unstale":"2024-03-18T00:49:03.088756+0000","last_undegraded":"2024-03-18T00:49:03.088756+0000","last_fullsized":"2024-03-18T00:49:03.088756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:25.504652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087171+0000","last_change":"2024-03-18T00:48:06.550413+0000","last_active":"2024-03-18T00:49:03.087171+0000","last_peered":"2024-03-18T00:49:03.087171+0000","last_clean":"2024-03-18T00:49:03.087171+0000","last_became_active":"2024-03-18T00:48:06.547727+0000","last_became_peered":"2024-03-18T00:48:06.547727+0000","last_unstale":"2024-03-18T00:49:03.087171+0000","last_undegraded":"2024-03-18T00:49:03.087171+0000","last_fullsized":"2024-03-18T00:49:03.087171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:36:48.878417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:20.098103+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:49:20.098103+0000","last_peered":"2024-03-18T00:49:20.098103+0000","last_clean":"2024-03-18T00:49:20.098103+0000","last_became_active":"2024-03-18T00:48:05.525607+0000","last_became_peered":"2024-03-18T00:48:05.525607+0000","last_unstale":"2024-03-18T00:49:20.098103+0000","last_undegraded":"2024-03-18T00:49:20.098103+0000","last_fullsized":"2024-03-18T00:49:20.098103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:32.457447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.091221+0000","last_change":"2024-03-18T00:49:03.091221+0000","last_active":"2024-03-18T00:49:03.091221+0000","last_peered":"2024-03-18T00:49:03.091221+0000","last_clean":"2024-03-18T00:49:03.091221+0000","last_became_active":"2024-03-18T00:49:03.090559+0000","last_became_peered":"2024-03-18T00:49:03.090559+0000","last_unstale":"2024-03-18T00:49:03.091221+0000","last_undegraded":"2024-03-18T00:49:03.091221+0000","last_fullsized":"2024-03-18T00:49:03.091221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:02:54.890187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.417640+0000","last_change":"2024-03-18T00:49:03.417640+0000","last_active":"2024-03-18T00:49:03.417640+0000","last_peered":"2024-03-18T00:49:03.417640+0000","last_clean":"2024-03-18T00:49:03.417640+0000","last_became_active":"2024-03-18T00:49:03.405217+0000","last_became_peered":"2024-03-18T00:49:03.405217+0000","last_unstale":"2024-03-18T00:49:03.417640+0000","last_undegraded":"2024-03-18T00:49:03.417640+0000","last_fullsized":"2024-03-18T00:49:03.417640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:00:13.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088573+0000","last_change":"2024-03-18T00:48:06.544084+0000","last_active":"2024-03-18T00:49:03.088573+0000","last_peered":"2024-03-18T00:49:03.088573+0000","last_clean":"2024-03-18T00:49:03.088573+0000","last_became_active":"2024-03-18T00:48:06.543419+0000","last_became_peered":"2024-03-18T00:48:06.543419+0000","last_unstale":"2024-03-18T00:49:03.088573+0000","last_undegraded":"2024-03-18T00:49:03.088573+0000","last_fullsized":"2024-03-18T00:49:03.088573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:15:28.527473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087635+0000","last_change":"2024-03-18T00:48:05.528347+0000","last_active":"2024-03-18T00:49:03.087635+0000","last_peered":"2024-03-18T00:49:03.087635+0000","last_clean":"2024-03-18T00:49:03.087635+0000","last_became_active":"2024-03-18T00:48:05.527761+0000","last_became_peered":"2024-03-18T00:48:05.527761+0000","last_unstale":"2024-03-18T00:49:03.087635+0000","last_undegraded":"2024-03-18T00:49:03.087635+0000","last_fullsized":"2024-03-18T00:49:03.087635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:36:58.562495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.085039+0000","last_change":"2024-03-18T00:48:06.541862+0000","last_active":"2024-03-18T00:49:03.085039+0000","last_peered":"2024-03-18T00:49:03.085039+0000","last_clean":"2024-03-18T00:49:03.085039+0000","last_became_active":"2024-03-18T00:48:06.541488+0000","last_became_peered":"2024-03-18T00:48:06.541488+0000","last_unstale":"2024-03-18T00:49:03.085039+0000","last_undegraded":"2024-03-18T00:49:03.085039+0000","last_fullsized":"2024-03-18T00:49:03.085039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:47:40.724244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084618+0000","last_change":"2024-03-18T00:48:05.535498+0000","last_active":"2024-03-18T00:49:03.084618+0000","last_peered":"2024-03-18T00:49:03.084618+0000","last_clean":"2024-03-18T00:49:03.084618+0000","last_became_active":"2024-03-18T00:48:05.534907+0000","last_became_peered":"2024-03-18T00:48:05.534907+0000","last_unstale":"2024-03-18T00:49:03.084618+0000","last_undegraded":"2024-03-18T00:49:03.084618+0000","last_fullsized":"2024-03-18T00:49:03.084618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:47:50.196929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088336+0000","last_change":"2024-03-18T00:48:06.534423+0000","last_active":"2024-03-18T00:49:03.088336+0000","last_peered":"2024-03-18T00:49:03.088336+0000","last_clean":"2024-03-18T00:49:03.088336+0000","last_became_active":"2024-03-18T00:48:06.534091+0000","last_became_peered":"2024-03-18T00:48:06.534091+0000","last_unstale":"2024-03-18T00:49:03.088336+0000","last_undegraded":"2024-03-18T00:49:03.088336+0000","last_fullsized":"2024-03-18T00:49:03.088336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:39:08.755634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:16.110279+0000","last_change":"2024-03-18T00:48:05.532891+0000","last_active":"2024-03-18T00:49:16.110279+0000","last_peered":"2024-03-18T00:49:16.110279+0000","last_clean":"2024-03-18T00:49:16.110279+0000","last_became_active":"2024-03-18T00:48:05.532515+0000","last_became_peered":"2024-03-18T00:48:05.532515+0000","last_unstale":"2024-03-18T00:49:16.110279+0000","last_undegraded":"2024-03-18T00:49:16.110279+0000","last_fullsized":"2024-03-18T00:49:16.110279+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:46:52.056837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088513+0000","last_change":"2024-03-18T00:48:06.542992+0000","last_active":"2024-03-18T00:49:03.088513+0000","last_peered":"2024-03-18T00:49:03.088513+0000","last_clean":"2024-03-18T00:49:03.088513+0000","last_became_active":"2024-03-18T00:48:06.542617+0000","last_became_peered":"2024-03-18T00:48:06.542617+0000","last_unstale":"2024-03-18T00:49:03.088513+0000","last_undegraded":"2024-03-18T00:49:03.088513+0000","last_fullsized":"2024-03-18T00:49:03.088513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:22.736118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088272+0000","last_change":"2024-03-18T00:48:06.536018+0000","last_active":"2024-03-18T00:49:03.088272+0000","last_peered":"2024-03-18T00:49:03.088272+0000","last_clean":"2024-03-18T00:49:03.088272+0000","last_became_active":"2024-03-18T00:48:06.535656+0000","last_became_peered":"2024-03-18T00:48:06.535656+0000","last_unstale":"2024-03-18T00:49:03.088272+0000","last_undegraded":"2024-03-18T00:49:03.088272+0000","last_fullsized":"2024-03-18T00:49:03.088272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:33:00.492887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:16.093080+0000","last_change":"2024-03-18T00:48:05.536635+0000","last_active":"2024-03-18T00:49:16.093080+0000","last_peered":"2024-03-18T00:49:16.093080+0000","last_clean":"2024-03-18T00:49:16.093080+0000","last_became_active":"2024-03-18T00:48:05.535941+0000","last_became_peered":"2024-03-18T00:48:05.535941+0000","last_unstale":"2024-03-18T00:49:16.093080+0000","last_undegraded":"2024-03-18T00:49:16.093080+0000","last_fullsized":"2024-03-18T00:49:16.093080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:20:52.167217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325914+0000","last_change":"2024-03-18T00:48:06.546266+0000","last_active":"2024-03-18T00:49:03.325914+0000","last_peered":"2024-03-18T00:49:03.325914+0000","last_clean":"2024-03-18T00:49:03.325914+0000","last_became_active":"2024-03-18T00:48:06.543565+0000","last_became_peered":"2024-03-18T00:48:06.543565+0000","last_unstale":"2024-03-18T00:49:03.325914+0000","last_undegraded":"2024-03-18T00:49:03.325914+0000","last_fullsized":"2024-03-18T00:49:03.325914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:49:09.673212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.403989+0000","last_change":"2024-03-18T00:48:05.526184+0000","last_active":"2024-03-18T00:49:03.403989+0000","last_peered":"2024-03-18T00:49:03.403989+0000","last_clean":"2024-03-18T00:49:03.403989+0000","last_became_active":"2024-03-18T00:48:05.525011+0000","last_became_peered":"2024-03-18T00:48:05.525011+0000","last_unstale":"2024-03-18T00:49:03.403989+0000","last_undegraded":"2024-03-18T00:49:03.403989+0000","last_fullsized":"2024-03-18T00:49:03.403989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:12.690966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":344,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087104+0000","last_change":"2024-03-18T00:46:53.288768+0000","last_active":"2024-03-18T00:49:03.087104+0000","last_peered":"2024-03-18T00:49:03.087104+0000","last_clean":"2024-03-18T00:49:03.087104+0000","last_became_active":"2024-03-18T00:46:53.288394+0000","last_became_peered":"2024-03-18T00:46:53.288394+0000","last_unstale":"2024-03-18T00:49:03.087104+0000","last_undegraded":"2024-03-18T00:49:03.087104+0000","last_fullsized":"2024-03-18T00:49:03.087104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_clean_scrub_stamp":"2024-03-18T00:43:28.173150+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:49.949940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087727+0000","last_change":"2024-03-18T00:48:06.534383+0000","last_active":"2024-03-18T00:49:03.087727+0000","last_peered":"2024-03-18T00:49:03.087727+0000","last_clean":"2024-03-18T00:49:03.087727+0000","last_became_active":"2024-03-18T00:48:06.534085+0000","last_became_peered":"2024-03-18T00:48:06.534085+0000","last_unstale":"2024-03-18T00:49:03.087727+0000","last_undegraded":"2024-03-18T00:49:03.087727+0000","last_fullsized":"2024-03-18T00:49:03.087727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:54.021830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.407661+0000","last_change":"2024-03-18T00:49:03.407661+0000","last_active":"2024-03-18T00:49:03.407661+0000","last_peered":"2024-03-18T00:49:03.407661+0000","last_clean":"2024-03-18T00:49:03.407661+0000","last_became_active":"2024-03-18T00:49:03.405134+0000","last_became_peered":"2024-03-18T00:49:03.405134+0000","last_unstale":"2024-03-18T00:49:03.407661+0000","last_undegraded":"2024-03-18T00:49:03.407661+0000","last_fullsized":"2024-03-18T00:49:03.407661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:29:03.201556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.098300+0000","last_change":"2024-03-18T00:49:03.097885+0000","last_active":"2024-03-18T00:49:03.098300+0000","last_peered":"2024-03-18T00:49:03.098300+0000","last_clean":"2024-03-18T00:49:03.098300+0000","last_became_active":"2024-03-18T00:49:03.090574+0000","last_became_peered":"2024-03-18T00:49:03.090574+0000","last_unstale":"2024-03-18T00:49:03.098300+0000","last_undegraded":"2024-03-18T00:49:03.098300+0000","last_fullsized":"2024-03-18T00:49:03.098300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:22:30.194349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088082+0000","last_change":"2024-03-18T00:48:06.530108+0000","last_active":"2024-03-18T00:49:03.088082+0000","last_peered":"2024-03-18T00:49:03.088082+0000","last_clean":"2024-03-18T00:49:03.088082+0000","last_became_active":"2024-03-18T00:48:06.529768+0000","last_became_peered":"2024-03-18T00:48:06.529768+0000","last_unstale":"2024-03-18T00:49:03.088082+0000","last_undegraded":"2024-03-18T00:49:03.088082+0000","last_fullsized":"2024-03-18T00:49:03.088082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:23:09.150305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084656+0000","last_change":"2024-03-18T00:48:05.535703+0000","last_active":"2024-03-18T00:49:03.084656+0000","last_peered":"2024-03-18T00:49:03.084656+0000","last_clean":"2024-03-18T00:49:03.084656+0000","last_became_active":"2024-03-18T00:48:05.535086+0000","last_became_peered":"2024-03-18T00:48:05.535086+0000","last_unstale":"2024-03-18T00:49:03.084656+0000","last_undegraded":"2024-03-18T00:49:03.084656+0000","last_fullsized":"2024-03-18T00:49:03.084656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:10:24.172353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325842+0000","last_change":"2024-03-18T00:48:06.541281+0000","last_active":"2024-03-18T00:49:03.325842+0000","last_peered":"2024-03-18T00:49:03.325842+0000","last_clean":"2024-03-18T00:49:03.325842+0000","last_became_active":"2024-03-18T00:48:06.540946+0000","last_became_peered":"2024-03-18T00:48:06.540946+0000","last_unstale":"2024-03-18T00:49:03.325842+0000","last_undegraded":"2024-03-18T00:49:03.325842+0000","last_fullsized":"2024-03-18T00:49:03.325842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:02.987661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.404075+0000","last_change":"2024-03-18T00:48:05.531685+0000","last_active":"2024-03-18T00:49:03.404075+0000","last_peered":"2024-03-18T00:49:03.404075+0000","last_clean":"2024-03-18T00:49:03.404075+0000","last_became_active":"2024-03-18T00:48:05.531239+0000","last_became_peered":"2024-03-18T00:48:05.531239+0000","last_unstale":"2024-03-18T00:49:03.404075+0000","last_undegraded":"2024-03-18T00:49:03.404075+0000","last_fullsized":"2024-03-18T00:49:03.404075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:26:35.034842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086824+0000","last_change":"2024-03-18T00:48:06.546783+0000","last_active":"2024-03-18T00:49:03.086824+0000","last_peered":"2024-03-18T00:49:03.086824+0000","last_clean":"2024-03-18T00:49:03.086824+0000","last_became_active":"2024-03-18T00:48:06.546411+0000","last_became_peered":"2024-03-18T00:48:06.546411+0000","last_unstale":"2024-03-18T00:49:03.086824+0000","last_undegraded":"2024-03-18T00:49:03.086824+0000","last_fullsized":"2024-03-18T00:49:03.086824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:39:34.432104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089245+0000","last_change":"2024-03-18T00:48:05.534701+0000","last_active":"2024-03-18T00:49:03.089245+0000","last_peered":"2024-03-18T00:49:03.089245+0000","last_clean":"2024-03-18T00:49:03.089245+0000","last_became_active":"2024-03-18T00:48:05.534300+0000","last_became_peered":"2024-03-18T00:48:05.534300+0000","last_unstale":"2024-03-18T00:49:03.089245+0000","last_undegraded":"2024-03-18T00:49:03.089245+0000","last_fullsized":"2024-03-18T00:49:03.089245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:10.580001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086905+0000","last_change":"2024-03-18T00:48:06.528286+0000","last_active":"2024-03-18T00:49:03.086905+0000","last_peered":"2024-03-18T00:49:03.086905+0000","last_clean":"2024-03-18T00:49:03.086905+0000","last_became_active":"2024-03-18T00:48:06.527947+0000","last_became_peered":"2024-03-18T00:48:06.527947+0000","last_unstale":"2024-03-18T00:49:03.086905+0000","last_undegraded":"2024-03-18T00:49:03.086905+0000","last_fullsized":"2024-03-18T00:49:03.086905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:34.953923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325566+0000","last_change":"2024-03-18T00:48:05.526530+0000","last_active":"2024-03-18T00:49:03.325566+0000","last_peered":"2024-03-18T00:49:03.325566+0000","last_clean":"2024-03-18T00:49:03.325566+0000","last_became_active":"2024-03-18T00:48:05.526210+0000","last_became_peered":"2024-03-18T00:48:05.526210+0000","last_unstale":"2024-03-18T00:49:03.325566+0000","last_undegraded":"2024-03-18T00:49:03.325566+0000","last_fullsized":"2024-03-18T00:49:03.325566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:11:11.094469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084783+0000","last_change":"2024-03-18T00:48:06.535316+0000","last_active":"2024-03-18T00:49:03.084783+0000","last_peered":"2024-03-18T00:49:03.084783+0000","last_clean":"2024-03-18T00:49:03.084783+0000","last_became_active":"2024-03-18T00:48:06.534945+0000","last_became_peered":"2024-03-18T00:48:06.534945+0000","last_unstale":"2024-03-18T00:49:03.084783+0000","last_undegraded":"2024-03-18T00:49:03.084783+0000","last_fullsized":"2024-03-18T00:49:03.084783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:04:32.570585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087264+0000","last_change":"2024-03-18T00:48:05.529300+0000","last_active":"2024-03-18T00:49:03.087264+0000","last_peered":"2024-03-18T00:49:03.087264+0000","last_clean":"2024-03-18T00:49:03.087264+0000","last_became_active":"2024-03-18T00:48:05.528935+0000","last_became_peered":"2024-03-18T00:48:05.528935+0000","last_unstale":"2024-03-18T00:49:03.087264+0000","last_undegraded":"2024-03-18T00:49:03.087264+0000","last_fullsized":"2024-03-18T00:49:03.087264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:29:45.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325587+0000","last_change":"2024-03-18T00:48:06.536426+0000","last_active":"2024-03-18T00:49:03.325587+0000","last_peered":"2024-03-18T00:49:03.325587+0000","last_clean":"2024-03-18T00:49:03.325587+0000","last_became_active":"2024-03-18T00:48:06.536005+0000","last_became_peered":"2024-03-18T00:48:06.536005+0000","last_unstale":"2024-03-18T00:49:03.325587+0000","last_undegraded":"2024-03-18T00:49:03.325587+0000","last_fullsized":"2024-03-18T00:49:03.325587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:40:52.028794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087438+0000","last_change":"2024-03-18T00:48:05.536495+0000","last_active":"2024-03-18T00:49:03.087438+0000","last_peered":"2024-03-18T00:49:03.087438+0000","last_clean":"2024-03-18T00:49:03.087438+0000","last_became_active":"2024-03-18T00:48:05.536072+0000","last_became_peered":"2024-03-18T00:48:05.536072+0000","last_unstale":"2024-03-18T00:49:03.087438+0000","last_undegraded":"2024-03-18T00:49:03.087438+0000","last_fullsized":"2024-03-18T00:49:03.087438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:25.528261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087389+0000","last_change":"2024-03-18T00:48:06.552900+0000","last_active":"2024-03-18T00:49:03.087389+0000","last_peered":"2024-03-18T00:49:03.087389+0000","last_clean":"2024-03-18T00:49:03.087389+0000","last_became_active":"2024-03-18T00:48:06.552568+0000","last_became_peered":"2024-03-18T00:48:06.552568+0000","last_unstale":"2024-03-18T00:49:03.087389+0000","last_undegraded":"2024-03-18T00:49:03.087389+0000","last_fullsized":"2024-03-18T00:49:03.087389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:53:25.254177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:15.096495+0000","last_change":"2024-03-18T00:48:05.537327+0000","last_active":"2024-03-18T00:49:15.096495+0000","last_peered":"2024-03-18T00:49:15.096495+0000","last_clean":"2024-03-18T00:49:15.096495+0000","last_became_active":"2024-03-18T00:48:05.536825+0000","last_became_peered":"2024-03-18T00:48:05.536825+0000","last_unstale":"2024-03-18T00:49:15.096495+0000","last_undegraded":"2024-03-18T00:49:15.096495+0000","last_fullsized":"2024-03-18T00:49:15.096495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:03.141142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087502+0000","last_change":"2024-03-18T00:48:06.542593+0000","last_active":"2024-03-18T00:49:03.087502+0000","last_peered":"2024-03-18T00:49:03.087502+0000","last_clean":"2024-03-18T00:49:03.087502+0000","last_became_active":"2024-03-18T00:48:06.541624+0000","last_became_peered":"2024-03-18T00:48:06.541624+0000","last_unstale":"2024-03-18T00:49:03.087502+0000","last_undegraded":"2024-03-18T00:49:03.087502+0000","last_fullsized":"2024-03-18T00:49:03.087502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:10:56.905221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089025+0000","last_change":"2024-03-18T00:48:05.529140+0000","last_active":"2024-03-18T00:49:03.089025+0000","last_peered":"2024-03-18T00:49:03.089025+0000","last_clean":"2024-03-18T00:49:03.089025+0000","last_became_active":"2024-03-18T00:48:05.528682+0000","last_became_peered":"2024-03-18T00:48:05.528682+0000","last_unstale":"2024-03-18T00:49:03.089025+0000","last_undegraded":"2024-03-18T00:49:03.089025+0000","last_fullsized":"2024-03-18T00:49:03.089025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:57:52.663150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087272+0000","last_change":"2024-03-18T00:48:06.546027+0000","last_active":"2024-03-18T00:49:03.087272+0000","last_peered":"2024-03-18T00:49:03.087272+0000","last_clean":"2024-03-18T00:49:03.087272+0000","last_became_active":"2024-03-18T00:48:06.545670+0000","last_became_peered":"2024-03-18T00:48:06.545670+0000","last_unstale":"2024-03-18T00:49:03.087272+0000","last_undegraded":"2024-03-18T00:49:03.087272+0000","last_fullsized":"2024-03-18T00:49:03.087272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:47:05.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087890+0000","last_change":"2024-03-18T00:48:05.526929+0000","last_active":"2024-03-18T00:49:03.087890+0000","last_peered":"2024-03-18T00:49:03.087890+0000","last_clean":"2024-03-18T00:49:03.087890+0000","last_became_active":"2024-03-18T00:48:05.526530+0000","last_became_peered":"2024-03-18T00:48:05.526530+0000","last_unstale":"2024-03-18T00:49:03.087890+0000","last_undegraded":"2024-03-18T00:49:03.087890+0000","last_fullsized":"2024-03-18T00:49:03.087890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:51:25.110895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087753+0000","last_change":"2024-03-18T00:48:06.541223+0000","last_active":"2024-03-18T00:49:03.087753+0000","last_peered":"2024-03-18T00:49:03.087753+0000","last_clean":"2024-03-18T00:49:03.087753+0000","last_became_active":"2024-03-18T00:48:06.540898+0000","last_became_peered":"2024-03-18T00:48:06.540898+0000","last_unstale":"2024-03-18T00:49:03.087753+0000","last_undegraded":"2024-03-18T00:49:03.087753+0000","last_fullsized":"2024-03-18T00:49:03.087753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:20:22.749609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084879+0000","last_change":"2024-03-18T00:48:06.532906+0000","last_active":"2024-03-18T00:49:03.084879+0000","last_peered":"2024-03-18T00:49:03.084879+0000","last_clean":"2024-03-18T00:49:03.084879+0000","last_became_active":"2024-03-18T00:48:06.532583+0000","last_became_peered":"2024-03-18T00:48:06.532583+0000","last_unstale":"2024-03-18T00:49:03.084879+0000","last_undegraded":"2024-03-18T00:49:03.084879+0000","last_fullsized":"2024-03-18T00:49:03.084879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:06.699615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087137+0000","last_change":"2024-03-18T00:48:06.530020+0000","last_active":"2024-03-18T00:49:03.087137+0000","last_peered":"2024-03-18T00:49:03.087137+0000","last_clean":"2024-03-18T00:49:03.087137+0000","last_became_active":"2024-03-18T00:48:06.529709+0000","last_became_peered":"2024-03-18T00:48:06.529709+0000","last_unstale":"2024-03-18T00:49:03.087137+0000","last_undegraded":"2024-03-18T00:49:03.087137+0000","last_fullsized":"2024-03-18T00:49:03.087137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:07.305537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087254+0000","last_change":"2024-03-18T00:48:06.537658+0000","last_active":"2024-03-18T00:49:03.087254+0000","last_peered":"2024-03-18T00:49:03.087254+0000","last_clean":"2024-03-18T00:49:03.087254+0000","last_became_active":"2024-03-18T00:48:06.537339+0000","last_became_peered":"2024-03-18T00:48:06.537339+0000","last_unstale":"2024-03-18T00:49:03.087254+0000","last_undegraded":"2024-03-18T00:49:03.087254+0000","last_fullsized":"2024-03-18T00:49:03.087254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:18:25.753168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086737+0000","last_change":"2024-03-18T00:48:06.544514+0000","last_active":"2024-03-18T00:49:03.086737+0000","last_peered":"2024-03-18T00:49:03.086737+0000","last_clean":"2024-03-18T00:49:03.086737+0000","last_became_active":"2024-03-18T00:48:06.544165+0000","last_became_peered":"2024-03-18T00:48:06.544165+0000","last_unstale":"2024-03-18T00:49:03.086737+0000","last_undegraded":"2024-03-18T00:49:03.086737+0000","last_fullsized":"2024-03-18T00:49:03.086737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:48:02.498940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.090610+0000","last_change":"2024-03-18T00:49:03.090610+0000","last_active":"2024-03-18T00:49:03.090610+0000","last_peered":"2024-03-18T00:49:03.090610+0000","last_clean":"2024-03-18T00:49:03.090610+0000","last_became_active":"2024-03-18T00:49:03.090014+0000","last_became_peered":"2024-03-18T00:49:03.090014+0000","last_unstale":"2024-03-18T00:49:03.090610+0000","last_undegraded":"2024-03-18T00:49:03.090610+0000","last_fullsized":"2024-03-18T00:49:03.090610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:55:12.210223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.085194+0000","last_change":"2024-03-18T00:48:06.533837+0000","last_active":"2024-03-18T00:49:03.085194+0000","last_peered":"2024-03-18T00:49:03.085194+0000","last_clean":"2024-03-18T00:49:03.085194+0000","last_became_active":"2024-03-18T00:48:06.533418+0000","last_became_peered":"2024-03-18T00:48:06.533418+0000","last_unstale":"2024-03-18T00:49:03.085194+0000","last_undegraded":"2024-03-18T00:49:03.085194+0000","last_fullsized":"2024-03-18T00:49:03.085194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:10.873312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087070+0000","last_change":"2024-03-18T00:48:06.527394+0000","last_active":"2024-03-18T00:49:03.087070+0000","last_peered":"2024-03-18T00:49:03.087070+0000","last_clean":"2024-03-18T00:49:03.087070+0000","last_became_active":"2024-03-18T00:48:06.526879+0000","last_became_peered":"2024-03-18T00:48:06.526879+0000","last_unstale":"2024-03-18T00:49:03.087070+0000","last_undegraded":"2024-03-18T00:49:03.087070+0000","last_fullsized":"2024-03-18T00:49:03.087070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:23:07.397259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325740+0000","last_change":"2024-03-18T00:48:06.542645+0000","last_active":"2024-03-18T00:49:03.325740+0000","last_peered":"2024-03-18T00:49:03.325740+0000","last_clean":"2024-03-18T00:49:03.325740+0000","last_became_active":"2024-03-18T00:48:06.542073+0000","last_became_peered":"2024-03-18T00:48:06.542073+0000","last_unstale":"2024-03-18T00:49:03.325740+0000","last_undegraded":"2024-03-18T00:49:03.325740+0000","last_fullsized":"2024-03-18T00:49:03.325740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:08:44.479318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088173+0000","last_change":"2024-03-18T00:48:06.541698+0000","last_active":"2024-03-18T00:49:03.088173+0000","last_peered":"2024-03-18T00:49:03.088173+0000","last_clean":"2024-03-18T00:49:03.088173+0000","last_became_active":"2024-03-18T00:48:06.541378+0000","last_became_peered":"2024-03-18T00:48:06.541378+0000","last_unstale":"2024-03-18T00:49:03.088173+0000","last_undegraded":"2024-03-18T00:49:03.088173+0000","last_fullsized":"2024-03-18T00:49:03.088173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:42:23.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325481+0000","last_change":"2024-03-18T00:48:06.533184+0000","last_active":"2024-03-18T00:49:03.325481+0000","last_peered":"2024-03-18T00:49:03.325481+0000","last_clean":"2024-03-18T00:49:03.325481+0000","last_became_active":"2024-03-18T00:48:06.532034+0000","last_became_peered":"2024-03-18T00:48:06.532034+0000","last_unstale":"2024-03-18T00:49:03.325481+0000","last_undegraded":"2024-03-18T00:49:03.325481+0000","last_fullsized":"2024-03-18T00:49:03.325481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:45.265799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.090002+0000","last_change":"2024-03-18T00:48:06.545941+0000","last_active":"2024-03-18T00:49:03.090002+0000","last_peered":"2024-03-18T00:49:03.090002+0000","last_clean":"2024-03-18T00:49:03.090002+0000","last_became_active":"2024-03-18T00:48:06.545402+0000","last_became_peered":"2024-03-18T00:48:06.545402+0000","last_unstale":"2024-03-18T00:49:03.090002+0000","last_undegraded":"2024-03-18T00:49:03.090002+0000","last_fullsized":"2024-03-18T00:49:03.090002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:11:45.512849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087949+0000","last_change":"2024-03-18T00:48:06.541513+0000","last_active":"2024-03-18T00:49:03.087949+0000","last_peered":"2024-03-18T00:49:03.087949+0000","last_clean":"2024-03-18T00:49:03.087949+0000","last_became_active":"2024-03-18T00:48:06.541128+0000","last_became_peered":"2024-03-18T00:48:06.541128+0000","last_unstale":"2024-03-18T00:49:03.087949+0000","last_undegraded":"2024-03-18T00:49:03.087949+0000","last_fullsized":"2024-03-18T00:49:03.087949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:42:10.635836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084786+0000","last_change":"2024-03-18T00:48:06.542316+0000","last_active":"2024-03-18T00:49:03.084786+0000","last_peered":"2024-03-18T00:49:03.084786+0000","last_clean":"2024-03-18T00:49:03.084786+0000","last_became_active":"2024-03-18T00:48:06.541914+0000","last_became_peered":"2024-03-18T00:48:06.541914+0000","last_unstale":"2024-03-18T00:49:03.084786+0000","last_undegraded":"2024-03-18T00:49:03.084786+0000","last_fullsized":"2024-03-18T00:49:03.084786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:46.293240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249372,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67859,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":658,"num_read_kb":431,"num_write":236,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1237,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":203577,"data_compressed":5589,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351003,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":461165,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":1112115,"data_compressed":43768,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776166,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":459970,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":460236,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1052866,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":459687,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1024000,"data_stored":521646,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":987136,"data_stored":461026,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":522145,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":459512,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888128,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":459660,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":1051531,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62798,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63297,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":62782,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T00:49:22.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T00:49:22.909 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:23.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:23 smithi049 ceph-mon[32130]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:23.360 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:23 smithi049 ceph-mon[32130]: from='client.24794 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:49:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:23 smithi107 ceph-mon[36270]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:23 smithi107 ceph-mon[36270]: from='client.24794 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:49:23.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:23 smithi106 ceph-mon[35899]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:23.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:23 smithi106 ceph-mon[35899]: from='client.24794 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.612+0000 7f1021700700 1 -- 172.21.15.49:0/4059290996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f101c06b5e0 msgr2=0x7f101c106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.612+0000 7f1021700700 1 --2- 172.21.15.49:0/4059290996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f101c06b5e0 0x7f101c106f50 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f10100099c0 tx=0x7f1010009cd0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.613+0000 7f1021700700 1 -- 172.21.15.49:0/4059290996 shutdown_connections 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.613+0000 7f1021700700 1 --2- 172.21.15.49:0/4059290996 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f101c06ac30 0x7f101c06b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.613+0000 7f1021700700 1 --2- 172.21.15.49:0/4059290996 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f101c107680 0x7f101c10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.613+0000 7f1021700700 1 --2- 172.21.15.49:0/4059290996 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f101c06b5e0 0x7f101c106f50 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.613+0000 7f1021700700 1 -- 172.21.15.49:0/4059290996 >> 172.21.15.49:0/4059290996 conn(0x7f101c077220 msgr2=0x7f101c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.613+0000 7f1021700700 1 -- 172.21.15.49:0/4059290996 shutdown_connections 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.613+0000 7f1021700700 1 -- 172.21.15.49:0/4059290996 wait complete. 2024-03-18T00:49:24.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.613+0000 7f1021700700 1 Processor -- start 2024-03-18T00:49:24.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.613+0000 7f1021700700 1 -- start start 2024-03-18T00:49:24.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f1021700700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f101c06ac30 0x7f101c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:24.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f1021700700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f101c06b5e0 0x7f101c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:24.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f1021700700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f101c107680 0x7f101c198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:24.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f1021700700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f101c1125d0 con 0x7f101c107680 2024-03-18T00:49:24.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f1021700700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f101c112450 con 0x7f101c06ac30 2024-03-18T00:49:24.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f1021700700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f101c112750 con 0x7f101c06b5e0 2024-03-18T00:49:24.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101b7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f101c107680 0x7f101c198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101a7fc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f101c06b5e0 0x7f101c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101a7fc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f101c06b5e0 0x7f101c19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58176/0 (socket says 172.21.15.49:58176) 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101a7fc700 1 -- 172.21.15.49:0/3445248379 learned_addr learned my addr 172.21.15.49:0/3445248379 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101affd700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f101c06ac30 0x7f101c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101b7fe700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f101c107680 0x7f101c198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55230/0 (socket says 172.21.15.49:55230) 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101a7fc700 1 -- 172.21.15.49:0/3445248379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f101c06ac30 msgr2=0x7f101c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101a7fc700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f101c06ac30 0x7f101c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101a7fc700 1 -- 172.21.15.49:0/3445248379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f101c107680 msgr2=0x7f101c198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101a7fc700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f101c107680 0x7f101c198890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101a7fc700 1 -- 172.21.15.49:0/3445248379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c198ff0 con 0x7f101c06b5e0 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f101a7fc700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f101c06b5e0 0x7f101c19ec10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f101000b580 tx=0x7f1010003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f1003fff700 1 -- 172.21.15.49:0/3445248379 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010003680 con 0x7f101c06b5e0 2024-03-18T00:49:24.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.614+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101c199280 con 0x7f101c06b5e0 2024-03-18T00:49:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.615+0000 7f1003fff700 1 -- 172.21.15.49:0/3445248379 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10100055a0 con 0x7f101c06b5e0 2024-03-18T00:49:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.615+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101c1a55a0 con 0x7f101c06b5e0 2024-03-18T00:49:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.615+0000 7f1003fff700 1 -- 172.21.15.49:0/3445248379 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101000fd00 con 0x7f101c06b5e0 2024-03-18T00:49:24.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.615+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f101c068f90 con 0x7f101c06b5e0 2024-03-18T00:49:24.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.617+0000 7f1003fff700 1 -- 172.21.15.49:0/3445248379 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f101000f460 con 0x7f101c06b5e0 2024-03-18T00:49:24.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.617+0000 7f1003fff700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f10040a2ac0 0x7f10040a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:24.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.617+0000 7f101affd700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f10040a2ac0 0x7f10040a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:24.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.617+0000 7f1003fff700 1 -- 172.21.15.49:0/3445248379 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f10100cfac0 con 0x7f101c06b5e0 2024-03-18T00:49:24.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.618+0000 7f101affd700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f10040a2ac0 0x7f10040a4f80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f101c1999d0 tx=0x7f100c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:24.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.626+0000 7f1003fff700 1 -- 172.21.15.49:0/3445248379 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1010013070 con 0x7f101c06b5e0 2024-03-18T00:49:24.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.826+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f101c199dc0 con 0x7f10040a2ac0 2024-03-18T00:49:24.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.833+0000 7f1003fff700 1 -- 172.21.15.49:0/3445248379 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259044 (secure 0 0 0) 0x7f101c199dc0 con 0x7f10040a2ac0 2024-03-18T00:49:24.837 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:24.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f10040a2ac0 msgr2=0x7f10040a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:24.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f10040a2ac0 0x7f10040a4f80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f101c1999d0 tx=0x7f100c009260 comp rx=0 tx=0).stop 2024-03-18T00:49:24.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f101c06b5e0 msgr2=0x7f101c19ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:24.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f101c06b5e0 0x7f101c19ec10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f101000b580 tx=0x7f1010003a90 comp rx=0 tx=0).stop 2024-03-18T00:49:24.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 shutdown_connections 2024-03-18T00:49:24.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f101c107680 0x7f101c198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f101c06ac30 0x7f101c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f10040a2ac0 0x7f10040a4f80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 --2- 172.21.15.49:0/3445248379 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f101c06b5e0 0x7f101c19ec10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:24.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 >> 172.21.15.49:0/3445248379 conn(0x7f101c077220 msgr2=0x7f101c0ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:24.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 shutdown_connections 2024-03-18T00:49:24.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:24.837+0000 7f1021700700 1 -- 172.21.15.49:0/3445248379 wait complete. 2024-03-18T00:49:24.840 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-03-18T00:49:25.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:25 smithi107 ceph-mon[36270]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:25.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:25 smithi049 ceph-mon[32130]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:25.585 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":329,"stamp":"2024-03-18T00:49:23.479395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658461,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":804,"num_read_kb":545,"num_write":419,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181973,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":344,"ondisk_log_size":344,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356868,"kb_used_data":11796,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584956,"statfs":{"total":1151940427776,"available":1151574994944,"internally_reserved":0,"allocated":12079104,"data_stored":7481559,"data_compressed":183621,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003231"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.085235+0000","last_change":"2024-03-18T00:48:06.536034+0000","last_active":"2024-03-18T00:49:03.085235+0000","last_peered":"2024-03-18T00:49:03.085235+0000","last_clean":"2024-03-18T00:49:03.085235+0000","last_became_active":"2024-03-18T00:48:06.535694+0000","last_became_peered":"2024-03-18T00:48:06.535694+0000","last_unstale":"2024-03-18T00:49:03.085235+0000","last_undegraded":"2024-03-18T00:49:03.085235+0000","last_fullsized":"2024-03-18T00:49:03.085235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:24:33.785669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086935+0000","last_change":"2024-03-18T00:48:06.536741+0000","last_active":"2024-03-18T00:49:03.086935+0000","last_peered":"2024-03-18T00:49:03.086935+0000","last_clean":"2024-03-18T00:49:03.086935+0000","last_became_active":"2024-03-18T00:48:06.535865+0000","last_became_peered":"2024-03-18T00:48:06.535865+0000","last_unstale":"2024-03-18T00:49:03.086935+0000","last_undegraded":"2024-03-18T00:49:03.086935+0000","last_fullsized":"2024-03-18T00:49:03.086935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:58.104983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086845+0000","last_change":"2024-03-18T00:48:06.537162+0000","last_active":"2024-03-18T00:49:03.086845+0000","last_peered":"2024-03-18T00:49:03.086845+0000","last_clean":"2024-03-18T00:49:03.086845+0000","last_became_active":"2024-03-18T00:48:06.536835+0000","last_became_peered":"2024-03-18T00:48:06.536835+0000","last_unstale":"2024-03-18T00:49:03.086845+0000","last_undegraded":"2024-03-18T00:49:03.086845+0000","last_fullsized":"2024-03-18T00:49:03.086845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:15:08.519180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.404142+0000","last_change":"2024-03-18T00:48:06.540084+0000","last_active":"2024-03-18T00:49:03.404142+0000","last_peered":"2024-03-18T00:49:03.404142+0000","last_clean":"2024-03-18T00:49:03.404142+0000","last_became_active":"2024-03-18T00:48:06.539736+0000","last_became_peered":"2024-03-18T00:48:06.539736+0000","last_unstale":"2024-03-18T00:49:03.404142+0000","last_undegraded":"2024-03-18T00:49:03.404142+0000","last_fullsized":"2024-03-18T00:49:03.404142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:37:51.748603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087335+0000","last_change":"2024-03-18T00:48:06.540022+0000","last_active":"2024-03-18T00:49:03.087335+0000","last_peered":"2024-03-18T00:49:03.087335+0000","last_clean":"2024-03-18T00:49:03.087335+0000","last_became_active":"2024-03-18T00:48:06.539665+0000","last_became_peered":"2024-03-18T00:48:06.539665+0000","last_unstale":"2024-03-18T00:49:03.087335+0000","last_undegraded":"2024-03-18T00:49:03.087335+0000","last_fullsized":"2024-03-18T00:49:03.087335+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:55:52.448279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088241+0000","last_change":"2024-03-18T00:48:06.538672+0000","last_active":"2024-03-18T00:49:03.088241+0000","last_peered":"2024-03-18T00:49:03.088241+0000","last_clean":"2024-03-18T00:49:03.088241+0000","last_became_active":"2024-03-18T00:48:06.538189+0000","last_became_peered":"2024-03-18T00:48:06.538189+0000","last_unstale":"2024-03-18T00:49:03.088241+0000","last_undegraded":"2024-03-18T00:49:03.088241+0000","last_fullsized":"2024-03-18T00:49:03.088241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:09:04.672255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086945+0000","last_change":"2024-03-18T00:48:06.546417+0000","last_active":"2024-03-18T00:49:03.086945+0000","last_peered":"2024-03-18T00:49:03.086945+0000","last_clean":"2024-03-18T00:49:03.086945+0000","last_became_active":"2024-03-18T00:48:06.545759+0000","last_became_peered":"2024-03-18T00:48:06.545759+0000","last_unstale":"2024-03-18T00:49:03.086945+0000","last_undegraded":"2024-03-18T00:49:03.086945+0000","last_fullsized":"2024-03-18T00:49:03.086945+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:56.759028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325436+0000","last_change":"2024-03-18T00:48:06.545875+0000","last_active":"2024-03-18T00:49:03.325436+0000","last_peered":"2024-03-18T00:49:03.325436+0000","last_clean":"2024-03-18T00:49:03.325436+0000","last_became_active":"2024-03-18T00:48:06.545316+0000","last_became_peered":"2024-03-18T00:48:06.545316+0000","last_unstale":"2024-03-18T00:49:03.325436+0000","last_undegraded":"2024-03-18T00:49:03.325436+0000","last_fullsized":"2024-03-18T00:49:03.325436+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:22:40.998671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089525+0000","last_change":"2024-03-18T00:48:06.533947+0000","last_active":"2024-03-18T00:49:03.089525+0000","last_peered":"2024-03-18T00:49:03.089525+0000","last_clean":"2024-03-18T00:49:03.089525+0000","last_became_active":"2024-03-18T00:48:06.533615+0000","last_became_peered":"2024-03-18T00:48:06.533615+0000","last_unstale":"2024-03-18T00:49:03.089525+0000","last_undegraded":"2024-03-18T00:49:03.089525+0000","last_fullsized":"2024-03-18T00:49:03.089525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:32:16.750269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088372+0000","last_change":"2024-03-18T00:48:06.537614+0000","last_active":"2024-03-18T00:49:03.088372+0000","last_peered":"2024-03-18T00:49:03.088372+0000","last_clean":"2024-03-18T00:49:03.088372+0000","last_became_active":"2024-03-18T00:48:06.536617+0000","last_became_peered":"2024-03-18T00:48:06.536617+0000","last_unstale":"2024-03-18T00:49:03.088372+0000","last_undegraded":"2024-03-18T00:49:03.088372+0000","last_fullsized":"2024-03-18T00:49:03.088372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:52.222603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087412+0000","last_change":"2024-03-18T00:48:06.540363+0000","last_active":"2024-03-18T00:49:03.087412+0000","last_peered":"2024-03-18T00:49:03.087412+0000","last_clean":"2024-03-18T00:49:03.087412+0000","last_became_active":"2024-03-18T00:48:06.540011+0000","last_became_peered":"2024-03-18T00:48:06.540011+0000","last_unstale":"2024-03-18T00:49:03.087412+0000","last_undegraded":"2024-03-18T00:49:03.087412+0000","last_fullsized":"2024-03-18T00:49:03.087412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:23:10.034444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.404019+0000","last_change":"2024-03-18T00:48:06.532611+0000","last_active":"2024-03-18T00:49:03.404019+0000","last_peered":"2024-03-18T00:49:03.404019+0000","last_clean":"2024-03-18T00:49:03.404019+0000","last_became_active":"2024-03-18T00:48:06.532301+0000","last_became_peered":"2024-03-18T00:48:06.532301+0000","last_unstale":"2024-03-18T00:49:03.404019+0000","last_undegraded":"2024-03-18T00:49:03.404019+0000","last_fullsized":"2024-03-18T00:49:03.404019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:40.795105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088078+0000","last_change":"2024-03-18T00:48:06.531993+0000","last_active":"2024-03-18T00:49:03.088078+0000","last_peered":"2024-03-18T00:49:03.088078+0000","last_clean":"2024-03-18T00:49:03.088078+0000","last_became_active":"2024-03-18T00:48:06.531677+0000","last_became_peered":"2024-03-18T00:48:06.531677+0000","last_unstale":"2024-03-18T00:49:03.088078+0000","last_undegraded":"2024-03-18T00:49:03.088078+0000","last_fullsized":"2024-03-18T00:49:03.088078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:48:24.865299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087046+0000","last_change":"2024-03-18T00:48:06.536830+0000","last_active":"2024-03-18T00:49:03.087046+0000","last_peered":"2024-03-18T00:49:03.087046+0000","last_clean":"2024-03-18T00:49:03.087046+0000","last_became_active":"2024-03-18T00:48:06.535908+0000","last_became_peered":"2024-03-18T00:48:06.535908+0000","last_unstale":"2024-03-18T00:49:03.087046+0000","last_undegraded":"2024-03-18T00:49:03.087046+0000","last_fullsized":"2024-03-18T00:49:03.087046+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:23:26.155130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086601+0000","last_change":"2024-03-18T00:48:06.543641+0000","last_active":"2024-03-18T00:49:03.086601+0000","last_peered":"2024-03-18T00:49:03.086601+0000","last_clean":"2024-03-18T00:49:03.086601+0000","last_became_active":"2024-03-18T00:48:06.543293+0000","last_became_peered":"2024-03-18T00:48:06.543293+0000","last_unstale":"2024-03-18T00:49:03.086601+0000","last_undegraded":"2024-03-18T00:49:03.086601+0000","last_fullsized":"2024-03-18T00:49:03.086601+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:52.406255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089103+0000","last_change":"2024-03-18T00:48:06.544732+0000","last_active":"2024-03-18T00:49:03.089103+0000","last_peered":"2024-03-18T00:49:03.089103+0000","last_clean":"2024-03-18T00:49:03.089103+0000","last_became_active":"2024-03-18T00:48:06.544400+0000","last_became_peered":"2024-03-18T00:48:06.544400+0000","last_unstale":"2024-03-18T00:49:03.089103+0000","last_undegraded":"2024-03-18T00:49:03.089103+0000","last_fullsized":"2024-03-18T00:49:03.089103+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:23:39.655434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.085669+0000","last_change":"2024-03-18T00:48:06.536595+0000","last_active":"2024-03-18T00:49:03.085669+0000","last_peered":"2024-03-18T00:49:03.085669+0000","last_clean":"2024-03-18T00:49:03.085669+0000","last_became_active":"2024-03-18T00:48:06.535874+0000","last_became_peered":"2024-03-18T00:48:06.535874+0000","last_unstale":"2024-03-18T00:49:03.085669+0000","last_undegraded":"2024-03-18T00:49:03.085669+0000","last_fullsized":"2024-03-18T00:49:03.085669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:55.603614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084750+0000","last_change":"2024-03-18T00:48:06.546676+0000","last_active":"2024-03-18T00:49:03.084750+0000","last_peered":"2024-03-18T00:49:03.084750+0000","last_clean":"2024-03-18T00:49:03.084750+0000","last_became_active":"2024-03-18T00:48:06.546356+0000","last_became_peered":"2024-03-18T00:48:06.546356+0000","last_unstale":"2024-03-18T00:49:03.084750+0000","last_undegraded":"2024-03-18T00:49:03.084750+0000","last_fullsized":"2024-03-18T00:49:03.084750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:04:35.467415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086966+0000","last_change":"2024-03-18T00:48:06.532184+0000","last_active":"2024-03-18T00:49:03.086966+0000","last_peered":"2024-03-18T00:49:03.086966+0000","last_clean":"2024-03-18T00:49:03.086966+0000","last_became_active":"2024-03-18T00:48:06.531816+0000","last_became_peered":"2024-03-18T00:48:06.531816+0000","last_unstale":"2024-03-18T00:49:03.086966+0000","last_undegraded":"2024-03-18T00:49:03.086966+0000","last_fullsized":"2024-03-18T00:49:03.086966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:09:03.113024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":101,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:18.111759+0000","last_change":"2024-03-18T00:48:05.530213+0000","last_active":"2024-03-18T00:49:18.111759+0000","last_peered":"2024-03-18T00:49:18.111759+0000","last_clean":"2024-03-18T00:49:18.111759+0000","last_became_active":"2024-03-18T00:48:05.529897+0000","last_became_peered":"2024-03-18T00:48:05.529897+0000","last_unstale":"2024-03-18T00:49:18.111759+0000","last_undegraded":"2024-03-18T00:49:18.111759+0000","last_fullsized":"2024-03-18T00:49:18.111759+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:49.003362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325786+0000","last_change":"2024-03-18T00:48:06.544437+0000","last_active":"2024-03-18T00:49:03.325786+0000","last_peered":"2024-03-18T00:49:03.325786+0000","last_clean":"2024-03-18T00:49:03.325786+0000","last_became_active":"2024-03-18T00:48:06.544125+0000","last_became_peered":"2024-03-18T00:48:06.544125+0000","last_unstale":"2024-03-18T00:49:03.325786+0000","last_undegraded":"2024-03-18T00:49:03.325786+0000","last_fullsized":"2024-03-18T00:49:03.325786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:56:42.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325937+0000","last_change":"2024-03-18T00:48:05.536249+0000","last_active":"2024-03-18T00:49:03.325937+0000","last_peered":"2024-03-18T00:49:03.325937+0000","last_clean":"2024-03-18T00:49:03.325937+0000","last_became_active":"2024-03-18T00:48:05.535954+0000","last_became_peered":"2024-03-18T00:48:05.535954+0000","last_unstale":"2024-03-18T00:49:03.325937+0000","last_undegraded":"2024-03-18T00:49:03.325937+0000","last_fullsized":"2024-03-18T00:49:03.325937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:18:52.341453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086984+0000","last_change":"2024-03-18T00:48:06.541377+0000","last_active":"2024-03-18T00:49:03.086984+0000","last_peered":"2024-03-18T00:49:03.086984+0000","last_clean":"2024-03-18T00:49:03.086984+0000","last_became_active":"2024-03-18T00:48:06.540937+0000","last_became_peered":"2024-03-18T00:48:06.540937+0000","last_unstale":"2024-03-18T00:49:03.086984+0000","last_undegraded":"2024-03-18T00:49:03.086984+0000","last_fullsized":"2024-03-18T00:49:03.086984+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:51:55.268517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088568+0000","last_change":"2024-03-18T00:48:05.534372+0000","last_active":"2024-03-18T00:49:03.088568+0000","last_peered":"2024-03-18T00:49:03.088568+0000","last_clean":"2024-03-18T00:49:03.088568+0000","last_became_active":"2024-03-18T00:48:05.534034+0000","last_became_peered":"2024-03-18T00:48:05.534034+0000","last_unstale":"2024-03-18T00:49:03.088568+0000","last_undegraded":"2024-03-18T00:49:03.088568+0000","last_fullsized":"2024-03-18T00:49:03.088568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:58:55.013167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089575+0000","last_change":"2024-03-18T00:48:06.527276+0000","last_active":"2024-03-18T00:49:03.089575+0000","last_peered":"2024-03-18T00:49:03.089575+0000","last_clean":"2024-03-18T00:49:03.089575+0000","last_became_active":"2024-03-18T00:48:06.526904+0000","last_became_peered":"2024-03-18T00:48:06.526904+0000","last_unstale":"2024-03-18T00:49:03.089575+0000","last_undegraded":"2024-03-18T00:49:03.089575+0000","last_fullsized":"2024-03-18T00:49:03.089575+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:16:10.731256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087607+0000","last_change":"2024-03-18T00:48:05.528188+0000","last_active":"2024-03-18T00:49:03.087607+0000","last_peered":"2024-03-18T00:49:03.087607+0000","last_clean":"2024-03-18T00:49:03.087607+0000","last_became_active":"2024-03-18T00:48:05.527788+0000","last_became_peered":"2024-03-18T00:48:05.527788+0000","last_unstale":"2024-03-18T00:49:03.087607+0000","last_undegraded":"2024-03-18T00:49:03.087607+0000","last_fullsized":"2024-03-18T00:49:03.087607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:47.939771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.092920+0000","last_change":"2024-03-18T00:49:03.092920+0000","last_active":"2024-03-18T00:49:03.092920+0000","last_peered":"2024-03-18T00:49:03.092920+0000","last_clean":"2024-03-18T00:49:03.092920+0000","last_became_active":"2024-03-18T00:49:03.092569+0000","last_became_peered":"2024-03-18T00:49:03.092569+0000","last_unstale":"2024-03-18T00:49:03.092920+0000","last_undegraded":"2024-03-18T00:49:03.092920+0000","last_fullsized":"2024-03-18T00:49:03.092920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:15:02.246688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:20.110376+0000","last_change":"2024-03-18T00:49:03.100514+0000","last_active":"2024-03-18T00:49:20.110376+0000","last_peered":"2024-03-18T00:49:20.110376+0000","last_clean":"2024-03-18T00:49:20.110376+0000","last_became_active":"2024-03-18T00:49:03.090898+0000","last_became_peered":"2024-03-18T00:49:03.090898+0000","last_unstale":"2024-03-18T00:49:20.110376+0000","last_undegraded":"2024-03-18T00:49:20.110376+0000","last_fullsized":"2024-03-18T00:49:20.110376+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:19:39.943819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087085+0000","last_change":"2024-03-18T00:48:06.537405+0000","last_active":"2024-03-18T00:49:03.087085+0000","last_peered":"2024-03-18T00:49:03.087085+0000","last_clean":"2024-03-18T00:49:03.087085+0000","last_became_active":"2024-03-18T00:48:06.536919+0000","last_became_peered":"2024-03-18T00:48:06.536919+0000","last_unstale":"2024-03-18T00:49:03.087085+0000","last_undegraded":"2024-03-18T00:49:03.087085+0000","last_fullsized":"2024-03-18T00:49:03.087085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:31:59.863571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088122+0000","last_change":"2024-03-18T00:48:05.528588+0000","last_active":"2024-03-18T00:49:03.088122+0000","last_peered":"2024-03-18T00:49:03.088122+0000","last_clean":"2024-03-18T00:49:03.088122+0000","last_became_active":"2024-03-18T00:48:05.528263+0000","last_became_peered":"2024-03-18T00:48:05.528263+0000","last_unstale":"2024-03-18T00:49:03.088122+0000","last_undegraded":"2024-03-18T00:49:03.088122+0000","last_fullsized":"2024-03-18T00:49:03.088122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:18:10.283278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088860+0000","last_change":"2024-03-18T00:48:06.546323+0000","last_active":"2024-03-18T00:49:03.088860+0000","last_peered":"2024-03-18T00:49:03.088860+0000","last_clean":"2024-03-18T00:49:03.088860+0000","last_became_active":"2024-03-18T00:48:06.545956+0000","last_became_peered":"2024-03-18T00:48:06.545956+0000","last_unstale":"2024-03-18T00:49:03.088860+0000","last_undegraded":"2024-03-18T00:49:03.088860+0000","last_fullsized":"2024-03-18T00:49:03.088860+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:32.618724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087983+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:49:03.087983+0000","last_peered":"2024-03-18T00:49:03.087983+0000","last_clean":"2024-03-18T00:49:03.087983+0000","last_became_active":"2024-03-18T00:48:05.525590+0000","last_became_peered":"2024-03-18T00:48:05.525590+0000","last_unstale":"2024-03-18T00:49:03.087983+0000","last_undegraded":"2024-03-18T00:49:03.087983+0000","last_fullsized":"2024-03-18T00:49:03.087983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:19.357331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325522+0000","last_change":"2024-03-18T00:48:06.538798+0000","last_active":"2024-03-18T00:49:03.325522+0000","last_peered":"2024-03-18T00:49:03.325522+0000","last_clean":"2024-03-18T00:49:03.325522+0000","last_became_active":"2024-03-18T00:48:06.538457+0000","last_became_peered":"2024-03-18T00:48:06.538457+0000","last_unstale":"2024-03-18T00:49:03.325522+0000","last_undegraded":"2024-03-18T00:49:03.325522+0000","last_fullsized":"2024-03-18T00:49:03.325522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:10.738683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088047+0000","last_change":"2024-03-18T00:48:05.528094+0000","last_active":"2024-03-18T00:49:03.088047+0000","last_peered":"2024-03-18T00:49:03.088047+0000","last_clean":"2024-03-18T00:49:03.088047+0000","last_became_active":"2024-03-18T00:48:05.527730+0000","last_became_peered":"2024-03-18T00:48:05.527730+0000","last_unstale":"2024-03-18T00:49:03.088047+0000","last_undegraded":"2024-03-18T00:49:03.088047+0000","last_fullsized":"2024-03-18T00:49:03.088047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:48:27.707505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084567+0000","last_change":"2024-03-18T00:48:06.548676+0000","last_active":"2024-03-18T00:49:03.084567+0000","last_peered":"2024-03-18T00:49:03.084567+0000","last_clean":"2024-03-18T00:49:03.084567+0000","last_became_active":"2024-03-18T00:48:06.547693+0000","last_became_peered":"2024-03-18T00:48:06.547693+0000","last_unstale":"2024-03-18T00:49:03.084567+0000","last_undegraded":"2024-03-18T00:49:03.084567+0000","last_fullsized":"2024-03-18T00:49:03.084567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:13.849175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":161,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:20.102318+0000","last_change":"2024-03-18T00:48:05.523639+0000","last_active":"2024-03-18T00:49:20.102318+0000","last_peered":"2024-03-18T00:49:20.102318+0000","last_clean":"2024-03-18T00:49:20.102318+0000","last_became_active":"2024-03-18T00:48:05.523269+0000","last_became_peered":"2024-03-18T00:48:05.523269+0000","last_unstale":"2024-03-18T00:49:20.102318+0000","last_undegraded":"2024-03-18T00:49:20.102318+0000","last_fullsized":"2024-03-18T00:49:20.102318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:39.961247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089528+0000","last_change":"2024-03-18T00:48:06.537786+0000","last_active":"2024-03-18T00:49:03.089528+0000","last_peered":"2024-03-18T00:49:03.089528+0000","last_clean":"2024-03-18T00:49:03.089528+0000","last_became_active":"2024-03-18T00:48:06.537488+0000","last_became_peered":"2024-03-18T00:48:06.537488+0000","last_unstale":"2024-03-18T00:49:03.089528+0000","last_undegraded":"2024-03-18T00:49:03.089528+0000","last_fullsized":"2024-03-18T00:49:03.089528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:45:02.095610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.096001+0000","last_change":"2024-03-18T00:49:03.096001+0000","last_active":"2024-03-18T00:49:03.096001+0000","last_peered":"2024-03-18T00:49:03.096001+0000","last_clean":"2024-03-18T00:49:03.096001+0000","last_became_active":"2024-03-18T00:49:03.090868+0000","last_became_peered":"2024-03-18T00:49:03.090868+0000","last_unstale":"2024-03-18T00:49:03.096001+0000","last_undegraded":"2024-03-18T00:49:03.096001+0000","last_fullsized":"2024-03-18T00:49:03.096001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:03.930010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086987+0000","last_change":"2024-03-18T00:48:06.545416+0000","last_active":"2024-03-18T00:49:03.086987+0000","last_peered":"2024-03-18T00:49:03.086987+0000","last_clean":"2024-03-18T00:49:03.086987+0000","last_became_active":"2024-03-18T00:48:06.545087+0000","last_became_peered":"2024-03-18T00:48:06.545087+0000","last_unstale":"2024-03-18T00:49:03.086987+0000","last_undegraded":"2024-03-18T00:49:03.086987+0000","last_fullsized":"2024-03-18T00:49:03.086987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:02:25.309990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089159+0000","last_change":"2024-03-18T00:48:05.533088+0000","last_active":"2024-03-18T00:49:03.089159+0000","last_peered":"2024-03-18T00:49:03.089159+0000","last_clean":"2024-03-18T00:49:03.089159+0000","last_became_active":"2024-03-18T00:48:05.532308+0000","last_became_peered":"2024-03-18T00:48:05.532308+0000","last_unstale":"2024-03-18T00:49:03.089159+0000","last_undegraded":"2024-03-18T00:49:03.089159+0000","last_fullsized":"2024-03-18T00:49:03.089159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:35.071283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.092567+0000","last_change":"2024-03-18T00:49:03.092567+0000","last_active":"2024-03-18T00:49:03.092567+0000","last_peered":"2024-03-18T00:49:03.092567+0000","last_clean":"2024-03-18T00:49:03.092567+0000","last_became_active":"2024-03-18T00:49:03.092297+0000","last_became_peered":"2024-03-18T00:49:03.092297+0000","last_unstale":"2024-03-18T00:49:03.092567+0000","last_undegraded":"2024-03-18T00:49:03.092567+0000","last_fullsized":"2024-03-18T00:49:03.092567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:44.567584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'30","reported_seq":193,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:20.107887+0000","last_change":"2024-03-18T00:48:05.527557+0000","last_active":"2024-03-18T00:49:20.107887+0000","last_peered":"2024-03-18T00:49:20.107887+0000","last_clean":"2024-03-18T00:49:20.107887+0000","last_became_active":"2024-03-18T00:48:05.527167+0000","last_became_peered":"2024-03-18T00:48:05.527167+0000","last_unstale":"2024-03-18T00:49:20.107887+0000","last_undegraded":"2024-03-18T00:49:20.107887+0000","last_fullsized":"2024-03-18T00:49:20.107887+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:02:04.105091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":144,"num_read_kb":132,"num_write":32,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088401+0000","last_change":"2024-03-18T00:48:06.543153+0000","last_active":"2024-03-18T00:49:03.088401+0000","last_peered":"2024-03-18T00:49:03.088401+0000","last_clean":"2024-03-18T00:49:03.088401+0000","last_became_active":"2024-03-18T00:48:06.541353+0000","last_became_peered":"2024-03-18T00:48:06.541353+0000","last_unstale":"2024-03-18T00:49:03.088401+0000","last_undegraded":"2024-03-18T00:49:03.088401+0000","last_fullsized":"2024-03-18T00:49:03.088401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:35.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086981+0000","last_change":"2024-03-18T00:48:05.530560+0000","last_active":"2024-03-18T00:49:03.086981+0000","last_peered":"2024-03-18T00:49:03.086981+0000","last_clean":"2024-03-18T00:49:03.086981+0000","last_became_active":"2024-03-18T00:48:05.528858+0000","last_became_peered":"2024-03-18T00:48:05.528858+0000","last_unstale":"2024-03-18T00:49:03.086981+0000","last_undegraded":"2024-03-18T00:49:03.086981+0000","last_fullsized":"2024-03-18T00:49:03.086981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:03:01.496919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.090951+0000","last_change":"2024-03-18T00:49:03.090951+0000","last_active":"2024-03-18T00:49:03.090951+0000","last_peered":"2024-03-18T00:49:03.090951+0000","last_clean":"2024-03-18T00:49:03.090951+0000","last_became_active":"2024-03-18T00:49:03.090578+0000","last_became_peered":"2024-03-18T00:49:03.090578+0000","last_unstale":"2024-03-18T00:49:03.090951+0000","last_undegraded":"2024-03-18T00:49:03.090951+0000","last_fullsized":"2024-03-18T00:49:03.090951+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:04:12.703545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088756+0000","last_change":"2024-03-18T00:48:05.527902+0000","last_active":"2024-03-18T00:49:03.088756+0000","last_peered":"2024-03-18T00:49:03.088756+0000","last_clean":"2024-03-18T00:49:03.088756+0000","last_became_active":"2024-03-18T00:48:05.527495+0000","last_became_peered":"2024-03-18T00:48:05.527495+0000","last_unstale":"2024-03-18T00:49:03.088756+0000","last_undegraded":"2024-03-18T00:49:03.088756+0000","last_fullsized":"2024-03-18T00:49:03.088756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:25.504652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087171+0000","last_change":"2024-03-18T00:48:06.550413+0000","last_active":"2024-03-18T00:49:03.087171+0000","last_peered":"2024-03-18T00:49:03.087171+0000","last_clean":"2024-03-18T00:49:03.087171+0000","last_became_active":"2024-03-18T00:48:06.547727+0000","last_became_peered":"2024-03-18T00:48:06.547727+0000","last_unstale":"2024-03-18T00:49:03.087171+0000","last_undegraded":"2024-03-18T00:49:03.087171+0000","last_fullsized":"2024-03-18T00:49:03.087171+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:36:48.878417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:20.098103+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:49:20.098103+0000","last_peered":"2024-03-18T00:49:20.098103+0000","last_clean":"2024-03-18T00:49:20.098103+0000","last_became_active":"2024-03-18T00:48:05.525607+0000","last_became_peered":"2024-03-18T00:48:05.525607+0000","last_unstale":"2024-03-18T00:49:20.098103+0000","last_undegraded":"2024-03-18T00:49:20.098103+0000","last_fullsized":"2024-03-18T00:49:20.098103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:32.457447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.091221+0000","last_change":"2024-03-18T00:49:03.091221+0000","last_active":"2024-03-18T00:49:03.091221+0000","last_peered":"2024-03-18T00:49:03.091221+0000","last_clean":"2024-03-18T00:49:03.091221+0000","last_became_active":"2024-03-18T00:49:03.090559+0000","last_became_peered":"2024-03-18T00:49:03.090559+0000","last_unstale":"2024-03-18T00:49:03.091221+0000","last_undegraded":"2024-03-18T00:49:03.091221+0000","last_fullsized":"2024-03-18T00:49:03.091221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:02:54.890187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.417640+0000","last_change":"2024-03-18T00:49:03.417640+0000","last_active":"2024-03-18T00:49:03.417640+0000","last_peered":"2024-03-18T00:49:03.417640+0000","last_clean":"2024-03-18T00:49:03.417640+0000","last_became_active":"2024-03-18T00:49:03.405217+0000","last_became_peered":"2024-03-18T00:49:03.405217+0000","last_unstale":"2024-03-18T00:49:03.417640+0000","last_undegraded":"2024-03-18T00:49:03.417640+0000","last_fullsized":"2024-03-18T00:49:03.417640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:00:13.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088573+0000","last_change":"2024-03-18T00:48:06.544084+0000","last_active":"2024-03-18T00:49:03.088573+0000","last_peered":"2024-03-18T00:49:03.088573+0000","last_clean":"2024-03-18T00:49:03.088573+0000","last_became_active":"2024-03-18T00:48:06.543419+0000","last_became_peered":"2024-03-18T00:48:06.543419+0000","last_unstale":"2024-03-18T00:49:03.088573+0000","last_undegraded":"2024-03-18T00:49:03.088573+0000","last_fullsized":"2024-03-18T00:49:03.088573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:15:28.527473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087635+0000","last_change":"2024-03-18T00:48:05.528347+0000","last_active":"2024-03-18T00:49:03.087635+0000","last_peered":"2024-03-18T00:49:03.087635+0000","last_clean":"2024-03-18T00:49:03.087635+0000","last_became_active":"2024-03-18T00:48:05.527761+0000","last_became_peered":"2024-03-18T00:48:05.527761+0000","last_unstale":"2024-03-18T00:49:03.087635+0000","last_undegraded":"2024-03-18T00:49:03.087635+0000","last_fullsized":"2024-03-18T00:49:03.087635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:36:58.562495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.085039+0000","last_change":"2024-03-18T00:48:06.541862+0000","last_active":"2024-03-18T00:49:03.085039+0000","last_peered":"2024-03-18T00:49:03.085039+0000","last_clean":"2024-03-18T00:49:03.085039+0000","last_became_active":"2024-03-18T00:48:06.541488+0000","last_became_peered":"2024-03-18T00:48:06.541488+0000","last_unstale":"2024-03-18T00:49:03.085039+0000","last_undegraded":"2024-03-18T00:49:03.085039+0000","last_fullsized":"2024-03-18T00:49:03.085039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:47:40.724244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084618+0000","last_change":"2024-03-18T00:48:05.535498+0000","last_active":"2024-03-18T00:49:03.084618+0000","last_peered":"2024-03-18T00:49:03.084618+0000","last_clean":"2024-03-18T00:49:03.084618+0000","last_became_active":"2024-03-18T00:48:05.534907+0000","last_became_peered":"2024-03-18T00:48:05.534907+0000","last_unstale":"2024-03-18T00:49:03.084618+0000","last_undegraded":"2024-03-18T00:49:03.084618+0000","last_fullsized":"2024-03-18T00:49:03.084618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:47:50.196929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088336+0000","last_change":"2024-03-18T00:48:06.534423+0000","last_active":"2024-03-18T00:49:03.088336+0000","last_peered":"2024-03-18T00:49:03.088336+0000","last_clean":"2024-03-18T00:49:03.088336+0000","last_became_active":"2024-03-18T00:48:06.534091+0000","last_became_peered":"2024-03-18T00:48:06.534091+0000","last_unstale":"2024-03-18T00:49:03.088336+0000","last_undegraded":"2024-03-18T00:49:03.088336+0000","last_fullsized":"2024-03-18T00:49:03.088336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:39:08.755634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:21.115501+0000","last_change":"2024-03-18T00:48:05.532891+0000","last_active":"2024-03-18T00:49:21.115501+0000","last_peered":"2024-03-18T00:49:21.115501+0000","last_clean":"2024-03-18T00:49:21.115501+0000","last_became_active":"2024-03-18T00:48:05.532515+0000","last_became_peered":"2024-03-18T00:48:05.532515+0000","last_unstale":"2024-03-18T00:49:21.115501+0000","last_undegraded":"2024-03-18T00:49:21.115501+0000","last_fullsized":"2024-03-18T00:49:21.115501+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:46:52.056837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088513+0000","last_change":"2024-03-18T00:48:06.542992+0000","last_active":"2024-03-18T00:49:03.088513+0000","last_peered":"2024-03-18T00:49:03.088513+0000","last_clean":"2024-03-18T00:49:03.088513+0000","last_became_active":"2024-03-18T00:48:06.542617+0000","last_became_peered":"2024-03-18T00:48:06.542617+0000","last_unstale":"2024-03-18T00:49:03.088513+0000","last_undegraded":"2024-03-18T00:49:03.088513+0000","last_fullsized":"2024-03-18T00:49:03.088513+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:22.736118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088272+0000","last_change":"2024-03-18T00:48:06.536018+0000","last_active":"2024-03-18T00:49:03.088272+0000","last_peered":"2024-03-18T00:49:03.088272+0000","last_clean":"2024-03-18T00:49:03.088272+0000","last_became_active":"2024-03-18T00:48:06.535656+0000","last_became_peered":"2024-03-18T00:48:06.535656+0000","last_unstale":"2024-03-18T00:49:03.088272+0000","last_undegraded":"2024-03-18T00:49:03.088272+0000","last_fullsized":"2024-03-18T00:49:03.088272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:33:00.492887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:21.098749+0000","last_change":"2024-03-18T00:48:05.536635+0000","last_active":"2024-03-18T00:49:21.098749+0000","last_peered":"2024-03-18T00:49:21.098749+0000","last_clean":"2024-03-18T00:49:21.098749+0000","last_became_active":"2024-03-18T00:48:05.535941+0000","last_became_peered":"2024-03-18T00:48:05.535941+0000","last_unstale":"2024-03-18T00:49:21.098749+0000","last_undegraded":"2024-03-18T00:49:21.098749+0000","last_fullsized":"2024-03-18T00:49:21.098749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:20:52.167217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325914+0000","last_change":"2024-03-18T00:48:06.546266+0000","last_active":"2024-03-18T00:49:03.325914+0000","last_peered":"2024-03-18T00:49:03.325914+0000","last_clean":"2024-03-18T00:49:03.325914+0000","last_became_active":"2024-03-18T00:48:06.543565+0000","last_became_peered":"2024-03-18T00:48:06.543565+0000","last_unstale":"2024-03-18T00:49:03.325914+0000","last_undegraded":"2024-03-18T00:49:03.325914+0000","last_fullsized":"2024-03-18T00:49:03.325914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:49:09.673212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.403989+0000","last_change":"2024-03-18T00:48:05.526184+0000","last_active":"2024-03-18T00:49:03.403989+0000","last_peered":"2024-03-18T00:49:03.403989+0000","last_clean":"2024-03-18T00:49:03.403989+0000","last_became_active":"2024-03-18T00:48:05.525011+0000","last_became_peered":"2024-03-18T00:48:05.525011+0000","last_unstale":"2024-03-18T00:49:03.403989+0000","last_undegraded":"2024-03-18T00:49:03.403989+0000","last_fullsized":"2024-03-18T00:49:03.403989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:12.690966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":344,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087104+0000","last_change":"2024-03-18T00:46:53.288768+0000","last_active":"2024-03-18T00:49:03.087104+0000","last_peered":"2024-03-18T00:49:03.087104+0000","last_clean":"2024-03-18T00:49:03.087104+0000","last_became_active":"2024-03-18T00:46:53.288394+0000","last_became_peered":"2024-03-18T00:46:53.288394+0000","last_unstale":"2024-03-18T00:49:03.087104+0000","last_undegraded":"2024-03-18T00:49:03.087104+0000","last_fullsized":"2024-03-18T00:49:03.087104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_clean_scrub_stamp":"2024-03-18T00:43:28.173150+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:49.949940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087727+0000","last_change":"2024-03-18T00:48:06.534383+0000","last_active":"2024-03-18T00:49:03.087727+0000","last_peered":"2024-03-18T00:49:03.087727+0000","last_clean":"2024-03-18T00:49:03.087727+0000","last_became_active":"2024-03-18T00:48:06.534085+0000","last_became_peered":"2024-03-18T00:48:06.534085+0000","last_unstale":"2024-03-18T00:49:03.087727+0000","last_undegraded":"2024-03-18T00:49:03.087727+0000","last_fullsized":"2024-03-18T00:49:03.087727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:54.021830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.407661+0000","last_change":"2024-03-18T00:49:03.407661+0000","last_active":"2024-03-18T00:49:03.407661+0000","last_peered":"2024-03-18T00:49:03.407661+0000","last_clean":"2024-03-18T00:49:03.407661+0000","last_became_active":"2024-03-18T00:49:03.405134+0000","last_became_peered":"2024-03-18T00:49:03.405134+0000","last_unstale":"2024-03-18T00:49:03.407661+0000","last_undegraded":"2024-03-18T00:49:03.407661+0000","last_fullsized":"2024-03-18T00:49:03.407661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:29:03.201556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.098300+0000","last_change":"2024-03-18T00:49:03.097885+0000","last_active":"2024-03-18T00:49:03.098300+0000","last_peered":"2024-03-18T00:49:03.098300+0000","last_clean":"2024-03-18T00:49:03.098300+0000","last_became_active":"2024-03-18T00:49:03.090574+0000","last_became_peered":"2024-03-18T00:49:03.090574+0000","last_unstale":"2024-03-18T00:49:03.098300+0000","last_undegraded":"2024-03-18T00:49:03.098300+0000","last_fullsized":"2024-03-18T00:49:03.098300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:22:30.194349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088082+0000","last_change":"2024-03-18T00:48:06.530108+0000","last_active":"2024-03-18T00:49:03.088082+0000","last_peered":"2024-03-18T00:49:03.088082+0000","last_clean":"2024-03-18T00:49:03.088082+0000","last_became_active":"2024-03-18T00:48:06.529768+0000","last_became_peered":"2024-03-18T00:48:06.529768+0000","last_unstale":"2024-03-18T00:49:03.088082+0000","last_undegraded":"2024-03-18T00:49:03.088082+0000","last_fullsized":"2024-03-18T00:49:03.088082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:23:09.150305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084656+0000","last_change":"2024-03-18T00:48:05.535703+0000","last_active":"2024-03-18T00:49:03.084656+0000","last_peered":"2024-03-18T00:49:03.084656+0000","last_clean":"2024-03-18T00:49:03.084656+0000","last_became_active":"2024-03-18T00:48:05.535086+0000","last_became_peered":"2024-03-18T00:48:05.535086+0000","last_unstale":"2024-03-18T00:49:03.084656+0000","last_undegraded":"2024-03-18T00:49:03.084656+0000","last_fullsized":"2024-03-18T00:49:03.084656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:10:24.172353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325842+0000","last_change":"2024-03-18T00:48:06.541281+0000","last_active":"2024-03-18T00:49:03.325842+0000","last_peered":"2024-03-18T00:49:03.325842+0000","last_clean":"2024-03-18T00:49:03.325842+0000","last_became_active":"2024-03-18T00:48:06.540946+0000","last_became_peered":"2024-03-18T00:48:06.540946+0000","last_unstale":"2024-03-18T00:49:03.325842+0000","last_undegraded":"2024-03-18T00:49:03.325842+0000","last_fullsized":"2024-03-18T00:49:03.325842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:02.987661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.404075+0000","last_change":"2024-03-18T00:48:05.531685+0000","last_active":"2024-03-18T00:49:03.404075+0000","last_peered":"2024-03-18T00:49:03.404075+0000","last_clean":"2024-03-18T00:49:03.404075+0000","last_became_active":"2024-03-18T00:48:05.531239+0000","last_became_peered":"2024-03-18T00:48:05.531239+0000","last_unstale":"2024-03-18T00:49:03.404075+0000","last_undegraded":"2024-03-18T00:49:03.404075+0000","last_fullsized":"2024-03-18T00:49:03.404075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:26:35.034842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086824+0000","last_change":"2024-03-18T00:48:06.546783+0000","last_active":"2024-03-18T00:49:03.086824+0000","last_peered":"2024-03-18T00:49:03.086824+0000","last_clean":"2024-03-18T00:49:03.086824+0000","last_became_active":"2024-03-18T00:48:06.546411+0000","last_became_peered":"2024-03-18T00:48:06.546411+0000","last_unstale":"2024-03-18T00:49:03.086824+0000","last_undegraded":"2024-03-18T00:49:03.086824+0000","last_fullsized":"2024-03-18T00:49:03.086824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:39:34.432104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089245+0000","last_change":"2024-03-18T00:48:05.534701+0000","last_active":"2024-03-18T00:49:03.089245+0000","last_peered":"2024-03-18T00:49:03.089245+0000","last_clean":"2024-03-18T00:49:03.089245+0000","last_became_active":"2024-03-18T00:48:05.534300+0000","last_became_peered":"2024-03-18T00:48:05.534300+0000","last_unstale":"2024-03-18T00:49:03.089245+0000","last_undegraded":"2024-03-18T00:49:03.089245+0000","last_fullsized":"2024-03-18T00:49:03.089245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:10.580001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086905+0000","last_change":"2024-03-18T00:48:06.528286+0000","last_active":"2024-03-18T00:49:03.086905+0000","last_peered":"2024-03-18T00:49:03.086905+0000","last_clean":"2024-03-18T00:49:03.086905+0000","last_became_active":"2024-03-18T00:48:06.527947+0000","last_became_peered":"2024-03-18T00:48:06.527947+0000","last_unstale":"2024-03-18T00:49:03.086905+0000","last_undegraded":"2024-03-18T00:49:03.086905+0000","last_fullsized":"2024-03-18T00:49:03.086905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:34.953923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325566+0000","last_change":"2024-03-18T00:48:05.526530+0000","last_active":"2024-03-18T00:49:03.325566+0000","last_peered":"2024-03-18T00:49:03.325566+0000","last_clean":"2024-03-18T00:49:03.325566+0000","last_became_active":"2024-03-18T00:48:05.526210+0000","last_became_peered":"2024-03-18T00:48:05.526210+0000","last_unstale":"2024-03-18T00:49:03.325566+0000","last_undegraded":"2024-03-18T00:49:03.325566+0000","last_fullsized":"2024-03-18T00:49:03.325566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:11:11.094469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084783+0000","last_change":"2024-03-18T00:48:06.535316+0000","last_active":"2024-03-18T00:49:03.084783+0000","last_peered":"2024-03-18T00:49:03.084783+0000","last_clean":"2024-03-18T00:49:03.084783+0000","last_became_active":"2024-03-18T00:48:06.534945+0000","last_became_peered":"2024-03-18T00:48:06.534945+0000","last_unstale":"2024-03-18T00:49:03.084783+0000","last_undegraded":"2024-03-18T00:49:03.084783+0000","last_fullsized":"2024-03-18T00:49:03.084783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:04:32.570585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087264+0000","last_change":"2024-03-18T00:48:05.529300+0000","last_active":"2024-03-18T00:49:03.087264+0000","last_peered":"2024-03-18T00:49:03.087264+0000","last_clean":"2024-03-18T00:49:03.087264+0000","last_became_active":"2024-03-18T00:48:05.528935+0000","last_became_peered":"2024-03-18T00:48:05.528935+0000","last_unstale":"2024-03-18T00:49:03.087264+0000","last_undegraded":"2024-03-18T00:49:03.087264+0000","last_fullsized":"2024-03-18T00:49:03.087264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:29:45.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325587+0000","last_change":"2024-03-18T00:48:06.536426+0000","last_active":"2024-03-18T00:49:03.325587+0000","last_peered":"2024-03-18T00:49:03.325587+0000","last_clean":"2024-03-18T00:49:03.325587+0000","last_became_active":"2024-03-18T00:48:06.536005+0000","last_became_peered":"2024-03-18T00:48:06.536005+0000","last_unstale":"2024-03-18T00:49:03.325587+0000","last_undegraded":"2024-03-18T00:49:03.325587+0000","last_fullsized":"2024-03-18T00:49:03.325587+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:40:52.028794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087438+0000","last_change":"2024-03-18T00:48:05.536495+0000","last_active":"2024-03-18T00:49:03.087438+0000","last_peered":"2024-03-18T00:49:03.087438+0000","last_clean":"2024-03-18T00:49:03.087438+0000","last_became_active":"2024-03-18T00:48:05.536072+0000","last_became_peered":"2024-03-18T00:48:05.536072+0000","last_unstale":"2024-03-18T00:49:03.087438+0000","last_undegraded":"2024-03-18T00:49:03.087438+0000","last_fullsized":"2024-03-18T00:49:03.087438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:25.528261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087389+0000","last_change":"2024-03-18T00:48:06.552900+0000","last_active":"2024-03-18T00:49:03.087389+0000","last_peered":"2024-03-18T00:49:03.087389+0000","last_clean":"2024-03-18T00:49:03.087389+0000","last_became_active":"2024-03-18T00:48:06.552568+0000","last_became_peered":"2024-03-18T00:48:06.552568+0000","last_unstale":"2024-03-18T00:49:03.087389+0000","last_undegraded":"2024-03-18T00:49:03.087389+0000","last_fullsized":"2024-03-18T00:49:03.087389+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:53:25.254177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:20.102766+0000","last_change":"2024-03-18T00:48:05.537327+0000","last_active":"2024-03-18T00:49:20.102766+0000","last_peered":"2024-03-18T00:49:20.102766+0000","last_clean":"2024-03-18T00:49:20.102766+0000","last_became_active":"2024-03-18T00:48:05.536825+0000","last_became_peered":"2024-03-18T00:48:05.536825+0000","last_unstale":"2024-03-18T00:49:20.102766+0000","last_undegraded":"2024-03-18T00:49:20.102766+0000","last_fullsized":"2024-03-18T00:49:20.102766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:03.141142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087502+0000","last_change":"2024-03-18T00:48:06.542593+0000","last_active":"2024-03-18T00:49:03.087502+0000","last_peered":"2024-03-18T00:49:03.087502+0000","last_clean":"2024-03-18T00:49:03.087502+0000","last_became_active":"2024-03-18T00:48:06.541624+0000","last_became_peered":"2024-03-18T00:48:06.541624+0000","last_unstale":"2024-03-18T00:49:03.087502+0000","last_undegraded":"2024-03-18T00:49:03.087502+0000","last_fullsized":"2024-03-18T00:49:03.087502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:10:56.905221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089025+0000","last_change":"2024-03-18T00:48:05.529140+0000","last_active":"2024-03-18T00:49:03.089025+0000","last_peered":"2024-03-18T00:49:03.089025+0000","last_clean":"2024-03-18T00:49:03.089025+0000","last_became_active":"2024-03-18T00:48:05.528682+0000","last_became_peered":"2024-03-18T00:48:05.528682+0000","last_unstale":"2024-03-18T00:49:03.089025+0000","last_undegraded":"2024-03-18T00:49:03.089025+0000","last_fullsized":"2024-03-18T00:49:03.089025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:57:52.663150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087272+0000","last_change":"2024-03-18T00:48:06.546027+0000","last_active":"2024-03-18T00:49:03.087272+0000","last_peered":"2024-03-18T00:49:03.087272+0000","last_clean":"2024-03-18T00:49:03.087272+0000","last_became_active":"2024-03-18T00:48:06.545670+0000","last_became_peered":"2024-03-18T00:48:06.545670+0000","last_unstale":"2024-03-18T00:49:03.087272+0000","last_undegraded":"2024-03-18T00:49:03.087272+0000","last_fullsized":"2024-03-18T00:49:03.087272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:47:05.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087890+0000","last_change":"2024-03-18T00:48:05.526929+0000","last_active":"2024-03-18T00:49:03.087890+0000","last_peered":"2024-03-18T00:49:03.087890+0000","last_clean":"2024-03-18T00:49:03.087890+0000","last_became_active":"2024-03-18T00:48:05.526530+0000","last_became_peered":"2024-03-18T00:48:05.526530+0000","last_unstale":"2024-03-18T00:49:03.087890+0000","last_undegraded":"2024-03-18T00:49:03.087890+0000","last_fullsized":"2024-03-18T00:49:03.087890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:51:25.110895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087753+0000","last_change":"2024-03-18T00:48:06.541223+0000","last_active":"2024-03-18T00:49:03.087753+0000","last_peered":"2024-03-18T00:49:03.087753+0000","last_clean":"2024-03-18T00:49:03.087753+0000","last_became_active":"2024-03-18T00:48:06.540898+0000","last_became_peered":"2024-03-18T00:48:06.540898+0000","last_unstale":"2024-03-18T00:49:03.087753+0000","last_undegraded":"2024-03-18T00:49:03.087753+0000","last_fullsized":"2024-03-18T00:49:03.087753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:20:22.749609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084879+0000","last_change":"2024-03-18T00:48:06.532906+0000","last_active":"2024-03-18T00:49:03.084879+0000","last_peered":"2024-03-18T00:49:03.084879+0000","last_clean":"2024-03-18T00:49:03.084879+0000","last_became_active":"2024-03-18T00:48:06.532583+0000","last_became_peered":"2024-03-18T00:48:06.532583+0000","last_unstale":"2024-03-18T00:49:03.084879+0000","last_undegraded":"2024-03-18T00:49:03.084879+0000","last_fullsized":"2024-03-18T00:49:03.084879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:06.699615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087137+0000","last_change":"2024-03-18T00:48:06.530020+0000","last_active":"2024-03-18T00:49:03.087137+0000","last_peered":"2024-03-18T00:49:03.087137+0000","last_clean":"2024-03-18T00:49:03.087137+0000","last_became_active":"2024-03-18T00:48:06.529709+0000","last_became_peered":"2024-03-18T00:48:06.529709+0000","last_unstale":"2024-03-18T00:49:03.087137+0000","last_undegraded":"2024-03-18T00:49:03.087137+0000","last_fullsized":"2024-03-18T00:49:03.087137+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:07.305537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087254+0000","last_change":"2024-03-18T00:48:06.537658+0000","last_active":"2024-03-18T00:49:03.087254+0000","last_peered":"2024-03-18T00:49:03.087254+0000","last_clean":"2024-03-18T00:49:03.087254+0000","last_became_active":"2024-03-18T00:48:06.537339+0000","last_became_peered":"2024-03-18T00:48:06.537339+0000","last_unstale":"2024-03-18T00:49:03.087254+0000","last_undegraded":"2024-03-18T00:49:03.087254+0000","last_fullsized":"2024-03-18T00:49:03.087254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:18:25.753168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086737+0000","last_change":"2024-03-18T00:48:06.544514+0000","last_active":"2024-03-18T00:49:03.086737+0000","last_peered":"2024-03-18T00:49:03.086737+0000","last_clean":"2024-03-18T00:49:03.086737+0000","last_became_active":"2024-03-18T00:48:06.544165+0000","last_became_peered":"2024-03-18T00:48:06.544165+0000","last_unstale":"2024-03-18T00:49:03.086737+0000","last_undegraded":"2024-03-18T00:49:03.086737+0000","last_fullsized":"2024-03-18T00:49:03.086737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:48:02.498940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.090610+0000","last_change":"2024-03-18T00:49:03.090610+0000","last_active":"2024-03-18T00:49:03.090610+0000","last_peered":"2024-03-18T00:49:03.090610+0000","last_clean":"2024-03-18T00:49:03.090610+0000","last_became_active":"2024-03-18T00:49:03.090014+0000","last_became_peered":"2024-03-18T00:49:03.090014+0000","last_unstale":"2024-03-18T00:49:03.090610+0000","last_undegraded":"2024-03-18T00:49:03.090610+0000","last_fullsized":"2024-03-18T00:49:03.090610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:55:12.210223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.085194+0000","last_change":"2024-03-18T00:48:06.533837+0000","last_active":"2024-03-18T00:49:03.085194+0000","last_peered":"2024-03-18T00:49:03.085194+0000","last_clean":"2024-03-18T00:49:03.085194+0000","last_became_active":"2024-03-18T00:48:06.533418+0000","last_became_peered":"2024-03-18T00:48:06.533418+0000","last_unstale":"2024-03-18T00:49:03.085194+0000","last_undegraded":"2024-03-18T00:49:03.085194+0000","last_fullsized":"2024-03-18T00:49:03.085194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:10.873312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087070+0000","last_change":"2024-03-18T00:48:06.527394+0000","last_active":"2024-03-18T00:49:03.087070+0000","last_peered":"2024-03-18T00:49:03.087070+0000","last_clean":"2024-03-18T00:49:03.087070+0000","last_became_active":"2024-03-18T00:48:06.526879+0000","last_became_peered":"2024-03-18T00:48:06.526879+0000","last_unstale":"2024-03-18T00:49:03.087070+0000","last_undegraded":"2024-03-18T00:49:03.087070+0000","last_fullsized":"2024-03-18T00:49:03.087070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:23:07.397259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325740+0000","last_change":"2024-03-18T00:48:06.542645+0000","last_active":"2024-03-18T00:49:03.325740+0000","last_peered":"2024-03-18T00:49:03.325740+0000","last_clean":"2024-03-18T00:49:03.325740+0000","last_became_active":"2024-03-18T00:48:06.542073+0000","last_became_peered":"2024-03-18T00:48:06.542073+0000","last_unstale":"2024-03-18T00:49:03.325740+0000","last_undegraded":"2024-03-18T00:49:03.325740+0000","last_fullsized":"2024-03-18T00:49:03.325740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:08:44.479318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088173+0000","last_change":"2024-03-18T00:48:06.541698+0000","last_active":"2024-03-18T00:49:03.088173+0000","last_peered":"2024-03-18T00:49:03.088173+0000","last_clean":"2024-03-18T00:49:03.088173+0000","last_became_active":"2024-03-18T00:48:06.541378+0000","last_became_peered":"2024-03-18T00:48:06.541378+0000","last_unstale":"2024-03-18T00:49:03.088173+0000","last_undegraded":"2024-03-18T00:49:03.088173+0000","last_fullsized":"2024-03-18T00:49:03.088173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:42:23.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325481+0000","last_change":"2024-03-18T00:48:06.533184+0000","last_active":"2024-03-18T00:49:03.325481+0000","last_peered":"2024-03-18T00:49:03.325481+0000","last_clean":"2024-03-18T00:49:03.325481+0000","last_became_active":"2024-03-18T00:48:06.532034+0000","last_became_peered":"2024-03-18T00:48:06.532034+0000","last_unstale":"2024-03-18T00:49:03.325481+0000","last_undegraded":"2024-03-18T00:49:03.325481+0000","last_fullsized":"2024-03-18T00:49:03.325481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:45.265799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.090002+0000","last_change":"2024-03-18T00:48:06.545941+0000","last_active":"2024-03-18T00:49:03.090002+0000","last_peered":"2024-03-18T00:49:03.090002+0000","last_clean":"2024-03-18T00:49:03.090002+0000","last_became_active":"2024-03-18T00:48:06.545402+0000","last_became_peered":"2024-03-18T00:48:06.545402+0000","last_unstale":"2024-03-18T00:49:03.090002+0000","last_undegraded":"2024-03-18T00:49:03.090002+0000","last_fullsized":"2024-03-18T00:49:03.090002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:11:45.512849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087949+0000","last_change":"2024-03-18T00:48:06.541513+0000","last_active":"2024-03-18T00:49:03.087949+0000","last_peered":"2024-03-18T00:49:03.087949+0000","last_clean":"2024-03-18T00:49:03.087949+0000","last_became_active":"2024-03-18T00:48:06.541128+0000","last_became_peered":"2024-03-18T00:48:06.541128+0000","last_unstale":"2024-03-18T00:49:03.087949+0000","last_undegraded":"2024-03-18T00:49:03.087949+0000","last_fullsized":"2024-03-18T00:49:03.087949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:42:10.635836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084786+0000","last_change":"2024-03-18T00:48:06.542316+0000","last_active":"2024-03-18T00:49:03.084786+0000","last_peered":"2024-03-18T00:49:03.084786+0000","last_clean":"2024-03-18T00:49:03.084786+0000","last_became_active":"2024-03-18T00:48:06.541914+0000","last_became_peered":"2024-03-18T00:48:06.541914+0000","last_unstale":"2024-03-18T00:49:03.084786+0000","last_undegraded":"2024-03-18T00:49:03.084786+0000","last_fullsized":"2024-03-18T00:49:03.084786+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:46.293240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249372,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67859,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":678,"num_read_kb":436,"num_write":236,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1237,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":203577,"data_compressed":5589,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":461165,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":1112115,"data_compressed":43768,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776166,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":459970,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037805,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":970752,"data_stored":460236,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299442,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1052866,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462967,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":459687,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574928,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1024000,"data_stored":521646,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":987136,"data_stored":461026,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":522145,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":459512,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888128,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":459660,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":1051531,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62798,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":63297,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":62782,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T00:49:25.590 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-18T00:49:25.590 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T00:49:25.590 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-18T00:49:25.590 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T00:49:25.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:25 smithi106 ceph-mon[35899]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:25.843 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:26.565 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:26 smithi049 ceph-mon[32130]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:49:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:26 smithi107 ceph-mon[36270]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:49:26.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:26 smithi106 ceph-mon[35899]: from='client.24829 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:49:27.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.471+0000 7fc95919f700 1 -- 172.21.15.49:0/2205668444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc954101e40 msgr2=0x7fc954105e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:27.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.471+0000 7fc95919f700 1 --2- 172.21.15.49:0/2205668444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc954101e40 0x7fc954105e00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc9440099c0 tx=0x7fc944009cd0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.471+0000 7fc95919f700 1 -- 172.21.15.49:0/2205668444 shutdown_connections 2024-03-18T00:49:27.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.471+0000 7fc95919f700 1 --2- 172.21.15.49:0/2205668444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9541064e0 0x7fc95410e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.471+0000 7fc95919f700 1 --2- 172.21.15.49:0/2205668444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc954101490 0x7fc954101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.471+0000 7fc95919f700 1 --2- 172.21.15.49:0/2205668444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc954101e40 0x7fc954105e00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.471+0000 7fc95919f700 1 -- 172.21.15.49:0/2205668444 >> 172.21.15.49:0/2205668444 conn(0x7fc954077220 msgr2=0x7fc954077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:27.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.471+0000 7fc95919f700 1 -- 172.21.15.49:0/2205668444 shutdown_connections 2024-03-18T00:49:27.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.471+0000 7fc95919f700 1 -- 172.21.15.49:0/2205668444 wait complete. 2024-03-18T00:49:27.474 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:27 smithi049 ceph-mon[32130]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T00:49:27.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.472+0000 7fc95919f700 1 Processor -- start 2024-03-18T00:49:27.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95919f700 1 -- start start 2024-03-18T00:49:27.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95919f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc954101490 0x7fc954102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:27.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95919f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc954101e40 0x7fc9541033e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:27.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95919f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9541064e0 0x7fc954103920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:27.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95919f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc9541114c0 con 0x7fc954101e40 2024-03-18T00:49:27.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95919f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc954111340 con 0x7fc9541064e0 2024-03-18T00:49:27.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95919f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc954111640 con 0x7fc954101490 2024-03-18T00:49:27.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc954101e40 0x7fc9541033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95359e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9541064e0 0x7fc954103920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc952d9d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc954101490 0x7fc954102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95359e700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9541064e0 0x7fc954103920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:49484/0 (socket says 172.21.15.49:49484) 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95259c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc954101e40 0x7fc9541033e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55246/0 (socket says 172.21.15.49:55246) 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95359e700 1 -- 172.21.15.49:0/2040018782 learned_addr learned my addr 172.21.15.49:0/2040018782 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc952d9d700 1 -- 172.21.15.49:0/2040018782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9541064e0 msgr2=0x7fc954103920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc952d9d700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9541064e0 0x7fc954103920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc952d9d700 1 -- 172.21.15.49:0/2040018782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc954101e40 msgr2=0x7fc9541033e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc952d9d700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc954101e40 0x7fc9541033e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc952d9d700 1 -- 172.21.15.49:0/2040018782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9541a6f90 con 0x7fc954101490 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.473+0000 7fc95359e700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9541064e0 0x7fc954103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:27.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.474+0000 7fc952d9d700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc954101490 0x7fc954102ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00e9f0 tx=0x7fc93c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:27.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.474+0000 7fc94bfff700 1 -- 172.21.15.49:0/2040018782 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c00c9c0 con 0x7fc954101490 2024-03-18T00:49:27.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.474+0000 7fc94bfff700 1 -- 172.21.15.49:0/2040018782 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc93c00cb60 con 0x7fc954101490 2024-03-18T00:49:27.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.474+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9541a7280 con 0x7fc954101490 2024-03-18T00:49:27.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.474+0000 7fc94bfff700 1 -- 172.21.15.49:0/2040018782 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93c018680 con 0x7fc954101490 2024-03-18T00:49:27.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.474+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9541a7790 con 0x7fc954101490 2024-03-18T00:49:27.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.475+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc954068f90 con 0x7fc954101490 2024-03-18T00:49:27.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.476+0000 7fc94bfff700 1 -- 172.21.15.49:0/2040018782 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc93c047a90 con 0x7fc954101490 2024-03-18T00:49:27.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.477+0000 7fc94bfff700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9400a2920 0x7fc9400a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:27.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.477+0000 7fc95259c700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9400a2920 0x7fc9400a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:27.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.477+0000 7fc94bfff700 1 -- 172.21.15.49:0/2040018782 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc93c0ced30 con 0x7fc954101490 2024-03-18T00:49:27.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.477+0000 7fc95259c700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9400a2920 0x7fc9400a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc944000c00 tx=0x7fc944005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:27.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.485+0000 7fc94bfff700 1 -- 172.21.15.49:0/2040018782 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc93c09bbf0 con 0x7fc954101490 2024-03-18T00:49:27.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:27 smithi107 ceph-mon[36270]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T00:49:27.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:27 smithi106 ceph-mon[35899]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T00:49:27.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.735+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc9541a7a60 con 0x7fc954101490 2024-03-18T00:49:27.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.735+0000 7fc94bfff700 1 -- 172.21.15.49:0/2040018782 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fc93c09b4f0 con 0x7fc954101490 2024-03-18T00:49:27.737 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:27.737 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9400a2920 msgr2=0x7fc9400a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9400a2920 0x7fc9400a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc944000c00 tx=0x7fc944005fb0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc954101490 msgr2=0x7fc954102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc954101490 0x7fc954102ea0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00e9f0 tx=0x7fc93c00ed00 comp rx=0 tx=0).stop 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 shutdown_connections 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc9400a2920 0x7fc9400a4de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc954101490 0x7fc954102ea0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc9541064e0 0x7fc954103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 --2- 172.21.15.49:0/2040018782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc954101e40 0x7fc9541033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 >> 172.21.15.49:0/2040018782 conn(0x7fc954077220 msgr2=0x7fc954105080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 shutdown_connections 2024-03-18T00:49:27.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:27.738+0000 7fc95919f700 1 -- 172.21.15.49:0/2040018782 wait complete. 2024-03-18T00:49:28.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:28 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2040018782' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T00:49:28.428 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-18T00:49:28.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:49:28.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.544+0000 7f82d9fe6640 1 -- 172.21.15.49:0/2437092277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d41018f0 msgr2=0x7f82d4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:28.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.544+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/2437092277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d41018f0 0x7f82d4101d70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f82c4009860 tx=0x7f82c402bb70 comp rx=0 tx=0).stop 2024-03-18T00:49:28.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.544+0000 7f82d9fe6640 1 -- 172.21.15.49:0/2437092277 shutdown_connections 2024-03-18T00:49:28.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.544+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/2437092277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d41018f0 0x7f82d4101d70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.544+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/2437092277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d41022b0 0x7f82d4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.544+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/2437092277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82d4100700 0x7f82d4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.544+0000 7f82d9fe6640 1 -- 172.21.15.49:0/2437092277 >> 172.21.15.49:0/2437092277 conn(0x7f82d40fbc70 msgr2=0x7f82d40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.544+0000 7f82d9fe6640 1 -- 172.21.15.49:0/2437092277 shutdown_connections 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.544+0000 7f82d9fe6640 1 -- 172.21.15.49:0/2437092277 wait complete. 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d9fe6640 1 Processor -- start 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d9fe6640 1 -- start start 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d9fe6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82d4100700 0x7f82d410b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d9fe6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d41018f0 0x7f82d41096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d9fe6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d41022b0 0x7f82d4109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d9fe6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f82d410d4a0 con 0x7f82d4100700 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d9fe6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f82d410d320 con 0x7f82d41022b0 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d9fe6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f82d410d620 con 0x7f82d41018f0 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82d4100700 0x7f82d410b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d41018f0 0x7f82d41096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82d4100700 0x7f82d410b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52718/0 (socket says 172.21.15.49:52718) 2024-03-18T00:49:28.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d37fe640 1 -- 172.21.15.49:0/1439676306 learned_addr learned my addr 172.21.15.49:0/1439676306 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d3fff640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d41022b0 0x7f82d4109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d37fe640 1 -- 172.21.15.49:0/1439676306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d41018f0 msgr2=0x7f82d41096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d37fe640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d41018f0 0x7f82d41096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d37fe640 1 -- 172.21.15.49:0/1439676306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d41022b0 msgr2=0x7f82d4109c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d37fe640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d41022b0 0x7f82d4109c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d37fe640 1 -- 172.21.15.49:0/1439676306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d410a490 con 0x7f82d4100700 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.545+0000 7f82d2ffd640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d41018f0 0x7f82d41096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.546+0000 7f82d37fe640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82d4100700 0x7f82d410b080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00e9a0 tx=0x7f82bc00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.546+0000 7f82d0ff9640 1 -- 172.21.15.49:0/1439676306 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82bc00cd80 con 0x7f82d4100700 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.546+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d41a1c40 con 0x7f82d4100700 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.546+0000 7f82d0ff9640 1 -- 172.21.15.49:0/1439676306 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82bc010910 con 0x7f82d4100700 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.546+0000 7f82d0ff9640 1 -- 172.21.15.49:0/1439676306 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82bc018a40 con 0x7f82d4100700 2024-03-18T00:49:28.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.546+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d41a2160 con 0x7f82d4100700 2024-03-18T00:49:28.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.547+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d40685d0 con 0x7f82d4100700 2024-03-18T00:49:28.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.548+0000 7f82d0ff9640 1 -- 172.21.15.49:0/1439676306 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f82bc010ac0 con 0x7f82d4100700 2024-03-18T00:49:28.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.548+0000 7f82d0ff9640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82c00a2890 0x7f82c00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:28.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.549+0000 7f82d2ffd640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82c00a2890 0x7f82c00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:28.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.549+0000 7f82d0ff9640 1 -- 172.21.15.49:0/1439676306 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f82bc0cef00 con 0x7f82d4100700 2024-03-18T00:49:28.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.549+0000 7f82d2ffd640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82c00a2890 0x7f82c00a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f82c402c180 tx=0x7f82c402e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:28.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.553+0000 7f82d0ff9640 1 -- 172.21.15.49:0/1439676306 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f82bc053740 con 0x7f82d4100700 2024-03-18T00:49:28.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:28 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2040018782' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T00:49:28.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:28 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2040018782' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T00:49:28.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.740+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82d4062410 con 0x7f82d4100700 2024-03-18T00:49:28.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.742+0000 7f82d0ff9640 1 -- 172.21.15.49:0/1439676306 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7f82bc09be20 con 0x7f82d4100700 2024-03-18T00:49:28.742 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:28.742 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:49:28.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82c00a2890 msgr2=0x7f82c00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:28.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82c00a2890 0x7f82c00a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f82c402c180 tx=0x7f82c402e040 comp rx=0 tx=0).stop 2024-03-18T00:49:28.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82d4100700 msgr2=0x7f82d410b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:28.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82d4100700 0x7f82d410b080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00e9a0 tx=0x7f82bc00ee60 comp rx=0 tx=0).stop 2024-03-18T00:49:28.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 shutdown_connections 2024-03-18T00:49:28.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82c00a2890 0x7f82c00a4d50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82d41022b0 0x7f82d4109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82d41018f0 0x7f82d41096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 --2- 172.21.15.49:0/1439676306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82d4100700 0x7f82d410b080 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.744+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 >> 172.21.15.49:0/1439676306 conn(0x7f82d40fbc70 msgr2=0x7f82d4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:28.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.745+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 shutdown_connections 2024-03-18T00:49:28.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.745+0000 7f82d9fe6640 1 -- 172.21.15.49:0/1439676306 wait complete. 2024-03-18T00:49:28.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T00:48:04.316609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T00:48:05.271815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:49:28.757 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:49:28.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.871+0000 7f4d66752640 1 -- 172.21.15.49:0/2394385744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d60102e00 msgr2=0x7f4d60103220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:28.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.871+0000 7f4d66752640 1 --2- 172.21.15.49:0/2394385744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d60102e00 0x7f4d60103220 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009860 tx=0x7f4d5002bb70 comp rx=0 tx=0).stop 2024-03-18T00:49:28.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.871+0000 7f4d66752640 1 -- 172.21.15.49:0/2394385744 shutdown_connections 2024-03-18T00:49:28.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.871+0000 7f4d66752640 1 --2- 172.21.15.49:0/2394385744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d60100230 0x7f4d60109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.871+0000 7f4d66752640 1 --2- 172.21.15.49:0/2394385744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d600ff870 0x7f4d600ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.871+0000 7f4d66752640 1 --2- 172.21.15.49:0/2394385744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d60102e00 0x7f4d60103220 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.871+0000 7f4d66752640 1 -- 172.21.15.49:0/2394385744 >> 172.21.15.49:0/2394385744 conn(0x7f4d600fb3d0 msgr2=0x7f4d600fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.871+0000 7f4d66752640 1 -- 172.21.15.49:0/2394385744 shutdown_connections 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.871+0000 7f4d66752640 1 -- 172.21.15.49:0/2394385744 wait complete. 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.872+0000 7f4d66752640 1 Processor -- start 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.872+0000 7f4d66752640 1 -- start start 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.872+0000 7f4d66752640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d600ff870 0x7f4d60197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.872+0000 7f4d66752640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d60100230 0x7f4d601980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.872+0000 7f4d66752640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d60102e00 0x7f4d6019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.872+0000 7f4d66752640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6010d430 con 0x7f4d600ff870 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.872+0000 7f4d66752640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6010d2b0 con 0x7f4d60100230 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.872+0000 7f4d66752640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6010d5b0 con 0x7f4d60102e00 2024-03-18T00:49:28.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d600ff870 0x7f4d60197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d600ff870 0x7f4d60197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52734/0 (socket says 172.21.15.49:52734) 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5ffff640 1 -- 172.21.15.49:0/44415158 learned_addr learned my addr 172.21.15.49:0/44415158 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5f7fe640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d60100230 0x7f4d601980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d64cef640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d60102e00 0x7f4d6019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5ffff640 1 -- 172.21.15.49:0/44415158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d60102e00 msgr2=0x7f4d6019f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5ffff640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d60102e00 0x7f4d6019f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5ffff640 1 -- 172.21.15.49:0/44415158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d60100230 msgr2=0x7f4d601980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5ffff640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d60100230 0x7f4d601980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5ffff640 1 -- 172.21.15.49:0/44415158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6019f780 con 0x7f4d600ff870 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5f7fe640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d60100230 0x7f4d601980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5ffff640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d600ff870 0x7f4d60197b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4d500349e0 tx=0x7f4d50032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:28.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5d7fa640 1 -- 172.21.15.49:0/44415158 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d50041020 con 0x7f4d600ff870 2024-03-18T00:49:28.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d6019fa10 con 0x7f4d600ff870 2024-03-18T00:49:28.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5d7fa640 1 -- 172.21.15.49:0/44415158 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d5002c9f0 con 0x7f4d600ff870 2024-03-18T00:49:28.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d5d7fa640 1 -- 172.21.15.49:0/44415158 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d50039040 con 0x7f4d600ff870 2024-03-18T00:49:28.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.873+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d6019ff00 con 0x7f4d600ff870 2024-03-18T00:49:28.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.874+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d60108db0 con 0x7f4d600ff870 2024-03-18T00:49:28.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.876+0000 7f4d5d7fa640 1 -- 172.21.15.49:0/44415158 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4d50044020 con 0x7f4d600ff870 2024-03-18T00:49:28.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.876+0000 7f4d5d7fa640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4d480a2890 0x7f4d480a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:28.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.876+0000 7f4d5f7fe640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4d480a2890 0x7f4d480a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:28.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.876+0000 7f4d5d7fa640 1 -- 172.21.15.49:0/44415158 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4d500f0710 con 0x7f4d600ff870 2024-03-18T00:49:28.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.877+0000 7f4d5f7fe640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4d480a2890 0x7f4d480a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4d540095f0 tx=0x7f4d54005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:28.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:28.880+0000 7f4d5d7fa640 1 -- 172.21.15.49:0/44415158 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d5004b020 con 0x7f4d600ff870 2024-03-18T00:49:29.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.062+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4d6002d340 con 0x7f4d600ff870 2024-03-18T00:49:29.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.063+0000 7f4d5d7fa640 1 -- 172.21.15.49:0/44415158 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f4d500bd740 con 0x7f4d600ff870 2024-03-18T00:49:29.063 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4d480a2890 msgr2=0x7f4d480a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4d480a2890 0x7f4d480a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4d540095f0 tx=0x7f4d54005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d600ff870 msgr2=0x7f4d60197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d600ff870 0x7f4d60197b60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4d500349e0 tx=0x7f4d50032430 comp rx=0 tx=0).stop 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 shutdown_connections 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4d480a2890 0x7f4d480a4d50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4d60100230 0x7f4d601980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d60102e00 0x7f4d6019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 --2- 172.21.15.49:0/44415158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4d600ff870 0x7f4d60197b60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 >> 172.21.15.49:0/44415158 conn(0x7f4d600fb3d0 msgr2=0x7f4d60105f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 shutdown_connections 2024-03-18T00:49:29.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.065+0000 7f4d66752640 1 -- 172.21.15.49:0/44415158 wait complete. 2024-03-18T00:49:29.076 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T00:49:29.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.190+0000 7f15c4a12640 1 -- 172.21.15.49:0/2962453136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 msgr2=0x7f15c0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.190+0000 7f15c4a12640 1 --2- 172.21.15.49:0/2962453136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 0x7f15c0100b00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f15b0009900 tx=0x7f15b002bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:29.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.190+0000 7f15c4a12640 1 -- 172.21.15.49:0/2962453136 shutdown_connections 2024-03-18T00:49:29.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.190+0000 7f15c4a12640 1 --2- 172.21.15.49:0/2962453136 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15c0102290 0x7f15c0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.190+0000 7f15c4a12640 1 --2- 172.21.15.49:0/2962453136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15c01018d0 0x7f15c0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.190+0000 7f15c4a12640 1 --2- 172.21.15.49:0/2962453136 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 0x7f15c0100b00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.190+0000 7f15c4a12640 1 -- 172.21.15.49:0/2962453136 >> 172.21.15.49:0/2962453136 conn(0x7f15c00fbc70 msgr2=0x7f15c00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:29.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.190+0000 7f15c4a12640 1 -- 172.21.15.49:0/2962453136 shutdown_connections 2024-03-18T00:49:29.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.190+0000 7f15c4a12640 1 -- 172.21.15.49:0/2962453136 wait complete. 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15c4a12640 1 Processor -- start 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15c4a12640 1 -- start start 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15c4a12640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 0x7f15c0197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15c4a12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15c01018d0 0x7f15c01980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15c4a12640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15c0102290 0x7f15c019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15c4a12640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f15c010c350 con 0x7f15c01006e0 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15c4a12640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f15c010c1d0 con 0x7f15c0102290 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15c4a12640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f15c010c4d0 con 0x7f15c01018d0 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15be59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 0x7f15c0197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15be59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 0x7f15c0197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52750/0 (socket says 172.21.15.49:52750) 2024-03-18T00:49:29.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15be59c640 1 -- 172.21.15.49:0/55253321 learned_addr learned my addr 172.21.15.49:0/55253321 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15bdd9b640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15c01018d0 0x7f15c01980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15bed9d640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15c0102290 0x7f15c019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15be59c640 1 -- 172.21.15.49:0/55253321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15c01018d0 msgr2=0x7f15c01980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15be59c640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15c01018d0 0x7f15c01980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15be59c640 1 -- 172.21.15.49:0/55253321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15c0102290 msgr2=0x7f15c019f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15be59c640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15c0102290 0x7f15c019f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.191+0000 7f15be59c640 1 -- 172.21.15.49:0/55253321 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15c019f790 con 0x7f15c01006e0 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.192+0000 7f15bed9d640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15c0102290 0x7f15c019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.192+0000 7f15bdd9b640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15c01018d0 0x7f15c01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:29.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.192+0000 7f15be59c640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 0x7f15c0197b70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f15b0000c00 tx=0x7f15b002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:29.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.192+0000 7f15af7fe640 1 -- 172.21.15.49:0/55253321 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b0041070 con 0x7f15c01006e0 2024-03-18T00:49:29.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.192+0000 7f15af7fe640 1 -- 172.21.15.49:0/55253321 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15b0004450 con 0x7f15c01006e0 2024-03-18T00:49:29.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.192+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15c019fa20 con 0x7f15c01006e0 2024-03-18T00:49:29.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.192+0000 7f15af7fe640 1 -- 172.21.15.49:0/55253321 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b0034440 con 0x7f15c01006e0 2024-03-18T00:49:29.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.192+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15c019ff10 con 0x7f15c01006e0 2024-03-18T00:49:29.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.193+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15c01a0140 con 0x7f15c01006e0 2024-03-18T00:49:29.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.194+0000 7f15af7fe640 1 -- 172.21.15.49:0/55253321 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f15b0044070 con 0x7f15c01006e0 2024-03-18T00:49:29.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.195+0000 7f15af7fe640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f15a80a2890 0x7f15a80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.195+0000 7f15bdd9b640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f15a80a2890 0x7f15a80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.195+0000 7f15af7fe640 1 -- 172.21.15.49:0/55253321 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f15b00f07d0 con 0x7f15c01006e0 2024-03-18T00:49:29.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.195+0000 7f15bdd9b640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f15a80a2890 0x7f15a80a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f15c0199080 tx=0x7f15b4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:29.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.202+0000 7f15af7fe640 1 -- 172.21.15.49:0/55253321 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f15b00bd240 con 0x7f15c01006e0 2024-03-18T00:49:29.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:29 smithi049 ceph-mon[32130]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:29.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:29 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1439676306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:29.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:29 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/44415158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:29.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.388+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f15c01a0310 con 0x7f15c01006e0 2024-03-18T00:49:29.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.388+0000 7f15af7fe640 1 -- 172.21.15.49:0/55253321 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f15b00c12e0 con 0x7f15c01006e0 2024-03-18T00:49:29.388 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 32 2024-03-18T00:49:29.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f15a80a2890 msgr2=0x7f15a80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f15a80a2890 0x7f15a80a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f15c0199080 tx=0x7f15b4005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:29.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 msgr2=0x7f15c0197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 0x7f15c0197b70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f15b0000c00 tx=0x7f15b002c640 comp rx=0 tx=0).stop 2024-03-18T00:49:29.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 shutdown_connections 2024-03-18T00:49:29.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f15a80a2890 0x7f15a80a4d50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f15c0102290 0x7f15c019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15c01018d0 0x7f15c01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 --2- 172.21.15.49:0/55253321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f15c01006e0 0x7f15c0197b70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 >> 172.21.15.49:0/55253321 conn(0x7f15c00fbc70 msgr2=0x7f15c00fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:29.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 shutdown_connections 2024-03-18T00:49:29.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.390+0000 7f15c4a12640 1 -- 172.21.15.49:0/55253321 wait complete. 2024-03-18T00:49:29.401 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T00:49:29.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.513+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/831265680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0104bd0 msgr2=0x7fbbc0106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.513+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/831265680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0104bd0 0x7fbbc0106fc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009860 tx=0x7fbbb002bb70 comp rx=0 tx=0).stop 2024-03-18T00:49:29.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.513+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/831265680 shutdown_connections 2024-03-18T00:49:29.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.513+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/831265680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbc0107500 0x7fbbc0109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.513+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/831265680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0104bd0 0x7fbbc0106fc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.513+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/831265680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbc01022a0 0x7fbbc0104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.513+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/831265680 >> 172.21.15.49:0/831265680 conn(0x7fbbc00fbc70 msgr2=0x7fbbc00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.513+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/831265680 shutdown_connections 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.513+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/831265680 wait complete. 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc7f3b640 1 Processor -- start 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc7f3b640 1 -- start start 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc7f3b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbc01022a0 0x7fbbc0197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc7f3b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbc0104bd0 0x7fbbc01980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc7f3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0107500 0x7fbbc019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc7f3b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc010d460 con 0x7fbbc0104bd0 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc7f3b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc010d2e0 con 0x7fbbc01022a0 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc7f3b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc010d5e0 con 0x7fbbc0107500 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc54d6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbc0104bd0 0x7fbbc01980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc5cd7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbc01022a0 0x7fbbc0197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc54d6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbc0104bd0 0x7fbbc01980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52766/0 (socket says 172.21.15.49:52766) 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc54d6640 1 -- 172.21.15.49:0/3124293965 learned_addr learned my addr 172.21.15.49:0/3124293965 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc64d8640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0107500 0x7fbbc019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc54d6640 1 -- 172.21.15.49:0/3124293965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0107500 msgr2=0x7fbbc019f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc54d6640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0107500 0x7fbbc019f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc54d6640 1 -- 172.21.15.49:0/3124293965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbc01022a0 msgr2=0x7fbbc0197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc54d6640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbc01022a0 0x7fbbc0197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc54d6640 1 -- 172.21.15.49:0/3124293965 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc019f7b0 con 0x7fbbc0104bd0 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.514+0000 7fbbc5cd7640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbc01022a0 0x7fbbc0197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.515+0000 7fbbc54d6640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbc0104bd0 0x7fbbc01980d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbbb002c0b0 tx=0x7fbbb0032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.515+0000 7fbbb6ffd640 1 -- 172.21.15.49:0/3124293965 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0041020 con 0x7fbbc0104bd0 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.515+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc019fa40 con 0x7fbbc0104bd0 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.515+0000 7fbbb6ffd640 1 -- 172.21.15.49:0/3124293965 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbb002c9f0 con 0x7fbbc0104bd0 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.515+0000 7fbbb6ffd640 1 -- 172.21.15.49:0/3124293965 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0039040 con 0x7fbbc0104bd0 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.515+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc019ff30 con 0x7fbbc0104bd0 2024-03-18T00:49:29.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.516+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc00fd760 con 0x7fbbc0104bd0 2024-03-18T00:49:29.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.517+0000 7fbbb6ffd640 1 -- 172.21.15.49:0/3124293965 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbbb0044020 con 0x7fbbc0104bd0 2024-03-18T00:49:29.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.517+0000 7fbbb6ffd640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbbac0a2890 0x7fbbac0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.518+0000 7fbbc5cd7640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbbac0a2890 0x7fbbac0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.518+0000 7fbbb6ffd640 1 -- 172.21.15.49:0/3124293965 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbbb00f0780 con 0x7fbbc0104bd0 2024-03-18T00:49:29.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.518+0000 7fbbc5cd7640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbbac0a2890 0x7fbbac0a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009fc0 tx=0x7fbbbc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:29.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.521+0000 7fbbb6ffd640 1 -- 172.21.15.49:0/3124293965 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbbb004b020 con 0x7fbbc0104bd0 2024-03-18T00:49:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:29 smithi107 ceph-mon[36270]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:29 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1439676306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:29 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/44415158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:29.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:29 smithi106 ceph-mon[35899]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:29.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:29 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1439676306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:29.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:29 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/44415158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:29.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.705+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbbc002d340 con 0x7fbbc0104bd0 2024-03-18T00:49:29.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.705+0000 7fbbb6ffd640 1 -- 172.21.15.49:0/3124293965 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fbbb00bd7b0 con 0x7fbbc0104bd0 2024-03-18T00:49:29.705 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 64 2024-03-18T00:49:29.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbbac0a2890 msgr2=0x7fbbac0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbbac0a2890 0x7fbbac0a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009fc0 tx=0x7fbbbc005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:29.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbc0104bd0 msgr2=0x7fbbc01980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbc0104bd0 0x7fbbc01980d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbbb002c0b0 tx=0x7fbbb0032430 comp rx=0 tx=0).stop 2024-03-18T00:49:29.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 shutdown_connections 2024-03-18T00:49:29.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbbac0a2890 0x7fbbac0a4d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbbc0107500 0x7fbbc019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbbc0104bd0 0x7fbbc01980d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 --2- 172.21.15.49:0/3124293965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbbc01022a0 0x7fbbc0197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 >> 172.21.15.49:0/3124293965 conn(0x7fbbc00fbc70 msgr2=0x7fbbc0102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:29.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 shutdown_connections 2024-03-18T00:49:29.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.708+0000 7fbbc7f3b640 1 -- 172.21.15.49:0/3124293965 wait complete. 2024-03-18T00:49:29.719 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:49:29.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.828+0000 7fea597a9640 1 -- 172.21.15.49:0/4016374979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea541022d0 msgr2=0x7fea54108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.828+0000 7fea597a9640 1 --2- 172.21.15.49:0/4016374979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea541022d0 0x7fea54108b60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009ca0 tx=0x7fea4c02c0f0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.828+0000 7fea597a9640 1 -- 172.21.15.49:0/4016374979 shutdown_connections 2024-03-18T00:49:29.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.828+0000 7fea597a9640 1 --2- 172.21.15.49:0/4016374979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea541022d0 0x7fea54108b60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.828+0000 7fea597a9640 1 --2- 172.21.15.49:0/4016374979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea54101910 0x7fea54101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.828+0000 7fea597a9640 1 --2- 172.21.15.49:0/4016374979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea54100720 0x7fea54100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.828+0000 7fea597a9640 1 -- 172.21.15.49:0/4016374979 >> 172.21.15.49:0/4016374979 conn(0x7fea540fbc90 msgr2=0x7fea540fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.828+0000 7fea597a9640 1 -- 172.21.15.49:0/4016374979 shutdown_connections 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.828+0000 7fea597a9640 1 -- 172.21.15.49:0/4016374979 wait complete. 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea597a9640 1 Processor -- start 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea597a9640 1 -- start start 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea597a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea54100720 0x7fea5406a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea597a9640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea54101910 0x7fea5419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea597a9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea541022d0 0x7fea5419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea597a9640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fea5410c3a0 con 0x7fea54101910 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea597a9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fea5410c220 con 0x7fea541022d0 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea597a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea5410c520 con 0x7fea54100720 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea4bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea54101910 0x7fea5419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea4bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea54101910 0x7fea5419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52784/0 (socket says 172.21.15.49:52784) 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea52ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea54100720 0x7fea5406a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea4bfff640 1 -- 172.21.15.49:0/3858059277 learned_addr learned my addr 172.21.15.49:0/3858059277 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.829+0000 7fea537fe640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea541022d0 0x7fea5419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea52ffd640 1 -- 172.21.15.49:0/3858059277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea541022d0 msgr2=0x7fea5419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea52ffd640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea541022d0 0x7fea5419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea52ffd640 1 -- 172.21.15.49:0/3858059277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea54101910 msgr2=0x7fea5419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea52ffd640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea54101910 0x7fea5419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea52ffd640 1 -- 172.21.15.49:0/3858059277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5419f740 con 0x7fea54100720 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea52ffd640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea54100720 0x7fea5406a980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea4400e9c0 tx=0x7fea4400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea50ff9640 1 -- 172.21.15.49:0/3858059277 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4400ccc0 con 0x7fea54100720 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea50ff9640 1 -- 172.21.15.49:0/3858059277 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea440035f0 con 0x7fea54100720 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea5419f9d0 con 0x7fea54100720 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea50ff9640 1 -- 172.21.15.49:0/3858059277 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4400ce60 con 0x7fea54100720 2024-03-18T00:49:29.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.830+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea5419fed0 con 0x7fea54100720 2024-03-18T00:49:29.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.831+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea541079e0 con 0x7fea54100720 2024-03-18T00:49:29.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.832+0000 7fea50ff9640 1 -- 172.21.15.49:0/3858059277 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fea440037a0 con 0x7fea54100720 2024-03-18T00:49:29.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.832+0000 7fea50ff9640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea400a2890 0x7fea400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:29.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.833+0000 7fea4bfff640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea400a2890 0x7fea400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:29.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.833+0000 7fea50ff9640 1 -- 172.21.15.49:0/3858059277 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fea440cf110 con 0x7fea54100720 2024-03-18T00:49:29.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.833+0000 7fea4bfff640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea400a2890 0x7fea400a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0095f0 tx=0x7fea3c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:29.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:29.836+0000 7fea50ff9640 1 -- 172.21.15.49:0/3858059277 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea4409c030 con 0x7fea54100720 2024-03-18T00:49:30.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.059+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea5402d400 con 0x7fea54100720 2024-03-18T00:49:30.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.060+0000 7fea50ff9640 1 -- 172.21.15.49:0/3858059277 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11932 (secure 0 0 0) 0x7fea4409b930 con 0x7fea54100720 2024-03-18T00:49:30.060 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:30.060 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:48:12.008019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24499,24506,24481,24470,24463]},"id":1}]} 2024-03-18T00:49:30.060 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 24 2024-03-18T00:49:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea400a2890 msgr2=0x7fea400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea400a2890 0x7fea400a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0095f0 tx=0x7fea3c005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea54100720 msgr2=0x7fea5406a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea54100720 0x7fea5406a980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea4400e9c0 tx=0x7fea4400ee80 comp rx=0 tx=0).stop 2024-03-18T00:49:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 shutdown_connections 2024-03-18T00:49:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea400a2890 0x7fea400a4d50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea541022d0 0x7fea5419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea54101910 0x7fea5419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 --2- 172.21.15.49:0/3858059277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea54100720 0x7fea5406a980 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 >> 172.21.15.49:0/3858059277 conn(0x7fea540fbc90 msgr2=0x7fea54104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:30.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 shutdown_connections 2024-03-18T00:49:30.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.062+0000 7fea597a9640 1 -- 172.21.15.49:0/3858059277 wait complete. 2024-03-18T00:49:30.073 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:49:30.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.182+0000 7f2804c36640 1 -- 172.21.15.49:0/797147070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2800100720 msgr2=0x7f2800100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.182+0000 7f2804c36640 1 --2- 172.21.15.49:0/797147070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2800100720 0x7f2800100b40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f27f8009860 tx=0x7f27f802bb70 comp rx=0 tx=0).stop 2024-03-18T00:49:30.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.182+0000 7f2804c36640 1 -- 172.21.15.49:0/797147070 shutdown_connections 2024-03-18T00:49:30.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.182+0000 7f2804c36640 1 --2- 172.21.15.49:0/797147070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f28001022d0 0x7f2800108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.182+0000 7f2804c36640 1 --2- 172.21.15.49:0/797147070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2800101910 0x7f2800101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.182+0000 7f2804c36640 1 --2- 172.21.15.49:0/797147070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2800100720 0x7f2800100b40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.182+0000 7f2804c36640 1 -- 172.21.15.49:0/797147070 >> 172.21.15.49:0/797147070 conn(0x7f28000fbc90 msgr2=0x7f28000fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:30.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.182+0000 7f2804c36640 1 -- 172.21.15.49:0/797147070 shutdown_connections 2024-03-18T00:49:30.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.182+0000 7f2804c36640 1 -- 172.21.15.49:0/797147070 wait complete. 2024-03-18T00:49:30.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f2804c36640 1 Processor -- start 2024-03-18T00:49:30.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f2804c36640 1 -- start start 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f2804c36640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2800100720 0x7f2800197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f2804c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2800101910 0x7f28001980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f2804c36640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28001022d0 0x7f280019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f2804c36640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f280010c370 con 0x7f2800100720 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f2804c36640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f280010c1f0 con 0x7f28001022d0 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f2804c36640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f280010c4f0 con 0x7f2800101910 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fe59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2800100720 0x7f2800197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fe59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2800100720 0x7f2800197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52804/0 (socket says 172.21.15.49:52804) 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fe59c640 1 -- 172.21.15.49:0/2379677063 learned_addr learned my addr 172.21.15.49:0/2379677063 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fdd9b640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2800101910 0x7f28001980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fed9d640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28001022d0 0x7f280019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fe59c640 1 -- 172.21.15.49:0/2379677063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2800101910 msgr2=0x7f28001980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fe59c640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2800101910 0x7f28001980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fe59c640 1 -- 172.21.15.49:0/2379677063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28001022d0 msgr2=0x7f280019f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fe59c640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28001022d0 0x7f280019f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.183+0000 7f27fe59c640 1 -- 172.21.15.49:0/2379677063 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f280019f790 con 0x7f2800100720 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.184+0000 7f27fe59c640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2800100720 0x7f2800197b70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f27f802c0b0 tx=0x7f27f8032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.184+0000 7f27f77fe640 1 -- 172.21.15.49:0/2379677063 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8041020 con 0x7f2800100720 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.184+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f280019fa20 con 0x7f2800100720 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.184+0000 7f27f77fe640 1 -- 172.21.15.49:0/2379677063 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27f802c9f0 con 0x7f2800100720 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.184+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f280019ff60 con 0x7f2800100720 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.184+0000 7f27f77fe640 1 -- 172.21.15.49:0/2379677063 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27f8039040 con 0x7f2800100720 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.185+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28001a0250 con 0x7f2800100720 2024-03-18T00:49:30.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.186+0000 7f27f77fe640 1 -- 172.21.15.49:0/2379677063 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f27f8044020 con 0x7f2800100720 2024-03-18T00:49:30.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.186+0000 7f27f77fe640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27ec0a2890 0x7f27ec0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.187+0000 7f27fdd9b640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27ec0a2890 0x7f27ec0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.187+0000 7f27f77fe640 1 -- 172.21.15.49:0/2379677063 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f27f80f0430 con 0x7f2800100720 2024-03-18T00:49:30.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.187+0000 7f27fdd9b640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27ec0a2890 0x7f27ec0a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2800199080 tx=0x7f27e8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:30.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.191+0000 7f27f77fe640 1 -- 172.21.15.49:0/2379677063 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f27f804b020 con 0x7f2800100720 2024-03-18T00:49:30.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:30 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/55253321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:30.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:30 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3124293965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:30.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:30 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3858059277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:30.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.379+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28001a0420 con 0x7f2800100720 2024-03-18T00:49:30.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.381+0000 7f27f77fe640 1 -- 172.21.15.49:0/2379677063 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7f27f80bd460 con 0x7f2800100720 2024-03-18T00:49:30.381 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:30.381 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:49:30.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27ec0a2890 msgr2=0x7f27ec0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27ec0a2890 0x7f27ec0a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2800199080 tx=0x7f27e8009260 comp rx=0 tx=0).stop 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2800100720 msgr2=0x7f2800197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2800100720 0x7f2800197b70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f27f802c0b0 tx=0x7f27f8032430 comp rx=0 tx=0).stop 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 shutdown_connections 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27ec0a2890 0x7f27ec0a4d50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f28001022d0 0x7f280019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2800101910 0x7f28001980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 --2- 172.21.15.49:0/2379677063 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2800100720 0x7f2800197b70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 >> 172.21.15.49:0/2379677063 conn(0x7f28000fbc90 msgr2=0x7f2800069f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 shutdown_connections 2024-03-18T00:49:30.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.383+0000 7f2804c36640 1 -- 172.21.15.49:0/2379677063 wait complete. 2024-03-18T00:49:30.394 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T00:48:04.316609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T00:48:05.271815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:49:30.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.505+0000 7f12deca1640 1 -- 172.21.15.49:0/536553695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12d8102290 msgr2=0x7f12d8108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.505+0000 7f12deca1640 1 --2- 172.21.15.49:0/536553695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12d8102290 0x7f12d8108b20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009860 tx=0x7f12c002bb70 comp rx=0 tx=0).stop 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.505+0000 7f12deca1640 1 -- 172.21.15.49:0/536553695 shutdown_connections 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.505+0000 7f12deca1640 1 --2- 172.21.15.49:0/536553695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12d8102290 0x7f12d8108b20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.505+0000 7f12deca1640 1 --2- 172.21.15.49:0/536553695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12d81018d0 0x7f12d8101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.505+0000 7f12deca1640 1 --2- 172.21.15.49:0/536553695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12d81006e0 0x7f12d8100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.505+0000 7f12deca1640 1 -- 172.21.15.49:0/536553695 >> 172.21.15.49:0/536553695 conn(0x7f12d80fbc70 msgr2=0x7f12d80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.505+0000 7f12deca1640 1 -- 172.21.15.49:0/536553695 shutdown_connections 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.505+0000 7f12deca1640 1 -- 172.21.15.49:0/536553695 wait complete. 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12deca1640 1 Processor -- start 2024-03-18T00:49:30.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12deca1640 1 -- start start 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12deca1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12d81006e0 0x7f12d81969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12deca1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12d81018d0 0x7f12d8196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12deca1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12d8102290 0x7f12d819df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12deca1640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f12d810c340 con 0x7f12d8102290 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12deca1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f12d810c1c0 con 0x7f12d81006e0 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12deca1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f12d810c4c0 con 0x7f12d81018d0 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dd23e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12d8102290 0x7f12d819df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dd23e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12d8102290 0x7f12d819df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52820/0 (socket says 172.21.15.49:52820) 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dd23e640 1 -- 172.21.15.49:0/1895084637 learned_addr learned my addr 172.21.15.49:0/1895084637 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dca3d640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12d81006e0 0x7f12d81969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12d7fff640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12d81018d0 0x7f12d8196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dd23e640 1 -- 172.21.15.49:0/1895084637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12d81018d0 msgr2=0x7f12d8196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dd23e640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12d81018d0 0x7f12d8196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dd23e640 1 -- 172.21.15.49:0/1895084637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12d81006e0 msgr2=0x7f12d81969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dd23e640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12d81006e0 0x7f12d81969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dd23e640 1 -- 172.21.15.49:0/1895084637 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12d819e5d0 con 0x7f12d8102290 2024-03-18T00:49:30.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.506+0000 7f12dca3d640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12d81006e0 0x7f12d81969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:30.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.507+0000 7f12dd23e640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12d8102290 0x7f12d819df60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f12c002c180 tx=0x7f12c0032460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:30.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.507+0000 7f12d5ffb640 1 -- 172.21.15.49:0/1895084637 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c0039040 con 0x7f12d8102290 2024-03-18T00:49:30.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.507+0000 7f12d5ffb640 1 -- 172.21.15.49:0/1895084637 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12c0045070 con 0x7f12d8102290 2024-03-18T00:49:30.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.507+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12d8068940 con 0x7f12d8102290 2024-03-18T00:49:30.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.507+0000 7f12d5ffb640 1 -- 172.21.15.49:0/1895084637 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12c00046c0 con 0x7f12d8102290 2024-03-18T00:49:30.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.507+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12d8068e30 con 0x7f12d8102290 2024-03-18T00:49:30.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.508+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12d80685d0 con 0x7f12d8102290 2024-03-18T00:49:30.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.509+0000 7f12d5ffb640 1 -- 172.21.15.49:0/1895084637 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f12c0004340 con 0x7f12d8102290 2024-03-18T00:49:30.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.509+0000 7f12d5ffb640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12c40a2890 0x7f12c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.510+0000 7f12d5ffb640 1 -- 172.21.15.49:0/1895084637 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f12c00f27c0 con 0x7f12d8102290 2024-03-18T00:49:30.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.510+0000 7f12dca3d640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12c40a2890 0x7f12c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.510+0000 7f12dca3d640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12c40a2890 0x7f12c40a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f12d8101730 tx=0x7f12d0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:30.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.514+0000 7f12d5ffb640 1 -- 172.21.15.49:0/1895084637 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f12c00bf7f0 con 0x7f12d8102290 2024-03-18T00:49:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:30 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/55253321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:30 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3124293965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:30 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3858059277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:30.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:30 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/55253321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:30.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:30 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3124293965' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:30.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:30 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3858059277' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:30.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.696+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f12d819e800 con 0x7f12d8102290 2024-03-18T00:49:30.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.696+0000 7f12d5ffb640 1 -- 172.21.15.49:0/1895084637 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f12c00bf0f0 con 0x7f12d8102290 2024-03-18T00:49:30.696 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:49:30.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12c40a2890 msgr2=0x7f12c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12c40a2890 0x7f12c40a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f12d8101730 tx=0x7f12d0009500 comp rx=0 tx=0).stop 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12d8102290 msgr2=0x7f12d819df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12d8102290 0x7f12d819df60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f12c002c180 tx=0x7f12c0032460 comp rx=0 tx=0).stop 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 shutdown_connections 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f12c40a2890 0x7f12c40a4d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f12d8102290 0x7f12d819df60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f12d81018d0 0x7f12d8196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 --2- 172.21.15.49:0/1895084637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f12d81006e0 0x7f12d81969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 >> 172.21.15.49:0/1895084637 conn(0x7f12d80fbc70 msgr2=0x7f12d8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 shutdown_connections 2024-03-18T00:49:30.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.698+0000 7f12deca1640 1 -- 172.21.15.49:0/1895084637 wait complete. 2024-03-18T00:49:30.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T00:49:30.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.821+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2584522958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e18102290 msgr2=0x7f0e18108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.821+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2584522958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e18102290 0x7f0e18108b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0e10009be0 tx=0x7f0e1002bef0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.821+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2584522958 shutdown_connections 2024-03-18T00:49:30.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.821+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2584522958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e18102290 0x7f0e18108b20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.821+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2584522958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e181018d0 0x7f0e18101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.821+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2584522958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e181006e0 0x7f0e18100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.821+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2584522958 >> 172.21.15.49:0/2584522958 conn(0x7f0e180fbc50 msgr2=0x7f0e180fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.821+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2584522958 shutdown_connections 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.821+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2584522958 wait complete. 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e1dbd4640 1 Processor -- start 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e1dbd4640 1 -- start start 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e1dbd4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e181006e0 0x7f0e18199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e1dbd4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e181018d0 0x7f0e1819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e1dbd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e18102290 0x7f0e181a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e1dbd4640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1810d480 con 0x7f0e181006e0 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e1dbd4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1810d300 con 0x7f0e181018d0 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e1dbd4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0e1810d600 con 0x7f0e18102290 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e177fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e181006e0 0x7f0e18199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e177fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e181006e0 0x7f0e18199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52830/0 (socket says 172.21.15.49:52830) 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e177fe640 1 -- 172.21.15.49:0/2385452918 learned_addr learned my addr 172.21.15.49:0/2385452918 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e16ffd640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e181018d0 0x7f0e1819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e17fff640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e18102290 0x7f0e181a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e177fe640 1 -- 172.21.15.49:0/2385452918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e18102290 msgr2=0x7f0e181a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e177fe640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e18102290 0x7f0e181a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e177fe640 1 -- 172.21.15.49:0/2385452918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e181018d0 msgr2=0x7f0e1819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:30.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e177fe640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e181018d0 0x7f0e1819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.822+0000 7f0e177fe640 1 -- 172.21.15.49:0/2385452918 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e181a18d0 con 0x7f0e181006e0 2024-03-18T00:49:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.823+0000 7f0e177fe640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e181006e0 0x7f0e18199c90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000e9c0 tx=0x7f0e0000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.823+0000 7f0e14ff9640 1 -- 172.21.15.49:0/2385452918 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0000ccc0 con 0x7f0e181006e0 2024-03-18T00:49:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.823+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e181a1bc0 con 0x7f0e181006e0 2024-03-18T00:49:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.823+0000 7f0e14ff9640 1 -- 172.21.15.49:0/2385452918 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e000035f0 con 0x7f0e181006e0 2024-03-18T00:49:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.823+0000 7f0e14ff9640 1 -- 172.21.15.49:0/2385452918 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e0000ce60 con 0x7f0e181006e0 2024-03-18T00:49:30.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.823+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e181a2110 con 0x7f0e181006e0 2024-03-18T00:49:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.824+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e181a2340 con 0x7f0e181006e0 2024-03-18T00:49:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.825+0000 7f0e14ff9640 1 -- 172.21.15.49:0/2385452918 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0e000037a0 con 0x7f0e181006e0 2024-03-18T00:49:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.826+0000 7f0e14ff9640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e040a2890 0x7f0e040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.826+0000 7f0e16ffd640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e040a2890 0x7f0e040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.826+0000 7f0e14ff9640 1 -- 172.21.15.49:0/2385452918 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0e000cefd0 con 0x7f0e181006e0 2024-03-18T00:49:30.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.826+0000 7f0e16ffd640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e040a2890 0x7f0e040a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0e1819b190 tx=0x7f0e08009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:30.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:30.830+0000 7f0e14ff9640 1 -- 172.21.15.49:0/2385452918 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e0009bef0 con 0x7f0e181006e0 2024-03-18T00:49:31.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.016+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0e181a25b0 con 0x7f0e181006e0 2024-03-18T00:49:31.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.017+0000 7f0e14ff9640 1 -- 172.21.15.49:0/2385452918 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f0e0009b7f0 con 0x7f0e181006e0 2024-03-18T00:49:31.017 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 32 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e040a2890 msgr2=0x7f0e040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e040a2890 0x7f0e040a4d50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0e1819b190 tx=0x7f0e08009260 comp rx=0 tx=0).stop 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e181006e0 msgr2=0x7f0e18199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e181006e0 0x7f0e18199c90 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000e9c0 tx=0x7f0e0000edc0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 shutdown_connections 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e040a2890 0x7f0e040a4d50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e181018d0 0x7f0e1819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e18102290 0x7f0e181a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 --2- 172.21.15.49:0/2385452918 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e181006e0 0x7f0e18199c90 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 >> 172.21.15.49:0/2385452918 conn(0x7f0e180fbc50 msgr2=0x7f0e181094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:31.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 shutdown_connections 2024-03-18T00:49:31.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.019+0000 7f0e1dbd4640 1 -- 172.21.15.49:0/2385452918 wait complete. 2024-03-18T00:49:31.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T00:49:31.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.141+0000 7faf64a62640 1 -- 172.21.15.49:0/4016974936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf601018d0 msgr2=0x7faf60101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.141+0000 7faf64a62640 1 --2- 172.21.15.49:0/4016974936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf601018d0 0x7faf60101d50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faf48009860 tx=0x7faf4802bb70 comp rx=0 tx=0).stop 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.141+0000 7faf64a62640 1 -- 172.21.15.49:0/4016974936 shutdown_connections 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.141+0000 7faf64a62640 1 --2- 172.21.15.49:0/4016974936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf601018d0 0x7faf60101d50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.141+0000 7faf64a62640 1 --2- 172.21.15.49:0/4016974936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf60102290 0x7faf60108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.141+0000 7faf64a62640 1 --2- 172.21.15.49:0/4016974936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf601006e0 0x7faf60100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.141+0000 7faf64a62640 1 -- 172.21.15.49:0/4016974936 >> 172.21.15.49:0/4016974936 conn(0x7faf600fbc70 msgr2=0x7faf600fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.141+0000 7faf64a62640 1 -- 172.21.15.49:0/4016974936 shutdown_connections 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.141+0000 7faf64a62640 1 -- 172.21.15.49:0/4016974936 wait complete. 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf64a62640 1 Processor -- start 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf64a62640 1 -- start start 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf64a62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf601006e0 0x7faf60197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf64a62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf601018d0 0x7faf60198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf64a62640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf60102290 0x7faf6019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf64a62640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7faf6010c340 con 0x7faf601006e0 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf64a62640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7faf6010c1c0 con 0x7faf60102290 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf64a62640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faf6010c4c0 con 0x7faf601018d0 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf5e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf601006e0 0x7faf60197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf5e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf601006e0 0x7faf60197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52848/0 (socket says 172.21.15.49:52848) 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf5e59c640 1 -- 172.21.15.49:0/1071244328 learned_addr learned my addr 172.21.15.49:0/1071244328 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.142+0000 7faf5dd9b640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf601018d0 0x7faf60198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf5ed9d640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf60102290 0x7faf6019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf5e59c640 1 -- 172.21.15.49:0/1071244328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf601018d0 msgr2=0x7faf60198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf5e59c640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf601018d0 0x7faf60198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf5e59c640 1 -- 172.21.15.49:0/1071244328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf60102290 msgr2=0x7faf6019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf5e59c640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf60102290 0x7faf6019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf5e59c640 1 -- 172.21.15.49:0/1071244328 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf6019f730 con 0x7faf601006e0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf5e59c640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf601006e0 0x7faf60197b10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faf5800e9a0 tx=0x7faf5800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf577fe640 1 -- 172.21.15.49:0/1071244328 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5800ccc0 con 0x7faf601006e0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf577fe640 1 -- 172.21.15.49:0/1071244328 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf58010910 con 0x7faf601006e0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf6019fa20 con 0x7faf601006e0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf577fe640 1 -- 172.21.15.49:0/1071244328 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf58018a30 con 0x7faf601006e0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.143+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf6019ff90 con 0x7faf601006e0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.144+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf601a01c0 con 0x7faf601006e0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.145+0000 7faf577fe640 1 -- 172.21.15.49:0/1071244328 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7faf5801f020 con 0x7faf601006e0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.146+0000 7faf577fe640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faf4c0a2910 0x7faf4c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.146+0000 7faf5dd9b640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faf4c0a2910 0x7faf4c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.146+0000 7faf577fe640 1 -- 172.21.15.49:0/1071244328 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7faf580ced70 con 0x7faf601006e0 2024-03-18T00:49:31.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.146+0000 7faf5dd9b640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faf4c0a2910 0x7faf4c0a4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7faf4802c180 tx=0x7faf48034ad0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:31.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.150+0000 7faf577fe640 1 -- 172.21.15.49:0/1071244328 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faf5809bc90 con 0x7faf601006e0 2024-03-18T00:49:31.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.333+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faf601a0390 con 0x7faf601006e0 2024-03-18T00:49:31.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.333+0000 7faf577fe640 1 -- 172.21.15.49:0/1071244328 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7faf5809b590 con 0x7faf601006e0 2024-03-18T00:49:31.333 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 64 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faf4c0a2910 msgr2=0x7faf4c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faf4c0a2910 0x7faf4c0a4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7faf4802c180 tx=0x7faf48034ad0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf601006e0 msgr2=0x7faf60197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf601006e0 0x7faf60197b10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faf5800e9a0 tx=0x7faf5800eda0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 shutdown_connections 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faf4c0a2910 0x7faf4c0a4dd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faf60102290 0x7faf6019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faf601018d0 0x7faf60198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 --2- 172.21.15.49:0/1071244328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faf601006e0 0x7faf60197b10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 >> 172.21.15.49:0/1071244328 conn(0x7faf600fbc70 msgr2=0x7faf600fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 shutdown_connections 2024-03-18T00:49:31.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.336+0000 7faf64a62640 1 -- 172.21.15.49:0/1071244328 wait complete. 2024-03-18T00:49:31.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:31 smithi049 ceph-mon[32130]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:31.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:31 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2379677063' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:31.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:31 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1895084637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:31.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:31 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2385452918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:31.350 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:49:31.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.461+0000 7f9b87a61640 1 -- 172.21.15.49:0/3507809656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b801022c0 msgr2=0x7f9b801046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.461+0000 7f9b87a61640 1 --2- 172.21.15.49:0/3507809656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b801022c0 0x7f9b801046b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009900 tx=0x7f9b7c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:31.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.464+0000 7f9b87a61640 1 -- 172.21.15.49:0/3507809656 shutdown_connections 2024-03-18T00:49:31.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.464+0000 7f9b87a61640 1 --2- 172.21.15.49:0/3507809656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b80107520 0x7f9b80109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.464+0000 7f9b87a61640 1 --2- 172.21.15.49:0/3507809656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b80104bf0 0x7f9b80106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.464+0000 7f9b87a61640 1 --2- 172.21.15.49:0/3507809656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b801022c0 0x7f9b801046b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.464+0000 7f9b87a61640 1 -- 172.21.15.49:0/3507809656 >> 172.21.15.49:0/3507809656 conn(0x7f9b800fbc70 msgr2=0x7f9b800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:31.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.465+0000 7f9b87a61640 1 -- 172.21.15.49:0/3507809656 shutdown_connections 2024-03-18T00:49:31.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.465+0000 7f9b87a61640 1 -- 172.21.15.49:0/3507809656 wait complete. 2024-03-18T00:49:31.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.466+0000 7f9b87a61640 1 Processor -- start 2024-03-18T00:49:31.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.466+0000 7f9b87a61640 1 -- start start 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.466+0000 7f9b87a61640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b801022c0 0x7f9b80195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.466+0000 7f9b87a61640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b80104bf0 0x7f9b80195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b87a61640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b80107520 0x7f9b801a57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b87a61640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010d4c0 con 0x7f9b80107520 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b857fd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b801022c0 0x7f9b80195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b857fd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b801022c0 0x7f9b80195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:57194/0 (socket says 172.21.15.49:57194) 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b87a61640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010d340 con 0x7f9b801022c0 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b857fd640 1 -- 172.21.15.49:0/2166961607 learned_addr learned my addr 172.21.15.49:0/2166961607 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8010d640 con 0x7f9b80104bf0 2024-03-18T00:49:31.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b85ffe640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b80107520 0x7f9b801a57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b85ffe640 1 -- 172.21.15.49:0/2166961607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b80104bf0 msgr2=0x7f9b80195ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:49:31.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b85ffe640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b80104bf0 0x7f9b80195ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b85ffe640 1 -- 172.21.15.49:0/2166961607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b801022c0 msgr2=0x7f9b80195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b85ffe640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b801022c0 0x7f9b80195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.467+0000 7f9b85ffe640 1 -- 172.21.15.49:0/2166961607 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b801a5db0 con 0x7f9b80107520 2024-03-18T00:49:31.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.468+0000 7f9b85ffe640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b80107520 0x7f9b801a57f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800ea80 tx=0x7f9b7800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:31.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.472+0000 7f9b767fc640 1 -- 172.21.15.49:0/2166961607 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7800cea0 con 0x7f9b80107520 2024-03-18T00:49:31.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.473+0000 7f9b767fc640 1 -- 172.21.15.49:0/2166961607 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b78005020 con 0x7f9b80107520 2024-03-18T00:49:31.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.473+0000 7f9b767fc640 1 -- 172.21.15.49:0/2166961607 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b780106d0 con 0x7f9b80107520 2024-03-18T00:49:31.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.473+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b801a60a0 con 0x7f9b80107520 2024-03-18T00:49:31.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.473+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b801a65a0 con 0x7f9b80107520 2024-03-18T00:49:31.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.475+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b64005290 con 0x7f9b80107520 2024-03-18T00:49:31.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.477+0000 7f9b767fc640 1 -- 172.21.15.49:0/2166961607 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9b78004a40 con 0x7f9b80107520 2024-03-18T00:49:31.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.478+0000 7f9b767fc640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9b6c0a2a60 0x7f9b6c0a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.478+0000 7f9b767fc640 1 -- 172.21.15.49:0/2166961607 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9b780cf030 con 0x7f9b80107520 2024-03-18T00:49:31.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.483+0000 7f9b857fd640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9b6c0a2a60 0x7f9b6c0a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.486+0000 7f9b767fc640 1 -- 172.21.15.49:0/2166961607 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9b7809bef0 con 0x7f9b80107520 2024-03-18T00:49:31.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.489+0000 7f9b857fd640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9b6c0a2a60 0x7f9b6c0a4f20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c02c8e0 tx=0x7f9b7c02df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:31 smithi107 ceph-mon[36270]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:31 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2379677063' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:31 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1895084637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:31 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2385452918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:31.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:31 smithi106 ceph-mon[35899]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:31.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:31 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2379677063' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:31.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:31 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1895084637' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:31.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:31 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2385452918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:31.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.722+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b64005850 con 0x7f9b80107520 2024-03-18T00:49:31.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.723+0000 7f9b767fc640 1 -- 172.21.15.49:0/2166961607 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11932 (secure 0 0 0) 0x7f9b7809b7f0 con 0x7f9b80107520 2024-03-18T00:49:31.723 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:31.723 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:48:12.008019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24499,24506,24481,24470,24463]},"id":1}]} 2024-03-18T00:49:31.723 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 24 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9b6c0a2a60 msgr2=0x7f9b6c0a4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9b6c0a2a60 0x7f9b6c0a4f20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c02c8e0 tx=0x7f9b7c02df90 comp rx=0 tx=0).stop 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b80107520 msgr2=0x7f9b801a57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b80107520 0x7f9b801a57f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800ea80 tx=0x7f9b7800ef40 comp rx=0 tx=0).stop 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 shutdown_connections 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9b6c0a2a60 0x7f9b6c0a4f20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9b80107520 0x7f9b801a57f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9b80104bf0 0x7f9b80195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 --2- 172.21.15.49:0/2166961607 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9b801022c0 0x7f9b80195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 >> 172.21.15.49:0/2166961607 conn(0x7f9b800fbc70 msgr2=0x7f9b80101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:31.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 shutdown_connections 2024-03-18T00:49:31.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.725+0000 7f9b87a61640 1 -- 172.21.15.49:0/2166961607 wait complete. 2024-03-18T00:49:31.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:49:31.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.844+0000 7f8a780e8640 1 -- 172.21.15.49:0/3469509433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 msgr2=0x7f8a70101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.844+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3469509433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 0x7f8a70101df0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009900 tx=0x7f8a6c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:31.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 -- 172.21.15.49:0/3469509433 shutdown_connections 2024-03-18T00:49:31.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3469509433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a70102330 0x7f8a70108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3469509433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 0x7f8a70101df0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3469509433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70100780 0x7f8a70100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 -- 172.21.15.49:0/3469509433 >> 172.21.15.49:0/3469509433 conn(0x7f8a700fbd10 msgr2=0x7f8a700fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 -- 172.21.15.49:0/3469509433 shutdown_connections 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 -- 172.21.15.49:0/3469509433 wait complete. 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 Processor -- start 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 -- start start 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70100780 0x7f8a70197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 0x7f8a70198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a70102330 0x7f8a7019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7010c390 con 0x7f8a70101970 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7010c210 con 0x7f8a70102330 2024-03-18T00:49:31.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a780e8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7010c510 con 0x7f8a70100780 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.845+0000 7f8a75683640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 0x7f8a70198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a75683640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 0x7f8a70198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52886/0 (socket says 172.21.15.49:52886) 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a75683640 1 -- 172.21.15.49:0/3107494989 learned_addr learned my addr 172.21.15.49:0/3107494989 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a75e84640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70100780 0x7f8a70197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a76685640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a70102330 0x7f8a7019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a75683640 1 -- 172.21.15.49:0/3107494989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70100780 msgr2=0x7f8a70197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a75683640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70100780 0x7f8a70197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a75683640 1 -- 172.21.15.49:0/3107494989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a70102330 msgr2=0x7f8a7019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a75683640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a70102330 0x7f8a7019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a75683640 1 -- 172.21.15.49:0/3107494989 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7019f720 con 0x7f8a70101970 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a76685640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a70102330 0x7f8a7019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a75683640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 0x7f8a70198040 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009230 tx=0x7f8a6c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:31.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a66ffd640 1 -- 172.21.15.49:0/3107494989 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c041070 con 0x7f8a70101970 2024-03-18T00:49:31.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a66ffd640 1 -- 172.21.15.49:0/3107494989 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a6c004450 con 0x7f8a70101970 2024-03-18T00:49:31.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7019f9b0 con 0x7f8a70101970 2024-03-18T00:49:31.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a66ffd640 1 -- 172.21.15.49:0/3107494989 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c003e20 con 0x7f8a70101970 2024-03-18T00:49:31.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.846+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7019fea0 con 0x7f8a70101970 2024-03-18T00:49:31.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.847+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a701a00d0 con 0x7f8a70101970 2024-03-18T00:49:31.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.848+0000 7f8a66ffd640 1 -- 172.21.15.49:0/3107494989 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8a6c044070 con 0x7f8a70101970 2024-03-18T00:49:31.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.849+0000 7f8a66ffd640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:31.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.849+0000 7f8a75e84640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:31.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.849+0000 7f8a66ffd640 1 -- 172.21.15.49:0/3107494989 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8a6c0f0750 con 0x7f8a70101970 2024-03-18T00:49:31.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.849+0000 7f8a75e84640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8a701017d0 tx=0x7f8a60005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:31.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:31.853+0000 7f8a66ffd640 1 -- 172.21.15.49:0/3107494989 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a6c0bd780 con 0x7f8a70101970 2024-03-18T00:49:32.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.038+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a701a03c0 con 0x7f8a70101970 2024-03-18T00:49:32.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.039+0000 7f8a66ffd640 1 -- 172.21.15.49:0/3107494989 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7f8a6c0bd080 con 0x7f8a70101970 2024-03-18T00:49:32.040 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:32.040 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a5c0a2890 msgr2=0x7f8a5c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8a701017d0 tx=0x7f8a60005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 msgr2=0x7f8a70198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 0x7f8a70198040 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c009230 tx=0x7f8a6c02c650 comp rx=0 tx=0).stop 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 shutdown_connections 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a5c0a2890 0x7f8a5c0a4d50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a70102330 0x7f8a7019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a70101970 0x7f8a70198040 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 --2- 172.21.15.49:0/3107494989 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a70100780 0x7f8a70197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.041+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 >> 172.21.15.49:0/3107494989 conn(0x7f8a700fbd10 msgr2=0x7f8a700fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.042+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 shutdown_connections 2024-03-18T00:49:32.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.042+0000 7f8a780e8640 1 -- 172.21.15.49:0/3107494989 wait complete. 2024-03-18T00:49:32.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.161+0000 7f7ce76b5640 1 -- 172.21.15.49:0/1572222374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0102330 msgr2=0x7f7ce0108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.161+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/1572222374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0102330 0x7f7ce0108bc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7cd8009ca0 tx=0x7f7cd802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.161+0000 7f7ce76b5640 1 -- 172.21.15.49:0/1572222374 shutdown_connections 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.161+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/1572222374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce0101970 0x7f7ce0101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.161+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/1572222374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0102330 0x7f7ce0108bc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.161+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/1572222374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce0100780 0x7f7ce0100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.161+0000 7f7ce76b5640 1 -- 172.21.15.49:0/1572222374 >> 172.21.15.49:0/1572222374 conn(0x7f7ce00fbd10 msgr2=0x7f7ce00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.161+0000 7f7ce76b5640 1 -- 172.21.15.49:0/1572222374 shutdown_connections 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.161+0000 7f7ce76b5640 1 -- 172.21.15.49:0/1572222374 wait complete. 2024-03-18T00:49:32.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce76b5640 1 Processor -- start 2024-03-18T00:49:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce76b5640 1 -- start start 2024-03-18T00:49:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce76b5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce0100780 0x7f7ce01969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce76b5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0101970 0x7f7ce0196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce76b5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce0102330 0x7f7ce019df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce76b5640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce010c370 con 0x7f7ce0101970 2024-03-18T00:49:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce76b5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce010c1f0 con 0x7f7ce0102330 2024-03-18T00:49:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce76b5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce010c4f0 con 0x7f7ce0100780 2024-03-18T00:49:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce4c50640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0101970 0x7f7ce0196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce4c50640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0101970 0x7f7ce0196ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52898/0 (socket says 172.21.15.49:52898) 2024-03-18T00:49:32.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce4c50640 1 -- 172.21.15.49:0/2007635439 learned_addr learned my addr 172.21.15.49:0/2007635439 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce5451640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce0100780 0x7f7ce01969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce5c52640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce0102330 0x7f7ce019df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce4c50640 1 -- 172.21.15.49:0/2007635439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce0100780 msgr2=0x7f7ce01969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce4c50640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce0100780 0x7f7ce01969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce4c50640 1 -- 172.21.15.49:0/2007635439 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce0102330 msgr2=0x7f7ce019df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce4c50640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce0102330 0x7f7ce019df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce4c50640 1 -- 172.21.15.49:0/2007635439 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce019e5d0 con 0x7f7ce0101970 2024-03-18T00:49:32.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.162+0000 7f7ce4c50640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0101970 0x7f7ce0196ef0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00e9f0 tx=0x7f7cdc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.163+0000 7f7cd27fc640 1 -- 172.21.15.49:0/2007635439 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc00cd80 con 0x7f7ce0101970 2024-03-18T00:49:32.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.163+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce019e8c0 con 0x7f7ce0101970 2024-03-18T00:49:32.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.163+0000 7f7cd27fc640 1 -- 172.21.15.49:0/2007635439 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cdc010910 con 0x7f7ce0101970 2024-03-18T00:49:32.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.163+0000 7f7cd27fc640 1 -- 172.21.15.49:0/2007635439 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc018990 con 0x7f7ce0101970 2024-03-18T00:49:32.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.163+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce019ee10 con 0x7f7ce0101970 2024-03-18T00:49:32.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.164+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce0107b20 con 0x7f7ce0101970 2024-03-18T00:49:32.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.165+0000 7f7cd27fc640 1 -- 172.21.15.49:0/2007635439 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7cdc010ac0 con 0x7f7ce0101970 2024-03-18T00:49:32.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.165+0000 7f7cd27fc640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7ccc0a2890 0x7f7ccc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.165+0000 7f7ce5451640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7ccc0a2890 0x7f7ccc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.165+0000 7f7cd27fc640 1 -- 172.21.15.49:0/2007635439 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7cdc0cef60 con 0x7f7ce0101970 2024-03-18T00:49:32.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.166+0000 7f7ce5451640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7ccc0a2890 0x7f7ccc0a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40095f0 tx=0x7f7cd4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.170+0000 7f7cd27fc640 1 -- 172.21.15.49:0/2007635439 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7cdc09b830 con 0x7f7ce0101970 2024-03-18T00:49:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:32 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1071244328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:32 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2166961607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:32 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3107494989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:32.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.395+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ce002d1b0 con 0x7f7ce0101970 2024-03-18T00:49:32.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.396+0000 7f7cd27fc640 1 -- 172.21.15.49:0/2007635439 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11932 (secure 0 0 0) 0x7f7cdc09f8d0 con 0x7f7ce0101970 2024-03-18T00:49:32.397 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:32.397 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:48:12.008019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24499,24506,24481,24470,24463]},"id":1}]} 2024-03-18T00:49:32.397 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 24 2024-03-18T00:49:32.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7ccc0a2890 msgr2=0x7f7ccc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7ccc0a2890 0x7f7ccc0a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40095f0 tx=0x7f7cd4005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0101970 msgr2=0x7f7ce0196ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0101970 0x7f7ce0196ef0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00e9f0 tx=0x7f7cdc00eeb0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 shutdown_connections 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7ccc0a2890 0x7f7ccc0a4d50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7ce0102330 0x7f7ce019df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7ce0101970 0x7f7ce0196ef0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 --2- 172.21.15.49:0/2007635439 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7ce0100780 0x7f7ce01969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 >> 172.21.15.49:0/2007635439 conn(0x7f7ce00fbd10 msgr2=0x7f7ce0104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.398+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 shutdown_connections 2024-03-18T00:49:32.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.399+0000 7f7ce76b5640 1 -- 172.21.15.49:0/2007635439 wait complete. 2024-03-18T00:49:32.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T00:48:05.881816+0000', 'modified': '2024-03-18T00:48:12.008019+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24506, 'mds_1': 24499, 'mds_2': 24481, 'mds_3': 24470, 'mds_4': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14613': {'gid': 14613, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.49:6839/1964983089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6838', 'nonce': 1964983089}, {'type': 'v1', 'addr': '172.21.15.49:6839', 'nonce': 1964983089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14625': {'gid': 14625, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.49:6841/3327527918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6840', 'nonce': 3327527918}, {'type': 'v1', 'addr': '172.21.15.49:6841', 'nonce': 3327527918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24404': {'gid': 24404, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6833/701158707', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 701158707}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 701158707}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6833/1442063907', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6832', 'nonce': 1442063907}, {'type': 'v1', 'addr': '172.21.15.107:6833', 'nonce': 1442063907}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6835/353176461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6834', 'nonce': 353176461}, {'type': 'v1', 'addr': '172.21.15.106:6835', 'nonce': 353176461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 4, 'incarnation': 22, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.107:6835/863606455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 863606455}, {'type': 'v1', 'addr': '172.21.15.107:6835', 'nonce': 863606455}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'h', 'rank': 3, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.106:6837/3542707642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6836', 'nonce': 3542707642}, {'type': 'v1', 'addr': '172.21.15.106:6837', 'nonce': 3542707642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.107:6837/3043451403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 3043451403}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 3043451403}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.107:6839/2909446244', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 2909446244}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 2909446244}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.106:6839/2088471086', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6838', 'nonce': 2088471086}, {'type': 'v1', 'addr': '172.21.15.106:6839', 'nonce': 2088471086}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24506, 'qdb_cluster': [24499, 24506, 24481, 24470, 24463]} 2024-03-18T00:49:32.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T00:49:32.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.519+0000 7f52712b2640 1 -- 172.21.15.49:0/1328941261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f526c1018d0 msgr2=0x7f526c101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.519+0000 7f52712b2640 1 --2- 172.21.15.49:0/1328941261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f526c1018d0 0x7f526c101d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5254009900 tx=0x7f525402bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.519+0000 7f52712b2640 1 -- 172.21.15.49:0/1328941261 shutdown_connections 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.519+0000 7f52712b2640 1 --2- 172.21.15.49:0/1328941261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f526c102290 0x7f526c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.519+0000 7f52712b2640 1 --2- 172.21.15.49:0/1328941261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f526c1018d0 0x7f526c101d50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.519+0000 7f52712b2640 1 --2- 172.21.15.49:0/1328941261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f526c1006e0 0x7f526c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.519+0000 7f52712b2640 1 -- 172.21.15.49:0/1328941261 >> 172.21.15.49:0/1328941261 conn(0x7f526c0fbc70 msgr2=0x7f526c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.519+0000 7f52712b2640 1 -- 172.21.15.49:0/1328941261 shutdown_connections 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.519+0000 7f52712b2640 1 -- 172.21.15.49:0/1328941261 wait complete. 2024-03-18T00:49:32.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f52712b2640 1 Processor -- start 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f52712b2640 1 -- start start 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f52712b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f526c1006e0 0x7f526c197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f52712b2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f526c1018d0 0x7f526c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f52712b2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f526c102290 0x7f526c19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f52712b2640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f526c10d480 con 0x7f526c1018d0 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f52712b2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f526c10d300 con 0x7f526c102290 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f52712b2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f526c10d600 con 0x7f526c1006e0 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f526a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f526c1018d0 0x7f526c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f526a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f526c1018d0 0x7f526c1980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52912/0 (socket says 172.21.15.49:52912) 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f526a7fc640 1 -- 172.21.15.49:0/300917279 learned_addr learned my addr 172.21.15.49:0/300917279 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f526b7fe640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f526c102290 0x7f526c19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.520+0000 7f526affd640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f526c1006e0 0x7f526c197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f526a7fc640 1 -- 172.21.15.49:0/300917279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f526c1006e0 msgr2=0x7f526c197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f526a7fc640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f526c1006e0 0x7f526c197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f526a7fc640 1 -- 172.21.15.49:0/300917279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f526c102290 msgr2=0x7f526c19f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f526a7fc640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f526c102290 0x7f526c19f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f526a7fc640 1 -- 172.21.15.49:0/300917279 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f526c19f780 con 0x7f526c1018d0 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f526b7fe640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f526c102290 0x7f526c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f526a7fc640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f526c1018d0 0x7f526c1980a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5254009230 tx=0x7f525402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f5263fff640 1 -- 172.21.15.49:0/300917279 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254041070 con 0x7f526c1018d0 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f5263fff640 1 -- 172.21.15.49:0/300917279 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5254004450 con 0x7f526c1018d0 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f52712b2640 1 -- 172.21.15.49:0/300917279 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f526c19fa10 con 0x7f526c1018d0 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f5263fff640 1 -- 172.21.15.49:0/300917279 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5254003db0 con 0x7f526c1018d0 2024-03-18T00:49:32.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.521+0000 7f52712b2640 1 -- 172.21.15.49:0/300917279 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f526c19ff00 con 0x7f526c1018d0 2024-03-18T00:49:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f5263fff640 1 -- 172.21.15.49:0/300917279 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5254044070 con 0x7f526c1018d0 2024-03-18T00:49:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f5273f7c740 1 Processor -- start 2024-03-18T00:49:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f5273f7c740 1 -- start start 2024-03-18T00:49:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a76ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 0x55b0f6a7c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a7c730 0x55b0f6a7ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f5273f7c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b0f69dac80 con 0x55b0f6a76bd0 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f5273f7c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6983aa0 con 0x55b0f6a7c730 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f5273f7c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b0f69d6d20 con 0x55b0f6a775c0 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.524+0000 7f526b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a76ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a76ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52928/0 (socket says 172.21.15.49:52928) 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526b7fe640 1 -- 172.21.15.49:0/2312787389 learned_addr learned my addr 172.21.15.49:0/2312787389 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526a7fc640 1 --2- 172.21.15.49:0/2312787389 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a7c730 0x55b0f6a7ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526affd640 1 --2- 172.21.15.49:0/2312787389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 0x55b0f6a7c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f5263fff640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52580a2890 0x7f52580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526affd640 1 -- 172.21.15.49:0/2312787389 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a7c730 msgr2=0x55b0f6a7ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526affd640 1 --2- 172.21.15.49:0/2312787389 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a7c730 0x55b0f6a7ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526affd640 1 -- 172.21.15.49:0/2312787389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a76bd0 msgr2=0x55b0f6a76ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526affd640 1 --2- 172.21.15.49:0/2312787389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a76ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526affd640 1 -- 172.21.15.49:0/2312787389 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0f69a5ff0 con 0x55b0f6a775c0 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526b7fe640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52580a2890 0x7f52580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f5263fff640 1 -- 172.21.15.49:0/300917279 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f52540f06a0 con 0x7f526c1018d0 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526affd640 1 --2- 172.21.15.49:0/2312787389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 0x55b0f6a7c0c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5264009bd0 tx=0x7f5264011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfd34f0f348ced00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f52617fa640 1 -- 172.21.15.49:0/2312787389 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5264004230 con 0x55b0f6a775c0 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f526b7fe640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52580a2890 0x7f52580a4d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55b0f6a77e20 tx=0x7f525c004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f52617fa640 1 -- 172.21.15.49:0/2312787389 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5264009e40 con 0x55b0f6a775c0 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.525+0000 7f52617fa640 1 -- 172.21.15.49:0/2312787389 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52640197c0 con 0x55b0f6a775c0 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 -- 172.21.15.49:0/2312787389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 msgr2=0x55b0f6a7c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2312787389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 0x55b0f6a7c0c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5264009bd0 tx=0x7f5264011810 comp rx=0 tx=0).stop 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 -- 172.21.15.49:0/2312787389 shutdown_connections 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2312787389 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a7c730 0x55b0f6a7ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2312787389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 0x55b0f6a7c0c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2312787389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a76ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 -- 172.21.15.49:0/2312787389 >> 172.21.15.49:0/2312787389 conn(0x55b0f6a72140 msgr2=0x55b0f6a72540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 -- 172.21.15.49:0/2312787389 shutdown_connections 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 -- 172.21.15.49:0/2312787389 wait complete. 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 -- start start 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.526+0000 7f5273f7c740 1 Processor -- start 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a80d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 0x55b0f6a812c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a88300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f5273f7c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6a7f6e0 con 0x55b0f6a7c730 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f5273f7c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6996ef0 con 0x55b0f6a775c0 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f5273f7c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6a7f860 con 0x55b0f6a76bd0 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f5273f7c740 10 client.0 _ll_register_callbacks cb 0x55b0f6991dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a88300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a80d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a88300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52938/0 (socket says 172.21.15.49:52938) 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a80d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44412/0 (socket says 172.21.15.49:44412) 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526b7fe640 1 -- 172.21.15.49:0/1980615812 learned_addr learned my addr 172.21.15.49:0/1980615812 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526a7fc640 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 0x55b0f6a812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526b7fe640 1 -- 172.21.15.49:0/1980615812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 msgr2=0x55b0f6a80d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526b7fe640 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a80d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526b7fe640 1 -- 172.21.15.49:0/1980615812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 msgr2=0x55b0f6a812c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526b7fe640 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 0x55b0f6a812c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526b7fe640 1 -- 172.21.15.49:0/1980615812 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0f69a3a30 con 0x55b0f6a7c730 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526affd640 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a80d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:32.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f526b7fe640 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a88300 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f525c00ffd0 tx=0x7f525c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0e554ea5cf9a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f52527fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.527+0000 7f52527fc640 1 -- 172.21.15.49:0/1980615812 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c026070 con 0x55b0f6a7c730 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f5273f7c740 1 -- 172.21.15.49:0/1980615812 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b0f69e81c0 con 0x55b0f6a7c730 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f52527fc640 1 -- 172.21.15.49:0/1980615812 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f525c021970 con 0x55b0f6a7c730 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f52527fc640 1 -- 172.21.15.49:0/1980615812 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f525c032050 con 0x55b0f6a7c730 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f5273f7c740 10 client.15051 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f5273f7c740 1 -- 172.21.15.49:0/1980615812 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b0f66ef2d0 con 0x55b0f6a7c730 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f52527fc640 1 -- 172.21.15.49:0/1980615812 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b0f69e81c0 con 0x55b0f6a7c730 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f52527fc640 1 -- 172.21.15.49:0/1980615812 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f525c01f050 con 0x55b0f6a7c730 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f5273f7c740 10 client.15051 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f5273f7c740 10 client.15051 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f5273f7c740 20 client.15051 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:32.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f5273f7c740 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b0f6a8d8b0 0x55b0f6a8ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f5273f7c740 4 client.15051 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f5273f7c740 1 -- 172.21.15.49:0/1980615812 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0f69e2510 con 0x55b0f6a8d8b0 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.528+0000 7f526affd640 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b0f6a8d8b0 0x55b0f6a8ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.529+0000 7f52527fc640 10 client.15051 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.529+0000 7f526affd640 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b0f6a8d8b0 0x55b0f6a8ed20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a36a0339fc4e5194 server_cookie=a0cfe85ee00438f5 in_seq=0 out_seq=0 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.529+0000 7f52527fc640 1 -- 172.21.15.49:0/1980615812 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b0f69e2510 con 0x55b0f6a8d8b0 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.529+0000 7f52527fc640 10 client.15051 handle_command_reply: tid=0 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 client.15051 shutdown 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 -- 172.21.15.49:0/1980615812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 msgr2=0x55b0f6a88300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a88300 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f525c00ffd0 tx=0x7f525c004a40 comp rx=0 tx=0).stop 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 -- 172.21.15.49:0/1980615812 shutdown_connections 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b0f6a8d8b0 0x55b0f6a8ed20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 0x55b0f6a812c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a88300 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f52527fc640 0 client.15051 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:49:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 --2- 172.21.15.49:0/1980615812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a80d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 -- 172.21.15.49:0/1980615812 >> 172.21.15.49:0/1980615812 conn(0x55b0f6a72140 msgr2=0x55b0f6a71380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 -- 172.21.15.49:0/1980615812 shutdown_connections 2024-03-18T00:49:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 1 -- 172.21.15.49:0/1980615812 wait complete. 2024-03-18T00:49:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.530+0000 7f5273f7c740 20 client.15051 trim_cache size 0 max 16384 2024-03-18T00:49:32.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f5273f7c740 1 Processor -- start 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f5273f7c740 1 -- start start 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a7eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 0x55b0f6a88220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a78060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f5273f7c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6a8d060 con 0x55b0f6a7c730 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f5273f7c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6a8d730 con 0x55b0f6a76bd0 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f5273f7c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6a8de00 con 0x55b0f6a775c0 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f526b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a78060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f526b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a78060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52948/0 (socket says 172.21.15.49:52948) 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f526b7fe640 1 -- 172.21.15.49:0/937929648 learned_addr learned my addr 172.21.15.49:0/937929648 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f526affd640 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a7eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.548+0000 7f526a7fc640 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 0x55b0f6a88220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f526b7fe640 1 -- 172.21.15.49:0/937929648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 msgr2=0x55b0f6a88220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f526b7fe640 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 0x55b0f6a88220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f526b7fe640 1 -- 172.21.15.49:0/937929648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a76bd0 msgr2=0x55b0f6a7eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f526b7fe640 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a7eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f526b7fe640 1 -- 172.21.15.49:0/937929648 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0f67b96b0 con 0x55b0f6a7c730 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f526affd640 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a7eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f526b7fe640 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a78060 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f525c00a010 tx=0x7f525c02a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f413daaab6e2c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f5260ff9640 1 -- 172.21.15.49:0/937929648 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c026070 con 0x55b0f6a7c730 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f5260ff9640 1 -- 172.21.15.49:0/937929648 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f525c021770 con 0x55b0f6a7c730 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.549+0000 7f5260ff9640 1 -- 172.21.15.49:0/937929648 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c0226e0 con 0x55b0f6a7c730 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 -- 172.21.15.49:0/937929648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 msgr2=0x55b0f6a78060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a78060 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f525c00a010 tx=0x7f525c02a7b0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 -- 172.21.15.49:0/937929648 shutdown_connections 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6a78060 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a775c0 0x55b0f6a88220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 --2- 172.21.15.49:0/937929648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a7eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 -- 172.21.15.49:0/937929648 >> 172.21.15.49:0/937929648 conn(0x55b0f6a72140 msgr2=0x55b0f6a80d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 -- 172.21.15.49:0/937929648 shutdown_connections 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 -- 172.21.15.49:0/937929648 wait complete. 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 -- start start 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 Processor -- start 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a7af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 0x55b0f6a88220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6af1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6a8eba0 con 0x55b0f6a7c730 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6a8e4d0 con 0x55b0f6a775c0 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b0f6a8f270 con 0x55b0f6a76bd0 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f5273f7c740 10 client.0 _ll_register_callbacks cb 0x7f525c00f790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f526b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6af1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f526b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6af1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52952/0 (socket says 172.21.15.49:52952) 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f526b7fe640 1 -- 172.21.15.49:0/2876945564 learned_addr learned my addr 172.21.15.49:0/2876945564 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f526affd640 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a7af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.550+0000 7f526a7fc640 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 0x55b0f6a88220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f526b7fe640 1 -- 172.21.15.49:0/2876945564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 msgr2=0x55b0f6a7af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f526b7fe640 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a7af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f526b7fe640 1 -- 172.21.15.49:0/2876945564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 msgr2=0x55b0f6a88220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f526b7fe640 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 0x55b0f6a88220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f526b7fe640 1 -- 172.21.15.49:0/2876945564 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0f6af1bd0 con 0x55b0f6a7c730 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f526b7fe640 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6af1610 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f525c000f80 tx=0x7f525c011650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1876f701f53722 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f52527fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f52527fc640 1 -- 172.21.15.49:0/2876945564 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f525c026040 con 0x55b0f6a7c730 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f52527fc640 1 -- 172.21.15.49:0/2876945564 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f525c021d30 con 0x55b0f6a7c730 2024-03-18T00:49:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f5273f7c740 1 -- 172.21.15.49:0/2876945564 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f525c021770 con 0x55b0f6a7c730 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f52527fc640 1 -- 172.21.15.49:0/2876945564 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f525c00fe50 con 0x55b0f6a7c730 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f5273f7c740 10 client.15057 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f5273f7c740 1 -- 172.21.15.49:0/2876945564 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b0f6af2ed0 con 0x55b0f6a7c730 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f52527fc640 1 -- 172.21.15.49:0/2876945564 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f525c021770 con 0x55b0f6a7c730 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.551+0000 7f52527fc640 1 -- 172.21.15.49:0/2876945564 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f525c01f040 con 0x55b0f6a7c730 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 10 client.15057 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 10 client.15057 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 20 client.15057 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b0f6af77b0 0x55b0f6af64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 4 client.15057 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 -- 172.21.15.49:0/2876945564 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "status"}) -- 0x55b0f6960a00 con 0x55b0f6af77b0 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f526affd640 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b0f6af77b0 0x55b0f6af64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f52527fc640 10 client.15057 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:49:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f526affd640 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b0f6af77b0 0x55b0f6af64c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1232ed00520017f7 server_cookie=339d51d06c5b658 in_seq=0 out_seq=0 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f52527fc640 1 -- 172.21.15.49:0/2876945564 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b0f6960a00 con 0x55b0f6af77b0 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f52527fc640 10 client.15057 handle_command_reply: tid=0 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 client.15057 shutdown 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 -- 172.21.15.49:0/2876945564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 msgr2=0x55b0f6af1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6af1610 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f525c000f80 tx=0x7f525c011650 comp rx=0 tx=0).stop 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 -- 172.21.15.49:0/2876945564 shutdown_connections 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b0f6af77b0 0x55b0f6af64c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b0f6a775c0 0x55b0f6a88220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b0f6a7c730 0x55b0f6af1610 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 --2- 172.21.15.49:0/2876945564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b0f6a76bd0 0x55b0f6a7af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f52527fc640 0 client.15057 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.552+0000 7f5273f7c740 1 -- 172.21.15.49:0/2876945564 >> 172.21.15.49:0/2876945564 conn(0x55b0f6a72140 msgr2=0x55b0f6a753a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f5273f7c740 1 -- 172.21.15.49:0/2876945564 shutdown_connections 2024-03-18T00:49:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f5273f7c740 1 -- 172.21.15.49:0/2876945564 wait complete. 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f5273f7c740 20 client.15057 trim_cache size 0 max 16384 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 4, 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24463, 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 81.549500554000005, 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:32.561 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:32.562 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 109.77469691899999 2024-03-18T00:49:32.562 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:32.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 -- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52580a2890 msgr2=0x7f52580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52580a2890 0x7f52580a4d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x55b0f6a77e20 tx=0x7f525c004ab0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 -- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f526c1018d0 msgr2=0x7f526c1980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f526c1018d0 0x7f526c1980a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5254009230 tx=0x7f525402c650 comp rx=0 tx=0).stop 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 -- 172.21.15.49:0/300917279 shutdown_connections 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52580a2890 0x7f52580a4d50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f526c102290 0x7f526c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f526c1018d0 0x7f526c1980a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 --2- 172.21.15.49:0/300917279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f526c1006e0 0x7f526c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 -- 172.21.15.49:0/300917279 >> 172.21.15.49:0/300917279 conn(0x7f526c0fbc70 msgr2=0x7f526c104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 -- 172.21.15.49:0/300917279 shutdown_connections 2024-03-18T00:49:32.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.553+0000 7f52712b2640 1 -- 172.21.15.49:0/300917279 wait complete. 2024-03-18T00:49:32.565 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 -- 172.21.15.49:0/2698093897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1018f0 msgr2=0x7fa9dc101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2698093897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1018f0 0x7fa9dc101d70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc009860 tx=0x7fa9cc02bb70 comp rx=0 tx=0).stop 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 -- 172.21.15.49:0/2698093897 shutdown_connections 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2698093897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc1022b0 0x7fa9dc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2698093897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1018f0 0x7fa9dc101d70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2698093897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc100700 0x7fa9dc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 -- 172.21.15.49:0/2698093897 >> 172.21.15.49:0/2698093897 conn(0x7fa9dc0fbc90 msgr2=0x7fa9dc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 -- 172.21.15.49:0/2698093897 shutdown_connections 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 -- 172.21.15.49:0/2698093897 wait complete. 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 Processor -- start 2024-03-18T00:49:32.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.674+0000 7fa9e226c640 1 -- start start 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e226c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc100700 0x7fa9dc199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e226c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc1018f0 0x7fa9dc19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e226c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1022b0 0x7fa9dc1a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e226c640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc10c360 con 0x7fa9dc1022b0 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e226c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc10c1e0 con 0x7fa9dc100700 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e226c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa9dc10c4e0 con 0x7fa9dc1018f0 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e0809640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1022b0 0x7fa9dc1a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e0809640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1022b0 0x7fa9dc1a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52970/0 (socket says 172.21.15.49:52970) 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e0809640 1 -- 172.21.15.49:0/2765688206 learned_addr learned my addr 172.21.15.49:0/2765688206 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc100700 0x7fa9dc199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc1018f0 0x7fa9dc19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e0809640 1 -- 172.21.15.49:0/2765688206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc1018f0 msgr2=0x7fa9dc19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e0809640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc1018f0 0x7fa9dc19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e0809640 1 -- 172.21.15.49:0/2765688206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc100700 msgr2=0x7fa9dc199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e0809640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc100700 0x7fa9dc199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e0809640 1 -- 172.21.15.49:0/2765688206 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9dc1a1970 con 0x7fa9dc1022b0 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e0809640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1022b0 0x7fa9dc1a1300 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400efd0 tx=0x7fa9d400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9d97fa640 1 -- 172.21.15.49:0/2765688206 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4003b70 con 0x7fa9dc1022b0 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9d97fa640 1 -- 172.21.15.49:0/2765688206 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4003d10 con 0x7fa9dc1022b0 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e226c640 1 -- 172.21.15.49:0/2765688206 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9dc1a1c60 con 0x7fa9dc1022b0 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9d97fa640 1 -- 172.21.15.49:0/2765688206 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40109f0 con 0x7fa9dc1022b0 2024-03-18T00:49:32.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.675+0000 7fa9e226c640 1 -- 172.21.15.49:0/2765688206 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9dc1a2180 con 0x7fa9dc1022b0 2024-03-18T00:49:32.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:32 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1071244328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:32.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:32 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2166961607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:32.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:32 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3107494989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:32.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.677+0000 7fa9d97fa640 1 -- 172.21.15.49:0/2765688206 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fa9d4004020 con 0x7fa9dc1022b0 2024-03-18T00:49:32.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9e4e58740 1 Processor -- start 2024-03-18T00:49:32.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9e4e58740 1 -- start start 2024-03-18T00:49:32.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8ebd0 0x55b961f8eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8f5c0 0x55b961f940c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 0x55b961f96bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b961ef2c80 con 0x55b961f8f5c0 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b961e9baa0 con 0x55b961f94730 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b961eeed20 con 0x55b961f8ebd0 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9d97fa640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9c80a2910 0x7fa9c80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8f5c0 0x55b961f940c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8f5c0 0x55b961f940c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52972/0 (socket says 172.21.15.49:52972) 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9dbfff640 1 -- 172.21.15.49:0/1994520191 learned_addr learned my addr 172.21.15.49:0/1994520191 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/1994520191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8ebd0 0x55b961f8eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9e0809640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9c80a2910 0x7fa9c80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/1994520191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 0x55b961f96bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9d97fa640 1 -- 172.21.15.49:0/2765688206 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa9d40cefa0 con 0x7fa9dc1022b0 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9dbfff640 1 -- 172.21.15.49:0/1994520191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8ebd0 msgr2=0x55b961f8eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/1994520191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8ebd0 0x55b961f8eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9dbfff640 1 -- 172.21.15.49:0/1994520191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 msgr2=0x55b961f96bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/1994520191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 0x55b961f96bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.678+0000 7fa9dbfff640 1 -- 172.21.15.49:0/1994520191 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b961ebdff0 con 0x55b961f8f5c0 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/1994520191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8f5c0 0x55b961f940c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009c80 tx=0x7fa9c4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea6a08fd0543d2b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9e0809640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9c80a2910 0x7fa9c80a4dd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40a3a30 tx=0x7fa9d4086040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9d27fc640 1 -- 172.21.15.49:0/1994520191 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4009d70 con 0x55b961f8f5c0 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9d27fc640 1 -- 172.21.15.49:0/1994520191 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9c4005500 con 0x55b961f8f5c0 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9d27fc640 1 -- 172.21.15.49:0/1994520191 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4019670 con 0x55b961f8f5c0 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9e4e58740 1 -- 172.21.15.49:0/1994520191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8f5c0 msgr2=0x55b961f940c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/1994520191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8f5c0 0x55b961f940c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009c80 tx=0x7fa9c4011810 comp rx=0 tx=0).stop 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9e4e58740 1 -- 172.21.15.49:0/1994520191 shutdown_connections 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/1994520191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 0x55b961f96bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/1994520191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8f5c0 0x55b961f940c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/1994520191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8ebd0 0x55b961f8eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.679+0000 7fa9e4e58740 1 -- 172.21.15.49:0/1994520191 >> 172.21.15.49:0/1994520191 conn(0x55b961f8a140 msgr2=0x55b961f8a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 -- 172.21.15.49:0/1994520191 shutdown_connections 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 -- 172.21.15.49:0/1994520191 wait complete. 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 -- start start 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 Processor -- start 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f98d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 0x55b961f992c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 0x55b961fa0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b961f976e0 con 0x55b961f8ebd0 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b961eaeef0 con 0x55b961f8f5c0 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b961f97860 con 0x55b961f94730 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9e4e58740 10 client.0 _ll_register_callbacks cb 0x55b961ea9dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:32.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f98d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f98d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52986/0 (socket says 172.21.15.49:52986) 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9db7fe640 1 -- 172.21.15.49:0/4271347111 learned_addr learned my addr 172.21.15.49:0/4271347111 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.680+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 0x55b961f992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 0x55b961fa0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9db7fe640 1 -- 172.21.15.49:0/4271347111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 msgr2=0x55b961fa0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 0x55b961fa0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9db7fe640 1 -- 172.21.15.49:0/4271347111 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 msgr2=0x55b961f992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 0x55b961f992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9db7fe640 1 -- 172.21.15.49:0/4271347111 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b961ebba30 con 0x55b961f8ebd0 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f98d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc032570 tx=0x7fa9cc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc44462ffbaa3be4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:32.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/4271347111 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc049030 con 0x55b961f8ebd0 2024-03-18T00:49:32.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/4271347111 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa9cc004100 con 0x55b961f8ebd0 2024-03-18T00:49:32.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4271347111 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9c4005500 con 0x55b961f8ebd0 2024-03-18T00:49:32.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/4271347111 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa9cc054070 con 0x55b961f8ebd0 2024-03-18T00:49:32.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9e4e58740 10 client.15069 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:32.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4271347111 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b961c072d0 con 0x55b961f8ebd0 2024-03-18T00:49:32.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/4271347111 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9c4005500 con 0x55b961f8ebd0 2024-03-18T00:49:32.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.681+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/4271347111 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fa9cc03f030 con 0x55b961f8ebd0 2024-03-18T00:49:32.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.682+0000 7fa9e4e58740 10 client.15069 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:32.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.682+0000 7fa9e4e58740 10 client.15069 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T00:49:32.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.682+0000 7fa9e4e58740 20 client.15069 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:32.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.682+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b961fa5840 0x55b961fa6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.682+0000 7fa9e4e58740 4 client.15069 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:32.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.682+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4271347111 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b961f001c0 con 0x55b961fa5840 2024-03-18T00:49:32.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.682+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b961fa5840 0x55b961fa6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.682+0000 7fa9bb7fe640 10 client.15069 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:49:32.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.682+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b961fa5840 0x55b961fa6cb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=94e53905d672f202 server_cookie=c1dd55b5ccd9fe4f in_seq=0 out_seq=0 2024-03-18T00:49:32.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/4271347111 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b961f001c0 con 0x55b961fa5840 2024-03-18T00:49:32.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9bb7fe640 10 client.15069 handle_command_reply: tid=0 2024-03-18T00:49:32.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9e4e58740 1 client.15069 shutdown 2024-03-18T00:49:32.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4271347111 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 msgr2=0x55b961f98d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f98d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc032570 tx=0x7fa9cc005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:32.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4271347111 shutdown_connections 2024-03-18T00:49:32.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b961fa5840 0x55b961fa6cb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 0x55b961f992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 0x55b961fa0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4271347111 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f98d80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9bb7fe640 0 client.15069 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:49:32.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.683+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4271347111 >> 172.21.15.49:0/4271347111 conn(0x55b961f8a140 msgr2=0x55b961f89380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.684+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4271347111 shutdown_connections 2024-03-18T00:49:32.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.684+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4271347111 wait complete. 2024-03-18T00:49:32.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.684+0000 7fa9e4e58740 20 client.15069 trim_cache size 0 max 16384 2024-03-18T00:49:32.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:32 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1071244328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:32.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:32 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2166961607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:32.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:32 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3107494989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:32.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9e4e58740 1 Processor -- start 2024-03-18T00:49:32.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9e4e58740 1 -- start start 2024-03-18T00:49:32.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f96b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 0x55b961fa02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 0x55b961f990a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b961fa4ff0 con 0x55b961f8ebd0 2024-03-18T00:49:32.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b961fa56c0 con 0x55b961f8f5c0 2024-03-18T00:49:32.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b961fa5d90 con 0x55b961f94730 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f96b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f96b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52998/0 (socket says 172.21.15.49:52998) 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 -- 172.21.15.49:0/4214894653 learned_addr learned my addr 172.21.15.49:0/4214894653 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/4214894653 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 0x55b961fa02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/4214894653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 0x55b961f990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 -- 172.21.15.49:0/4214894653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 msgr2=0x55b961f990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/4214894653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 0x55b961f990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 -- 172.21.15.49:0/4214894653 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 msgr2=0x55b961fa02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/4214894653 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 0x55b961fa02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 -- 172.21.15.49:0/4214894653 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b961cd16b0 con 0x55b961f8ebd0 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.702+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/4214894653 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f96b20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc034b10 tx=0x7fa9cc005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b849e6d5167e82cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9d1ffb640 1 -- 172.21.15.49:0/4214894653 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc000e60 con 0x55b961f8ebd0 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9d1ffb640 1 -- 172.21.15.49:0/4214894653 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9cc032940 con 0x55b961f8ebd0 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9d1ffb640 1 -- 172.21.15.49:0/4214894653 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc049040 con 0x55b961f8ebd0 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4214894653 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 msgr2=0x55b961f96b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4214894653 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f96b20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc034b10 tx=0x7fa9cc005bd0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4214894653 shutdown_connections 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4214894653 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f8f5c0 0x55b961fa02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4214894653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f94730 0x55b961f990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/4214894653 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f96b20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4214894653 >> 172.21.15.49:0/4214894653 conn(0x55b961f8a140 msgr2=0x55b961f98990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4214894653 shutdown_connections 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 -- 172.21.15.49:0/4214894653 wait complete. 2024-03-18T00:49:32.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.703+0000 7fa9e4e58740 1 -- start start 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9e4e58740 1 Processor -- start 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f9b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8f5c0 0x55b961f9b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9e4e58740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 0x55b961f9e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b961fa6b30 con 0x55b961f8ebd0 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b961fa6460 con 0x55b961f94730 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9e4e58740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b961fa7200 con 0x55b961f8f5c0 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9e4e58740 10 client.0 _ll_register_callbacks cb 0x7fa9cc0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f9b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f9b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53006/0 (socket says 172.21.15.49:53006) 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 -- 172.21.15.49:0/2159428726 learned_addr learned my addr 172.21.15.49:0/2159428726 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8f5c0 0x55b961f9b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 0x55b961f9e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 -- 172.21.15.49:0/2159428726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8f5c0 msgr2=0x55b961f9b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8f5c0 0x55b961f9b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 -- 172.21.15.49:0/2159428726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 msgr2=0x55b961f9e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 0x55b961f9e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 -- 172.21.15.49:0/2159428726 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b961f9eed0 con 0x55b961f8ebd0 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9db7fe640 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f9b3d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc02c0b0 tx=0x7fa9cc02bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c12c76e39a8706 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.704+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/2159428726 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc049040 con 0x55b961f8ebd0 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/2159428726 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa9cc004100 con 0x55b961f8ebd0 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9e4e58740 1 -- 172.21.15.49:0/2159428726 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9cc032940 con 0x55b961f8ebd0 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/2159428726 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa9cc039040 con 0x55b961f8ebd0 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9e4e58740 10 client.15075 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9e4e58740 1 -- 172.21.15.49:0/2159428726 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b961fa01d0 con 0x55b961f8ebd0 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/2159428726 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9cc032940 con 0x55b961f8ebd0 2024-03-18T00:49:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/2159428726 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fa9cc03f040 con 0x55b961f8ebd0 2024-03-18T00:49:32.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9e4e58740 10 client.15075 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:32.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9e4e58740 10 client.15075 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T00:49:32.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9e4e58740 20 client.15075 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:32.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b96200f6d0 0x55b96200e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9e4e58740 4 client.15075 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:32.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9e4e58740 1 -- 172.21.15.49:0/2159428726 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "status"}) -- 0x55b961efa510 con 0x55b96200f6d0 2024-03-18T00:49:32.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.705+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b96200f6d0 0x55b96200e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9bb7fe640 10 client.15075 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:49:32.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9dbfff640 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b96200f6d0 0x55b96200e3e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b3628a5b474b9625 server_cookie=5504a99b7624f741 in_seq=0 out_seq=0 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9bb7fe640 1 -- 172.21.15.49:0/2159428726 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b961efa510 con 0x55b96200f6d0 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9bb7fe640 10 client.15075 handle_command_reply: tid=0 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 client.15075 shutdown 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 -- 172.21.15.49:0/2159428726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 msgr2=0x55b961f9b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f9b3d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc02c0b0 tx=0x7fa9cc02bb70 comp rx=0 tx=0).stop 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 -- 172.21.15.49:0/2159428726 shutdown_connections 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b96200f6d0 0x55b96200e3e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b961f94730 0x55b961f9e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b961f8f5c0 0x55b961f9b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 --2- 172.21.15.49:0/2159428726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b961f8ebd0 0x55b961f9b3d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9bb7fe640 0 client.15075 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 -- 172.21.15.49:0/2159428726 >> 172.21.15.49:0/2159428726 conn(0x55b961f8a140 msgr2=0x55b961f8d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 -- 172.21.15.49:0/2159428726 shutdown_connections 2024-03-18T00:49:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 1 -- 172.21.15.49:0/2159428726 wait complete. 2024-03-18T00:49:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.706+0000 7fa9e4e58740 20 client.15075 trim_cache size 0 max 16384 2024-03-18T00:49:32.714 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:32.714 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:32.714 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 3, 2024-03-18T00:49:32.714 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24470, 2024-03-18T00:49:32.714 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:32.714 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:32.714 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:32.714 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 82.709792651000001, 2024-03-18T00:49:32.715 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:32.715 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:32.715 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:32.715 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 104.07909589400001 2024-03-18T00:49:32.715 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 -- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9c80a2910 msgr2=0x7fa9c80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9c80a2910 0x7fa9c80a4dd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40a3a30 tx=0x7fa9d4086040 comp rx=0 tx=0).stop 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 -- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1022b0 msgr2=0x7fa9dc1a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1022b0 0x7fa9dc1a1300 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa9d400efd0 tx=0x7fa9d400c6a0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 -- 172.21.15.49:0/2765688206 shutdown_connections 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9c80a2910 0x7fa9c80a4dd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9dc1022b0 0x7fa9dc1a1300 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9dc1018f0 0x7fa9dc19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 --2- 172.21.15.49:0/2765688206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9dc100700 0x7fa9dc199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 -- 172.21.15.49:0/2765688206 >> 172.21.15.49:0/2765688206 conn(0x7fa9dc0fbc90 msgr2=0x7fa9dc0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 -- 172.21.15.49:0/2765688206 shutdown_connections 2024-03-18T00:49:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.707+0000 7fa9e226c640 1 -- 172.21.15.49:0/2765688206 wait complete. 2024-03-18T00:49:32.717 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T00:49:32.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.826+0000 7fbd47d53640 1 -- 172.21.15.49:0/2943039826 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd40101800 msgr2=0x7fbd40101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.826+0000 7fbd47d53640 1 --2- 172.21.15.49:0/2943039826 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd40101800 0x7fbd40101c20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009900 tx=0x7fbd3c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:32.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.826+0000 7fbd47d53640 1 -- 172.21.15.49:0/2943039826 shutdown_connections 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.826+0000 7fbd47d53640 1 --2- 172.21.15.49:0/2943039826 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd401033b0 0x7fbd40109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.826+0000 7fbd47d53640 1 --2- 172.21.15.49:0/2943039826 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd401029f0 0x7fbd40102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.826+0000 7fbd47d53640 1 --2- 172.21.15.49:0/2943039826 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd40101800 0x7fbd40101c20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.826+0000 7fbd47d53640 1 -- 172.21.15.49:0/2943039826 >> 172.21.15.49:0/2943039826 conn(0x7fbd400fcd70 msgr2=0x7fbd400ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.826+0000 7fbd47d53640 1 -- 172.21.15.49:0/2943039826 shutdown_connections 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.826+0000 7fbd47d53640 1 -- 172.21.15.49:0/2943039826 wait complete. 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd47d53640 1 Processor -- start 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd47d53640 1 -- start start 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd47d53640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd40101800 0x7fbd40197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd47d53640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd401029f0 0x7fbd40198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd47d53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd401033b0 0x7fbd4019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd47d53640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4010d470 con 0x7fbd401029f0 2024-03-18T00:49:32.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd47d53640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4010d2f0 con 0x7fbd40101800 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd47d53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4010d5f0 con 0x7fbd401033b0 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd452ee640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd401029f0 0x7fbd40198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd452ee640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd401029f0 0x7fbd40198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53024/0 (socket says 172.21.15.49:53024) 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd452ee640 1 -- 172.21.15.49:0/3070629530 learned_addr learned my addr 172.21.15.49:0/3070629530 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd45aef640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd40101800 0x7fbd40197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd462f0640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd401033b0 0x7fbd4019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd452ee640 1 -- 172.21.15.49:0/3070629530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd401033b0 msgr2=0x7fbd4019f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd452ee640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd401033b0 0x7fbd4019f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd452ee640 1 -- 172.21.15.49:0/3070629530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd40101800 msgr2=0x7fbd40197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd452ee640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd40101800 0x7fbd40197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.827+0000 7fbd452ee640 1 -- 172.21.15.49:0/3070629530 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4019f790 con 0x7fbd401029f0 2024-03-18T00:49:32.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.828+0000 7fbd452ee640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd401029f0 0x7fbd40198100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000e9f0 tx=0x7fbd3000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.828+0000 7fbd36ffd640 1 -- 172.21.15.49:0/3070629530 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3000ccc0 con 0x7fbd401029f0 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.828+0000 7fbd47d53640 1 -- 172.21.15.49:0/3070629530 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4019fa80 con 0x7fbd401029f0 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.828+0000 7fbd36ffd640 1 -- 172.21.15.49:0/3070629530 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd3000ce60 con 0x7fbd401029f0 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.828+0000 7fbd36ffd640 1 -- 172.21.15.49:0/3070629530 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30010690 con 0x7fbd401029f0 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.828+0000 7fbd47d53640 1 -- 172.21.15.49:0/3070629530 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd4019ffd0 con 0x7fbd401029f0 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.830+0000 7fbd36ffd640 1 -- 172.21.15.49:0/3070629530 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fbd300046c0 con 0x7fbd401029f0 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.830+0000 7fbd4aa1b740 1 Processor -- start 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd4aa1b740 1 -- start start 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a30fb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a31004d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a3102fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c4a305ac80 con 0x55c4a30fafe0 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3003aa0 con 0x55c4a30fb9d0 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3056d20 con 0x55c4a3100b40 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd36ffd640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd2c0a2890 0x7fbd2c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd45aef640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a31004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd452ee640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a3102fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd462f0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a30fb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd45aef640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a31004d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:57344/0 (socket says 172.21.15.49:57344) 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd45aef640 1 -- 172.21.15.49:0/3761974839 learned_addr learned my addr 172.21.15.49:0/3761974839 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd462f0640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd2c0a2890 0x7fbd2c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd36ffd640 1 -- 172.21.15.49:0/3070629530 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbd300cefc0 con 0x7fbd401029f0 2024-03-18T00:49:32.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd45aef640 1 -- 172.21.15.49:0/3761974839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 msgr2=0x55c4a3102fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd45aef640 1 --2- 172.21.15.49:0/3761974839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a3102fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd45aef640 1 -- 172.21.15.49:0/3761974839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 msgr2=0x55c4a30fb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd45aef640 1 --2- 172.21.15.49:0/3761974839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a30fb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.831+0000 7fbd45aef640 1 -- 172.21.15.49:0/3761974839 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a3025ff0 con 0x55c4a30fb9d0 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.832+0000 7fbd45aef640 1 --2- 172.21.15.49:0/3761974839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a31004d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009900 tx=0x7fbd3c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ed8b4436ee45ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.832+0000 7fbd462f0640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd2c0a2890 0x7fbd2c0a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55c4a30fc230 tx=0x7fbd38009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.832+0000 7fbd2bfff640 1 -- 172.21.15.49:0/3761974839 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c02cb80 con 0x55c4a30fb9d0 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.832+0000 7fbd2bfff640 1 -- 172.21.15.49:0/3761974839 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd3c005040 con 0x55c4a30fb9d0 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.832+0000 7fbd2bfff640 1 -- 172.21.15.49:0/3761974839 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c004920 con 0x55c4a30fb9d0 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.832+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3761974839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 msgr2=0x55c4a31004d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.832+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3761974839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a31004d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009900 tx=0x7fbd3c009560 comp rx=0 tx=0).stop 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3761974839 shutdown_connections 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3761974839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a31004d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3761974839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a3102fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3761974839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a30fb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3761974839 >> 172.21.15.49:0/3761974839 conn(0x55c4a30f2140 msgr2=0x55c4a30f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3761974839 shutdown_connections 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3761974839 wait complete. 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 -- start start 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 Processor -- start 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fafe0 0x55c4a31051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fb9d0 0x55c4a3105700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a310c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.833+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3103a90 con 0x55c4a30fb9d0 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3016ef0 con 0x55c4a30fafe0 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3103c10 con 0x55c4a3100b40 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd4aa1b740 10 client.0 _ll_register_callbacks cb 0x55c4a3011dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd452ee640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fb9d0 0x55c4a3105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd452ee640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fb9d0 0x55c4a3105700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53044/0 (socket says 172.21.15.49:53044) 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd452ee640 1 -- 172.21.15.49:0/3132640528 learned_addr learned my addr 172.21.15.49:0/3132640528 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd45aef640 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fafe0 0x55c4a31051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd462f0640 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a310c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd452ee640 1 -- 172.21.15.49:0/3132640528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 msgr2=0x55c4a310c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd452ee640 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a310c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd452ee640 1 -- 172.21.15.49:0/3132640528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fafe0 msgr2=0x55c4a31051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd452ee640 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fafe0 0x55c4a31051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd452ee640 1 -- 172.21.15.49:0/3132640528 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a3023a30 con 0x55c4a30fb9d0 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd452ee640 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fb9d0 0x55c4a3105700 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbd300981b0 tx=0x7fbd3006b3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf9faf237b012b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd297fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd297fa640 1 -- 172.21.15.49:0/3132640528 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd30066ac0 con 0x55c4a30fb9d0 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd297fa640 1 -- 172.21.15.49:0/3132640528 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbd30097e50 con 0x55c4a30fb9d0 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3132640528 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd3c005040 con 0x55c4a30fb9d0 2024-03-18T00:49:32.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd297fa640 1 -- 172.21.15.49:0/3132640528 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbd3009b7e0 con 0x55c4a30fb9d0 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd4aa1b740 10 client.15087 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3132640528 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4a2d6f2d0 con 0x55c4a30fb9d0 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.834+0000 7fbd297fa640 1 -- 172.21.15.49:0/3132640528 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd3c005040 con 0x55c4a30fb9d0 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd297fa640 1 -- 172.21.15.49:0/3132640528 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fbd300a3af0 con 0x55c4a30fb9d0 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd4aa1b740 10 client.15087 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd4aa1b740 10 client.15087 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd4aa1b740 20 client.15087 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55c4a3111ca0 0x55c4a3113110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd4aa1b740 4 client.15087 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3132640528 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4a30681c0 con 0x55c4a3111ca0 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd45aef640 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55c4a3111ca0 0x55c4a3113110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd45aef640 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55c4a3111ca0 0x55c4a3113110 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1cb9c9bd5ff0a865 server_cookie=abbf1e8d71e3fac3 in_seq=0 out_seq=0 2024-03-18T00:49:32.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.835+0000 7fbd297fa640 10 client.15087 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd297fa640 1 -- 172.21.15.49:0/3132640528 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fbd3c0509c0 con 0x55c4a3111ca0 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd297fa640 10 client.15087 handle_command_reply: tid=0 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 client.15087 shutdown 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3132640528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fb9d0 msgr2=0x55c4a3105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fb9d0 0x55c4a3105700 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbd300981b0 tx=0x7fbd3006b3c0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3132640528 shutdown_connections 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55c4a3111ca0 0x55c4a3113110 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a310c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fb9d0 0x55c4a3105700 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/3132640528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fafe0 0x55c4a31051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd297fa640 0 client.15087 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3132640528 >> 172.21.15.49:0/3132640528 conn(0x55c4a30f2140 msgr2=0x55c4a30f1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3132640528 shutdown_connections 2024-03-18T00:49:32.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/3132640528 wait complete. 2024-03-18T00:49:32.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.836+0000 7fbd4aa1b740 20 client.15087 trim_cache size 0 max 16384 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd4aa1b740 1 Processor -- start 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd4aa1b740 1 -- start start 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3104750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a310c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a31001b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3111450 con 0x55c4a30fafe0 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3111b20 con 0x55c4a30fb9d0 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c4a31121f0 con 0x55c4a3100b40 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd45aef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3104750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd45aef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3104750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53058/0 (socket says 172.21.15.49:53058) 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd45aef640 1 -- 172.21.15.49:0/2558558732 learned_addr learned my addr 172.21.15.49:0/2558558732 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd452ee640 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a310c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd462f0640 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a31001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd45aef640 1 -- 172.21.15.49:0/2558558732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 msgr2=0x55c4a31001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd45aef640 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a31001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd45aef640 1 -- 172.21.15.49:0/2558558732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 msgr2=0x55c4a310c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd45aef640 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a310c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd45aef640 1 -- 172.21.15.49:0/2558558732 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4a2e396b0 con 0x55c4a30fafe0 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd452ee640 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a310c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd45aef640 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3104750 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c02c170 tx=0x7fbd3c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2faf965f22b4c8ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.855+0000 7fbd2b7fe640 1 -- 172.21.15.49:0/2558558732 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c054070 con 0x55c4a30fafe0 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd2b7fe640 1 -- 172.21.15.49:0/2558558732 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd3c04b040 con 0x55c4a30fafe0 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd2b7fe640 1 -- 172.21.15.49:0/2558558732 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c005560 con 0x55c4a30fafe0 2024-03-18T00:49:32.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/2558558732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 msgr2=0x55c4a3104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3104750 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c02c170 tx=0x7fbd3c000bc0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/2558558732 shutdown_connections 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a310c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a31001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/2558558732 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3104750 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/2558558732 >> 172.21.15.49:0/2558558732 conn(0x55c4a30f2140 msgr2=0x55c4a3104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/2558558732 shutdown_connections 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/2558558732 wait complete. 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 -- start start 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.856+0000 7fbd4aa1b740 1 Processor -- start 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3107750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a3107c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd4aa1b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a310acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3112f90 con 0x55c4a30fafe0 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c4a31128c0 con 0x55c4a30fb9d0 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd4aa1b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c4a3113660 con 0x55c4a3100b40 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd4aa1b740 10 client.0 _ll_register_callbacks cb 0x7fbd3000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd45aef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3107750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd45aef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3107750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53066/0 (socket says 172.21.15.49:53066) 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd45aef640 1 -- 172.21.15.49:0/1152264621 learned_addr learned my addr 172.21.15.49:0/1152264621 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd462f0640 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a310acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd452ee640 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a3107c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd45aef640 1 -- 172.21.15.49:0/1152264621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 msgr2=0x55c4a310acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd45aef640 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a310acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd45aef640 1 -- 172.21.15.49:0/1152264621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 msgr2=0x55c4a3107c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd45aef640 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a3107c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd45aef640 1 -- 172.21.15.49:0/1152264621 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4a310b2a0 con 0x55c4a30fafe0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd45aef640 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3107750 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009900 tx=0x7fbd3c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ec319cc7ac15c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd297fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.857+0000 7fbd297fa640 1 -- 172.21.15.49:0/1152264621 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c054070 con 0x55c4a30fafe0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd297fa640 1 -- 172.21.15.49:0/1152264621 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbd3c0054e0 con 0x55c4a30fafe0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/1152264621 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd3c04b040 con 0x55c4a30fafe0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd297fa640 1 -- 172.21.15.49:0/1152264621 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fbd3c004040 con 0x55c4a30fafe0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd4aa1b740 10 client.15093 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/1152264621 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4a310c5a0 con 0x55c4a30fafe0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd297fa640 1 -- 172.21.15.49:0/1152264621 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd3c04b040 con 0x55c4a30fafe0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd297fa640 1 -- 172.21.15.49:0/1152264621 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fbd3c0046c0 con 0x55c4a30fafe0 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd4aa1b740 10 client.15093 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:32.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd4aa1b740 10 client.15093 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd4aa1b740 20 client.15093 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55c4a317b870 0x55c4a317a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd4aa1b740 4 client.15093 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/1152264621 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "status"}) -- 0x55c4a3062510 con 0x55c4a317b870 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.858+0000 7fbd452ee640 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55c4a317b870 0x55c4a317a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd297fa640 10 client.15093 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd452ee640 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55c4a317b870 0x55c4a317a580 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=272e9b51acf4458c server_cookie=3141460323b27bb0 in_seq=0 out_seq=0 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd297fa640 1 -- 172.21.15.49:0/1152264621 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c4a3062510 con 0x55c4a317b870 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd297fa640 10 client.15093 handle_command_reply: tid=0 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 client.15093 shutdown 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/1152264621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 msgr2=0x55c4a3107750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3107750 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c009900 tx=0x7fbd3c005dd0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/1152264621 shutdown_connections 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55c4a317b870 0x55c4a317a580 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c4a30fb9d0 0x55c4a3107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c4a3100b40 0x55c4a310acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 --2- 172.21.15.49:0/1152264621 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c4a30fafe0 0x55c4a3107750 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd297fa640 0 client.15093 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/1152264621 >> 172.21.15.49:0/1152264621 conn(0x55c4a30f2140 msgr2=0x55c4a3104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/1152264621 shutdown_connections 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 1 -- 172.21.15.49:0/1152264621 wait complete. 2024-03-18T00:49:32.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd4aa1b740 20 client.15093 trim_cache size 0 max 16384 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 2, 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24481, 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 83.871778011999993, 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:32.867 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 107.283131673 2024-03-18T00:49:32.868 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:32.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd47d53640 1 -- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd2c0a2890 msgr2=0x7fbd2c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd47d53640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd2c0a2890 0x7fbd2c0a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x55c4a30fc230 tx=0x7fbd38009500 comp rx=0 tx=0).stop 2024-03-18T00:49:32.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd47d53640 1 -- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd401029f0 msgr2=0x7fbd40198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.859+0000 7fbd47d53640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd401029f0 0x7fbd40198100 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbd3000e9f0 tx=0x7fbd3000edf0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.860+0000 7fbd47d53640 1 -- 172.21.15.49:0/3070629530 shutdown_connections 2024-03-18T00:49:32.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.860+0000 7fbd47d53640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd2c0a2890 0x7fbd2c0a4d50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.860+0000 7fbd47d53640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd401033b0 0x7fbd4019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.860+0000 7fbd47d53640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd401029f0 0x7fbd40198100 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.860+0000 7fbd47d53640 1 --2- 172.21.15.49:0/3070629530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd40101800 0x7fbd40197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.860+0000 7fbd47d53640 1 -- 172.21.15.49:0/3070629530 >> 172.21.15.49:0/3070629530 conn(0x7fbd400fcd70 msgr2=0x7fbd40105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.860+0000 7fbd47d53640 1 -- 172.21.15.49:0/3070629530 shutdown_connections 2024-03-18T00:49:32.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.860+0000 7fbd47d53640 1 -- 172.21.15.49:0/3070629530 wait complete. 2024-03-18T00:49:32.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.978+0000 7f85f4910640 1 -- 172.21.15.49:0/1289105001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f01006e0 msgr2=0x7f85f0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.978+0000 7f85f4910640 1 --2- 172.21.15.49:0/1289105001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f01006e0 0x7f85f0100b00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009ca0 tx=0x7f85e802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.978+0000 7f85f4910640 1 -- 172.21.15.49:0/1289105001 shutdown_connections 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.978+0000 7f85f4910640 1 --2- 172.21.15.49:0/1289105001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f0102290 0x7f85f0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.978+0000 7f85f4910640 1 --2- 172.21.15.49:0/1289105001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f01018d0 0x7f85f0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.978+0000 7f85f4910640 1 --2- 172.21.15.49:0/1289105001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f01006e0 0x7f85f0100b00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.978+0000 7f85f4910640 1 -- 172.21.15.49:0/1289105001 >> 172.21.15.49:0/1289105001 conn(0x7f85f00fbc70 msgr2=0x7f85f00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 -- 172.21.15.49:0/1289105001 shutdown_connections 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 -- 172.21.15.49:0/1289105001 wait complete. 2024-03-18T00:49:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 Processor -- start 2024-03-18T00:49:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 -- start start 2024-03-18T00:49:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f01006e0 0x7f85f0197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f01018d0 0x7f85f01980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f0102290 0x7f85f019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f85f010c350 con 0x7f85f0102290 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f85f010c1d0 con 0x7f85f01018d0 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.979+0000 7f85f4910640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f85f010c4d0 con 0x7f85f01006e0 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85eed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f0102290 0x7f85f019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85eed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f0102290 0x7f85f019f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53078/0 (socket says 172.21.15.49:53078) 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85eed9d640 1 -- 172.21.15.49:0/2769725200 learned_addr learned my addr 172.21.15.49:0/2769725200 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85edd9b640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f01018d0 0x7f85f01980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85ee59c640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f01006e0 0x7f85f0197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85eed9d640 1 -- 172.21.15.49:0/2769725200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f01006e0 msgr2=0x7f85f0197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85eed9d640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f01006e0 0x7f85f0197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85eed9d640 1 -- 172.21.15.49:0/2769725200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f01018d0 msgr2=0x7f85f01980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85eed9d640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f01018d0 0x7f85f01980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85eed9d640 1 -- 172.21.15.49:0/2769725200 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f019f790 con 0x7f85f0102290 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85edd9b640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f01018d0 0x7f85f01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85eed9d640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f0102290 0x7f85f019f120 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f85e000e970 tx=0x7f85e000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85df7fe640 1 -- 172.21.15.49:0/2769725200 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e000ccc0 con 0x7f85f0102290 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85df7fe640 1 -- 172.21.15.49:0/2769725200 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85e000ce60 con 0x7f85f0102290 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85f4910640 1 -- 172.21.15.49:0/2769725200 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f019fa80 con 0x7f85f0102290 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85df7fe640 1 -- 172.21.15.49:0/2769725200 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e0010490 con 0x7f85f0102290 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.980+0000 7f85f4910640 1 -- 172.21.15.49:0/2769725200 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f019ffd0 con 0x7f85f0102290 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.982+0000 7f85df7fe640 1 -- 172.21.15.49:0/2769725200 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f85e0005020 con 0x7f85f0102290 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85f75da740 1 Processor -- start 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85f75da740 1 -- start start 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85f75da740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d3bd0 0x55cb784d3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85f75da740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784d90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85f75da740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85df7fe640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f85d80a2910 0x7f85d80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85f75da740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55cb78437c80 con 0x55cb784d9730 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85f75da740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55cb783e0aa0 con 0x55cb784d3bd0 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85f75da740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cb78433d20 con 0x55cb784d45c0 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85eed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85edd9b640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f85d80a2910 0x7f85d80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85ee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784d90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85ee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784d90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44592/0 (socket says 172.21.15.49:44592) 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85ee59c640 1 -- 172.21.15.49:0/466273042 learned_addr learned my addr 172.21.15.49:0/466273042 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.983+0000 7f85edd9b640 1 --2- 172.21.15.49:0/466273042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d3bd0 0x55cb784d3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85df7fe640 1 -- 172.21.15.49:0/2769725200 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f85e00ceff0 con 0x7f85f0102290 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85ee59c640 1 -- 172.21.15.49:0/466273042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d3bd0 msgr2=0x55cb784d3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85ee59c640 1 --2- 172.21.15.49:0/466273042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d3bd0 0x55cb784d3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85ee59c640 1 -- 172.21.15.49:0/466273042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 msgr2=0x55cb784dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85ee59c640 1 --2- 172.21.15.49:0/466273042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85ee59c640 1 -- 172.21.15.49:0/466273042 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb78402ff0 con 0x55cb784d45c0 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85edd9b640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f85d80a2910 0x7f85d80a4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f85f0199080 tx=0x7f85e4009f20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85ee59c640 1 --2- 172.21.15.49:0/466273042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784d90c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f85f0101730 tx=0x7f85e80323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9d8049a42d03693 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85dcff9640 1 -- 172.21.15.49:0/466273042 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e8043070 con 0x55cb784d45c0 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85dcff9640 1 -- 172.21.15.49:0/466273042 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85e8032c90 con 0x55cb784d45c0 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.984+0000 7f85dcff9640 1 -- 172.21.15.49:0/466273042 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e8039040 con 0x55cb784d45c0 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 -- 172.21.15.49:0/466273042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 msgr2=0x55cb784d90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 --2- 172.21.15.49:0/466273042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784d90c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f85f0101730 tx=0x7f85e80323d0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 -- 172.21.15.49:0/466273042 shutdown_connections 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 --2- 172.21.15.49:0/466273042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 --2- 172.21.15.49:0/466273042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784d90c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 --2- 172.21.15.49:0/466273042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d3bd0 0x55cb784d3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 -- 172.21.15.49:0/466273042 >> 172.21.15.49:0/466273042 conn(0x55cb784cf140 msgr2=0x55cb784cf540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 -- 172.21.15.49:0/466273042 shutdown_connections 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 -- 172.21.15.49:0/466273042 wait complete. 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 -- start start 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.985+0000 7f85f75da740 1 Processor -- start 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85f75da740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784ddd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85f75da740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784de240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85f75da740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 0x55cb784e5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85f75da740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55cb784dc680 con 0x55cb784d3bd0 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85f75da740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55cb783f3ef0 con 0x55cb784d9730 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85f75da740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cb784dc800 con 0x55cb784d45c0 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85f75da740 10 client.0 _ll_register_callbacks cb 0x55cb783eedc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85ee59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784ddd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85ee59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784ddd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53104/0 (socket says 172.21.15.49:53104) 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85ee59c640 1 -- 172.21.15.49:0/1807332566 learned_addr learned my addr 172.21.15.49:0/1807332566 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85edd9b640 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784de240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85eed9d640 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 0x55cb784e5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85ee59c640 1 -- 172.21.15.49:0/1807332566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 msgr2=0x55cb784de240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85ee59c640 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784de240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85ee59c640 1 -- 172.21.15.49:0/1807332566 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 msgr2=0x55cb784e5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85ee59c640 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 0x55cb784e5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85ee59c640 1 -- 172.21.15.49:0/1807332566 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb78400a30 con 0x55cb784d3bd0 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85ee59c640 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784ddd00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009ca0 tx=0x7f85e802c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7287f262ac9444a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85d5ffb640 1 -- 172.21.15.49:0/1807332566 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e804c070 con 0x55cb784d3bd0 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85d5ffb640 1 -- 172.21.15.49:0/1807332566 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f85e8048650 con 0x55cb784d3bd0 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85f75da740 1 -- 172.21.15.49:0/1807332566 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85e8032c90 con 0x55cb784d3bd0 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.986+0000 7f85d5ffb640 1 -- 172.21.15.49:0/1807332566 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f85e8057050 con 0x55cb784d3bd0 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85f75da740 10 client.15105 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85f75da740 1 -- 172.21.15.49:0/1807332566 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cb7814c2d0 con 0x55cb784d3bd0 2024-03-18T00:49:32.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85d5ffb640 1 -- 172.21.15.49:0/1807332566 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85e8032c90 con 0x55cb784d3bd0 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85d5ffb640 1 -- 172.21.15.49:0/1807332566 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f85e803f750 con 0x55cb784d3bd0 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85f75da740 10 client.15105 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85f75da740 10 client.15105 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85f75da740 20 client.15105 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85f75da740 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55cb784ea210 0x55cb784eb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85f75da740 4 client.15105 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85f75da740 1 -- 172.21.15.49:0/1807332566 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cb784451c0 con 0x55cb784ea210 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.987+0000 7f85edd9b640 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55cb784ea210 0x55cb784eb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.988+0000 7f85d5ffb640 10 client.15105 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.988+0000 7f85edd9b640 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55cb784ea210 0x55cb784eb680 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b8fb0728ed3cd7c9 server_cookie=1c7126079f2b7524 in_seq=0 out_seq=0 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85d5ffb640 1 -- 172.21.15.49:0/1807332566 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cb784451c0 con 0x55cb784ea210 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85d5ffb640 10 client.15105 handle_command_reply: tid=0 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 client.15105 shutdown 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 -- 172.21.15.49:0/1807332566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 msgr2=0x55cb784ddd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:32.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784ddd00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009ca0 tx=0x7f85e802c850 comp rx=0 tx=0).stop 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 -- 172.21.15.49:0/1807332566 shutdown_connections 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55cb784ea210 0x55cb784eb680 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 0x55cb784e5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784de240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85d5ffb640 0 client.15105 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 --2- 172.21.15.49:0/1807332566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784ddd00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 -- 172.21.15.49:0/1807332566 >> 172.21.15.49:0/1807332566 conn(0x55cb784cf140 msgr2=0x55cb784ce380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 -- 172.21.15.49:0/1807332566 shutdown_connections 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 1 -- 172.21.15.49:0/1807332566 wait complete. 2024-03-18T00:49:32.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:32.989+0000 7f85f75da740 20 client.15105 trim_cache size 0 max 16384 2024-03-18T00:49:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85f75da740 1 Processor -- start 2024-03-18T00:49:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85f75da740 1 -- start start 2024-03-18T00:49:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85f75da740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d3bd0 0x55cb784d2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85f75da740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d45c0 0x55cb784e5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85f75da740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dcff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85f75da740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55cb784e99c0 con 0x55cb784d9730 2024-03-18T00:49:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85f75da740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55cb784ea090 con 0x55cb784d45c0 2024-03-18T00:49:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85f75da740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cb784ea760 con 0x55cb784d3bd0 2024-03-18T00:49:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85eed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85ee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d3bd0 0x55cb784d2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.007+0000 7f85eed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dcff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53110/0 (socket says 172.21.15.49:53110) 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85edd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d45c0 0x55cb784e5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85eed9d640 1 -- 172.21.15.49:0/3391285116 learned_addr learned my addr 172.21.15.49:0/3391285116 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85eed9d640 1 -- 172.21.15.49:0/3391285116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d3bd0 msgr2=0x55cb784d2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85eed9d640 1 --2- 172.21.15.49:0/3391285116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d3bd0 0x55cb784d2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85eed9d640 1 -- 172.21.15.49:0/3391285116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d45c0 msgr2=0x55cb784e5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85eed9d640 1 --2- 172.21.15.49:0/3391285116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d45c0 0x55cb784e5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85eed9d640 1 -- 172.21.15.49:0/3391285116 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cb782166b0 con 0x55cb784d9730 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85eed9d640 1 --2- 172.21.15.49:0/3391285116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dcff0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f85e000d010 tx=0x7f85e0086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8f26254de84f5d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85d7fff640 1 -- 172.21.15.49:0/3391285116 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e009bf10 con 0x55cb784d9730 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85d7fff640 1 -- 172.21.15.49:0/3391285116 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85e009b810 con 0x55cb784d9730 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.008+0000 7f85d7fff640 1 -- 172.21.15.49:0/3391285116 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e0097e80 con 0x55cb784d9730 2024-03-18T00:49:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 -- 172.21.15.49:0/3391285116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 msgr2=0x55cb784dcff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 --2- 172.21.15.49:0/3391285116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dcff0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f85e000d010 tx=0x7f85e0086040 comp rx=0 tx=0).stop 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 -- 172.21.15.49:0/3391285116 shutdown_connections 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 --2- 172.21.15.49:0/3391285116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d45c0 0x55cb784e5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 --2- 172.21.15.49:0/3391285116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d9730 0x55cb784dcff0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 --2- 172.21.15.49:0/3391285116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d3bd0 0x55cb784d2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 -- 172.21.15.49:0/3391285116 >> 172.21.15.49:0/3391285116 conn(0x55cb784cf140 msgr2=0x55cb784d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 -- 172.21.15.49:0/3391285116 shutdown_connections 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 -- 172.21.15.49:0/3391285116 wait complete. 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 -- start start 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 Processor -- start 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784e0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784e1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 0x55cb784e4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55cb784eb500 con 0x55cb784d3bd0 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55cb784eae30 con 0x55cb784d9730 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cb784ebbd0 con 0x55cb784d45c0 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85f75da740 10 client.0 _ll_register_callbacks cb 0x7f85e0009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85ee59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784e0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85ee59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784e0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53126/0 (socket says 172.21.15.49:53126) 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85ee59c640 1 -- 172.21.15.49:0/899241789 learned_addr learned my addr 172.21.15.49:0/899241789 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85edd9b640 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784e1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.009+0000 7f85eed9d640 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 0x55cb784e4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85ee59c640 1 -- 172.21.15.49:0/899241789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 msgr2=0x55cb784e1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85ee59c640 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784e1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85ee59c640 1 -- 172.21.15.49:0/899241789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 msgr2=0x55cb784e4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85ee59c640 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 0x55cb784e4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85ee59c640 1 -- 172.21.15.49:0/899241789 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cb784e47f0 con 0x55cb784d3bd0 2024-03-18T00:49:33.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85ee59c640 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784e0cf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f85e80324c0 tx=0x7f85e802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459b2418025204ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85d5ffb640 1 -- 172.21.15.49:0/899241789 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e804c040 con 0x55cb784d3bd0 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85d5ffb640 1 -- 172.21.15.49:0/899241789 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f85e803f640 con 0x55cb784d3bd0 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85f75da740 1 -- 172.21.15.49:0/899241789 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85e009b810 con 0x55cb784d3bd0 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85d5ffb640 1 -- 172.21.15.49:0/899241789 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f85e8039070 con 0x55cb784d3bd0 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85f75da740 10 client.15111 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85f75da740 1 -- 172.21.15.49:0/899241789 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55cb78550300 con 0x55cb784d3bd0 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85d5ffb640 1 -- 172.21.15.49:0/899241789 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85e009b810 con 0x55cb784d3bd0 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85d5ffb640 1 -- 172.21.15.49:0/899241789 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f85e8034b90 con 0x55cb784d3bd0 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.010+0000 7f85f75da740 10 client.15111 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 10 client.15111 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 20 client.15111 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55cb78554ea0 0x55cb78553bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 4 client.15111 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 -- 172.21.15.49:0/899241789 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "status"}) -- 0x55cb7843f510 con 0x55cb78554ea0 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85edd9b640 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55cb78554ea0 0x55cb78553bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85d5ffb640 10 client.15111 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85edd9b640 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55cb78554ea0 0x55cb78553bb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ab633d715323944d server_cookie=6c6d0c19a4b84355 in_seq=0 out_seq=0 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85d5ffb640 1 -- 172.21.15.49:0/899241789 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cb7843f510 con 0x55cb78554ea0 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85d5ffb640 10 client.15111 handle_command_reply: tid=0 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 client.15111 shutdown 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 -- 172.21.15.49:0/899241789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 msgr2=0x55cb784e0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784e0cf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f85e80324c0 tx=0x7f85e802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 -- 172.21.15.49:0/899241789 shutdown_connections 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55cb78554ea0 0x55cb78553bb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cb784d9730 0x55cb784e4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cb784d45c0 0x55cb784e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85d5ffb640 0 client.15111 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 --2- 172.21.15.49:0/899241789 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cb784d3bd0 0x55cb784e0cf0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 -- 172.21.15.49:0/899241789 >> 172.21.15.49:0/899241789 conn(0x55cb784cf140 msgr2=0x55cb784d2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.011+0000 7f85f75da740 1 -- 172.21.15.49:0/899241789 shutdown_connections 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f75da740 1 -- 172.21.15.49:0/899241789 wait complete. 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f75da740 20 client.15111 trim_cache size 0 max 16384 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 1, 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24499, 2024-03-18T00:49:33.018 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 85.036901010999998, 2024-03-18T00:49:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:33.019 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 101.523656293 2024-03-18T00:49:33.019 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:33.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 -- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f85d80a2910 msgr2=0x7f85d80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f85d80a2910 0x7f85d80a4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f85f0199080 tx=0x7f85e4009f20 comp rx=0 tx=0).stop 2024-03-18T00:49:33.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 -- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f0102290 msgr2=0x7f85f019f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f0102290 0x7f85f019f120 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f85e000e970 tx=0x7f85e000ed70 comp rx=0 tx=0).stop 2024-03-18T00:49:33.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 -- 172.21.15.49:0/2769725200 shutdown_connections 2024-03-18T00:49:33.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f85d80a2910 0x7f85d80a4dd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f85f01018d0 0x7f85f01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f85f0102290 0x7f85f019f120 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 --2- 172.21.15.49:0/2769725200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f85f01006e0 0x7f85f0197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 -- 172.21.15.49:0/2769725200 >> 172.21.15.49:0/2769725200 conn(0x7f85f00fbc70 msgr2=0x7f85f00fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:33.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 -- 172.21.15.49:0/2769725200 shutdown_connections 2024-03-18T00:49:33.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.012+0000 7f85f4910640 1 -- 172.21.15.49:0/2769725200 wait complete. 2024-03-18T00:49:33.023 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T00:49:33.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.132+0000 7fc6da27e640 1 -- 172.21.15.49:0/414372917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d4102340 msgr2=0x7fc6d4104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.132+0000 7fc6da27e640 1 --2- 172.21.15.49:0/414372917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d4102340 0x7fc6d4104730 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009900 tx=0x7fc6c402bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:33.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.132+0000 7fc6da27e640 1 -- 172.21.15.49:0/414372917 shutdown_connections 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.132+0000 7fc6da27e640 1 --2- 172.21.15.49:0/414372917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6d4104c70 0x7fc6d4107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.132+0000 7fc6da27e640 1 --2- 172.21.15.49:0/414372917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6d41075a0 0x7fc6d4109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.132+0000 7fc6da27e640 1 --2- 172.21.15.49:0/414372917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d4102340 0x7fc6d4104730 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.132+0000 7fc6da27e640 1 -- 172.21.15.49:0/414372917 >> 172.21.15.49:0/414372917 conn(0x7fc6d40fbc90 msgr2=0x7fc6d40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.132+0000 7fc6da27e640 1 -- 172.21.15.49:0/414372917 shutdown_connections 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.132+0000 7fc6da27e640 1 -- 172.21.15.49:0/414372917 wait complete. 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6da27e640 1 Processor -- start 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6da27e640 1 -- start start 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6da27e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6d4102340 0x7fc6d4197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6da27e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6d4104c70 0x7fc6d41980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6da27e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d41075a0 0x7fc6d419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6da27e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410d4b0 con 0x7fc6d41075a0 2024-03-18T00:49:33.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6da27e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410d330 con 0x7fc6d4102340 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6da27e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410d630 con 0x7fc6d4104c70 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6d881b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d41075a0 0x7fc6d419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6d881b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d41075a0 0x7fc6d419f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53142/0 (socket says 172.21.15.49:53142) 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.133+0000 7fc6d881b640 1 -- 172.21.15.49:0/1890228531 learned_addr learned my addr 172.21.15.49:0/1890228531 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6d4102340 0x7fc6d4197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6d4104c70 0x7fc6d41980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d881b640 1 -- 172.21.15.49:0/1890228531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6d4104c70 msgr2=0x7fc6d41980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d881b640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6d4104c70 0x7fc6d41980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d881b640 1 -- 172.21.15.49:0/1890228531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6d4102340 msgr2=0x7fc6d4197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d881b640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6d4102340 0x7fc6d4197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d881b640 1 -- 172.21.15.49:0/1890228531 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d419f780 con 0x7fc6d41075a0 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6d4102340 0x7fc6d4197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d881b640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d41075a0 0x7fc6d419f110 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00eb10 tx=0x7fc6cc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:33.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d17fa640 1 -- 172.21.15.49:0/1890228531 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc004b20 con 0x7fc6d41075a0 2024-03-18T00:49:33.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d17fa640 1 -- 172.21.15.49:0/1890228531 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc004cc0 con 0x7fc6d41075a0 2024-03-18T00:49:33.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6da27e640 1 -- 172.21.15.49:0/1890228531 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d419fa70 con 0x7fc6d41075a0 2024-03-18T00:49:33.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6d17fa640 1 -- 172.21.15.49:0/1890228531 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc010760 con 0x7fc6d41075a0 2024-03-18T00:49:33.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.134+0000 7fc6da27e640 1 -- 172.21.15.49:0/1890228531 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d419ffc0 con 0x7fc6d41075a0 2024-03-18T00:49:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.136+0000 7fc6d17fa640 1 -- 172.21.15.49:0/1890228531 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc6cc005020 con 0x7fc6d41075a0 2024-03-18T00:49:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6dce62740 1 Processor -- start 2024-03-18T00:49:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6dce62740 1 -- start start 2024-03-18T00:49:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d17fa640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6bc0a2890 0x7fc6bc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afacbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 0x55e9afad10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afad1730 0x55e9afad3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6dce62740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e9afa2fc80 con 0x55e9afad1730 2024-03-18T00:49:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6dce62740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e9af9d8aa0 con 0x55e9afacc5c0 2024-03-18T00:49:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6dce62740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e9afa2bd20 con 0x55e9afacbbd0 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6bc0a2890 0x7fc6bc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d881b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afad1730 0x55e9afad3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 0x55e9afad10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afacbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d881b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afad1730 0x55e9afad3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53146/0 (socket says 172.21.15.49:53146) 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d881b640 1 -- 172.21.15.49:0/2494514580 learned_addr learned my addr 172.21.15.49:0/2494514580 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d17fa640 1 -- 172.21.15.49:0/1890228531 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc6cc0cf2f0 con 0x7fc6d41075a0 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d881b640 1 -- 172.21.15.49:0/2494514580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 msgr2=0x55e9afacbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d881b640 1 --2- 172.21.15.49:0/2494514580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afacbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d881b640 1 -- 172.21.15.49:0/2494514580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 msgr2=0x55e9afad10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d881b640 1 --2- 172.21.15.49:0/2494514580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 0x55e9afad10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d881b640 1 -- 172.21.15.49:0/2494514580 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9af9faff0 con 0x55e9afad1730 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/2494514580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afacbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.137+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6bc0a2890 0x7fc6bc0a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4199070 tx=0x7fc6c8009b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6d881b640 1 --2- 172.21.15.49:0/2494514580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afad1730 0x55e9afad3bc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0049d0 tx=0x7fc6cc00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a009ce254755370b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6ba7fc640 1 -- 172.21.15.49:0/2494514580 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc09b650 con 0x55e9afad1730 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6ba7fc640 1 -- 172.21.15.49:0/2494514580 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc09c210 con 0x55e9afad1730 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6ba7fc640 1 -- 172.21.15.49:0/2494514580 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc09b470 con 0x55e9afad1730 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6dce62740 1 -- 172.21.15.49:0/2494514580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afad1730 msgr2=0x55e9afad3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6dce62740 1 --2- 172.21.15.49:0/2494514580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afad1730 0x55e9afad3bc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc0049d0 tx=0x7fc6cc00cf70 comp rx=0 tx=0).stop 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6dce62740 1 -- 172.21.15.49:0/2494514580 shutdown_connections 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6dce62740 1 --2- 172.21.15.49:0/2494514580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 0x55e9afad10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6dce62740 1 --2- 172.21.15.49:0/2494514580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afad1730 0x55e9afad3bc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6dce62740 1 --2- 172.21.15.49:0/2494514580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afacbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6dce62740 1 -- 172.21.15.49:0/2494514580 >> 172.21.15.49:0/2494514580 conn(0x55e9afac7140 msgr2=0x55e9afac7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6dce62740 1 -- 172.21.15.49:0/2494514580 shutdown_connections 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.138+0000 7fc6dce62740 1 -- 172.21.15.49:0/2494514580 wait complete. 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6dce62740 1 -- start start 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6dce62740 1 Processor -- start 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afad5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacc5c0 0x55e9afad6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6dce62740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e9afad4680 con 0x55e9afacbbd0 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6dce62740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e9af9ebef0 con 0x55e9afad1730 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6dce62740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e9afad4800 con 0x55e9afacc5c0 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6dce62740 10 client.0 _ll_register_callbacks cb 0x55e9af9e6dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afad5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afad5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53152/0 (socket says 172.21.15.49:53152) 2024-03-18T00:49:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6d3fff640 1 -- 172.21.15.49:0/1351732391 learned_addr learned my addr 172.21.15.49:0/1351732391 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6d881b640 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.139+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacc5c0 0x55e9afad6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6d3fff640 1 -- 172.21.15.49:0/1351732391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacc5c0 msgr2=0x55e9afad6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacc5c0 0x55e9afad6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6d3fff640 1 -- 172.21.15.49:0/1351732391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 msgr2=0x55e9afadd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6d3fff640 1 -- 172.21.15.49:0/1351732391 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9af9f8a30 con 0x55e9afacbbd0 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6d881b640 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacc5c0 0x55e9afad6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afad5cd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4036040 tx=0x7fc6c4005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c4a6d14438d4b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6b37fe640 1 -- 172.21.15.49:0/1351732391 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4005d50 con 0x55e9afacbbd0 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6b37fe640 1 -- 172.21.15.49:0/1351732391 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc6c40046c0 con 0x55e9afacbbd0 2024-03-18T00:49:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6dce62740 1 -- 172.21.15.49:0/1351732391 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6cc09c210 con 0x55e9afacbbd0 2024-03-18T00:49:33.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6b37fe640 1 -- 172.21.15.49:0/1351732391 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc6c40048c0 con 0x55e9afacbbd0 2024-03-18T00:49:33.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6dce62740 10 client.15123 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:33.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6dce62740 1 -- 172.21.15.49:0/1351732391 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e9af7442d0 con 0x55e9afacbbd0 2024-03-18T00:49:33.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.140+0000 7fc6b37fe640 1 -- 172.21.15.49:0/1351732391 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6cc09c210 con 0x55e9afacbbd0 2024-03-18T00:49:33.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6b37fe640 1 -- 172.21.15.49:0/1351732391 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fc6c4003ad0 con 0x55e9afacbbd0 2024-03-18T00:49:33.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6dce62740 10 client.15123 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:33.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6dce62740 10 client.15123 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T00:49:33.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6dce62740 20 client.15123 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6dce62740 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e9afae27c0 0x55e9afae3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6dce62740 4 client.15123 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6dce62740 1 -- 172.21.15.49:0/1351732391 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9afa3d1c0 con 0x55e9afae27c0 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e9afae27c0 0x55e9afae3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e9afae27c0 0x55e9afae3c30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aca31c995eccd77 server_cookie=2f9b4e2dbbe695c8 in_seq=0 out_seq=0 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.141+0000 7fc6b37fe640 10 client.15123 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6b37fe640 1 -- 172.21.15.49:0/1351732391 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e9afa3d1c0 con 0x55e9afae27c0 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6b37fe640 10 client.15123 handle_command_reply: tid=0 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 client.15123 shutdown 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 -- 172.21.15.49:0/1351732391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 msgr2=0x55e9afad5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afad5cd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4036040 tx=0x7fc6c4005250 comp rx=0 tx=0).stop 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 -- 172.21.15.49:0/1351732391 shutdown_connections 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e9afae27c0 0x55e9afae3c30 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacc5c0 0x55e9afad6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6b37fe640 0 client.15123 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:49:33.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 --2- 172.21.15.49:0/1351732391 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afad5cd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 -- 172.21.15.49:0/1351732391 >> 172.21.15.49:0/1351732391 conn(0x55e9afac7140 msgr2=0x55e9afac6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:33.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 -- 172.21.15.49:0/1351732391 shutdown_connections 2024-03-18T00:49:33.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 1 -- 172.21.15.49:0/1351732391 wait complete. 2024-03-18T00:49:33.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.142+0000 7fc6dce62740 20 client.15123 trim_cache size 0 max 16384 2024-03-18T00:49:33.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.160+0000 7fc6dce62740 1 Processor -- start 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.160+0000 7fc6dce62740 1 -- start start 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afac5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 0x55e9afadd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afad1730 0x55e9afac6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6dce62740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e9afae1f70 con 0x55e9afacbbd0 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6dce62740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e9afae2640 con 0x55e9afacc5c0 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6dce62740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e9afae2d10 con 0x55e9afad1730 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afac5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afac5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53164/0 (socket says 172.21.15.49:53164) 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d3fff640 1 -- 172.21.15.49:0/669600418 learned_addr learned my addr 172.21.15.49:0/669600418 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/669600418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 0x55e9afadd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d881b640 1 --2- 172.21.15.49:0/669600418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afad1730 0x55e9afac6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d3fff640 1 -- 172.21.15.49:0/669600418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afad1730 msgr2=0x55e9afac6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/669600418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afad1730 0x55e9afac6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d3fff640 1 -- 172.21.15.49:0/669600418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 msgr2=0x55e9afadd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/669600418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 0x55e9afadd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d3fff640 1 -- 172.21.15.49:0/669600418 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9af80e6b0 con 0x55e9afacbbd0 2024-03-18T00:49:33.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/669600418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afac5af0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4003ef0 tx=0x7fc6c4004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4aa40ceb86423e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6b9ffb640 1 -- 172.21.15.49:0/669600418 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4000e60 con 0x55e9afacbbd0 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6b9ffb640 1 -- 172.21.15.49:0/669600418 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c4041070 con 0x55e9afacbbd0 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.161+0000 7fc6b9ffb640 1 -- 172.21.15.49:0/669600418 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4004a60 con 0x55e9afacbbd0 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 -- 172.21.15.49:0/669600418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 msgr2=0x55e9afac5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 --2- 172.21.15.49:0/669600418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afac5af0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4003ef0 tx=0x7fc6c4004500 comp rx=0 tx=0).stop 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 -- 172.21.15.49:0/669600418 shutdown_connections 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 --2- 172.21.15.49:0/669600418 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afacc5c0 0x55e9afadd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 --2- 172.21.15.49:0/669600418 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afad1730 0x55e9afac6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 --2- 172.21.15.49:0/669600418 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacbbd0 0x55e9afac5af0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 -- 172.21.15.49:0/669600418 >> 172.21.15.49:0/669600418 conn(0x55e9afac7140 msgr2=0x55e9afae1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 -- 172.21.15.49:0/669600418 shutdown_connections 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 -- 172.21.15.49:0/669600418 wait complete. 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 -- start start 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.162+0000 7fc6dce62740 1 Processor -- start 2024-03-18T00:49:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afad8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacc5c0 0x55e9afad8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6dce62740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6dce62740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e9afae3ab0 con 0x55e9afacc5c0 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6dce62740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e9afae33e0 con 0x55e9afad1730 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6dce62740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e9afae4180 con 0x55e9afacbbd0 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6dce62740 10 client.0 _ll_register_callbacks cb 0x7fc6c4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afad8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacc5c0 0x55e9afad8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afad8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44644/0 (socket says 172.21.15.49:44644) 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d3fff640 1 -- 172.21.15.49:0/61773882 learned_addr learned my addr 172.21.15.49:0/61773882 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d881b640 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d37fe640 1 -- 172.21.15.49:0/61773882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 msgr2=0x55e9afad8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afad8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d37fe640 1 -- 172.21.15.49:0/61773882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 msgr2=0x55e9afadbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d37fe640 1 -- 172.21.15.49:0/61773882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9afadc140 con 0x55e9afacc5c0 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afad8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d881b640 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6d37fe640 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacc5c0 0x55e9afad8ba0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55e9afae2c80 tx=0x7fc6c801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16cb824cbaba16a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.163+0000 7fc6b37fe640 1 -- 172.21.15.49:0/61773882 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c802c4f0 con 0x55e9afacc5c0 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6b37fe640 1 -- 172.21.15.49:0/61773882 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc6c801a070 con 0x55e9afacc5c0 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6dce62740 1 -- 172.21.15.49:0/61773882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6c4041070 con 0x55e9afacc5c0 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6b37fe640 1 -- 172.21.15.49:0/61773882 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fc6c802c690 con 0x55e9afacc5c0 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6dce62740 10 client.15129 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6dce62740 1 -- 172.21.15.49:0/61773882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e9afadd4b0 con 0x55e9afacc5c0 2024-03-18T00:49:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6b37fe640 1 -- 172.21.15.49:0/61773882 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c4041070 con 0x55e9afacc5c0 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6b37fe640 1 -- 172.21.15.49:0/61773882 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fc6c8005530 con 0x55e9afacc5c0 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6dce62740 10 client.15129 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6dce62740 10 client.15129 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6dce62740 20 client.15129 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6dce62740 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e9afb4cf20 0x55e9afb4bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6dce62740 4 client.15129 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6dce62740 1 -- 172.21.15.49:0/61773882 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "status"}) -- 0x55e9afa37510 con 0x55e9afb4cf20 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.164+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e9afb4cf20 0x55e9afb4bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6d3fff640 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e9afb4cf20 0x55e9afb4bc30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=740aaa0b001bcfc server_cookie=2a62ce5fd02b34e1 in_seq=0 out_seq=0 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6b37fe640 10 client.15129 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6b37fe640 1 -- 172.21.15.49:0/61773882 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55e9afa37510 con 0x55e9afb4cf20 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6b37fe640 10 client.15129 handle_command_reply: tid=0 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 client.15129 shutdown 2024-03-18T00:49:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 -- 172.21.15.49:0/61773882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacc5c0 msgr2=0x55e9afad8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacc5c0 0x55e9afad8ba0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55e9afae2c80 tx=0x7fc6c801d000 comp rx=0 tx=0).stop 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 -- 172.21.15.49:0/61773882 shutdown_connections 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e9afb4cf20 0x55e9afb4bc30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e9afad1730 0x55e9afadbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e9afacc5c0 0x55e9afad8ba0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 --2- 172.21.15.49:0/61773882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e9afacbbd0 0x55e9afad8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6b37fe640 0 client.15129 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 -- 172.21.15.49:0/61773882 >> 172.21.15.49:0/61773882 conn(0x55e9afac7140 msgr2=0x55e9afaca3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 -- 172.21.15.49:0/61773882 shutdown_connections 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 1 -- 172.21.15.49:0/61773882 wait complete. 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.165+0000 7fc6dce62740 20 client.15129 trim_cache size 0 max 16384 2024-03-18T00:49:33.172 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 0, 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24506, 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 86.650085875000002, 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 93.207994631000005 2024-03-18T00:49:33.173 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 -- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6bc0a2890 msgr2=0x7fc6bc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6bc0a2890 0x7fc6bc0a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4199070 tx=0x7fc6c8009b60 comp rx=0 tx=0).stop 2024-03-18T00:49:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 -- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d41075a0 msgr2=0x7fc6d419f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d41075a0 0x7fc6d419f110 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00eb10 tx=0x7fc6cc00c2d0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 -- 172.21.15.49:0/1890228531 shutdown_connections 2024-03-18T00:49:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6bc0a2890 0x7fc6bc0a4d50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc6d41075a0 0x7fc6d419f110 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc6d4104c70 0x7fc6d41980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 --2- 172.21.15.49:0/1890228531 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc6d4102340 0x7fc6d4197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 -- 172.21.15.49:0/1890228531 >> 172.21.15.49:0/1890228531 conn(0x7fc6d40fbc90 msgr2=0x7fc6d4102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 -- 172.21.15.49:0/1890228531 shutdown_connections 2024-03-18T00:49:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:33.166+0000 7fc6da27e640 1 -- 172.21.15.49:0/1890228531 wait complete. 2024-03-18T00:49:33.176 INFO:tasks.cephadm:Setup complete, yielding 2024-03-18T00:49:33.177 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T00:49:33.191 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi049.front.sepia.ceph.com 2024-03-18T00:49:33.191 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-18T00:49:33.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:33 smithi049 ceph-mon[32130]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:33.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:33 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2007635439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:33.448 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:33.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:33 smithi107 ceph-mon[36270]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:33.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:33 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2007635439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:33.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:33 smithi106 ceph-mon[35899]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:33.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:33 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2007635439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:34.525 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:34 smithi049 ceph-mon[32130]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T00:49:34.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:34 smithi107 ceph-mon[36270]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T00:49:34.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:34 smithi106 ceph-mon[35899]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-18T00:49:35.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.092+0000 7fa9bcad8700 1 -- 172.21.15.49:0/1052649071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102e10 msgr2=0x7fa9b81096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:35.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.092+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/1052649071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102e10 0x7fa9b81096c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00099c0 tx=0x7fa9b0009cd0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.092+0000 7fa9bcad8700 1 -- 172.21.15.49:0/1052649071 shutdown_connections 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.092+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/1052649071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102e10 0x7fa9b81096c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.092+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/1052649071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9b8101270 0x7fa9b8101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.092+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/1052649071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b8102470 0x7fa9b81028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.092+0000 7fa9bcad8700 1 -- 172.21.15.49:0/1052649071 >> 172.21.15.49:0/1052649071 conn(0x7fa9b80fc830 msgr2=0x7fa9b80fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.092+0000 7fa9bcad8700 1 -- 172.21.15.49:0/1052649071 shutdown_connections 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.093+0000 7fa9bcad8700 1 -- 172.21.15.49:0/1052649071 wait complete. 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.093+0000 7fa9bcad8700 1 Processor -- start 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.093+0000 7fa9bcad8700 1 -- start start 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.093+0000 7fa9bcad8700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9b8101270 0x7fa9b81973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.093+0000 7fa9bcad8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102470 0x7fa9b8197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.093+0000 7fa9bcad8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b8102e10 0x7fa9b819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.093+0000 7fa9bcad8700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b810ce00 con 0x7fa9b8102470 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.093+0000 7fa9bcad8700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b810cc80 con 0x7fa9b8101270 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.093+0000 7fa9bcad8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b810cf80 con 0x7fa9b8102e10 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b5d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102470 0x7fa9b8197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b5d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102470 0x7fa9b8197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53190/0 (socket says 172.21.15.49:53190) 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b5d9b700 1 -- 172.21.15.49:0/2281518958 learned_addr learned my addr 172.21.15.49:0/2281518958 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b659c700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9b8101270 0x7fa9b81973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b6d9d700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b8102e10 0x7fa9b819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b5d9b700 1 -- 172.21.15.49:0/2281518958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b8102e10 msgr2=0x7fa9b819e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b5d9b700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b8102e10 0x7fa9b819e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b5d9b700 1 -- 172.21.15.49:0/2281518958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9b8101270 msgr2=0x7fa9b81973c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b5d9b700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9b8101270 0x7fa9b81973c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b5d9b700 1 -- 172.21.15.49:0/2281518958 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b819f020 con 0x7fa9b8102470 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9b5d9b700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102470 0x7fa9b8197900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000eb00 tx=0x7fa9a000ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9af7fe700 1 -- 172.21.15.49:0/2281518958 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a000b9c0 con 0x7fa9b8102470 2024-03-18T00:49:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b806a9d0 con 0x7fa9b8102470 2024-03-18T00:49:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9af7fe700 1 -- 172.21.15.49:0/2281518958 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a0004d10 con 0x7fa9b8102470 2024-03-18T00:49:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9af7fe700 1 -- 172.21.15.49:0/2281518958 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0005700 con 0x7fa9b8102470 2024-03-18T00:49:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.094+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b806aec0 con 0x7fa9b8102470 2024-03-18T00:49:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.095+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b8068f90 con 0x7fa9b8102470 2024-03-18T00:49:35.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.097+0000 7fa9af7fe700 1 -- 172.21.15.49:0/2281518958 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa9a0004750 con 0x7fa9b8102470 2024-03-18T00:49:35.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.097+0000 7fa9af7fe700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9a40a2890 0x7fa9a40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:35.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.097+0000 7fa9af7fe700 1 -- 172.21.15.49:0/2281518958 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa9a00d0310 con 0x7fa9b8102470 2024-03-18T00:49:35.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.097+0000 7fa9b659c700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9a40a2890 0x7fa9a40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:35.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.098+0000 7fa9b659c700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9a40a2890 0x7fa9a40a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa9b81022d0 tx=0x7fa9a800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:35.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.101+0000 7fa9af7fe700 1 -- 172.21.15.49:0/2281518958 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa9a009d390 con 0x7fa9b8102470 2024-03-18T00:49:35.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.326+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fa9b806b120 con 0x7fa9a40a2890 2024-03-18T00:49:35.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.327+0000 7fa9af7fe700 1 -- 172.21.15.49:0/2281518958 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fa9b806b120 con 0x7fa9a40a2890 2024-03-18T00:49:35.327 INFO:teuthology.orchestra.run.smithi049.stdout:Backend: cephadm 2024-03-18T00:49:35.327 INFO:teuthology.orchestra.run.smithi049.stdout:Available: Yes 2024-03-18T00:49:35.327 INFO:teuthology.orchestra.run.smithi049.stdout:Paused: No 2024-03-18T00:49:35.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9a40a2890 msgr2=0x7fa9a40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:35.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9a40a2890 0x7fa9a40a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa9b81022d0 tx=0x7fa9a800b040 comp rx=0 tx=0).stop 2024-03-18T00:49:35.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102470 msgr2=0x7fa9b8197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:35.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102470 0x7fa9b8197900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000eb00 tx=0x7fa9a000ee10 comp rx=0 tx=0).stop 2024-03-18T00:49:35.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 shutdown_connections 2024-03-18T00:49:35.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9a40a2890 0x7fa9a40a4d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9b8102e10 0x7fa9b819e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9b8101270 0x7fa9b81973c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 --2- 172.21.15.49:0/2281518958 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9b8102470 0x7fa9b8197900 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:35.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 >> 172.21.15.49:0/2281518958 conn(0x7fa9b80fc830 msgr2=0x7fa9b8105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:35.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 shutdown_connections 2024-03-18T00:49:35.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:35.329+0000 7fa9bcad8700 1 -- 172.21.15.49:0/2281518958 wait complete. 2024-03-18T00:49:35.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:35 smithi049 ceph-mon[32130]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T00:49:35.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:35 smithi107 ceph-mon[36270]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T00:49:35.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:35 smithi106 ceph-mon[35899]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T00:49:36.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-18T00:49:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:37 smithi049 ceph-mon[32130]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:37 smithi049 ceph-mon[32130]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T00:49:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:37 smithi107 ceph-mon[36270]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:37 smithi107 ceph-mon[36270]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T00:49:37.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:37 smithi106 ceph-mon[35899]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:37.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:37 smithi106 ceph-mon[35899]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T00:49:37.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/826971811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4102d80 msgr2=0x7f2cd4105170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/826971811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4102d80 0x7f2cd4105170 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8009990 tx=0x7f2cc8009ca0 comp rx=0 tx=0).stop 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/826971811 shutdown_connections 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/826971811 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cd4107fe0 0x7f2cd410a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/826971811 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4102d80 0x7f2cd4105170 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/826971811 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd41056b0 0x7f2cd4107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/826971811 >> 172.21.15.49:0/826971811 conn(0x7f2cd40fc790 msgr2=0x7f2cd40febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/826971811 shutdown_connections 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/826971811 wait complete. 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.977+0000 7f2cdb0d0700 1 Processor -- start 2024-03-18T00:49:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cdb0d0700 1 -- start start 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cdb0d0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd4102d80 0x7f2cd41026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cdb0d0700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cd41056b0 0x7f2cd4100d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cdb0d0700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4107fe0 0x7f2cd4101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cdb0d0700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd410df00 con 0x7f2cd4107fe0 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cdb0d0700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd410dd80 con 0x7f2cd41056b0 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cdb0d0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd410e080 con 0x7f2cd4102d80 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd8e6c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd4102d80 0x7f2cd41026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd966d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4107fe0 0x7f2cd4101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd3fff700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cd41056b0 0x7f2cd4100d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd966d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4107fe0 0x7f2cd4101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41992/0 (socket says 172.21.15.49:41992) 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd966d700 1 -- 172.21.15.49:0/827224836 learned_addr learned my addr 172.21.15.49:0/827224836 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd8e6c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd4102d80 0x7f2cd41026e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:50030/0 (socket says 172.21.15.49:50030) 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd966d700 1 -- 172.21.15.49:0/827224836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd4102d80 msgr2=0x7f2cd41026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd966d700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd4102d80 0x7f2cd41026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd966d700 1 -- 172.21.15.49:0/827224836 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cd41056b0 msgr2=0x7f2cd4100d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd966d700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cd41056b0 0x7f2cd4100d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd966d700 1 -- 172.21.15.49:0/827224836 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd4101b20 con 0x7f2cd4107fe0 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd3fff700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cd41056b0 0x7f2cd4100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd966d700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4107fe0 0x7f2cd4101270 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00ef90 tx=0x7f2ccc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.978+0000 7f2cd1ffb700 1 -- 172.21.15.49:0/827224836 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc0092e0 con 0x7f2cd4107fe0 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.979+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd41a2620 con 0x7f2cd4107fe0 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.979+0000 7f2cd1ffb700 1 -- 172.21.15.49:0/827224836 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc004bb0 con 0x7f2cd4107fe0 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.979+0000 7f2cd1ffb700 1 -- 172.21.15.49:0/827224836 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc004280 con 0x7f2cd4107fe0 2024-03-18T00:49:37.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.979+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd41a2ae0 con 0x7f2cd4107fe0 2024-03-18T00:49:37.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.980+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cd4068f90 con 0x7f2cd4107fe0 2024-03-18T00:49:37.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.981+0000 7f2cd1ffb700 1 -- 172.21.15.49:0/827224836 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2ccc01b030 con 0x7f2cd4107fe0 2024-03-18T00:49:37.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.981+0000 7f2cd1ffb700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2cc40a2890 0x7f2cc40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:37.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.982+0000 7f2cd8e6c700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2cc40a2890 0x7f2cc40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:37.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.982+0000 7f2cd1ffb700 1 -- 172.21.15.49:0/827224836 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2ccc0ce600 con 0x7f2cd4107fe0 2024-03-18T00:49:37.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.982+0000 7f2cd8e6c700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2cc40a2890 0x7f2cc40a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800b2e0 tx=0x7f2cc8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:37.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:37.986+0000 7f2cd1ffb700 1 -- 172.21.15.49:0/827224836 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2ccc022020 con 0x7f2cd4107fe0 2024-03-18T00:49:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.202+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f2cd4062710 con 0x7f2cc40a2890 2024-03-18T00:49:38.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.216+0000 7f2cd1ffb700 1 -- 172.21.15.49:0/827224836 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f2cd4062710 con 0x7f2cc40a2890 2024-03-18T00:49:38.216 INFO:teuthology.orchestra.run.smithi049.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T00:49:38.216 INFO:teuthology.orchestra.run.smithi049.stdout:mds.a smithi049 running (101s) 18s ago 101s 17.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 240276f0c266 2024-03-18T00:49:38.217 INFO:teuthology.orchestra.run.smithi049.stdout:mds.b smithi106 running (98s) 31s ago 98s 25.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b e291b8ea36c8 2024-03-18T00:49:38.217 INFO:teuthology.orchestra.run.smithi049.stdout:mds.c smithi107 running (113s) 31s ago 112s 20.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b b6e77d53936f 2024-03-18T00:49:38.217 INFO:teuthology.orchestra.run.smithi049.stdout:mds.d smithi049 running (2m) 18s ago 2m 15.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 60f6373da184 2024-03-18T00:49:38.217 INFO:teuthology.orchestra.run.smithi049.stdout:mds.e smithi106 running (118s) 31s ago 118s 18.0M - 19.0.0-2162-g3d78f5ba be86dc8cb22b ebdb9d656f99 2024-03-18T00:49:38.217 INFO:teuthology.orchestra.run.smithi049.stdout:mds.f smithi107 running (115s) 31s ago 115s 17.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 72008fe08c4a 2024-03-18T00:49:38.218 INFO:teuthology.orchestra.run.smithi049.stdout:mds.g smithi049 running (104s) 18s ago 103s 16.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 9ebcde9fa222 2024-03-18T00:49:38.218 INFO:teuthology.orchestra.run.smithi049.stdout:mds.h smithi106 running (110s) 31s ago 109s 18.8M - 19.0.0-2162-g3d78f5ba be86dc8cb22b c065f4492ff7 2024-03-18T00:49:38.218 INFO:teuthology.orchestra.run.smithi049.stdout:mds.i smithi107 running (107s) 31s ago 106s 17.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b eea5ffc66de3 2024-03-18T00:49:38.218 INFO:teuthology.orchestra.run.smithi049.stdout:mds.j smithi049 running (2m) 18s ago 2m 16.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b ae2cab6fbc21 2024-03-18T00:49:38.218 INFO:teuthology.orchestra.run.smithi049.stdout:mds.k smithi106 running (2m) 31s ago 2m 18.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 6a9c5dec5093 2024-03-18T00:49:38.219 INFO:teuthology.orchestra.run.smithi049.stdout:mds.l smithi107 running (2m) 31s ago 2m 18.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 88fa35199912 2024-03-18T00:49:38.219 INFO:teuthology.orchestra.run.smithi049.stdout:mgr.x smithi049 *:9283,8765 running (9m) 18s ago 9m 513M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 440c098891eb 2024-03-18T00:49:38.219 INFO:teuthology.orchestra.run.smithi049.stdout:mgr.y smithi106 *:8443,8765 running (7m) 31s ago 7m 452M - 19.0.0-2162-g3d78f5ba be86dc8cb22b c2df8d4435b8 2024-03-18T00:49:38.219 INFO:teuthology.orchestra.run.smithi049.stdout:mgr.z smithi107 *:8443,8765 running (7m) 31s ago 7m 451M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 2d7aad30a0be 2024-03-18T00:49:38.219 INFO:teuthology.orchestra.run.smithi049.stdout:mon.a smithi049 running (9m) 18s ago 9m 50.5M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b fdaeca8c5935 2024-03-18T00:49:38.219 INFO:teuthology.orchestra.run.smithi049.stdout:mon.b smithi106 running (7m) 31s ago 7m 39.6M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b d10ddbeaacf3 2024-03-18T00:49:38.220 INFO:teuthology.orchestra.run.smithi049.stdout:mon.c smithi107 running (7m) 31s ago 7m 41.6M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 92247214c5cb 2024-03-18T00:49:38.220 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 smithi049 running (7m) 18s ago 7m 56.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b f4ecaa5a576c 2024-03-18T00:49:38.220 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 smithi106 running (6m) 31s ago 6m 55.6M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 8c41a7ba8306 2024-03-18T00:49:38.220 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 smithi107 running (6m) 31s ago 6m 46.6M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 1930f3f0c0e9 2024-03-18T00:49:38.220 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 smithi049 running (5m) 18s ago 5m 41.7M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 6b0a81a592fa 2024-03-18T00:49:38.220 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 smithi106 running (5m) 31s ago 5m 44.7M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 352b3ce8d877 2024-03-18T00:49:38.221 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 smithi107 running (5m) 31s ago 5m 46.3M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b fd7d36ad148c 2024-03-18T00:49:38.221 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6 smithi049 running (4m) 18s ago 4m 42.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 25ca0569e228 2024-03-18T00:49:38.221 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 smithi106 running (4m) 31s ago 4m 54.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 42dad86d6f94 2024-03-18T00:49:38.221 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8 smithi107 running (3m) 31s ago 3m 53.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b e2ebf7f994ab 2024-03-18T00:49:38.221 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 smithi049 running (3m) 18s ago 3m 44.7M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b d8c9d417f635 2024-03-18T00:49:38.221 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10 smithi106 running (2m) 31s ago 2m 41.9M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 09a4e019cad4 2024-03-18T00:49:38.222 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11 smithi107 running (2m) 31s ago 2m 44.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 05804a5b1443 2024-03-18T00:49:38.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2cc40a2890 msgr2=0x7f2cc40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:38.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2cc40a2890 0x7f2cc40a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800b2e0 tx=0x7f2cc8005fd0 comp rx=0 tx=0).stop 2024-03-18T00:49:38.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4107fe0 msgr2=0x7f2cd4101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:38.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4107fe0 0x7f2cd4101270 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00ef90 tx=0x7f2ccc00c3a0 comp rx=0 tx=0).stop 2024-03-18T00:49:38.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 shutdown_connections 2024-03-18T00:49:38.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2cc40a2890 0x7f2cc40a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:38.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2cd4102d80 0x7f2cd41026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:38.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2cd4107fe0 0x7f2cd4101270 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:38.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 --2- 172.21.15.49:0/827224836 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2cd41056b0 0x7f2cd4100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:38.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 >> 172.21.15.49:0/827224836 conn(0x7f2cd40fc790 msgr2=0x7f2cd41037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:38.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 shutdown_connections 2024-03-18T00:49:38.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:38.218+0000 7f2cdb0d0700 1 -- 172.21.15.49:0/827224836 wait complete. 2024-03-18T00:49:39.011 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-18T00:49:39.271 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:39.541 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:39 smithi049 ceph-mon[32130]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:49:39.541 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:39 smithi049 ceph-mon[32130]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:39 smithi107 ceph-mon[36270]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:49:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:39 smithi107 ceph-mon[36270]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:39.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:39 smithi106 ceph-mon[35899]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:49:39.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:39 smithi106 ceph-mon[35899]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:41.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.074+0000 7f1bdcd12700 1 -- 172.21.15.49:0/1411501761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd8100610 msgr2=0x7f1bd8100a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:41.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.074+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/1411501761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd8100610 0x7f1bd8100a30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8009990 tx=0x7f1bc8009ca0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.074+0000 7f1bdcd12700 1 -- 172.21.15.49:0/1411501761 shutdown_connections 2024-03-18T00:49:41.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.074+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/1411501761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd8103f90 0x7f1bd810a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.074+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/1411501761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd8100610 0x7f1bd8100a30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.074+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/1411501761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd81035f0 0x7f1bd8103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.074+0000 7f1bdcd12700 1 -- 172.21.15.49:0/1411501761 >> 172.21.15.49:0/1411501761 conn(0x7f1bd80fbf70 msgr2=0x7f1bd80fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.074+0000 7f1bdcd12700 1 -- 172.21.15.49:0/1411501761 shutdown_connections 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.074+0000 7f1bdcd12700 1 -- 172.21.15.49:0/1411501761 wait complete. 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bdcd12700 1 Processor -- start 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bdcd12700 1 -- start start 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bdcd12700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd8100610 0x7f1bd819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bdcd12700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd81035f0 0x7f1bd819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bdcd12700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd8103f90 0x7f1bd81a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bdcd12700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd810df10 con 0x7f1bd81035f0 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bdcd12700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd810dd90 con 0x7f1bd8100610 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bdcd12700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd810e090 con 0x7f1bd8103f90 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd5d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd81035f0 0x7f1bd819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:41.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd5d9b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd81035f0 0x7f1bd819ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42010/0 (socket says 172.21.15.49:42010) 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd5d9b700 1 -- 172.21.15.49:0/2359684866 learned_addr learned my addr 172.21.15.49:0/2359684866 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd6d9d700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd8103f90 0x7f1bd81a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd659c700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd8100610 0x7f1bd819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd5d9b700 1 -- 172.21.15.49:0/2359684866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd8103f90 msgr2=0x7f1bd81a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd5d9b700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd8103f90 0x7f1bd81a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd5d9b700 1 -- 172.21.15.49:0/2359684866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd8100610 msgr2=0x7f1bd819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd5d9b700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd8100610 0x7f1bd819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.075+0000 7f1bd5d9b700 1 -- 172.21.15.49:0/2359684866 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd81a2390 con 0x7f1bd81035f0 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.076+0000 7f1bd5d9b700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd81035f0 0x7f1bd819ac70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00d980 tx=0x7f1bcc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.076+0000 7f1bc77fe700 1 -- 172.21.15.49:0/2359684866 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc016070 con 0x7f1bd81035f0 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.076+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd81a2680 con 0x7f1bd81035f0 2024-03-18T00:49:41.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.076+0000 7f1bc77fe700 1 -- 172.21.15.49:0/2359684866 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bcc009db0 con 0x7f1bd81035f0 2024-03-18T00:49:41.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.076+0000 7f1bc77fe700 1 -- 172.21.15.49:0/2359684866 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc00f3c0 con 0x7f1bd81035f0 2024-03-18T00:49:41.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.076+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd81a2c10 con 0x7f1bd81035f0 2024-03-18T00:49:41.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.077+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd8068f90 con 0x7f1bd81035f0 2024-03-18T00:49:41.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.078+0000 7f1bc77fe700 1 -- 172.21.15.49:0/2359684866 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1bcc019070 con 0x7f1bd81035f0 2024-03-18T00:49:41.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.079+0000 7f1bc77fe700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bc00a2890 0x7f1bc00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:41.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.079+0000 7f1bd659c700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bc00a2890 0x7f1bc00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:41.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.079+0000 7f1bc77fe700 1 -- 172.21.15.49:0/2359684866 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f1bcc0ce530 con 0x7f1bd81035f0 2024-03-18T00:49:41.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.079+0000 7f1bd659c700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bc00a2890 0x7f1bc00a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8003890 tx=0x7f1bc8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:41.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.082+0000 7f1bc77fe700 1 -- 172.21.15.49:0/2359684866 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1bcc09b5b0 con 0x7f1bd81035f0 2024-03-18T00:49:41.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.298+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f1bd8102b00 con 0x7f1bc00a2890 2024-03-18T00:49:41.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.304+0000 7f1bc77fe700 1 -- 172.21.15.49:0/2359684866 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f1bd8102b00 con 0x7f1bc00a2890 2024-03-18T00:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-18T00:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout:mds.all 12/12 34s ago 2m smithi049=a;smithi049=d;smithi049=g;smithi049=j;smithi106=b;smithi106=e;smithi106=h;smithi106=k;smithi107=c;smithi107=f;smithi107=i;smithi107=l;count:12 2024-03-18T00:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout:mgr 3/3 34s ago 7m smithi049=x;smithi106=y;smithi107=z;count:3 2024-03-18T00:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout:mon 3/3 34s ago 7m smithi049:172.21.15.49=a;smithi106:172.21.15.106=b;smithi107:172.21.15.107=c;count:3 2024-03-18T00:49:41.304 INFO:teuthology.orchestra.run.smithi049.stdout:osd 12 34s ago - 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bc00a2890 msgr2=0x7f1bc00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bc00a2890 0x7f1bc00a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1bc8003890 tx=0x7f1bc8005fd0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd81035f0 msgr2=0x7f1bd819ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd81035f0 0x7f1bd819ac70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00d980 tx=0x7f1bcc00dc90 comp rx=0 tx=0).stop 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 shutdown_connections 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bc00a2890 0x7f1bc00a4d50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bd8103f90 0x7f1bd81a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bd8100610 0x7f1bd819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 --2- 172.21.15.49:0/2359684866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bd81035f0 0x7f1bd819ac70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:41.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 >> 172.21.15.49:0/2359684866 conn(0x7f1bd80fbf70 msgr2=0x7f1bd81013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:41.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 shutdown_connections 2024-03-18T00:49:41.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:41.306+0000 7f1bdcd12700 1 -- 172.21.15.49:0/2359684866 wait complete. 2024-03-18T00:49:41.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:41 smithi049 ceph-mon[32130]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T00:49:41.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:41 smithi107 ceph-mon[36270]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T00:49:41.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:41 smithi106 ceph-mon[35899]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-18T00:49:42.021 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-18T00:49:42.279 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:43 smithi049 ceph-mon[32130]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:43.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:43 smithi049 ceph-mon[32130]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T00:49:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:43 smithi107 ceph-mon[36270]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:43 smithi107 ceph-mon[36270]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T00:49:43.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:43 smithi106 ceph-mon[35899]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:43.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:43 smithi106 ceph-mon[35899]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-18T00:49:44.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.127+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/3204636917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 msgr2=0x7fb0c0107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:44.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.127+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/3204636917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 0x7fb0c0107b50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009990 tx=0x7fb0b0009ca0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.128+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/3204636917 shutdown_connections 2024-03-18T00:49:44.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.128+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/3204636917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0102e30 0x7fb0c0105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.128+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/3204636917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0108090 0x7fb0c010a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.128+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/3204636917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 0x7fb0c0107b50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.128+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/3204636917 >> 172.21.15.49:0/3204636917 conn(0x7fb0c00fc7a0 msgr2=0x7fb0c00febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.128+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/3204636917 shutdown_connections 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.128+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/3204636917 wait complete. 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.128+0000 7fb0c7a5a700 1 Processor -- start 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.128+0000 7fb0c7a5a700 1 -- start start 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c7a5a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0102e30 0x7fb0c01985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c7a5a700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 0x7fb0c0198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c7a5a700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0108090 0x7fb0c019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c7a5a700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010df60 con 0x7fb0c0105760 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c7a5a700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010dde0 con 0x7fb0c0108090 2024-03-18T00:49:44.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c7a5a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c010e0e0 con 0x7fb0c0102e30 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c4ff5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 0x7fb0c0198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c57f6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0102e30 0x7fb0c01985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c4ff5700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 0x7fb0c0198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42038/0 (socket says 172.21.15.49:42038) 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c4ff5700 1 -- 172.21.15.49:0/1881677606 learned_addr learned my addr 172.21.15.49:0/1881677606 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c5ff7700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0108090 0x7fb0c019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c4ff5700 1 -- 172.21.15.49:0/1881677606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0102e30 msgr2=0x7fb0c01985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c4ff5700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0102e30 0x7fb0c01985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c4ff5700 1 -- 172.21.15.49:0/1881677606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0108090 msgr2=0x7fb0c019fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c4ff5700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0108090 0x7fb0c019fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c4ff5700 1 -- 172.21.15.49:0/1881677606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c01a0220 con 0x7fb0c0105760 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c57f6700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0102e30 0x7fb0c01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c4ff5700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 0x7fb0c0198b00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0005f50 tx=0x7fb0b000bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0b67fc700 1 -- 172.21.15.49:0/1881677606 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0026070 con 0x7fb0c0105760 2024-03-18T00:49:44.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c01a04b0 con 0x7fb0c0105760 2024-03-18T00:49:44.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0b67fc700 1 -- 172.21.15.49:0/1881677606 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b0003d70 con 0x7fb0c0105760 2024-03-18T00:49:44.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.129+0000 7fb0b67fc700 1 -- 172.21.15.49:0/1881677606 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0021410 con 0x7fb0c0105760 2024-03-18T00:49:44.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.130+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c01a0960 con 0x7fb0c0105760 2024-03-18T00:49:44.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.130+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0c0068f90 con 0x7fb0c0105760 2024-03-18T00:49:44.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.132+0000 7fb0b67fc700 1 -- 172.21.15.49:0/1881677606 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb0b0029070 con 0x7fb0c0105760 2024-03-18T00:49:44.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.132+0000 7fb0b67fc700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb0ac0a2890 0x7fb0ac0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:44.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.132+0000 7fb0c57f6700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb0ac0a2890 0x7fb0ac0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:44.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.132+0000 7fb0b67fc700 1 -- 172.21.15.49:0/1881677606 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb0b00cfa70 con 0x7fb0c0105760 2024-03-18T00:49:44.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.133+0000 7fb0c57f6700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb0ac0a2890 0x7fb0ac0a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc011490 tx=0x7fb0bc009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:44.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.136+0000 7fb0b67fc700 1 -- 172.21.15.49:0/1881677606 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb0b0013070 con 0x7fb0c0105760 2024-03-18T00:49:44.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.349+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fb0c0062710 con 0x7fb0ac0a2890 2024-03-18T00:49:44.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.350+0000 7fb0b67fc700 1 -- 172.21.15.49:0/1881677606 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fb0c0062710 con 0x7fb0ac0a2890 2024-03-18T00:49:44.350 INFO:teuthology.orchestra.run.smithi049.stdout:HOST ADDR LABELS STATUS 2024-03-18T00:49:44.350 INFO:teuthology.orchestra.run.smithi049.stdout:smithi049 172.21.15.49 2024-03-18T00:49:44.350 INFO:teuthology.orchestra.run.smithi049.stdout:smithi106 172.21.15.106 2024-03-18T00:49:44.350 INFO:teuthology.orchestra.run.smithi049.stdout:smithi107 172.21.15.107 2024-03-18T00:49:44.350 INFO:teuthology.orchestra.run.smithi049.stdout:3 hosts in cluster 2024-03-18T00:49:44.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.352+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb0ac0a2890 msgr2=0x7fb0ac0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:44.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.352+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb0ac0a2890 0x7fb0ac0a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc011490 tx=0x7fb0bc009260 comp rx=0 tx=0).stop 2024-03-18T00:49:44.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.352+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 msgr2=0x7fb0c0198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:44.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.352+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 0x7fb0c0198b00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0005f50 tx=0x7fb0b000bbb0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.353+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 shutdown_connections 2024-03-18T00:49:44.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.353+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb0ac0a2890 0x7fb0ac0a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.353+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb0c0102e30 0x7fb0c01985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.353+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb0c0108090 0x7fb0c019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.353+0000 7fb0c7a5a700 1 --2- 172.21.15.49:0/1881677606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb0c0105760 0x7fb0c0198b00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:44.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.353+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 >> 172.21.15.49:0/1881677606 conn(0x7fb0c00fc7a0 msgr2=0x7fb0c0103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:44.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.353+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 shutdown_connections 2024-03-18T00:49:44.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:44.353+0000 7fb0c7a5a700 1 -- 172.21.15.49:0/1881677606 wait complete. 2024-03-18T00:49:45.180 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-18T00:49:45.437 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:45.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:45 smithi049 ceph-mon[32130]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:49:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:45 smithi107 ceph-mon[36270]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:49:45.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:45 smithi106 ceph-mon[35899]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T00:49:46.557 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:46 smithi049 ceph-mon[32130]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:46.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:46 smithi107 ceph-mon[36270]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:46.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:46 smithi106 ceph-mon[35899]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:47.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.119+0000 7fcc1387f700 1 -- 172.21.15.49:0/3502896624 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 msgr2=0x7fcc0c102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:47.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.119+0000 7fcc1387f700 1 --2- 172.21.15.49:0/3502896624 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 0x7fcc0c102720 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0099c0 tx=0x7fcbfc009cd0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.119+0000 7fcc1387f700 1 -- 172.21.15.49:0/3502896624 shutdown_connections 2024-03-18T00:49:47.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.119+0000 7fcc1387f700 1 --2- 172.21.15.49:0/3502896624 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc0c103ea0 0x7fcc0c10a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.119+0000 7fcc1387f700 1 --2- 172.21.15.49:0/3502896624 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 0x7fcc0c102720 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.119+0000 7fcc1387f700 1 --2- 172.21.15.49:0/3502896624 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc0c103500 0x7fcc0c103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.119+0000 7fcc1387f700 1 -- 172.21.15.49:0/3502896624 >> 172.21.15.49:0/3502896624 conn(0x7fcc0c0fd8e0 msgr2=0x7fcc0c0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.119+0000 7fcc1387f700 1 -- 172.21.15.49:0/3502896624 shutdown_connections 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.119+0000 7fcc1387f700 1 -- 172.21.15.49:0/3502896624 wait complete. 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1387f700 1 Processor -- start 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1387f700 1 -- start start 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1387f700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 0x7fcc0c198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1387f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc0c103500 0x7fcc0c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1387f700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc0c103ea0 0x7fcc0c19fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1387f700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c10dee0 con 0x7fcc0c102300 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1387f700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c10dd60 con 0x7fcc0c103ea0 2024-03-18T00:49:47.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1387f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c10e060 con 0x7fcc0c103500 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1161b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 0x7fcc0c198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1161b700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 0x7fcc0c198550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42056/0 (socket says 172.21.15.49:42056) 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1161b700 1 -- 172.21.15.49:0/270905710 learned_addr learned my addr 172.21.15.49:0/270905710 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc10e1a700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc0c103500 0x7fcc0c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc11e1c700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc0c103ea0 0x7fcc0c19fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc10e1a700 1 -- 172.21.15.49:0/270905710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc0c103ea0 msgr2=0x7fcc0c19fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc10e1a700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc0c103ea0 0x7fcc0c19fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc10e1a700 1 -- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 msgr2=0x7fcc0c198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc10e1a700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 0x7fcc0c198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc10e1a700 1 -- 172.21.15.49:0/270905710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c1a01b0 con 0x7fcc0c103500 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.120+0000 7fcc1161b700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 0x7fcc0c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:47.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.121+0000 7fcc10e1a700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc0c103500 0x7fcc0c198a90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800ea20 tx=0x7fcc0800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.121+0000 7fcc027fc700 1 -- 172.21.15.49:0/270905710 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0800cbd0 con 0x7fcc0c103500 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.121+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0c1a04a0 con 0x7fcc0c103500 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.121+0000 7fcc027fc700 1 -- 172.21.15.49:0/270905710 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc08004500 con 0x7fcc0c103500 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.121+0000 7fcc027fc700 1 -- 172.21.15.49:0/270905710 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc08005020 con 0x7fcc0c103500 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.121+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0c1a09e0 con 0x7fcc0c103500 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.122+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc0c068f90 con 0x7fcc0c103500 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.123+0000 7fcc027fc700 1 -- 172.21.15.49:0/270905710 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcc080052a0 con 0x7fcc0c103500 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.123+0000 7fcc027fc700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbf80a2890 0x7fcbf80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.124+0000 7fcc1161b700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbf80a2890 0x7fcbf80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:47.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.124+0000 7fcc027fc700 1 -- 172.21.15.49:0/270905710 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcc080ce970 con 0x7fcc0c103500 2024-03-18T00:49:47.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.124+0000 7fcc1161b700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbf80a2890 0x7fcbf80a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc000c00 tx=0x7fcbfc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:47.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.127+0000 7fcc027fc700 1 -- 172.21.15.49:0/270905710 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcc0809b990 con 0x7fcc0c103500 2024-03-18T00:49:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.342+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fcc0c062710 con 0x7fcbf80a2890 2024-03-18T00:49:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.344+0000 7fcc027fc700 1 -- 172.21.15.49:0/270905710 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fcc0c062710 con 0x7fcbf80a2890 2024-03-18T00:49:47.344 INFO:teuthology.orchestra.run.smithi049.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-18T00:49:47.344 INFO:teuthology.orchestra.run.smithi049.stdout:smithi049 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747500D2480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-18T00:49:47.344 INFO:teuthology.orchestra.run.smithi049.stdout:smithi106 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204013R400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-18T00:49:47.344 INFO:teuthology.orchestra.run.smithi049.stdout:smithi107 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NK400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-18T00:49:47.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbf80a2890 msgr2=0x7fcbf80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:47.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbf80a2890 0x7fcbf80a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc000c00 tx=0x7fcbfc005fb0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc0c103500 msgr2=0x7fcc0c198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc0c103500 0x7fcc0c198a90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800ea20 tx=0x7fcc0800ed30 comp rx=0 tx=0).stop 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 shutdown_connections 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc0c103ea0 0x7fcc0c19fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc0c102300 0x7fcc0c198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbf80a2890 0x7fcbf80a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 --2- 172.21.15.49:0/270905710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc0c103500 0x7fcc0c198a90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 >> 172.21.15.49:0/270905710 conn(0x7fcc0c0fd8e0 msgr2=0x7fcc0c106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 shutdown_connections 2024-03-18T00:49:47.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:47.346+0000 7fcc1387f700 1 -- 172.21.15.49:0/270905710 wait complete. 2024-03-18T00:49:47.596 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:47 smithi106 ceph-mon[35899]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-18T00:49:47.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:47 smithi049 ceph-mon[32130]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-18T00:49:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:47 smithi107 ceph-mon[36270]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-18T00:49:48.012 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T00:49:48.022 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi049.front.sepia.ceph.com 2024-03-18T00:49:48.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T00:49:48.277 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:49.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:49 smithi049 ceph-mon[32130]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:49.550 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:49 smithi049 ceph-mon[32130]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-18T00:49:49.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:49 smithi107 ceph-mon[36270]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:49.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:49 smithi107 ceph-mon[36270]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-18T00:49:49.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:49 smithi106 ceph-mon[35899]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T00:49:49.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:49 smithi106 ceph-mon[35899]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-18T00:49:50.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.084+0000 7fca48853700 1 -- 172.21.15.49:0/905275867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca44103900 msgr2=0x7fca44103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.084+0000 7fca48853700 1 --2- 172.21.15.49:0/905275867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca44103900 0x7fca44103d20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fca34009990 tx=0x7fca34009ca0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.084+0000 7fca48853700 1 -- 172.21.15.49:0/905275867 shutdown_connections 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.084+0000 7fca48853700 1 --2- 172.21.15.49:0/905275867 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca44100d20 0x7fca4410a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.084+0000 7fca48853700 1 --2- 172.21.15.49:0/905275867 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca44103900 0x7fca44103d20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.084+0000 7fca48853700 1 --2- 172.21.15.49:0/905275867 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca44100380 0x7fca441007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.084+0000 7fca48853700 1 -- 172.21.15.49:0/905275867 >> 172.21.15.49:0/905275867 conn(0x7fca440fbf20 msgr2=0x7fca440fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.084+0000 7fca48853700 1 -- 172.21.15.49:0/905275867 shutdown_connections 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.084+0000 7fca48853700 1 -- 172.21.15.49:0/905275867 wait complete. 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca48853700 1 Processor -- start 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca48853700 1 -- start start 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca48853700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca44100380 0x7fca44198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca48853700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca44100d20 0x7fca44198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:50.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca48853700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca44103900 0x7fca4419fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca48853700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fca4410df00 con 0x7fca44103900 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca48853700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fca4410dd80 con 0x7fca44100d20 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca48853700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fca4410e080 con 0x7fca44100380 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca4259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca44100380 0x7fca44198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca42d9d700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca44103900 0x7fca4419fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca4259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca44100380 0x7fca44198540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:49556/0 (socket says 172.21.15.49:49556) 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca4259c700 1 -- 172.21.15.49:0/723817658 learned_addr learned my addr 172.21.15.49:0/723817658 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca41d9b700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca44100d20 0x7fca44198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca4259c700 1 -- 172.21.15.49:0/723817658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca44100d20 msgr2=0x7fca44198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca4259c700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca44100d20 0x7fca44198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca4259c700 1 -- 172.21.15.49:0/723817658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca44103900 msgr2=0x7fca4419fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca4259c700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca44103900 0x7fca4419fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.085+0000 7fca4259c700 1 -- 172.21.15.49:0/723817658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca441a01a0 con 0x7fca44100380 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.086+0000 7fca4259c700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca44100380 0x7fca44198540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fca3400bb50 tx=0x7fca3400bb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.086+0000 7fca337fe700 1 -- 172.21.15.49:0/723817658 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34026070 con 0x7fca44100380 2024-03-18T00:49:50.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.086+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca441a0430 con 0x7fca44100380 2024-03-18T00:49:50.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.086+0000 7fca337fe700 1 -- 172.21.15.49:0/723817658 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca3400f460 con 0x7fca44100380 2024-03-18T00:49:50.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.086+0000 7fca337fe700 1 -- 172.21.15.49:0/723817658 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34021670 con 0x7fca44100380 2024-03-18T00:49:50.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.086+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca441a0880 con 0x7fca44100380 2024-03-18T00:49:50.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.087+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca44068f90 con 0x7fca44100380 2024-03-18T00:49:50.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.088+0000 7fca337fe700 1 -- 172.21.15.49:0/723817658 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fca34029070 con 0x7fca44100380 2024-03-18T00:49:50.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.088+0000 7fca337fe700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca2c0a2890 0x7fca2c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:50.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.089+0000 7fca337fe700 1 -- 172.21.15.49:0/723817658 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fca340cf960 con 0x7fca44100380 2024-03-18T00:49:50.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.089+0000 7fca41d9b700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca2c0a2890 0x7fca2c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:50.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.089+0000 7fca41d9b700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca2c0a2890 0x7fca2c0a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fca38009c60 tx=0x7fca38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:50.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.093+0000 7fca337fe700 1 -- 172.21.15.49:0/723817658 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca3405d740 con 0x7fca44100380 2024-03-18T00:49:50.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.344+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fca44062ee0 con 0x7fca44100380 2024-03-18T00:49:50.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.344+0000 7fca337fe700 1 -- 172.21.15.49:0/723817658 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3090 (secure 0 0 0) 0x7fca34013070 con 0x7fca44100380 2024-03-18T00:49:50.344 INFO:teuthology.orchestra.run.smithi049.stdout:e24 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:legacy client fscid: 1 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:Filesystem 'cephfs' (1) 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:fs_name cephfs 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 24 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-03-18T00:48:05.881816+0000 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:modified 2024-03-18T00:48:12.008019+0000 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:tableserver 0 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:root 0 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:session_timeout 300 2024-03-18T00:49:50.345 INFO:teuthology.orchestra.run.smithi049.stdout:session_autoclose 300 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:max_file_size 1099511627776 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:max_xattr_size 65536 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:required_client_features {} 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:last_failure 0 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:last_failure_osd_epoch 0 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:max_mds 5 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:in 0,1,2,3,4 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:up {0=24506,1=24499,2=24481,3=24470,4=24463} 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:failed 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:damaged 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:stopped 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:data_pools [3] 2024-03-18T00:49:50.346 INFO:teuthology.orchestra.run.smithi049.stdout:metadata_pool 2 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:inline_data disabled 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:balancer 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:bal_rank_mask -1 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:standby_count_wanted 0 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:qdb_cluster leader: 24506 members: 24499,24506,24481,24470,24463 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.b{0:24506} state up:active seq 3 addr [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.e{0:24431} state up:standby-replay seq 1 addr [v2:172.21.15.106:6834/353176461,v1:172.21.15.106:6835/353176461] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.i{1:24499} state up:active seq 6 addr [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.l{1:24421} state up:standby-replay seq 1 addr [v2:172.21.15.107:6832/1442063907,v1:172.21.15.107:6833/1442063907] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.c{2:24481} state up:active seq 7 addr [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.k{2:24404} state up:standby-replay seq 1 addr [v2:172.21.15.106:6832/701158707,v1:172.21.15.106:6833/701158707] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.h{3:24470} state up:active seq 7 addr [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.a{3:14625} state up:standby-replay seq 1 addr [v2:172.21.15.49:6840/3327527918,v1:172.21.15.49:6841/3327527918] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.f{4:24463} state up:active seq 9 addr [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.g{4:14613} state up:standby-replay seq 1 addr [v2:172.21.15.49:6838/1964983089,v1:172.21.15.49:6839/1964983089] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.347 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:50.348 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:50.348 INFO:teuthology.orchestra.run.smithi049.stdout:Standby daemons: 2024-03-18T00:49:50.348 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:50.348 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.j{-1:14535} state up:standby seq 1 addr [v2:172.21.15.49:6834/1901706381,v1:172.21.15.49:6835/1901706381] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.348 INFO:teuthology.orchestra.run.smithi049.stdout:[mds.d{-1:14547} state up:standby seq 1 addr [v2:172.21.15.49:6836/535636420,v1:172.21.15.49:6837/535636420] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca2c0a2890 msgr2=0x7fca2c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca2c0a2890 0x7fca2c0a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fca38009c60 tx=0x7fca38009260 comp rx=0 tx=0).stop 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca44100380 msgr2=0x7fca44198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca44100380 0x7fca44198540 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fca3400bb50 tx=0x7fca3400bb80 comp rx=0 tx=0).stop 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 shutdown_connections 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca2c0a2890 0x7fca2c0a4d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca44100380 0x7fca44198540 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca44103900 0x7fca4419fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 --2- 172.21.15.49:0/723817658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca44100d20 0x7fca44198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 >> 172.21.15.49:0/723817658 conn(0x7fca440fbf20 msgr2=0x7fca441069f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 shutdown_connections 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:50.347+0000 7fca48853700 1 -- 172.21.15.49:0/723817658 wait complete. 2024-03-18T00:49:50.349 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 24 2024-03-18T00:49:51.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-18T00:49:51.285 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:51.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:51 smithi049 ceph-mon[32130]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:51.571 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:51 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/723817658' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T00:49:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:51 smithi107 ceph-mon[36270]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:51 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/723817658' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T00:49:51.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:51 smithi106 ceph-mon[35899]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:51.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:51 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/723817658' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T00:49:53.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.143+0000 7f5d901e7700 1 -- 172.21.15.49:0/382256346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d881035a0 msgr2=0x7f5d88103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.143+0000 7f5d901e7700 1 --2- 172.21.15.49:0/382256346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d881035a0 0x7f5d88103a00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009990 tx=0x7f5d78009ca0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.143+0000 7f5d901e7700 1 -- 172.21.15.49:0/382256346 shutdown_connections 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.143+0000 7f5d901e7700 1 --2- 172.21.15.49:0/382256346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d881005c0 0x7f5d881009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.143+0000 7f5d901e7700 1 --2- 172.21.15.49:0/382256346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d88103f40 0x7f5d8810a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.143+0000 7f5d901e7700 1 --2- 172.21.15.49:0/382256346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d881035a0 0x7f5d88103a00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.143+0000 7f5d901e7700 1 -- 172.21.15.49:0/382256346 >> 172.21.15.49:0/382256346 conn(0x7f5d880fbee0 msgr2=0x7f5d880fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.143+0000 7f5d901e7700 1 -- 172.21.15.49:0/382256346 shutdown_connections 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.143+0000 7f5d901e7700 1 -- 172.21.15.49:0/382256346 wait complete. 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d901e7700 1 Processor -- start 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d901e7700 1 -- start start 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d901e7700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d881005c0 0x7f5d88102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:53.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d901e7700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d881035a0 0x7f5d88101550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d901e7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d88103f40 0x7f5d88101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d901e7700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8810df30 con 0x7f5d881005c0 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d901e7700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8810ddb0 con 0x7f5d881035a0 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d901e7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5d8810e0b0 con 0x7f5d88103f40 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d8df83700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d881005c0 0x7f5d88102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d8df83700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d881005c0 0x7f5d88102f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55020/0 (socket says 172.21.15.49:55020) 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d8df83700 1 -- 172.21.15.49:0/1412092726 learned_addr learned my addr 172.21.15.49:0/1412092726 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d8d782700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d881035a0 0x7f5d88101550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.144+0000 7f5d8e784700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d88103f40 0x7f5d88101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d8d782700 1 -- 172.21.15.49:0/1412092726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d88103f40 msgr2=0x7f5d88101a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d8d782700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d88103f40 0x7f5d88101a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d8d782700 1 -- 172.21.15.49:0/1412092726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d881005c0 msgr2=0x7f5d88102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d8d782700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d881005c0 0x7f5d88102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d8d782700 1 -- 172.21.15.49:0/1412092726 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d88102340 con 0x7f5d881035a0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d8d782700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d881035a0 0x7f5d88101550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800b7b0 tx=0x7f5d7800bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d7effd700 1 -- 172.21.15.49:0/1412092726 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d78026070 con 0x7f5d881035a0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d881a2670 con 0x7f5d881035a0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d7effd700 1 -- 172.21.15.49:0/1412092726 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d7800f460 con 0x7f5d881035a0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d7effd700 1 -- 172.21.15.49:0/1412092726 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d78021670 con 0x7f5d881035a0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.145+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d881a2ac0 con 0x7f5d881035a0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.146+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d88068f90 con 0x7f5d881035a0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.147+0000 7f5d7effd700 1 -- 172.21.15.49:0/1412092726 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5d78029070 con 0x7f5d881035a0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.148+0000 7f5d7effd700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5d740a2890 0x7f5d740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.148+0000 7f5d8df83700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5d740a2890 0x7f5d740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.148+0000 7f5d7effd700 1 -- 172.21.15.49:0/1412092726 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5d780cf840 con 0x7f5d881035a0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.148+0000 7f5d8df83700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5d740a2890 0x7f5d740a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5d84005950 tx=0x7f5d84009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:53.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.153+0000 7f5d7effd700 1 -- 172.21.15.49:0/1412092726 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5d7805d740 con 0x7f5d881035a0 2024-03-18T00:49:53.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.361+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f5d8802db70 con 0x7f5d881035a0 2024-03-18T00:49:53.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.362+0000 7f5d7effd700 1 -- 172.21.15.49:0/1412092726 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) ==== 56+0+5331 (secure 0 0 0) 0x7f5d78013070 con 0x7f5d881035a0 2024-03-18T00:49:53.362 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 88 2024-03-18T00:49:53.362 INFO:teuthology.orchestra.run.smithi049.stdout:fsid ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T00:49:53.362 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-03-18T00:40:03.036875+0000 2024-03-18T00:49:53.362 INFO:teuthology.orchestra.run.smithi049.stdout:modified 2024-03-18T00:49:03.081599+0000 2024-03-18T00:49:53.362 INFO:teuthology.orchestra.run.smithi049.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-18T00:49:53.362 INFO:teuthology.orchestra.run.smithi049.stdout:crush_version 26 2024-03-18T00:49:53.362 INFO:teuthology.orchestra.run.smithi049.stdout:full_ratio 0.95 2024-03-18T00:49:53.362 INFO:teuthology.orchestra.run.smithi049.stdout:backfillfull_ratio 0.9 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:nearfull_ratio 0.85 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:require_min_compat_client luminous 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:min_compat_client luminous 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:require_osd_release squid 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:stretch_mode_enabled false 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:max_osd 12 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.0 up in weight 1 up_from 8 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6802/2159617645,v1:172.21.15.49:6803/2159617645] [v2:172.21.15.49:6804/2159617645,v1:172.21.15.49:6805/2159617645] exists,up ba2a7ba2-663e-47ea-99f4-f190dece19d2 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6800/3847617930,v1:172.21.15.106:6801/3847617930] [v2:172.21.15.106:6802/3847617930,v1:172.21.15.106:6803/3847617930] exists,up e16bd7a7-f97d-4c1c-b014-c3252101dd2a 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6800/4213760131,v1:172.21.15.107:6801/4213760131] [v2:172.21.15.107:6802/4213760131,v1:172.21.15.107:6803/4213760131] exists,up 2fc48643-7a8d-4a34-9dcc-8b5b9098abe9 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.3 up in weight 1 up_from 26 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6810/725028995,v1:172.21.15.49:6811/725028995] [v2:172.21.15.49:6812/725028995,v1:172.21.15.49:6813/725028995] exists,up 0adbb9e1-881b-4d18-8933-6ac789b3d595 2024-03-18T00:49:53.363 INFO:teuthology.orchestra.run.smithi049.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6808/2755514783,v1:172.21.15.106:6809/2755514783] [v2:172.21.15.106:6810/2755514783,v1:172.21.15.106:6811/2755514783] exists,up d95b4131-8533-4e43-b25c-c081adff6639 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.5 up in weight 1 up_from 39 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6808/391404983,v1:172.21.15.107:6809/391404983] [v2:172.21.15.107:6810/391404983,v1:172.21.15.107:6811/391404983] exists,up 41055562-3e13-4785-9ba2-de16fcd415f1 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.6 up in weight 1 up_from 47 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6818/24691072,v1:172.21.15.49:6819/24691072] [v2:172.21.15.49:6820/24691072,v1:172.21.15.49:6821/24691072] exists,up 8945dfe3-e4b2-49bc-81bd-ed8ef849eab6 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.7 up in weight 1 up_from 52 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6816/4213628543,v1:172.21.15.106:6817/4213628543] [v2:172.21.15.106:6818/4213628543,v1:172.21.15.106:6819/4213628543] exists,up dc773d75-ff12-45ce-ac11-27de406961c7 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.8 up in weight 1 up_from 60 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6816/3032692345,v1:172.21.15.107:6817/3032692345] [v2:172.21.15.107:6818/3032692345,v1:172.21.15.107:6819/3032692345] exists,up 0e583e9c-fd87-4cdf-9b71-ced791a6419e 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.9 up in weight 1 up_from 68 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.49:6826/1286831410,v1:172.21.15.49:6827/1286831410] [v2:172.21.15.49:6828/1286831410,v1:172.21.15.49:6829/1286831410] exists,up a435579c-62d2-409c-83ce-6c44d620a592 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.106:6824/3868441447,v1:172.21.15.106:6825/3868441447] [v2:172.21.15.106:6826/3868441447,v1:172.21.15.106:6827/3868441447] exists,up f13f5e88-b234-4c16-a11b-b65a9100334b 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6824/1232543995,v1:172.21.15.107:6825/1232543995] [v2:172.21.15.107:6826/1232543995,v1:172.21.15.107:6827/1232543995] exists,up 95663d7f-7dd2-4eb8-bcf1-e7f5923105d4 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 2.0 [8,4] 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 2.5 [8,11] 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 2.10 [2,4] 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 2.16 [2,6] 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 2.1b [5,11] 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 3.10 [5,9] 2024-03-18T00:49:53.364 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 3.12 [3,9] 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 3.14 [4,6] 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 3.1d [4,8] 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:pg_upmap_items 3.24 [4,9] 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/3253512442 expires 2024-03-19T00:41:01.376745+0000 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6801/1899027445 expires 2024-03-19T00:41:01.376745+0000 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/378341991 expires 2024-03-19T00:41:01.376745+0000 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6800/1899027445 expires 2024-03-19T00:41:01.376745+0000 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/1473779746 expires 2024-03-19T00:40:34.217610+0000 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/620710678 expires 2024-03-19T00:40:34.217610+0000 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6800/1672783865 expires 2024-03-19T00:40:34.217610+0000 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/3065759231 expires 2024-03-19T00:40:34.217610+0000 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:0/406635835 expires 2024-03-19T00:41:01.376745+0000 2024-03-18T00:49:53.365 INFO:teuthology.orchestra.run.smithi049.stdout:blocklist 172.21.15.49:6801/1672783865 expires 2024-03-19T00:40:34.217610+0000 2024-03-18T00:49:53.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5d740a2890 msgr2=0x7f5d740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:53.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5d740a2890 0x7f5d740a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5d84005950 tx=0x7f5d84009260 comp rx=0 tx=0).stop 2024-03-18T00:49:53.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d881035a0 msgr2=0x7f5d88101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:53.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d881035a0 0x7f5d88101550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d7800b7b0 tx=0x7f5d7800bbb0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 shutdown_connections 2024-03-18T00:49:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5d740a2890 0x7f5d740a4d50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d88103f40 0x7f5d88101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5d881005c0 0x7f5d88102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 --2- 172.21.15.49:0/1412092726 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5d881035a0 0x7f5d88101550 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 >> 172.21.15.49:0/1412092726 conn(0x7f5d880fbee0 msgr2=0x7f5d881068f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 shutdown_connections 2024-03-18T00:49:53.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:53.364+0000 7f5d901e7700 1 -- 172.21.15.49:0/1412092726 wait complete. 2024-03-18T00:49:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:53 smithi049 ceph-mon[32130]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:53.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:53 smithi107 ceph-mon[36270]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:53.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:53 smithi106 ceph-mon[35899]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:49:54.053 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-18T00:49:54.068 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:49:54.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.185+0000 7f6cab5b2640 1 -- 172.21.15.49:0/2525751341 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 msgr2=0x7f6ca4101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.185+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/2525751341 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 0x7f6ca4101d70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009900 tx=0x7f6c9802bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:54.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.185+0000 7f6cab5b2640 1 -- 172.21.15.49:0/2525751341 shutdown_connections 2024-03-18T00:49:54.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.185+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/2525751341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ca41022b0 0x7f6ca4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.185+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/2525751341 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 0x7f6ca4101d70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.185+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/2525751341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ca4100700 0x7f6ca4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.185+0000 7f6cab5b2640 1 -- 172.21.15.49:0/2525751341 >> 172.21.15.49:0/2525751341 conn(0x7f6ca40fbc70 msgr2=0x7f6ca40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:54.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.185+0000 7f6cab5b2640 1 -- 172.21.15.49:0/2525751341 shutdown_connections 2024-03-18T00:49:54.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.185+0000 7f6cab5b2640 1 -- 172.21.15.49:0/2525751341 wait complete. 2024-03-18T00:49:54.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6cab5b2640 1 Processor -- start 2024-03-18T00:49:54.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6cab5b2640 1 -- start start 2024-03-18T00:49:54.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6cab5b2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ca4100700 0x7f6ca410b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6cab5b2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 0x7f6ca41096c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6cab5b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ca41022b0 0x7f6ca4109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6cab5b2640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca410d4a0 con 0x7f6ca41018f0 2024-03-18T00:49:54.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6cab5b2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca410d320 con 0x7f6ca4100700 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6cab5b2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca410d620 con 0x7f6ca41022b0 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6ca8b4d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 0x7f6ca41096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6ca8b4d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 0x7f6ca41096c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55024/0 (socket says 172.21.15.49:55024) 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6ca8b4d640 1 -- 172.21.15.49:0/1002364631 learned_addr learned my addr 172.21.15.49:0/1002364631 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6ca934e640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ca4100700 0x7f6ca410b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.186+0000 7f6ca9b4f640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ca41022b0 0x7f6ca4109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6ca8b4d640 1 -- 172.21.15.49:0/1002364631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ca41022b0 msgr2=0x7f6ca4109c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6ca8b4d640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ca41022b0 0x7f6ca4109c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6ca8b4d640 1 -- 172.21.15.49:0/1002364631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ca4100700 msgr2=0x7f6ca410b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6ca8b4d640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ca4100700 0x7f6ca410b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6ca8b4d640 1 -- 172.21.15.49:0/1002364631 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca410a490 con 0x7f6ca41018f0 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6ca934e640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ca4100700 0x7f6ca410b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:54.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6ca9b4f640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ca41022b0 0x7f6ca4109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:54.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6ca8b4d640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 0x7f6ca41096c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009230 tx=0x7f6c9802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:54.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6c967fc640 1 -- 172.21.15.49:0/1002364631 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98041070 con 0x7f6ca41018f0 2024-03-18T00:49:54.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6c967fc640 1 -- 172.21.15.49:0/1002364631 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c98004450 con 0x7f6ca41018f0 2024-03-18T00:49:54.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca41a1bb0 con 0x7f6ca41018f0 2024-03-18T00:49:54.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6c967fc640 1 -- 172.21.15.49:0/1002364631 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003e20 con 0x7f6ca41018f0 2024-03-18T00:49:54.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.187+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca41a2100 con 0x7f6ca41018f0 2024-03-18T00:49:54.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.188+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ca4107a20 con 0x7f6ca41018f0 2024-03-18T00:49:54.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.189+0000 7f6c967fc640 1 -- 172.21.15.49:0/1002364631 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6c98044070 con 0x7f6ca41018f0 2024-03-18T00:49:54.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.190+0000 7f6c967fc640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c900a2800 0x7f6c900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.190+0000 7f6ca934e640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c900a2800 0x7f6c900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.190+0000 7f6c967fc640 1 -- 172.21.15.49:0/1002364631 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6c980f0980 con 0x7f6ca41018f0 2024-03-18T00:49:54.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.190+0000 7f6ca934e640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c900a2800 0x7f6c900a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4101750 tx=0x7f6ca0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:54.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.193+0000 7f6c967fc640 1 -- 172.21.15.49:0/1002364631 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6c980bd3b0 con 0x7f6ca41018f0 2024-03-18T00:49:54.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.379+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ca402d1b0 con 0x7f6ca41018f0 2024-03-18T00:49:54.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.381+0000 7f6c967fc640 1 -- 172.21.15.49:0/1002364631 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7f6c980c1450 con 0x7f6ca41018f0 2024-03-18T00:49:54.381 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:54.381 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c900a2800 msgr2=0x7f6c900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c900a2800 0x7f6c900a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4101750 tx=0x7f6ca0005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 msgr2=0x7f6ca41096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 0x7f6ca41096c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6c98009230 tx=0x7f6c9802c650 comp rx=0 tx=0).stop 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 shutdown_connections 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6c900a2800 0x7f6c900a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6ca41022b0 0x7f6ca4109c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6ca41018f0 0x7f6ca41096c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 --2- 172.21.15.49:0/1002364631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6ca4100700 0x7f6ca410b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 >> 172.21.15.49:0/1002364631 conn(0x7f6ca40fbc70 msgr2=0x7f6ca4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:54.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 shutdown_connections 2024-03-18T00:49:54.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.383+0000 7f6cab5b2640 1 -- 172.21.15.49:0/1002364631 wait complete. 2024-03-18T00:49:54.394 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T00:48:04.316609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T00:48:05.271815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:49:54.394 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:49:54.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.505+0000 7f3ce5f48640 1 -- 172.21.15.49:0/2141470847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce01018f0 msgr2=0x7f3ce0101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.505+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/2141470847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce01018f0 0x7f3ce0101d70 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009900 tx=0x7f3cd402bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:54.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.505+0000 7f3ce5f48640 1 -- 172.21.15.49:0/2141470847 shutdown_connections 2024-03-18T00:49:54.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.505+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/2141470847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce01022b0 0x7f3ce0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.505+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/2141470847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce01018f0 0x7f3ce0101d70 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.505+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/2141470847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ce0100700 0x7f3ce0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.505+0000 7f3ce5f48640 1 -- 172.21.15.49:0/2141470847 >> 172.21.15.49:0/2141470847 conn(0x7f3ce00fbc90 msgr2=0x7f3ce00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.505+0000 7f3ce5f48640 1 -- 172.21.15.49:0/2141470847 shutdown_connections 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.505+0000 7f3ce5f48640 1 -- 172.21.15.49:0/2141470847 wait complete. 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3ce5f48640 1 Processor -- start 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3ce5f48640 1 -- start start 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3ce5f48640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce0100700 0x7f3ce006a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3ce5f48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce01018f0 0x7f3ce019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3ce5f48640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ce01022b0 0x7f3ce019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3ce5f48640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce010c380 con 0x7f3ce0100700 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3ce5f48640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce010c200 con 0x7f3ce01022b0 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3ce5f48640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce010c500 con 0x7f3ce01018f0 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3cdf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce0100700 0x7f3ce006a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3cdf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce0100700 0x7f3ce006a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55036/0 (socket says 172.21.15.49:55036) 2024-03-18T00:49:54.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3cdf7fe640 1 -- 172.21.15.49:0/1685933086 learned_addr learned my addr 172.21.15.49:0/1685933086 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3cdeffd640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce01018f0 0x7f3ce019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3cdffff640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ce01022b0 0x7f3ce019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3cdf7fe640 1 -- 172.21.15.49:0/1685933086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce01018f0 msgr2=0x7f3ce019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.506+0000 7f3cdf7fe640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce01018f0 0x7f3ce019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.507+0000 7f3cdf7fe640 1 -- 172.21.15.49:0/1685933086 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ce01022b0 msgr2=0x7f3ce019ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.507+0000 7f3cdf7fe640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ce01022b0 0x7f3ce019ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.507+0000 7f3cdf7fe640 1 -- 172.21.15.49:0/1685933086 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce019f7a0 con 0x7f3ce0100700 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.507+0000 7f3cdf7fe640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce0100700 0x7f3ce006a920 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000e9c0 tx=0x7f3cd000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.507+0000 7f3cdcff9640 1 -- 172.21.15.49:0/1685933086 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd000cd80 con 0x7f3ce0100700 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.507+0000 7f3cdcff9640 1 -- 172.21.15.49:0/1685933086 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cd00035f0 con 0x7f3ce0100700 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.507+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce019fa30 con 0x7f3ce0100700 2024-03-18T00:49:54.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.507+0000 7f3cdcff9640 1 -- 172.21.15.49:0/1685933086 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0010690 con 0x7f3ce0100700 2024-03-18T00:49:54.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.507+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce019ff50 con 0x7f3ce0100700 2024-03-18T00:49:54.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.508+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce0107a20 con 0x7f3ce0100700 2024-03-18T00:49:54.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.509+0000 7f3cdcff9640 1 -- 172.21.15.49:0/1685933086 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3cd00037a0 con 0x7f3ce0100700 2024-03-18T00:49:54.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.509+0000 7f3cdcff9640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3cc80a2800 0x7f3cc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.510+0000 7f3cdeffd640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3cc80a2800 0x7f3cc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.510+0000 7f3cdcff9640 1 -- 172.21.15.49:0/1685933086 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3cd00cee90 con 0x7f3ce0100700 2024-03-18T00:49:54.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.510+0000 7f3cdeffd640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3cc80a2800 0x7f3cc80a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3cd402c7b0 tx=0x7f3cd4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:54.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.514+0000 7f3cdcff9640 1 -- 172.21.15.49:0/1685933086 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3cd009beb0 con 0x7f3ce0100700 2024-03-18T00:49:54.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:54 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1412092726' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T00:49:54.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:54 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1412092726' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T00:49:54.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:54 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1412092726' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T00:49:54.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.696+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3ce0062480 con 0x7f3ce0100700 2024-03-18T00:49:54.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.696+0000 7f3cdcff9640 1 -- 172.21.15.49:0/1685933086 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f3cd009b7b0 con 0x7f3ce0100700 2024-03-18T00:49:54.696 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:49:54.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3cc80a2800 msgr2=0x7f3cc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3cc80a2800 0x7f3cc80a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3cd402c7b0 tx=0x7f3cd4005fd0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce0100700 msgr2=0x7f3ce006a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce0100700 0x7f3ce006a920 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000e9c0 tx=0x7f3cd000ee80 comp rx=0 tx=0).stop 2024-03-18T00:49:54.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 shutdown_connections 2024-03-18T00:49:54.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3cc80a2800 0x7f3cc80a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ce01022b0 0x7f3ce019ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ce01018f0 0x7f3ce019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 --2- 172.21.15.49:0/1685933086 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ce0100700 0x7f3ce006a920 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 >> 172.21.15.49:0/1685933086 conn(0x7f3ce00fbc90 msgr2=0x7f3ce0104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:54.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 shutdown_connections 2024-03-18T00:49:54.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.698+0000 7f3ce5f48640 1 -- 172.21.15.49:0/1685933086 wait complete. 2024-03-18T00:49:54.709 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T00:49:54.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 -- 172.21.15.49:0/3019318361 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a01018d0 msgr2=0x7fe9a0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/3019318361 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a01018d0 0x7fe9a0101d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009900 tx=0x7fe98c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:54.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 -- 172.21.15.49:0/3019318361 shutdown_connections 2024-03-18T00:49:54.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/3019318361 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9a0102290 0x7fe9a0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/3019318361 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a01018d0 0x7fe9a0101d50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/3019318361 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9a01006e0 0x7fe9a0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 -- 172.21.15.49:0/3019318361 >> 172.21.15.49:0/3019318361 conn(0x7fe9a00fbc70 msgr2=0x7fe9a00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 -- 172.21.15.49:0/3019318361 shutdown_connections 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 -- 172.21.15.49:0/3019318361 wait complete. 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 Processor -- start 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 -- start start 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9a01006e0 0x7fe9a0197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a01018d0 0x7fe9a0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9a0102290 0x7fe9a019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a010d480 con 0x7fe9a0102290 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a010d300 con 0x7fe9a01006e0 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.821+0000 7fe9a6adc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a010d600 con 0x7fe9a01018d0 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a5079640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9a0102290 0x7fe9a019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a5079640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9a0102290 0x7fe9a019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55044/0 (socket says 172.21.15.49:55044) 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a5079640 1 -- 172.21.15.49:0/1516056199 learned_addr learned my addr 172.21.15.49:0/1516056199 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:54.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a4878640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9a01006e0 0x7fe9a0197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe99ffff640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a01018d0 0x7fe9a0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a5079640 1 -- 172.21.15.49:0/1516056199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a01018d0 msgr2=0x7fe9a0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a5079640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a01018d0 0x7fe9a0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a5079640 1 -- 172.21.15.49:0/1516056199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9a01006e0 msgr2=0x7fe9a0197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a5079640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9a01006e0 0x7fe9a0197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a5079640 1 -- 172.21.15.49:0/1516056199 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a019f720 con 0x7fe9a0102290 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a5079640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9a0102290 0x7fe9a019f0b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe99800eb10 tx=0x7fe99800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe99dffb640 1 -- 172.21.15.49:0/1516056199 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe998004b20 con 0x7fe9a0102290 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a019fa10 con 0x7fe9a0102290 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe99dffb640 1 -- 172.21.15.49:0/1516056199 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe998004cc0 con 0x7fe9a0102290 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe99dffb640 1 -- 172.21.15.49:0/1516056199 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe998010760 con 0x7fe9a0102290 2024-03-18T00:49:54.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.822+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a019ff30 con 0x7fe9a0102290 2024-03-18T00:49:54.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.823+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a0107a00 con 0x7fe9a0102290 2024-03-18T00:49:54.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.825+0000 7fe99dffb640 1 -- 172.21.15.49:0/1516056199 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe998005000 con 0x7fe9a0102290 2024-03-18T00:49:54.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.825+0000 7fe99dffb640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe9900a2800 0x7fe9900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:54.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.825+0000 7fe9a4878640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe9900a2800 0x7fe9900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:54.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.825+0000 7fe99dffb640 1 -- 172.21.15.49:0/1516056199 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe9980cf530 con 0x7fe9a0102290 2024-03-18T00:49:54.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.826+0000 7fe9a4878640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe9900a2800 0x7fe9900a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe9940095f0 tx=0x7fe994005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:54.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:54.829+0000 7fe99dffb640 1 -- 172.21.15.49:0/1516056199 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe99809c550 con 0x7fe9a0102290 2024-03-18T00:49:55.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.014+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe9a002d340 con 0x7fe9a0102290 2024-03-18T00:49:55.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.015+0000 7fe99dffb640 1 -- 172.21.15.49:0/1516056199 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fe99809be50 con 0x7fe9a0102290 2024-03-18T00:49:55.015 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 32 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe9900a2800 msgr2=0x7fe9900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe9900a2800 0x7fe9900a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe9940095f0 tx=0x7fe994005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9a0102290 msgr2=0x7fe9a019f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9a0102290 0x7fe9a019f0b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe99800eb10 tx=0x7fe99800c2d0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 shutdown_connections 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe9900a2800 0x7fe9900a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe9a0102290 0x7fe9a019f0b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe9a01018d0 0x7fe9a0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 --2- 172.21.15.49:0/1516056199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe9a01006e0 0x7fe9a0197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 >> 172.21.15.49:0/1516056199 conn(0x7fe9a00fbc70 msgr2=0x7fe9a0104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:55.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 shutdown_connections 2024-03-18T00:49:55.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.017+0000 7fe9a6adc640 1 -- 172.21.15.49:0/1516056199 wait complete. 2024-03-18T00:49:55.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T00:49:55.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 -- 172.21.15.49:0/899942255 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac0100700 msgr2=0x7f5ac0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 --2- 172.21.15.49:0/899942255 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac0100700 0x7f5ac0100b20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0009900 tx=0x7f5ab002bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 -- 172.21.15.49:0/899942255 shutdown_connections 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 --2- 172.21.15.49:0/899942255 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ac01022b0 0x7f5ac0108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 --2- 172.21.15.49:0/899942255 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac01018f0 0x7f5ac0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 --2- 172.21.15.49:0/899942255 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac0100700 0x7f5ac0100b20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 -- 172.21.15.49:0/899942255 >> 172.21.15.49:0/899942255 conn(0x7f5ac00fbc90 msgr2=0x7f5ac00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 -- 172.21.15.49:0/899942255 shutdown_connections 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 -- 172.21.15.49:0/899942255 wait complete. 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 Processor -- start 2024-03-18T00:49:55.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.143+0000 7f5ac5412640 1 -- start start 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5ac5412640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ac0100700 0x7f5ac0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5ac5412640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac01018f0 0x7f5ac0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5ac5412640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac01022b0 0x7f5ac019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5ac5412640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac010d470 con 0x7f5ac01022b0 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5ac5412640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac010d2f0 con 0x7f5ac0100700 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5ac5412640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac010d5f0 con 0x7f5ac01018f0 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac01022b0 0x7f5ac019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac01022b0 0x7f5ac019f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55052/0 (socket says 172.21.15.49:55052) 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abf7fe640 1 -- 172.21.15.49:0/1501425139 learned_addr learned my addr 172.21.15.49:0/1501425139 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abe7fc640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac01018f0 0x7f5ac0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abeffd640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ac0100700 0x7f5ac0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abf7fe640 1 -- 172.21.15.49:0/1501425139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac01018f0 msgr2=0x7f5ac0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abf7fe640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac01018f0 0x7f5ac0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abf7fe640 1 -- 172.21.15.49:0/1501425139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ac0100700 msgr2=0x7f5ac0197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abf7fe640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ac0100700 0x7f5ac0197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abf7fe640 1 -- 172.21.15.49:0/1501425139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac019f730 con 0x7f5ac01022b0 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5abf7fe640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac01022b0 0x7f5ac019f0c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800eb10 tx=0x7f5ab800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5aa7fff640 1 -- 172.21.15.49:0/1501425139 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8004b20 con 0x7f5ac01022b0 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5aa7fff640 1 -- 172.21.15.49:0/1501425139 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ab8004cc0 con 0x7f5ac01022b0 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac019fa20 con 0x7f5ac01022b0 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5aa7fff640 1 -- 172.21.15.49:0/1501425139 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8010760 con 0x7f5ac01022b0 2024-03-18T00:49:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.144+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac019ff20 con 0x7f5ac01022b0 2024-03-18T00:49:55.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.145+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac0107af0 con 0x7f5ac01022b0 2024-03-18T00:49:55.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.146+0000 7f5aa7fff640 1 -- 172.21.15.49:0/1501425139 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5ab8005000 con 0x7f5ac01022b0 2024-03-18T00:49:55.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.147+0000 7f5aa7fff640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.147+0000 7f5abeffd640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.147+0000 7f5aa7fff640 1 -- 172.21.15.49:0/1501425139 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5ab80cf220 con 0x7f5ac01022b0 2024-03-18T00:49:55.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.148+0000 7f5abeffd640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0000c00 tx=0x7f5ab0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:55.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.151+0000 7f5aa7fff640 1 -- 172.21.15.49:0/1501425139 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ab8053740 con 0x7f5ac01022b0 2024-03-18T00:49:55.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.334+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5ac002d400 con 0x7f5ac01022b0 2024-03-18T00:49:55.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.334+0000 7f5aa7fff640 1 -- 172.21.15.49:0/1501425139 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f5ab809c2a0 con 0x7f5ac01022b0 2024-03-18T00:49:55.335 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 64 2024-03-18T00:49:55.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5aa80a2800 msgr2=0x7f5aa80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5ab0000c00 tx=0x7f5ab0005fd0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac01022b0 msgr2=0x7f5ac019f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac01022b0 0x7f5ac019f0c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800eb10 tx=0x7f5ab800c2d0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 shutdown_connections 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5aa80a2800 0x7f5aa80a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ac01022b0 0x7f5ac019f0c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ac01018f0 0x7f5ac0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 --2- 172.21.15.49:0/1501425139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ac0100700 0x7f5ac0197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.336+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 >> 172.21.15.49:0/1501425139 conn(0x7f5ac00fbc90 msgr2=0x7f5ac0104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.337+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 shutdown_connections 2024-03-18T00:49:55.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.337+0000 7f5ac5412640 1 -- 172.21.15.49:0/1501425139 wait complete. 2024-03-18T00:49:55.346 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:55 smithi049 ceph-mon[32130]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:55.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:55 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1002364631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:55.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:55 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1685933086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:55.347 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:55 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1516056199' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:55.348 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:49:55.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.456+0000 7fb1672f8640 1 -- 172.21.15.49:0/3806334699 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 msgr2=0x7fb160102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.456+0000 7fb1672f8640 1 --2- 172.21.15.49:0/3806334699 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 0x7fb160102e80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb150009900 tx=0x7fb15002bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.456+0000 7fb1672f8640 1 -- 172.21.15.49:0/3806334699 shutdown_connections 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.456+0000 7fb1672f8640 1 --2- 172.21.15.49:0/3806334699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1601033c0 0x7fb160109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.456+0000 7fb1672f8640 1 --2- 172.21.15.49:0/3806334699 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 0x7fb160102e80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.456+0000 7fb1672f8640 1 --2- 172.21.15.49:0/3806334699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb160101810 0x7fb160101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.456+0000 7fb1672f8640 1 -- 172.21.15.49:0/3806334699 >> 172.21.15.49:0/3806334699 conn(0x7fb1600fcdc0 msgr2=0x7fb1600ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.456+0000 7fb1672f8640 1 -- 172.21.15.49:0/3806334699 shutdown_connections 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.456+0000 7fb1672f8640 1 -- 172.21.15.49:0/3806334699 wait complete. 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb1672f8640 1 Processor -- start 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb1672f8640 1 -- start start 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb1672f8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb160101810 0x7fb160197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb1672f8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 0x7fb160198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb1672f8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1601033c0 0x7fb16019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb1672f8640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb16010d470 con 0x7fb160102a00 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb1672f8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb16010d2f0 con 0x7fb1601033c0 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb1672f8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb16010d5f0 con 0x7fb160101810 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb164893640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 0x7fb160198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb164893640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 0x7fb160198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55062/0 (socket says 172.21.15.49:55062) 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb164893640 1 -- 172.21.15.49:0/2707551920 learned_addr learned my addr 172.21.15.49:0/2707551920 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb165895640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1601033c0 0x7fb16019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.457+0000 7fb165094640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb160101810 0x7fb160197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb164893640 1 -- 172.21.15.49:0/2707551920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb160101810 msgr2=0x7fb160197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb164893640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb160101810 0x7fb160197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb164893640 1 -- 172.21.15.49:0/2707551920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1601033c0 msgr2=0x7fb16019f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb164893640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1601033c0 0x7fb16019f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb164893640 1 -- 172.21.15.49:0/2707551920 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16019f730 con 0x7fb160102a00 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb164893640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 0x7fb160198050 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb150009230 tx=0x7fb15002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:55.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb1567fc640 1 -- 172.21.15.49:0/2707551920 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb150041070 con 0x7fb160102a00 2024-03-18T00:49:55.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb1567fc640 1 -- 172.21.15.49:0/2707551920 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb150004450 con 0x7fb160102a00 2024-03-18T00:49:55.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb1567fc640 1 -- 172.21.15.49:0/2707551920 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb150003e20 con 0x7fb160102a00 2024-03-18T00:49:55.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16019f9c0 con 0x7fb160102a00 2024-03-18T00:49:55.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.458+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16019fe60 con 0x7fb160102a00 2024-03-18T00:49:55.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.459+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb160108b30 con 0x7fb160102a00 2024-03-18T00:49:55.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.460+0000 7fb1567fc640 1 -- 172.21.15.49:0/2707551920 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb150044070 con 0x7fb160102a00 2024-03-18T00:49:55.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.461+0000 7fb1567fc640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb14c0a2880 0x7fb14c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.461+0000 7fb165094640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb14c0a2880 0x7fb14c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.461+0000 7fb1567fc640 1 -- 172.21.15.49:0/2707551920 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb1500f08e0 con 0x7fb160102a00 2024-03-18T00:49:55.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.461+0000 7fb165094640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb14c0a2880 0x7fb14c0a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb160102860 tx=0x7fb15c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:55.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.464+0000 7fb1567fc640 1 -- 172.21.15.49:0/2707551920 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb1500bd310 con 0x7fb160102a00 2024-03-18T00:49:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:55 smithi107 ceph-mon[36270]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:55 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1002364631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:55 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1685933086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:55 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1516056199' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:55.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.689+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb16002d370 con 0x7fb160102a00 2024-03-18T00:49:55.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.689+0000 7fb1567fc640 1 -- 172.21.15.49:0/2707551920 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11932 (secure 0 0 0) 0x7fb1500c13b0 con 0x7fb160102a00 2024-03-18T00:49:55.689 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:55.689 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:48:12.008019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24499,24506,24481,24470,24463]},"id":1}]} 2024-03-18T00:49:55.690 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 24 2024-03-18T00:49:55.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.691+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb14c0a2880 msgr2=0x7fb14c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.691+0000 7fb1672f8640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb14c0a2880 0x7fb14c0a4d40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb160102860 tx=0x7fb15c005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.691+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 msgr2=0x7fb160198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.691+0000 7fb1672f8640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 0x7fb160198050 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb150009230 tx=0x7fb15002c650 comp rx=0 tx=0).stop 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.692+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 shutdown_connections 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.692+0000 7fb1672f8640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb14c0a2880 0x7fb14c0a4d40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.692+0000 7fb1672f8640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1601033c0 0x7fb16019f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.692+0000 7fb1672f8640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb160102a00 0x7fb160198050 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.692+0000 7fb1672f8640 1 --2- 172.21.15.49:0/2707551920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb160101810 0x7fb160197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.692+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 >> 172.21.15.49:0/2707551920 conn(0x7fb1600fcdc0 msgr2=0x7fb160105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.692+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 shutdown_connections 2024-03-18T00:49:55.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.692+0000 7fb1672f8640 1 -- 172.21.15.49:0/2707551920 wait complete. 2024-03-18T00:49:55.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:55 smithi106 ceph-mon[35899]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:55.697 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:55 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1002364631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:55.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:55 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1685933086' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:55.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:55 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1516056199' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:55.704 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:49:55.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.812+0000 7fb520952640 1 -- 172.21.15.49:0/3368168014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 msgr2=0x7fb51c1046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.812+0000 7fb520952640 1 --2- 172.21.15.49:0/3368168014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 0x7fb51c1046f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb50c009900 tx=0x7fb50c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.812+0000 7fb520952640 1 -- 172.21.15.49:0/3368168014 shutdown_connections 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.812+0000 7fb520952640 1 --2- 172.21.15.49:0/3368168014 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb51c104c30 0x7fb51c107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.812+0000 7fb520952640 1 --2- 172.21.15.49:0/3368168014 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb51c107560 0x7fb51c1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.812+0000 7fb520952640 1 --2- 172.21.15.49:0/3368168014 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 0x7fb51c1046f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.812+0000 7fb520952640 1 -- 172.21.15.49:0/3368168014 >> 172.21.15.49:0/3368168014 conn(0x7fb51c0fbd10 msgr2=0x7fb51c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 -- 172.21.15.49:0/3368168014 shutdown_connections 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 -- 172.21.15.49:0/3368168014 wait complete. 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 Processor -- start 2024-03-18T00:49:55.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 -- start start 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 0x7fb51c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb51c104c30 0x7fb51c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb51c107560 0x7fb51c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c10d490 con 0x7fb51c102300 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c10d310 con 0x7fb51c107560 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb520952640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb51c10d610 con 0x7fb51c104c30 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.813+0000 7fb51a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 0x7fb51c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 0x7fb51c197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55084/0 (socket says 172.21.15.49:55084) 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51a59c640 1 -- 172.21.15.49:0/1867069459 learned_addr learned my addr 172.21.15.49:0/1867069459 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51ad9d640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb51c107560 0x7fb51c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb519d9b640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb51c104c30 0x7fb51c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51a59c640 1 -- 172.21.15.49:0/1867069459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb51c104c30 msgr2=0x7fb51c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51a59c640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb51c104c30 0x7fb51c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51a59c640 1 -- 172.21.15.49:0/1867069459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb51c107560 msgr2=0x7fb51c19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51a59c640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb51c107560 0x7fb51c19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51a59c640 1 -- 172.21.15.49:0/1867069459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb51c19f760 con 0x7fb51c102300 2024-03-18T00:49:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51ad9d640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb51c107560 0x7fb51c19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb51a59c640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 0x7fb51c197b40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb50c009230 tx=0x7fb50c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb50b7fe640 1 -- 172.21.15.49:0/1867069459 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c041070 con 0x7fb51c102300 2024-03-18T00:49:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb50b7fe640 1 -- 172.21.15.49:0/1867069459 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb50c004450 con 0x7fb51c102300 2024-03-18T00:49:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb50b7fe640 1 -- 172.21.15.49:0/1867069459 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb50c003db0 con 0x7fb51c102300 2024-03-18T00:49:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb51c19f9f0 con 0x7fb51c102300 2024-03-18T00:49:55.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.814+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb51c19fee0 con 0x7fb51c102300 2024-03-18T00:49:55.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.815+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb51c0fd800 con 0x7fb51c102300 2024-03-18T00:49:55.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.816+0000 7fb50b7fe640 1 -- 172.21.15.49:0/1867069459 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb50c044070 con 0x7fb51c102300 2024-03-18T00:49:55.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.817+0000 7fb50b7fe640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5040a2880 0x7fb5040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:55.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.817+0000 7fb519d9b640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5040a2880 0x7fb5040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:55.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.817+0000 7fb50b7fe640 1 -- 172.21.15.49:0/1867069459 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb50c0f0950 con 0x7fb51c102300 2024-03-18T00:49:55.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.817+0000 7fb519d9b640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5040a2880 0x7fb5040a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb5100095f0 tx=0x7fb510005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:55.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:55.821+0000 7fb50b7fe640 1 -- 172.21.15.49:0/1867069459 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb50c0bd9d0 con 0x7fb51c102300 2024-03-18T00:49:56.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.004+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb51c02d380 con 0x7fb51c102300 2024-03-18T00:49:56.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.006+0000 7fb50b7fe640 1 -- 172.21.15.49:0/1867069459 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7fb50c0bd2d0 con 0x7fb51c102300 2024-03-18T00:49:56.006 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:56.006 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:49:56.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5040a2880 msgr2=0x7fb5040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5040a2880 0x7fb5040a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb5100095f0 tx=0x7fb510005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:56.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 msgr2=0x7fb51c197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 0x7fb51c197b40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb50c009230 tx=0x7fb50c02c650 comp rx=0 tx=0).stop 2024-03-18T00:49:56.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 shutdown_connections 2024-03-18T00:49:56.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb5040a2880 0x7fb5040a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb51c107560 0x7fb51c19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb51c104c30 0x7fb51c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 --2- 172.21.15.49:0/1867069459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb51c102300 0x7fb51c197b40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 >> 172.21.15.49:0/1867069459 conn(0x7fb51c0fbd10 msgr2=0x7fb51c102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:56.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 shutdown_connections 2024-03-18T00:49:56.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.008+0000 7fb520952640 1 -- 172.21.15.49:0/1867069459 wait complete. 2024-03-18T00:49:56.019 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T00:48:04.316609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T00:48:05.271815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:49:56.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:49:56.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.134+0000 7f5c1486a640 1 -- 172.21.15.49:0/1234255291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c10102290 msgr2=0x7f5c10108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.134+0000 7f5c1486a640 1 --2- 172.21.15.49:0/1234255291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c10102290 0x7f5c10108b20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009ca0 tx=0x7f5c0802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.134+0000 7f5c1486a640 1 -- 172.21.15.49:0/1234255291 shutdown_connections 2024-03-18T00:49:56.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.134+0000 7f5c1486a640 1 --2- 172.21.15.49:0/1234255291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c101018d0 0x7f5c10101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.134+0000 7f5c1486a640 1 --2- 172.21.15.49:0/1234255291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c10102290 0x7f5c10108b20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.134+0000 7f5c1486a640 1 --2- 172.21.15.49:0/1234255291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c101006e0 0x7f5c10100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.134+0000 7f5c1486a640 1 -- 172.21.15.49:0/1234255291 >> 172.21.15.49:0/1234255291 conn(0x7f5c100fbc70 msgr2=0x7f5c100fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.134+0000 7f5c1486a640 1 -- 172.21.15.49:0/1234255291 shutdown_connections 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.134+0000 7f5c1486a640 1 -- 172.21.15.49:0/1234255291 wait complete. 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c1486a640 1 Processor -- start 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c1486a640 1 -- start start 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c1486a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c101006e0 0x7f5c10197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c1486a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c101018d0 0x7f5c10198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c1486a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10102290 0x7f5c1019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c1486a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1010c340 con 0x7f5c101006e0 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c1486a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1010c1c0 con 0x7f5c101018d0 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c1486a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1010c4c0 con 0x7f5c10102290 2024-03-18T00:49:56.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c0e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c101006e0 0x7f5c10197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c0e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c101006e0 0x7f5c10197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55094/0 (socket says 172.21.15.49:55094) 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c0e59c640 1 -- 172.21.15.49:0/3350425674 learned_addr learned my addr 172.21.15.49:0/3350425674 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.135+0000 7f5c0dd9b640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c101018d0 0x7f5c10198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5c0ed9d640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10102290 0x7f5c1019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5c0e59c640 1 -- 172.21.15.49:0/3350425674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10102290 msgr2=0x7f5c1019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5c0e59c640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10102290 0x7f5c1019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5c0e59c640 1 -- 172.21.15.49:0/3350425674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c101018d0 msgr2=0x7f5c10198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5c0e59c640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c101018d0 0x7f5c10198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5c0e59c640 1 -- 172.21.15.49:0/3350425674 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1019f6f0 con 0x7f5c101006e0 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5c0e59c640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c101006e0 0x7f5c10197ad0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000e9c0 tx=0x7f5c0000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5bff7fe640 1 -- 172.21.15.49:0/3350425674 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000cde0 con 0x7f5c101006e0 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c1019f9e0 con 0x7f5c101006e0 2024-03-18T00:49:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5bff7fe640 1 -- 172.21.15.49:0/3350425674 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c00010910 con 0x7f5c101006e0 2024-03-18T00:49:56.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5bff7fe640 1 -- 172.21.15.49:0/3350425674 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00018a10 con 0x7f5c101006e0 2024-03-18T00:49:56.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.136+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c1019ff80 con 0x7f5c101006e0 2024-03-18T00:49:56.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.138+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c100685d0 con 0x7f5c101006e0 2024-03-18T00:49:56.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.138+0000 7f5bff7fe640 1 -- 172.21.15.49:0/3350425674 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5c0001f020 con 0x7f5c101006e0 2024-03-18T00:49:56.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.139+0000 7f5bff7fe640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bf80a2800 0x7f5bf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.139+0000 7f5c0dd9b640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bf80a2800 0x7f5bf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.139+0000 7f5bff7fe640 1 -- 172.21.15.49:0/3350425674 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5c000cec40 con 0x7f5c101006e0 2024-03-18T00:49:56.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.139+0000 7f5c0dd9b640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bf80a2800 0x7f5bf80a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5c10198fe0 tx=0x7f5c04005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:56.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.143+0000 7f5bff7fe640 1 -- 172.21.15.49:0/3350425674 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5c0009bcc0 con 0x7f5c101006e0 2024-03-18T00:49:56.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.351+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5c101a0880 con 0x7f5c101006e0 2024-03-18T00:49:56.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.352+0000 7f5bff7fe640 1 -- 172.21.15.49:0/3350425674 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f5c0009b5c0 con 0x7f5c101006e0 2024-03-18T00:49:56.352 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bf80a2800 msgr2=0x7f5bf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bf80a2800 0x7f5bf80a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5c10198fe0 tx=0x7f5c04005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c101006e0 msgr2=0x7f5c10197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c101006e0 0x7f5c10197ad0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000e9c0 tx=0x7f5c0000ee80 comp rx=0 tx=0).stop 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 shutdown_connections 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bf80a2800 0x7f5bf80a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5c101018d0 0x7f5c10198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5c10102290 0x7f5c1019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 --2- 172.21.15.49:0/3350425674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5c101006e0 0x7f5c10197ad0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 >> 172.21.15.49:0/3350425674 conn(0x7f5c100fbc70 msgr2=0x7f5c100fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:56.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 shutdown_connections 2024-03-18T00:49:56.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.354+0000 7f5c1486a640 1 -- 172.21.15.49:0/3350425674 wait complete. 2024-03-18T00:49:56.365 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T00:49:56.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.479+0000 7f9465c7e640 1 -- 172.21.15.49:0/1961588447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94600ffe60 msgr2=0x7f9460100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.479+0000 7f9465c7e640 1 --2- 172.21.15.49:0/1961588447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94600ffe60 0x7f9460100280 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9450009be0 tx=0x7f945002bef0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.479+0000 7f9465c7e640 1 -- 172.21.15.49:0/1961588447 shutdown_connections 2024-03-18T00:49:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.479+0000 7f9465c7e640 1 --2- 172.21.15.49:0/1961588447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9460100fb0 0x7f9460101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.479+0000 7f9465c7e640 1 --2- 172.21.15.49:0/1961588447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9460101970 0x7f9460108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.479+0000 7f9465c7e640 1 --2- 172.21.15.49:0/1961588447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94600ffe60 0x7f9460100280 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.479+0000 7f9465c7e640 1 -- 172.21.15.49:0/1961588447 >> 172.21.15.49:0/1961588447 conn(0x7f94600fb3d0 msgr2=0x7f94600fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.479+0000 7f9465c7e640 1 -- 172.21.15.49:0/1961588447 shutdown_connections 2024-03-18T00:49:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.479+0000 7f9465c7e640 1 -- 172.21.15.49:0/1961588447 wait complete. 2024-03-18T00:49:56.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.480+0000 7f9465c7e640 1 Processor -- start 2024-03-18T00:49:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.480+0000 7f9465c7e640 1 -- start start 2024-03-18T00:49:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.480+0000 7f9465c7e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94600ffe60 0x7f9460197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.480+0000 7f9465c7e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9460100fb0 0x7f9460198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.480+0000 7f9465c7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9460101970 0x7f946019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.480+0000 7f9465c7e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f946010d480 con 0x7f9460100fb0 2024-03-18T00:49:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.480+0000 7f9465c7e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f946010d300 con 0x7f94600ffe60 2024-03-18T00:49:56.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.480+0000 7f9465c7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f946010d600 con 0x7f9460101970 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.480+0000 7f945effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9460100fb0 0x7f9460198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94600ffe60 0x7f9460197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9460100fb0 0x7f9460198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55104/0 (socket says 172.21.15.49:55104) 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94600ffe60 0x7f9460197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:39762/0 (socket says 172.21.15.49:39762) 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945effd640 1 -- 172.21.15.49:0/4291271947 learned_addr learned my addr 172.21.15.49:0/4291271947 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945ffff640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9460101970 0x7f946019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945effd640 1 -- 172.21.15.49:0/4291271947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9460101970 msgr2=0x7f946019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945effd640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9460101970 0x7f946019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945effd640 1 -- 172.21.15.49:0/4291271947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94600ffe60 msgr2=0x7f9460197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945effd640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94600ffe60 0x7f9460197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945effd640 1 -- 172.21.15.49:0/4291271947 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946019f720 con 0x7f9460100fb0 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945f7fe640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94600ffe60 0x7f9460197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945effd640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9460100fb0 0x7f9460198040 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f945400e9c0 tx=0x7f945400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945cff9640 1 -- 172.21.15.49:0/4291271947 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945400ccc0 con 0x7f9460100fb0 2024-03-18T00:49:56.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f946019fa10 con 0x7f9460100fb0 2024-03-18T00:49:56.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945cff9640 1 -- 172.21.15.49:0/4291271947 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94540035f0 con 0x7f9460100fb0 2024-03-18T00:49:56.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f945cff9640 1 -- 172.21.15.49:0/4291271947 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f945400ce60 con 0x7f9460100fb0 2024-03-18T00:49:56.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.481+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f946019ffb0 con 0x7f9460100fb0 2024-03-18T00:49:56.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.482+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9460107170 con 0x7f9460100fb0 2024-03-18T00:49:56.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.484+0000 7f945cff9640 1 -- 172.21.15.49:0/4291271947 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f94540037a0 con 0x7f9460100fb0 2024-03-18T00:49:56.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.485+0000 7f945cff9640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f94480a2800 0x7f94480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.485+0000 7f945cff9640 1 -- 172.21.15.49:0/4291271947 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f94540cef00 con 0x7f9460100fb0 2024-03-18T00:49:56.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.485+0000 7f945f7fe640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f94480a2800 0x7f94480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.485+0000 7f945f7fe640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f94480a2800 0x7f94480a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9450006010 tx=0x7f9450000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:56.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.488+0000 7f945cff9640 1 -- 172.21.15.49:0/4291271947 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f945409c010 con 0x7f9460100fb0 2024-03-18T00:49:56.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:56 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1501425139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:56.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:56 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2707551920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:56.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:56 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1867069459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:56.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.674+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f946002d370 con 0x7f9460100fb0 2024-03-18T00:49:56.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.674+0000 7f945cff9640 1 -- 172.21.15.49:0/4291271947 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f945409b910 con 0x7f9460100fb0 2024-03-18T00:49:56.674 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 32 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.676+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f94480a2800 msgr2=0x7f94480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.676+0000 7f9465c7e640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f94480a2800 0x7f94480a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9450006010 tx=0x7f9450000f40 comp rx=0 tx=0).stop 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.676+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9460100fb0 msgr2=0x7f9460198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.676+0000 7f9465c7e640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9460100fb0 0x7f9460198040 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f945400e9c0 tx=0x7f945400edc0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.677+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 shutdown_connections 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.677+0000 7f9465c7e640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f94480a2800 0x7f94480a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.677+0000 7f9465c7e640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9460101970 0x7f946019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.677+0000 7f9465c7e640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9460100fb0 0x7f9460198040 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.677+0000 7f9465c7e640 1 --2- 172.21.15.49:0/4291271947 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f94600ffe60 0x7f9460197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.677+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 >> 172.21.15.49:0/4291271947 conn(0x7f94600fb3d0 msgr2=0x7f9460104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:56.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.677+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 shutdown_connections 2024-03-18T00:49:56.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.677+0000 7f9465c7e640 1 -- 172.21.15.49:0/4291271947 wait complete. 2024-03-18T00:49:56.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:56 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1501425139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:56 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2707551920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:56.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:56 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1867069459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:56.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T00:49:56.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:56 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1501425139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:56 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2707551920' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:56.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:56 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1867069459' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:56.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.815+0000 7fa17b153640 1 -- 172.21.15.49:0/3399424518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 msgr2=0x7fa174108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.815+0000 7fa17b153640 1 --2- 172.21.15.49:0/3399424518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 0x7fa174108b20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009ca0 tx=0x7fa16c02c100 comp rx=0 tx=0).stop 2024-03-18T00:49:56.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.815+0000 7fa17b153640 1 -- 172.21.15.49:0/3399424518 shutdown_connections 2024-03-18T00:49:56.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.815+0000 7fa17b153640 1 --2- 172.21.15.49:0/3399424518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741018d0 0x7fa174101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.815+0000 7fa17b153640 1 --2- 172.21.15.49:0/3399424518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 0x7fa174108b20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.815+0000 7fa17b153640 1 --2- 172.21.15.49:0/3399424518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1741006e0 0x7fa174100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.815+0000 7fa17b153640 1 -- 172.21.15.49:0/3399424518 >> 172.21.15.49:0/3399424518 conn(0x7fa1740fbc70 msgr2=0x7fa1740fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.815+0000 7fa17b153640 1 -- 172.21.15.49:0/3399424518 shutdown_connections 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.815+0000 7fa17b153640 1 -- 172.21.15.49:0/3399424518 wait complete. 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa17b153640 1 Processor -- start 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa17b153640 1 -- start start 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa17b153640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1741006e0 0x7fa174197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa17b153640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741018d0 0x7fa174198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa17b153640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 0x7fa17419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa17b153640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa17410c360 con 0x7fa174102290 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa17b153640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa17410c1e0 con 0x7fa1741018d0 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa17b153640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa17410c4e0 con 0x7fa1741006e0 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa1796f0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 0x7fa17419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa1796f0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 0x7fa17419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55108/0 (socket says 172.21.15.49:55108) 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa1796f0640 1 -- 172.21.15.49:0/2483012844 learned_addr learned my addr 172.21.15.49:0/2483012844 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa178eef640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1741006e0 0x7fa174197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa173fff640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741018d0 0x7fa174198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa1796f0640 1 -- 172.21.15.49:0/2483012844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1741006e0 msgr2=0x7fa174197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa1796f0640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1741006e0 0x7fa174197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa1796f0640 1 -- 172.21.15.49:0/2483012844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741018d0 msgr2=0x7fa174198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa1796f0640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741018d0 0x7fa174198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa1796f0640 1 -- 172.21.15.49:0/2483012844 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17419f720 con 0x7fa174102290 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.816+0000 7fa1796f0640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 0x7fa17419f0b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa16c037040 tx=0x7fa16c035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.817+0000 7fa171ffb640 1 -- 172.21.15.49:0/2483012844 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c039040 con 0x7fa174102290 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.817+0000 7fa171ffb640 1 -- 172.21.15.49:0/2483012844 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa16c044070 con 0x7fa174102290 2024-03-18T00:49:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.817+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa17419f9b0 con 0x7fa174102290 2024-03-18T00:49:56.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.817+0000 7fa171ffb640 1 -- 172.21.15.49:0/2483012844 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c0043a0 con 0x7fa174102290 2024-03-18T00:49:56.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.817+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa17419fea0 con 0x7fa174102290 2024-03-18T00:49:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.818+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1741a00d0 con 0x7fa174102290 2024-03-18T00:49:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.819+0000 7fa171ffb640 1 -- 172.21.15.49:0/2483012844 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa16c004540 con 0x7fa174102290 2024-03-18T00:49:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.820+0000 7fa171ffb640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1640a2800 0x7fa1640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.820+0000 7fa171ffb640 1 -- 172.21.15.49:0/2483012844 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa16c0f0fe0 con 0x7fa174102290 2024-03-18T00:49:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.820+0000 7fa178eef640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1640a2800 0x7fa1640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.820+0000 7fa178eef640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1640a2800 0x7fa1640a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa174101730 tx=0x7fa168005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:56.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:56.824+0000 7fa171ffb640 1 -- 172.21.15.49:0/2483012844 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa16c0be100 con 0x7fa174102290 2024-03-18T00:49:57.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.007+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa1741a0460 con 0x7fa174102290 2024-03-18T00:49:57.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.008+0000 7fa171ffb640 1 -- 172.21.15.49:0/2483012844 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fa16c0bda00 con 0x7fa174102290 2024-03-18T00:49:57.009 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 64 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1640a2800 msgr2=0x7fa1640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1640a2800 0x7fa1640a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa174101730 tx=0x7fa168005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 msgr2=0x7fa17419f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 0x7fa17419f0b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa16c037040 tx=0x7fa16c035d90 comp rx=0 tx=0).stop 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 shutdown_connections 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1640a2800 0x7fa1640a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1741018d0 0x7fa174198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa174102290 0x7fa17419f0b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 --2- 172.21.15.49:0/2483012844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1741006e0 0x7fa174197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 >> 172.21.15.49:0/2483012844 conn(0x7fa1740fbc70 msgr2=0x7fa1740fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 shutdown_connections 2024-03-18T00:49:57.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.010+0000 7fa17b153640 1 -- 172.21.15.49:0/2483012844 wait complete. 2024-03-18T00:49:57.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:49:57.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.129+0000 7fcc33116640 1 -- 172.21.15.49:0/1972539963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 msgr2=0x7fcc2c104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.129+0000 7fcc33116640 1 --2- 172.21.15.49:0/1972539963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 0x7fcc2c104710 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009900 tx=0x7fcc2002bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:57.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.129+0000 7fcc33116640 1 -- 172.21.15.49:0/1972539963 shutdown_connections 2024-03-18T00:49:57.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.129+0000 7fcc33116640 1 --2- 172.21.15.49:0/1972539963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc2c107580 0x7fcc2c109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.129+0000 7fcc33116640 1 --2- 172.21.15.49:0/1972539963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c104c50 0x7fcc2c107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.129+0000 7fcc33116640 1 --2- 172.21.15.49:0/1972539963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 0x7fcc2c104710 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.129+0000 7fcc33116640 1 -- 172.21.15.49:0/1972539963 >> 172.21.15.49:0/1972539963 conn(0x7fcc2c0fbc70 msgr2=0x7fcc2c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.129+0000 7fcc33116640 1 -- 172.21.15.49:0/1972539963 shutdown_connections 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.129+0000 7fcc33116640 1 -- 172.21.15.49:0/1972539963 wait complete. 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc33116640 1 Processor -- start 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc33116640 1 -- start start 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc33116640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 0x7fcc2c101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc33116640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc2c104c50 0x7fcc2c1002d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc33116640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c107580 0x7fcc2c100810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc33116640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c10d470 con 0x7fcc2c102320 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc33116640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c10d2f0 con 0x7fcc2c104c50 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc33116640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c10d5f0 con 0x7fcc2c107580 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc30eb2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 0x7fcc2c101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc30eb2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 0x7fcc2c101c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55116/0 (socket says 172.21.15.49:55116) 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc30eb2640 1 -- 172.21.15.49:0/39466370 learned_addr learned my addr 172.21.15.49:0/39466370 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc316b3640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c107580 0x7fcc2c100810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc2bfff640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc2c104c50 0x7fcc2c1002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc30eb2640 1 -- 172.21.15.49:0/39466370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c107580 msgr2=0x7fcc2c100810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc30eb2640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c107580 0x7fcc2c100810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc30eb2640 1 -- 172.21.15.49:0/39466370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc2c104c50 msgr2=0x7fcc2c1002d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc30eb2640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc2c104c50 0x7fcc2c1002d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.130+0000 7fcc30eb2640 1 -- 172.21.15.49:0/39466370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2c1010a0 con 0x7fcc2c102320 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.131+0000 7fcc30eb2640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 0x7fcc2c101c90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009230 tx=0x7fcc2002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.131+0000 7fcc29ffb640 1 -- 172.21.15.49:0/39466370 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc20041070 con 0x7fcc2c102320 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.131+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2c1a1c40 con 0x7fcc2c102320 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.131+0000 7fcc29ffb640 1 -- 172.21.15.49:0/39466370 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc20004450 con 0x7fcc2c102320 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.131+0000 7fcc29ffb640 1 -- 172.21.15.49:0/39466370 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc20003db0 con 0x7fcc2c102320 2024-03-18T00:49:57.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.131+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2c1a2190 con 0x7fcc2c102320 2024-03-18T00:49:57.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.132+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc2c0fd760 con 0x7fcc2c102320 2024-03-18T00:49:57.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.133+0000 7fcc29ffb640 1 -- 172.21.15.49:0/39466370 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcc20044070 con 0x7fcc2c102320 2024-03-18T00:49:57.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.134+0000 7fcc29ffb640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcc1c0a2800 0x7fcc1c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.134+0000 7fcc2bfff640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcc1c0a2800 0x7fcc1c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.134+0000 7fcc29ffb640 1 -- 172.21.15.49:0/39466370 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcc200f09c0 con 0x7fcc2c102320 2024-03-18T00:49:57.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.134+0000 7fcc2bfff640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcc1c0a2800 0x7fcc1c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcc180095f0 tx=0x7fcc18005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.137+0000 7fcc29ffb640 1 -- 172.21.15.49:0/39466370 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcc200bda40 con 0x7fcc2c102320 2024-03-18T00:49:57.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.363+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc2c02d370 con 0x7fcc2c102320 2024-03-18T00:49:57.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.364+0000 7fcc29ffb640 1 -- 172.21.15.49:0/39466370 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11932 (secure 0 0 0) 0x7fcc200bd340 con 0x7fcc2c102320 2024-03-18T00:49:57.365 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:57.365 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:48:12.008019+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24499,24506,24481,24470,24463]},"id":1}]} 2024-03-18T00:49:57.365 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 24 2024-03-18T00:49:57.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcc1c0a2800 msgr2=0x7fcc1c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcc1c0a2800 0x7fcc1c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcc180095f0 tx=0x7fcc18005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:57.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 msgr2=0x7fcc2c101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 0x7fcc2c101c90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009230 tx=0x7fcc2002c650 comp rx=0 tx=0).stop 2024-03-18T00:49:57.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 shutdown_connections 2024-03-18T00:49:57.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcc1c0a2800 0x7fcc1c0a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcc2c104c50 0x7fcc2c1002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc2c107580 0x7fcc2c100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 --2- 172.21.15.49:0/39466370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcc2c102320 0x7fcc2c101c90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 >> 172.21.15.49:0/39466370 conn(0x7fcc2c0fbc70 msgr2=0x7fcc2c102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 shutdown_connections 2024-03-18T00:49:57.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.366+0000 7fcc33116640 1 -- 172.21.15.49:0/39466370 wait complete. 2024-03-18T00:49:57.378 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:49:57.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 -- 172.21.15.49:0/3078505586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84141074e0 msgr2=0x7f8414109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 --2- 172.21.15.49:0/3078505586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84141074e0 0x7f8414109930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8404009ca0 tx=0x7f840402c0f0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 -- 172.21.15.49:0/3078505586 shutdown_connections 2024-03-18T00:49:57.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 --2- 172.21.15.49:0/3078505586 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8414104bb0 0x7f8414106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 --2- 172.21.15.49:0/3078505586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f84141074e0 0x7f8414109930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 --2- 172.21.15.49:0/3078505586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8414102280 0x7f8414104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 -- 172.21.15.49:0/3078505586 >> 172.21.15.49:0/3078505586 conn(0x7f84140fbc70 msgr2=0x7f84140fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 -- 172.21.15.49:0/3078505586 shutdown_connections 2024-03-18T00:49:57.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 -- 172.21.15.49:0/3078505586 wait complete. 2024-03-18T00:49:57.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 Processor -- start 2024-03-18T00:49:57.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.487+0000 7f841a663640 1 -- start start 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f841a663640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414102280 0x7f8414197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f841a663640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8414104bb0 0x7f8414197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f841a663640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84141074e0 0x7f841419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f841a663640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f841410d4a0 con 0x7f8414104bb0 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f841a663640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f841410d320 con 0x7f84141074e0 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f841a663640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f841410d620 con 0x7f8414102280 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f84137fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8414104bb0 0x7f8414197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f84137fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8414104bb0 0x7f8414197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55138/0 (socket says 172.21.15.49:55138) 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f8413fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414102280 0x7f8414197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f84137fe640 1 -- 172.21.15.49:0/1883517170 learned_addr learned my addr 172.21.15.49:0/1883517170 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f8418c00640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84141074e0 0x7f841419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f84137fe640 1 -- 172.21.15.49:0/1883517170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414102280 msgr2=0x7f8414197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f84137fe640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414102280 0x7f8414197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f84137fe640 1 -- 172.21.15.49:0/1883517170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84141074e0 msgr2=0x7f841419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f84137fe640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84141074e0 0x7f841419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f84137fe640 1 -- 172.21.15.49:0/1883517170 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841419f6d0 con 0x7f8414104bb0 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f8418c00640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84141074e0 0x7f841419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f8413fff640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414102280 0x7f8414197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.488+0000 7f84137fe640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8414104bb0 0x7f8414197ff0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e9f0 tx=0x7f83fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.489+0000 7f84117fa640 1 -- 172.21.15.49:0/1883517170 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc00ccc0 con 0x7f8414104bb0 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.489+0000 7f84117fa640 1 -- 172.21.15.49:0/1883517170 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83fc005500 con 0x7f8414104bb0 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.489+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f841419f9c0 con 0x7f8414104bb0 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.489+0000 7f84117fa640 1 -- 172.21.15.49:0/1883517170 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc010690 con 0x7f8414104bb0 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.489+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f841419fec0 con 0x7f8414104bb0 2024-03-18T00:49:57.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.490+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84140fd760 con 0x7f8414104bb0 2024-03-18T00:49:57.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.491+0000 7f84117fa640 1 -- 172.21.15.49:0/1883517170 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f83fc0046c0 con 0x7f8414104bb0 2024-03-18T00:49:57.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.491+0000 7f84117fa640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f84000a2800 0x7f84000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.491+0000 7f8413fff640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f84000a2800 0x7f84000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.491+0000 7f84117fa640 1 -- 172.21.15.49:0/1883517170 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f83fc0cede0 con 0x7f8414104bb0 2024-03-18T00:49:57.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.492+0000 7f8413fff640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f84000a2800 0x7f84000a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f840c0095f0 tx=0x7f840c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.496+0000 7f84117fa640 1 -- 172.21.15.49:0/1883517170 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f83fc09be60 con 0x7f8414104bb0 2024-03-18T00:49:57.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:57 smithi049 ceph-mon[32130]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T00:49:57.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:57 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3350425674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:57.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:57 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/4291271947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:57.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:57 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2483012844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:57 smithi107 ceph-mon[36270]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T00:49:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:57 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3350425674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:57 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/4291271947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:57 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2483012844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:57.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.683+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84140623e0 con 0x7f8414104bb0 2024-03-18T00:49:57.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.685+0000 7f84117fa640 1 -- 172.21.15.49:0/1883517170 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7f83fc09b760 con 0x7f8414104bb0 2024-03-18T00:49:57.686 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:49:57.686 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:49:57.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f84000a2800 msgr2=0x7f84000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f84000a2800 0x7f84000a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f840c0095f0 tx=0x7f840c005f90 comp rx=0 tx=0).stop 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8414104bb0 msgr2=0x7f8414197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8414104bb0 0x7f8414197ff0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e9f0 tx=0x7f83fc00eeb0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 shutdown_connections 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f84000a2800 0x7f84000a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f84141074e0 0x7f841419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8414104bb0 0x7f8414197ff0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 --2- 172.21.15.49:0/1883517170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8414102280 0x7f8414197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 >> 172.21.15.49:0/1883517170 conn(0x7f84140fbc70 msgr2=0x7f8414102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 shutdown_connections 2024-03-18T00:49:57.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.687+0000 7f841a663640 1 -- 172.21.15.49:0/1883517170 wait complete. 2024-03-18T00:49:57.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:57 smithi106 ceph-mon[35899]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T00:49:57.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:57 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3350425674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:49:57.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:57 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/4291271947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:49:57.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:57 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2483012844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:49:57.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T00:48:05.881816+0000', 'modified': '2024-03-18T00:48:12.008019+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24506, 'mds_1': 24499, 'mds_2': 24481, 'mds_3': 24470, 'mds_4': 24463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14613': {'gid': 14613, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.49:6839/1964983089', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6838', 'nonce': 1964983089}, {'type': 'v1', 'addr': '172.21.15.49:6839', 'nonce': 1964983089}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14625': {'gid': 14625, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.49:6841/3327527918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.49:6840', 'nonce': 3327527918}, {'type': 'v1', 'addr': '172.21.15.49:6841', 'nonce': 3327527918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24404': {'gid': 24404, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6833/701158707', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6832', 'nonce': 701158707}, {'type': 'v1', 'addr': '172.21.15.106:6833', 'nonce': 701158707}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24421': {'gid': 24421, 'name': 'l', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6833/1442063907', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6832', 'nonce': 1442063907}, {'type': 'v1', 'addr': '172.21.15.107:6833', 'nonce': 1442063907}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.106:6835/353176461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6834', 'nonce': 353176461}, {'type': 'v1', 'addr': '172.21.15.106:6835', 'nonce': 353176461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'f', 'rank': 4, 'incarnation': 22, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.107:6835/863606455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 863606455}, {'type': 'v1', 'addr': '172.21.15.107:6835', 'nonce': 863606455}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24470': {'gid': 24470, 'name': 'h', 'rank': 3, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.106:6837/3542707642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6836', 'nonce': 3542707642}, {'type': 'v1', 'addr': '172.21.15.106:6837', 'nonce': 3542707642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.107:6837/3043451403', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 3043451403}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 3043451403}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24499': {'gid': 24499, 'name': 'i', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.107:6839/2909446244', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 2909446244}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 2909446244}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.106:6839/2088471086', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.106:6838', 'nonce': 2088471086}, {'type': 'v1', 'addr': '172.21.15.106:6839', 'nonce': 2088471086}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24506, 'qdb_cluster': [24499, 24506, 24481, 24470, 24463]} 2024-03-18T00:49:57.700 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T00:49:57.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-18T00:49:57.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.808+0000 7fa26a189640 1 -- 172.21.15.49:0/210378221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2641029f0 msgr2=0x7fa264102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.808+0000 7fa26a189640 1 --2- 172.21.15.49:0/210378221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2641029f0 0x7fa264102e70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa258009900 tx=0x7fa25802bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:57.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.808+0000 7fa26a189640 1 -- 172.21.15.49:0/210378221 shutdown_connections 2024-03-18T00:49:57.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.808+0000 7fa26a189640 1 --2- 172.21.15.49:0/210378221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2641033b0 0x7fa264109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.808+0000 7fa26a189640 1 --2- 172.21.15.49:0/210378221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2641029f0 0x7fa264102e70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.808+0000 7fa26a189640 1 --2- 172.21.15.49:0/210378221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa264101800 0x7fa264101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.808+0000 7fa26a189640 1 -- 172.21.15.49:0/210378221 >> 172.21.15.49:0/210378221 conn(0x7fa2640fcd70 msgr2=0x7fa2640ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.808+0000 7fa26a189640 1 -- 172.21.15.49:0/210378221 shutdown_connections 2024-03-18T00:49:57.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.808+0000 7fa26a189640 1 -- 172.21.15.49:0/210378221 wait complete. 2024-03-18T00:49:57.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa26a189640 1 Processor -- start 2024-03-18T00:49:57.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa26a189640 1 -- start start 2024-03-18T00:49:57.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa26a189640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa264101800 0x7fa264075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa26a189640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2641029f0 0x7fa264073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa26a189640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2641033b0 0x7fa264074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa26a189640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa26410d480 con 0x7fa2641033b0 2024-03-18T00:49:57.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa26a189640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa26410d300 con 0x7fa264101800 2024-03-18T00:49:57.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa26a189640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa26410d600 con 0x7fa2641029f0 2024-03-18T00:49:57.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa263fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2641033b0 0x7fa264074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa2637fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa264101800 0x7fa264075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa2637fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa264101800 0x7fa264075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:50552/0 (socket says 172.21.15.49:50552) 2024-03-18T00:49:57.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa2637fe640 1 -- 172.21.15.49:0/522761523 learned_addr learned my addr 172.21.15.49:0/522761523 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.809+0000 7fa263fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2641033b0 0x7fa264074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49640/0 (socket says 172.21.15.49:49640) 2024-03-18T00:49:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa262ffd640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2641029f0 0x7fa264073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa2637fe640 1 -- 172.21.15.49:0/522761523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2641029f0 msgr2=0x7fa264073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa2637fe640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2641029f0 0x7fa264073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa2637fe640 1 -- 172.21.15.49:0/522761523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2641033b0 msgr2=0x7fa264074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa2637fe640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2641033b0 0x7fa264074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa2637fe640 1 -- 172.21.15.49:0/522761523 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2640749a0 con 0x7fa264101800 2024-03-18T00:49:57.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa2637fe640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa264101800 0x7fa264075590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa25400e9c0 tx=0x7fa25400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa260ff9640 1 -- 172.21.15.49:0/522761523 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25400cd80 con 0x7fa264101800 2024-03-18T00:49:57.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa260ff9640 1 -- 172.21.15.49:0/522761523 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2540035f0 con 0x7fa264101800 2024-03-18T00:49:57.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa26a189640 1 -- 172.21.15.49:0/522761523 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2641a1c10 con 0x7fa264101800 2024-03-18T00:49:57.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa260ff9640 1 -- 172.21.15.49:0/522761523 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254010690 con 0x7fa264101800 2024-03-18T00:49:57.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.810+0000 7fa26a189640 1 -- 172.21.15.49:0/522761523 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2641a2130 con 0x7fa264101800 2024-03-18T00:49:57.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa260ff9640 1 -- 172.21.15.49:0/522761523 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa2540037a0 con 0x7fa264101800 2024-03-18T00:49:57.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa26cd7b740 1 Processor -- start 2024-03-18T00:49:57.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa26cd7b740 1 -- start start 2024-03-18T00:49:57.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e29bd0 0x55b143e29ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 0x55b143e2f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2f730 0x55b143e31bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b143d8dc80 con 0x55b143e2f730 2024-03-18T00:49:57.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b143d36aa0 con 0x55b143e29bd0 2024-03-18T00:49:57.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b143d89d20 con 0x55b143e2a5c0 2024-03-18T00:49:57.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.813+0000 7fa260ff9640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa24c0a2800 0x7fa24c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa262ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2f730 0x55b143e31bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa263fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e29bd0 0x55b143e29ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa262ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2f730 0x55b143e31bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49654/0 (socket says 172.21.15.49:49654) 2024-03-18T00:49:57.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa262ffd640 1 -- 172.21.15.49:0/1937888696 learned_addr learned my addr 172.21.15.49:0/1937888696 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa2637fe640 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 0x55b143e2f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa263fff640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa24c0a2800 0x7fa24c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa260ff9640 1 -- 172.21.15.49:0/522761523 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa2540cee70 con 0x7fa264101800 2024-03-18T00:49:57.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa262ffd640 1 -- 172.21.15.49:0/1937888696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 msgr2=0x55b143e2f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa262ffd640 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 0x55b143e2f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa262ffd640 1 -- 172.21.15.49:0/1937888696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e29bd0 msgr2=0x55b143e29ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa262ffd640 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e29bd0 0x55b143e29ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa262ffd640 1 -- 172.21.15.49:0/1937888696 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b143d58ff0 con 0x55b143e2f730 2024-03-18T00:49:57.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa263fff640 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e29bd0 0x55b143e29ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:57.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa2637fe640 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 0x55b143e2f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:49:57.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa262ffd640 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2f730 0x55b143e31bc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa258009900 tx=0x7fa258009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd99e00dcc385be server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa263fff640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa24c0a2800 0x7fa24c0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55b143e2ae20 tx=0x7fa25c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa249ffb640 1 -- 172.21.15.49:0/1937888696 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25802cb80 con 0x55b143e2f730 2024-03-18T00:49:57.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa249ffb640 1 -- 172.21.15.49:0/1937888696 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa258005040 con 0x55b143e2f730 2024-03-18T00:49:57.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.814+0000 7fa249ffb640 1 -- 172.21.15.49:0/1937888696 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa258004920 con 0x55b143e2f730 2024-03-18T00:49:57.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 -- 172.21.15.49:0/1937888696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2f730 msgr2=0x55b143e31bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2f730 0x55b143e31bc0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa258009900 tx=0x7fa258009560 comp rx=0 tx=0).stop 2024-03-18T00:49:57.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 -- 172.21.15.49:0/1937888696 shutdown_connections 2024-03-18T00:49:57.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2f730 0x55b143e31bc0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 0x55b143e2f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/1937888696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e29bd0 0x55b143e29ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 -- 172.21.15.49:0/1937888696 >> 172.21.15.49:0/1937888696 conn(0x55b143e25140 msgr2=0x55b143e25540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 -- 172.21.15.49:0/1937888696 shutdown_connections 2024-03-18T00:49:57.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 -- 172.21.15.49:0/1937888696 wait complete. 2024-03-18T00:49:57.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.815+0000 7fa26cd7b740 1 -- start start 2024-03-18T00:49:57.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa26cd7b740 1 Processor -- start 2024-03-18T00:49:57.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 0x55b143e33d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e34290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e3b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b143e32680 con 0x55b143e2a5c0 2024-03-18T00:49:57.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b143d49ef0 con 0x55b143e2f730 2024-03-18T00:49:57.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b143e32800 con 0x55b143e29bd0 2024-03-18T00:49:57.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa26cd7b740 10 client.0 _ll_register_callbacks cb 0x55b143d44dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:57.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa262ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e34290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa262ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e34290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49658/0 (socket says 172.21.15.49:49658) 2024-03-18T00:49:57.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa262ffd640 1 -- 172.21.15.49:0/2020177153 learned_addr learned my addr 172.21.15.49:0/2020177153 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa2637fe640 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 0x55b143e33d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa263fff640 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e3b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa262ffd640 1 -- 172.21.15.49:0/2020177153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 msgr2=0x55b143e33d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa262ffd640 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 0x55b143e33d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa262ffd640 1 -- 172.21.15.49:0/2020177153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 msgr2=0x55b143e3b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa262ffd640 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e3b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa262ffd640 1 -- 172.21.15.49:0/2020177153 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b143d56a30 con 0x55b143e2a5c0 2024-03-18T00:49:57.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa262ffd640 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e34290 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa25802c170 tx=0x7fa258005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bafc62ab33809f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.816+0000 7fa242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:57.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa242ffd640 1 -- 172.21.15.49:0/2020177153 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa258004ad0 con 0x55b143e2a5c0 2024-03-18T00:49:57.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa242ffd640 1 -- 172.21.15.49:0/2020177153 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa258004c70 con 0x55b143e2a5c0 2024-03-18T00:49:57.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2020177153 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa258005040 con 0x55b143e2a5c0 2024-03-18T00:49:57.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa242ffd640 1 -- 172.21.15.49:0/2020177153 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa258004e70 con 0x55b143e2a5c0 2024-03-18T00:49:57.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa26cd7b740 10 client.15249 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:57.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2020177153 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b143aa22d0 con 0x55b143e2a5c0 2024-03-18T00:49:57.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa242ffd640 1 -- 172.21.15.49:0/2020177153 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa258005040 con 0x55b143e2a5c0 2024-03-18T00:49:57.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa242ffd640 1 -- 172.21.15.49:0/2020177153 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fa258034440 con 0x55b143e2a5c0 2024-03-18T00:49:57.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa26cd7b740 10 client.15249 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:57.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa26cd7b740 10 client.15249 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T00:49:57.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa26cd7b740 20 client.15249 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:57.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b143e402f0 0x55b143e41760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa26cd7b740 4 client.15249 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:57.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2020177153 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b143d9b1c0 con 0x55b143e402f0 2024-03-18T00:49:57.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.817+0000 7fa2637fe640 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b143e402f0 0x55b143e41760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.818+0000 7fa242ffd640 10 client.15249 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:49:57.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.818+0000 7fa2637fe640 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b143e402f0 0x55b143e41760 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=91fc640d5a2a35b1 server_cookie=81bbf8dd30b15780 in_seq=0 out_seq=0 2024-03-18T00:49:57.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa242ffd640 1 -- 172.21.15.49:0/2020177153 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b143d9b1c0 con 0x55b143e402f0 2024-03-18T00:49:57.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa242ffd640 10 client.15249 handle_command_reply: tid=0 2024-03-18T00:49:57.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 client.15249 shutdown 2024-03-18T00:49:57.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2020177153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 msgr2=0x55b143e34290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e34290 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa25802c170 tx=0x7fa258005420 comp rx=0 tx=0).stop 2024-03-18T00:49:57.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2020177153 shutdown_connections 2024-03-18T00:49:57.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b143e402f0 0x55b143e41760 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e3b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e34290 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa242ffd640 0 client.15249 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:49:57.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2020177153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 0x55b143e33d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2020177153 >> 172.21.15.49:0/2020177153 conn(0x55b143e25140 msgr2=0x55b143e24380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2020177153 shutdown_connections 2024-03-18T00:49:57.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2020177153 wait complete. 2024-03-18T00:49:57.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.819+0000 7fa26cd7b740 20 client.15249 trim_cache size 0 max 16384 2024-03-18T00:49:57.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa26cd7b740 1 Processor -- start 2024-03-18T00:49:57.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa26cd7b740 1 -- start start 2024-03-18T00:49:57.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e29bd0 0x55b143e25520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 0x55b143e3b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e31b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b143e3faa0 con 0x55b143e29bd0 2024-03-18T00:49:57.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b143e40170 con 0x55b143e2f730 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b143e40840 con 0x55b143e2a5c0 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa2637fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e29bd0 0x55b143e25520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa2637fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e29bd0 0x55b143e25520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49660/0 (socket says 172.21.15.49:49660) 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa2637fe640 1 -- 172.21.15.49:0/2158511894 learned_addr learned my addr 172.21.15.49:0/2158511894 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa262ffd640 1 --2- 172.21.15.49:0/2158511894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 0x55b143e3b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.837+0000 7fa263fff640 1 --2- 172.21.15.49:0/2158511894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e31b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.838+0000 7fa2637fe640 1 -- 172.21.15.49:0/2158511894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 msgr2=0x55b143e3b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.838+0000 7fa2637fe640 1 --2- 172.21.15.49:0/2158511894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 0x55b143e3b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.838+0000 7fa2637fe640 1 -- 172.21.15.49:0/2158511894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 msgr2=0x55b143e31b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.838+0000 7fa2637fe640 1 --2- 172.21.15.49:0/2158511894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e31b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.838+0000 7fa2637fe640 1 -- 172.21.15.49:0/2158511894 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b143b6c6b0 con 0x55b143e29bd0 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.838+0000 7fa2637fe640 1 --2- 172.21.15.49:0/2158511894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e29bd0 0x55b143e25520 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa25400c6a0 tx=0x7fa254007b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1ae85e04b5586 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.838+0000 7fa2497fa640 1 -- 172.21.15.49:0/2158511894 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25409f940 con 0x55b143e29bd0 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.838+0000 7fa2497fa640 1 -- 172.21.15.49:0/2158511894 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2540a3f90 con 0x55b143e29bd0 2024-03-18T00:49:57.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.838+0000 7fa2497fa640 1 -- 172.21.15.49:0/2158511894 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa254097e60 con 0x55b143e29bd0 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2158511894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e29bd0 msgr2=0x55b143e25520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2158511894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e29bd0 0x55b143e25520 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa25400c6a0 tx=0x7fa254007b90 comp rx=0 tx=0).stop 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2158511894 shutdown_connections 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2158511894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e31b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2158511894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e2a5c0 0x55b143e3b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/2158511894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e29bd0 0x55b143e25520 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2158511894 >> 172.21.15.49:0/2158511894 conn(0x55b143e25140 msgr2=0x55b143e33d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2158511894 shutdown_connections 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 -- 172.21.15.49:0/2158511894 wait complete. 2024-03-18T00:49:57.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 -- start start 2024-03-18T00:49:57.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 Processor -- start 2024-03-18T00:49:57.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 0x55b143e367e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e36d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e39d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b143e415e0 con 0x55b143e2a5c0 2024-03-18T00:49:57.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b143e40f10 con 0x55b143e2f730 2024-03-18T00:49:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b143e41cb0 con 0x55b143e29bd0 2024-03-18T00:49:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa26cd7b740 10 client.0 _ll_register_callbacks cb 0x7fa258009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa262ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e36d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa262ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e36d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49664/0 (socket says 172.21.15.49:49664) 2024-03-18T00:49:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa262ffd640 1 -- 172.21.15.49:0/3423195184 learned_addr learned my addr 172.21.15.49:0/3423195184 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa2637fe640 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 0x55b143e367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.839+0000 7fa263fff640 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e39d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa262ffd640 1 -- 172.21.15.49:0/3423195184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 msgr2=0x55b143e367e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa262ffd640 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 0x55b143e367e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa262ffd640 1 -- 172.21.15.49:0/3423195184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 msgr2=0x55b143e39d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa262ffd640 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e39d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa262ffd640 1 -- 172.21.15.49:0/3423195184 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b143e3a330 con 0x55b143e2a5c0 2024-03-18T00:49:57.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa2637fe640 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 0x55b143e367e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:57.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa262ffd640 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e36d20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa258009900 tx=0x7fa25802c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8628e27733b6b08e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:57.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa242ffd640 1 -- 172.21.15.49:0/3423195184 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25805d060 con 0x55b143e2a5c0 2024-03-18T00:49:57.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa242ffd640 1 -- 172.21.15.49:0/3423195184 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa258005690 con 0x55b143e2a5c0 2024-03-18T00:49:57.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa26cd7b740 1 -- 172.21.15.49:0/3423195184 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2540a3f90 con 0x55b143e2a5c0 2024-03-18T00:49:57.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa242ffd640 1 -- 172.21.15.49:0/3423195184 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa258034650 con 0x55b143e2a5c0 2024-03-18T00:49:57.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa26cd7b740 10 client.15255 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:57.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa26cd7b740 1 -- 172.21.15.49:0/3423195184 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b143e3b630 con 0x55b143e2a5c0 2024-03-18T00:49:57.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.840+0000 7fa242ffd640 1 -- 172.21.15.49:0/3423195184 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2540a3f90 con 0x55b143e2a5c0 2024-03-18T00:49:57.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa242ffd640 1 -- 172.21.15.49:0/3423195184 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fa258004130 con 0x55b143e2a5c0 2024-03-18T00:49:57.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 10 client.15255 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:57.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 10 client.15255 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-18T00:49:57.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 20 client.15255 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:57.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b143eaa9e0 0x55b143ea96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 4 client.15255 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 1 -- 172.21.15.49:0/3423195184 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "status"}) -- 0x55b143d95510 con 0x55b143eaa9e0 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa2637fe640 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b143eaa9e0 0x55b143ea96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa2637fe640 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b143eaa9e0 0x55b143ea96f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86510a8a4bd3ff35 server_cookie=be0f70de573b4bf6 in_seq=0 out_seq=0 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa242ffd640 10 client.15255 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa242ffd640 1 -- 172.21.15.49:0/3423195184 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b143d95510 con 0x55b143eaa9e0 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa242ffd640 10 client.15255 handle_command_reply: tid=0 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 1 client.15255 shutdown 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 1 -- 172.21.15.49:0/3423195184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 msgr2=0x55b143e36d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e36d20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa258009900 tx=0x7fa25802c330 comp rx=0 tx=0).stop 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 1 -- 172.21.15.49:0/3423195184 shutdown_connections 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.841+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b143eaa9e0 0x55b143ea96f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b143e2f730 0x55b143e39d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b143e2a5c0 0x55b143e36d20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26cd7b740 1 --2- 172.21.15.49:0/3423195184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b143e29bd0 0x55b143e367e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa242ffd640 0 client.15255 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:49:57.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26cd7b740 1 -- 172.21.15.49:0/3423195184 >> 172.21.15.49:0/3423195184 conn(0x55b143e25140 msgr2=0x55b143e283a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26cd7b740 1 -- 172.21.15.49:0/3423195184 shutdown_connections 2024-03-18T00:49:57.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26cd7b740 1 -- 172.21.15.49:0/3423195184 wait complete. 2024-03-18T00:49:57.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26cd7b740 20 client.15255 trim_cache size 0 max 16384 2024-03-18T00:49:57.861 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 4, 2024-03-18T00:49:57.861 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24463, 2024-03-18T00:49:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 106.839480239, 2024-03-18T00:49:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:57.862 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 135.06467560600001 2024-03-18T00:49:57.862 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 -- 172.21.15.49:0/522761523 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa24c0a2800 msgr2=0x7fa24c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa24c0a2800 0x7fa24c0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x55b143e2ae20 tx=0x7fa25c009500 comp rx=0 tx=0).stop 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 -- 172.21.15.49:0/522761523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa264101800 msgr2=0x7fa264075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa264101800 0x7fa264075590 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa25400e9c0 tx=0x7fa25400ee80 comp rx=0 tx=0).stop 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 -- 172.21.15.49:0/522761523 shutdown_connections 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa24c0a2800 0x7fa24c0a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2641033b0 0x7fa264074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2641029f0 0x7fa264073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 --2- 172.21.15.49:0/522761523 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa264101800 0x7fa264075590 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 -- 172.21.15.49:0/522761523 >> 172.21.15.49:0/522761523 conn(0x7fa2640fcd70 msgr2=0x7fa264105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 -- 172.21.15.49:0/522761523 shutdown_connections 2024-03-18T00:49:57.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.842+0000 7fa26a189640 1 -- 172.21.15.49:0/522761523 wait complete. 2024-03-18T00:49:57.865 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T00:49:57.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.973+0000 7f76d8cc6640 1 -- 172.21.15.49:0/2308622997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 msgr2=0x7f76d4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.973+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/2308622997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 0x7f76d4100b20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009900 tx=0x7f76c402bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:57.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.973+0000 7f76d8cc6640 1 -- 172.21.15.49:0/2308622997 shutdown_connections 2024-03-18T00:49:57.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.973+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/2308622997 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d41022b0 0x7f76d4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.973+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/2308622997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76d41018f0 0x7f76d4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.973+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/2308622997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 0x7f76d4100b20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.973+0000 7f76d8cc6640 1 -- 172.21.15.49:0/2308622997 >> 172.21.15.49:0/2308622997 conn(0x7f76d40fbc90 msgr2=0x7f76d40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.973+0000 7f76d8cc6640 1 -- 172.21.15.49:0/2308622997 shutdown_connections 2024-03-18T00:49:57.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.973+0000 7f76d8cc6640 1 -- 172.21.15.49:0/2308622997 wait complete. 2024-03-18T00:49:57.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d8cc6640 1 Processor -- start 2024-03-18T00:49:57.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d8cc6640 1 -- start start 2024-03-18T00:49:57.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d8cc6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 0x7f76d4197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d8cc6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d41018f0 0x7f76d41980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d8cc6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76d41022b0 0x7f76d419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d8cc6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f76d410c370 con 0x7f76d4100700 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d8cc6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f76d410c1f0 con 0x7f76d41018f0 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d8cc6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f76d410c4f0 con 0x7f76d41022b0 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 0x7f76d4197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 0x7f76d4197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49682/0 (socket says 172.21.15.49:49682) 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d259c640 1 -- 172.21.15.49:0/4072861054 learned_addr learned my addr 172.21.15.49:0/4072861054 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d41018f0 0x7f76d41980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.974+0000 7f76d2d9d640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76d41022b0 0x7f76d419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76d259c640 1 -- 172.21.15.49:0/4072861054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76d41022b0 msgr2=0x7f76d419f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76d259c640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76d41022b0 0x7f76d419f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76d259c640 1 -- 172.21.15.49:0/4072861054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d41018f0 msgr2=0x7f76d41980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76d259c640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d41018f0 0x7f76d41980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76d259c640 1 -- 172.21.15.49:0/4072861054 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d419f790 con 0x7f76d4100700 2024-03-18T00:49:57.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d41018f0 0x7f76d41980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:57.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76d259c640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 0x7f76d4197b70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f76c4000c00 tx=0x7f76c402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76cbfff640 1 -- 172.21.15.49:0/4072861054 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4041070 con 0x7f76d4100700 2024-03-18T00:49:57.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76cbfff640 1 -- 172.21.15.49:0/4072861054 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c4004450 con 0x7f76d4100700 2024-03-18T00:49:57.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76d8cc6640 1 -- 172.21.15.49:0/4072861054 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d419fa20 con 0x7f76d4100700 2024-03-18T00:49:57.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76cbfff640 1 -- 172.21.15.49:0/4072861054 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4034440 con 0x7f76d4100700 2024-03-18T00:49:57.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.975+0000 7f76d8cc6640 1 -- 172.21.15.49:0/4072861054 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d419ff10 con 0x7f76d4100700 2024-03-18T00:49:57.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.977+0000 7f76cbfff640 1 -- 172.21.15.49:0/4072861054 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f76c4044070 con 0x7f76d4100700 2024-03-18T00:49:57.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76db994740 1 Processor -- start 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76db994740 1 -- start start 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76db994740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb748d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76db994740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb74924d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76db994740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 0x55afb7494fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76db994740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55afb73ecc80 con 0x55afb748d9d0 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76db994740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55afb7395aa0 con 0x55afb7492b40 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76db994740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55afb73e8d20 con 0x55afb748cfe0 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76d259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb74924d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76cbfff640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f76c00a2800 0x7f76c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76d259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb74924d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49700/0 (socket says 172.21.15.49:49700) 2024-03-18T00:49:57.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76d259c640 1 -- 172.21.15.49:0/4114643984 learned_addr learned my addr 172.21.15.49:0/4114643984 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 0x55afb7494fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76d2d9d640 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb748d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.978+0000 7f76d2d9d640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f76c00a2800 0x7f76c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76cbfff640 1 -- 172.21.15.49:0/4072861054 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f76c40f0540 con 0x7f76d4100700 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76c9d9b640 1 -- 172.21.15.49:0/4114643984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 msgr2=0x55afb748d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb748d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76c9d9b640 1 -- 172.21.15.49:0/4114643984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 msgr2=0x55afb74924d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb74924d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76c9d9b640 1 -- 172.21.15.49:0/4114643984 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afb73b7ff0 con 0x55afb7492b40 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76d259c640 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb74924d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76d2d9d640 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb748d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:57.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 0x55afb7494fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009fa0 tx=0x7f76bc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=638960701210deb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76d2d9d640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f76c00a2800 0x7f76c00a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55afb748e230 tx=0x7f76cc009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76c8d99640 1 -- 172.21.15.49:0/4114643984 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc004310 con 0x55afb7492b40 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76c8d99640 1 -- 172.21.15.49:0/4114643984 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76bc0044b0 con 0x55afb7492b40 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.979+0000 7f76c8d99640 1 -- 172.21.15.49:0/4114643984 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0035f0 con 0x55afb7492b40 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 -- 172.21.15.49:0/4114643984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 msgr2=0x55afb7494fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 0x55afb7494fd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f76bc009fa0 tx=0x7f76bc011810 comp rx=0 tx=0).stop 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 -- 172.21.15.49:0/4114643984 shutdown_connections 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 0x55afb7494fd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb74924d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 --2- 172.21.15.49:0/4114643984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb748d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 -- 172.21.15.49:0/4114643984 >> 172.21.15.49:0/4114643984 conn(0x55afb7484140 msgr2=0x55afb7484540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 -- 172.21.15.49:0/4114643984 shutdown_connections 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 -- 172.21.15.49:0/4114643984 wait complete. 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 -- start start 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 Processor -- start 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.980+0000 7f76db994740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb7497130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76db994740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb7497670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76db994740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 0x55afb749e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76db994740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55afb7495af0 con 0x55afb748d9d0 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76db994740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55afb73a8ef0 con 0x55afb7492b40 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76db994740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55afb7495c70 con 0x55afb748cfe0 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76db994740 10 client.0 _ll_register_callbacks cb 0x55afb73a3dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76c9d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb7497670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76c9d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb7497670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49712/0 (socket says 172.21.15.49:49712) 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76c9d9b640 1 -- 172.21.15.49:0/2107452793 learned_addr learned my addr 172.21.15.49:0/2107452793 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76d259c640 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb7497130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76d2d9d640 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 0x55afb749e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76c9d9b640 1 -- 172.21.15.49:0/2107452793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 msgr2=0x55afb7497130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb7497130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76c9d9b640 1 -- 172.21.15.49:0/2107452793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 msgr2=0x55afb749e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 0x55afb749e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76c9d9b640 1 -- 172.21.15.49:0/2107452793 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afb73b5a30 con 0x55afb748d9d0 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76d259c640 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb7497130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb7497670 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f76bc004970 tx=0x7f76bc0049a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5def3c1c12eb5cb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:57.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76b9ffb640 1 -- 172.21.15.49:0/2107452793 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc028020 con 0x55afb748d9d0 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76db994740 1 -- 172.21.15.49:0/2107452793 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76bc0044b0 con 0x55afb748d9d0 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76b9ffb640 1 -- 172.21.15.49:0/2107452793 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f76bc005240 con 0x55afb748d9d0 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76b9ffb640 1 -- 172.21.15.49:0/2107452793 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f76bc033050 con 0x55afb748d9d0 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.981+0000 7f76db994740 10 client.15267 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76db994740 1 -- 172.21.15.49:0/2107452793 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55afb71012d0 con 0x55afb748d9d0 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76b9ffb640 1 -- 172.21.15.49:0/2107452793 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76bc0044b0 con 0x55afb748d9d0 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76b9ffb640 1 -- 172.21.15.49:0/2107452793 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f76bc018070 con 0x55afb748d9d0 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76db994740 10 client.15267 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76db994740 10 client.15267 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76db994740 20 client.15267 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76db994740 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55afb74a38f0 0x55afb74a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76db994740 4 client.15267 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:57.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76db994740 1 -- 172.21.15.49:0/2107452793 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afb73fa1c0 con 0x55afb74a38f0 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76d259c640 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55afb74a38f0 0x55afb74a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.982+0000 7f76d259c640 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55afb74a38f0 0x55afb74a4d60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1de5c09ab5f95e20 server_cookie=2bd9730cf2bc9fb in_seq=0 out_seq=0 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76b9ffb640 10 client.15267 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76b9ffb640 1 -- 172.21.15.49:0/2107452793 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55afb73fa1c0 con 0x55afb74a38f0 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76b9ffb640 10 client.15267 handle_command_reply: tid=0 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 client.15267 shutdown 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 -- 172.21.15.49:0/2107452793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 msgr2=0x55afb7497670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb7497670 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f76bc004970 tx=0x7f76bc0049a0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 -- 172.21.15.49:0/2107452793 shutdown_connections 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55afb74a38f0 0x55afb74a4d60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb7492b40 0x55afb749e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748d9d0 0x55afb7497670 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76b9ffb640 0 client.15267 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:49:57.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 --2- 172.21.15.49:0/2107452793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb748cfe0 0x55afb7497130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:57.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 -- 172.21.15.49:0/2107452793 >> 172.21.15.49:0/2107452793 conn(0x55afb7484140 msgr2=0x55afb7483380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:57.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 -- 172.21.15.49:0/2107452793 shutdown_connections 2024-03-18T00:49:57.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.983+0000 7f76db994740 1 -- 172.21.15.49:0/2107452793 wait complete. 2024-03-18T00:49:57.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:57.984+0000 7f76db994740 20 client.15267 trim_cache size 0 max 16384 2024-03-18T00:49:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76db994740 1 Processor -- start 2024-03-18T00:49:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76db994740 1 -- start start 2024-03-18T00:49:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76db994740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb74975d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76db994740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 0x55afb749e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76db994740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 0x55afb7496040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76db994740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55afb74a30a0 con 0x55afb748cfe0 2024-03-18T00:49:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76db994740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55afb74a3770 con 0x55afb748d9d0 2024-03-18T00:49:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76db994740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55afb74a3e40 con 0x55afb7492b40 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb74975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb74975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49714/0 (socket says 172.21.15.49:49714) 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d259c640 1 -- 172.21.15.49:0/2382707084 learned_addr learned my addr 172.21.15.49:0/2382707084 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/2382707084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 0x55afb749e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d2d9d640 1 --2- 172.21.15.49:0/2382707084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 0x55afb7496040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d259c640 1 -- 172.21.15.49:0/2382707084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 msgr2=0x55afb7496040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d259c640 1 --2- 172.21.15.49:0/2382707084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 0x55afb7496040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d259c640 1 -- 172.21.15.49:0/2382707084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 msgr2=0x55afb749e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d259c640 1 --2- 172.21.15.49:0/2382707084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 0x55afb749e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d259c640 1 -- 172.21.15.49:0/2382707084 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afb71cb6b0 con 0x55afb748cfe0 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76d259c640 1 --2- 172.21.15.49:0/2382707084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb74975d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f76c4005ab0 tx=0x7f76c40edef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb3c0767175a93e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76bbfff640 1 -- 172.21.15.49:0/2382707084 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c40bcb40 con 0x55afb748cfe0 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.002+0000 7f76bbfff640 1 -- 172.21.15.49:0/2382707084 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c4004c80 con 0x55afb748cfe0 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76bbfff640 1 -- 172.21.15.49:0/2382707084 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c40bc960 con 0x55afb748cfe0 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 -- 172.21.15.49:0/2382707084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 msgr2=0x55afb74975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 --2- 172.21.15.49:0/2382707084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb74975d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f76c4005ab0 tx=0x7f76c40edef0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 -- 172.21.15.49:0/2382707084 shutdown_connections 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 --2- 172.21.15.49:0/2382707084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 0x55afb749e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 --2- 172.21.15.49:0/2382707084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 0x55afb7496040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 --2- 172.21.15.49:0/2382707084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb74975d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 -- 172.21.15.49:0/2382707084 >> 172.21.15.49:0/2382707084 conn(0x55afb7484140 msgr2=0x55afb747f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 -- 172.21.15.49:0/2382707084 shutdown_connections 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 -- 172.21.15.49:0/2382707084 wait complete. 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 -- start start 2024-03-18T00:49:58.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.003+0000 7f76db994740 1 Processor -- start 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb7490040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 0x55afb749e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 0x55afb74905c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55afb74a4be0 con 0x55afb748cfe0 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55afb74a4510 con 0x55afb748d9d0 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55afb74a52b0 con 0x55afb7492b40 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 10 client.0 _ll_register_callbacks cb 0x7f76bc011f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb7490040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb7490040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49726/0 (socket says 172.21.15.49:49726) 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d259c640 1 -- 172.21.15.49:0/1813863089 learned_addr learned my addr 172.21.15.49:0/1813863089 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 0x55afb749e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d2d9d640 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 0x55afb74905c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d259c640 1 -- 172.21.15.49:0/1813863089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 msgr2=0x55afb74905c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d259c640 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 0x55afb74905c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d259c640 1 -- 172.21.15.49:0/1813863089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 msgr2=0x55afb749e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d259c640 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 0x55afb749e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d259c640 1 -- 172.21.15.49:0/1813863089 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afb7490df0 con 0x55afb748cfe0 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76d259c640 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb7490040 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f76c402e040 tx=0x7f76c4005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3846e19a89f0032f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76b9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76b9ffb640 1 -- 172.21.15.49:0/1813863089 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c40d9020 con 0x55afb748cfe0 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76b9ffb640 1 -- 172.21.15.49:0/1813863089 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f76c40035f0 con 0x55afb748cfe0 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 1 -- 172.21.15.49:0/1813863089 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76c4004c80 con 0x55afb748cfe0 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76b9ffb640 1 -- 172.21.15.49:0/1813863089 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f76c404b070 con 0x55afb748cfe0 2024-03-18T00:49:58.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 10 client.15273 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.004+0000 7f76db994740 1 -- 172.21.15.49:0/1813863089 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55afb75086f0 con 0x55afb748cfe0 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76b9ffb640 1 -- 172.21.15.49:0/1813863089 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c4004c80 con 0x55afb748cfe0 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76b9ffb640 1 -- 172.21.15.49:0/1813863089 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f76c40c0040 con 0x55afb748cfe0 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76db994740 10 client.15273 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76db994740 10 client.15273 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76db994740 20 client.15273 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76db994740 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55afb750d1b0 0x55afb750bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76db994740 4 client.15273 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76db994740 1 -- 172.21.15.49:0/1813863089 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "status"}) -- 0x55afb73f4510 con 0x55afb750d1b0 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55afb750d1b0 0x55afb750bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76c9d9b640 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55afb750d1b0 0x55afb750bec0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=916908b6c7d48a99 server_cookie=613196b2f8749309 in_seq=0 out_seq=0 2024-03-18T00:49:58.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.005+0000 7f76b9ffb640 10 client.15273 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76b9ffb640 1 -- 172.21.15.49:0/1813863089 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55afb73f4510 con 0x55afb750d1b0 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76b9ffb640 10 client.15273 handle_command_reply: tid=0 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 client.15273 shutdown 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 -- 172.21.15.49:0/1813863089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 msgr2=0x55afb7490040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb7490040 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f76c402e040 tx=0x7f76c4005670 comp rx=0 tx=0).stop 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 -- 172.21.15.49:0/1813863089 shutdown_connections 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55afb750d1b0 0x55afb750bec0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afb748d9d0 0x55afb749e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afb7492b40 0x55afb74905c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76b9ffb640 0 client.15273 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 --2- 172.21.15.49:0/1813863089 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afb748cfe0 0x55afb7490040 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 -- 172.21.15.49:0/1813863089 >> 172.21.15.49:0/1813863089 conn(0x55afb7484140 msgr2=0x55afb7496040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 -- 172.21.15.49:0/1813863089 shutdown_connections 2024-03-18T00:49:58.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 1 -- 172.21.15.49:0/1813863089 wait complete. 2024-03-18T00:49:58.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76db994740 20 client.15273 trim_cache size 0 max 16384 2024-03-18T00:49:58.016 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:58.016 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:58.016 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 3, 2024-03-18T00:49:58.016 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24470, 2024-03-18T00:49:58.016 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:58.016 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:58.017 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:58.017 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 108.01062031799999, 2024-03-18T00:49:58.017 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:58.017 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:58.017 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:58.017 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 129.37992313800001 2024-03-18T00:49:58.017 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:58.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76d8cc6640 1 -- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f76c00a2800 msgr2=0x7f76c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f76c00a2800 0x7f76c00a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x55afb748e230 tx=0x7f76cc009500 comp rx=0 tx=0).stop 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76d8cc6640 1 -- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 msgr2=0x7f76d4197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.006+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 0x7f76d4197b70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f76c4000c00 tx=0x7f76c402c640 comp rx=0 tx=0).stop 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.007+0000 7f76d8cc6640 1 -- 172.21.15.49:0/4072861054 shutdown_connections 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.007+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f76c00a2800 0x7f76c00a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.007+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f76d41018f0 0x7f76d41980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.007+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76d41022b0 0x7f76d419f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.007+0000 7f76d8cc6640 1 --2- 172.21.15.49:0/4072861054 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f76d4100700 0x7f76d4197b70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.007+0000 7f76d8cc6640 1 -- 172.21.15.49:0/4072861054 >> 172.21.15.49:0/4072861054 conn(0x7f76d40fbc90 msgr2=0x7f76d4069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.007+0000 7f76d8cc6640 1 -- 172.21.15.49:0/4072861054 shutdown_connections 2024-03-18T00:49:58.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.007+0000 7f76d8cc6640 1 -- 172.21.15.49:0/4072861054 wait complete. 2024-03-18T00:49:58.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T00:49:58.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 -- 172.21.15.49:0/3222715275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21381022b0 msgr2=0x7f2138108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 --2- 172.21.15.49:0/3222715275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21381022b0 0x7f2138108b40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2130009ca0 tx=0x7f213002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 -- 172.21.15.49:0/3222715275 shutdown_connections 2024-03-18T00:49:58.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 --2- 172.21.15.49:0/3222715275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21381018f0 0x7f2138101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 --2- 172.21.15.49:0/3222715275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f21381022b0 0x7f2138108b40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 --2- 172.21.15.49:0/3222715275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2138100700 0x7f2138100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 -- 172.21.15.49:0/3222715275 >> 172.21.15.49:0/3222715275 conn(0x7f21380fbc90 msgr2=0x7f21380fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 -- 172.21.15.49:0/3222715275 shutdown_connections 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 -- 172.21.15.49:0/3222715275 wait complete. 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 Processor -- start 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 -- start start 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2138100700 0x7f213806a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21381018f0 0x7f213819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.128+0000 7f213fb48640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21381022b0 0x7f213819ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213fb48640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f213810c380 con 0x7f2138100700 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213fb48640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f213810c200 con 0x7f21381022b0 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213fb48640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f213810c500 con 0x7f21381018f0 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d8e4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2138100700 0x7f213806a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d8e4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2138100700 0x7f213806a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49746/0 (socket says 172.21.15.49:49746) 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d8e4640 1 -- 172.21.15.49:0/1362669376 learned_addr learned my addr 172.21.15.49:0/1362669376 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d0e3640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21381018f0 0x7f213819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213e0e5640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21381022b0 0x7f213819ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d8e4640 1 -- 172.21.15.49:0/1362669376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21381018f0 msgr2=0x7f213819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d8e4640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21381018f0 0x7f213819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d8e4640 1 -- 172.21.15.49:0/1362669376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21381022b0 msgr2=0x7f213819ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d8e4640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21381022b0 0x7f213819ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d8e4640 1 -- 172.21.15.49:0/1362669376 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f213819f6c0 con 0x7f2138100700 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213d8e4640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2138100700 0x7f213806a940 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f212c00e9c0 tx=0x7f212c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f212affd640 1 -- 172.21.15.49:0/1362669376 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c00cd80 con 0x7f2138100700 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f212affd640 1 -- 172.21.15.49:0/1362669376 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f212c0035f0 con 0x7f2138100700 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213fb48640 1 -- 172.21.15.49:0/1362669376 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f213819f9b0 con 0x7f2138100700 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f212affd640 1 -- 172.21.15.49:0/1362669376 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c010690 con 0x7f2138100700 2024-03-18T00:49:58.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.129+0000 7f213fb48640 1 -- 172.21.15.49:0/1362669376 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f213819feb0 con 0x7f2138100700 2024-03-18T00:49:58.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.132+0000 7f212affd640 1 -- 172.21.15.49:0/1362669376 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f212c0037a0 con 0x7f2138100700 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.132+0000 7f214280c740 1 Processor -- start 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.132+0000 7f214280c740 1 -- start start 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.132+0000 7f214280c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d8bbd0 0x55ae82d8bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.132+0000 7f214280c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 0x55ae82d910c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.132+0000 7f214280c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d91730 0x55ae82d93bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.132+0000 7f214280c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ae82cefc80 con 0x55ae82d91730 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.132+0000 7f214280c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ae82c98aa0 con 0x55ae82d8c5c0 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.132+0000 7f214280c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ae82cebd20 con 0x55ae82d8bbd0 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f212affd640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21240a2800 0x7f21240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d8bbd0 0x55ae82d8bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213d0e3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d91730 0x55ae82d93bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d8bbd0 0x55ae82d8bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:54416/0 (socket says 172.21.15.49:54416) 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 -- 172.21.15.49:0/3767454725 learned_addr learned my addr 172.21.15.49:0/3767454725 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213d8e4640 1 --2- 172.21.15.49:0/3767454725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 0x55ae82d910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21240a2800 0x7f21240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 -- 172.21.15.49:0/3767454725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 msgr2=0x55ae82d910c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 --2- 172.21.15.49:0/3767454725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 0x55ae82d910c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 -- 172.21.15.49:0/3767454725 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d91730 msgr2=0x55ae82d93bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f212affd640 1 -- 172.21.15.49:0/1362669376 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f212c0cee90 con 0x7f2138100700 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 --2- 172.21.15.49:0/3767454725 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d91730 0x55ae82d93bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 -- 172.21.15.49:0/3767454725 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae82cbaff0 con 0x55ae82d8bbd0 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213d0e3640 1 --2- 172.21.15.49:0/3767454725 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d91730 0x55ae82d93bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 --2- 172.21.15.49:0/3767454725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d8bbd0 0x55ae82d8bff0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2130005e30 tx=0x7f2130032c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf1dd741a3ca225f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f213e0e5640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21240a2800 0x7f21240a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2130046000 tx=0x7f2130048000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f2123fff640 1 -- 172.21.15.49:0/3767454725 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2130039040 con 0x55ae82d8bbd0 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f2123fff640 1 -- 172.21.15.49:0/3767454725 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2130003b70 con 0x55ae82d8bbd0 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.133+0000 7f2123fff640 1 -- 172.21.15.49:0/3767454725 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2130043660 con 0x55ae82d8bbd0 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.134+0000 7f214280c740 1 -- 172.21.15.49:0/3767454725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d8bbd0 msgr2=0x55ae82d8bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.134+0000 7f214280c740 1 --2- 172.21.15.49:0/3767454725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d8bbd0 0x55ae82d8bff0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2130005e30 tx=0x7f2130032c90 comp rx=0 tx=0).stop 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.134+0000 7f214280c740 1 -- 172.21.15.49:0/3767454725 shutdown_connections 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.134+0000 7f214280c740 1 --2- 172.21.15.49:0/3767454725 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 0x55ae82d910c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.134+0000 7f214280c740 1 --2- 172.21.15.49:0/3767454725 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d91730 0x55ae82d93bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.134+0000 7f214280c740 1 --2- 172.21.15.49:0/3767454725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d8bbd0 0x55ae82d8bff0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.134+0000 7f214280c740 1 -- 172.21.15.49:0/3767454725 >> 172.21.15.49:0/3767454725 conn(0x55ae82d87140 msgr2=0x55ae82d87540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.134+0000 7f214280c740 1 -- 172.21.15.49:0/3767454725 shutdown_connections 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.134+0000 7f214280c740 1 -- 172.21.15.49:0/3767454725 wait complete. 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f214280c740 1 -- start start 2024-03-18T00:49:58.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f214280c740 1 Processor -- start 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f214280c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 0x55ae82d95de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f214280c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82d96320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f214280c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82d9d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f214280c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ae82d94680 con 0x55ae82d8c5c0 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f214280c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ae82cabef0 con 0x55ae82d8bbd0 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f214280c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ae82d94800 con 0x55ae82d91730 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f214280c740 10 client.0 _ll_register_callbacks cb 0x55ae82ca6dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d0e3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82d96320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d0e3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82d96320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49762/0 (socket says 172.21.15.49:49762) 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d0e3640 1 -- 172.21.15.49:0/2682508052 learned_addr learned my addr 172.21.15.49:0/2682508052 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d8e4640 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 0x55ae82d95de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213e0e5640 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82d9d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d0e3640 1 -- 172.21.15.49:0/2682508052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 msgr2=0x55ae82d9d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d0e3640 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82d9d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d0e3640 1 -- 172.21.15.49:0/2682508052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 msgr2=0x55ae82d95de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d0e3640 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 0x55ae82d95de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d0e3640 1 -- 172.21.15.49:0/2682508052 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae82cb8a30 con 0x55ae82d8c5c0 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.135+0000 7f213d8e4640 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 0x55ae82d95de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f213d0e3640 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82d96320 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2134012720 tx=0x7f2134012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8575af52ccbe39c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f21217fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f21217fa640 1 -- 172.21.15.49:0/2682508052 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2134004030 con 0x55ae82d8c5c0 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f21217fa640 1 -- 172.21.15.49:0/2682508052 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f21340041d0 con 0x55ae82d8c5c0 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f214280c740 1 -- 172.21.15.49:0/2682508052 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2130003b70 con 0x55ae82d8c5c0 2024-03-18T00:49:58.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f21217fa640 1 -- 172.21.15.49:0/2682508052 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2134004410 con 0x55ae82d8c5c0 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f214280c740 10 client.15285 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f214280c740 1 -- 172.21.15.49:0/2682508052 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ae82a042d0 con 0x55ae82d8c5c0 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f21217fa640 1 -- 172.21.15.49:0/2682508052 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2130003b70 con 0x55ae82d8c5c0 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f21217fa640 1 -- 172.21.15.49:0/2682508052 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f2134003ba0 con 0x55ae82d8c5c0 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f214280c740 10 client.15285 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f214280c740 10 client.15285 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f214280c740 20 client.15285 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f214280c740 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55ae82da2920 0x55ae82da3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f214280c740 4 client.15285 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.136+0000 7f214280c740 1 -- 172.21.15.49:0/2682508052 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae82cfd1c0 con 0x55ae82da2920 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.137+0000 7f213d8e4640 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55ae82da2920 0x55ae82da3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.137+0000 7f21217fa640 10 client.15285 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.137+0000 7f213d8e4640 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55ae82da2920 0x55ae82da3d90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e7d9ebc788cb5b1 server_cookie=bd3438ceea2d281a in_seq=0 out_seq=0 2024-03-18T00:49:58.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f21217fa640 1 -- 172.21.15.49:0/2682508052 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ae82cfd1c0 con 0x55ae82da2920 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f21217fa640 10 client.15285 handle_command_reply: tid=0 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 client.15285 shutdown 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 -- 172.21.15.49:0/2682508052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 msgr2=0x55ae82d96320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82d96320 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2134012720 tx=0x7f2134012be0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 -- 172.21.15.49:0/2682508052 shutdown_connections 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55ae82da2920 0x55ae82da3d90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82d9d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82d96320 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 --2- 172.21.15.49:0/2682508052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 0x55ae82d95de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f21217fa640 0 client.15285 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 -- 172.21.15.49:0/2682508052 >> 172.21.15.49:0/2682508052 conn(0x55ae82d87140 msgr2=0x55ae82d86380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 -- 172.21.15.49:0/2682508052 shutdown_connections 2024-03-18T00:49:58.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 1 -- 172.21.15.49:0/2682508052 wait complete. 2024-03-18T00:49:58.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.138+0000 7f214280c740 20 client.15285 trim_cache size 0 max 16384 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.156+0000 7f214280c740 1 Processor -- start 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.156+0000 7f214280c740 1 -- start start 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f214280c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8bbd0 0x55ae82d959f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f214280c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 0x55ae82d95f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f214280c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82d85af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f214280c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ae82da20d0 con 0x55ae82d8bbd0 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f214280c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ae82da27a0 con 0x55ae82d8c5c0 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f214280c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ae82da2e70 con 0x55ae82d91730 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d8e4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8bbd0 0x55ae82d959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d8e4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8bbd0 0x55ae82d959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49778/0 (socket says 172.21.15.49:49778) 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d8e4640 1 -- 172.21.15.49:0/603696350 learned_addr learned my addr 172.21.15.49:0/603696350 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213e0e5640 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82d85af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d0e3640 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 0x55ae82d95f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d8e4640 1 -- 172.21.15.49:0/603696350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 msgr2=0x55ae82d85af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d8e4640 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82d85af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d8e4640 1 -- 172.21.15.49:0/603696350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 msgr2=0x55ae82d95f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d8e4640 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 0x55ae82d95f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d8e4640 1 -- 172.21.15.49:0/603696350 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae82ace6b0 con 0x55ae82d8bbd0 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d0e3640 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 0x55ae82d95f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f213d8e4640 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8bbd0 0x55ae82d959f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f212c0981e0 tx=0x7f212c0af040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6306a22704e68218 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f21237fe640 1 -- 172.21.15.49:0/603696350 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c09bf10 con 0x55ae82d8bbd0 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f21237fe640 1 -- 172.21.15.49:0/603696350 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f212c09b810 con 0x55ae82d8bbd0 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.157+0000 7f21237fe640 1 -- 172.21.15.49:0/603696350 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c0860b0 con 0x55ae82d8bbd0 2024-03-18T00:49:58.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 -- 172.21.15.49:0/603696350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8bbd0 msgr2=0x55ae82d959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8bbd0 0x55ae82d959f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f212c0981e0 tx=0x7f212c0af040 comp rx=0 tx=0).stop 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 -- 172.21.15.49:0/603696350 shutdown_connections 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8c5c0 0x55ae82d95f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82d85af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 --2- 172.21.15.49:0/603696350 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8bbd0 0x55ae82d959f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 -- 172.21.15.49:0/603696350 >> 172.21.15.49:0/603696350 conn(0x55ae82d87140 msgr2=0x55ae82d90e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 -- 172.21.15.49:0/603696350 shutdown_connections 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 -- 172.21.15.49:0/603696350 wait complete. 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 -- start start 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.158+0000 7f214280c740 1 Processor -- start 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 0x55ae82e06680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82e08bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82e0b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ae82da3c10 con 0x55ae82d8c5c0 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ae82da3540 con 0x55ae82d8bbd0 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ae82da42e0 con 0x55ae82d91730 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 10 client.0 _ll_register_callbacks cb 0x7f2134009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d0e3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82e08bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d0e3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82e08bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49792/0 (socket says 172.21.15.49:49792) 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d0e3640 1 -- 172.21.15.49:0/487148852 learned_addr learned my addr 172.21.15.49:0/487148852 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213e0e5640 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82e0b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d8e4640 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 0x55ae82e06680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d0e3640 1 -- 172.21.15.49:0/487148852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 msgr2=0x55ae82e0b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d0e3640 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82e0b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d0e3640 1 -- 172.21.15.49:0/487148852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 msgr2=0x55ae82e06680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d0e3640 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 0x55ae82e06680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d0e3640 1 -- 172.21.15.49:0/487148852 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae82e0b900 con 0x55ae82d8c5c0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f213d0e3640 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82e08bd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f21340126f0 tx=0x7f2134010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c7d1b2941117bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f21217fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f21217fa640 1 -- 172.21.15.49:0/487148852 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2134005310 con 0x55ae82d8c5c0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f21217fa640 1 -- 172.21.15.49:0/487148852 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f21340054b0 con 0x55ae82d8c5c0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 1 -- 172.21.15.49:0/487148852 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f212c09b810 con 0x55ae82d8c5c0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f21217fa640 1 -- 172.21.15.49:0/487148852 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2134036070 con 0x55ae82d8c5c0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 10 client.15291 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.159+0000 7f214280c740 1 -- 172.21.15.49:0/487148852 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ae82e0cc70 con 0x55ae82d8c5c0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f21217fa640 1 -- 172.21.15.49:0/487148852 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f212c09b810 con 0x55ae82d8c5c0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f21217fa640 1 -- 172.21.15.49:0/487148852 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f2134004740 con 0x55ae82d8c5c0 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f214280c740 10 client.15291 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:58.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f214280c740 10 client.15291 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f214280c740 20 client.15291 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f214280c740 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55ae82e11860 0x55ae82e10570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f214280c740 4 client.15291 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f214280c740 1 -- 172.21.15.49:0/487148852 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "status"}) -- 0x55ae82cf7510 con 0x55ae82e11860 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f213d8e4640 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55ae82e11860 0x55ae82e10570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f213d8e4640 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55ae82e11860 0x55ae82e10570 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2fedc8758850c6ed server_cookie=7a53eb91cdcf93ed in_seq=0 out_seq=0 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.160+0000 7f21217fa640 10 client.15291 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f21217fa640 1 -- 172.21.15.49:0/487148852 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ae82cf7510 con 0x55ae82e11860 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f21217fa640 10 client.15291 handle_command_reply: tid=0 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 client.15291 shutdown 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 -- 172.21.15.49:0/487148852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 msgr2=0x55ae82e08bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82e08bd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f21340126f0 tx=0x7f2134010d70 comp rx=0 tx=0).stop 2024-03-18T00:49:58.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 -- 172.21.15.49:0/487148852 shutdown_connections 2024-03-18T00:49:58.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55ae82e11860 0x55ae82e10570 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ae82d91730 0x55ae82e0b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f21217fa640 0 client.15291 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:49:58.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ae82d8c5c0 0x55ae82e08bd0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 --2- 172.21.15.49:0/487148852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ae82d8bbd0 0x55ae82e06680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 -- 172.21.15.49:0/487148852 >> 172.21.15.49:0/487148852 conn(0x55ae82d87140 msgr2=0x55ae82d959f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 -- 172.21.15.49:0/487148852 shutdown_connections 2024-03-18T00:49:58.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 1 -- 172.21.15.49:0/487148852 wait complete. 2024-03-18T00:49:58.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f214280c740 20 client.15291 trim_cache size 0 max 16384 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 2, 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24481, 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 109.174625369, 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 132.58597837900001 2024-03-18T00:49:58.171 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:58.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f213fb48640 1 -- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21240a2800 msgr2=0x7f21240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f213fb48640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21240a2800 0x7f21240a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2130046000 tx=0x7f2130048000 comp rx=0 tx=0).stop 2024-03-18T00:49:58.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f213fb48640 1 -- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2138100700 msgr2=0x7f213806a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.161+0000 7f213fb48640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2138100700 0x7f213806a940 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f212c00e9c0 tx=0x7f212c00ee80 comp rx=0 tx=0).stop 2024-03-18T00:49:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.162+0000 7f213fb48640 1 -- 172.21.15.49:0/1362669376 shutdown_connections 2024-03-18T00:49:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.162+0000 7f213fb48640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f21240a2800 0x7f21240a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.162+0000 7f213fb48640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f21381022b0 0x7f213819ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.162+0000 7f213fb48640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f21381018f0 0x7f213819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.162+0000 7f213fb48640 1 --2- 172.21.15.49:0/1362669376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2138100700 0x7f213806a940 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.162+0000 7f213fb48640 1 -- 172.21.15.49:0/1362669376 >> 172.21.15.49:0/1362669376 conn(0x7f21380fbc90 msgr2=0x7f2138104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.162+0000 7f213fb48640 1 -- 172.21.15.49:0/1362669376 shutdown_connections 2024-03-18T00:49:58.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.162+0000 7f213fb48640 1 -- 172.21.15.49:0/1362669376 wait complete. 2024-03-18T00:49:58.174 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T00:49:58.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.284+0000 7f53b2ef1640 1 -- 172.21.15.49:0/711731956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac100700 msgr2=0x7f53ac100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.284+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/711731956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac100700 0x7f53ac100b20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009900 tx=0x7f53a002bc30 comp rx=0 tx=0).stop 2024-03-18T00:49:58.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.284+0000 7f53b2ef1640 1 -- 172.21.15.49:0/711731956 shutdown_connections 2024-03-18T00:49:58.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.284+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/711731956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac1022b0 0x7f53ac108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.284+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/711731956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ac1018f0 0x7f53ac101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.284+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/711731956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac100700 0x7f53ac100b20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.284+0000 7f53b2ef1640 1 -- 172.21.15.49:0/711731956 >> 172.21.15.49:0/711731956 conn(0x7f53ac0fbc90 msgr2=0x7f53ac0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.284+0000 7f53b2ef1640 1 -- 172.21.15.49:0/711731956 shutdown_connections 2024-03-18T00:49:58.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.284+0000 7f53b2ef1640 1 -- 172.21.15.49:0/711731956 wait complete. 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b2ef1640 1 Processor -- start 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b2ef1640 1 -- start start 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b2ef1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ac100700 0x7f53ac199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b2ef1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac1018f0 0x7f53ac19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b2ef1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac1022b0 0x7f53ac1a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b2ef1640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac10c350 con 0x7f53ac1022b0 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b2ef1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac10c1d0 con 0x7f53ac1018f0 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b2ef1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac10c4d0 con 0x7f53ac100700 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b148e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac1022b0 0x7f53ac1a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b148e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac1022b0 0x7f53ac1a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49820/0 (socket says 172.21.15.49:49820) 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b148e640 1 -- 172.21.15.49:0/326105276 learned_addr learned my addr 172.21.15.49:0/326105276 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ac100700 0x7f53ac199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.285+0000 7f53abfff640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac1018f0 0x7f53ac19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.286+0000 7f53b148e640 1 -- 172.21.15.49:0/326105276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ac100700 msgr2=0x7f53ac199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.286+0000 7f53b148e640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ac100700 0x7f53ac199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.286+0000 7f53b148e640 1 -- 172.21.15.49:0/326105276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac1018f0 msgr2=0x7f53ac19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.286+0000 7f53b148e640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac1018f0 0x7f53ac19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.286+0000 7f53b148e640 1 -- 172.21.15.49:0/326105276 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac1a1960 con 0x7f53ac1022b0 2024-03-18T00:49:58.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.286+0000 7f53b148e640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac1022b0 0x7f53ac1a12f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f53a400eb10 tx=0x7f53a400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.287+0000 7f53a9ffb640 1 -- 172.21.15.49:0/326105276 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4004b20 con 0x7f53ac1022b0 2024-03-18T00:49:58.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.287+0000 7f53a9ffb640 1 -- 172.21.15.49:0/326105276 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a4004cc0 con 0x7f53ac1022b0 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.287+0000 7f53b2ef1640 1 -- 172.21.15.49:0/326105276 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac1a1c50 con 0x7f53ac1022b0 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.287+0000 7f53a9ffb640 1 -- 172.21.15.49:0/326105276 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4010760 con 0x7f53ac1022b0 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.287+0000 7f53b2ef1640 1 -- 172.21.15.49:0/326105276 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac1a21a0 con 0x7f53ac1022b0 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.289+0000 7f53a9ffb640 1 -- 172.21.15.49:0/326105276 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f53a4005020 con 0x7f53ac1022b0 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.289+0000 7f53a9ffb640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f539c0a2880 0x7f539c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.289+0000 7f53b5bbb740 1 Processor -- start 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.289+0000 7f53b5bbb740 1 -- start start 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.289+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f539c0a2880 0x7f539c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.289+0000 7f53a9ffb640 1 -- 172.21.15.49:0/326105276 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f53a40cef70 con 0x7f53ac1022b0 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 0x5580ec15dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec1630c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 0x5580ec165bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5580ec0c1c80 con 0x5580ec15e5c0 2024-03-18T00:49:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5580ec06aaa0 con 0x5580ec163730 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580ec0bdd20 con 0x5580ec15dbd0 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53abfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec1630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53abfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec1630c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49836/0 (socket says 172.21.15.49:49836) 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53abfff640 1 -- 172.21.15.49:0/2724274382 learned_addr learned my addr 172.21.15.49:0/2724274382 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53b148e640 1 --2- 172.21.15.49:0/2724274382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 0x5580ec165bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/2724274382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 0x5580ec15dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53abfff640 1 -- 172.21.15.49:0/2724274382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 msgr2=0x5580ec15dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53abfff640 1 --2- 172.21.15.49:0/2724274382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 0x5580ec15dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53abfff640 1 -- 172.21.15.49:0/2724274382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 msgr2=0x5580ec165bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53abfff640 1 --2- 172.21.15.49:0/2724274382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 0x5580ec165bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f539c0a2880 0x7f539c0a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f53a0000c00 tx=0x7f53a0005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53abfff640 1 -- 172.21.15.49:0/2724274382 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580ec08cff0 con 0x5580ec15e5c0 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f53abfff640 1 --2- 172.21.15.49:0/2724274382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec1630c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5398009c90 tx=0x7f5398011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11fd8a8df158cab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f5396ffd640 1 -- 172.21.15.49:0/2724274382 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539801e070 con 0x5580ec15e5c0 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f5396ffd640 1 -- 172.21.15.49:0/2724274382 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5398019b10 con 0x5580ec15e5c0 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.290+0000 7f5396ffd640 1 -- 172.21.15.49:0/2724274382 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539800ea70 con 0x5580ec15e5c0 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2724274382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 msgr2=0x5580ec1630c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2724274382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec1630c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5398009c90 tx=0x7f5398011810 comp rx=0 tx=0).stop 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2724274382 shutdown_connections 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2724274382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 0x5580ec165bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2724274382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec1630c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2724274382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 0x5580ec15dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2724274382 >> 172.21.15.49:0/2724274382 conn(0x5580ec159140 msgr2=0x5580ec159540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2724274382 shutdown_connections 2024-03-18T00:49:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2724274382 wait complete. 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.291+0000 7f53b5bbb740 1 -- start start 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b5bbb740 1 Processor -- start 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 0x5580ec167df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec168330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 0x5580ec16f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5580ec1666e0 con 0x5580ec15e5c0 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5580ec07def0 con 0x5580ec163730 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580ec166860 con 0x5580ec15dbd0 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b5bbb740 10 client.0 _ll_register_callbacks cb 0x5580ec078dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53abfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec168330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53abfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec168330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49844/0 (socket says 172.21.15.49:49844) 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b0c8d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 0x5580ec167df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53abfff640 1 -- 172.21.15.49:0/1915488988 learned_addr learned my addr 172.21.15.49:0/1915488988 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b148e640 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 0x5580ec16f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53abfff640 1 -- 172.21.15.49:0/1915488988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 msgr2=0x5580ec167df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53abfff640 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 0x5580ec167df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53abfff640 1 -- 172.21.15.49:0/1915488988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 msgr2=0x5580ec16f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53abfff640 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 0x5580ec16f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53abfff640 1 -- 172.21.15.49:0/1915488988 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580ec08aa30 con 0x5580ec15e5c0 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.292+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 0x5580ec167df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53abfff640 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec168330 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5398011d10 tx=0x7f5398019aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee740c1e4ffc545 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f538bfff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f538bfff640 1 -- 172.21.15.49:0/1915488988 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539801e050 con 0x5580ec15e5c0 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f538bfff640 1 -- 172.21.15.49:0/1915488988 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5398005450 con 0x5580ec15e5c0 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b5bbb740 1 -- 172.21.15.49:0/1915488988 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5398019b10 con 0x5580ec15e5c0 2024-03-18T00:49:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f538bfff640 1 -- 172.21.15.49:0/1915488988 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5398005650 con 0x5580ec15e5c0 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b5bbb740 10 client.15303 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b5bbb740 1 -- 172.21.15.49:0/1915488988 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5580ebdd62d0 con 0x5580ec15e5c0 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f538bfff640 1 -- 172.21.15.49:0/1915488988 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5398019b10 con 0x5580ec15e5c0 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f538bfff640 1 -- 172.21.15.49:0/1915488988 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f539802d030 con 0x5580ec15e5c0 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b5bbb740 10 client.15303 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b5bbb740 10 client.15303 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b5bbb740 20 client.15303 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5580ec1745e0 0x5580ec175a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b5bbb740 4 client.15303 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b5bbb740 1 -- 172.21.15.49:0/1915488988 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580ec0cf1c0 con 0x5580ec1745e0 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.293+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5580ec1745e0 0x5580ec175a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.294+0000 7f538bfff640 10 client.15303 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:49:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.294+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5580ec1745e0 0x5580ec175a50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=88bbb94396136fcf server_cookie=225f9172d70e4927 in_seq=0 out_seq=0 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.294+0000 7f538bfff640 1 -- 172.21.15.49:0/1915488988 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5580ec0cf1c0 con 0x5580ec1745e0 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.294+0000 7f538bfff640 10 client.15303 handle_command_reply: tid=0 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 client.15303 shutdown 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 -- 172.21.15.49:0/1915488988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 msgr2=0x5580ec168330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec168330 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5398011d10 tx=0x7f5398019aa0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 -- 172.21.15.49:0/1915488988 shutdown_connections 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5580ec1745e0 0x5580ec175a50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec163730 0x5580ec16f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15e5c0 0x5580ec168330 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f538bfff640 0 client.15303 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/1915488988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15dbd0 0x5580ec167df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 -- 172.21.15.49:0/1915488988 >> 172.21.15.49:0/1915488988 conn(0x5580ec159140 msgr2=0x5580ec158380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 -- 172.21.15.49:0/1915488988 shutdown_connections 2024-03-18T00:49:58.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 1 -- 172.21.15.49:0/1915488988 wait complete. 2024-03-18T00:49:58.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.295+0000 7f53b5bbb740 20 client.15303 trim_cache size 0 max 16384 2024-03-18T00:49:58.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b5bbb740 1 Processor -- start 2024-03-18T00:49:58.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b5bbb740 1 -- start start 2024-03-18T00:49:58.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15dbd0 0x5580ec167a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15e5c0 0x5580ec16f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec163730 0x5580ec167f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5580ec173d90 con 0x5580ec15dbd0 2024-03-18T00:49:58.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5580ec174460 con 0x5580ec15e5c0 2024-03-18T00:49:58.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580ec174b30 con 0x5580ec163730 2024-03-18T00:49:58.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b0c8d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15dbd0 0x5580ec167a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.313+0000 7f53b0c8d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15dbd0 0x5580ec167a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49846/0 (socket says 172.21.15.49:49846) 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53b0c8d640 1 -- 172.21.15.49:0/773746146 learned_addr learned my addr 172.21.15.49:0/773746146 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53b148e640 1 --2- 172.21.15.49:0/773746146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec163730 0x5580ec167f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53abfff640 1 --2- 172.21.15.49:0/773746146 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15e5c0 0x5580ec16f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53b0c8d640 1 -- 172.21.15.49:0/773746146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec163730 msgr2=0x5580ec167f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/773746146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec163730 0x5580ec167f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53b0c8d640 1 -- 172.21.15.49:0/773746146 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15e5c0 msgr2=0x5580ec16f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/773746146 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15e5c0 0x5580ec16f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53b0c8d640 1 -- 172.21.15.49:0/773746146 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580ebea06b0 con 0x5580ec15dbd0 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/773746146 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15dbd0 0x5580ec167a00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f53a002c7b0 tx=0x7f53a0046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=771b1e70ce6c6215 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53967fc640 1 -- 172.21.15.49:0/773746146 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0005240 con 0x5580ec15dbd0 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53967fc640 1 -- 172.21.15.49:0/773746146 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a0004ba0 con 0x5580ec15dbd0 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.314+0000 7f53967fc640 1 -- 172.21.15.49:0/773746146 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0056340 con 0x5580ec15dbd0 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 -- 172.21.15.49:0/773746146 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15dbd0 msgr2=0x5580ec167a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/773746146 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15dbd0 0x5580ec167a00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f53a002c7b0 tx=0x7f53a0046000 comp rx=0 tx=0).stop 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 -- 172.21.15.49:0/773746146 shutdown_connections 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/773746146 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15e5c0 0x5580ec16f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/773746146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec163730 0x5580ec167f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/773746146 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec15dbd0 0x5580ec167a00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 -- 172.21.15.49:0/773746146 >> 172.21.15.49:0/773746146 conn(0x5580ec159140 msgr2=0x5580ec159520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 -- 172.21.15.49:0/773746146 shutdown_connections 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 -- 172.21.15.49:0/773746146 wait complete. 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 -- start start 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 Processor -- start 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15dbd0 0x5580ec16a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15e5c0 0x5580ec16aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec163730 0x5580ec16daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5580ec1758d0 con 0x5580ec163730 2024-03-18T00:49:58.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5580ec175200 con 0x5580ec15dbd0 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.315+0000 7f53b5bbb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580ec175fa0 con 0x5580ec15e5c0 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b5bbb740 10 client.0 _ll_register_callbacks cb 0x7f539801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b148e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec163730 0x5580ec16daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b0c8d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15dbd0 0x5580ec16a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b0c8d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15dbd0 0x5580ec16a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:50736/0 (socket says 172.21.15.49:50736) 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b0c8d640 1 -- 172.21.15.49:0/2507119760 learned_addr learned my addr 172.21.15.49:0/2507119760 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53abfff640 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15e5c0 0x5580ec16aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b0c8d640 1 -- 172.21.15.49:0/2507119760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15e5c0 msgr2=0x5580ec16aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15e5c0 0x5580ec16aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b0c8d640 1 -- 172.21.15.49:0/2507119760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec163730 msgr2=0x5580ec16daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec163730 0x5580ec16daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b0c8d640 1 -- 172.21.15.49:0/2507119760 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580ec16e090 con 0x5580ec15dbd0 2024-03-18T00:49:58.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b0c8d640 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15dbd0 0x5580ec16a540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f53a002c170 tx=0x7f53a0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97e774ba59a594c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f538bfff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f538bfff640 1 -- 172.21.15.49:0/2507119760 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0005000 con 0x5580ec15dbd0 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f538bfff640 1 -- 172.21.15.49:0/2507119760 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f53a00051a0 con 0x5580ec15dbd0 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2507119760 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a0004ba0 con 0x5580ec15dbd0 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f538bfff640 1 -- 172.21.15.49:0/2507119760 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f53a00053a0 con 0x5580ec15dbd0 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b5bbb740 10 client.25148 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.316+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2507119760 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5580ec16f390 con 0x5580ec15dbd0 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f538bfff640 1 -- 172.21.15.49:0/2507119760 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a0004ba0 con 0x5580ec15dbd0 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f538bfff640 1 -- 172.21.15.49:0/2507119760 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f53a0003780 con 0x5580ec15dbd0 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f53b5bbb740 10 client.25148 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f53b5bbb740 10 client.25148 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f53b5bbb740 20 client.25148 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:58.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5580ec1de5f0 0x5580ec1dd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f53b5bbb740 4 client.25148 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2507119760 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "status"}) -- 0x5580ec0c9510 con 0x5580ec1de5f0 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f53abfff640 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5580ec1de5f0 0x5580ec1dd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f53abfff640 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5580ec1de5f0 0x5580ec1dd300 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=20125777a810133d server_cookie=7c4565076ec3d262 in_seq=0 out_seq=0 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f538bfff640 10 client.25148 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f538bfff640 1 -- 172.21.15.49:0/2507119760 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5580ec0c9510 con 0x5580ec1de5f0 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f538bfff640 10 client.25148 handle_command_reply: tid=0 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.317+0000 7f53b5bbb740 1 client.25148 shutdown 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2507119760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15dbd0 msgr2=0x5580ec16a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15dbd0 0x5580ec16a540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f53a002c170 tx=0x7f53a0005fd0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2507119760 shutdown_connections 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5580ec1de5f0 0x5580ec1dd300 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580ec163730 0x5580ec16daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580ec15e5c0 0x5580ec16aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f538bfff640 0 client.25148 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:49:58.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 --2- 172.21.15.49:0/2507119760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580ec15dbd0 0x5580ec16a540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2507119760 >> 172.21.15.49:0/2507119760 conn(0x5580ec159140 msgr2=0x5580ec167a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2507119760 shutdown_connections 2024-03-18T00:49:58.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 1 -- 172.21.15.49:0/2507119760 wait complete. 2024-03-18T00:49:58.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b5bbb740 20 client.25148 trim_cache size 0 max 16384 2024-03-18T00:49:58.328 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:58.328 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:58.328 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 1, 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24499, 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 110.34413661400001, 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 126.83089187 2024-03-18T00:49:58.329 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:58.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 -- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f539c0a2880 msgr2=0x7f539c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f539c0a2880 0x7f539c0a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f53a0000c00 tx=0x7f53a0005d50 comp rx=0 tx=0).stop 2024-03-18T00:49:58.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 -- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac1022b0 msgr2=0x7f53ac1a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac1022b0 0x7f53ac1a12f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f53a400eb10 tx=0x7f53a400c2d0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 -- 172.21.15.49:0/326105276 shutdown_connections 2024-03-18T00:49:58.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f539c0a2880 0x7f539c0a4d40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53ac1018f0 0x7f53ac19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53ac1022b0 0x7f53ac1a12f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 --2- 172.21.15.49:0/326105276 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53ac100700 0x7f53ac199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.318+0000 7f53b2ef1640 1 -- 172.21.15.49:0/326105276 >> 172.21.15.49:0/326105276 conn(0x7f53ac0fbc90 msgr2=0x7f53ac0fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.319+0000 7f53b2ef1640 1 -- 172.21.15.49:0/326105276 shutdown_connections 2024-03-18T00:49:58.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.319+0000 7f53b2ef1640 1 -- 172.21.15.49:0/326105276 wait complete. 2024-03-18T00:49:58.332 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T00:49:58.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.440+0000 7f72fb309640 1 -- 172.21.15.49:0/3058662833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f4102290 msgr2=0x7f72f4108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.440+0000 7f72fb309640 1 --2- 172.21.15.49:0/3058662833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f4102290 0x7f72f4108b20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f72ec009ca0 tx=0x7f72ec02c0f0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.440+0000 7f72fb309640 1 -- 172.21.15.49:0/3058662833 shutdown_connections 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.440+0000 7f72fb309640 1 --2- 172.21.15.49:0/3058662833 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f4102290 0x7f72f4108b20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.440+0000 7f72fb309640 1 --2- 172.21.15.49:0/3058662833 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f41018d0 0x7f72f4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.440+0000 7f72fb309640 1 --2- 172.21.15.49:0/3058662833 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72f41006e0 0x7f72f4100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.440+0000 7f72fb309640 1 -- 172.21.15.49:0/3058662833 >> 172.21.15.49:0/3058662833 conn(0x7f72f40fbc70 msgr2=0x7f72f40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.440+0000 7f72fb309640 1 -- 172.21.15.49:0/3058662833 shutdown_connections 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.440+0000 7f72fb309640 1 -- 172.21.15.49:0/3058662833 wait complete. 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72fb309640 1 Processor -- start 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72fb309640 1 -- start start 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72fb309640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72f41006e0 0x7f72f40694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72fb309640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f41018d0 0x7f72f419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72fb309640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f4102290 0x7f72f419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72fb309640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f72f410c360 con 0x7f72f41018d0 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72fb309640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f72f410c1e0 con 0x7f72f41006e0 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72fb309640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f72f410c4e0 con 0x7f72f4102290 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f88a4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f41018d0 0x7f72f419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f88a4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f41018d0 0x7f72f419c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49860/0 (socket says 172.21.15.49:49860) 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f88a4640 1 -- 172.21.15.49:0/173641839 learned_addr learned my addr 172.21.15.49:0/173641839 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f90a5640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72f41006e0 0x7f72f40694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f98a6640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f4102290 0x7f72f419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f88a4640 1 -- 172.21.15.49:0/173641839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f4102290 msgr2=0x7f72f419ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f88a4640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f4102290 0x7f72f419ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f88a4640 1 -- 172.21.15.49:0/173641839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72f41006e0 msgr2=0x7f72f40694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f88a4640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72f41006e0 0x7f72f40694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f88a4640 1 -- 172.21.15.49:0/173641839 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f419f750 con 0x7f72f41018d0 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.441+0000 7f72f90a5640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72f41006e0 0x7f72f40694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.442+0000 7f72f88a4640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f41018d0 0x7f72f419c820 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e9f0 tx=0x7f72e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.442+0000 7f72ea7fc640 1 -- 172.21.15.49:0/173641839 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e400cd80 con 0x7f72f41018d0 2024-03-18T00:49:58.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.442+0000 7f72ea7fc640 1 -- 172.21.15.49:0/173641839 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72e4005500 con 0x7f72f41018d0 2024-03-18T00:49:58.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.442+0000 7f72ea7fc640 1 -- 172.21.15.49:0/173641839 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e4010690 con 0x7f72f41018d0 2024-03-18T00:49:58.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.442+0000 7f72fb309640 1 -- 172.21.15.49:0/173641839 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f419f9e0 con 0x7f72f41018d0 2024-03-18T00:49:58.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.442+0000 7f72fb309640 1 -- 172.21.15.49:0/173641839 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f419fee0 con 0x7f72f41018d0 2024-03-18T00:49:58.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.444+0000 7f72ea7fc640 1 -- 172.21.15.49:0/173641839 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f72e40046c0 con 0x7f72f41018d0 2024-03-18T00:49:58.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.444+0000 7f72fdfd5740 1 Processor -- start 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.444+0000 7f72fdfd5740 1 -- start start 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.444+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9603ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.444+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e96090c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.444+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 0x55a9e960bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.444+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a9e9567c80 con 0x55a9e9609730 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.444+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a9e9510aa0 con 0x55a9e96045c0 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.444+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a9e9563d20 con 0x55a9e9603bd0 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72ea7fc640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72e00a2800 0x7f72e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f98a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9603ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f88a4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 0x55a9e960bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f90a5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e96090c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f98a6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9603ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:54544/0 (socket says 172.21.15.49:54544) 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f98a6640 1 -- 172.21.15.49:0/978157819 learned_addr learned my addr 172.21.15.49:0/978157819 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f98a6640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72e00a2800 0x7f72e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72ea7fc640 1 -- 172.21.15.49:0/173641839 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f72e40cee70 con 0x7f72f41018d0 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f90a5640 1 -- 172.21.15.49:0/978157819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 msgr2=0x55a9e9603ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f90a5640 1 --2- 172.21.15.49:0/978157819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9603ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f90a5640 1 -- 172.21.15.49:0/978157819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 msgr2=0x55a9e960bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f90a5640 1 --2- 172.21.15.49:0/978157819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 0x55a9e960bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f90a5640 1 -- 172.21.15.49:0/978157819 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e9532ff0 con 0x55a9e96045c0 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f98a6640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72e00a2800 0x7f72e00a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55a9e9604e20 tx=0x7f72ec032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72f90a5640 1 --2- 172.21.15.49:0/978157819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e96090c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009fa0 tx=0x7f72f00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a97733097b955844 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72df7fe640 1 -- 172.21.15.49:0/978157819 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f0004310 con 0x55a9e96045c0 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72df7fe640 1 -- 172.21.15.49:0/978157819 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72f00044b0 con 0x55a9e96045c0 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.445+0000 7f72df7fe640 1 -- 172.21.15.49:0/978157819 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f00035f0 con 0x55a9e96045c0 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 -- 172.21.15.49:0/978157819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 msgr2=0x55a9e96090c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/978157819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e96090c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009fa0 tx=0x7f72f00118d0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 -- 172.21.15.49:0/978157819 shutdown_connections 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/978157819 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e96090c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/978157819 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 0x55a9e960bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/978157819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9603ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 -- 172.21.15.49:0/978157819 >> 172.21.15.49:0/978157819 conn(0x55a9e95ff140 msgr2=0x55a9e95ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 -- 172.21.15.49:0/978157819 shutdown_connections 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 -- 172.21.15.49:0/978157819 wait complete. 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 -- start start 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.446+0000 7f72fdfd5740 1 Processor -- start 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e960ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e960e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 0x55a9e9615340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a9e960c6e0 con 0x55a9e9609730 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a9e9523ef0 con 0x55a9e96045c0 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a9e960c860 con 0x55a9e9603bd0 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72fdfd5740 10 client.0 _ll_register_callbacks cb 0x55a9e951edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f98a6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 0x55a9e9615340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f90a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e960ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f88a4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e960e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f90a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e960ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:54558/0 (socket says 172.21.15.49:54558) 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f98a6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 0x55a9e9615340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49888/0 (socket says 172.21.15.49:49888) 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f90a5640 1 -- 172.21.15.49:0/3160828251 learned_addr learned my addr 172.21.15.49:0/3160828251 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f88a4640 1 -- 172.21.15.49:0/3160828251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 msgr2=0x55a9e960ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f88a4640 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e960ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f88a4640 1 -- 172.21.15.49:0/3160828251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 msgr2=0x55a9e9615340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f88a4640 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 0x55a9e9615340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f88a4640 1 -- 172.21.15.49:0/3160828251 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e9530a30 con 0x55a9e96045c0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f90a5640 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e960ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72f88a4640 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e960e300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f72e40cc9a0 tx=0x7f72e406a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b60fbf26af638ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.447+0000 7f72dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72dcff9640 1 -- 172.21.15.49:0/3160828251 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e40a0860 con 0x55a9e96045c0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72dcff9640 1 -- 172.21.15.49:0/3160828251 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f72e40860f0 con 0x55a9e96045c0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72fdfd5740 1 -- 172.21.15.49:0/3160828251 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72f00044b0 con 0x55a9e96045c0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72dcff9640 1 -- 172.21.15.49:0/3160828251 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f72e409bef0 con 0x55a9e96045c0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72fdfd5740 10 client.25160 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72fdfd5740 1 -- 172.21.15.49:0/3160828251 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a9e927c2d0 con 0x55a9e96045c0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72dcff9640 1 -- 172.21.15.49:0/3160828251 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72f00044b0 con 0x55a9e96045c0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72dcff9640 1 -- 172.21.15.49:0/3160828251 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f72e40bb040 con 0x55a9e96045c0 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72fdfd5740 10 client.25160 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:58.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72fdfd5740 10 client.25160 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72fdfd5740 20 client.25160 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a9e961a790 0x55a9e961bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72fdfd5740 4 client.25160 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72fdfd5740 1 -- 172.21.15.49:0/3160828251 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9e95751c0 con 0x55a9e961a790 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.448+0000 7f72f90a5640 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a9e961a790 0x55a9e961bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.449+0000 7f72f90a5640 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a9e961a790 0x55a9e961bc00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc2ef3a76c44f9bb server_cookie=a76c330fc36a69e7 in_seq=0 out_seq=0 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.449+0000 7f72dcff9640 10 client.25160 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.449+0000 7f72dcff9640 1 -- 172.21.15.49:0/3160828251 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a9e95751c0 con 0x55a9e961a790 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.449+0000 7f72dcff9640 10 client.25160 handle_command_reply: tid=0 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.449+0000 7f72fdfd5740 1 client.25160 shutdown 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.449+0000 7f72fdfd5740 1 -- 172.21.15.49:0/3160828251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 msgr2=0x55a9e960e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.449+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e960e300 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f72e40cc9a0 tx=0x7f72e406a710 comp rx=0 tx=0).stop 2024-03-18T00:49:58.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72fdfd5740 1 -- 172.21.15.49:0/3160828251 shutdown_connections 2024-03-18T00:49:58.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a9e961a790 0x55a9e961bc00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e960e300 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9609730 0x55a9e9615340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72dcff9640 0 client.25160 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:49:58.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/3160828251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e960ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72fdfd5740 1 -- 172.21.15.49:0/3160828251 >> 172.21.15.49:0/3160828251 conn(0x55a9e95ff140 msgr2=0x55a9e95fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72fdfd5740 1 -- 172.21.15.49:0/3160828251 shutdown_connections 2024-03-18T00:49:58.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72fdfd5740 1 -- 172.21.15.49:0/3160828251 wait complete. 2024-03-18T00:49:58.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.450+0000 7f72fdfd5740 20 client.25160 trim_cache size 0 max 16384 2024-03-18T00:49:58.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72fdfd5740 1 Processor -- start 2024-03-18T00:49:58.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72fdfd5740 1 -- start start 2024-03-18T00:49:58.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9603bd0 0x55a9e960bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e9615300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9609730 0x55a9e960ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a9e9619f40 con 0x55a9e9603bd0 2024-03-18T00:49:58.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a9e961a610 con 0x55a9e96045c0 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a9e961ace0 con 0x55a9e9609730 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f90a5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9603bd0 0x55a9e960bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f90a5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9603bd0 0x55a9e960bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49894/0 (socket says 172.21.15.49:49894) 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f90a5640 1 -- 172.21.15.49:0/1052777974 learned_addr learned my addr 172.21.15.49:0/1052777974 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f98a6640 1 --2- 172.21.15.49:0/1052777974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9609730 0x55a9e960ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f88a4640 1 --2- 172.21.15.49:0/1052777974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e9615300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f90a5640 1 -- 172.21.15.49:0/1052777974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9609730 msgr2=0x55a9e960ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f90a5640 1 --2- 172.21.15.49:0/1052777974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9609730 0x55a9e960ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f90a5640 1 -- 172.21.15.49:0/1052777974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 msgr2=0x55a9e9615300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f90a5640 1 --2- 172.21.15.49:0/1052777974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e9615300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.468+0000 7f72f90a5640 1 -- 172.21.15.49:0/1052777974 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e93466b0 con 0x55a9e9603bd0 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72f90a5640 1 --2- 172.21.15.49:0/1052777974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9603bd0 0x55a9e960bb20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f72f000a010 tx=0x7f72f0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01aa6086aadce52 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72deffd640 1 -- 172.21.15.49:0/1052777974 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f001d040 con 0x55a9e9603bd0 2024-03-18T00:49:58.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72deffd640 1 -- 172.21.15.49:0/1052777974 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72f0009b50 con 0x55a9e9603bd0 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72deffd640 1 -- 172.21.15.49:0/1052777974 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72f000f740 con 0x55a9e9603bd0 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72fdfd5740 1 -- 172.21.15.49:0/1052777974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9603bd0 msgr2=0x55a9e960bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/1052777974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9603bd0 0x55a9e960bb20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f72f000a010 tx=0x7f72f0009310 comp rx=0 tx=0).stop 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72fdfd5740 1 -- 172.21.15.49:0/1052777974 shutdown_connections 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/1052777974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e96045c0 0x55a9e9615300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/1052777974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9609730 0x55a9e960ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/1052777974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e9603bd0 0x55a9e960bb20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72fdfd5740 1 -- 172.21.15.49:0/1052777974 >> 172.21.15.49:0/1052777974 conn(0x55a9e95ff140 msgr2=0x55a9e95ff520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72fdfd5740 1 -- 172.21.15.49:0/1052777974 shutdown_connections 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.469+0000 7f72fdfd5740 1 -- 172.21.15.49:0/1052777974 wait complete. 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72fdfd5740 1 -- start start 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72fdfd5740 1 Processor -- start 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9610710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e96045c0 0x55a9e9610c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72fdfd5740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e9609730 0x55a9e9613c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a9e961ba80 con 0x55a9e96045c0 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a9e961b3b0 con 0x55a9e9609730 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72fdfd5740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a9e961c150 con 0x55a9e9603bd0 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72fdfd5740 10 client.0 _ll_register_callbacks cb 0x7f72e4004ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72f88a4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e96045c0 0x55a9e9610c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72f90a5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9610710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72f88a4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e96045c0 0x55a9e9610c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49900/0 (socket says 172.21.15.49:49900) 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72f88a4640 1 -- 172.21.15.49:0/2002412048 learned_addr learned my addr 172.21.15.49:0/2002412048 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.470+0000 7f72f98a6640 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e9609730 0x55a9e9613c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72f88a4640 1 -- 172.21.15.49:0/2002412048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 msgr2=0x55a9e9610710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72f88a4640 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9610710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72f88a4640 1 -- 172.21.15.49:0/2002412048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e9609730 msgr2=0x55a9e9613c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72f88a4640 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e9609730 0x55a9e9613c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72f88a4640 1 -- 172.21.15.49:0/2002412048 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e9614260 con 0x55a9e96045c0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72f90a5640 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9610710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72f88a4640 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e96045c0 0x55a9e9610c50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f72e40981c0 tx=0x7f72e409bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456d02f4da0745fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72dcff9640 1 -- 172.21.15.49:0/2002412048 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e406e060 con 0x55a9e96045c0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72dcff9640 1 -- 172.21.15.49:0/2002412048 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f72e40860f0 con 0x55a9e96045c0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72fdfd5740 1 -- 172.21.15.49:0/2002412048 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72f0009b50 con 0x55a9e96045c0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72dcff9640 1 -- 172.21.15.49:0/2002412048 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f72e40a0860 con 0x55a9e96045c0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72fdfd5740 10 client.15327 fetch_fsmap learned FSMap version 24 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72fdfd5740 1 -- 172.21.15.49:0/2002412048 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a9e96155d0 con 0x55a9e96045c0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72dcff9640 1 -- 172.21.15.49:0/2002412048 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72f0009b50 con 0x55a9e96045c0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72dcff9640 1 -- 172.21.15.49:0/2002412048 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f72e40bb040 con 0x55a9e96045c0 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72fdfd5740 10 client.15327 fetch_fsmap finished waiting for FSMap version 24 2024-03-18T00:49:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72fdfd5740 10 client.15327 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72fdfd5740 20 client.15327 populate_metadata read hostname 'smithi049' 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a9e9684ea0 0x55a9e9683bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72fdfd5740 4 client.15327 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.471+0000 7f72fdfd5740 1 -- 172.21.15.49:0/2002412048 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "status"}) -- 0x55a9e956f510 con 0x55a9e9684ea0 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72f90a5640 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a9e9684ea0 0x55a9e9683bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72f90a5640 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a9e9684ea0 0x55a9e9683bb0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d59c0e241a5bee01 server_cookie=ae7a4a9803881819 in_seq=0 out_seq=0 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72dcff9640 10 client.15327 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72dcff9640 1 -- 172.21.15.49:0/2002412048 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a9e956f510 con 0x55a9e9684ea0 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72dcff9640 10 client.15327 handle_command_reply: tid=0 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 client.15327 shutdown 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 -- 172.21.15.49:0/2002412048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e96045c0 msgr2=0x55a9e9610c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e96045c0 0x55a9e9610c50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f72e40981c0 tx=0x7f72e409bdf0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 -- 172.21.15.49:0/2002412048 shutdown_connections 2024-03-18T00:49:58.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a9e9684ea0 0x55a9e9683bb0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a9e9609730 0x55a9e9613c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72dcff9640 0 client.15327 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a9e96045c0 0x55a9e9610c50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 --2- 172.21.15.49:0/2002412048 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a9e9603bd0 0x55a9e9610710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 -- 172.21.15.49:0/2002412048 >> 172.21.15.49:0/2002412048 conn(0x55a9e95ff140 msgr2=0x55a9e960dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 -- 172.21.15.49:0/2002412048 shutdown_connections 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 1 -- 172.21.15.49:0/2002412048 wait complete. 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.472+0000 7f72fdfd5740 20 client.15327 trim_cache size 0 max 16384 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:49:58.480 INFO:teuthology.orchestra.run.smithi049.stdout: "cluster_fsid": "ec390cde-e4bf-11ee-95c9-87774f69a715", 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "whoami": 0, 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "id": 24506, 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "want_state": "up:active", 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "state": "up:active", 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "fs_name": "cephfs", 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "rank_uptime": 111.95821209899999, 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "mdsmap_epoch": 24, 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch": 88, 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_epoch_barrier": 86, 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout: "uptime": 118.51612629100001 2024-03-18T00:49:58.481 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:49:58.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 -- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72e00a2800 msgr2=0x7f72e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72e00a2800 0x7f72e00a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x55a9e9604e20 tx=0x7f72ec032590 comp rx=0 tx=0).stop 2024-03-18T00:49:58.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 -- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f41018d0 msgr2=0x7f72f419c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:49:58.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f41018d0 0x7f72f419c820 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e9f0 tx=0x7f72e400edf0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 -- 172.21.15.49:0/173641839 shutdown_connections 2024-03-18T00:49:58.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72e00a2800 0x7f72e00a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72f4102290 0x7f72f419ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72f41018d0 0x7f72f419c820 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 --2- 172.21.15.49:0/173641839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72f41006e0 0x7f72f40694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:49:58.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 -- 172.21.15.49:0/173641839 >> 172.21.15.49:0/173641839 conn(0x7f72f40fbc70 msgr2=0x7f72f4104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:49:58.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 -- 172.21.15.49:0/173641839 shutdown_connections 2024-03-18T00:49:58.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:49:58.473+0000 7f72fb309640 1 -- 172.21.15.49:0/173641839 wait complete. 2024-03-18T00:49:58.484 INFO:teuthology.run_tasks:Running task kclient... 2024-03-18T00:49:58.509 INFO:tasks.kclient:Mounting kernel clients... 2024-03-18T00:49:58.509 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-18T00:49:58.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:49:58.510 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list 2024-03-18T00:49:58.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:49:58.525 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link delete ceph-brx 2024-03-18T00:49:58.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:58 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/39466370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:58.593 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:58 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1883517170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:58.599 INFO:teuthology.orchestra.run.smithi049.stderr:Cannot find device "ceph-brx" 2024-03-18T00:49:58.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T00:49:58.600 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-18T00:49:58.601 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T00:49:58.602 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T00:49:58.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T00:49:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:58 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/39466370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:58 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1883517170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:58.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:58 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/39466370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:49:58.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:58 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1883517170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:49:58.896 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T00:49:59.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:49:59 smithi049 ceph-mon[32130]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:59.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:49:59 smithi107 ceph-mon[36270]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:49:59.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:49:59 smithi106 ceph-mon[35899]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:50:00.359 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:00 smithi049 ceph-mon[32130]: overall HEALTH_OK 2024-03-18T00:50:00.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:00 smithi107 ceph-mon[36270]: overall HEALTH_OK 2024-03-18T00:50:00.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:00 smithi106 ceph-mon[35899]: overall HEALTH_OK 2024-03-18T00:50:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 -- 172.21.15.49:0/648664161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd0102f60 msgr2=0x7f4bd0106f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 --2- 172.21.15.49:0/648664161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd0102f60 0x7f4bd0106f30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00099c0 tx=0x7f4bc0009cd0 comp rx=0 tx=0).stop 2024-03-18T00:50:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 -- 172.21.15.49:0/648664161 shutdown_connections 2024-03-18T00:50:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 --2- 172.21.15.49:0/648664161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0107660 0x7f4bd010f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 --2- 172.21.15.49:0/648664161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd01025b0 0x7f4bd0102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 --2- 172.21.15.49:0/648664161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd0102f60 0x7f4bd0106f30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 -- 172.21.15.49:0/648664161 >> 172.21.15.49:0/648664161 conn(0x7f4bd00793f0 msgr2=0x7f4bd0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 -- 172.21.15.49:0/648664161 shutdown_connections 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 -- 172.21.15.49:0/648664161 wait complete. 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 Processor -- start 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.741+0000 7f4bd762c700 1 -- start start 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd762c700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd01025b0 0x7f4bd00745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd762c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0102f60 0x7f4bd0074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd762c700 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd0107660 0x7f4bd0075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd762c700 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd01125b0 con 0x7f4bd01025b0 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd762c700 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0112430 con 0x7f4bd0107660 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd762c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd0112730 con 0x7f4bd0102f60 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd53c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd01025b0 0x7f4bd00745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd4bc7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0102f60 0x7f4bd0074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd53c8700 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd01025b0 0x7f4bd00745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49912/0 (socket says 172.21.15.49:49912) 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd4bc7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0102f60 0x7f4bd0074b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:54588/0 (socket says 172.21.15.49:54588) 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd53c8700 1 -- 172.21.15.49:0/3464257464 learned_addr learned my addr 172.21.15.49:0/3464257464 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd5bc9700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd0107660 0x7f4bd0075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd53c8700 1 -- 172.21.15.49:0/3464257464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0102f60 msgr2=0x7f4bd0074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd53c8700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0102f60 0x7f4bd0074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd53c8700 1 -- 172.21.15.49:0/3464257464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd0107660 msgr2=0x7f4bd0075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd53c8700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd0107660 0x7f4bd0075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd53c8700 1 -- 172.21.15.49:0/3464257464 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd01a6fd0 con 0x7f4bd01025b0 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd53c8700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd01025b0 0x7f4bd00745d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00e9a0 tx=0x7f4bcc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bc67fc700 1 -- 172.21.15.49:0/3464257464 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00ca10 con 0x7f4bd01025b0 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd01a72c0 con 0x7f4bd01025b0 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bc67fc700 1 -- 172.21.15.49:0/3464257464 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bcc00cbb0 con 0x7f4bd01025b0 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bc67fc700 1 -- 172.21.15.49:0/3464257464 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc018680 con 0x7f4bd01025b0 2024-03-18T00:50:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.742+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd01a7820 con 0x7f4bd01025b0 2024-03-18T00:50:00.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.743+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd0068f90 con 0x7f4bd01025b0 2024-03-18T00:50:00.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.745+0000 7f4bc67fc700 1 -- 172.21.15.49:0/3464257464 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4bcc010430 con 0x7f4bd01025b0 2024-03-18T00:50:00.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.746+0000 7f4bc67fc700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bbc0a2890 0x7f4bbc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:00.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.746+0000 7f4bd4bc7700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bbc0a2890 0x7f4bbc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:00.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.746+0000 7f4bc67fc700 1 -- 172.21.15.49:0/3464257464 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4bcc0cecc0 con 0x7f4bd01025b0 2024-03-18T00:50:00.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.746+0000 7f4bd4bc7700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bbc0a2890 0x7f4bbc0a4d50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009990 tx=0x7f4bc0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:00.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:00.750+0000 7f4bc67fc700 1 -- 172.21.15.49:0/3464257464 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4bcc09bd40 con 0x7f4bd01025b0 2024-03-18T00:50:01.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.000+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4bd002db70 con 0x7f4bd01025b0 2024-03-18T00:50:01.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.001+0000 7f4bc67fc700 1 -- 172.21.15.49:0/3464257464 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+73 (secure 0 0 0) 0x7f4bcc097030 con 0x7f4bd01025b0 2024-03-18T00:50:01.002 INFO:teuthology.orchestra.run.smithi049.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-18T00:50:01.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bbc0a2890 msgr2=0x7f4bbc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bbc0a2890 0x7f4bbc0a4d50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009990 tx=0x7f4bc0005fb0 comp rx=0 tx=0).stop 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd01025b0 msgr2=0x7f4bd00745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd01025b0 0x7f4bd00745d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00e9a0 tx=0x7f4bcc00ecb0 comp rx=0 tx=0).stop 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 shutdown_connections 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4bd0107660 0x7f4bd0075050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4bd01025b0 0x7f4bd00745d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bbc0a2890 0x7f4bbc0a4d50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 --2- 172.21.15.49:0/3464257464 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4bd0102f60 0x7f4bd0074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 >> 172.21.15.49:0/3464257464 conn(0x7f4bd00793f0 msgr2=0x7f4bd0108790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 shutdown_connections 2024-03-18T00:50:01.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01.003+0000 7f4bd762c700 1 -- 172.21.15.49:0/3464257464 wait complete. 2024-03-18T00:50:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:01 smithi049 ceph-mon[32130]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:50:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:01 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3464257464' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T00:50:01.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:01 smithi107 ceph-mon[36270]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:50:01.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:01 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3464257464' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T00:50:01.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:01 smithi106 ceph-mon[35899]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:50:01.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:01 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3464257464' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T00:50:01.764 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T00:50:01.764 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T00:50:01.764 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi049.front.sepia.ceph.com 2024-03-18T00:50:01.764 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T00:50:01.764 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T00:50:01.764 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T00:50:01.764 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T00:50:01.765 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T00:50:01.765 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T00:50:01.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:50:01.765 DEBUG:teuthology.orchestra.run.smithi049:> ip addr 2024-03-18T00:50:01.782 INFO:teuthology.orchestra.run.smithi049.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T00:50:01.783 INFO:teuthology.orchestra.run.smithi049.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T00:50:01.783 INFO:teuthology.orchestra.run.smithi049.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T00:50:01.783 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2024-03-18T00:50:01.783 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 ::1/128 scope host 2024-03-18T00:50:01.783 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2024-03-18T00:50:01.783 INFO:teuthology.orchestra.run.smithi049.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T00:50:01.783 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:6b:ff:84 brd ff:ff:ff:ff:ff:ff 2024-03-18T00:50:01.783 INFO:teuthology.orchestra.run.smithi049.stdout: altname enp7s0f0 2024-03-18T00:50:01.783 INFO:teuthology.orchestra.run.smithi049.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:6b:ff:85 brd ff:ff:ff:ff:ff:ff 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: altname enp7s0f1 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:bd:15:66 brd ff:ff:ff:ff:ff:ff 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: altname enp1s0f0 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: inet 172.21.15.49/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft 42147sec preferred_lft 42147sec 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: inet6 fe80::ec4:7aff:febd:1566/64 scope link noprefixroute 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: valid_lft forever preferred_lft forever 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: link/ether 0c:c4:7a:bd:15:67 brd ff:ff:ff:ff:ff:ff 2024-03-18T00:50:01.784 INFO:teuthology.orchestra.run.smithi049.stdout: altname enp1s0f1 2024-03-18T00:50:01.785 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T00:50:01.785 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T00:50:01.785 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2024-03-18T00:50:01.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link add name ceph-brx type bridge 2024-03-18T00:50:01.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip addr flush dev ceph-brx 2024-03-18T00:50:01.785 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link set ceph-brx up 2024-03-18T00:50:01.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T00:50:01.786 DEBUG:teuthology.orchestra.run.smithi049:> ') 2024-03-18T00:50:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T00:50:02.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T00:50:02.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:50:02.009 DEBUG:teuthology.orchestra.run.smithi049:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T00:50:02.040 INFO:teuthology.orchestra.run.smithi049.stdout:1 2024-03-18T00:50:02.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:50:02.042 DEBUG:teuthology.orchestra.run.smithi049:> route 2024-03-18T00:50:02.115 INFO:teuthology.orchestra.run.smithi049.stdout:Kernel IP routing table 2024-03-18T00:50:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T00:50:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T00:50:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T00:50:02.116 INFO:teuthology.orchestra.run.smithi049.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T00:50:02.116 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T00:50:02.116 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2024-03-18T00:50:02.116 DEBUG:teuthology.orchestra.run.smithi049:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T00:50:02.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T00:50:02.117 DEBUG:teuthology.orchestra.run.smithi049:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T00:50:02.117 DEBUG:teuthology.orchestra.run.smithi049:> ') 2024-03-18T00:50:02.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T00:50:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T00:50:02.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:50:02.449 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list 2024-03-18T00:50:02.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:50:02.465 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list-id 2024-03-18T00:50:02.521 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T00:50:02.522 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2024-03-18T00:50:02.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T00:50:02.522 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T00:50:02.522 DEBUG:teuthology.orchestra.run.smithi049:> ') 2024-03-18T00:50:02.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T00:50:02.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T00:50:02.654 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T00:50:02.654 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T00:50:02.654 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2024-03-18T00:50:02.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T00:50:02.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T00:50:02.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T00:50:02.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T00:50:02.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T00:50:02.655 DEBUG:teuthology.orchestra.run.smithi049:> ') 2024-03-18T00:50:02.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T00:50:02.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T00:50:02.815 DEBUG:teuthology.orchestra.run.smithi049:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T00:50:02.815 DEBUG:teuthology.orchestra.run.smithi049:> set -e 2024-03-18T00:50:02.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link set brx.0 up 2024-03-18T00:50:02.816 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T00:50:02.816 DEBUG:teuthology.orchestra.run.smithi049:> ') 2024-03-18T00:50:02.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T00:50:02.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T00:50:02.905 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T00:50:02.905 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T00:50:02.920 INFO:teuthology.orchestra.run.smithi049.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T00:50:02.920 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T00:50:02.921 DEBUG:teuthology.orchestra.run.smithi049:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T00:50:02.976 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-18T00:50:02.976 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-18T00:50:02.976 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/d7393f6b-0735-4414-ac84-f375918aae72 2024-03-18T00:50:02.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:50:02.976 DEBUG:teuthology.orchestra.run.smithi049:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/d7393f6b-0735-4414-ac84-f375918aae72 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-18T00:50:03.065 INFO:teuthology.orchestra.run.smithi049.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-18T00:50:03.065 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-18T00:50:03.178 INFO:teuthology.orchestra.run.smithi049.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=legacy,nowsync,nofallback 2024-03-18T00:50:03.178 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync". 2024-03-18T00:50:03.178 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: resolved to: "172.21.15.49:6789,172.21.15.106:6789,172.21.15.107:6789" 2024-03-18T00:50:03.178 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: trying mount with new device syntax: 0@ec390cde-e4bf-11ee-95c9-87774f69a715.cephfs=/volumes/qa/sv_1/d7393f6b-0735-4414-ac84-f375918aae72 2024-03-18T00:50:03.179 INFO:teuthology.orchestra.run.smithi049.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=legacy,nowsync,name=0,key=0,mon_addr=172.21.15.49:6789/172.21.15.106:6789/172.21.15.107:6789" will pass to kernel 2024-03-18T00:50:03.179 INFO:teuthology.orchestra.run.smithi049.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-18T00:50:03.179 INFO:teuthology.orchestra.run.smithi049.stdout: You just mounted a file system that supports labels which does not 2024-03-18T00:50:03.179 INFO:teuthology.orchestra.run.smithi049.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-18T00:50:03.179 INFO:teuthology.orchestra.run.smithi049.stdout: applications will generate AVC messages and not be allowed access to 2024-03-18T00:50:03.179 INFO:teuthology.orchestra.run.smithi049.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-18T00:50:03.181 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-18T00:50:03.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:50:03.181 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T00:50:03.216 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T00:50:03.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T00:50:03.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T00:50:03.316 INFO:teuthology.orchestra.run.smithi049.stdout:client25201 2024-03-18T00:50:03.317 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:50:03.317 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T00:50:03.335 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T00:50:03.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T00:50:03.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T00:50:03.431 INFO:teuthology.orchestra.run.smithi049.stdout:client25201 2024-03-18T00:50:03.432 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T00:50:03.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T00:50:03.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T00:50:03.488 INFO:teuthology.orchestra.run.smithi049.stdout:client25201 2024-03-18T00:50:03.488 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715.client25201/status') 2024-03-18T00:50:03.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T00:50:03.551 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:03 smithi049 ceph-mon[32130]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:50:03.553 INFO:teuthology.orchestra.run.smithi049.stdout:instance: client.25201 (3)172.21.15.49:0/3289765253 2024-03-18T00:50:03.554 INFO:teuthology.orchestra.run.smithi049.stdout:blocklisted: false 2024-03-18T00:50:03.554 INFO:teuthology.orchestra.run.smithi049.stderr:0+1 records in 2024-03-18T00:50:03.554 INFO:teuthology.orchestra.run.smithi049.stderr:0+1 records out 2024-03-18T00:50:03.554 INFO:teuthology.orchestra.run.smithi049.stderr:71 bytes copied, 2.2308e-05 s, 3.2 MB/s 2024-03-18T00:50:03.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T00:50:03.561 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T00:50:03.572 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T00:50:03.573 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi049.front.sepia.ceph.com 2024-03-18T00:50:03.573 DEBUG:teuthology.orchestra.run.smithi049:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-18T00:50:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:03 smithi107 ceph-mon[36270]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:50:03.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.681+0000 7fee72ff3640 1 -- 172.21.15.49:0/2872594515 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee6c101940 msgr2=0x7fee6c101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:03.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.681+0000 7fee72ff3640 1 --2- 172.21.15.49:0/2872594515 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee6c101940 0x7fee6c101dc0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fee5c009900 tx=0x7fee5c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:03.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.681+0000 7fee72ff3640 1 -- 172.21.15.49:0/2872594515 shutdown_connections 2024-03-18T00:50:03.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.681+0000 7fee72ff3640 1 --2- 172.21.15.49:0/2872594515 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee6c102300 0x7fee6c108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:03.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.681+0000 7fee72ff3640 1 --2- 172.21.15.49:0/2872594515 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee6c101940 0x7fee6c101dc0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.681+0000 7fee72ff3640 1 --2- 172.21.15.49:0/2872594515 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee6c100750 0x7fee6c100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.681+0000 7fee72ff3640 1 -- 172.21.15.49:0/2872594515 >> 172.21.15.49:0/2872594515 conn(0x7fee6c0fbcc0 msgr2=0x7fee6c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.681+0000 7fee72ff3640 1 -- 172.21.15.49:0/2872594515 shutdown_connections 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.681+0000 7fee72ff3640 1 -- 172.21.15.49:0/2872594515 wait complete. 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee72ff3640 1 Processor -- start 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee72ff3640 1 -- start start 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee72ff3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee6c100750 0x7fee6c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee72ff3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee6c101940 0x7fee6c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee72ff3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee6c102300 0x7fee6c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee72ff3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c10c3d0 con 0x7fee6c102300 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee72ff3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c10c250 con 0x7fee6c100750 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee72ff3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c10c550 con 0x7fee6c101940 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee71590640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee6c102300 0x7fee6c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:03.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee70d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee6c100750 0x7fee6c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee63fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee6c101940 0x7fee6c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee70d8f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee6c100750 0x7fee6c197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:50820/0 (socket says 172.21.15.49:50820) 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee70d8f640 1 -- 172.21.15.49:0/3668289430 learned_addr learned my addr 172.21.15.49:0/3668289430 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee63fff640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee6c101940 0x7fee6c198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:54602/0 (socket says 172.21.15.49:54602) 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee71590640 1 -- 172.21.15.49:0/3668289430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee6c101940 msgr2=0x7fee6c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee71590640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee6c101940 0x7fee6c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee71590640 1 -- 172.21.15.49:0/3668289430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee6c100750 msgr2=0x7fee6c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee71590640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee6c100750 0x7fee6c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee71590640 1 -- 172.21.15.49:0/3668289430 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c19f710 con 0x7fee6c102300 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee63fff640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee6c101940 0x7fee6c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.682+0000 7fee70d8f640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee6c100750 0x7fee6c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.683+0000 7fee71590640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee6c102300 0x7fee6c19f0a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fee6400ec50 tx=0x7fee6400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:03.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.683+0000 7fee61ffb640 1 -- 172.21.15.49:0/3668289430 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64003b70 con 0x7fee6c102300 2024-03-18T00:50:03.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.683+0000 7fee61ffb640 1 -- 172.21.15.49:0/3668289430 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee64003d10 con 0x7fee6c102300 2024-03-18T00:50:03.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.683+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee6c19fa00 con 0x7fee6c102300 2024-03-18T00:50:03.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.683+0000 7fee61ffb640 1 -- 172.21.15.49:0/3668289430 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64010890 con 0x7fee6c102300 2024-03-18T00:50:03.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.683+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee6c19fed0 con 0x7fee6c102300 2024-03-18T00:50:03.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.684+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee6c1a00a0 con 0x7fee6c102300 2024-03-18T00:50:03.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.685+0000 7fee61ffb640 1 -- 172.21.15.49:0/3668289430 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fee64003fc0 con 0x7fee6c102300 2024-03-18T00:50:03.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.685+0000 7fee61ffb640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fee540a28a0 0x7fee540a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:03.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.686+0000 7fee70d8f640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fee540a28a0 0x7fee540a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:03.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.686+0000 7fee61ffb640 1 -- 172.21.15.49:0/3668289430 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fee640cf000 con 0x7fee6c102300 2024-03-18T00:50:03.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.686+0000 7fee70d8f640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fee540a28a0 0x7fee540a4d60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fee680058c0 tx=0x7fee68005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:03.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.691+0000 7fee61ffb640 1 -- 172.21.15.49:0/3668289430 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fee6409c080 con 0x7fee6c102300 2024-03-18T00:50:03.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:03 smithi106 ceph-mon[35899]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T00:50:03.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:03.912+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fee6c1a0270 con 0x7fee6c102300 2024-03-18T00:50:04.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.301+0000 7fee61ffb640 1 -- 172.21.15.49:0/3668289430 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7fee6409b980 con 0x7fee6c102300 2024-03-18T00:50:04.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fee540a28a0 msgr2=0x7fee540a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fee540a28a0 0x7fee540a4d60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fee680058c0 tx=0x7fee68005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:04.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee6c102300 msgr2=0x7fee6c19f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee6c102300 0x7fee6c19f0a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fee6400ec50 tx=0x7fee6400c6a0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 shutdown_connections 2024-03-18T00:50:04.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fee540a28a0 0x7fee540a4d60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fee6c102300 0x7fee6c19f0a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fee6c101940 0x7fee6c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 --2- 172.21.15.49:0/3668289430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fee6c100750 0x7fee6c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 >> 172.21.15.49:0/3668289430 conn(0x7fee6c0fbcc0 msgr2=0x7fee6c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:04.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.304+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 shutdown_connections 2024-03-18T00:50:04.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.305+0000 7fee72ff3640 1 -- 172.21.15.49:0/3668289430 wait complete. 2024-03-18T00:50:04.317 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-18T00:50:04.327 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-18T00:50:04.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:50:04.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.478+0000 7f530e6ea640 1 -- 172.21.15.49:0/2823898412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 msgr2=0x7f5308101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.478+0000 7f530e6ea640 1 --2- 172.21.15.49:0/2823898412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 0x7f5308101d70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009900 tx=0x7f52f802bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:04.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.478+0000 7f530e6ea640 1 -- 172.21.15.49:0/2823898412 shutdown_connections 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.478+0000 7f530e6ea640 1 --2- 172.21.15.49:0/2823898412 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f53081022b0 0x7f5308108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.478+0000 7f530e6ea640 1 --2- 172.21.15.49:0/2823898412 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 0x7f5308101d70 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.478+0000 7f530e6ea640 1 --2- 172.21.15.49:0/2823898412 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5308100700 0x7f5308100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.478+0000 7f530e6ea640 1 -- 172.21.15.49:0/2823898412 >> 172.21.15.49:0/2823898412 conn(0x7f53080fbc70 msgr2=0x7f53080fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.478+0000 7f530e6ea640 1 -- 172.21.15.49:0/2823898412 shutdown_connections 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.478+0000 7f530e6ea640 1 -- 172.21.15.49:0/2823898412 wait complete. 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f530e6ea640 1 Processor -- start 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f530e6ea640 1 -- start start 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f530e6ea640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5308100700 0x7f5308197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f530e6ea640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 0x7f5308198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f530e6ea640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53081022b0 0x7f530819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f530e6ea640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f530810c360 con 0x7f53081018f0 2024-03-18T00:50:04.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f530e6ea640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f530810c1e0 con 0x7f5308100700 2024-03-18T00:50:04.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f530e6ea640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f530810c4e0 con 0x7f53081022b0 2024-03-18T00:50:04.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f53077fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 0x7f5308198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:04.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.479+0000 7f53077fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 0x7f5308198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49950/0 (socket says 172.21.15.49:49950) 2024-03-18T00:50:04.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f5307fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5308100700 0x7f5308197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:04.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53077fe640 1 -- 172.21.15.49:0/1957798045 learned_addr learned my addr 172.21.15.49:0/1957798045 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:04.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f530cc87640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53081022b0 0x7f530819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:04.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53077fe640 1 -- 172.21.15.49:0/1957798045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53081022b0 msgr2=0x7f530819f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53077fe640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53081022b0 0x7f530819f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53077fe640 1 -- 172.21.15.49:0/1957798045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5308100700 msgr2=0x7f5308197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53077fe640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5308100700 0x7f5308197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53077fe640 1 -- 172.21.15.49:0/1957798045 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530819f720 con 0x7f53081018f0 2024-03-18T00:50:04.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f5307fff640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5308100700 0x7f5308197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:04.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53077fe640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 0x7f5308198040 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009230 tx=0x7f52f802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:04.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53057fa640 1 -- 172.21.15.49:0/1957798045 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8041070 con 0x7f53081018f0 2024-03-18T00:50:04.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53057fa640 1 -- 172.21.15.49:0/1957798045 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52f8004450 con 0x7f53081018f0 2024-03-18T00:50:04.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530819f9b0 con 0x7f53081018f0 2024-03-18T00:50:04.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f53057fa640 1 -- 172.21.15.49:0/1957798045 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8003e20 con 0x7f53081018f0 2024-03-18T00:50:04.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.480+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530819fef0 con 0x7f53081018f0 2024-03-18T00:50:04.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.481+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53081a00c0 con 0x7f53081018f0 2024-03-18T00:50:04.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.482+0000 7f53057fa640 1 -- 172.21.15.49:0/1957798045 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f52f8044070 con 0x7f53081018f0 2024-03-18T00:50:04.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.483+0000 7f53057fa640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52f40a2880 0x7f52f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:04.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.483+0000 7f5307fff640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52f40a2880 0x7f52f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:04.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.483+0000 7f53057fa640 1 -- 172.21.15.49:0/1957798045 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f52f80f0590 con 0x7f53081018f0 2024-03-18T00:50:04.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.484+0000 7f5307fff640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52f40a2880 0x7f52f40a4d40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5308101750 tx=0x7f52f0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:04.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.488+0000 7f53057fa640 1 -- 172.21.15.49:0/1957798045 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f52f80bcfc0 con 0x7f53081018f0 2024-03-18T00:50:04.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:04 smithi049 ceph-mon[32130]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:50:04.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:04 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3668289430' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T00:50:04.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.673+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53081a0450 con 0x7f53081018f0 2024-03-18T00:50:04.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.674+0000 7f53057fa640 1 -- 172.21.15.49:0/1957798045 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7f52f80c1060 con 0x7f53081018f0 2024-03-18T00:50:04.674 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:04.675 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52f40a2880 msgr2=0x7f52f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52f40a2880 0x7f52f40a4d40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5308101750 tx=0x7f52f0005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 msgr2=0x7f5308198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 0x7f5308198040 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009230 tx=0x7f52f802c650 comp rx=0 tx=0).stop 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 shutdown_connections 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52f40a2880 0x7f52f40a4d40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53081022b0 0x7f530819f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f53081018f0 0x7f5308198040 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 --2- 172.21.15.49:0/1957798045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5308100700 0x7f5308197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 >> 172.21.15.49:0/1957798045 conn(0x7f53080fbc70 msgr2=0x7f53080fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:04.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 shutdown_connections 2024-03-18T00:50:04.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.677+0000 7f530e6ea640 1 -- 172.21.15.49:0/1957798045 wait complete. 2024-03-18T00:50:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:04 smithi107 ceph-mon[36270]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:50:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:04 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3668289430' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T00:50:04.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T00:48:04.316609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T00:48:05.271815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:50:04.688 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:50:04.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:04 smithi106 ceph-mon[35899]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T00:50:04.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:04 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3668289430' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T00:50:04.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 -- 172.21.15.49:0/2049778147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 msgr2=0x7efff4109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 --2- 172.21.15.49:0/2049778147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 0x7efff4109d00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7effec009cd0 tx=0x7effec02c0f0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 -- 172.21.15.49:0/2049778147 shutdown_connections 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 --2- 172.21.15.49:0/2049778147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 0x7efff4109d00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 --2- 172.21.15.49:0/2049778147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff4102ab0 0x7efff4102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 --2- 172.21.15.49:0/2049778147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efff40ffae0 0x7efff40fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 -- 172.21.15.49:0/2049778147 >> 172.21.15.49:0/2049778147 conn(0x7efff40fb410 msgr2=0x7efff40fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 -- 172.21.15.49:0/2049778147 shutdown_connections 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 -- 172.21.15.49:0/2049778147 wait complete. 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 Processor -- start 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.800+0000 7efffb709640 1 -- start start 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efffb709640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efff40ffae0 0x7efff4197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efffb709640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff4102ab0 0x7efff4198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efffb709640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 0x7efff419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:04.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efffb709640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efff410d4c0 con 0x7efff4103470 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efffb709640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efff410d340 con 0x7efff40ffae0 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efffb709640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efff410d640 con 0x7efff4102ab0 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff9ca6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 0x7efff419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff94a5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efff40ffae0 0x7efff4197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff9ca6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 0x7efff419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49954/0 (socket says 172.21.15.49:49954) 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff8ca4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff4102ab0 0x7efff4198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff9ca6640 1 -- 172.21.15.49:0/485941153 learned_addr learned my addr 172.21.15.49:0/485941153 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff9ca6640 1 -- 172.21.15.49:0/485941153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff4102ab0 msgr2=0x7efff4198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff9ca6640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff4102ab0 0x7efff4198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff9ca6640 1 -- 172.21.15.49:0/485941153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efff40ffae0 msgr2=0x7efff4197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff9ca6640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efff40ffae0 0x7efff4197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff9ca6640 1 -- 172.21.15.49:0/485941153 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff419f720 con 0x7efff4103470 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efff9ca6640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 0x7efff419f0b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7effec009ca0 tx=0x7effec0346d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:04.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7effea7fc640 1 -- 172.21.15.49:0/485941153 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec041020 con 0x7efff4103470 2024-03-18T00:50:04.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7effea7fc640 1 -- 172.21.15.49:0/485941153 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effec032890 con 0x7efff4103470 2024-03-18T00:50:04.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff419f9b0 con 0x7efff4103470 2024-03-18T00:50:04.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7effea7fc640 1 -- 172.21.15.49:0/485941153 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effec039040 con 0x7efff4103470 2024-03-18T00:50:04.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.801+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff419fef0 con 0x7efff4103470 2024-03-18T00:50:04.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.802+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff4108be0 con 0x7efff4103470 2024-03-18T00:50:04.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.803+0000 7effea7fc640 1 -- 172.21.15.49:0/485941153 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7effec044020 con 0x7efff4103470 2024-03-18T00:50:04.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.804+0000 7effea7fc640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7effe00a2880 0x7effe00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:04.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.804+0000 7effea7fc640 1 -- 172.21.15.49:0/485941153 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7effec0f07a0 con 0x7efff4103470 2024-03-18T00:50:04.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.804+0000 7efff94a5640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7effe00a2880 0x7effe00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:04.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.804+0000 7efff94a5640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7effe00a2880 0x7effe00a4d40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efff4069b90 tx=0x7efff0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:04.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.808+0000 7effea7fc640 1 -- 172.21.15.49:0/485941153 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7effec04b020 con 0x7efff4103470 2024-03-18T00:50:04.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.992+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efff40623e0 con 0x7efff4103470 2024-03-18T00:50:04.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.993+0000 7effea7fc640 1 -- 172.21.15.49:0/485941153 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7effec0098b0 con 0x7efff4103470 2024-03-18T00:50:04.993 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7effe00a2880 msgr2=0x7effe00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7effe00a2880 0x7effe00a4d40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7efff4069b90 tx=0x7efff0005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 msgr2=0x7efff419f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 0x7efff419f0b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7effec009ca0 tx=0x7effec0346d0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 shutdown_connections 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7effe00a2880 0x7effe00a4d40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efff4103470 0x7efff419f0b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efff4102ab0 0x7efff4198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 --2- 172.21.15.49:0/485941153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efff40ffae0 0x7efff4197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 >> 172.21.15.49:0/485941153 conn(0x7efff40fb410 msgr2=0x7efff4105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 shutdown_connections 2024-03-18T00:50:04.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:04.995+0000 7efffb709640 1 -- 172.21.15.49:0/485941153 wait complete. 2024-03-18T00:50:05.006 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T00:50:05.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.117+0000 7f9d6e13b640 1 -- 172.21.15.49:0/4051675816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681018d0 msgr2=0x7f9d68101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.117+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/4051675816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681018d0 0x7f9d68101d50 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c009900 tx=0x7f9d5c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:05.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.117+0000 7f9d6e13b640 1 -- 172.21.15.49:0/4051675816 shutdown_connections 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.117+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/4051675816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d68102290 0x7f9d68108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.117+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/4051675816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681018d0 0x7f9d68101d50 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.117+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/4051675816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d681006e0 0x7f9d68100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.117+0000 7f9d6e13b640 1 -- 172.21.15.49:0/4051675816 >> 172.21.15.49:0/4051675816 conn(0x7f9d680fbc70 msgr2=0x7f9d680fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.117+0000 7f9d6e13b640 1 -- 172.21.15.49:0/4051675816 shutdown_connections 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.117+0000 7f9d6e13b640 1 -- 172.21.15.49:0/4051675816 wait complete. 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d6e13b640 1 Processor -- start 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d6e13b640 1 -- start start 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d6e13b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681006e0 0x7f9d68197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d6e13b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d681018d0 0x7f9d68198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d6e13b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d68102290 0x7f9d6819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d6e13b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6810c340 con 0x7f9d681006e0 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d6e13b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6810c1c0 con 0x7f9d68102290 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d6e13b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6810c4c0 con 0x7f9d681018d0 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d677fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681006e0 0x7f9d68197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d677fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681006e0 0x7f9d68197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49966/0 (socket says 172.21.15.49:49966) 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d677fe640 1 -- 172.21.15.49:0/2767717748 learned_addr learned my addr 172.21.15.49:0/2767717748 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d67fff640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d68102290 0x7f9d6819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.118+0000 7f9d66ffd640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d681018d0 0x7f9d68198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d677fe640 1 -- 172.21.15.49:0/2767717748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d681018d0 msgr2=0x7f9d68198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d677fe640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d681018d0 0x7f9d68198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d677fe640 1 -- 172.21.15.49:0/2767717748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d68102290 msgr2=0x7f9d6819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d677fe640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d68102290 0x7f9d6819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d677fe640 1 -- 172.21.15.49:0/2767717748 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6819f740 con 0x7f9d681006e0 2024-03-18T00:50:05.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d677fe640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681006e0 0x7f9d68197b20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800e9c0 tx=0x7f9d5800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:05.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d64ff9640 1 -- 172.21.15.49:0/2767717748 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5800cd80 con 0x7f9d681006e0 2024-03-18T00:50:05.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d64ff9640 1 -- 172.21.15.49:0/2767717748 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d580035f0 con 0x7f9d681006e0 2024-03-18T00:50:05.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6819fa30 con 0x7f9d681006e0 2024-03-18T00:50:05.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d64ff9640 1 -- 172.21.15.49:0/2767717748 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58010690 con 0x7f9d681006e0 2024-03-18T00:50:05.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.119+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d6819ffa0 con 0x7f9d681006e0 2024-03-18T00:50:05.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.120+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d681a0230 con 0x7f9d681006e0 2024-03-18T00:50:05.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.121+0000 7f9d64ff9640 1 -- 172.21.15.49:0/2767717748 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9d580037a0 con 0x7f9d681006e0 2024-03-18T00:50:05.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.122+0000 7f9d64ff9640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9d500a2880 0x7f9d500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.122+0000 7f9d66ffd640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9d500a2880 0x7f9d500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.122+0000 7f9d64ff9640 1 -- 172.21.15.49:0/2767717748 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9d580cedb0 con 0x7f9d681006e0 2024-03-18T00:50:05.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.123+0000 7f9d66ffd640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9d500a2880 0x7f9d500a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c02c7b0 tx=0x7f9d5c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:05.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.127+0000 7f9d64ff9640 1 -- 172.21.15.49:0/2767717748 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9d5809be30 con 0x7f9d681006e0 2024-03-18T00:50:05.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.309+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9d681a09a0 con 0x7f9d681006e0 2024-03-18T00:50:05.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.309+0000 7f9d64ff9640 1 -- 172.21.15.49:0/2767717748 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f9d5809b730 con 0x7f9d681006e0 2024-03-18T00:50:05.309 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 32 2024-03-18T00:50:05.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9d500a2880 msgr2=0x7f9d500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9d500a2880 0x7f9d500a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c02c7b0 tx=0x7f9d5c005fd0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681006e0 msgr2=0x7f9d68197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681006e0 0x7f9d68197b20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9d5800e9c0 tx=0x7f9d5800ee80 comp rx=0 tx=0).stop 2024-03-18T00:50:05.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 shutdown_connections 2024-03-18T00:50:05.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9d500a2880 0x7f9d500a4d40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9d68102290 0x7f9d6819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d681018d0 0x7f9d68198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 --2- 172.21.15.49:0/2767717748 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9d681006e0 0x7f9d68197b20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 >> 172.21.15.49:0/2767717748 conn(0x7f9d680fbc70 msgr2=0x7f9d680fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:05.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 shutdown_connections 2024-03-18T00:50:05.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.311+0000 7f9d6e13b640 1 -- 172.21.15.49:0/2767717748 wait complete. 2024-03-18T00:50:05.322 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T00:50:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.433+0000 7fa105790640 1 -- 172.21.15.49:0/1545037710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1001018f0 msgr2=0x7fa100101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.433+0000 7fa105790640 1 --2- 172.21.15.49:0/1545037710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1001018f0 0x7fa100101d70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009860 tx=0x7fa0e802bb70 comp rx=0 tx=0).stop 2024-03-18T00:50:05.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.433+0000 7fa105790640 1 -- 172.21.15.49:0/1545037710 shutdown_connections 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.433+0000 7fa105790640 1 --2- 172.21.15.49:0/1545037710 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1001018f0 0x7fa100101d70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.433+0000 7fa105790640 1 --2- 172.21.15.49:0/1545037710 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1001022b0 0x7fa100108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.433+0000 7fa105790640 1 --2- 172.21.15.49:0/1545037710 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa100100700 0x7fa100100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.433+0000 7fa105790640 1 -- 172.21.15.49:0/1545037710 >> 172.21.15.49:0/1545037710 conn(0x7fa1000fbc90 msgr2=0x7fa1000fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.433+0000 7fa105790640 1 -- 172.21.15.49:0/1545037710 shutdown_connections 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.433+0000 7fa105790640 1 -- 172.21.15.49:0/1545037710 wait complete. 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa105790640 1 Processor -- start 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa105790640 1 -- start start 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa105790640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa100100700 0x7fa100196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa105790640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1001018f0 0x7fa100196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa105790640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1001022b0 0x7fa10019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa105790640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa10010c380 con 0x7fa1001022b0 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa105790640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa10010c200 con 0x7fa100100700 2024-03-18T00:50:05.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa105790640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa10010c500 con 0x7fa1001018f0 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa0ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1001022b0 0x7fa10019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa0ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1001022b0 0x7fa10019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49982/0 (socket says 172.21.15.49:49982) 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa0ff7fe640 1 -- 172.21.15.49:0/717144606 learned_addr learned my addr 172.21.15.49:0/717144606 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa0fe7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1001018f0 0x7fa100196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.434+0000 7fa0feffd640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa100100700 0x7fa100196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0ff7fe640 1 -- 172.21.15.49:0/717144606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1001018f0 msgr2=0x7fa100196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0ff7fe640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1001018f0 0x7fa100196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0ff7fe640 1 -- 172.21.15.49:0/717144606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa100100700 msgr2=0x7fa100196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0ff7fe640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa100100700 0x7fa100196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0ff7fe640 1 -- 172.21.15.49:0/717144606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10019e620 con 0x7fa1001022b0 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0feffd640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa100100700 0x7fa100196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0fe7fc640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1001018f0 0x7fa100196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0ff7fe640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1001022b0 0x7fa10019dfb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800e970 tx=0x7fa0f800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:05.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0f7fff640 1 -- 172.21.15.49:0/717144606 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f800ccc0 con 0x7fa1001022b0 2024-03-18T00:50:05.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10019e910 con 0x7fa1001022b0 2024-03-18T00:50:05.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0f7fff640 1 -- 172.21.15.49:0/717144606 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f800ce60 con 0x7fa1001022b0 2024-03-18T00:50:05.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa0f7fff640 1 -- 172.21.15.49:0/717144606 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8010450 con 0x7fa1001022b0 2024-03-18T00:50:05.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.435+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa10019ee80 con 0x7fa1001022b0 2024-03-18T00:50:05.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.436+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa100107a20 con 0x7fa1001022b0 2024-03-18T00:50:05.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.437+0000 7fa0f7fff640 1 -- 172.21.15.49:0/717144606 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa0f8005020 con 0x7fa1001022b0 2024-03-18T00:50:05.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.438+0000 7fa0f7fff640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.438+0000 7fa0feffd640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.438+0000 7fa0f7fff640 1 -- 172.21.15.49:0/717144606 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa0f80cee90 con 0x7fa1001022b0 2024-03-18T00:50:05.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.438+0000 7fa0feffd640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00098d0 tx=0x7fa0f0010040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.443+0000 7fa0f7fff640 1 -- 172.21.15.49:0/717144606 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa0f809bef0 con 0x7fa1001022b0 2024-03-18T00:50:05.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:05 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3668289430' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T00:50:05.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:05 smithi049 ceph-mon[32130]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:50:05.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:05 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1957798045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:05.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:05 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/485941153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:50:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.627+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa1000623e0 con 0x7fa1001022b0 2024-03-18T00:50:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.628+0000 7fa0f7fff640 1 -- 172.21.15.49:0/717144606 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fa0f809b7f0 con 0x7fa1001022b0 2024-03-18T00:50:05.627 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 64 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0ec0a2800 msgr2=0x7fa0ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00098d0 tx=0x7fa0f0010040 comp rx=0 tx=0).stop 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1001022b0 msgr2=0x7fa10019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1001022b0 0x7fa10019dfb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800e970 tx=0x7fa0f800ed70 comp rx=0 tx=0).stop 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 shutdown_connections 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0ec0a2800 0x7fa0ec0a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1001022b0 0x7fa10019dfb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1001018f0 0x7fa100196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 --2- 172.21.15.49:0/717144606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa100100700 0x7fa100196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 >> 172.21.15.49:0/717144606 conn(0x7fa1000fbc90 msgr2=0x7fa100104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:05.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 shutdown_connections 2024-03-18T00:50:05.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.630+0000 7fa105790640 1 -- 172.21.15.49:0/717144606 wait complete. 2024-03-18T00:50:05.641 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:50:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:05 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3668289430' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T00:50:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:05 smithi107 ceph-mon[36270]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:50:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:05 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1957798045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:05 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/485941153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:50:05.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:05 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3668289430' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T00:50:05.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:05 smithi106 ceph-mon[35899]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:50:05.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:05 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1957798045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:05.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:05 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/485941153' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:50:05.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.749+0000 7fd02cdba640 1 -- 172.21.15.49:0/593266535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd028100780 msgr2=0x7fd028100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.749+0000 7fd02cdba640 1 --2- 172.21.15.49:0/593266535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd028100780 0x7fd028100ba0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd018009860 tx=0x7fd01802bb70 comp rx=0 tx=0).stop 2024-03-18T00:50:05.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.749+0000 7fd02cdba640 1 -- 172.21.15.49:0/593266535 shutdown_connections 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.749+0000 7fd02cdba640 1 --2- 172.21.15.49:0/593266535 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd028102330 0x7fd028108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.749+0000 7fd02cdba640 1 --2- 172.21.15.49:0/593266535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd028101970 0x7fd028101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.749+0000 7fd02cdba640 1 --2- 172.21.15.49:0/593266535 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd028100780 0x7fd028100ba0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.749+0000 7fd02cdba640 1 -- 172.21.15.49:0/593266535 >> 172.21.15.49:0/593266535 conn(0x7fd0280fbd10 msgr2=0x7fd0280fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.749+0000 7fd02cdba640 1 -- 172.21.15.49:0/593266535 shutdown_connections 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.749+0000 7fd02cdba640 1 -- 172.21.15.49:0/593266535 wait complete. 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.750+0000 7fd02cdba640 1 Processor -- start 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.750+0000 7fd02cdba640 1 -- start start 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.750+0000 7fd02cdba640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd028100780 0x7fd02810b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.750+0000 7fd02cdba640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd028101970 0x7fd028109740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.750+0000 7fd02cdba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd028102330 0x7fd028109c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.750+0000 7fd02cdba640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd02810d470 con 0x7fd028100780 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.750+0000 7fd02cdba640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd02810d2f0 con 0x7fd028101970 2024-03-18T00:50:05.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.750+0000 7fd02cdba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd02810d5f0 con 0x7fd028102330 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd028100780 0x7fd02810b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd028100780 0x7fd02810b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49992/0 (socket says 172.21.15.49:49992) 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02659c640 1 -- 172.21.15.49:0/1177206195 learned_addr learned my addr 172.21.15.49:0/1177206195 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd026d9d640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd028102330 0x7fd028109c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd01ffff640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd028101970 0x7fd028109740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02659c640 1 -- 172.21.15.49:0/1177206195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd028102330 msgr2=0x7fd028109c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02659c640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd028102330 0x7fd028109c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02659c640 1 -- 172.21.15.49:0/1177206195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd028101970 msgr2=0x7fd028109740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02659c640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd028101970 0x7fd028109740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02659c640 1 -- 172.21.15.49:0/1177206195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02810a510 con 0x7fd028100780 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd026d9d640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd028102330 0x7fd028109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd01ffff640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd028101970 0x7fd028109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:05.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02659c640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd028100780 0x7fd02810b100 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd01802c0b0 tx=0x7fd018032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:05.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd01f7fe640 1 -- 172.21.15.49:0/1177206195 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd018041020 con 0x7fd028100780 2024-03-18T00:50:05.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0281a1be0 con 0x7fd028100780 2024-03-18T00:50:05.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd01f7fe640 1 -- 172.21.15.49:0/1177206195 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd01802c9f0 con 0x7fd028100780 2024-03-18T00:50:05.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd01f7fe640 1 -- 172.21.15.49:0/1177206195 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd018039040 con 0x7fd028100780 2024-03-18T00:50:05.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.751+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0281a2130 con 0x7fd028100780 2024-03-18T00:50:05.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.752+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd028107aa0 con 0x7fd028100780 2024-03-18T00:50:05.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.753+0000 7fd01f7fe640 1 -- 172.21.15.49:0/1177206195 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd018044020 con 0x7fd028100780 2024-03-18T00:50:05.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.754+0000 7fd01f7fe640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd0140a2800 0x7fd0140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:05.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.754+0000 7fd01ffff640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd0140a2800 0x7fd0140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:05.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.754+0000 7fd01f7fe640 1 -- 172.21.15.49:0/1177206195 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd0180f0370 con 0x7fd028100780 2024-03-18T00:50:05.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.755+0000 7fd01ffff640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd0140a2800 0x7fd0140a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd02810ae70 tx=0x7fd010005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:05.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.759+0000 7fd01f7fe640 1 -- 172.21.15.49:0/1177206195 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd01804b020 con 0x7fd028100780 2024-03-18T00:50:05.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.948+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0280623e0 con 0x7fd028100780 2024-03-18T00:50:05.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.950+0000 7fd01f7fe640 1 -- 172.21.15.49:0/1177206195 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7fd0180bd3f0 con 0x7fd028100780 2024-03-18T00:50:05.950 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:05.954 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:50:05.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd0140a2800 msgr2=0x7fd0140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd0140a2800 0x7fd0140a4cc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd02810ae70 tx=0x7fd010005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd028100780 msgr2=0x7fd02810b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd028100780 0x7fd02810b100 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd01802c0b0 tx=0x7fd018032430 comp rx=0 tx=0).stop 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 shutdown_connections 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd0140a2800 0x7fd0140a4cc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd028101970 0x7fd028109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd028102330 0x7fd028109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 --2- 172.21.15.49:0/1177206195 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd028100780 0x7fd02810b100 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 >> 172.21.15.49:0/1177206195 conn(0x7fd0280fbd10 msgr2=0x7fd028104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.952+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 shutdown_connections 2024-03-18T00:50:05.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:05.953+0000 7fd02cdba640 1 -- 172.21.15.49:0/1177206195 wait complete. 2024-03-18T00:50:05.964 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T00:48:04.316609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T00:48:05.271815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:50:05.964 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:50:06.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 -- 172.21.15.49:0/1919316474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8104c10 msgr2=0x7ff4e8107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 --2- 172.21.15.49:0/1919316474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8104c10 0x7ff4e8107000 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009900 tx=0x7ff4dc02bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:06.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 -- 172.21.15.49:0/1919316474 shutdown_connections 2024-03-18T00:50:06.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 --2- 172.21.15.49:0/1919316474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e8107540 0x7ff4e8109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 --2- 172.21.15.49:0/1919316474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8104c10 0x7ff4e8107000 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 --2- 172.21.15.49:0/1919316474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e81022e0 0x7ff4e81046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 -- 172.21.15.49:0/1919316474 >> 172.21.15.49:0/1919316474 conn(0x7ff4e80fbc90 msgr2=0x7ff4e80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 -- 172.21.15.49:0/1919316474 shutdown_connections 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 -- 172.21.15.49:0/1919316474 wait complete. 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.076+0000 7ff4ee976640 1 Processor -- start 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ee976640 1 -- start start 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ee976640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e81022e0 0x7ff4e8197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ee976640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e8104c10 0x7ff4e8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ee976640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8107540 0x7ff4e819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ee976640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e810d470 con 0x7ff4e8107540 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ee976640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e810d2f0 con 0x7ff4e81022e0 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ee976640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e810d5f0 con 0x7ff4e8104c10 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ecf13640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8107540 0x7ff4e819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e8104c10 0x7ff4e8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ecf13640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8107540 0x7ff4e819f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50002/0 (socket says 172.21.15.49:50002) 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4e77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e8104c10 0x7ff4e8198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:54730/0 (socket says 172.21.15.49:54730) 2024-03-18T00:50:06.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ecf13640 1 -- 172.21.15.49:0/327556985 learned_addr learned my addr 172.21.15.49:0/327556985 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4e7fff640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e81022e0 0x7ff4e8197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ecf13640 1 -- 172.21.15.49:0/327556985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e8104c10 msgr2=0x7ff4e8198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ecf13640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e8104c10 0x7ff4e8198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ecf13640 1 -- 172.21.15.49:0/327556985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e81022e0 msgr2=0x7ff4e8197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ecf13640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e81022e0 0x7ff4e8197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ecf13640 1 -- 172.21.15.49:0/327556985 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e819f740 con 0x7ff4e8107540 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4e77fe640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e8104c10 0x7ff4e8198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4e7fff640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e81022e0 0x7ff4e8197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.077+0000 7ff4ecf13640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8107540 0x7ff4e819f0d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000eb10 tx=0x7ff4e000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.078+0000 7ff4e57fa640 1 -- 172.21.15.49:0/327556985 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0004b20 con 0x7ff4e8107540 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.078+0000 7ff4e57fa640 1 -- 172.21.15.49:0/327556985 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4e0004cc0 con 0x7ff4e8107540 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.078+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e819fa30 con 0x7ff4e8107540 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.078+0000 7ff4e57fa640 1 -- 172.21.15.49:0/327556985 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e0010760 con 0x7ff4e8107540 2024-03-18T00:50:06.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.078+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e819ff00 con 0x7ff4e8107540 2024-03-18T00:50:06.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.079+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4e80fd780 con 0x7ff4e8107540 2024-03-18T00:50:06.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.080+0000 7ff4e57fa640 1 -- 172.21.15.49:0/327556985 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff4e0005020 con 0x7ff4e8107540 2024-03-18T00:50:06.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.080+0000 7ff4e57fa640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4d00a2800 0x7ff4d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.081+0000 7ff4e7fff640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4d00a2800 0x7ff4d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.081+0000 7ff4e57fa640 1 -- 172.21.15.49:0/327556985 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff4e00ceff0 con 0x7ff4e8107540 2024-03-18T00:50:06.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.081+0000 7ff4e7fff640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4d00a2800 0x7ff4d00a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8009fc0 tx=0x7ff4d8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:06.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.085+0000 7ff4e57fa640 1 -- 172.21.15.49:0/327556985 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff4e009c070 con 0x7ff4e8107540 2024-03-18T00:50:06.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.268+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff4e80623e0 con 0x7ff4e8107540 2024-03-18T00:50:06.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.268+0000 7ff4e57fa640 1 -- 172.21.15.49:0/327556985 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7ff4e009b970 con 0x7ff4e8107540 2024-03-18T00:50:06.268 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:50:06.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4d00a2800 msgr2=0x7ff4d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4d00a2800 0x7ff4d00a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7ff4d8009fc0 tx=0x7ff4d8005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:06.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8107540 msgr2=0x7ff4e819f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8107540 0x7ff4e819f0d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff4e000eb10 tx=0x7ff4e000c2d0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 shutdown_connections 2024-03-18T00:50:06.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4d00a2800 0x7ff4d00a4cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff4e8107540 0x7ff4e819f0d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4e8104c10 0x7ff4e8198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 --2- 172.21.15.49:0/327556985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff4e81022e0 0x7ff4e8197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.270+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 >> 172.21.15.49:0/327556985 conn(0x7ff4e80fbc90 msgr2=0x7ff4e8102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:06.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.271+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 shutdown_connections 2024-03-18T00:50:06.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.271+0000 7ff4ee976640 1 -- 172.21.15.49:0/327556985 wait complete. 2024-03-18T00:50:06.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T00:50:06.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 -- 172.21.15.49:0/3432540020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82041018f0 msgr2=0x7f8204101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 --2- 172.21.15.49:0/3432540020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82041018f0 0x7f8204101d70 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009900 tx=0x7f81f802bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:06.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 -- 172.21.15.49:0/3432540020 shutdown_connections 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 --2- 172.21.15.49:0/3432540020 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82041022b0 0x7f8204108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 --2- 172.21.15.49:0/3432540020 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82041018f0 0x7f8204101d70 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 --2- 172.21.15.49:0/3432540020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8204100700 0x7f8204100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 -- 172.21.15.49:0/3432540020 >> 172.21.15.49:0/3432540020 conn(0x7f82040fbc90 msgr2=0x7f82040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 -- 172.21.15.49:0/3432540020 shutdown_connections 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 -- 172.21.15.49:0/3432540020 wait complete. 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 Processor -- start 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.395+0000 7f820a46e640 1 -- start start 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f820a46e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8204100700 0x7f8204197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f820a46e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82041018f0 0x7f8204198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f820a46e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82041022b0 0x7f820419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f820a46e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f820410c380 con 0x7f8204100700 2024-03-18T00:50:06.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f820a46e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f820410c200 con 0x7f82041022b0 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f820a46e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f820410c500 con 0x7f82041018f0 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8203fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8204100700 0x7f8204197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8203fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8204100700 0x7f8204197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50026/0 (socket says 172.21.15.49:50026) 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f82037fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82041018f0 0x7f8204198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8203fff640 1 -- 172.21.15.49:0/3060950114 learned_addr learned my addr 172.21.15.49:0/3060950114 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8208a0b640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82041022b0 0x7f820419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8203fff640 1 -- 172.21.15.49:0/3060950114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82041018f0 msgr2=0x7f8204198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8203fff640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82041018f0 0x7f8204198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8203fff640 1 -- 172.21.15.49:0/3060950114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82041022b0 msgr2=0x7f820419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8203fff640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82041022b0 0x7f820419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8203fff640 1 -- 172.21.15.49:0/3060950114 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820419f770 con 0x7f8204100700 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f82037fe640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82041018f0 0x7f8204198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.396+0000 7f8203fff640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8204100700 0x7f8204197b50 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f81f400e9c0 tx=0x7f81f400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:06.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.397+0000 7f82017fa640 1 -- 172.21.15.49:0/3060950114 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f400cd80 con 0x7f8204100700 2024-03-18T00:50:06.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.397+0000 7f82017fa640 1 -- 172.21.15.49:0/3060950114 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81f40035f0 con 0x7f8204100700 2024-03-18T00:50:06.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.397+0000 7f82017fa640 1 -- 172.21.15.49:0/3060950114 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4010690 con 0x7f8204100700 2024-03-18T00:50:06.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.397+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820419fa60 con 0x7f8204100700 2024-03-18T00:50:06.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.397+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f820419ff80 con 0x7f8204100700 2024-03-18T00:50:06.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.398+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82041a01b0 con 0x7f8204100700 2024-03-18T00:50:06.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.399+0000 7f82017fa640 1 -- 172.21.15.49:0/3060950114 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f81f40037a0 con 0x7f8204100700 2024-03-18T00:50:06.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.399+0000 7f82017fa640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81ec0a2880 0x7f81ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.399+0000 7f82037fe640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81ec0a2880 0x7f81ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.399+0000 7f82017fa640 1 -- 172.21.15.49:0/3060950114 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f81f40ced50 con 0x7f8204100700 2024-03-18T00:50:06.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.400+0000 7f82037fe640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81ec0a2880 0x7f81ec0a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f81f802c7b0 tx=0x7f81f8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:06.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.404+0000 7f82017fa640 1 -- 172.21.15.49:0/3060950114 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81f40d2030 con 0x7f8204100700 2024-03-18T00:50:06.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.590+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f82041a0420 con 0x7f8204100700 2024-03-18T00:50:06.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.591+0000 7f82017fa640 1 -- 172.21.15.49:0/3060950114 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f81f409bdd0 con 0x7f8204100700 2024-03-18T00:50:06.591 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 32 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81ec0a2880 msgr2=0x7f81ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81ec0a2880 0x7f81ec0a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f81f802c7b0 tx=0x7f81f8005fd0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8204100700 msgr2=0x7f8204197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8204100700 0x7f8204197b50 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f81f400e9c0 tx=0x7f81f400ee80 comp rx=0 tx=0).stop 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 shutdown_connections 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81ec0a2880 0x7f81ec0a4d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82041022b0 0x7f820419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82041018f0 0x7f8204198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 --2- 172.21.15.49:0/3060950114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8204100700 0x7f8204197b50 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 >> 172.21.15.49:0/3060950114 conn(0x7f82040fbc90 msgr2=0x7f8204069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 shutdown_connections 2024-03-18T00:50:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.593+0000 7f820a46e640 1 -- 172.21.15.49:0/3060950114 wait complete. 2024-03-18T00:50:06.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:06 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2767717748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:50:06.602 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:06 smithi049 ceph-mon[32130]: pgmap v350: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T00:50:06.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:06 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/717144606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:50:06.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:06 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1177206195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:06.603 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:06 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/327556985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:50:06.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T00:50:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:06 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2767717748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:50:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:06 smithi107 ceph-mon[36270]: pgmap v350: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T00:50:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:06 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/717144606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:50:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:06 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1177206195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:06.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:06 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/327556985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:50:06.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:06 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2767717748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:50:06.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:06 smithi106 ceph-mon[35899]: pgmap v350: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T00:50:06.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:06 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/717144606' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:50:06.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:06 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1177206195' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:06.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:06 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/327556985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:50:06.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.715+0000 7fa9acec6640 1 -- 172.21.15.49:0/1599974416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a8100700 msgr2=0x7fa9a8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.715+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1599974416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a8100700 0x7fa9a8100b20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa998009900 tx=0x7fa99802bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:06.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.715+0000 7fa9acec6640 1 -- 172.21.15.49:0/1599974416 shutdown_connections 2024-03-18T00:50:06.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.715+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1599974416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9a81018f0 0x7fa9a8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.715+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1599974416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a81022b0 0x7fa9a8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.715+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1599974416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a8100700 0x7fa9a8100b20 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.715+0000 7fa9acec6640 1 -- 172.21.15.49:0/1599974416 >> 172.21.15.49:0/1599974416 conn(0x7fa9a80fbc90 msgr2=0x7fa9a80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.715+0000 7fa9acec6640 1 -- 172.21.15.49:0/1599974416 shutdown_connections 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.715+0000 7fa9acec6640 1 -- 172.21.15.49:0/1599974416 wait complete. 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.716+0000 7fa9acec6640 1 Processor -- start 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.716+0000 7fa9acec6640 1 -- start start 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.716+0000 7fa9acec6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9a8100700 0x7fa9a8197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.716+0000 7fa9acec6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a81018f0 0x7fa9a81980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.716+0000 7fa9acec6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a81022b0 0x7fa9a819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.716+0000 7fa9acec6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a810c350 con 0x7fa9a81022b0 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.716+0000 7fa9acec6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a810c1d0 con 0x7fa9a8100700 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.716+0000 7fa9acec6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a810c4d0 con 0x7fa9a81018f0 2024-03-18T00:50:06.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a81022b0 0x7fa9a819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a81022b0 0x7fa9a819f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50050/0 (socket says 172.21.15.49:50050) 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a6d9d640 1 -- 172.21.15.49:0/1131757428 learned_addr learned my addr 172.21.15.49:0/1131757428 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a659c640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9a8100700 0x7fa9a8197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a5d9b640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a81018f0 0x7fa9a81980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a6d9d640 1 -- 172.21.15.49:0/1131757428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a81018f0 msgr2=0x7fa9a81980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a6d9d640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a81018f0 0x7fa9a81980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a6d9d640 1 -- 172.21.15.49:0/1131757428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9a8100700 msgr2=0x7fa9a8197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a6d9d640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9a8100700 0x7fa9a8197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a6d9d640 1 -- 172.21.15.49:0/1131757428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a819f770 con 0x7fa9a81022b0 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9a6d9d640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a81022b0 0x7fa9a819f100 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000eb10 tx=0x7fa9a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9977fe640 1 -- 172.21.15.49:0/1131757428 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0004b20 con 0x7fa9a81022b0 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9977fe640 1 -- 172.21.15.49:0/1131757428 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a0004cc0 con 0x7fa9a81022b0 2024-03-18T00:50:06.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a819fa60 con 0x7fa9a81022b0 2024-03-18T00:50:06.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9977fe640 1 -- 172.21.15.49:0/1131757428 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0010760 con 0x7fa9a81022b0 2024-03-18T00:50:06.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.717+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a819ff60 con 0x7fa9a81022b0 2024-03-18T00:50:06.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.718+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a81a0190 con 0x7fa9a81022b0 2024-03-18T00:50:06.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.719+0000 7fa9977fe640 1 -- 172.21.15.49:0/1131757428 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa9a0005020 con 0x7fa9a81022b0 2024-03-18T00:50:06.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.720+0000 7fa9977fe640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9900a2800 0x7fa9900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:06.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.720+0000 7fa9a659c640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9900a2800 0x7fa9900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:06.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.720+0000 7fa9977fe640 1 -- 172.21.15.49:0/1131757428 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa9a00ceff0 con 0x7fa9a81022b0 2024-03-18T00:50:06.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.720+0000 7fa9a659c640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9900a2800 0x7fa9900a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa998000c00 tx=0x7fa998005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:06.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.724+0000 7fa9977fe640 1 -- 172.21.15.49:0/1131757428 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa9a009c070 con 0x7fa9a81022b0 2024-03-18T00:50:06.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.909+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa9a81a0360 con 0x7fa9a81022b0 2024-03-18T00:50:06.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.909+0000 7fa9977fe640 1 -- 172.21.15.49:0/1131757428 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fa9a009b970 con 0x7fa9a81022b0 2024-03-18T00:50:06.909 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 64 2024-03-18T00:50:06.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9900a2800 msgr2=0x7fa9900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9900a2800 0x7fa9900a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa998000c00 tx=0x7fa998005fd0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a81022b0 msgr2=0x7fa9a819f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:06.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a81022b0 0x7fa9a819f100 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000eb10 tx=0x7fa9a000c2d0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 shutdown_connections 2024-03-18T00:50:06.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa9900a2800 0x7fa9900a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa9a81022b0 0x7fa9a819f100 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa9a81018f0 0x7fa9a81980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 --2- 172.21.15.49:0/1131757428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa9a8100700 0x7fa9a8197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:06.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 >> 172.21.15.49:0/1131757428 conn(0x7fa9a80fbc90 msgr2=0x7fa9a80fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:06.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 shutdown_connections 2024-03-18T00:50:06.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:06.911+0000 7fa9acec6640 1 -- 172.21.15.49:0/1131757428 wait complete. 2024-03-18T00:50:06.922 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-18T00:50:06.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:50:07.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.032+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1412189769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8b8100700 msgr2=0x7fe8b8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.032+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1412189769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8b8100700 0x7fe8b8100b20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009900 tx=0x7fe8ac02bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:07.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.032+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1412189769 shutdown_connections 2024-03-18T00:50:07.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.032+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1412189769 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8b81022b0 0x7fe8b8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.032+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1412189769 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8b81018f0 0x7fe8b8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.032+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1412189769 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8b8100700 0x7fe8b8100b20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.032+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1412189769 >> 172.21.15.49:0/1412189769 conn(0x7fe8b80fbc70 msgr2=0x7fe8b80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.032+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1412189769 shutdown_connections 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.032+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1412189769 wait complete. 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8bdbb2640 1 Processor -- start 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8bdbb2640 1 -- start start 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8bdbb2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8b8100700 0x7fe8b80694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8bdbb2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8b81018f0 0x7fe8b819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8bdbb2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8b81022b0 0x7fe8b819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8bdbb2640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b810c350 con 0x7fe8b81018f0 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8bdbb2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b810c1d0 con 0x7fe8b8100700 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8bdbb2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b810c4d0 con 0x7fe8b81022b0 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8b81018f0 0x7fe8b819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8b81018f0 0x7fe8b819c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50068/0 (socket says 172.21.15.49:50068) 2024-03-18T00:50:07.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b6ffd640 1 -- 172.21.15.49:0/1348078428 learned_addr learned my addr 172.21.15.49:0/1348078428 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b77fe640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8b8100700 0x7fe8b80694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b7fff640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8b81022b0 0x7fe8b819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b6ffd640 1 -- 172.21.15.49:0/1348078428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8b81022b0 msgr2=0x7fe8b819eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b6ffd640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8b81022b0 0x7fe8b819eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b6ffd640 1 -- 172.21.15.49:0/1348078428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8b8100700 msgr2=0x7fe8b80694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b6ffd640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8b8100700 0x7fe8b80694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b6ffd640 1 -- 172.21.15.49:0/1348078428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b819f7d0 con 0x7fe8b81018f0 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.033+0000 7fe8b77fe640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8b8100700 0x7fe8b80694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.034+0000 7fe8b6ffd640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8b81018f0 0x7fe8b819c8a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800e9f0 tx=0x7fe8a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.034+0000 7fe8b4ff9640 1 -- 172.21.15.49:0/1348078428 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a800ccc0 con 0x7fe8b81018f0 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.034+0000 7fe8b4ff9640 1 -- 172.21.15.49:0/1348078428 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8a8010910 con 0x7fe8b81018f0 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.034+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b819fa60 con 0x7fe8b81018f0 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.034+0000 7fe8b4ff9640 1 -- 172.21.15.49:0/1348078428 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a80188a0 con 0x7fe8b81018f0 2024-03-18T00:50:07.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.034+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b81a0000 con 0x7fe8b81018f0 2024-03-18T00:50:07.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.035+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8b8107ac0 con 0x7fe8b81018f0 2024-03-18T00:50:07.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.036+0000 7fe8b4ff9640 1 -- 172.21.15.49:0/1348078428 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe8a8010ac0 con 0x7fe8b81018f0 2024-03-18T00:50:07.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.037+0000 7fe8b4ff9640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8a00a2880 0x7fe8a00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.037+0000 7fe8b4ff9640 1 -- 172.21.15.49:0/1348078428 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe8a80cf2b0 con 0x7fe8b81018f0 2024-03-18T00:50:07.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.037+0000 7fe8b77fe640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8a00a2880 0x7fe8a00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.037+0000 7fe8b77fe640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8a00a2880 0x7fe8a00a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac000c00 tx=0x7fe8ac005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:07.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.040+0000 7fe8b4ff9640 1 -- 172.21.15.49:0/1348078428 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe8a809c330 con 0x7fe8b81018f0 2024-03-18T00:50:07.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.264+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8b802d1b0 con 0x7fe8b81018f0 2024-03-18T00:50:07.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.265+0000 7fe8b4ff9640 1 -- 172.21.15.49:0/1348078428 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11932 (secure 0 0 0) 0x7fe8a809bc30 con 0x7fe8b81018f0 2024-03-18T00:50:07.265 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:07.265 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:50:03.914926+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24463,24470,24481,24499,24506]},"id":1}]} 2024-03-18T00:50:07.265 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 25 2024-03-18T00:50:07.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8a00a2880 msgr2=0x7fe8a00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8a00a2880 0x7fe8a00a4d40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac000c00 tx=0x7fe8ac005fd0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8b81018f0 msgr2=0x7fe8b819c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8b81018f0 0x7fe8b819c8a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe8a800e9f0 tx=0x7fe8a800edf0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 shutdown_connections 2024-03-18T00:50:07.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8a00a2880 0x7fe8a00a4d40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe8b81022b0 0x7fe8b819eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe8b81018f0 0x7fe8b819c8a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 --2- 172.21.15.49:0/1348078428 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe8b8100700 0x7fe8b80694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 >> 172.21.15.49:0/1348078428 conn(0x7fe8b80fbc70 msgr2=0x7fe8b8104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:07.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 shutdown_connections 2024-03-18T00:50:07.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.267+0000 7fe8bdbb2640 1 -- 172.21.15.49:0/1348078428 wait complete. 2024-03-18T00:50:07.278 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-18T00:50:07.278 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-18T00:50:07.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T00:50:07.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.404+0000 7f8f9cbab640 1 -- 172.21.15.49:0/3203378550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981029f0 msgr2=0x7f8f98102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.404+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/3203378550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981029f0 0x7f8f98102e70 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009900 tx=0x7f8f8002bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:07.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.404+0000 7f8f9cbab640 1 -- 172.21.15.49:0/3203378550 shutdown_connections 2024-03-18T00:50:07.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.404+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/3203378550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f981033b0 0x7f8f98109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.404+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/3203378550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981029f0 0x7f8f98102e70 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.404+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/3203378550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f98101800 0x7f8f98101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.404+0000 7f8f9cbab640 1 -- 172.21.15.49:0/3203378550 >> 172.21.15.49:0/3203378550 conn(0x7f8f980fcd90 msgr2=0x7f8f980ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.404+0000 7f8f9cbab640 1 -- 172.21.15.49:0/3203378550 shutdown_connections 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.404+0000 7f8f9cbab640 1 -- 172.21.15.49:0/3203378550 wait complete. 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f9cbab640 1 Processor -- start 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f9cbab640 1 -- start start 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f9cbab640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f98101800 0x7f8f98197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f9cbab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f981029f0 0x7f8f98198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f9cbab640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981033b0 0x7f8f9819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f9cbab640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9810d480 con 0x7f8f981033b0 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f9cbab640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9810d300 con 0x7f8f98101800 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f9cbab640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9810d600 con 0x7f8f981029f0 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f96d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981033b0 0x7f8f9819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f9659c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f98101800 0x7f8f98197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f96d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981033b0 0x7f8f9819f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50082/0 (socket says 172.21.15.49:50082) 2024-03-18T00:50:07.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f8ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f981029f0 0x7f8f98198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.405+0000 7f8f96d9d640 1 -- 172.21.15.49:0/779508797 learned_addr learned my addr 172.21.15.49:0/779508797 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f96d9d640 1 -- 172.21.15.49:0/779508797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f981029f0 msgr2=0x7f8f98198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f96d9d640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f981029f0 0x7f8f98198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f96d9d640 1 -- 172.21.15.49:0/779508797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f98101800 msgr2=0x7f8f98197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f96d9d640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f98101800 0x7f8f98197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f96d9d640 1 -- 172.21.15.49:0/779508797 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9819f740 con 0x7f8f981033b0 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f9659c640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f98101800 0x7f8f98197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f96d9d640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981033b0 0x7f8f9819f0d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000eb10 tx=0x7f8f9000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f8f7fe640 1 -- 172.21.15.49:0/779508797 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f90004b20 con 0x7f8f981033b0 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f8f7fe640 1 -- 172.21.15.49:0/779508797 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f90004cc0 con 0x7f8f981033b0 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f98073a70 con 0x7f8f981033b0 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f8f7fe640 1 -- 172.21.15.49:0/779508797 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f90010760 con 0x7f8f981033b0 2024-03-18T00:50:07.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.406+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f98073f40 con 0x7f8f981033b0 2024-03-18T00:50:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.407+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f98108b20 con 0x7f8f981033b0 2024-03-18T00:50:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.408+0000 7f8f8f7fe640 1 -- 172.21.15.49:0/779508797 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8f90005020 con 0x7f8f981033b0 2024-03-18T00:50:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.409+0000 7f8f8f7fe640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f840a2800 0x7f8f840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.409+0000 7f8f9659c640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f840a2800 0x7f8f840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.409+0000 7f8f8f7fe640 1 -- 172.21.15.49:0/779508797 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8f900ceff0 con 0x7f8f981033b0 2024-03-18T00:50:07.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.409+0000 7f8f9659c640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f840a2800 0x7f8f840a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8f98102850 tx=0x7f8f88012040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:07.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.413+0000 7f8f8f7fe640 1 -- 172.21.15.49:0/779508797 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f9009c070 con 0x7f8f981033b0 2024-03-18T00:50:07.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.596+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8f98061c20 con 0x7f8f840a2800 2024-03-18T00:50:07.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.603+0000 7f8f8f7fe640 1 -- 172.21.15.49:0/779508797 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259166 (secure 0 0 0) 0x7f8f98061c20 con 0x7f8f840a2800 2024-03-18T00:50:07.607 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:07.608 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-03-18T00:50:07.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.606+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f840a2800 msgr2=0x7f8f840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.606+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f840a2800 0x7f8f840a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8f98102850 tx=0x7f8f88012040 comp rx=0 tx=0).stop 2024-03-18T00:50:07.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.606+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981033b0 msgr2=0x7f8f9819f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.606+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981033b0 0x7f8f9819f0d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000eb10 tx=0x7f8f9000c2d0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.607+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 shutdown_connections 2024-03-18T00:50:07.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.607+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f840a2800 0x7f8f840a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.607+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f981033b0 0x7f8f9819f0d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.607+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f981029f0 0x7f8f98198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.607+0000 7f8f9cbab640 1 --2- 172.21.15.49:0/779508797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f98101800 0x7f8f98197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.607+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 >> 172.21.15.49:0/779508797 conn(0x7f8f980fcd90 msgr2=0x7f8f98105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:07.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.607+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 shutdown_connections 2024-03-18T00:50:07.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.607+0000 7f8f9cbab640 1 -- 172.21.15.49:0/779508797 wait complete. 2024-03-18T00:50:07.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3060950114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:50:07.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1131757428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:50:07.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:07 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1348078428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:50:07.618 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-03-18T00:50:07.490063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663781,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1254,"num_read_kb":779,"num_write":435,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181973,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":857,"ondisk_log_size":857,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356796,"kb_used_data":11804,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585028,"statfs":{"total":1151940427776,"available":1151575068672,"internally_reserved":0,"allocated":12087296,"data_stored":7492589,"data_compressed":183621,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4930,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":123,"num_read_kb":62,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002823"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504686+0000","last_change":"2024-03-18T00:48:06.536034+0000","last_active":"2024-03-18T00:49:31.504686+0000","last_peered":"2024-03-18T00:49:31.504686+0000","last_clean":"2024-03-18T00:49:31.504686+0000","last_became_active":"2024-03-18T00:48:06.535694+0000","last_became_peered":"2024-03-18T00:48:06.535694+0000","last_unstale":"2024-03-18T00:49:31.504686+0000","last_undegraded":"2024-03-18T00:49:31.504686+0000","last_fullsized":"2024-03-18T00:49:31.504686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:24:33.785669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502451+0000","last_change":"2024-03-18T00:48:06.536741+0000","last_active":"2024-03-18T00:49:31.502451+0000","last_peered":"2024-03-18T00:49:31.502451+0000","last_clean":"2024-03-18T00:49:31.502451+0000","last_became_active":"2024-03-18T00:48:06.535865+0000","last_became_peered":"2024-03-18T00:48:06.535865+0000","last_unstale":"2024-03-18T00:49:31.502451+0000","last_undegraded":"2024-03-18T00:49:31.502451+0000","last_fullsized":"2024-03-18T00:49:31.502451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:58.104983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501939+0000","last_change":"2024-03-18T00:48:06.537162+0000","last_active":"2024-03-18T00:49:31.501939+0000","last_peered":"2024-03-18T00:49:31.501939+0000","last_clean":"2024-03-18T00:49:31.501939+0000","last_became_active":"2024-03-18T00:48:06.536835+0000","last_became_peered":"2024-03-18T00:48:06.536835+0000","last_unstale":"2024-03-18T00:49:31.501939+0000","last_undegraded":"2024-03-18T00:49:31.501939+0000","last_fullsized":"2024-03-18T00:49:31.501939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:15:08.519180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.486309+0000","last_change":"2024-03-18T00:48:06.540084+0000","last_active":"2024-03-18T00:49:31.486309+0000","last_peered":"2024-03-18T00:49:31.486309+0000","last_clean":"2024-03-18T00:49:31.486309+0000","last_became_active":"2024-03-18T00:48:06.539736+0000","last_became_peered":"2024-03-18T00:48:06.539736+0000","last_unstale":"2024-03-18T00:49:31.486309+0000","last_undegraded":"2024-03-18T00:49:31.486309+0000","last_fullsized":"2024-03-18T00:49:31.486309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:37:51.748603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497212+0000","last_change":"2024-03-18T00:48:06.540022+0000","last_active":"2024-03-18T00:49:31.497212+0000","last_peered":"2024-03-18T00:49:31.497212+0000","last_clean":"2024-03-18T00:49:31.497212+0000","last_became_active":"2024-03-18T00:48:06.539665+0000","last_became_peered":"2024-03-18T00:48:06.539665+0000","last_unstale":"2024-03-18T00:49:31.497212+0000","last_undegraded":"2024-03-18T00:49:31.497212+0000","last_fullsized":"2024-03-18T00:49:31.497212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:55:52.448279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506091+0000","last_change":"2024-03-18T00:48:06.538672+0000","last_active":"2024-03-18T00:49:31.506091+0000","last_peered":"2024-03-18T00:49:31.506091+0000","last_clean":"2024-03-18T00:49:31.506091+0000","last_became_active":"2024-03-18T00:48:06.538189+0000","last_became_peered":"2024-03-18T00:48:06.538189+0000","last_unstale":"2024-03-18T00:49:31.506091+0000","last_undegraded":"2024-03-18T00:49:31.506091+0000","last_fullsized":"2024-03-18T00:49:31.506091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:09:04.672255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506665+0000","last_change":"2024-03-18T00:48:06.546417+0000","last_active":"2024-03-18T00:49:31.506665+0000","last_peered":"2024-03-18T00:49:31.506665+0000","last_clean":"2024-03-18T00:49:31.506665+0000","last_became_active":"2024-03-18T00:48:06.545759+0000","last_became_peered":"2024-03-18T00:48:06.545759+0000","last_unstale":"2024-03-18T00:49:31.506665+0000","last_undegraded":"2024-03-18T00:49:31.506665+0000","last_fullsized":"2024-03-18T00:49:31.506665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:56.759028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.508198+0000","last_change":"2024-03-18T00:48:06.545875+0000","last_active":"2024-03-18T00:49:31.508198+0000","last_peered":"2024-03-18T00:49:31.508198+0000","last_clean":"2024-03-18T00:49:31.508198+0000","last_became_active":"2024-03-18T00:48:06.545316+0000","last_became_peered":"2024-03-18T00:48:06.545316+0000","last_unstale":"2024-03-18T00:49:31.508198+0000","last_undegraded":"2024-03-18T00:49:31.508198+0000","last_fullsized":"2024-03-18T00:49:31.508198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:22:40.998671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.512026+0000","last_change":"2024-03-18T00:48:06.533947+0000","last_active":"2024-03-18T00:49:31.512026+0000","last_peered":"2024-03-18T00:49:31.512026+0000","last_clean":"2024-03-18T00:49:31.512026+0000","last_became_active":"2024-03-18T00:48:06.533615+0000","last_became_peered":"2024-03-18T00:48:06.533615+0000","last_unstale":"2024-03-18T00:49:31.512026+0000","last_undegraded":"2024-03-18T00:49:31.512026+0000","last_fullsized":"2024-03-18T00:49:31.512026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:32:16.750269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507310+0000","last_change":"2024-03-18T00:48:06.537614+0000","last_active":"2024-03-18T00:49:31.507310+0000","last_peered":"2024-03-18T00:49:31.507310+0000","last_clean":"2024-03-18T00:49:31.507310+0000","last_became_active":"2024-03-18T00:48:06.536617+0000","last_became_peered":"2024-03-18T00:48:06.536617+0000","last_unstale":"2024-03-18T00:49:31.507310+0000","last_undegraded":"2024-03-18T00:49:31.507310+0000","last_fullsized":"2024-03-18T00:49:31.507310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:52.222603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502441+0000","last_change":"2024-03-18T00:48:06.540363+0000","last_active":"2024-03-18T00:49:31.502441+0000","last_peered":"2024-03-18T00:49:31.502441+0000","last_clean":"2024-03-18T00:49:31.502441+0000","last_became_active":"2024-03-18T00:48:06.540011+0000","last_became_peered":"2024-03-18T00:48:06.540011+0000","last_unstale":"2024-03-18T00:49:31.502441+0000","last_undegraded":"2024-03-18T00:49:31.502441+0000","last_fullsized":"2024-03-18T00:49:31.502441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:23:10.034444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504057+0000","last_change":"2024-03-18T00:48:06.532611+0000","last_active":"2024-03-18T00:49:31.504057+0000","last_peered":"2024-03-18T00:49:31.504057+0000","last_clean":"2024-03-18T00:49:31.504057+0000","last_became_active":"2024-03-18T00:48:06.532301+0000","last_became_peered":"2024-03-18T00:48:06.532301+0000","last_unstale":"2024-03-18T00:49:31.504057+0000","last_undegraded":"2024-03-18T00:49:31.504057+0000","last_fullsized":"2024-03-18T00:49:31.504057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:40.795105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.508264+0000","last_change":"2024-03-18T00:48:06.531993+0000","last_active":"2024-03-18T00:49:31.508264+0000","last_peered":"2024-03-18T00:49:31.508264+0000","last_clean":"2024-03-18T00:49:31.508264+0000","last_became_active":"2024-03-18T00:48:06.531677+0000","last_became_peered":"2024-03-18T00:48:06.531677+0000","last_unstale":"2024-03-18T00:49:31.508264+0000","last_undegraded":"2024-03-18T00:49:31.508264+0000","last_fullsized":"2024-03-18T00:49:31.508264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:48:24.865299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501483+0000","last_change":"2024-03-18T00:48:06.536830+0000","last_active":"2024-03-18T00:49:31.501483+0000","last_peered":"2024-03-18T00:49:31.501483+0000","last_clean":"2024-03-18T00:49:31.501483+0000","last_became_active":"2024-03-18T00:48:06.535908+0000","last_became_peered":"2024-03-18T00:48:06.535908+0000","last_unstale":"2024-03-18T00:49:31.501483+0000","last_undegraded":"2024-03-18T00:49:31.501483+0000","last_fullsized":"2024-03-18T00:49:31.501483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:23:26.155130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.493074+0000","last_change":"2024-03-18T00:48:06.543641+0000","last_active":"2024-03-18T00:49:31.493074+0000","last_peered":"2024-03-18T00:49:31.493074+0000","last_clean":"2024-03-18T00:49:31.493074+0000","last_became_active":"2024-03-18T00:48:06.543293+0000","last_became_peered":"2024-03-18T00:48:06.543293+0000","last_unstale":"2024-03-18T00:49:31.493074+0000","last_undegraded":"2024-03-18T00:49:31.493074+0000","last_fullsized":"2024-03-18T00:49:31.493074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:52.406255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.510399+0000","last_change":"2024-03-18T00:48:06.544732+0000","last_active":"2024-03-18T00:49:31.510399+0000","last_peered":"2024-03-18T00:49:31.510399+0000","last_clean":"2024-03-18T00:49:31.510399+0000","last_became_active":"2024-03-18T00:48:06.544400+0000","last_became_peered":"2024-03-18T00:48:06.544400+0000","last_unstale":"2024-03-18T00:49:31.510399+0000","last_undegraded":"2024-03-18T00:49:31.510399+0000","last_fullsized":"2024-03-18T00:49:31.510399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:23:39.655434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504899+0000","last_change":"2024-03-18T00:48:06.536595+0000","last_active":"2024-03-18T00:49:31.504899+0000","last_peered":"2024-03-18T00:49:31.504899+0000","last_clean":"2024-03-18T00:49:31.504899+0000","last_became_active":"2024-03-18T00:48:06.535874+0000","last_became_peered":"2024-03-18T00:48:06.535874+0000","last_unstale":"2024-03-18T00:49:31.504899+0000","last_undegraded":"2024-03-18T00:49:31.504899+0000","last_fullsized":"2024-03-18T00:49:31.504899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:55.603614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.515187+0000","last_change":"2024-03-18T00:48:06.546676+0000","last_active":"2024-03-18T00:49:31.515187+0000","last_peered":"2024-03-18T00:49:31.515187+0000","last_clean":"2024-03-18T00:49:31.515187+0000","last_became_active":"2024-03-18T00:48:06.546356+0000","last_became_peered":"2024-03-18T00:48:06.546356+0000","last_unstale":"2024-03-18T00:49:31.515187+0000","last_undegraded":"2024-03-18T00:49:31.515187+0000","last_fullsized":"2024-03-18T00:49:31.515187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:04:35.467415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500375+0000","last_change":"2024-03-18T00:48:06.532184+0000","last_active":"2024-03-18T00:49:31.500375+0000","last_peered":"2024-03-18T00:49:31.500375+0000","last_clean":"2024-03-18T00:49:31.500375+0000","last_became_active":"2024-03-18T00:48:06.531816+0000","last_became_peered":"2024-03-18T00:48:06.531816+0000","last_unstale":"2024-03-18T00:49:31.500375+0000","last_undegraded":"2024-03-18T00:49:31.500375+0000","last_fullsized":"2024-03-18T00:49:31.500375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:09:03.113024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":146,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:03.159413+0000","last_change":"2024-03-18T00:48:05.530213+0000","last_active":"2024-03-18T00:50:03.159413+0000","last_peered":"2024-03-18T00:50:03.159413+0000","last_clean":"2024-03-18T00:50:03.159413+0000","last_became_active":"2024-03-18T00:48:05.529897+0000","last_became_peered":"2024-03-18T00:48:05.529897+0000","last_unstale":"2024-03-18T00:50:03.159413+0000","last_undegraded":"2024-03-18T00:50:03.159413+0000","last_fullsized":"2024-03-18T00:50:03.159413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:49.003362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506113+0000","last_change":"2024-03-18T00:48:06.544437+0000","last_active":"2024-03-18T00:49:31.506113+0000","last_peered":"2024-03-18T00:49:31.506113+0000","last_clean":"2024-03-18T00:49:31.506113+0000","last_became_active":"2024-03-18T00:48:06.544125+0000","last_became_peered":"2024-03-18T00:48:06.544125+0000","last_unstale":"2024-03-18T00:49:31.506113+0000","last_undegraded":"2024-03-18T00:49:31.506113+0000","last_fullsized":"2024-03-18T00:49:31.506113+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:56:42.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325994+0000","last_change":"2024-03-18T00:48:05.536249+0000","last_active":"2024-03-18T00:49:03.325994+0000","last_peered":"2024-03-18T00:49:03.325994+0000","last_clean":"2024-03-18T00:49:03.325994+0000","last_became_active":"2024-03-18T00:48:05.535954+0000","last_became_peered":"2024-03-18T00:48:05.535954+0000","last_unstale":"2024-03-18T00:49:03.325994+0000","last_undegraded":"2024-03-18T00:49:03.325994+0000","last_fullsized":"2024-03-18T00:49:03.325994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:18:52.341453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497426+0000","last_change":"2024-03-18T00:48:06.541377+0000","last_active":"2024-03-18T00:49:31.497426+0000","last_peered":"2024-03-18T00:49:31.497426+0000","last_clean":"2024-03-18T00:49:31.497426+0000","last_became_active":"2024-03-18T00:48:06.540937+0000","last_became_peered":"2024-03-18T00:48:06.540937+0000","last_unstale":"2024-03-18T00:49:31.497426+0000","last_undegraded":"2024-03-18T00:49:31.497426+0000","last_fullsized":"2024-03-18T00:49:31.497426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:51:55.268517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088568+0000","last_change":"2024-03-18T00:48:05.534372+0000","last_active":"2024-03-18T00:49:03.088568+0000","last_peered":"2024-03-18T00:49:03.088568+0000","last_clean":"2024-03-18T00:49:03.088568+0000","last_became_active":"2024-03-18T00:48:05.534034+0000","last_became_peered":"2024-03-18T00:48:05.534034+0000","last_unstale":"2024-03-18T00:49:03.088568+0000","last_undegraded":"2024-03-18T00:49:03.088568+0000","last_fullsized":"2024-03-18T00:49:03.088568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:58:55.013167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.513707+0000","last_change":"2024-03-18T00:48:06.527276+0000","last_active":"2024-03-18T00:49:31.513707+0000","last_peered":"2024-03-18T00:49:31.513707+0000","last_clean":"2024-03-18T00:49:31.513707+0000","last_became_active":"2024-03-18T00:48:06.526904+0000","last_became_peered":"2024-03-18T00:48:06.526904+0000","last_unstale":"2024-03-18T00:49:31.513707+0000","last_undegraded":"2024-03-18T00:49:31.513707+0000","last_fullsized":"2024-03-18T00:49:31.513707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:16:10.731256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087607+0000","last_change":"2024-03-18T00:48:05.528188+0000","last_active":"2024-03-18T00:49:03.087607+0000","last_peered":"2024-03-18T00:49:03.087607+0000","last_clean":"2024-03-18T00:49:03.087607+0000","last_became_active":"2024-03-18T00:48:05.527788+0000","last_became_peered":"2024-03-18T00:48:05.527788+0000","last_unstale":"2024-03-18T00:49:03.087607+0000","last_undegraded":"2024-03-18T00:49:03.087607+0000","last_fullsized":"2024-03-18T00:49:03.087607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:47.939771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507894+0000","last_change":"2024-03-18T00:49:03.092920+0000","last_active":"2024-03-18T00:49:31.507894+0000","last_peered":"2024-03-18T00:49:31.507894+0000","last_clean":"2024-03-18T00:49:31.507894+0000","last_became_active":"2024-03-18T00:49:03.092569+0000","last_became_peered":"2024-03-18T00:49:03.092569+0000","last_unstale":"2024-03-18T00:49:31.507894+0000","last_undegraded":"2024-03-18T00:49:31.507894+0000","last_fullsized":"2024-03-18T00:49:31.507894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:15:02.246688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'8","reported_seq":85,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.155786+0000","last_change":"2024-03-18T00:49:03.100514+0000","last_active":"2024-03-18T00:50:05.155786+0000","last_peered":"2024-03-18T00:50:05.155786+0000","last_clean":"2024-03-18T00:50:05.155786+0000","last_became_active":"2024-03-18T00:49:03.090898+0000","last_became_peered":"2024-03-18T00:49:03.090898+0000","last_unstale":"2024-03-18T00:50:05.155786+0000","last_undegraded":"2024-03-18T00:50:05.155786+0000","last_fullsized":"2024-03-18T00:50:05.155786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:19:39.943819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.494376+0000","last_change":"2024-03-18T00:48:06.537405+0000","last_active":"2024-03-18T00:49:31.494376+0000","last_peered":"2024-03-18T00:49:31.494376+0000","last_clean":"2024-03-18T00:49:31.494376+0000","last_became_active":"2024-03-18T00:48:06.536919+0000","last_became_peered":"2024-03-18T00:48:06.536919+0000","last_unstale":"2024-03-18T00:49:31.494376+0000","last_undegraded":"2024-03-18T00:49:31.494376+0000","last_fullsized":"2024-03-18T00:49:31.494376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:31:59.863571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088122+0000","last_change":"2024-03-18T00:48:05.528588+0000","last_active":"2024-03-18T00:49:03.088122+0000","last_peered":"2024-03-18T00:49:03.088122+0000","last_clean":"2024-03-18T00:49:03.088122+0000","last_became_active":"2024-03-18T00:48:05.528263+0000","last_became_peered":"2024-03-18T00:48:05.528263+0000","last_unstale":"2024-03-18T00:49:03.088122+0000","last_undegraded":"2024-03-18T00:49:03.088122+0000","last_fullsized":"2024-03-18T00:49:03.088122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:18:10.283278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506438+0000","last_change":"2024-03-18T00:48:06.546323+0000","last_active":"2024-03-18T00:49:31.506438+0000","last_peered":"2024-03-18T00:49:31.506438+0000","last_clean":"2024-03-18T00:49:31.506438+0000","last_became_active":"2024-03-18T00:48:06.545956+0000","last_became_peered":"2024-03-18T00:48:06.545956+0000","last_unstale":"2024-03-18T00:49:31.506438+0000","last_undegraded":"2024-03-18T00:49:31.506438+0000","last_fullsized":"2024-03-18T00:49:31.506438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:32.618724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087983+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:49:03.087983+0000","last_peered":"2024-03-18T00:49:03.087983+0000","last_clean":"2024-03-18T00:49:03.087983+0000","last_became_active":"2024-03-18T00:48:05.525590+0000","last_became_peered":"2024-03-18T00:48:05.525590+0000","last_unstale":"2024-03-18T00:49:03.087983+0000","last_undegraded":"2024-03-18T00:49:03.087983+0000","last_fullsized":"2024-03-18T00:49:03.087983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:19.357331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502517+0000","last_change":"2024-03-18T00:48:06.538798+0000","last_active":"2024-03-18T00:49:31.502517+0000","last_peered":"2024-03-18T00:49:31.502517+0000","last_clean":"2024-03-18T00:49:31.502517+0000","last_became_active":"2024-03-18T00:48:06.538457+0000","last_became_peered":"2024-03-18T00:48:06.538457+0000","last_unstale":"2024-03-18T00:49:31.502517+0000","last_undegraded":"2024-03-18T00:49:31.502517+0000","last_fullsized":"2024-03-18T00:49:31.502517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:10.738683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088047+0000","last_change":"2024-03-18T00:48:05.528094+0000","last_active":"2024-03-18T00:49:03.088047+0000","last_peered":"2024-03-18T00:49:03.088047+0000","last_clean":"2024-03-18T00:49:03.088047+0000","last_became_active":"2024-03-18T00:48:05.527730+0000","last_became_peered":"2024-03-18T00:48:05.527730+0000","last_unstale":"2024-03-18T00:49:03.088047+0000","last_undegraded":"2024-03-18T00:49:03.088047+0000","last_fullsized":"2024-03-18T00:49:03.088047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:48:27.707505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.498967+0000","last_change":"2024-03-18T00:48:06.548676+0000","last_active":"2024-03-18T00:49:31.498967+0000","last_peered":"2024-03-18T00:49:31.498967+0000","last_clean":"2024-03-18T00:49:31.498967+0000","last_became_active":"2024-03-18T00:48:06.547693+0000","last_became_peered":"2024-03-18T00:48:06.547693+0000","last_unstale":"2024-03-18T00:49:31.498967+0000","last_undegraded":"2024-03-18T00:49:31.498967+0000","last_fullsized":"2024-03-18T00:49:31.498967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:13.849175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'6","reported_seq":253,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.154991+0000","last_change":"2024-03-18T00:48:05.523639+0000","last_active":"2024-03-18T00:50:05.154991+0000","last_peered":"2024-03-18T00:50:05.154991+0000","last_clean":"2024-03-18T00:50:05.154991+0000","last_became_active":"2024-03-18T00:48:05.523269+0000","last_became_peered":"2024-03-18T00:48:05.523269+0000","last_unstale":"2024-03-18T00:50:05.154991+0000","last_undegraded":"2024-03-18T00:50:05.154991+0000","last_fullsized":"2024-03-18T00:50:05.154991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:39.961247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":228,"num_read_kb":228,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.514641+0000","last_change":"2024-03-18T00:48:06.537786+0000","last_active":"2024-03-18T00:49:31.514641+0000","last_peered":"2024-03-18T00:49:31.514641+0000","last_clean":"2024-03-18T00:49:31.514641+0000","last_became_active":"2024-03-18T00:48:06.537488+0000","last_became_peered":"2024-03-18T00:48:06.537488+0000","last_unstale":"2024-03-18T00:49:31.514641+0000","last_undegraded":"2024-03-18T00:49:31.514641+0000","last_fullsized":"2024-03-18T00:49:31.514641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:45:02.095610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.096001+0000","last_change":"2024-03-18T00:49:03.096001+0000","last_active":"2024-03-18T00:49:03.096001+0000","last_peered":"2024-03-18T00:49:03.096001+0000","last_clean":"2024-03-18T00:49:03.096001+0000","last_became_active":"2024-03-18T00:49:03.090868+0000","last_became_peered":"2024-03-18T00:49:03.090868+0000","last_unstale":"2024-03-18T00:49:03.096001+0000","last_undegraded":"2024-03-18T00:49:03.096001+0000","last_fullsized":"2024-03-18T00:49:03.096001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:03.930010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.494451+0000","last_change":"2024-03-18T00:48:06.545416+0000","last_active":"2024-03-18T00:49:31.494451+0000","last_peered":"2024-03-18T00:49:31.494451+0000","last_clean":"2024-03-18T00:49:31.494451+0000","last_became_active":"2024-03-18T00:48:06.545087+0000","last_became_peered":"2024-03-18T00:48:06.545087+0000","last_unstale":"2024-03-18T00:49:31.494451+0000","last_undegraded":"2024-03-18T00:49:31.494451+0000","last_fullsized":"2024-03-18T00:49:31.494451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:02:25.309990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089159+0000","last_change":"2024-03-18T00:48:05.533088+0000","last_active":"2024-03-18T00:49:03.089159+0000","last_peered":"2024-03-18T00:49:03.089159+0000","last_clean":"2024-03-18T00:49:03.089159+0000","last_became_active":"2024-03-18T00:48:05.532308+0000","last_became_peered":"2024-03-18T00:48:05.532308+0000","last_unstale":"2024-03-18T00:49:03.089159+0000","last_undegraded":"2024-03-18T00:49:03.089159+0000","last_fullsized":"2024-03-18T00:49:03.089159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:35.071283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.513851+0000","last_change":"2024-03-18T00:49:03.092567+0000","last_active":"2024-03-18T00:49:31.513851+0000","last_peered":"2024-03-18T00:49:31.513851+0000","last_clean":"2024-03-18T00:49:31.513851+0000","last_became_active":"2024-03-18T00:49:03.092297+0000","last_became_peered":"2024-03-18T00:49:03.092297+0000","last_unstale":"2024-03-18T00:49:31.513851+0000","last_undegraded":"2024-03-18T00:49:31.513851+0000","last_fullsized":"2024-03-18T00:49:31.513851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:44.567584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'36","reported_seq":292,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.147485+0000","last_change":"2024-03-18T00:48:05.527557+0000","last_active":"2024-03-18T00:50:05.147485+0000","last_peered":"2024-03-18T00:50:05.147485+0000","last_clean":"2024-03-18T00:50:05.147485+0000","last_became_active":"2024-03-18T00:48:05.527167+0000","last_became_peered":"2024-03-18T00:48:05.527167+0000","last_unstale":"2024-03-18T00:50:05.147485+0000","last_undegraded":"2024-03-18T00:50:05.147485+0000","last_fullsized":"2024-03-18T00:50:05.147485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:02:04.105091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65917,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":184,"num_write":38,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.499066+0000","last_change":"2024-03-18T00:48:06.543153+0000","last_active":"2024-03-18T00:49:31.499066+0000","last_peered":"2024-03-18T00:49:31.499066+0000","last_clean":"2024-03-18T00:49:31.499066+0000","last_became_active":"2024-03-18T00:48:06.541353+0000","last_became_peered":"2024-03-18T00:48:06.541353+0000","last_unstale":"2024-03-18T00:49:31.499066+0000","last_undegraded":"2024-03-18T00:49:31.499066+0000","last_fullsized":"2024-03-18T00:49:31.499066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:35.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086981+0000","last_change":"2024-03-18T00:48:05.530560+0000","last_active":"2024-03-18T00:49:03.086981+0000","last_peered":"2024-03-18T00:49:03.086981+0000","last_clean":"2024-03-18T00:49:03.086981+0000","last_became_active":"2024-03-18T00:48:05.528858+0000","last_became_peered":"2024-03-18T00:48:05.528858+0000","last_unstale":"2024-03-18T00:49:03.086981+0000","last_undegraded":"2024-03-18T00:49:03.086981+0000","last_fullsized":"2024-03-18T00:49:03.086981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:03:01.496919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497847+0000","last_change":"2024-03-18T00:49:03.090951+0000","last_active":"2024-03-18T00:49:31.497847+0000","last_peered":"2024-03-18T00:49:31.497847+0000","last_clean":"2024-03-18T00:49:31.497847+0000","last_became_active":"2024-03-18T00:49:03.090578+0000","last_became_peered":"2024-03-18T00:49:03.090578+0000","last_unstale":"2024-03-18T00:49:31.497847+0000","last_undegraded":"2024-03-18T00:49:31.497847+0000","last_fullsized":"2024-03-18T00:49:31.497847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:04:12.703545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088756+0000","last_change":"2024-03-18T00:48:05.527902+0000","last_active":"2024-03-18T00:49:03.088756+0000","last_peered":"2024-03-18T00:49:03.088756+0000","last_clean":"2024-03-18T00:49:03.088756+0000","last_became_active":"2024-03-18T00:48:05.527495+0000","last_became_peered":"2024-03-18T00:48:05.527495+0000","last_unstale":"2024-03-18T00:49:03.088756+0000","last_undegraded":"2024-03-18T00:49:03.088756+0000","last_fullsized":"2024-03-18T00:49:03.088756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:25.504652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504404+0000","last_change":"2024-03-18T00:48:06.550413+0000","last_active":"2024-03-18T00:49:31.504404+0000","last_peered":"2024-03-18T00:49:31.504404+0000","last_clean":"2024-03-18T00:49:31.504404+0000","last_became_active":"2024-03-18T00:48:06.547727+0000","last_became_peered":"2024-03-18T00:48:06.547727+0000","last_unstale":"2024-03-18T00:49:31.504404+0000","last_undegraded":"2024-03-18T00:49:31.504404+0000","last_fullsized":"2024-03-18T00:49:31.504404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:36:48.878417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.146430+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:50:05.146430+0000","last_peered":"2024-03-18T00:50:05.146430+0000","last_clean":"2024-03-18T00:50:05.146430+0000","last_became_active":"2024-03-18T00:48:05.525607+0000","last_became_peered":"2024-03-18T00:48:05.525607+0000","last_unstale":"2024-03-18T00:50:05.146430+0000","last_undegraded":"2024-03-18T00:50:05.146430+0000","last_fullsized":"2024-03-18T00:50:05.146430+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:32.457447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.498511+0000","last_change":"2024-03-18T00:49:03.091221+0000","last_active":"2024-03-18T00:49:31.498511+0000","last_peered":"2024-03-18T00:49:31.498511+0000","last_clean":"2024-03-18T00:49:31.498511+0000","last_became_active":"2024-03-18T00:49:03.090559+0000","last_became_peered":"2024-03-18T00:49:03.090559+0000","last_unstale":"2024-03-18T00:49:31.498511+0000","last_undegraded":"2024-03-18T00:49:31.498511+0000","last_fullsized":"2024-03-18T00:49:31.498511+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:02:54.890187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.417640+0000","last_change":"2024-03-18T00:49:03.417640+0000","last_active":"2024-03-18T00:49:03.417640+0000","last_peered":"2024-03-18T00:49:03.417640+0000","last_clean":"2024-03-18T00:49:03.417640+0000","last_became_active":"2024-03-18T00:49:03.405217+0000","last_became_peered":"2024-03-18T00:49:03.405217+0000","last_unstale":"2024-03-18T00:49:03.417640+0000","last_undegraded":"2024-03-18T00:49:03.417640+0000","last_fullsized":"2024-03-18T00:49:03.417640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:00:13.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507763+0000","last_change":"2024-03-18T00:48:06.544084+0000","last_active":"2024-03-18T00:49:31.507763+0000","last_peered":"2024-03-18T00:49:31.507763+0000","last_clean":"2024-03-18T00:49:31.507763+0000","last_became_active":"2024-03-18T00:48:06.543419+0000","last_became_peered":"2024-03-18T00:48:06.543419+0000","last_unstale":"2024-03-18T00:49:31.507763+0000","last_undegraded":"2024-03-18T00:49:31.507763+0000","last_fullsized":"2024-03-18T00:49:31.507763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:15:28.527473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087635+0000","last_change":"2024-03-18T00:48:05.528347+0000","last_active":"2024-03-18T00:49:03.087635+0000","last_peered":"2024-03-18T00:49:03.087635+0000","last_clean":"2024-03-18T00:49:03.087635+0000","last_became_active":"2024-03-18T00:48:05.527761+0000","last_became_peered":"2024-03-18T00:48:05.527761+0000","last_unstale":"2024-03-18T00:49:03.087635+0000","last_undegraded":"2024-03-18T00:49:03.087635+0000","last_fullsized":"2024-03-18T00:49:03.087635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:36:58.562495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.503220+0000","last_change":"2024-03-18T00:48:06.541862+0000","last_active":"2024-03-18T00:49:31.503220+0000","last_peered":"2024-03-18T00:49:31.503220+0000","last_clean":"2024-03-18T00:49:31.503220+0000","last_became_active":"2024-03-18T00:48:06.541488+0000","last_became_peered":"2024-03-18T00:48:06.541488+0000","last_unstale":"2024-03-18T00:49:31.503220+0000","last_undegraded":"2024-03-18T00:49:31.503220+0000","last_fullsized":"2024-03-18T00:49:31.503220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:47:40.724244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084618+0000","last_change":"2024-03-18T00:48:05.535498+0000","last_active":"2024-03-18T00:49:03.084618+0000","last_peered":"2024-03-18T00:49:03.084618+0000","last_clean":"2024-03-18T00:49:03.084618+0000","last_became_active":"2024-03-18T00:48:05.534907+0000","last_became_peered":"2024-03-18T00:48:05.534907+0000","last_unstale":"2024-03-18T00:49:03.084618+0000","last_undegraded":"2024-03-18T00:49:03.084618+0000","last_fullsized":"2024-03-18T00:49:03.084618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:47:50.196929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504894+0000","last_change":"2024-03-18T00:48:06.534423+0000","last_active":"2024-03-18T00:49:31.504894+0000","last_peered":"2024-03-18T00:49:31.504894+0000","last_clean":"2024-03-18T00:49:31.504894+0000","last_became_active":"2024-03-18T00:48:06.534091+0000","last_became_peered":"2024-03-18T00:48:06.534091+0000","last_unstale":"2024-03-18T00:49:31.504894+0000","last_undegraded":"2024-03-18T00:49:31.504894+0000","last_fullsized":"2024-03-18T00:49:31.504894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:39:08.755634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:01.157977+0000","last_change":"2024-03-18T00:48:05.532891+0000","last_active":"2024-03-18T00:50:01.157977+0000","last_peered":"2024-03-18T00:50:01.157977+0000","last_clean":"2024-03-18T00:50:01.157977+0000","last_became_active":"2024-03-18T00:48:05.532515+0000","last_became_peered":"2024-03-18T00:48:05.532515+0000","last_unstale":"2024-03-18T00:50:01.157977+0000","last_undegraded":"2024-03-18T00:50:01.157977+0000","last_fullsized":"2024-03-18T00:50:01.157977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:46:52.056837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506095+0000","last_change":"2024-03-18T00:48:06.542992+0000","last_active":"2024-03-18T00:49:31.506095+0000","last_peered":"2024-03-18T00:49:31.506095+0000","last_clean":"2024-03-18T00:49:31.506095+0000","last_became_active":"2024-03-18T00:48:06.542617+0000","last_became_peered":"2024-03-18T00:48:06.542617+0000","last_unstale":"2024-03-18T00:49:31.506095+0000","last_undegraded":"2024-03-18T00:49:31.506095+0000","last_fullsized":"2024-03-18T00:49:31.506095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:22.736118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506043+0000","last_change":"2024-03-18T00:48:06.536018+0000","last_active":"2024-03-18T00:49:31.506043+0000","last_peered":"2024-03-18T00:49:31.506043+0000","last_clean":"2024-03-18T00:49:31.506043+0000","last_became_active":"2024-03-18T00:48:06.535656+0000","last_became_peered":"2024-03-18T00:48:06.535656+0000","last_unstale":"2024-03-18T00:49:31.506043+0000","last_undegraded":"2024-03-18T00:49:31.506043+0000","last_fullsized":"2024-03-18T00:49:31.506043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:33:00.492887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:06.146970+0000","last_change":"2024-03-18T00:48:05.536635+0000","last_active":"2024-03-18T00:50:06.146970+0000","last_peered":"2024-03-18T00:50:06.146970+0000","last_clean":"2024-03-18T00:50:06.146970+0000","last_became_active":"2024-03-18T00:48:05.535941+0000","last_became_peered":"2024-03-18T00:48:05.535941+0000","last_unstale":"2024-03-18T00:50:06.146970+0000","last_undegraded":"2024-03-18T00:50:06.146970+0000","last_fullsized":"2024-03-18T00:50:06.146970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:20:52.167217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.515797+0000","last_change":"2024-03-18T00:48:06.546266+0000","last_active":"2024-03-18T00:49:31.515797+0000","last_peered":"2024-03-18T00:49:31.515797+0000","last_clean":"2024-03-18T00:49:31.515797+0000","last_became_active":"2024-03-18T00:48:06.543565+0000","last_became_peered":"2024-03-18T00:48:06.543565+0000","last_unstale":"2024-03-18T00:49:31.515797+0000","last_undegraded":"2024-03-18T00:49:31.515797+0000","last_fullsized":"2024-03-18T00:49:31.515797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:49:09.673212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.403989+0000","last_change":"2024-03-18T00:48:05.526184+0000","last_active":"2024-03-18T00:49:03.403989+0000","last_peered":"2024-03-18T00:49:03.403989+0000","last_clean":"2024-03-18T00:49:03.403989+0000","last_became_active":"2024-03-18T00:48:05.525011+0000","last_became_peered":"2024-03-18T00:48:05.525011+0000","last_unstale":"2024-03-18T00:49:03.403989+0000","last_undegraded":"2024-03-18T00:49:03.403989+0000","last_fullsized":"2024-03-18T00:49:03.403989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:12.690966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087462+0000","last_change":"2024-03-18T00:46:53.288768+0000","last_active":"2024-03-18T00:49:03.087462+0000","last_peered":"2024-03-18T00:49:03.087462+0000","last_clean":"2024-03-18T00:49:03.087462+0000","last_became_active":"2024-03-18T00:46:53.288394+0000","last_became_peered":"2024-03-18T00:46:53.288394+0000","last_unstale":"2024-03-18T00:49:03.087462+0000","last_undegraded":"2024-03-18T00:49:03.087462+0000","last_fullsized":"2024-03-18T00:49:03.087462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_clean_scrub_stamp":"2024-03-18T00:43:28.173150+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:49.949940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500421+0000","last_change":"2024-03-18T00:48:06.534383+0000","last_active":"2024-03-18T00:49:31.500421+0000","last_peered":"2024-03-18T00:49:31.500421+0000","last_clean":"2024-03-18T00:49:31.500421+0000","last_became_active":"2024-03-18T00:48:06.534085+0000","last_became_peered":"2024-03-18T00:48:06.534085+0000","last_unstale":"2024-03-18T00:49:31.500421+0000","last_undegraded":"2024-03-18T00:49:31.500421+0000","last_fullsized":"2024-03-18T00:49:31.500421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:54.021830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.407661+0000","last_change":"2024-03-18T00:49:03.407661+0000","last_active":"2024-03-18T00:49:03.407661+0000","last_peered":"2024-03-18T00:49:03.407661+0000","last_clean":"2024-03-18T00:49:03.407661+0000","last_became_active":"2024-03-18T00:49:03.405134+0000","last_became_peered":"2024-03-18T00:49:03.405134+0000","last_unstale":"2024-03-18T00:49:03.407661+0000","last_undegraded":"2024-03-18T00:49:03.407661+0000","last_fullsized":"2024-03-18T00:49:03.407661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:29:03.201556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.098300+0000","last_change":"2024-03-18T00:49:03.097885+0000","last_active":"2024-03-18T00:49:03.098300+0000","last_peered":"2024-03-18T00:49:03.098300+0000","last_clean":"2024-03-18T00:49:03.098300+0000","last_became_active":"2024-03-18T00:49:03.090574+0000","last_became_peered":"2024-03-18T00:49:03.090574+0000","last_unstale":"2024-03-18T00:49:03.098300+0000","last_undegraded":"2024-03-18T00:49:03.098300+0000","last_fullsized":"2024-03-18T00:49:03.098300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:22:30.194349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.499871+0000","last_change":"2024-03-18T00:48:06.530108+0000","last_active":"2024-03-18T00:49:31.499871+0000","last_peered":"2024-03-18T00:49:31.499871+0000","last_clean":"2024-03-18T00:49:31.499871+0000","last_became_active":"2024-03-18T00:48:06.529768+0000","last_became_peered":"2024-03-18T00:48:06.529768+0000","last_unstale":"2024-03-18T00:49:31.499871+0000","last_undegraded":"2024-03-18T00:49:31.499871+0000","last_fullsized":"2024-03-18T00:49:31.499871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:23:09.150305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'14","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.529747+0000","last_change":"2024-03-18T00:48:05.535703+0000","last_active":"2024-03-18T00:49:31.529747+0000","last_peered":"2024-03-18T00:49:31.529747+0000","last_clean":"2024-03-18T00:49:31.529747+0000","last_became_active":"2024-03-18T00:48:05.535086+0000","last_became_peered":"2024-03-18T00:48:05.535086+0000","last_unstale":"2024-03-18T00:49:31.529747+0000","last_undegraded":"2024-03-18T00:49:31.529747+0000","last_fullsized":"2024-03-18T00:49:31.529747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:10:24.172353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504017+0000","last_change":"2024-03-18T00:48:06.541281+0000","last_active":"2024-03-18T00:49:31.504017+0000","last_peered":"2024-03-18T00:49:31.504017+0000","last_clean":"2024-03-18T00:49:31.504017+0000","last_became_active":"2024-03-18T00:48:06.540946+0000","last_became_peered":"2024-03-18T00:48:06.540946+0000","last_unstale":"2024-03-18T00:49:31.504017+0000","last_undegraded":"2024-03-18T00:49:31.504017+0000","last_fullsized":"2024-03-18T00:49:31.504017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:02.987661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.404075+0000","last_change":"2024-03-18T00:48:05.531685+0000","last_active":"2024-03-18T00:49:03.404075+0000","last_peered":"2024-03-18T00:49:03.404075+0000","last_clean":"2024-03-18T00:49:03.404075+0000","last_became_active":"2024-03-18T00:48:05.531239+0000","last_became_peered":"2024-03-18T00:48:05.531239+0000","last_unstale":"2024-03-18T00:49:03.404075+0000","last_undegraded":"2024-03-18T00:49:03.404075+0000","last_fullsized":"2024-03-18T00:49:03.404075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:26:35.034842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502223+0000","last_change":"2024-03-18T00:48:06.546783+0000","last_active":"2024-03-18T00:49:31.502223+0000","last_peered":"2024-03-18T00:49:31.502223+0000","last_clean":"2024-03-18T00:49:31.502223+0000","last_became_active":"2024-03-18T00:48:06.546411+0000","last_became_peered":"2024-03-18T00:48:06.546411+0000","last_unstale":"2024-03-18T00:49:31.502223+0000","last_undegraded":"2024-03-18T00:49:31.502223+0000","last_fullsized":"2024-03-18T00:49:31.502223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:39:34.432104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089245+0000","last_change":"2024-03-18T00:48:05.534701+0000","last_active":"2024-03-18T00:49:03.089245+0000","last_peered":"2024-03-18T00:49:03.089245+0000","last_clean":"2024-03-18T00:49:03.089245+0000","last_became_active":"2024-03-18T00:48:05.534300+0000","last_became_peered":"2024-03-18T00:48:05.534300+0000","last_unstale":"2024-03-18T00:49:03.089245+0000","last_undegraded":"2024-03-18T00:49:03.089245+0000","last_fullsized":"2024-03-18T00:49:03.089245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:10.580001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.461798+0000","last_change":"2024-03-18T00:48:06.528286+0000","last_active":"2024-03-18T00:49:31.461798+0000","last_peered":"2024-03-18T00:49:31.461798+0000","last_clean":"2024-03-18T00:49:31.461798+0000","last_became_active":"2024-03-18T00:48:06.527947+0000","last_became_peered":"2024-03-18T00:48:06.527947+0000","last_unstale":"2024-03-18T00:49:31.461798+0000","last_undegraded":"2024-03-18T00:49:31.461798+0000","last_fullsized":"2024-03-18T00:49:31.461798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:34.953923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325566+0000","last_change":"2024-03-18T00:48:05.526530+0000","last_active":"2024-03-18T00:49:03.325566+0000","last_peered":"2024-03-18T00:49:03.325566+0000","last_clean":"2024-03-18T00:49:03.325566+0000","last_became_active":"2024-03-18T00:48:05.526210+0000","last_became_peered":"2024-03-18T00:48:05.526210+0000","last_unstale":"2024-03-18T00:49:03.325566+0000","last_undegraded":"2024-03-18T00:49:03.325566+0000","last_fullsized":"2024-03-18T00:49:03.325566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:11:11.094469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501176+0000","last_change":"2024-03-18T00:48:06.535316+0000","last_active":"2024-03-18T00:49:31.501176+0000","last_peered":"2024-03-18T00:49:31.501176+0000","last_clean":"2024-03-18T00:49:31.501176+0000","last_became_active":"2024-03-18T00:48:06.534945+0000","last_became_peered":"2024-03-18T00:48:06.534945+0000","last_unstale":"2024-03-18T00:49:31.501176+0000","last_undegraded":"2024-03-18T00:49:31.501176+0000","last_fullsized":"2024-03-18T00:49:31.501176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:04:32.570585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087310+0000","last_change":"2024-03-18T00:48:05.529300+0000","last_active":"2024-03-18T00:49:03.087310+0000","last_peered":"2024-03-18T00:49:03.087310+0000","last_clean":"2024-03-18T00:49:03.087310+0000","last_became_active":"2024-03-18T00:48:05.528935+0000","last_became_peered":"2024-03-18T00:48:05.528935+0000","last_unstale":"2024-03-18T00:49:03.087310+0000","last_undegraded":"2024-03-18T00:49:03.087310+0000","last_fullsized":"2024-03-18T00:49:03.087310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:29:45.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.498853+0000","last_change":"2024-03-18T00:48:06.536426+0000","last_active":"2024-03-18T00:49:31.498853+0000","last_peered":"2024-03-18T00:49:31.498853+0000","last_clean":"2024-03-18T00:49:31.498853+0000","last_became_active":"2024-03-18T00:48:06.536005+0000","last_became_peered":"2024-03-18T00:48:06.536005+0000","last_unstale":"2024-03-18T00:49:31.498853+0000","last_undegraded":"2024-03-18T00:49:31.498853+0000","last_fullsized":"2024-03-18T00:49:31.498853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:40:52.028794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087438+0000","last_change":"2024-03-18T00:48:05.536495+0000","last_active":"2024-03-18T00:49:03.087438+0000","last_peered":"2024-03-18T00:49:03.087438+0000","last_clean":"2024-03-18T00:49:03.087438+0000","last_became_active":"2024-03-18T00:48:05.536072+0000","last_became_peered":"2024-03-18T00:48:05.536072+0000","last_unstale":"2024-03-18T00:49:03.087438+0000","last_undegraded":"2024-03-18T00:49:03.087438+0000","last_fullsized":"2024-03-18T00:49:03.087438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:25.528261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.505766+0000","last_change":"2024-03-18T00:48:06.552900+0000","last_active":"2024-03-18T00:49:31.505766+0000","last_peered":"2024-03-18T00:49:31.505766+0000","last_clean":"2024-03-18T00:49:31.505766+0000","last_became_active":"2024-03-18T00:48:06.552568+0000","last_became_peered":"2024-03-18T00:48:06.552568+0000","last_unstale":"2024-03-18T00:49:31.505766+0000","last_undegraded":"2024-03-18T00:49:31.505766+0000","last_fullsized":"2024-03-18T00:49:31.505766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:53:25.254177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.155478+0000","last_change":"2024-03-18T00:48:05.537327+0000","last_active":"2024-03-18T00:50:05.155478+0000","last_peered":"2024-03-18T00:50:05.155478+0000","last_clean":"2024-03-18T00:50:05.155478+0000","last_became_active":"2024-03-18T00:48:05.536825+0000","last_became_peered":"2024-03-18T00:48:05.536825+0000","last_unstale":"2024-03-18T00:50:05.155478+0000","last_undegraded":"2024-03-18T00:50:05.155478+0000","last_fullsized":"2024-03-18T00:50:05.155478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:03.141142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.505260+0000","last_change":"2024-03-18T00:48:06.542593+0000","last_active":"2024-03-18T00:49:31.505260+0000","last_peered":"2024-03-18T00:49:31.505260+0000","last_clean":"2024-03-18T00:49:31.505260+0000","last_became_active":"2024-03-18T00:48:06.541624+0000","last_became_peered":"2024-03-18T00:48:06.541624+0000","last_unstale":"2024-03-18T00:49:31.505260+0000","last_undegraded":"2024-03-18T00:49:31.505260+0000","last_fullsized":"2024-03-18T00:49:31.505260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:10:56.905221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089025+0000","last_change":"2024-03-18T00:48:05.529140+0000","last_active":"2024-03-18T00:49:03.089025+0000","last_peered":"2024-03-18T00:49:03.089025+0000","last_clean":"2024-03-18T00:49:03.089025+0000","last_became_active":"2024-03-18T00:48:05.528682+0000","last_became_peered":"2024-03-18T00:48:05.528682+0000","last_unstale":"2024-03-18T00:49:03.089025+0000","last_undegraded":"2024-03-18T00:49:03.089025+0000","last_fullsized":"2024-03-18T00:49:03.089025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:57:52.663150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.508245+0000","last_change":"2024-03-18T00:48:06.546027+0000","last_active":"2024-03-18T00:49:31.508245+0000","last_peered":"2024-03-18T00:49:31.508245+0000","last_clean":"2024-03-18T00:49:31.508245+0000","last_became_active":"2024-03-18T00:48:06.545670+0000","last_became_peered":"2024-03-18T00:48:06.545670+0000","last_unstale":"2024-03-18T00:49:31.508245+0000","last_undegraded":"2024-03-18T00:49:31.508245+0000","last_fullsized":"2024-03-18T00:49:31.508245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:47:05.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087890+0000","last_change":"2024-03-18T00:48:05.526929+0000","last_active":"2024-03-18T00:49:03.087890+0000","last_peered":"2024-03-18T00:49:03.087890+0000","last_clean":"2024-03-18T00:49:03.087890+0000","last_became_active":"2024-03-18T00:48:05.526530+0000","last_became_peered":"2024-03-18T00:48:05.526530+0000","last_unstale":"2024-03-18T00:49:03.087890+0000","last_undegraded":"2024-03-18T00:49:03.087890+0000","last_fullsized":"2024-03-18T00:49:03.087890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:51:25.110895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504777+0000","last_change":"2024-03-18T00:48:06.541223+0000","last_active":"2024-03-18T00:49:31.504777+0000","last_peered":"2024-03-18T00:49:31.504777+0000","last_clean":"2024-03-18T00:49:31.504777+0000","last_became_active":"2024-03-18T00:48:06.540898+0000","last_became_peered":"2024-03-18T00:48:06.540898+0000","last_unstale":"2024-03-18T00:49:31.504777+0000","last_undegraded":"2024-03-18T00:49:31.504777+0000","last_fullsized":"2024-03-18T00:49:31.504777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:20:22.749609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506812+0000","last_change":"2024-03-18T00:48:06.532906+0000","last_active":"2024-03-18T00:49:31.506812+0000","last_peered":"2024-03-18T00:49:31.506812+0000","last_clean":"2024-03-18T00:49:31.506812+0000","last_became_active":"2024-03-18T00:48:06.532583+0000","last_became_peered":"2024-03-18T00:48:06.532583+0000","last_unstale":"2024-03-18T00:49:31.506812+0000","last_undegraded":"2024-03-18T00:49:31.506812+0000","last_fullsized":"2024-03-18T00:49:31.506812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:06.699615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506031+0000","last_change":"2024-03-18T00:48:06.530020+0000","last_active":"2024-03-18T00:49:31.506031+0000","last_peered":"2024-03-18T00:49:31.506031+0000","last_clean":"2024-03-18T00:49:31.506031+0000","last_became_active":"2024-03-18T00:48:06.529709+0000","last_became_peered":"2024-03-18T00:48:06.529709+0000","last_unstale":"2024-03-18T00:49:31.506031+0000","last_undegraded":"2024-03-18T00:49:31.506031+0000","last_fullsized":"2024-03-18T00:49:31.506031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:07.305537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500282+0000","last_change":"2024-03-18T00:48:06.537658+0000","last_active":"2024-03-18T00:49:31.500282+0000","last_peered":"2024-03-18T00:49:31.500282+0000","last_clean":"2024-03-18T00:49:31.500282+0000","last_became_active":"2024-03-18T00:48:06.537339+0000","last_became_peered":"2024-03-18T00:48:06.537339+0000","last_unstale":"2024-03-18T00:49:31.500282+0000","last_undegraded":"2024-03-18T00:49:31.500282+0000","last_fullsized":"2024-03-18T00:49:31.500282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:18:25.753168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497779+0000","last_change":"2024-03-18T00:48:06.544514+0000","last_active":"2024-03-18T00:49:31.497779+0000","last_peered":"2024-03-18T00:49:31.497779+0000","last_clean":"2024-03-18T00:49:31.497779+0000","last_became_active":"2024-03-18T00:48:06.544165+0000","last_became_peered":"2024-03-18T00:48:06.544165+0000","last_unstale":"2024-03-18T00:49:31.497779+0000","last_undegraded":"2024-03-18T00:49:31.497779+0000","last_fullsized":"2024-03-18T00:49:31.497779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:48:02.498940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501247+0000","last_change":"2024-03-18T00:49:03.090610+0000","last_active":"2024-03-18T00:49:31.501247+0000","last_peered":"2024-03-18T00:49:31.501247+0000","last_clean":"2024-03-18T00:49:31.501247+0000","last_became_active":"2024-03-18T00:49:03.090014+0000","last_became_peered":"2024-03-18T00:49:03.090014+0000","last_unstale":"2024-03-18T00:49:31.501247+0000","last_undegraded":"2024-03-18T00:49:31.501247+0000","last_fullsized":"2024-03-18T00:49:31.501247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:55:12.210223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.503021+0000","last_change":"2024-03-18T00:48:06.533837+0000","last_active":"2024-03-18T00:49:31.503021+0000","last_peered":"2024-03-18T00:49:31.503021+0000","last_clean":"2024-03-18T00:49:31.503021+0000","last_became_active":"2024-03-18T00:48:06.533418+0000","last_became_peered":"2024-03-18T00:48:06.533418+0000","last_unstale":"2024-03-18T00:49:31.503021+0000","last_undegraded":"2024-03-18T00:49:31.503021+0000","last_fullsized":"2024-03-18T00:49:31.503021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:10.873312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.499996+0000","last_change":"2024-03-18T00:48:06.527394+0000","last_active":"2024-03-18T00:49:31.499996+0000","last_peered":"2024-03-18T00:49:31.499996+0000","last_clean":"2024-03-18T00:49:31.499996+0000","last_became_active":"2024-03-18T00:48:06.526879+0000","last_became_peered":"2024-03-18T00:48:06.526879+0000","last_unstale":"2024-03-18T00:49:31.499996+0000","last_undegraded":"2024-03-18T00:49:31.499996+0000","last_fullsized":"2024-03-18T00:49:31.499996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:23:07.397259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500631+0000","last_change":"2024-03-18T00:48:06.542645+0000","last_active":"2024-03-18T00:49:31.500631+0000","last_peered":"2024-03-18T00:49:31.500631+0000","last_clean":"2024-03-18T00:49:31.500631+0000","last_became_active":"2024-03-18T00:48:06.542073+0000","last_became_peered":"2024-03-18T00:48:06.542073+0000","last_unstale":"2024-03-18T00:49:31.500631+0000","last_undegraded":"2024-03-18T00:49:31.500631+0000","last_fullsized":"2024-03-18T00:49:31.500631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:08:44.479318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507457+0000","last_change":"2024-03-18T00:48:06.541698+0000","last_active":"2024-03-18T00:49:31.507457+0000","last_peered":"2024-03-18T00:49:31.507457+0000","last_clean":"2024-03-18T00:49:31.507457+0000","last_became_active":"2024-03-18T00:48:06.541378+0000","last_became_peered":"2024-03-18T00:48:06.541378+0000","last_unstale":"2024-03-18T00:49:31.507457+0000","last_undegraded":"2024-03-18T00:49:31.507457+0000","last_fullsized":"2024-03-18T00:49:31.507457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:42:23.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502862+0000","last_change":"2024-03-18T00:48:06.533184+0000","last_active":"2024-03-18T00:49:31.502862+0000","last_peered":"2024-03-18T00:49:31.502862+0000","last_clean":"2024-03-18T00:49:31.502862+0000","last_became_active":"2024-03-18T00:48:06.532034+0000","last_became_peered":"2024-03-18T00:48:06.532034+0000","last_unstale":"2024-03-18T00:49:31.502862+0000","last_undegraded":"2024-03-18T00:49:31.502862+0000","last_fullsized":"2024-03-18T00:49:31.502862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:45.265799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.515864+0000","last_change":"2024-03-18T00:48:06.545941+0000","last_active":"2024-03-18T00:49:31.515864+0000","last_peered":"2024-03-18T00:49:31.515864+0000","last_clean":"2024-03-18T00:49:31.515864+0000","last_became_active":"2024-03-18T00:48:06.545402+0000","last_became_peered":"2024-03-18T00:48:06.545402+0000","last_unstale":"2024-03-18T00:49:31.515864+0000","last_undegraded":"2024-03-18T00:49:31.515864+0000","last_fullsized":"2024-03-18T00:49:31.515864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:11:45.512849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.505008+0000","last_change":"2024-03-18T00:48:06.541513+0000","last_active":"2024-03-18T00:49:31.505008+0000","last_peered":"2024-03-18T00:49:31.505008+0000","last_clean":"2024-03-18T00:49:31.505008+0000","last_became_active":"2024-03-18T00:48:06.541128+0000","last_became_peered":"2024-03-18T00:48:06.541128+0000","last_unstale":"2024-03-18T00:49:31.505008+0000","last_undegraded":"2024-03-18T00:49:31.505008+0000","last_fullsized":"2024-03-18T00:49:31.505008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:42:10.635836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501522+0000","last_change":"2024-03-18T00:48:06.542316+0000","last_active":"2024-03-18T00:49:31.501522+0000","last_peered":"2024-03-18T00:49:31.501522+0000","last_clean":"2024-03-18T00:49:31.501522+0000","last_became_active":"2024-03-18T00:48:06.541914+0000","last_became_peered":"2024-03-18T00:48:06.541914+0000","last_unstale":"2024-03-18T00:49:31.501522+0000","last_undegraded":"2024-03-18T00:49:31.501522+0000","last_fullsized":"2024-03-18T00:49:31.501522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:46.293240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249372,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73179,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1128,"num_read_kb":670,"num_write":252,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1237,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":214607,"data_compressed":5589,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":461614,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612651,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":1116986,"data_compressed":43768,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":459970,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":460236,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":1053007,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462976,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459687,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":521895,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":461026,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":527465,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":459512,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888137,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":459660,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724612,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1051531,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63047,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68617,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":67653,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T00:50:07.622 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T00:50:07.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:07 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3060950114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:50:07.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:07 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1131757428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:50:07.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:07 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1348078428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:50:07.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:07 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3060950114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:50:07.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:07 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1131757428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:50:07.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:07 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1348078428' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 -- 172.21.15.49:0/3679061800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa644104c70 msgr2=0x7fa644107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 --2- 172.21.15.49:0/3679061800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa644104c70 0x7fa644107060 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa640009860 tx=0x7fa64002bb70 comp rx=0 tx=0).stop 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 -- 172.21.15.49:0/3679061800 shutdown_connections 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 --2- 172.21.15.49:0/3679061800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa644104c70 0x7fa644107060 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 --2- 172.21.15.49:0/3679061800 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa6441075a0 0x7fa644109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 --2- 172.21.15.49:0/3679061800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644102340 0x7fa644104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 -- 172.21.15.49:0/3679061800 >> 172.21.15.49:0/3679061800 conn(0x7fa6440fbc70 msgr2=0x7fa6440fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 -- 172.21.15.49:0/3679061800 shutdown_connections 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 -- 172.21.15.49:0/3679061800 wait complete. 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 Processor -- start 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.768+0000 7fa64c57e640 1 -- start start 2024-03-18T00:50:07.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64c57e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa644102340 0x7fa644197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64c57e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644104c70 0x7fa644198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64c57e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6441075a0 0x7fa64419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64c57e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa64410d480 con 0x7fa644102340 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64c57e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa64410d300 con 0x7fa6441075a0 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64c57e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa64410d600 con 0x7fa644104c70 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64a31a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa644102340 0x7fa644197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64a31a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa644102340 0x7fa644197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58188/0 (socket says 172.21.15.49:58188) 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64a31a640 1 -- 172.21.15.49:0/2691287186 learned_addr learned my addr 172.21.15.49:0/2691287186 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa649b19640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644104c70 0x7fa644198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64ab1b640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6441075a0 0x7fa64419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64a31a640 1 -- 172.21.15.49:0/2691287186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644104c70 msgr2=0x7fa644198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64a31a640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644104c70 0x7fa644198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64a31a640 1 -- 172.21.15.49:0/2691287186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6441075a0 msgr2=0x7fa64419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64a31a640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6441075a0 0x7fa64419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64a31a640 1 -- 172.21.15.49:0/2691287186 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64419f770 con 0x7fa644102340 2024-03-18T00:50:07.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa649b19640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644104c70 0x7fa644198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:07.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64a31a640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa644102340 0x7fa644197b50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa63800e9f0 tx=0x7fa63800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:07.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa6377fe640 1 -- 172.21.15.49:0/2691287186 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63800cd80 con 0x7fa644102340 2024-03-18T00:50:07.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa6377fe640 1 -- 172.21.15.49:0/2691287186 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa638005500 con 0x7fa644102340 2024-03-18T00:50:07.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64419fa60 con 0x7fa644102340 2024-03-18T00:50:07.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa6377fe640 1 -- 172.21.15.49:0/2691287186 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638010690 con 0x7fa644102340 2024-03-18T00:50:07.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.769+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64419ff30 con 0x7fa644102340 2024-03-18T00:50:07.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.770+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6440fd760 con 0x7fa644102340 2024-03-18T00:50:07.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.772+0000 7fa6377fe640 1 -- 172.21.15.49:0/2691287186 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa6380046c0 con 0x7fa644102340 2024-03-18T00:50:07.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.772+0000 7fa6377fe640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa6300a2800 0x7fa6300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:07.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.773+0000 7fa649b19640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa6300a2800 0x7fa6300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:07.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.773+0000 7fa6377fe640 1 -- 172.21.15.49:0/2691287186 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa6380cee70 con 0x7fa644102340 2024-03-18T00:50:07.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.773+0000 7fa649b19640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa6300a2800 0x7fa6300a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa640009830 tx=0x7fa64002e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:07.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.777+0000 7fa6377fe640 1 -- 172.21.15.49:0/2691287186 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa63809bef0 con 0x7fa644102340 2024-03-18T00:50:07.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.958+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa644061c20 con 0x7fa6300a2800 2024-03-18T00:50:07.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.965+0000 7fa6377fe640 1 -- 172.21.15.49:0/2691287186 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259166 (secure 0 0 0) 0x7fa644061c20 con 0x7fa6300a2800 2024-03-18T00:50:07.967 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:07.969 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-03-18T00:50:07.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa6300a2800 msgr2=0x7fa6300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa6300a2800 0x7fa6300a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa640009830 tx=0x7fa64002e040 comp rx=0 tx=0).stop 2024-03-18T00:50:07.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa644102340 msgr2=0x7fa644197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:07.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa644102340 0x7fa644197b50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa63800e9f0 tx=0x7fa63800edf0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 shutdown_connections 2024-03-18T00:50:07.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa6300a2800 0x7fa6300a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa6441075a0 0x7fa64419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa644104c70 0x7fa644198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 --2- 172.21.15.49:0/2691287186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa644102340 0x7fa644197b50 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:07.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 >> 172.21.15.49:0/2691287186 conn(0x7fa6440fbc70 msgr2=0x7fa644102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:07.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 shutdown_connections 2024-03-18T00:50:07.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:07.969+0000 7fa64c57e640 1 -- 172.21.15.49:0/2691287186 wait complete. 2024-03-18T00:50:07.980 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-03-18T00:50:07.490063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663781,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1254,"num_read_kb":779,"num_write":435,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181973,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":857,"ondisk_log_size":857,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356796,"kb_used_data":11804,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585028,"statfs":{"total":1151940427776,"available":1151575068672,"internally_reserved":0,"allocated":12087296,"data_stored":7492589,"data_compressed":183621,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4930,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":123,"num_read_kb":62,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002823"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504686+0000","last_change":"2024-03-18T00:48:06.536034+0000","last_active":"2024-03-18T00:49:31.504686+0000","last_peered":"2024-03-18T00:49:31.504686+0000","last_clean":"2024-03-18T00:49:31.504686+0000","last_became_active":"2024-03-18T00:48:06.535694+0000","last_became_peered":"2024-03-18T00:48:06.535694+0000","last_unstale":"2024-03-18T00:49:31.504686+0000","last_undegraded":"2024-03-18T00:49:31.504686+0000","last_fullsized":"2024-03-18T00:49:31.504686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:24:33.785669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502451+0000","last_change":"2024-03-18T00:48:06.536741+0000","last_active":"2024-03-18T00:49:31.502451+0000","last_peered":"2024-03-18T00:49:31.502451+0000","last_clean":"2024-03-18T00:49:31.502451+0000","last_became_active":"2024-03-18T00:48:06.535865+0000","last_became_peered":"2024-03-18T00:48:06.535865+0000","last_unstale":"2024-03-18T00:49:31.502451+0000","last_undegraded":"2024-03-18T00:49:31.502451+0000","last_fullsized":"2024-03-18T00:49:31.502451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:58.104983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501939+0000","last_change":"2024-03-18T00:48:06.537162+0000","last_active":"2024-03-18T00:49:31.501939+0000","last_peered":"2024-03-18T00:49:31.501939+0000","last_clean":"2024-03-18T00:49:31.501939+0000","last_became_active":"2024-03-18T00:48:06.536835+0000","last_became_peered":"2024-03-18T00:48:06.536835+0000","last_unstale":"2024-03-18T00:49:31.501939+0000","last_undegraded":"2024-03-18T00:49:31.501939+0000","last_fullsized":"2024-03-18T00:49:31.501939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:15:08.519180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.486309+0000","last_change":"2024-03-18T00:48:06.540084+0000","last_active":"2024-03-18T00:49:31.486309+0000","last_peered":"2024-03-18T00:49:31.486309+0000","last_clean":"2024-03-18T00:49:31.486309+0000","last_became_active":"2024-03-18T00:48:06.539736+0000","last_became_peered":"2024-03-18T00:48:06.539736+0000","last_unstale":"2024-03-18T00:49:31.486309+0000","last_undegraded":"2024-03-18T00:49:31.486309+0000","last_fullsized":"2024-03-18T00:49:31.486309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:37:51.748603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497212+0000","last_change":"2024-03-18T00:48:06.540022+0000","last_active":"2024-03-18T00:49:31.497212+0000","last_peered":"2024-03-18T00:49:31.497212+0000","last_clean":"2024-03-18T00:49:31.497212+0000","last_became_active":"2024-03-18T00:48:06.539665+0000","last_became_peered":"2024-03-18T00:48:06.539665+0000","last_unstale":"2024-03-18T00:49:31.497212+0000","last_undegraded":"2024-03-18T00:49:31.497212+0000","last_fullsized":"2024-03-18T00:49:31.497212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:55:52.448279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506091+0000","last_change":"2024-03-18T00:48:06.538672+0000","last_active":"2024-03-18T00:49:31.506091+0000","last_peered":"2024-03-18T00:49:31.506091+0000","last_clean":"2024-03-18T00:49:31.506091+0000","last_became_active":"2024-03-18T00:48:06.538189+0000","last_became_peered":"2024-03-18T00:48:06.538189+0000","last_unstale":"2024-03-18T00:49:31.506091+0000","last_undegraded":"2024-03-18T00:49:31.506091+0000","last_fullsized":"2024-03-18T00:49:31.506091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:09:04.672255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506665+0000","last_change":"2024-03-18T00:48:06.546417+0000","last_active":"2024-03-18T00:49:31.506665+0000","last_peered":"2024-03-18T00:49:31.506665+0000","last_clean":"2024-03-18T00:49:31.506665+0000","last_became_active":"2024-03-18T00:48:06.545759+0000","last_became_peered":"2024-03-18T00:48:06.545759+0000","last_unstale":"2024-03-18T00:49:31.506665+0000","last_undegraded":"2024-03-18T00:49:31.506665+0000","last_fullsized":"2024-03-18T00:49:31.506665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:56.759028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.508198+0000","last_change":"2024-03-18T00:48:06.545875+0000","last_active":"2024-03-18T00:49:31.508198+0000","last_peered":"2024-03-18T00:49:31.508198+0000","last_clean":"2024-03-18T00:49:31.508198+0000","last_became_active":"2024-03-18T00:48:06.545316+0000","last_became_peered":"2024-03-18T00:48:06.545316+0000","last_unstale":"2024-03-18T00:49:31.508198+0000","last_undegraded":"2024-03-18T00:49:31.508198+0000","last_fullsized":"2024-03-18T00:49:31.508198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:22:40.998671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.512026+0000","last_change":"2024-03-18T00:48:06.533947+0000","last_active":"2024-03-18T00:49:31.512026+0000","last_peered":"2024-03-18T00:49:31.512026+0000","last_clean":"2024-03-18T00:49:31.512026+0000","last_became_active":"2024-03-18T00:48:06.533615+0000","last_became_peered":"2024-03-18T00:48:06.533615+0000","last_unstale":"2024-03-18T00:49:31.512026+0000","last_undegraded":"2024-03-18T00:49:31.512026+0000","last_fullsized":"2024-03-18T00:49:31.512026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:32:16.750269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507310+0000","last_change":"2024-03-18T00:48:06.537614+0000","last_active":"2024-03-18T00:49:31.507310+0000","last_peered":"2024-03-18T00:49:31.507310+0000","last_clean":"2024-03-18T00:49:31.507310+0000","last_became_active":"2024-03-18T00:48:06.536617+0000","last_became_peered":"2024-03-18T00:48:06.536617+0000","last_unstale":"2024-03-18T00:49:31.507310+0000","last_undegraded":"2024-03-18T00:49:31.507310+0000","last_fullsized":"2024-03-18T00:49:31.507310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:52.222603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502441+0000","last_change":"2024-03-18T00:48:06.540363+0000","last_active":"2024-03-18T00:49:31.502441+0000","last_peered":"2024-03-18T00:49:31.502441+0000","last_clean":"2024-03-18T00:49:31.502441+0000","last_became_active":"2024-03-18T00:48:06.540011+0000","last_became_peered":"2024-03-18T00:48:06.540011+0000","last_unstale":"2024-03-18T00:49:31.502441+0000","last_undegraded":"2024-03-18T00:49:31.502441+0000","last_fullsized":"2024-03-18T00:49:31.502441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:23:10.034444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504057+0000","last_change":"2024-03-18T00:48:06.532611+0000","last_active":"2024-03-18T00:49:31.504057+0000","last_peered":"2024-03-18T00:49:31.504057+0000","last_clean":"2024-03-18T00:49:31.504057+0000","last_became_active":"2024-03-18T00:48:06.532301+0000","last_became_peered":"2024-03-18T00:48:06.532301+0000","last_unstale":"2024-03-18T00:49:31.504057+0000","last_undegraded":"2024-03-18T00:49:31.504057+0000","last_fullsized":"2024-03-18T00:49:31.504057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:40.795105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.508264+0000","last_change":"2024-03-18T00:48:06.531993+0000","last_active":"2024-03-18T00:49:31.508264+0000","last_peered":"2024-03-18T00:49:31.508264+0000","last_clean":"2024-03-18T00:49:31.508264+0000","last_became_active":"2024-03-18T00:48:06.531677+0000","last_became_peered":"2024-03-18T00:48:06.531677+0000","last_unstale":"2024-03-18T00:49:31.508264+0000","last_undegraded":"2024-03-18T00:49:31.508264+0000","last_fullsized":"2024-03-18T00:49:31.508264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:48:24.865299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501483+0000","last_change":"2024-03-18T00:48:06.536830+0000","last_active":"2024-03-18T00:49:31.501483+0000","last_peered":"2024-03-18T00:49:31.501483+0000","last_clean":"2024-03-18T00:49:31.501483+0000","last_became_active":"2024-03-18T00:48:06.535908+0000","last_became_peered":"2024-03-18T00:48:06.535908+0000","last_unstale":"2024-03-18T00:49:31.501483+0000","last_undegraded":"2024-03-18T00:49:31.501483+0000","last_fullsized":"2024-03-18T00:49:31.501483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:23:26.155130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.493074+0000","last_change":"2024-03-18T00:48:06.543641+0000","last_active":"2024-03-18T00:49:31.493074+0000","last_peered":"2024-03-18T00:49:31.493074+0000","last_clean":"2024-03-18T00:49:31.493074+0000","last_became_active":"2024-03-18T00:48:06.543293+0000","last_became_peered":"2024-03-18T00:48:06.543293+0000","last_unstale":"2024-03-18T00:49:31.493074+0000","last_undegraded":"2024-03-18T00:49:31.493074+0000","last_fullsized":"2024-03-18T00:49:31.493074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:52.406255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.510399+0000","last_change":"2024-03-18T00:48:06.544732+0000","last_active":"2024-03-18T00:49:31.510399+0000","last_peered":"2024-03-18T00:49:31.510399+0000","last_clean":"2024-03-18T00:49:31.510399+0000","last_became_active":"2024-03-18T00:48:06.544400+0000","last_became_peered":"2024-03-18T00:48:06.544400+0000","last_unstale":"2024-03-18T00:49:31.510399+0000","last_undegraded":"2024-03-18T00:49:31.510399+0000","last_fullsized":"2024-03-18T00:49:31.510399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:23:39.655434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504899+0000","last_change":"2024-03-18T00:48:06.536595+0000","last_active":"2024-03-18T00:49:31.504899+0000","last_peered":"2024-03-18T00:49:31.504899+0000","last_clean":"2024-03-18T00:49:31.504899+0000","last_became_active":"2024-03-18T00:48:06.535874+0000","last_became_peered":"2024-03-18T00:48:06.535874+0000","last_unstale":"2024-03-18T00:49:31.504899+0000","last_undegraded":"2024-03-18T00:49:31.504899+0000","last_fullsized":"2024-03-18T00:49:31.504899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:55.603614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.515187+0000","last_change":"2024-03-18T00:48:06.546676+0000","last_active":"2024-03-18T00:49:31.515187+0000","last_peered":"2024-03-18T00:49:31.515187+0000","last_clean":"2024-03-18T00:49:31.515187+0000","last_became_active":"2024-03-18T00:48:06.546356+0000","last_became_peered":"2024-03-18T00:48:06.546356+0000","last_unstale":"2024-03-18T00:49:31.515187+0000","last_undegraded":"2024-03-18T00:49:31.515187+0000","last_fullsized":"2024-03-18T00:49:31.515187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:04:35.467415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500375+0000","last_change":"2024-03-18T00:48:06.532184+0000","last_active":"2024-03-18T00:49:31.500375+0000","last_peered":"2024-03-18T00:49:31.500375+0000","last_clean":"2024-03-18T00:49:31.500375+0000","last_became_active":"2024-03-18T00:48:06.531816+0000","last_became_peered":"2024-03-18T00:48:06.531816+0000","last_unstale":"2024-03-18T00:49:31.500375+0000","last_undegraded":"2024-03-18T00:49:31.500375+0000","last_fullsized":"2024-03-18T00:49:31.500375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:09:03.113024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":146,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:03.159413+0000","last_change":"2024-03-18T00:48:05.530213+0000","last_active":"2024-03-18T00:50:03.159413+0000","last_peered":"2024-03-18T00:50:03.159413+0000","last_clean":"2024-03-18T00:50:03.159413+0000","last_became_active":"2024-03-18T00:48:05.529897+0000","last_became_peered":"2024-03-18T00:48:05.529897+0000","last_unstale":"2024-03-18T00:50:03.159413+0000","last_undegraded":"2024-03-18T00:50:03.159413+0000","last_fullsized":"2024-03-18T00:50:03.159413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:49.003362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506113+0000","last_change":"2024-03-18T00:48:06.544437+0000","last_active":"2024-03-18T00:49:31.506113+0000","last_peered":"2024-03-18T00:49:31.506113+0000","last_clean":"2024-03-18T00:49:31.506113+0000","last_became_active":"2024-03-18T00:48:06.544125+0000","last_became_peered":"2024-03-18T00:48:06.544125+0000","last_unstale":"2024-03-18T00:49:31.506113+0000","last_undegraded":"2024-03-18T00:49:31.506113+0000","last_fullsized":"2024-03-18T00:49:31.506113+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:56:42.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325994+0000","last_change":"2024-03-18T00:48:05.536249+0000","last_active":"2024-03-18T00:49:03.325994+0000","last_peered":"2024-03-18T00:49:03.325994+0000","last_clean":"2024-03-18T00:49:03.325994+0000","last_became_active":"2024-03-18T00:48:05.535954+0000","last_became_peered":"2024-03-18T00:48:05.535954+0000","last_unstale":"2024-03-18T00:49:03.325994+0000","last_undegraded":"2024-03-18T00:49:03.325994+0000","last_fullsized":"2024-03-18T00:49:03.325994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:18:52.341453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497426+0000","last_change":"2024-03-18T00:48:06.541377+0000","last_active":"2024-03-18T00:49:31.497426+0000","last_peered":"2024-03-18T00:49:31.497426+0000","last_clean":"2024-03-18T00:49:31.497426+0000","last_became_active":"2024-03-18T00:48:06.540937+0000","last_became_peered":"2024-03-18T00:48:06.540937+0000","last_unstale":"2024-03-18T00:49:31.497426+0000","last_undegraded":"2024-03-18T00:49:31.497426+0000","last_fullsized":"2024-03-18T00:49:31.497426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:51:55.268517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088568+0000","last_change":"2024-03-18T00:48:05.534372+0000","last_active":"2024-03-18T00:49:03.088568+0000","last_peered":"2024-03-18T00:49:03.088568+0000","last_clean":"2024-03-18T00:49:03.088568+0000","last_became_active":"2024-03-18T00:48:05.534034+0000","last_became_peered":"2024-03-18T00:48:05.534034+0000","last_unstale":"2024-03-18T00:49:03.088568+0000","last_undegraded":"2024-03-18T00:49:03.088568+0000","last_fullsized":"2024-03-18T00:49:03.088568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:58:55.013167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.513707+0000","last_change":"2024-03-18T00:48:06.527276+0000","last_active":"2024-03-18T00:49:31.513707+0000","last_peered":"2024-03-18T00:49:31.513707+0000","last_clean":"2024-03-18T00:49:31.513707+0000","last_became_active":"2024-03-18T00:48:06.526904+0000","last_became_peered":"2024-03-18T00:48:06.526904+0000","last_unstale":"2024-03-18T00:49:31.513707+0000","last_undegraded":"2024-03-18T00:49:31.513707+0000","last_fullsized":"2024-03-18T00:49:31.513707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:16:10.731256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087607+0000","last_change":"2024-03-18T00:48:05.528188+0000","last_active":"2024-03-18T00:49:03.087607+0000","last_peered":"2024-03-18T00:49:03.087607+0000","last_clean":"2024-03-18T00:49:03.087607+0000","last_became_active":"2024-03-18T00:48:05.527788+0000","last_became_peered":"2024-03-18T00:48:05.527788+0000","last_unstale":"2024-03-18T00:49:03.087607+0000","last_undegraded":"2024-03-18T00:49:03.087607+0000","last_fullsized":"2024-03-18T00:49:03.087607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:47.939771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507894+0000","last_change":"2024-03-18T00:49:03.092920+0000","last_active":"2024-03-18T00:49:31.507894+0000","last_peered":"2024-03-18T00:49:31.507894+0000","last_clean":"2024-03-18T00:49:31.507894+0000","last_became_active":"2024-03-18T00:49:03.092569+0000","last_became_peered":"2024-03-18T00:49:03.092569+0000","last_unstale":"2024-03-18T00:49:31.507894+0000","last_undegraded":"2024-03-18T00:49:31.507894+0000","last_fullsized":"2024-03-18T00:49:31.507894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:15:02.246688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'8","reported_seq":85,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.155786+0000","last_change":"2024-03-18T00:49:03.100514+0000","last_active":"2024-03-18T00:50:05.155786+0000","last_peered":"2024-03-18T00:50:05.155786+0000","last_clean":"2024-03-18T00:50:05.155786+0000","last_became_active":"2024-03-18T00:49:03.090898+0000","last_became_peered":"2024-03-18T00:49:03.090898+0000","last_unstale":"2024-03-18T00:50:05.155786+0000","last_undegraded":"2024-03-18T00:50:05.155786+0000","last_fullsized":"2024-03-18T00:50:05.155786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:19:39.943819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.494376+0000","last_change":"2024-03-18T00:48:06.537405+0000","last_active":"2024-03-18T00:49:31.494376+0000","last_peered":"2024-03-18T00:49:31.494376+0000","last_clean":"2024-03-18T00:49:31.494376+0000","last_became_active":"2024-03-18T00:48:06.536919+0000","last_became_peered":"2024-03-18T00:48:06.536919+0000","last_unstale":"2024-03-18T00:49:31.494376+0000","last_undegraded":"2024-03-18T00:49:31.494376+0000","last_fullsized":"2024-03-18T00:49:31.494376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:31:59.863571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088122+0000","last_change":"2024-03-18T00:48:05.528588+0000","last_active":"2024-03-18T00:49:03.088122+0000","last_peered":"2024-03-18T00:49:03.088122+0000","last_clean":"2024-03-18T00:49:03.088122+0000","last_became_active":"2024-03-18T00:48:05.528263+0000","last_became_peered":"2024-03-18T00:48:05.528263+0000","last_unstale":"2024-03-18T00:49:03.088122+0000","last_undegraded":"2024-03-18T00:49:03.088122+0000","last_fullsized":"2024-03-18T00:49:03.088122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:18:10.283278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506438+0000","last_change":"2024-03-18T00:48:06.546323+0000","last_active":"2024-03-18T00:49:31.506438+0000","last_peered":"2024-03-18T00:49:31.506438+0000","last_clean":"2024-03-18T00:49:31.506438+0000","last_became_active":"2024-03-18T00:48:06.545956+0000","last_became_peered":"2024-03-18T00:48:06.545956+0000","last_unstale":"2024-03-18T00:49:31.506438+0000","last_undegraded":"2024-03-18T00:49:31.506438+0000","last_fullsized":"2024-03-18T00:49:31.506438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:32.618724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087983+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:49:03.087983+0000","last_peered":"2024-03-18T00:49:03.087983+0000","last_clean":"2024-03-18T00:49:03.087983+0000","last_became_active":"2024-03-18T00:48:05.525590+0000","last_became_peered":"2024-03-18T00:48:05.525590+0000","last_unstale":"2024-03-18T00:49:03.087983+0000","last_undegraded":"2024-03-18T00:49:03.087983+0000","last_fullsized":"2024-03-18T00:49:03.087983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:19.357331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502517+0000","last_change":"2024-03-18T00:48:06.538798+0000","last_active":"2024-03-18T00:49:31.502517+0000","last_peered":"2024-03-18T00:49:31.502517+0000","last_clean":"2024-03-18T00:49:31.502517+0000","last_became_active":"2024-03-18T00:48:06.538457+0000","last_became_peered":"2024-03-18T00:48:06.538457+0000","last_unstale":"2024-03-18T00:49:31.502517+0000","last_undegraded":"2024-03-18T00:49:31.502517+0000","last_fullsized":"2024-03-18T00:49:31.502517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:10.738683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088047+0000","last_change":"2024-03-18T00:48:05.528094+0000","last_active":"2024-03-18T00:49:03.088047+0000","last_peered":"2024-03-18T00:49:03.088047+0000","last_clean":"2024-03-18T00:49:03.088047+0000","last_became_active":"2024-03-18T00:48:05.527730+0000","last_became_peered":"2024-03-18T00:48:05.527730+0000","last_unstale":"2024-03-18T00:49:03.088047+0000","last_undegraded":"2024-03-18T00:49:03.088047+0000","last_fullsized":"2024-03-18T00:49:03.088047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:48:27.707505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.498967+0000","last_change":"2024-03-18T00:48:06.548676+0000","last_active":"2024-03-18T00:49:31.498967+0000","last_peered":"2024-03-18T00:49:31.498967+0000","last_clean":"2024-03-18T00:49:31.498967+0000","last_became_active":"2024-03-18T00:48:06.547693+0000","last_became_peered":"2024-03-18T00:48:06.547693+0000","last_unstale":"2024-03-18T00:49:31.498967+0000","last_undegraded":"2024-03-18T00:49:31.498967+0000","last_fullsized":"2024-03-18T00:49:31.498967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:13.849175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'6","reported_seq":253,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.154991+0000","last_change":"2024-03-18T00:48:05.523639+0000","last_active":"2024-03-18T00:50:05.154991+0000","last_peered":"2024-03-18T00:50:05.154991+0000","last_clean":"2024-03-18T00:50:05.154991+0000","last_became_active":"2024-03-18T00:48:05.523269+0000","last_became_peered":"2024-03-18T00:48:05.523269+0000","last_unstale":"2024-03-18T00:50:05.154991+0000","last_undegraded":"2024-03-18T00:50:05.154991+0000","last_fullsized":"2024-03-18T00:50:05.154991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:39.961247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":228,"num_read_kb":228,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.514641+0000","last_change":"2024-03-18T00:48:06.537786+0000","last_active":"2024-03-18T00:49:31.514641+0000","last_peered":"2024-03-18T00:49:31.514641+0000","last_clean":"2024-03-18T00:49:31.514641+0000","last_became_active":"2024-03-18T00:48:06.537488+0000","last_became_peered":"2024-03-18T00:48:06.537488+0000","last_unstale":"2024-03-18T00:49:31.514641+0000","last_undegraded":"2024-03-18T00:49:31.514641+0000","last_fullsized":"2024-03-18T00:49:31.514641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:45:02.095610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.096001+0000","last_change":"2024-03-18T00:49:03.096001+0000","last_active":"2024-03-18T00:49:03.096001+0000","last_peered":"2024-03-18T00:49:03.096001+0000","last_clean":"2024-03-18T00:49:03.096001+0000","last_became_active":"2024-03-18T00:49:03.090868+0000","last_became_peered":"2024-03-18T00:49:03.090868+0000","last_unstale":"2024-03-18T00:49:03.096001+0000","last_undegraded":"2024-03-18T00:49:03.096001+0000","last_fullsized":"2024-03-18T00:49:03.096001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:03.930010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.494451+0000","last_change":"2024-03-18T00:48:06.545416+0000","last_active":"2024-03-18T00:49:31.494451+0000","last_peered":"2024-03-18T00:49:31.494451+0000","last_clean":"2024-03-18T00:49:31.494451+0000","last_became_active":"2024-03-18T00:48:06.545087+0000","last_became_peered":"2024-03-18T00:48:06.545087+0000","last_unstale":"2024-03-18T00:49:31.494451+0000","last_undegraded":"2024-03-18T00:49:31.494451+0000","last_fullsized":"2024-03-18T00:49:31.494451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:02:25.309990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089159+0000","last_change":"2024-03-18T00:48:05.533088+0000","last_active":"2024-03-18T00:49:03.089159+0000","last_peered":"2024-03-18T00:49:03.089159+0000","last_clean":"2024-03-18T00:49:03.089159+0000","last_became_active":"2024-03-18T00:48:05.532308+0000","last_became_peered":"2024-03-18T00:48:05.532308+0000","last_unstale":"2024-03-18T00:49:03.089159+0000","last_undegraded":"2024-03-18T00:49:03.089159+0000","last_fullsized":"2024-03-18T00:49:03.089159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:35.071283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.513851+0000","last_change":"2024-03-18T00:49:03.092567+0000","last_active":"2024-03-18T00:49:31.513851+0000","last_peered":"2024-03-18T00:49:31.513851+0000","last_clean":"2024-03-18T00:49:31.513851+0000","last_became_active":"2024-03-18T00:49:03.092297+0000","last_became_peered":"2024-03-18T00:49:03.092297+0000","last_unstale":"2024-03-18T00:49:31.513851+0000","last_undegraded":"2024-03-18T00:49:31.513851+0000","last_fullsized":"2024-03-18T00:49:31.513851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:44.567584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'36","reported_seq":292,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.147485+0000","last_change":"2024-03-18T00:48:05.527557+0000","last_active":"2024-03-18T00:50:05.147485+0000","last_peered":"2024-03-18T00:50:05.147485+0000","last_clean":"2024-03-18T00:50:05.147485+0000","last_became_active":"2024-03-18T00:48:05.527167+0000","last_became_peered":"2024-03-18T00:48:05.527167+0000","last_unstale":"2024-03-18T00:50:05.147485+0000","last_undegraded":"2024-03-18T00:50:05.147485+0000","last_fullsized":"2024-03-18T00:50:05.147485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:02:04.105091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65917,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":184,"num_write":38,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.499066+0000","last_change":"2024-03-18T00:48:06.543153+0000","last_active":"2024-03-18T00:49:31.499066+0000","last_peered":"2024-03-18T00:49:31.499066+0000","last_clean":"2024-03-18T00:49:31.499066+0000","last_became_active":"2024-03-18T00:48:06.541353+0000","last_became_peered":"2024-03-18T00:48:06.541353+0000","last_unstale":"2024-03-18T00:49:31.499066+0000","last_undegraded":"2024-03-18T00:49:31.499066+0000","last_fullsized":"2024-03-18T00:49:31.499066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:35.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086981+0000","last_change":"2024-03-18T00:48:05.530560+0000","last_active":"2024-03-18T00:49:03.086981+0000","last_peered":"2024-03-18T00:49:03.086981+0000","last_clean":"2024-03-18T00:49:03.086981+0000","last_became_active":"2024-03-18T00:48:05.528858+0000","last_became_peered":"2024-03-18T00:48:05.528858+0000","last_unstale":"2024-03-18T00:49:03.086981+0000","last_undegraded":"2024-03-18T00:49:03.086981+0000","last_fullsized":"2024-03-18T00:49:03.086981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:03:01.496919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497847+0000","last_change":"2024-03-18T00:49:03.090951+0000","last_active":"2024-03-18T00:49:31.497847+0000","last_peered":"2024-03-18T00:49:31.497847+0000","last_clean":"2024-03-18T00:49:31.497847+0000","last_became_active":"2024-03-18T00:49:03.090578+0000","last_became_peered":"2024-03-18T00:49:03.090578+0000","last_unstale":"2024-03-18T00:49:31.497847+0000","last_undegraded":"2024-03-18T00:49:31.497847+0000","last_fullsized":"2024-03-18T00:49:31.497847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:04:12.703545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088756+0000","last_change":"2024-03-18T00:48:05.527902+0000","last_active":"2024-03-18T00:49:03.088756+0000","last_peered":"2024-03-18T00:49:03.088756+0000","last_clean":"2024-03-18T00:49:03.088756+0000","last_became_active":"2024-03-18T00:48:05.527495+0000","last_became_peered":"2024-03-18T00:48:05.527495+0000","last_unstale":"2024-03-18T00:49:03.088756+0000","last_undegraded":"2024-03-18T00:49:03.088756+0000","last_fullsized":"2024-03-18T00:49:03.088756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:25.504652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504404+0000","last_change":"2024-03-18T00:48:06.550413+0000","last_active":"2024-03-18T00:49:31.504404+0000","last_peered":"2024-03-18T00:49:31.504404+0000","last_clean":"2024-03-18T00:49:31.504404+0000","last_became_active":"2024-03-18T00:48:06.547727+0000","last_became_peered":"2024-03-18T00:48:06.547727+0000","last_unstale":"2024-03-18T00:49:31.504404+0000","last_undegraded":"2024-03-18T00:49:31.504404+0000","last_fullsized":"2024-03-18T00:49:31.504404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:36:48.878417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.146430+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:50:05.146430+0000","last_peered":"2024-03-18T00:50:05.146430+0000","last_clean":"2024-03-18T00:50:05.146430+0000","last_became_active":"2024-03-18T00:48:05.525607+0000","last_became_peered":"2024-03-18T00:48:05.525607+0000","last_unstale":"2024-03-18T00:50:05.146430+0000","last_undegraded":"2024-03-18T00:50:05.146430+0000","last_fullsized":"2024-03-18T00:50:05.146430+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:32.457447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.498511+0000","last_change":"2024-03-18T00:49:03.091221+0000","last_active":"2024-03-18T00:49:31.498511+0000","last_peered":"2024-03-18T00:49:31.498511+0000","last_clean":"2024-03-18T00:49:31.498511+0000","last_became_active":"2024-03-18T00:49:03.090559+0000","last_became_peered":"2024-03-18T00:49:03.090559+0000","last_unstale":"2024-03-18T00:49:31.498511+0000","last_undegraded":"2024-03-18T00:49:31.498511+0000","last_fullsized":"2024-03-18T00:49:31.498511+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:02:54.890187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.417640+0000","last_change":"2024-03-18T00:49:03.417640+0000","last_active":"2024-03-18T00:49:03.417640+0000","last_peered":"2024-03-18T00:49:03.417640+0000","last_clean":"2024-03-18T00:49:03.417640+0000","last_became_active":"2024-03-18T00:49:03.405217+0000","last_became_peered":"2024-03-18T00:49:03.405217+0000","last_unstale":"2024-03-18T00:49:03.417640+0000","last_undegraded":"2024-03-18T00:49:03.417640+0000","last_fullsized":"2024-03-18T00:49:03.417640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:00:13.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507763+0000","last_change":"2024-03-18T00:48:06.544084+0000","last_active":"2024-03-18T00:49:31.507763+0000","last_peered":"2024-03-18T00:49:31.507763+0000","last_clean":"2024-03-18T00:49:31.507763+0000","last_became_active":"2024-03-18T00:48:06.543419+0000","last_became_peered":"2024-03-18T00:48:06.543419+0000","last_unstale":"2024-03-18T00:49:31.507763+0000","last_undegraded":"2024-03-18T00:49:31.507763+0000","last_fullsized":"2024-03-18T00:49:31.507763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:15:28.527473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087635+0000","last_change":"2024-03-18T00:48:05.528347+0000","last_active":"2024-03-18T00:49:03.087635+0000","last_peered":"2024-03-18T00:49:03.087635+0000","last_clean":"2024-03-18T00:49:03.087635+0000","last_became_active":"2024-03-18T00:48:05.527761+0000","last_became_peered":"2024-03-18T00:48:05.527761+0000","last_unstale":"2024-03-18T00:49:03.087635+0000","last_undegraded":"2024-03-18T00:49:03.087635+0000","last_fullsized":"2024-03-18T00:49:03.087635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:36:58.562495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.503220+0000","last_change":"2024-03-18T00:48:06.541862+0000","last_active":"2024-03-18T00:49:31.503220+0000","last_peered":"2024-03-18T00:49:31.503220+0000","last_clean":"2024-03-18T00:49:31.503220+0000","last_became_active":"2024-03-18T00:48:06.541488+0000","last_became_peered":"2024-03-18T00:48:06.541488+0000","last_unstale":"2024-03-18T00:49:31.503220+0000","last_undegraded":"2024-03-18T00:49:31.503220+0000","last_fullsized":"2024-03-18T00:49:31.503220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:47:40.724244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084618+0000","last_change":"2024-03-18T00:48:05.535498+0000","last_active":"2024-03-18T00:49:03.084618+0000","last_peered":"2024-03-18T00:49:03.084618+0000","last_clean":"2024-03-18T00:49:03.084618+0000","last_became_active":"2024-03-18T00:48:05.534907+0000","last_became_peered":"2024-03-18T00:48:05.534907+0000","last_unstale":"2024-03-18T00:49:03.084618+0000","last_undegraded":"2024-03-18T00:49:03.084618+0000","last_fullsized":"2024-03-18T00:49:03.084618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:47:50.196929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504894+0000","last_change":"2024-03-18T00:48:06.534423+0000","last_active":"2024-03-18T00:49:31.504894+0000","last_peered":"2024-03-18T00:49:31.504894+0000","last_clean":"2024-03-18T00:49:31.504894+0000","last_became_active":"2024-03-18T00:48:06.534091+0000","last_became_peered":"2024-03-18T00:48:06.534091+0000","last_unstale":"2024-03-18T00:49:31.504894+0000","last_undegraded":"2024-03-18T00:49:31.504894+0000","last_fullsized":"2024-03-18T00:49:31.504894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:39:08.755634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:01.157977+0000","last_change":"2024-03-18T00:48:05.532891+0000","last_active":"2024-03-18T00:50:01.157977+0000","last_peered":"2024-03-18T00:50:01.157977+0000","last_clean":"2024-03-18T00:50:01.157977+0000","last_became_active":"2024-03-18T00:48:05.532515+0000","last_became_peered":"2024-03-18T00:48:05.532515+0000","last_unstale":"2024-03-18T00:50:01.157977+0000","last_undegraded":"2024-03-18T00:50:01.157977+0000","last_fullsized":"2024-03-18T00:50:01.157977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:46:52.056837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506095+0000","last_change":"2024-03-18T00:48:06.542992+0000","last_active":"2024-03-18T00:49:31.506095+0000","last_peered":"2024-03-18T00:49:31.506095+0000","last_clean":"2024-03-18T00:49:31.506095+0000","last_became_active":"2024-03-18T00:48:06.542617+0000","last_became_peered":"2024-03-18T00:48:06.542617+0000","last_unstale":"2024-03-18T00:49:31.506095+0000","last_undegraded":"2024-03-18T00:49:31.506095+0000","last_fullsized":"2024-03-18T00:49:31.506095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:22.736118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506043+0000","last_change":"2024-03-18T00:48:06.536018+0000","last_active":"2024-03-18T00:49:31.506043+0000","last_peered":"2024-03-18T00:49:31.506043+0000","last_clean":"2024-03-18T00:49:31.506043+0000","last_became_active":"2024-03-18T00:48:06.535656+0000","last_became_peered":"2024-03-18T00:48:06.535656+0000","last_unstale":"2024-03-18T00:49:31.506043+0000","last_undegraded":"2024-03-18T00:49:31.506043+0000","last_fullsized":"2024-03-18T00:49:31.506043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:33:00.492887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:06.146970+0000","last_change":"2024-03-18T00:48:05.536635+0000","last_active":"2024-03-18T00:50:06.146970+0000","last_peered":"2024-03-18T00:50:06.146970+0000","last_clean":"2024-03-18T00:50:06.146970+0000","last_became_active":"2024-03-18T00:48:05.535941+0000","last_became_peered":"2024-03-18T00:48:05.535941+0000","last_unstale":"2024-03-18T00:50:06.146970+0000","last_undegraded":"2024-03-18T00:50:06.146970+0000","last_fullsized":"2024-03-18T00:50:06.146970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:20:52.167217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.515797+0000","last_change":"2024-03-18T00:48:06.546266+0000","last_active":"2024-03-18T00:49:31.515797+0000","last_peered":"2024-03-18T00:49:31.515797+0000","last_clean":"2024-03-18T00:49:31.515797+0000","last_became_active":"2024-03-18T00:48:06.543565+0000","last_became_peered":"2024-03-18T00:48:06.543565+0000","last_unstale":"2024-03-18T00:49:31.515797+0000","last_undegraded":"2024-03-18T00:49:31.515797+0000","last_fullsized":"2024-03-18T00:49:31.515797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:49:09.673212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.403989+0000","last_change":"2024-03-18T00:48:05.526184+0000","last_active":"2024-03-18T00:49:03.403989+0000","last_peered":"2024-03-18T00:49:03.403989+0000","last_clean":"2024-03-18T00:49:03.403989+0000","last_became_active":"2024-03-18T00:48:05.525011+0000","last_became_peered":"2024-03-18T00:48:05.525011+0000","last_unstale":"2024-03-18T00:49:03.403989+0000","last_undegraded":"2024-03-18T00:49:03.403989+0000","last_fullsized":"2024-03-18T00:49:03.403989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:12.690966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087462+0000","last_change":"2024-03-18T00:46:53.288768+0000","last_active":"2024-03-18T00:49:03.087462+0000","last_peered":"2024-03-18T00:49:03.087462+0000","last_clean":"2024-03-18T00:49:03.087462+0000","last_became_active":"2024-03-18T00:46:53.288394+0000","last_became_peered":"2024-03-18T00:46:53.288394+0000","last_unstale":"2024-03-18T00:49:03.087462+0000","last_undegraded":"2024-03-18T00:49:03.087462+0000","last_fullsized":"2024-03-18T00:49:03.087462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_clean_scrub_stamp":"2024-03-18T00:43:28.173150+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:49.949940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500421+0000","last_change":"2024-03-18T00:48:06.534383+0000","last_active":"2024-03-18T00:49:31.500421+0000","last_peered":"2024-03-18T00:49:31.500421+0000","last_clean":"2024-03-18T00:49:31.500421+0000","last_became_active":"2024-03-18T00:48:06.534085+0000","last_became_peered":"2024-03-18T00:48:06.534085+0000","last_unstale":"2024-03-18T00:49:31.500421+0000","last_undegraded":"2024-03-18T00:49:31.500421+0000","last_fullsized":"2024-03-18T00:49:31.500421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:54.021830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.407661+0000","last_change":"2024-03-18T00:49:03.407661+0000","last_active":"2024-03-18T00:49:03.407661+0000","last_peered":"2024-03-18T00:49:03.407661+0000","last_clean":"2024-03-18T00:49:03.407661+0000","last_became_active":"2024-03-18T00:49:03.405134+0000","last_became_peered":"2024-03-18T00:49:03.405134+0000","last_unstale":"2024-03-18T00:49:03.407661+0000","last_undegraded":"2024-03-18T00:49:03.407661+0000","last_fullsized":"2024-03-18T00:49:03.407661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:29:03.201556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.098300+0000","last_change":"2024-03-18T00:49:03.097885+0000","last_active":"2024-03-18T00:49:03.098300+0000","last_peered":"2024-03-18T00:49:03.098300+0000","last_clean":"2024-03-18T00:49:03.098300+0000","last_became_active":"2024-03-18T00:49:03.090574+0000","last_became_peered":"2024-03-18T00:49:03.090574+0000","last_unstale":"2024-03-18T00:49:03.098300+0000","last_undegraded":"2024-03-18T00:49:03.098300+0000","last_fullsized":"2024-03-18T00:49:03.098300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:22:30.194349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.499871+0000","last_change":"2024-03-18T00:48:06.530108+0000","last_active":"2024-03-18T00:49:31.499871+0000","last_peered":"2024-03-18T00:49:31.499871+0000","last_clean":"2024-03-18T00:49:31.499871+0000","last_became_active":"2024-03-18T00:48:06.529768+0000","last_became_peered":"2024-03-18T00:48:06.529768+0000","last_unstale":"2024-03-18T00:49:31.499871+0000","last_undegraded":"2024-03-18T00:49:31.499871+0000","last_fullsized":"2024-03-18T00:49:31.499871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:23:09.150305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'14","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.529747+0000","last_change":"2024-03-18T00:48:05.535703+0000","last_active":"2024-03-18T00:49:31.529747+0000","last_peered":"2024-03-18T00:49:31.529747+0000","last_clean":"2024-03-18T00:49:31.529747+0000","last_became_active":"2024-03-18T00:48:05.535086+0000","last_became_peered":"2024-03-18T00:48:05.535086+0000","last_unstale":"2024-03-18T00:49:31.529747+0000","last_undegraded":"2024-03-18T00:49:31.529747+0000","last_fullsized":"2024-03-18T00:49:31.529747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:10:24.172353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504017+0000","last_change":"2024-03-18T00:48:06.541281+0000","last_active":"2024-03-18T00:49:31.504017+0000","last_peered":"2024-03-18T00:49:31.504017+0000","last_clean":"2024-03-18T00:49:31.504017+0000","last_became_active":"2024-03-18T00:48:06.540946+0000","last_became_peered":"2024-03-18T00:48:06.540946+0000","last_unstale":"2024-03-18T00:49:31.504017+0000","last_undegraded":"2024-03-18T00:49:31.504017+0000","last_fullsized":"2024-03-18T00:49:31.504017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:02.987661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.404075+0000","last_change":"2024-03-18T00:48:05.531685+0000","last_active":"2024-03-18T00:49:03.404075+0000","last_peered":"2024-03-18T00:49:03.404075+0000","last_clean":"2024-03-18T00:49:03.404075+0000","last_became_active":"2024-03-18T00:48:05.531239+0000","last_became_peered":"2024-03-18T00:48:05.531239+0000","last_unstale":"2024-03-18T00:49:03.404075+0000","last_undegraded":"2024-03-18T00:49:03.404075+0000","last_fullsized":"2024-03-18T00:49:03.404075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:26:35.034842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502223+0000","last_change":"2024-03-18T00:48:06.546783+0000","last_active":"2024-03-18T00:49:31.502223+0000","last_peered":"2024-03-18T00:49:31.502223+0000","last_clean":"2024-03-18T00:49:31.502223+0000","last_became_active":"2024-03-18T00:48:06.546411+0000","last_became_peered":"2024-03-18T00:48:06.546411+0000","last_unstale":"2024-03-18T00:49:31.502223+0000","last_undegraded":"2024-03-18T00:49:31.502223+0000","last_fullsized":"2024-03-18T00:49:31.502223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:39:34.432104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089245+0000","last_change":"2024-03-18T00:48:05.534701+0000","last_active":"2024-03-18T00:49:03.089245+0000","last_peered":"2024-03-18T00:49:03.089245+0000","last_clean":"2024-03-18T00:49:03.089245+0000","last_became_active":"2024-03-18T00:48:05.534300+0000","last_became_peered":"2024-03-18T00:48:05.534300+0000","last_unstale":"2024-03-18T00:49:03.089245+0000","last_undegraded":"2024-03-18T00:49:03.089245+0000","last_fullsized":"2024-03-18T00:49:03.089245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:10.580001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.461798+0000","last_change":"2024-03-18T00:48:06.528286+0000","last_active":"2024-03-18T00:49:31.461798+0000","last_peered":"2024-03-18T00:49:31.461798+0000","last_clean":"2024-03-18T00:49:31.461798+0000","last_became_active":"2024-03-18T00:48:06.527947+0000","last_became_peered":"2024-03-18T00:48:06.527947+0000","last_unstale":"2024-03-18T00:49:31.461798+0000","last_undegraded":"2024-03-18T00:49:31.461798+0000","last_fullsized":"2024-03-18T00:49:31.461798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:34.953923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325566+0000","last_change":"2024-03-18T00:48:05.526530+0000","last_active":"2024-03-18T00:49:03.325566+0000","last_peered":"2024-03-18T00:49:03.325566+0000","last_clean":"2024-03-18T00:49:03.325566+0000","last_became_active":"2024-03-18T00:48:05.526210+0000","last_became_peered":"2024-03-18T00:48:05.526210+0000","last_unstale":"2024-03-18T00:49:03.325566+0000","last_undegraded":"2024-03-18T00:49:03.325566+0000","last_fullsized":"2024-03-18T00:49:03.325566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:11:11.094469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501176+0000","last_change":"2024-03-18T00:48:06.535316+0000","last_active":"2024-03-18T00:49:31.501176+0000","last_peered":"2024-03-18T00:49:31.501176+0000","last_clean":"2024-03-18T00:49:31.501176+0000","last_became_active":"2024-03-18T00:48:06.534945+0000","last_became_peered":"2024-03-18T00:48:06.534945+0000","last_unstale":"2024-03-18T00:49:31.501176+0000","last_undegraded":"2024-03-18T00:49:31.501176+0000","last_fullsized":"2024-03-18T00:49:31.501176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:04:32.570585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087310+0000","last_change":"2024-03-18T00:48:05.529300+0000","last_active":"2024-03-18T00:49:03.087310+0000","last_peered":"2024-03-18T00:49:03.087310+0000","last_clean":"2024-03-18T00:49:03.087310+0000","last_became_active":"2024-03-18T00:48:05.528935+0000","last_became_peered":"2024-03-18T00:48:05.528935+0000","last_unstale":"2024-03-18T00:49:03.087310+0000","last_undegraded":"2024-03-18T00:49:03.087310+0000","last_fullsized":"2024-03-18T00:49:03.087310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:29:45.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.498853+0000","last_change":"2024-03-18T00:48:06.536426+0000","last_active":"2024-03-18T00:49:31.498853+0000","last_peered":"2024-03-18T00:49:31.498853+0000","last_clean":"2024-03-18T00:49:31.498853+0000","last_became_active":"2024-03-18T00:48:06.536005+0000","last_became_peered":"2024-03-18T00:48:06.536005+0000","last_unstale":"2024-03-18T00:49:31.498853+0000","last_undegraded":"2024-03-18T00:49:31.498853+0000","last_fullsized":"2024-03-18T00:49:31.498853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:40:52.028794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087438+0000","last_change":"2024-03-18T00:48:05.536495+0000","last_active":"2024-03-18T00:49:03.087438+0000","last_peered":"2024-03-18T00:49:03.087438+0000","last_clean":"2024-03-18T00:49:03.087438+0000","last_became_active":"2024-03-18T00:48:05.536072+0000","last_became_peered":"2024-03-18T00:48:05.536072+0000","last_unstale":"2024-03-18T00:49:03.087438+0000","last_undegraded":"2024-03-18T00:49:03.087438+0000","last_fullsized":"2024-03-18T00:49:03.087438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:25.528261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.505766+0000","last_change":"2024-03-18T00:48:06.552900+0000","last_active":"2024-03-18T00:49:31.505766+0000","last_peered":"2024-03-18T00:49:31.505766+0000","last_clean":"2024-03-18T00:49:31.505766+0000","last_became_active":"2024-03-18T00:48:06.552568+0000","last_became_peered":"2024-03-18T00:48:06.552568+0000","last_unstale":"2024-03-18T00:49:31.505766+0000","last_undegraded":"2024-03-18T00:49:31.505766+0000","last_fullsized":"2024-03-18T00:49:31.505766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:53:25.254177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.155478+0000","last_change":"2024-03-18T00:48:05.537327+0000","last_active":"2024-03-18T00:50:05.155478+0000","last_peered":"2024-03-18T00:50:05.155478+0000","last_clean":"2024-03-18T00:50:05.155478+0000","last_became_active":"2024-03-18T00:48:05.536825+0000","last_became_peered":"2024-03-18T00:48:05.536825+0000","last_unstale":"2024-03-18T00:50:05.155478+0000","last_undegraded":"2024-03-18T00:50:05.155478+0000","last_fullsized":"2024-03-18T00:50:05.155478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:03.141142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.505260+0000","last_change":"2024-03-18T00:48:06.542593+0000","last_active":"2024-03-18T00:49:31.505260+0000","last_peered":"2024-03-18T00:49:31.505260+0000","last_clean":"2024-03-18T00:49:31.505260+0000","last_became_active":"2024-03-18T00:48:06.541624+0000","last_became_peered":"2024-03-18T00:48:06.541624+0000","last_unstale":"2024-03-18T00:49:31.505260+0000","last_undegraded":"2024-03-18T00:49:31.505260+0000","last_fullsized":"2024-03-18T00:49:31.505260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:10:56.905221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089025+0000","last_change":"2024-03-18T00:48:05.529140+0000","last_active":"2024-03-18T00:49:03.089025+0000","last_peered":"2024-03-18T00:49:03.089025+0000","last_clean":"2024-03-18T00:49:03.089025+0000","last_became_active":"2024-03-18T00:48:05.528682+0000","last_became_peered":"2024-03-18T00:48:05.528682+0000","last_unstale":"2024-03-18T00:49:03.089025+0000","last_undegraded":"2024-03-18T00:49:03.089025+0000","last_fullsized":"2024-03-18T00:49:03.089025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:57:52.663150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.508245+0000","last_change":"2024-03-18T00:48:06.546027+0000","last_active":"2024-03-18T00:49:31.508245+0000","last_peered":"2024-03-18T00:49:31.508245+0000","last_clean":"2024-03-18T00:49:31.508245+0000","last_became_active":"2024-03-18T00:48:06.545670+0000","last_became_peered":"2024-03-18T00:48:06.545670+0000","last_unstale":"2024-03-18T00:49:31.508245+0000","last_undegraded":"2024-03-18T00:49:31.508245+0000","last_fullsized":"2024-03-18T00:49:31.508245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:47:05.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087890+0000","last_change":"2024-03-18T00:48:05.526929+0000","last_active":"2024-03-18T00:49:03.087890+0000","last_peered":"2024-03-18T00:49:03.087890+0000","last_clean":"2024-03-18T00:49:03.087890+0000","last_became_active":"2024-03-18T00:48:05.526530+0000","last_became_peered":"2024-03-18T00:48:05.526530+0000","last_unstale":"2024-03-18T00:49:03.087890+0000","last_undegraded":"2024-03-18T00:49:03.087890+0000","last_fullsized":"2024-03-18T00:49:03.087890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:51:25.110895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504777+0000","last_change":"2024-03-18T00:48:06.541223+0000","last_active":"2024-03-18T00:49:31.504777+0000","last_peered":"2024-03-18T00:49:31.504777+0000","last_clean":"2024-03-18T00:49:31.504777+0000","last_became_active":"2024-03-18T00:48:06.540898+0000","last_became_peered":"2024-03-18T00:48:06.540898+0000","last_unstale":"2024-03-18T00:49:31.504777+0000","last_undegraded":"2024-03-18T00:49:31.504777+0000","last_fullsized":"2024-03-18T00:49:31.504777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:20:22.749609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506812+0000","last_change":"2024-03-18T00:48:06.532906+0000","last_active":"2024-03-18T00:49:31.506812+0000","last_peered":"2024-03-18T00:49:31.506812+0000","last_clean":"2024-03-18T00:49:31.506812+0000","last_became_active":"2024-03-18T00:48:06.532583+0000","last_became_peered":"2024-03-18T00:48:06.532583+0000","last_unstale":"2024-03-18T00:49:31.506812+0000","last_undegraded":"2024-03-18T00:49:31.506812+0000","last_fullsized":"2024-03-18T00:49:31.506812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:06.699615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506031+0000","last_change":"2024-03-18T00:48:06.530020+0000","last_active":"2024-03-18T00:49:31.506031+0000","last_peered":"2024-03-18T00:49:31.506031+0000","last_clean":"2024-03-18T00:49:31.506031+0000","last_became_active":"2024-03-18T00:48:06.529709+0000","last_became_peered":"2024-03-18T00:48:06.529709+0000","last_unstale":"2024-03-18T00:49:31.506031+0000","last_undegraded":"2024-03-18T00:49:31.506031+0000","last_fullsized":"2024-03-18T00:49:31.506031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:07.305537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500282+0000","last_change":"2024-03-18T00:48:06.537658+0000","last_active":"2024-03-18T00:49:31.500282+0000","last_peered":"2024-03-18T00:49:31.500282+0000","last_clean":"2024-03-18T00:49:31.500282+0000","last_became_active":"2024-03-18T00:48:06.537339+0000","last_became_peered":"2024-03-18T00:48:06.537339+0000","last_unstale":"2024-03-18T00:49:31.500282+0000","last_undegraded":"2024-03-18T00:49:31.500282+0000","last_fullsized":"2024-03-18T00:49:31.500282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:18:25.753168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497779+0000","last_change":"2024-03-18T00:48:06.544514+0000","last_active":"2024-03-18T00:49:31.497779+0000","last_peered":"2024-03-18T00:49:31.497779+0000","last_clean":"2024-03-18T00:49:31.497779+0000","last_became_active":"2024-03-18T00:48:06.544165+0000","last_became_peered":"2024-03-18T00:48:06.544165+0000","last_unstale":"2024-03-18T00:49:31.497779+0000","last_undegraded":"2024-03-18T00:49:31.497779+0000","last_fullsized":"2024-03-18T00:49:31.497779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:48:02.498940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501247+0000","last_change":"2024-03-18T00:49:03.090610+0000","last_active":"2024-03-18T00:49:31.501247+0000","last_peered":"2024-03-18T00:49:31.501247+0000","last_clean":"2024-03-18T00:49:31.501247+0000","last_became_active":"2024-03-18T00:49:03.090014+0000","last_became_peered":"2024-03-18T00:49:03.090014+0000","last_unstale":"2024-03-18T00:49:31.501247+0000","last_undegraded":"2024-03-18T00:49:31.501247+0000","last_fullsized":"2024-03-18T00:49:31.501247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:55:12.210223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.503021+0000","last_change":"2024-03-18T00:48:06.533837+0000","last_active":"2024-03-18T00:49:31.503021+0000","last_peered":"2024-03-18T00:49:31.503021+0000","last_clean":"2024-03-18T00:49:31.503021+0000","last_became_active":"2024-03-18T00:48:06.533418+0000","last_became_peered":"2024-03-18T00:48:06.533418+0000","last_unstale":"2024-03-18T00:49:31.503021+0000","last_undegraded":"2024-03-18T00:49:31.503021+0000","last_fullsized":"2024-03-18T00:49:31.503021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:10.873312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.499996+0000","last_change":"2024-03-18T00:48:06.527394+0000","last_active":"2024-03-18T00:49:31.499996+0000","last_peered":"2024-03-18T00:49:31.499996+0000","last_clean":"2024-03-18T00:49:31.499996+0000","last_became_active":"2024-03-18T00:48:06.526879+0000","last_became_peered":"2024-03-18T00:48:06.526879+0000","last_unstale":"2024-03-18T00:49:31.499996+0000","last_undegraded":"2024-03-18T00:49:31.499996+0000","last_fullsized":"2024-03-18T00:49:31.499996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:23:07.397259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500631+0000","last_change":"2024-03-18T00:48:06.542645+0000","last_active":"2024-03-18T00:49:31.500631+0000","last_peered":"2024-03-18T00:49:31.500631+0000","last_clean":"2024-03-18T00:49:31.500631+0000","last_became_active":"2024-03-18T00:48:06.542073+0000","last_became_peered":"2024-03-18T00:48:06.542073+0000","last_unstale":"2024-03-18T00:49:31.500631+0000","last_undegraded":"2024-03-18T00:49:31.500631+0000","last_fullsized":"2024-03-18T00:49:31.500631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:08:44.479318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507457+0000","last_change":"2024-03-18T00:48:06.541698+0000","last_active":"2024-03-18T00:49:31.507457+0000","last_peered":"2024-03-18T00:49:31.507457+0000","last_clean":"2024-03-18T00:49:31.507457+0000","last_became_active":"2024-03-18T00:48:06.541378+0000","last_became_peered":"2024-03-18T00:48:06.541378+0000","last_unstale":"2024-03-18T00:49:31.507457+0000","last_undegraded":"2024-03-18T00:49:31.507457+0000","last_fullsized":"2024-03-18T00:49:31.507457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:42:23.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502862+0000","last_change":"2024-03-18T00:48:06.533184+0000","last_active":"2024-03-18T00:49:31.502862+0000","last_peered":"2024-03-18T00:49:31.502862+0000","last_clean":"2024-03-18T00:49:31.502862+0000","last_became_active":"2024-03-18T00:48:06.532034+0000","last_became_peered":"2024-03-18T00:48:06.532034+0000","last_unstale":"2024-03-18T00:49:31.502862+0000","last_undegraded":"2024-03-18T00:49:31.502862+0000","last_fullsized":"2024-03-18T00:49:31.502862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:45.265799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.515864+0000","last_change":"2024-03-18T00:48:06.545941+0000","last_active":"2024-03-18T00:49:31.515864+0000","last_peered":"2024-03-18T00:49:31.515864+0000","last_clean":"2024-03-18T00:49:31.515864+0000","last_became_active":"2024-03-18T00:48:06.545402+0000","last_became_peered":"2024-03-18T00:48:06.545402+0000","last_unstale":"2024-03-18T00:49:31.515864+0000","last_undegraded":"2024-03-18T00:49:31.515864+0000","last_fullsized":"2024-03-18T00:49:31.515864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:11:45.512849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.505008+0000","last_change":"2024-03-18T00:48:06.541513+0000","last_active":"2024-03-18T00:49:31.505008+0000","last_peered":"2024-03-18T00:49:31.505008+0000","last_clean":"2024-03-18T00:49:31.505008+0000","last_became_active":"2024-03-18T00:48:06.541128+0000","last_became_peered":"2024-03-18T00:48:06.541128+0000","last_unstale":"2024-03-18T00:49:31.505008+0000","last_undegraded":"2024-03-18T00:49:31.505008+0000","last_fullsized":"2024-03-18T00:49:31.505008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:42:10.635836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501522+0000","last_change":"2024-03-18T00:48:06.542316+0000","last_active":"2024-03-18T00:49:31.501522+0000","last_peered":"2024-03-18T00:49:31.501522+0000","last_clean":"2024-03-18T00:49:31.501522+0000","last_became_active":"2024-03-18T00:48:06.541914+0000","last_became_peered":"2024-03-18T00:48:06.541914+0000","last_unstale":"2024-03-18T00:49:31.501522+0000","last_undegraded":"2024-03-18T00:49:31.501522+0000","last_fullsized":"2024-03-18T00:49:31.501522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:46.293240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249372,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73179,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1128,"num_read_kb":670,"num_write":252,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1237,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":214607,"data_compressed":5589,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":461614,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612651,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":1116986,"data_compressed":43768,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":459970,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":460236,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":1053007,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462976,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459687,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":521895,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":461026,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":527465,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":459512,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888137,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":459660,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724612,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1051531,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63047,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68617,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":67653,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T00:50:07.984 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-18T00:50:07.984 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T00:50:08.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.129+0000 7fd88ff59640 1 -- 172.21.15.49:0/1262299701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8881018f0 msgr2=0x7fd888101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.129+0000 7fd88ff59640 1 --2- 172.21.15.49:0/1262299701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8881018f0 0x7fd888101d70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd878009900 tx=0x7fd87802bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:08.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.129+0000 7fd88ff59640 1 -- 172.21.15.49:0/1262299701 shutdown_connections 2024-03-18T00:50:08.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.129+0000 7fd88ff59640 1 --2- 172.21.15.49:0/1262299701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd8881022b0 0x7fd888108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.129+0000 7fd88ff59640 1 --2- 172.21.15.49:0/1262299701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8881018f0 0x7fd888101d70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.129+0000 7fd88ff59640 1 --2- 172.21.15.49:0/1262299701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd888100700 0x7fd888100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.129+0000 7fd88ff59640 1 -- 172.21.15.49:0/1262299701 >> 172.21.15.49:0/1262299701 conn(0x7fd8880fbc70 msgr2=0x7fd8880fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 -- 172.21.15.49:0/1262299701 shutdown_connections 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 -- 172.21.15.49:0/1262299701 wait complete. 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 Processor -- start 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 -- start start 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd888100700 0x7fd888197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8881018f0 0x7fd8881980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8881022b0 0x7fd88819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd88810c360 con 0x7fd8881018f0 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd88810c1e0 con 0x7fd8881022b0 2024-03-18T00:50:08.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88ff59640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd88810c4e0 con 0x7fd888100700 2024-03-18T00:50:08.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88d4f4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8881018f0 0x7fd8881980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88dcf5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd888100700 0x7fd888197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88dcf5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd888100700 0x7fd888197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:41578/0 (socket says 172.21.15.49:41578) 2024-03-18T00:50:08.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.130+0000 7fd88dcf5640 1 -- 172.21.15.49:0/344252050 learned_addr learned my addr 172.21.15.49:0/344252050 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:08.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88e4f6640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8881022b0 0x7fd88819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88dcf5640 1 -- 172.21.15.49:0/344252050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8881022b0 msgr2=0x7fd88819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88dcf5640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8881022b0 0x7fd88819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88dcf5640 1 -- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8881018f0 msgr2=0x7fd8881980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88dcf5640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8881018f0 0x7fd8881980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88dcf5640 1 -- 172.21.15.49:0/344252050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88819f730 con 0x7fd888100700 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88d4f4640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8881018f0 0x7fd8881980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88dcf5640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd888100700 0x7fd888197b60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd88000e9c0 tx=0x7fd88000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd87effd640 1 -- 172.21.15.49:0/344252050 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88000cde0 con 0x7fd888100700 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88819fa20 con 0x7fd888100700 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd87effd640 1 -- 172.21.15.49:0/344252050 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd880010910 con 0x7fd888100700 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd87effd640 1 -- 172.21.15.49:0/344252050 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd880018a10 con 0x7fd888100700 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.131+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88819ff90 con 0x7fd888100700 2024-03-18T00:50:08.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.132+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8881a01c0 con 0x7fd888100700 2024-03-18T00:50:08.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.133+0000 7fd87effd640 1 -- 172.21.15.49:0/344252050 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd88001f020 con 0x7fd888100700 2024-03-18T00:50:08.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.134+0000 7fd87effd640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd8740a2800 0x7fd8740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.134+0000 7fd88d4f4640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd8740a2800 0x7fd8740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.134+0000 7fd87effd640 1 -- 172.21.15.49:0/344252050 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd8800cedc0 con 0x7fd888100700 2024-03-18T00:50:08.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.134+0000 7fd88d4f4640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd8740a2800 0x7fd8740a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd87802c7b0 tx=0x7fd878005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:08.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.137+0000 7fd87effd640 1 -- 172.21.15.49:0/344252050 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd88009be40 con 0x7fd888100700 2024-03-18T00:50:08.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.318+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 --> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd8881a0390 con 0x7fd8740a2800 2024-03-18T00:50:08.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.326+0000 7fd87effd640 1 -- 172.21.15.49:0/344252050 <== mgr.14150 v2:172.21.15.49:6800/1246732627 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259166 (secure 0 0 0) 0x7fd8881a0390 con 0x7fd8740a2800 2024-03-18T00:50:08.328 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:08.330 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-03-18T00:50:08.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.328+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd8740a2800 msgr2=0x7fd8740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.328+0000 7fd88ff59640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd8740a2800 0x7fd8740a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd87802c7b0 tx=0x7fd878005fd0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd888100700 msgr2=0x7fd888197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd888100700 0x7fd888197b60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd88000e9c0 tx=0x7fd88000ee80 comp rx=0 tx=0).stop 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 shutdown_connections 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd8740a2800 0x7fd8740a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd8881022b0 0x7fd88819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd8881018f0 0x7fd8881980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 --2- 172.21.15.49:0/344252050 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd888100700 0x7fd888197b60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 >> 172.21.15.49:0/344252050 conn(0x7fd8880fbc70 msgr2=0x7fd8880fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 shutdown_connections 2024-03-18T00:50:08.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.329+0000 7fd88ff59640 1 -- 172.21.15.49:0/344252050 wait complete. 2024-03-18T00:50:08.340 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-03-18T00:50:07.490063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663781,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1254,"num_read_kb":779,"num_write":435,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1181973,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":857,"ondisk_log_size":857,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356796,"kb_used_data":11804,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585028,"statfs":{"total":1151940427776,"available":1151575068672,"internally_reserved":0,"allocated":12087296,"data_stored":7492589,"data_compressed":183621,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4930,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":123,"num_read_kb":62,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002823"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504686+0000","last_change":"2024-03-18T00:48:06.536034+0000","last_active":"2024-03-18T00:49:31.504686+0000","last_peered":"2024-03-18T00:49:31.504686+0000","last_clean":"2024-03-18T00:49:31.504686+0000","last_became_active":"2024-03-18T00:48:06.535694+0000","last_became_peered":"2024-03-18T00:48:06.535694+0000","last_unstale":"2024-03-18T00:49:31.504686+0000","last_undegraded":"2024-03-18T00:49:31.504686+0000","last_fullsized":"2024-03-18T00:49:31.504686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:24:33.785669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502451+0000","last_change":"2024-03-18T00:48:06.536741+0000","last_active":"2024-03-18T00:49:31.502451+0000","last_peered":"2024-03-18T00:49:31.502451+0000","last_clean":"2024-03-18T00:49:31.502451+0000","last_became_active":"2024-03-18T00:48:06.535865+0000","last_became_peered":"2024-03-18T00:48:06.535865+0000","last_unstale":"2024-03-18T00:49:31.502451+0000","last_undegraded":"2024-03-18T00:49:31.502451+0000","last_fullsized":"2024-03-18T00:49:31.502451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:53:58.104983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501939+0000","last_change":"2024-03-18T00:48:06.537162+0000","last_active":"2024-03-18T00:49:31.501939+0000","last_peered":"2024-03-18T00:49:31.501939+0000","last_clean":"2024-03-18T00:49:31.501939+0000","last_became_active":"2024-03-18T00:48:06.536835+0000","last_became_peered":"2024-03-18T00:48:06.536835+0000","last_unstale":"2024-03-18T00:49:31.501939+0000","last_undegraded":"2024-03-18T00:49:31.501939+0000","last_fullsized":"2024-03-18T00:49:31.501939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:15:08.519180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.486309+0000","last_change":"2024-03-18T00:48:06.540084+0000","last_active":"2024-03-18T00:49:31.486309+0000","last_peered":"2024-03-18T00:49:31.486309+0000","last_clean":"2024-03-18T00:49:31.486309+0000","last_became_active":"2024-03-18T00:48:06.539736+0000","last_became_peered":"2024-03-18T00:48:06.539736+0000","last_unstale":"2024-03-18T00:49:31.486309+0000","last_undegraded":"2024-03-18T00:49:31.486309+0000","last_fullsized":"2024-03-18T00:49:31.486309+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:37:51.748603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497212+0000","last_change":"2024-03-18T00:48:06.540022+0000","last_active":"2024-03-18T00:49:31.497212+0000","last_peered":"2024-03-18T00:49:31.497212+0000","last_clean":"2024-03-18T00:49:31.497212+0000","last_became_active":"2024-03-18T00:48:06.539665+0000","last_became_peered":"2024-03-18T00:48:06.539665+0000","last_unstale":"2024-03-18T00:49:31.497212+0000","last_undegraded":"2024-03-18T00:49:31.497212+0000","last_fullsized":"2024-03-18T00:49:31.497212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:55:52.448279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506091+0000","last_change":"2024-03-18T00:48:06.538672+0000","last_active":"2024-03-18T00:49:31.506091+0000","last_peered":"2024-03-18T00:49:31.506091+0000","last_clean":"2024-03-18T00:49:31.506091+0000","last_became_active":"2024-03-18T00:48:06.538189+0000","last_became_peered":"2024-03-18T00:48:06.538189+0000","last_unstale":"2024-03-18T00:49:31.506091+0000","last_undegraded":"2024-03-18T00:49:31.506091+0000","last_fullsized":"2024-03-18T00:49:31.506091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:09:04.672255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506665+0000","last_change":"2024-03-18T00:48:06.546417+0000","last_active":"2024-03-18T00:49:31.506665+0000","last_peered":"2024-03-18T00:49:31.506665+0000","last_clean":"2024-03-18T00:49:31.506665+0000","last_became_active":"2024-03-18T00:48:06.545759+0000","last_became_peered":"2024-03-18T00:48:06.545759+0000","last_unstale":"2024-03-18T00:49:31.506665+0000","last_undegraded":"2024-03-18T00:49:31.506665+0000","last_fullsized":"2024-03-18T00:49:31.506665+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:53:56.759028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.508198+0000","last_change":"2024-03-18T00:48:06.545875+0000","last_active":"2024-03-18T00:49:31.508198+0000","last_peered":"2024-03-18T00:49:31.508198+0000","last_clean":"2024-03-18T00:49:31.508198+0000","last_became_active":"2024-03-18T00:48:06.545316+0000","last_became_peered":"2024-03-18T00:48:06.545316+0000","last_unstale":"2024-03-18T00:49:31.508198+0000","last_undegraded":"2024-03-18T00:49:31.508198+0000","last_fullsized":"2024-03-18T00:49:31.508198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:22:40.998671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.512026+0000","last_change":"2024-03-18T00:48:06.533947+0000","last_active":"2024-03-18T00:49:31.512026+0000","last_peered":"2024-03-18T00:49:31.512026+0000","last_clean":"2024-03-18T00:49:31.512026+0000","last_became_active":"2024-03-18T00:48:06.533615+0000","last_became_peered":"2024-03-18T00:48:06.533615+0000","last_unstale":"2024-03-18T00:49:31.512026+0000","last_undegraded":"2024-03-18T00:49:31.512026+0000","last_fullsized":"2024-03-18T00:49:31.512026+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:32:16.750269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507310+0000","last_change":"2024-03-18T00:48:06.537614+0000","last_active":"2024-03-18T00:49:31.507310+0000","last_peered":"2024-03-18T00:49:31.507310+0000","last_clean":"2024-03-18T00:49:31.507310+0000","last_became_active":"2024-03-18T00:48:06.536617+0000","last_became_peered":"2024-03-18T00:48:06.536617+0000","last_unstale":"2024-03-18T00:49:31.507310+0000","last_undegraded":"2024-03-18T00:49:31.507310+0000","last_fullsized":"2024-03-18T00:49:31.507310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:02:52.222603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502441+0000","last_change":"2024-03-18T00:48:06.540363+0000","last_active":"2024-03-18T00:49:31.502441+0000","last_peered":"2024-03-18T00:49:31.502441+0000","last_clean":"2024-03-18T00:49:31.502441+0000","last_became_active":"2024-03-18T00:48:06.540011+0000","last_became_peered":"2024-03-18T00:48:06.540011+0000","last_unstale":"2024-03-18T00:49:31.502441+0000","last_undegraded":"2024-03-18T00:49:31.502441+0000","last_fullsized":"2024-03-18T00:49:31.502441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:23:10.034444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504057+0000","last_change":"2024-03-18T00:48:06.532611+0000","last_active":"2024-03-18T00:49:31.504057+0000","last_peered":"2024-03-18T00:49:31.504057+0000","last_clean":"2024-03-18T00:49:31.504057+0000","last_became_active":"2024-03-18T00:48:06.532301+0000","last_became_peered":"2024-03-18T00:48:06.532301+0000","last_unstale":"2024-03-18T00:49:31.504057+0000","last_undegraded":"2024-03-18T00:49:31.504057+0000","last_fullsized":"2024-03-18T00:49:31.504057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:54:40.795105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.508264+0000","last_change":"2024-03-18T00:48:06.531993+0000","last_active":"2024-03-18T00:49:31.508264+0000","last_peered":"2024-03-18T00:49:31.508264+0000","last_clean":"2024-03-18T00:49:31.508264+0000","last_became_active":"2024-03-18T00:48:06.531677+0000","last_became_peered":"2024-03-18T00:48:06.531677+0000","last_unstale":"2024-03-18T00:49:31.508264+0000","last_undegraded":"2024-03-18T00:49:31.508264+0000","last_fullsized":"2024-03-18T00:49:31.508264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:48:24.865299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501483+0000","last_change":"2024-03-18T00:48:06.536830+0000","last_active":"2024-03-18T00:49:31.501483+0000","last_peered":"2024-03-18T00:49:31.501483+0000","last_clean":"2024-03-18T00:49:31.501483+0000","last_became_active":"2024-03-18T00:48:06.535908+0000","last_became_peered":"2024-03-18T00:48:06.535908+0000","last_unstale":"2024-03-18T00:49:31.501483+0000","last_undegraded":"2024-03-18T00:49:31.501483+0000","last_fullsized":"2024-03-18T00:49:31.501483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:23:26.155130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.493074+0000","last_change":"2024-03-18T00:48:06.543641+0000","last_active":"2024-03-18T00:49:31.493074+0000","last_peered":"2024-03-18T00:49:31.493074+0000","last_clean":"2024-03-18T00:49:31.493074+0000","last_became_active":"2024-03-18T00:48:06.543293+0000","last_became_peered":"2024-03-18T00:48:06.543293+0000","last_unstale":"2024-03-18T00:49:31.493074+0000","last_undegraded":"2024-03-18T00:49:31.493074+0000","last_fullsized":"2024-03-18T00:49:31.493074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:52.406255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.510399+0000","last_change":"2024-03-18T00:48:06.544732+0000","last_active":"2024-03-18T00:49:31.510399+0000","last_peered":"2024-03-18T00:49:31.510399+0000","last_clean":"2024-03-18T00:49:31.510399+0000","last_became_active":"2024-03-18T00:48:06.544400+0000","last_became_peered":"2024-03-18T00:48:06.544400+0000","last_unstale":"2024-03-18T00:49:31.510399+0000","last_undegraded":"2024-03-18T00:49:31.510399+0000","last_fullsized":"2024-03-18T00:49:31.510399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:23:39.655434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504899+0000","last_change":"2024-03-18T00:48:06.536595+0000","last_active":"2024-03-18T00:49:31.504899+0000","last_peered":"2024-03-18T00:49:31.504899+0000","last_clean":"2024-03-18T00:49:31.504899+0000","last_became_active":"2024-03-18T00:48:06.535874+0000","last_became_peered":"2024-03-18T00:48:06.535874+0000","last_unstale":"2024-03-18T00:49:31.504899+0000","last_undegraded":"2024-03-18T00:49:31.504899+0000","last_fullsized":"2024-03-18T00:49:31.504899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:21:55.603614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.515187+0000","last_change":"2024-03-18T00:48:06.546676+0000","last_active":"2024-03-18T00:49:31.515187+0000","last_peered":"2024-03-18T00:49:31.515187+0000","last_clean":"2024-03-18T00:49:31.515187+0000","last_became_active":"2024-03-18T00:48:06.546356+0000","last_became_peered":"2024-03-18T00:48:06.546356+0000","last_unstale":"2024-03-18T00:49:31.515187+0000","last_undegraded":"2024-03-18T00:49:31.515187+0000","last_fullsized":"2024-03-18T00:49:31.515187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:04:35.467415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500375+0000","last_change":"2024-03-18T00:48:06.532184+0000","last_active":"2024-03-18T00:49:31.500375+0000","last_peered":"2024-03-18T00:49:31.500375+0000","last_clean":"2024-03-18T00:49:31.500375+0000","last_became_active":"2024-03-18T00:48:06.531816+0000","last_became_peered":"2024-03-18T00:48:06.531816+0000","last_unstale":"2024-03-18T00:49:31.500375+0000","last_undegraded":"2024-03-18T00:49:31.500375+0000","last_fullsized":"2024-03-18T00:49:31.500375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:09:03.113024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":146,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:03.159413+0000","last_change":"2024-03-18T00:48:05.530213+0000","last_active":"2024-03-18T00:50:03.159413+0000","last_peered":"2024-03-18T00:50:03.159413+0000","last_clean":"2024-03-18T00:50:03.159413+0000","last_became_active":"2024-03-18T00:48:05.529897+0000","last_became_peered":"2024-03-18T00:48:05.529897+0000","last_unstale":"2024-03-18T00:50:03.159413+0000","last_undegraded":"2024-03-18T00:50:03.159413+0000","last_fullsized":"2024-03-18T00:50:03.159413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:49.003362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":114,"num_read_kb":114,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506113+0000","last_change":"2024-03-18T00:48:06.544437+0000","last_active":"2024-03-18T00:49:31.506113+0000","last_peered":"2024-03-18T00:49:31.506113+0000","last_clean":"2024-03-18T00:49:31.506113+0000","last_became_active":"2024-03-18T00:48:06.544125+0000","last_became_peered":"2024-03-18T00:48:06.544125+0000","last_unstale":"2024-03-18T00:49:31.506113+0000","last_undegraded":"2024-03-18T00:49:31.506113+0000","last_fullsized":"2024-03-18T00:49:31.506113+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:56:42.917032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325994+0000","last_change":"2024-03-18T00:48:05.536249+0000","last_active":"2024-03-18T00:49:03.325994+0000","last_peered":"2024-03-18T00:49:03.325994+0000","last_clean":"2024-03-18T00:49:03.325994+0000","last_became_active":"2024-03-18T00:48:05.535954+0000","last_became_peered":"2024-03-18T00:48:05.535954+0000","last_unstale":"2024-03-18T00:49:03.325994+0000","last_undegraded":"2024-03-18T00:49:03.325994+0000","last_fullsized":"2024-03-18T00:49:03.325994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:18:52.341453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497426+0000","last_change":"2024-03-18T00:48:06.541377+0000","last_active":"2024-03-18T00:49:31.497426+0000","last_peered":"2024-03-18T00:49:31.497426+0000","last_clean":"2024-03-18T00:49:31.497426+0000","last_became_active":"2024-03-18T00:48:06.540937+0000","last_became_peered":"2024-03-18T00:48:06.540937+0000","last_unstale":"2024-03-18T00:49:31.497426+0000","last_undegraded":"2024-03-18T00:49:31.497426+0000","last_fullsized":"2024-03-18T00:49:31.497426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:51:55.268517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088568+0000","last_change":"2024-03-18T00:48:05.534372+0000","last_active":"2024-03-18T00:49:03.088568+0000","last_peered":"2024-03-18T00:49:03.088568+0000","last_clean":"2024-03-18T00:49:03.088568+0000","last_became_active":"2024-03-18T00:48:05.534034+0000","last_became_peered":"2024-03-18T00:48:05.534034+0000","last_unstale":"2024-03-18T00:49:03.088568+0000","last_undegraded":"2024-03-18T00:49:03.088568+0000","last_fullsized":"2024-03-18T00:49:03.088568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:58:55.013167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.513707+0000","last_change":"2024-03-18T00:48:06.527276+0000","last_active":"2024-03-18T00:49:31.513707+0000","last_peered":"2024-03-18T00:49:31.513707+0000","last_clean":"2024-03-18T00:49:31.513707+0000","last_became_active":"2024-03-18T00:48:06.526904+0000","last_became_peered":"2024-03-18T00:48:06.526904+0000","last_unstale":"2024-03-18T00:49:31.513707+0000","last_undegraded":"2024-03-18T00:49:31.513707+0000","last_fullsized":"2024-03-18T00:49:31.513707+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:16:10.731256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087607+0000","last_change":"2024-03-18T00:48:05.528188+0000","last_active":"2024-03-18T00:49:03.087607+0000","last_peered":"2024-03-18T00:49:03.087607+0000","last_clean":"2024-03-18T00:49:03.087607+0000","last_became_active":"2024-03-18T00:48:05.527788+0000","last_became_peered":"2024-03-18T00:48:05.527788+0000","last_unstale":"2024-03-18T00:49:03.087607+0000","last_undegraded":"2024-03-18T00:49:03.087607+0000","last_fullsized":"2024-03-18T00:49:03.087607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:47.939771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507894+0000","last_change":"2024-03-18T00:49:03.092920+0000","last_active":"2024-03-18T00:49:31.507894+0000","last_peered":"2024-03-18T00:49:31.507894+0000","last_clean":"2024-03-18T00:49:31.507894+0000","last_became_active":"2024-03-18T00:49:03.092569+0000","last_became_peered":"2024-03-18T00:49:03.092569+0000","last_unstale":"2024-03-18T00:49:31.507894+0000","last_undegraded":"2024-03-18T00:49:31.507894+0000","last_fullsized":"2024-03-18T00:49:31.507894+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:15:02.246688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'8","reported_seq":85,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.155786+0000","last_change":"2024-03-18T00:49:03.100514+0000","last_active":"2024-03-18T00:50:05.155786+0000","last_peered":"2024-03-18T00:50:05.155786+0000","last_clean":"2024-03-18T00:50:05.155786+0000","last_became_active":"2024-03-18T00:49:03.090898+0000","last_became_peered":"2024-03-18T00:49:03.090898+0000","last_unstale":"2024-03-18T00:50:05.155786+0000","last_undegraded":"2024-03-18T00:50:05.155786+0000","last_fullsized":"2024-03-18T00:50:05.155786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:19:39.943819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":2,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.494376+0000","last_change":"2024-03-18T00:48:06.537405+0000","last_active":"2024-03-18T00:49:31.494376+0000","last_peered":"2024-03-18T00:49:31.494376+0000","last_clean":"2024-03-18T00:49:31.494376+0000","last_became_active":"2024-03-18T00:48:06.536919+0000","last_became_peered":"2024-03-18T00:48:06.536919+0000","last_unstale":"2024-03-18T00:49:31.494376+0000","last_undegraded":"2024-03-18T00:49:31.494376+0000","last_fullsized":"2024-03-18T00:49:31.494376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:31:59.863571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088122+0000","last_change":"2024-03-18T00:48:05.528588+0000","last_active":"2024-03-18T00:49:03.088122+0000","last_peered":"2024-03-18T00:49:03.088122+0000","last_clean":"2024-03-18T00:49:03.088122+0000","last_became_active":"2024-03-18T00:48:05.528263+0000","last_became_peered":"2024-03-18T00:48:05.528263+0000","last_unstale":"2024-03-18T00:49:03.088122+0000","last_undegraded":"2024-03-18T00:49:03.088122+0000","last_fullsized":"2024-03-18T00:49:03.088122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:18:10.283278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506438+0000","last_change":"2024-03-18T00:48:06.546323+0000","last_active":"2024-03-18T00:49:31.506438+0000","last_peered":"2024-03-18T00:49:31.506438+0000","last_clean":"2024-03-18T00:49:31.506438+0000","last_became_active":"2024-03-18T00:48:06.545956+0000","last_became_peered":"2024-03-18T00:48:06.545956+0000","last_unstale":"2024-03-18T00:49:31.506438+0000","last_undegraded":"2024-03-18T00:49:31.506438+0000","last_fullsized":"2024-03-18T00:49:31.506438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:13:32.618724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087983+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:49:03.087983+0000","last_peered":"2024-03-18T00:49:03.087983+0000","last_clean":"2024-03-18T00:49:03.087983+0000","last_became_active":"2024-03-18T00:48:05.525590+0000","last_became_peered":"2024-03-18T00:48:05.525590+0000","last_unstale":"2024-03-18T00:49:03.087983+0000","last_undegraded":"2024-03-18T00:49:03.087983+0000","last_fullsized":"2024-03-18T00:49:03.087983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:12:19.357331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502517+0000","last_change":"2024-03-18T00:48:06.538798+0000","last_active":"2024-03-18T00:49:31.502517+0000","last_peered":"2024-03-18T00:49:31.502517+0000","last_clean":"2024-03-18T00:49:31.502517+0000","last_became_active":"2024-03-18T00:48:06.538457+0000","last_became_peered":"2024-03-18T00:48:06.538457+0000","last_unstale":"2024-03-18T00:49:31.502517+0000","last_undegraded":"2024-03-18T00:49:31.502517+0000","last_fullsized":"2024-03-18T00:49:31.502517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:39:10.738683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088047+0000","last_change":"2024-03-18T00:48:05.528094+0000","last_active":"2024-03-18T00:49:03.088047+0000","last_peered":"2024-03-18T00:49:03.088047+0000","last_clean":"2024-03-18T00:49:03.088047+0000","last_became_active":"2024-03-18T00:48:05.527730+0000","last_became_peered":"2024-03-18T00:48:05.527730+0000","last_unstale":"2024-03-18T00:49:03.088047+0000","last_undegraded":"2024-03-18T00:49:03.088047+0000","last_fullsized":"2024-03-18T00:49:03.088047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:48:27.707505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.498967+0000","last_change":"2024-03-18T00:48:06.548676+0000","last_active":"2024-03-18T00:49:31.498967+0000","last_peered":"2024-03-18T00:49:31.498967+0000","last_clean":"2024-03-18T00:49:31.498967+0000","last_became_active":"2024-03-18T00:48:06.547693+0000","last_became_peered":"2024-03-18T00:48:06.547693+0000","last_unstale":"2024-03-18T00:49:31.498967+0000","last_undegraded":"2024-03-18T00:49:31.498967+0000","last_fullsized":"2024-03-18T00:49:31.498967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:13.849175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'6","reported_seq":253,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.154991+0000","last_change":"2024-03-18T00:48:05.523639+0000","last_active":"2024-03-18T00:50:05.154991+0000","last_peered":"2024-03-18T00:50:05.154991+0000","last_clean":"2024-03-18T00:50:05.154991+0000","last_became_active":"2024-03-18T00:48:05.523269+0000","last_became_peered":"2024-03-18T00:48:05.523269+0000","last_unstale":"2024-03-18T00:50:05.154991+0000","last_undegraded":"2024-03-18T00:50:05.154991+0000","last_fullsized":"2024-03-18T00:50:05.154991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:47:39.961247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":228,"num_read_kb":228,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.514641+0000","last_change":"2024-03-18T00:48:06.537786+0000","last_active":"2024-03-18T00:49:31.514641+0000","last_peered":"2024-03-18T00:49:31.514641+0000","last_clean":"2024-03-18T00:49:31.514641+0000","last_became_active":"2024-03-18T00:48:06.537488+0000","last_became_peered":"2024-03-18T00:48:06.537488+0000","last_unstale":"2024-03-18T00:49:31.514641+0000","last_undegraded":"2024-03-18T00:49:31.514641+0000","last_fullsized":"2024-03-18T00:49:31.514641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:45:02.095610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.096001+0000","last_change":"2024-03-18T00:49:03.096001+0000","last_active":"2024-03-18T00:49:03.096001+0000","last_peered":"2024-03-18T00:49:03.096001+0000","last_clean":"2024-03-18T00:49:03.096001+0000","last_became_active":"2024-03-18T00:49:03.090868+0000","last_became_peered":"2024-03-18T00:49:03.090868+0000","last_unstale":"2024-03-18T00:49:03.096001+0000","last_undegraded":"2024-03-18T00:49:03.096001+0000","last_fullsized":"2024-03-18T00:49:03.096001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:42:03.930010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.494451+0000","last_change":"2024-03-18T00:48:06.545416+0000","last_active":"2024-03-18T00:49:31.494451+0000","last_peered":"2024-03-18T00:49:31.494451+0000","last_clean":"2024-03-18T00:49:31.494451+0000","last_became_active":"2024-03-18T00:48:06.545087+0000","last_became_peered":"2024-03-18T00:48:06.545087+0000","last_unstale":"2024-03-18T00:49:31.494451+0000","last_undegraded":"2024-03-18T00:49:31.494451+0000","last_fullsized":"2024-03-18T00:49:31.494451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:02:25.309990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089159+0000","last_change":"2024-03-18T00:48:05.533088+0000","last_active":"2024-03-18T00:49:03.089159+0000","last_peered":"2024-03-18T00:49:03.089159+0000","last_clean":"2024-03-18T00:49:03.089159+0000","last_became_active":"2024-03-18T00:48:05.532308+0000","last_became_peered":"2024-03-18T00:48:05.532308+0000","last_unstale":"2024-03-18T00:49:03.089159+0000","last_undegraded":"2024-03-18T00:49:03.089159+0000","last_fullsized":"2024-03-18T00:49:03.089159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:18:35.071283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.513851+0000","last_change":"2024-03-18T00:49:03.092567+0000","last_active":"2024-03-18T00:49:31.513851+0000","last_peered":"2024-03-18T00:49:31.513851+0000","last_clean":"2024-03-18T00:49:31.513851+0000","last_became_active":"2024-03-18T00:49:03.092297+0000","last_became_peered":"2024-03-18T00:49:03.092297+0000","last_unstale":"2024-03-18T00:49:31.513851+0000","last_undegraded":"2024-03-18T00:49:31.513851+0000","last_fullsized":"2024-03-18T00:49:31.513851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:10:44.567584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'36","reported_seq":292,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.147485+0000","last_change":"2024-03-18T00:48:05.527557+0000","last_active":"2024-03-18T00:50:05.147485+0000","last_peered":"2024-03-18T00:50:05.147485+0000","last_clean":"2024-03-18T00:50:05.147485+0000","last_became_active":"2024-03-18T00:48:05.527167+0000","last_became_peered":"2024-03-18T00:48:05.527167+0000","last_unstale":"2024-03-18T00:50:05.147485+0000","last_undegraded":"2024-03-18T00:50:05.147485+0000","last_fullsized":"2024-03-18T00:50:05.147485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:02:04.105091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65917,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":237,"num_read_kb":184,"num_write":38,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.499066+0000","last_change":"2024-03-18T00:48:06.543153+0000","last_active":"2024-03-18T00:49:31.499066+0000","last_peered":"2024-03-18T00:49:31.499066+0000","last_clean":"2024-03-18T00:49:31.499066+0000","last_became_active":"2024-03-18T00:48:06.541353+0000","last_became_peered":"2024-03-18T00:48:06.541353+0000","last_unstale":"2024-03-18T00:49:31.499066+0000","last_undegraded":"2024-03-18T00:49:31.499066+0000","last_fullsized":"2024-03-18T00:49:31.499066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:41:35.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.086981+0000","last_change":"2024-03-18T00:48:05.530560+0000","last_active":"2024-03-18T00:49:03.086981+0000","last_peered":"2024-03-18T00:49:03.086981+0000","last_clean":"2024-03-18T00:49:03.086981+0000","last_became_active":"2024-03-18T00:48:05.528858+0000","last_became_peered":"2024-03-18T00:48:05.528858+0000","last_unstale":"2024-03-18T00:49:03.086981+0000","last_undegraded":"2024-03-18T00:49:03.086981+0000","last_fullsized":"2024-03-18T00:49:03.086981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:03:01.496919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497847+0000","last_change":"2024-03-18T00:49:03.090951+0000","last_active":"2024-03-18T00:49:31.497847+0000","last_peered":"2024-03-18T00:49:31.497847+0000","last_clean":"2024-03-18T00:49:31.497847+0000","last_became_active":"2024-03-18T00:49:03.090578+0000","last_became_peered":"2024-03-18T00:49:03.090578+0000","last_unstale":"2024-03-18T00:49:31.497847+0000","last_undegraded":"2024-03-18T00:49:31.497847+0000","last_fullsized":"2024-03-18T00:49:31.497847+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:04:12.703545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.088756+0000","last_change":"2024-03-18T00:48:05.527902+0000","last_active":"2024-03-18T00:49:03.088756+0000","last_peered":"2024-03-18T00:49:03.088756+0000","last_clean":"2024-03-18T00:49:03.088756+0000","last_became_active":"2024-03-18T00:48:05.527495+0000","last_became_peered":"2024-03-18T00:48:05.527495+0000","last_unstale":"2024-03-18T00:49:03.088756+0000","last_undegraded":"2024-03-18T00:49:03.088756+0000","last_fullsized":"2024-03-18T00:49:03.088756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:03:25.504652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504404+0000","last_change":"2024-03-18T00:48:06.550413+0000","last_active":"2024-03-18T00:49:31.504404+0000","last_peered":"2024-03-18T00:49:31.504404+0000","last_clean":"2024-03-18T00:49:31.504404+0000","last_became_active":"2024-03-18T00:48:06.547727+0000","last_became_peered":"2024-03-18T00:48:06.547727+0000","last_unstale":"2024-03-18T00:49:31.504404+0000","last_undegraded":"2024-03-18T00:49:31.504404+0000","last_fullsized":"2024-03-18T00:49:31.504404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:36:48.878417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.146430+0000","last_change":"2024-03-18T00:48:05.525962+0000","last_active":"2024-03-18T00:50:05.146430+0000","last_peered":"2024-03-18T00:50:05.146430+0000","last_clean":"2024-03-18T00:50:05.146430+0000","last_became_active":"2024-03-18T00:48:05.525607+0000","last_became_peered":"2024-03-18T00:48:05.525607+0000","last_unstale":"2024-03-18T00:50:05.146430+0000","last_undegraded":"2024-03-18T00:50:05.146430+0000","last_fullsized":"2024-03-18T00:50:05.146430+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:49:32.457447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.498511+0000","last_change":"2024-03-18T00:49:03.091221+0000","last_active":"2024-03-18T00:49:31.498511+0000","last_peered":"2024-03-18T00:49:31.498511+0000","last_clean":"2024-03-18T00:49:31.498511+0000","last_became_active":"2024-03-18T00:49:03.090559+0000","last_became_peered":"2024-03-18T00:49:03.090559+0000","last_unstale":"2024-03-18T00:49:31.498511+0000","last_undegraded":"2024-03-18T00:49:31.498511+0000","last_fullsized":"2024-03-18T00:49:31.498511+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:02:54.890187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":61,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.417640+0000","last_change":"2024-03-18T00:49:03.417640+0000","last_active":"2024-03-18T00:49:03.417640+0000","last_peered":"2024-03-18T00:49:03.417640+0000","last_clean":"2024-03-18T00:49:03.417640+0000","last_became_active":"2024-03-18T00:49:03.405217+0000","last_became_peered":"2024-03-18T00:49:03.405217+0000","last_unstale":"2024-03-18T00:49:03.417640+0000","last_undegraded":"2024-03-18T00:49:03.417640+0000","last_fullsized":"2024-03-18T00:49:03.417640+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:00:13.329099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507763+0000","last_change":"2024-03-18T00:48:06.544084+0000","last_active":"2024-03-18T00:49:31.507763+0000","last_peered":"2024-03-18T00:49:31.507763+0000","last_clean":"2024-03-18T00:49:31.507763+0000","last_became_active":"2024-03-18T00:48:06.543419+0000","last_became_peered":"2024-03-18T00:48:06.543419+0000","last_unstale":"2024-03-18T00:49:31.507763+0000","last_undegraded":"2024-03-18T00:49:31.507763+0000","last_fullsized":"2024-03-18T00:49:31.507763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:15:28.527473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087635+0000","last_change":"2024-03-18T00:48:05.528347+0000","last_active":"2024-03-18T00:49:03.087635+0000","last_peered":"2024-03-18T00:49:03.087635+0000","last_clean":"2024-03-18T00:49:03.087635+0000","last_became_active":"2024-03-18T00:48:05.527761+0000","last_became_peered":"2024-03-18T00:48:05.527761+0000","last_unstale":"2024-03-18T00:49:03.087635+0000","last_undegraded":"2024-03-18T00:49:03.087635+0000","last_fullsized":"2024-03-18T00:49:03.087635+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:36:58.562495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.503220+0000","last_change":"2024-03-18T00:48:06.541862+0000","last_active":"2024-03-18T00:49:31.503220+0000","last_peered":"2024-03-18T00:49:31.503220+0000","last_clean":"2024-03-18T00:49:31.503220+0000","last_became_active":"2024-03-18T00:48:06.541488+0000","last_became_peered":"2024-03-18T00:48:06.541488+0000","last_unstale":"2024-03-18T00:49:31.503220+0000","last_undegraded":"2024-03-18T00:49:31.503220+0000","last_fullsized":"2024-03-18T00:49:31.503220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:47:40.724244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.084618+0000","last_change":"2024-03-18T00:48:05.535498+0000","last_active":"2024-03-18T00:49:03.084618+0000","last_peered":"2024-03-18T00:49:03.084618+0000","last_clean":"2024-03-18T00:49:03.084618+0000","last_became_active":"2024-03-18T00:48:05.534907+0000","last_became_peered":"2024-03-18T00:48:05.534907+0000","last_unstale":"2024-03-18T00:49:03.084618+0000","last_undegraded":"2024-03-18T00:49:03.084618+0000","last_fullsized":"2024-03-18T00:49:03.084618+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:47:50.196929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504894+0000","last_change":"2024-03-18T00:48:06.534423+0000","last_active":"2024-03-18T00:49:31.504894+0000","last_peered":"2024-03-18T00:49:31.504894+0000","last_clean":"2024-03-18T00:49:31.504894+0000","last_became_active":"2024-03-18T00:48:06.534091+0000","last_became_peered":"2024-03-18T00:48:06.534091+0000","last_unstale":"2024-03-18T00:49:31.504894+0000","last_undegraded":"2024-03-18T00:49:31.504894+0000","last_fullsized":"2024-03-18T00:49:31.504894+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:39:08.755634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":136,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:01.157977+0000","last_change":"2024-03-18T00:48:05.532891+0000","last_active":"2024-03-18T00:50:01.157977+0000","last_peered":"2024-03-18T00:50:01.157977+0000","last_clean":"2024-03-18T00:50:01.157977+0000","last_became_active":"2024-03-18T00:48:05.532515+0000","last_became_peered":"2024-03-18T00:48:05.532515+0000","last_unstale":"2024-03-18T00:50:01.157977+0000","last_undegraded":"2024-03-18T00:50:01.157977+0000","last_fullsized":"2024-03-18T00:50:01.157977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:46:52.056837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506095+0000","last_change":"2024-03-18T00:48:06.542992+0000","last_active":"2024-03-18T00:49:31.506095+0000","last_peered":"2024-03-18T00:49:31.506095+0000","last_clean":"2024-03-18T00:49:31.506095+0000","last_became_active":"2024-03-18T00:48:06.542617+0000","last_became_peered":"2024-03-18T00:48:06.542617+0000","last_unstale":"2024-03-18T00:49:31.506095+0000","last_undegraded":"2024-03-18T00:49:31.506095+0000","last_fullsized":"2024-03-18T00:49:31.506095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:56:22.736118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506043+0000","last_change":"2024-03-18T00:48:06.536018+0000","last_active":"2024-03-18T00:49:31.506043+0000","last_peered":"2024-03-18T00:49:31.506043+0000","last_clean":"2024-03-18T00:49:31.506043+0000","last_became_active":"2024-03-18T00:48:06.535656+0000","last_became_peered":"2024-03-18T00:48:06.535656+0000","last_unstale":"2024-03-18T00:49:31.506043+0000","last_undegraded":"2024-03-18T00:49:31.506043+0000","last_fullsized":"2024-03-18T00:49:31.506043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:33:00.492887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:06.146970+0000","last_change":"2024-03-18T00:48:05.536635+0000","last_active":"2024-03-18T00:50:06.146970+0000","last_peered":"2024-03-18T00:50:06.146970+0000","last_clean":"2024-03-18T00:50:06.146970+0000","last_became_active":"2024-03-18T00:48:05.535941+0000","last_became_peered":"2024-03-18T00:48:05.535941+0000","last_unstale":"2024-03-18T00:50:06.146970+0000","last_undegraded":"2024-03-18T00:50:06.146970+0000","last_fullsized":"2024-03-18T00:50:06.146970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:20:52.167217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.515797+0000","last_change":"2024-03-18T00:48:06.546266+0000","last_active":"2024-03-18T00:49:31.515797+0000","last_peered":"2024-03-18T00:49:31.515797+0000","last_clean":"2024-03-18T00:49:31.515797+0000","last_became_active":"2024-03-18T00:48:06.543565+0000","last_became_peered":"2024-03-18T00:48:06.543565+0000","last_unstale":"2024-03-18T00:49:31.515797+0000","last_undegraded":"2024-03-18T00:49:31.515797+0000","last_fullsized":"2024-03-18T00:49:31.515797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:49:09.673212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.403989+0000","last_change":"2024-03-18T00:48:05.526184+0000","last_active":"2024-03-18T00:49:03.403989+0000","last_peered":"2024-03-18T00:49:03.403989+0000","last_clean":"2024-03-18T00:49:03.403989+0000","last_became_active":"2024-03-18T00:48:05.525011+0000","last_became_peered":"2024-03-18T00:48:05.525011+0000","last_unstale":"2024-03-18T00:49:03.403989+0000","last_undegraded":"2024-03-18T00:49:03.403989+0000","last_fullsized":"2024-03-18T00:49:03.403989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:12.690966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087462+0000","last_change":"2024-03-18T00:46:53.288768+0000","last_active":"2024-03-18T00:49:03.087462+0000","last_peered":"2024-03-18T00:49:03.087462+0000","last_clean":"2024-03-18T00:49:03.087462+0000","last_became_active":"2024-03-18T00:46:53.288394+0000","last_became_peered":"2024-03-18T00:46:53.288394+0000","last_unstale":"2024-03-18T00:49:03.087462+0000","last_undegraded":"2024-03-18T00:49:03.087462+0000","last_fullsized":"2024-03-18T00:49:03.087462+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:43:28.173150+0000","last_clean_scrub_stamp":"2024-03-18T00:43:28.173150+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T03:38:49.949940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500421+0000","last_change":"2024-03-18T00:48:06.534383+0000","last_active":"2024-03-18T00:49:31.500421+0000","last_peered":"2024-03-18T00:49:31.500421+0000","last_clean":"2024-03-18T00:49:31.500421+0000","last_became_active":"2024-03-18T00:48:06.534085+0000","last_became_peered":"2024-03-18T00:48:06.534085+0000","last_unstale":"2024-03-18T00:49:31.500421+0000","last_undegraded":"2024-03-18T00:49:31.500421+0000","last_fullsized":"2024-03-18T00:49:31.500421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:25:54.021830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.407661+0000","last_change":"2024-03-18T00:49:03.407661+0000","last_active":"2024-03-18T00:49:03.407661+0000","last_peered":"2024-03-18T00:49:03.407661+0000","last_clean":"2024-03-18T00:49:03.407661+0000","last_became_active":"2024-03-18T00:49:03.405134+0000","last_became_peered":"2024-03-18T00:49:03.405134+0000","last_unstale":"2024-03-18T00:49:03.407661+0000","last_undegraded":"2024-03-18T00:49:03.407661+0000","last_fullsized":"2024-03-18T00:49:03.407661+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:29:03.201556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.098300+0000","last_change":"2024-03-18T00:49:03.097885+0000","last_active":"2024-03-18T00:49:03.098300+0000","last_peered":"2024-03-18T00:49:03.098300+0000","last_clean":"2024-03-18T00:49:03.098300+0000","last_became_active":"2024-03-18T00:49:03.090574+0000","last_became_peered":"2024-03-18T00:49:03.090574+0000","last_unstale":"2024-03-18T00:49:03.098300+0000","last_undegraded":"2024-03-18T00:49:03.098300+0000","last_fullsized":"2024-03-18T00:49:03.098300+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:22:30.194349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.499871+0000","last_change":"2024-03-18T00:48:06.530108+0000","last_active":"2024-03-18T00:49:31.499871+0000","last_peered":"2024-03-18T00:49:31.499871+0000","last_clean":"2024-03-18T00:49:31.499871+0000","last_became_active":"2024-03-18T00:48:06.529768+0000","last_became_peered":"2024-03-18T00:48:06.529768+0000","last_unstale":"2024-03-18T00:49:31.499871+0000","last_undegraded":"2024-03-18T00:49:31.499871+0000","last_fullsized":"2024-03-18T00:49:31.499871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:23:09.150305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'14","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.529747+0000","last_change":"2024-03-18T00:48:05.535703+0000","last_active":"2024-03-18T00:49:31.529747+0000","last_peered":"2024-03-18T00:49:31.529747+0000","last_clean":"2024-03-18T00:49:31.529747+0000","last_became_active":"2024-03-18T00:48:05.535086+0000","last_became_peered":"2024-03-18T00:48:05.535086+0000","last_unstale":"2024-03-18T00:49:31.529747+0000","last_undegraded":"2024-03-18T00:49:31.529747+0000","last_fullsized":"2024-03-18T00:49:31.529747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:10:24.172353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504017+0000","last_change":"2024-03-18T00:48:06.541281+0000","last_active":"2024-03-18T00:49:31.504017+0000","last_peered":"2024-03-18T00:49:31.504017+0000","last_clean":"2024-03-18T00:49:31.504017+0000","last_became_active":"2024-03-18T00:48:06.540946+0000","last_became_peered":"2024-03-18T00:48:06.540946+0000","last_unstale":"2024-03-18T00:49:31.504017+0000","last_undegraded":"2024-03-18T00:49:31.504017+0000","last_fullsized":"2024-03-18T00:49:31.504017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:21:02.987661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.404075+0000","last_change":"2024-03-18T00:48:05.531685+0000","last_active":"2024-03-18T00:49:03.404075+0000","last_peered":"2024-03-18T00:49:03.404075+0000","last_clean":"2024-03-18T00:49:03.404075+0000","last_became_active":"2024-03-18T00:48:05.531239+0000","last_became_peered":"2024-03-18T00:48:05.531239+0000","last_unstale":"2024-03-18T00:49:03.404075+0000","last_undegraded":"2024-03-18T00:49:03.404075+0000","last_fullsized":"2024-03-18T00:49:03.404075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:26:35.034842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502223+0000","last_change":"2024-03-18T00:48:06.546783+0000","last_active":"2024-03-18T00:49:31.502223+0000","last_peered":"2024-03-18T00:49:31.502223+0000","last_clean":"2024-03-18T00:49:31.502223+0000","last_became_active":"2024-03-18T00:48:06.546411+0000","last_became_peered":"2024-03-18T00:48:06.546411+0000","last_unstale":"2024-03-18T00:49:31.502223+0000","last_undegraded":"2024-03-18T00:49:31.502223+0000","last_fullsized":"2024-03-18T00:49:31.502223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T09:39:34.432104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089245+0000","last_change":"2024-03-18T00:48:05.534701+0000","last_active":"2024-03-18T00:49:03.089245+0000","last_peered":"2024-03-18T00:49:03.089245+0000","last_clean":"2024-03-18T00:49:03.089245+0000","last_became_active":"2024-03-18T00:48:05.534300+0000","last_became_peered":"2024-03-18T00:48:05.534300+0000","last_unstale":"2024-03-18T00:49:03.089245+0000","last_undegraded":"2024-03-18T00:49:03.089245+0000","last_fullsized":"2024-03-18T00:49:03.089245+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:06:10.580001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.461798+0000","last_change":"2024-03-18T00:48:06.528286+0000","last_active":"2024-03-18T00:49:31.461798+0000","last_peered":"2024-03-18T00:49:31.461798+0000","last_clean":"2024-03-18T00:49:31.461798+0000","last_became_active":"2024-03-18T00:48:06.527947+0000","last_became_peered":"2024-03-18T00:48:06.527947+0000","last_unstale":"2024-03-18T00:49:31.461798+0000","last_undegraded":"2024-03-18T00:49:31.461798+0000","last_fullsized":"2024-03-18T00:49:31.461798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:08:34.953923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.325566+0000","last_change":"2024-03-18T00:48:05.526530+0000","last_active":"2024-03-18T00:49:03.325566+0000","last_peered":"2024-03-18T00:49:03.325566+0000","last_clean":"2024-03-18T00:49:03.325566+0000","last_became_active":"2024-03-18T00:48:05.526210+0000","last_became_peered":"2024-03-18T00:48:05.526210+0000","last_unstale":"2024-03-18T00:49:03.325566+0000","last_undegraded":"2024-03-18T00:49:03.325566+0000","last_fullsized":"2024-03-18T00:49:03.325566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T05:11:11.094469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501176+0000","last_change":"2024-03-18T00:48:06.535316+0000","last_active":"2024-03-18T00:49:31.501176+0000","last_peered":"2024-03-18T00:49:31.501176+0000","last_clean":"2024-03-18T00:49:31.501176+0000","last_became_active":"2024-03-18T00:48:06.534945+0000","last_became_peered":"2024-03-18T00:48:06.534945+0000","last_unstale":"2024-03-18T00:49:31.501176+0000","last_undegraded":"2024-03-18T00:49:31.501176+0000","last_fullsized":"2024-03-18T00:49:31.501176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:04:32.570585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"88'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087310+0000","last_change":"2024-03-18T00:48:05.529300+0000","last_active":"2024-03-18T00:49:03.087310+0000","last_peered":"2024-03-18T00:49:03.087310+0000","last_clean":"2024-03-18T00:49:03.087310+0000","last_became_active":"2024-03-18T00:48:05.528935+0000","last_became_peered":"2024-03-18T00:48:05.528935+0000","last_unstale":"2024-03-18T00:49:03.087310+0000","last_undegraded":"2024-03-18T00:49:03.087310+0000","last_fullsized":"2024-03-18T00:49:03.087310+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:29:45.645566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.498853+0000","last_change":"2024-03-18T00:48:06.536426+0000","last_active":"2024-03-18T00:49:31.498853+0000","last_peered":"2024-03-18T00:49:31.498853+0000","last_clean":"2024-03-18T00:49:31.498853+0000","last_became_active":"2024-03-18T00:48:06.536005+0000","last_became_peered":"2024-03-18T00:48:06.536005+0000","last_unstale":"2024-03-18T00:49:31.498853+0000","last_undegraded":"2024-03-18T00:49:31.498853+0000","last_fullsized":"2024-03-18T00:49:31.498853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:40:52.028794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087438+0000","last_change":"2024-03-18T00:48:05.536495+0000","last_active":"2024-03-18T00:49:03.087438+0000","last_peered":"2024-03-18T00:49:03.087438+0000","last_clean":"2024-03-18T00:49:03.087438+0000","last_became_active":"2024-03-18T00:48:05.536072+0000","last_became_peered":"2024-03-18T00:48:05.536072+0000","last_unstale":"2024-03-18T00:49:03.087438+0000","last_undegraded":"2024-03-18T00:49:03.087438+0000","last_fullsized":"2024-03-18T00:49:03.087438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:03:25.528261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.505766+0000","last_change":"2024-03-18T00:48:06.552900+0000","last_active":"2024-03-18T00:49:31.505766+0000","last_peered":"2024-03-18T00:49:31.505766+0000","last_clean":"2024-03-18T00:49:31.505766+0000","last_became_active":"2024-03-18T00:48:06.552568+0000","last_became_peered":"2024-03-18T00:48:06.552568+0000","last_unstale":"2024-03-18T00:49:31.505766+0000","last_undegraded":"2024-03-18T00:49:31.505766+0000","last_fullsized":"2024-03-18T00:49:31.505766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T00:53:25.254177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:50:05.155478+0000","last_change":"2024-03-18T00:48:05.537327+0000","last_active":"2024-03-18T00:50:05.155478+0000","last_peered":"2024-03-18T00:50:05.155478+0000","last_clean":"2024-03-18T00:50:05.155478+0000","last_became_active":"2024-03-18T00:48:05.536825+0000","last_became_peered":"2024-03-18T00:48:05.536825+0000","last_unstale":"2024-03-18T00:50:05.155478+0000","last_undegraded":"2024-03-18T00:50:05.155478+0000","last_fullsized":"2024-03-18T00:50:05.155478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:50:03.141142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.505260+0000","last_change":"2024-03-18T00:48:06.542593+0000","last_active":"2024-03-18T00:49:31.505260+0000","last_peered":"2024-03-18T00:49:31.505260+0000","last_clean":"2024-03-18T00:49:31.505260+0000","last_became_active":"2024-03-18T00:48:06.541624+0000","last_became_peered":"2024-03-18T00:48:06.541624+0000","last_unstale":"2024-03-18T00:49:31.505260+0000","last_undegraded":"2024-03-18T00:49:31.505260+0000","last_fullsized":"2024-03-18T00:49:31.505260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:10:56.905221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.089025+0000","last_change":"2024-03-18T00:48:05.529140+0000","last_active":"2024-03-18T00:49:03.089025+0000","last_peered":"2024-03-18T00:49:03.089025+0000","last_clean":"2024-03-18T00:49:03.089025+0000","last_became_active":"2024-03-18T00:48:05.528682+0000","last_became_peered":"2024-03-18T00:48:05.528682+0000","last_unstale":"2024-03-18T00:49:03.089025+0000","last_undegraded":"2024-03-18T00:49:03.089025+0000","last_fullsized":"2024-03-18T00:49:03.089025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:57:52.663150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.508245+0000","last_change":"2024-03-18T00:48:06.546027+0000","last_active":"2024-03-18T00:49:31.508245+0000","last_peered":"2024-03-18T00:49:31.508245+0000","last_clean":"2024-03-18T00:49:31.508245+0000","last_became_active":"2024-03-18T00:48:06.545670+0000","last_became_peered":"2024-03-18T00:48:06.545670+0000","last_unstale":"2024-03-18T00:49:31.508245+0000","last_undegraded":"2024-03-18T00:49:31.508245+0000","last_fullsized":"2024-03-18T00:49:31.508245+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:47:05.888656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:03.087890+0000","last_change":"2024-03-18T00:48:05.526929+0000","last_active":"2024-03-18T00:49:03.087890+0000","last_peered":"2024-03-18T00:49:03.087890+0000","last_clean":"2024-03-18T00:49:03.087890+0000","last_became_active":"2024-03-18T00:48:05.526530+0000","last_became_peered":"2024-03-18T00:48:05.526530+0000","last_unstale":"2024-03-18T00:49:03.087890+0000","last_undegraded":"2024-03-18T00:49:03.087890+0000","last_fullsized":"2024-03-18T00:49:03.087890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:04.501534+0000","last_clean_scrub_stamp":"2024-03-18T00:48:04.501534+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:51:25.110895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.504777+0000","last_change":"2024-03-18T00:48:06.541223+0000","last_active":"2024-03-18T00:49:31.504777+0000","last_peered":"2024-03-18T00:49:31.504777+0000","last_clean":"2024-03-18T00:49:31.504777+0000","last_became_active":"2024-03-18T00:48:06.540898+0000","last_became_peered":"2024-03-18T00:48:06.540898+0000","last_unstale":"2024-03-18T00:49:31.504777+0000","last_undegraded":"2024-03-18T00:49:31.504777+0000","last_fullsized":"2024-03-18T00:49:31.504777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:20:22.749609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506812+0000","last_change":"2024-03-18T00:48:06.532906+0000","last_active":"2024-03-18T00:49:31.506812+0000","last_peered":"2024-03-18T00:49:31.506812+0000","last_clean":"2024-03-18T00:49:31.506812+0000","last_became_active":"2024-03-18T00:48:06.532583+0000","last_became_peered":"2024-03-18T00:48:06.532583+0000","last_unstale":"2024-03-18T00:49:31.506812+0000","last_undegraded":"2024-03-18T00:49:31.506812+0000","last_fullsized":"2024-03-18T00:49:31.506812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T08:45:06.699615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.506031+0000","last_change":"2024-03-18T00:48:06.530020+0000","last_active":"2024-03-18T00:49:31.506031+0000","last_peered":"2024-03-18T00:49:31.506031+0000","last_clean":"2024-03-18T00:49:31.506031+0000","last_became_active":"2024-03-18T00:48:06.529709+0000","last_became_peered":"2024-03-18T00:48:06.529709+0000","last_unstale":"2024-03-18T00:49:31.506031+0000","last_undegraded":"2024-03-18T00:49:31.506031+0000","last_fullsized":"2024-03-18T00:49:31.506031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:48:07.305537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500282+0000","last_change":"2024-03-18T00:48:06.537658+0000","last_active":"2024-03-18T00:49:31.500282+0000","last_peered":"2024-03-18T00:49:31.500282+0000","last_clean":"2024-03-18T00:49:31.500282+0000","last_became_active":"2024-03-18T00:48:06.537339+0000","last_became_peered":"2024-03-18T00:48:06.537339+0000","last_unstale":"2024-03-18T00:49:31.500282+0000","last_undegraded":"2024-03-18T00:49:31.500282+0000","last_fullsized":"2024-03-18T00:49:31.500282+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:18:25.753168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.497779+0000","last_change":"2024-03-18T00:48:06.544514+0000","last_active":"2024-03-18T00:49:31.497779+0000","last_peered":"2024-03-18T00:49:31.497779+0000","last_clean":"2024-03-18T00:49:31.497779+0000","last_became_active":"2024-03-18T00:48:06.544165+0000","last_became_peered":"2024-03-18T00:48:06.544165+0000","last_unstale":"2024-03-18T00:49:31.497779+0000","last_undegraded":"2024-03-18T00:49:31.497779+0000","last_fullsized":"2024-03-18T00:49:31.497779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T10:48:02.498940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501247+0000","last_change":"2024-03-18T00:49:03.090610+0000","last_active":"2024-03-18T00:49:31.501247+0000","last_peered":"2024-03-18T00:49:31.501247+0000","last_clean":"2024-03-18T00:49:31.501247+0000","last_became_active":"2024-03-18T00:49:03.090014+0000","last_became_peered":"2024-03-18T00:49:03.090014+0000","last_unstale":"2024-03-18T00:49:31.501247+0000","last_undegraded":"2024-03-18T00:49:31.501247+0000","last_fullsized":"2024-03-18T00:49:31.501247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T07:55:12.210223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.503021+0000","last_change":"2024-03-18T00:48:06.533837+0000","last_active":"2024-03-18T00:49:31.503021+0000","last_peered":"2024-03-18T00:49:31.503021+0000","last_clean":"2024-03-18T00:49:31.503021+0000","last_became_active":"2024-03-18T00:48:06.533418+0000","last_became_peered":"2024-03-18T00:48:06.533418+0000","last_unstale":"2024-03-18T00:49:31.503021+0000","last_undegraded":"2024-03-18T00:49:31.503021+0000","last_fullsized":"2024-03-18T00:49:31.503021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:00:10.873312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.499996+0000","last_change":"2024-03-18T00:48:06.527394+0000","last_active":"2024-03-18T00:49:31.499996+0000","last_peered":"2024-03-18T00:49:31.499996+0000","last_clean":"2024-03-18T00:49:31.499996+0000","last_became_active":"2024-03-18T00:48:06.526879+0000","last_became_peered":"2024-03-18T00:48:06.526879+0000","last_unstale":"2024-03-18T00:49:31.499996+0000","last_undegraded":"2024-03-18T00:49:31.499996+0000","last_fullsized":"2024-03-18T00:49:31.499996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T06:23:07.397259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.500631+0000","last_change":"2024-03-18T00:48:06.542645+0000","last_active":"2024-03-18T00:49:31.500631+0000","last_peered":"2024-03-18T00:49:31.500631+0000","last_clean":"2024-03-18T00:49:31.500631+0000","last_became_active":"2024-03-18T00:48:06.542073+0000","last_became_peered":"2024-03-18T00:48:06.542073+0000","last_unstale":"2024-03-18T00:49:31.500631+0000","last_undegraded":"2024-03-18T00:49:31.500631+0000","last_fullsized":"2024-03-18T00:49:31.500631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T12:08:44.479318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.507457+0000","last_change":"2024-03-18T00:48:06.541698+0000","last_active":"2024-03-18T00:49:31.507457+0000","last_peered":"2024-03-18T00:49:31.507457+0000","last_clean":"2024-03-18T00:49:31.507457+0000","last_became_active":"2024-03-18T00:48:06.541378+0000","last_became_peered":"2024-03-18T00:48:06.541378+0000","last_unstale":"2024-03-18T00:49:31.507457+0000","last_undegraded":"2024-03-18T00:49:31.507457+0000","last_fullsized":"2024-03-18T00:49:31.507457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T02:42:23.685603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.502862+0000","last_change":"2024-03-18T00:48:06.533184+0000","last_active":"2024-03-18T00:49:31.502862+0000","last_peered":"2024-03-18T00:49:31.502862+0000","last_clean":"2024-03-18T00:49:31.502862+0000","last_became_active":"2024-03-18T00:48:06.532034+0000","last_became_peered":"2024-03-18T00:48:06.532034+0000","last_unstale":"2024-03-18T00:49:31.502862+0000","last_undegraded":"2024-03-18T00:49:31.502862+0000","last_fullsized":"2024-03-18T00:49:31.502862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T11:33:45.265799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.515864+0000","last_change":"2024-03-18T00:48:06.545941+0000","last_active":"2024-03-18T00:49:31.515864+0000","last_peered":"2024-03-18T00:49:31.515864+0000","last_clean":"2024-03-18T00:49:31.515864+0000","last_became_active":"2024-03-18T00:48:06.545402+0000","last_became_peered":"2024-03-18T00:48:06.545402+0000","last_unstale":"2024-03-18T00:49:31.515864+0000","last_undegraded":"2024-03-18T00:49:31.515864+0000","last_fullsized":"2024-03-18T00:49:31.515864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:11:45.512849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.505008+0000","last_change":"2024-03-18T00:48:06.541513+0000","last_active":"2024-03-18T00:49:31.505008+0000","last_peered":"2024-03-18T00:49:31.505008+0000","last_clean":"2024-03-18T00:49:31.505008+0000","last_became_active":"2024-03-18T00:48:06.541128+0000","last_became_peered":"2024-03-18T00:48:06.541128+0000","last_unstale":"2024-03-18T00:49:31.505008+0000","last_undegraded":"2024-03-18T00:49:31.505008+0000","last_fullsized":"2024-03-18T00:49:31.505008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T01:42:10.635836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-18T00:49:31.501522+0000","last_change":"2024-03-18T00:48:06.542316+0000","last_active":"2024-03-18T00:49:31.501522+0000","last_peered":"2024-03-18T00:49:31.501522+0000","last_clean":"2024-03-18T00:49:31.501522+0000","last_became_active":"2024-03-18T00:48:06.541914+0000","last_became_peered":"2024-03-18T00:48:06.541914+0000","last_unstale":"2024-03-18T00:49:31.501522+0000","last_undegraded":"2024-03-18T00:49:31.501522+0000","last_fullsized":"2024-03-18T00:49:31.501522+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T00:48:05.506987+0000","last_clean_scrub_stamp":"2024-03-18T00:48:05.506987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T04:20:46.293240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":249372,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73179,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1128,"num_read_kb":670,"num_write":252,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1237,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":214607,"data_compressed":5589,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":218,"ondisk_log_size":218,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":461614,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612651,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1114112,"data_stored":1116986,"data_compressed":43768,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776175,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":459970,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037813,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":460236,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":1053007,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462976,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":459687,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":521895,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":461026,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":527465,"data_compressed":7676,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":459512,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888137,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":459660,"data_compressed":5813,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724612,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1051531,"data_compressed":41905,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36092,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63047,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":68617,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":67653,"data_compressed":1863,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T00:50:08.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:50:08.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.489+0000 7fc0fe087640 1 -- 172.21.15.49:0/3597923478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8100780 msgr2=0x7fc0f8100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.489+0000 7fc0fe087640 1 --2- 172.21.15.49:0/3597923478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8100780 0x7fc0f8100ba0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009900 tx=0x7fc0e802bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.489+0000 7fc0fe087640 1 -- 172.21.15.49:0/3597923478 shutdown_connections 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.489+0000 7fc0fe087640 1 --2- 172.21.15.49:0/3597923478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0f8102330 0x7fc0f8108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.489+0000 7fc0fe087640 1 --2- 172.21.15.49:0/3597923478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f8101970 0x7fc0f8101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.489+0000 7fc0fe087640 1 --2- 172.21.15.49:0/3597923478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8100780 0x7fc0f8100ba0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.489+0000 7fc0fe087640 1 -- 172.21.15.49:0/3597923478 >> 172.21.15.49:0/3597923478 conn(0x7fc0f80fbd10 msgr2=0x7fc0f80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.489+0000 7fc0fe087640 1 -- 172.21.15.49:0/3597923478 shutdown_connections 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.489+0000 7fc0fe087640 1 -- 172.21.15.49:0/3597923478 wait complete. 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0fe087640 1 Processor -- start 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0fe087640 1 -- start start 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0fe087640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f8100780 0x7fc0f81046c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0fe087640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8101970 0x7fc0f8102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0fe087640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0f8102330 0x7fc0f8103290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0fe087640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f810c380 con 0x7fc0f8101970 2024-03-18T00:50:08.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0fe087640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f810c200 con 0x7fc0f8102330 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0fe087640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f810c500 con 0x7fc0f8100780 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8101970 0x7fc0f8102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8101970 0x7fc0f8102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58224/0 (socket says 172.21.15.49:58224) 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f8100780 0x7fc0f81046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f6ffd640 1 -- 172.21.15.49:0/2005758306 learned_addr learned my addr 172.21.15.49:0/2005758306 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f7fff640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0f8102330 0x7fc0f8103290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f6ffd640 1 -- 172.21.15.49:0/2005758306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f8100780 msgr2=0x7fc0f81046c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f6ffd640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f8100780 0x7fc0f81046c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f6ffd640 1 -- 172.21.15.49:0/2005758306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0f8102330 msgr2=0x7fc0f8103290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f6ffd640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0f8102330 0x7fc0f8103290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.490+0000 7fc0f6ffd640 1 -- 172.21.15.49:0/2005758306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f8103b50 con 0x7fc0f8101970 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.491+0000 7fc0f6ffd640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8101970 0x7fc0f8102d50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00e9f0 tx=0x7fc0ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.491+0000 7fc0f4ff9640 1 -- 172.21.15.49:0/2005758306 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec00cd80 con 0x7fc0f8101970 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.491+0000 7fc0f4ff9640 1 -- 172.21.15.49:0/2005758306 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0ec005500 con 0x7fc0f8101970 2024-03-18T00:50:08.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.491+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f81a1cc0 con 0x7fc0f8101970 2024-03-18T00:50:08.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.491+0000 7fc0f4ff9640 1 -- 172.21.15.49:0/2005758306 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec010690 con 0x7fc0f8101970 2024-03-18T00:50:08.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.491+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f81a2190 con 0x7fc0f8101970 2024-03-18T00:50:08.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.492+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f81a2420 con 0x7fc0f8101970 2024-03-18T00:50:08.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.493+0000 7fc0f4ff9640 1 -- 172.21.15.49:0/2005758306 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc0ec0046c0 con 0x7fc0f8101970 2024-03-18T00:50:08.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.493+0000 7fc0f4ff9640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0e00a2880 0x7fc0e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.494+0000 7fc0f77fe640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0e00a2880 0x7fc0e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.494+0000 7fc0f4ff9640 1 -- 172.21.15.49:0/2005758306 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc0ec0cedf0 con 0x7fc0f8101970 2024-03-18T00:50:08.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.494+0000 7fc0f77fe640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0e00a2880 0x7fc0e00a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8000c00 tx=0x7fc0e8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:08.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.498+0000 7fc0f4ff9640 1 -- 172.21.15.49:0/2005758306 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc0ec09be70 con 0x7fc0f8101970 2024-03-18T00:50:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:08 smithi049 ceph-mon[32130]: pgmap v351: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T00:50:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:08 smithi049 ceph-mon[32130]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:50:08.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:08 smithi049 ceph-mon[32130]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:50:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:08 smithi107 ceph-mon[36270]: pgmap v351: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T00:50:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:08 smithi107 ceph-mon[36270]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:50:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:08 smithi107 ceph-mon[36270]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:50:08.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.684+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0f81a27b0 con 0x7fc0f8101970 2024-03-18T00:50:08.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.686+0000 7fc0f4ff9640 1 -- 172.21.15.49:0/2005758306 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7fc0ec09b770 con 0x7fc0f8101970 2024-03-18T00:50:08.686 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:08.686 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0e00a2880 msgr2=0x7fc0e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0e00a2880 0x7fc0e00a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8000c00 tx=0x7fc0e8005fd0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8101970 msgr2=0x7fc0f8102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8101970 0x7fc0f8102d50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00e9f0 tx=0x7fc0ec00edf0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 shutdown_connections 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0e00a2880 0x7fc0e00a4d40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0f8102330 0x7fc0f8103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0f8101970 0x7fc0f8102d50 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 --2- 172.21.15.49:0/2005758306 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0f8100780 0x7fc0f81046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 >> 172.21.15.49:0/2005758306 conn(0x7fc0f80fbd10 msgr2=0x7fc0f80fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:08.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 shutdown_connections 2024-03-18T00:50:08.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.688+0000 7fc0fe087640 1 -- 172.21.15.49:0/2005758306 wait complete. 2024-03-18T00:50:08.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:08 smithi106 ceph-mon[35899]: pgmap v351: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T00:50:08.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:08 smithi106 ceph-mon[35899]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:50:08.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:08 smithi106 ceph-mon[35899]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:50:08.700 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T00:48:04.316609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T00:48:05.271815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T00:50:08.700 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T00:50:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.812+0000 7f4321cdc640 1 -- 172.21.15.49:0/4217647557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 msgr2=0x7f431c106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.812+0000 7f4321cdc640 1 --2- 172.21.15.49:0/4217647557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 0x7f431c106ff0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f4310009900 tx=0x7f431002bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.812+0000 7f4321cdc640 1 -- 172.21.15.49:0/4217647557 shutdown_connections 2024-03-18T00:50:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.812+0000 7f4321cdc640 1 --2- 172.21.15.49:0/4217647557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f431c107530 0x7f431c109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.812+0000 7f4321cdc640 1 --2- 172.21.15.49:0/4217647557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 0x7f431c106ff0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.812+0000 7f4321cdc640 1 --2- 172.21.15.49:0/4217647557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f431c1022d0 0x7f431c1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.812+0000 7f4321cdc640 1 -- 172.21.15.49:0/4217647557 >> 172.21.15.49:0/4217647557 conn(0x7f431c0fbc80 msgr2=0x7f431c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.812+0000 7f4321cdc640 1 -- 172.21.15.49:0/4217647557 shutdown_connections 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.812+0000 7f4321cdc640 1 -- 172.21.15.49:0/4217647557 wait complete. 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f4321cdc640 1 Processor -- start 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f4321cdc640 1 -- start start 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f4321cdc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f431c1022d0 0x7f431c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f4321cdc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 0x7f431c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f4321cdc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f431c107530 0x7f431c1a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f4321cdc640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f431c10d440 con 0x7f431c104c00 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f4321cdc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f431c10d2c0 con 0x7f431c107530 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f4321cdc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f431c10d5c0 con 0x7f431c1022d0 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f431affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 0x7f431c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f431affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 0x7f431c19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58246/0 (socket says 172.21.15.49:58246) 2024-03-18T00:50:08.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f431affd640 1 -- 172.21.15.49:0/2554772140 learned_addr learned my addr 172.21.15.49:0/2554772140 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f431b7fe640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f431c1022d0 0x7f431c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.813+0000 7f431bfff640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f431c107530 0x7f431c1a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f431affd640 1 -- 172.21.15.49:0/2554772140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f431c1022d0 msgr2=0x7f431c199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f431affd640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f431c1022d0 0x7f431c199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f431affd640 1 -- 172.21.15.49:0/2554772140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f431c107530 msgr2=0x7f431c1a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f431affd640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f431c107530 0x7f431c1a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f431affd640 1 -- 172.21.15.49:0/2554772140 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431c1a19c0 con 0x7f431c104c00 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f431b7fe640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f431c1022d0 0x7f431c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f431bfff640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f431c107530 0x7f431c1a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f431affd640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 0x7f431c19a2f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4310000c00 tx=0x7f431002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:08.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f4318ff9640 1 -- 172.21.15.49:0/2554772140 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310041070 con 0x7f431c104c00 2024-03-18T00:50:08.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f4318ff9640 1 -- 172.21.15.49:0/2554772140 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4310004450 con 0x7f431c104c00 2024-03-18T00:50:08.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431c1a1c50 con 0x7f431c104c00 2024-03-18T00:50:08.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f4318ff9640 1 -- 172.21.15.49:0/2554772140 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310034440 con 0x7f431c104c00 2024-03-18T00:50:08.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.814+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f431c1a2140 con 0x7f431c104c00 2024-03-18T00:50:08.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.815+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f431c109020 con 0x7f431c104c00 2024-03-18T00:50:08.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.816+0000 7f4318ff9640 1 -- 172.21.15.49:0/2554772140 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4310044070 con 0x7f431c104c00 2024-03-18T00:50:08.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.817+0000 7f4318ff9640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43040a2800 0x7f43040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:08.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.817+0000 7f431b7fe640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43040a2800 0x7f43040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:08.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.817+0000 7f4318ff9640 1 -- 172.21.15.49:0/2554772140 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f43100f05a0 con 0x7f431c104c00 2024-03-18T00:50:08.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.817+0000 7f431b7fe640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43040a2800 0x7f43040a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f431c19b280 tx=0x7f430c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:08.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:08.822+0000 7f4318ff9640 1 -- 172.21.15.49:0/2554772140 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f43100bcfd0 con 0x7f431c104c00 2024-03-18T00:50:09.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.005+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f431c0623e0 con 0x7f431c104c00 2024-03-18T00:50:09.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.005+0000 7f4318ff9640 1 -- 172.21.15.49:0/2554772140 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f43100c1070 con 0x7f431c104c00 2024-03-18T00:50:09.006 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T00:50:09.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43040a2800 msgr2=0x7f43040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43040a2800 0x7f43040a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f431c19b280 tx=0x7f430c005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 msgr2=0x7f431c19a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 0x7f431c19a2f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f4310000c00 tx=0x7f431002c640 comp rx=0 tx=0).stop 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 shutdown_connections 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43040a2800 0x7f43040a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f431c107530 0x7f431c1a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f431c104c00 0x7f431c19a2f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 --2- 172.21.15.49:0/2554772140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f431c1022d0 0x7f431c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.007+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 >> 172.21.15.49:0/2554772140 conn(0x7f431c0fbc80 msgr2=0x7f431c101b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.008+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 shutdown_connections 2024-03-18T00:50:09.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.008+0000 7f4321cdc640 1 -- 172.21.15.49:0/2554772140 wait complete. 2024-03-18T00:50:09.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T00:50:09.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.134+0000 7fda37ab5640 1 -- 172.21.15.49:0/1217437924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda301018d0 msgr2=0x7fda30101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.134+0000 7fda37ab5640 1 --2- 172.21.15.49:0/1217437924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda301018d0 0x7fda30101d50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fda24009900 tx=0x7fda2402bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.134+0000 7fda37ab5640 1 -- 172.21.15.49:0/1217437924 shutdown_connections 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.134+0000 7fda37ab5640 1 --2- 172.21.15.49:0/1217437924 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda30102290 0x7fda30108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.134+0000 7fda37ab5640 1 --2- 172.21.15.49:0/1217437924 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda301018d0 0x7fda30101d50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.134+0000 7fda37ab5640 1 --2- 172.21.15.49:0/1217437924 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda301006e0 0x7fda30100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.134+0000 7fda37ab5640 1 -- 172.21.15.49:0/1217437924 >> 172.21.15.49:0/1217437924 conn(0x7fda300fbc70 msgr2=0x7fda300fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 -- 172.21.15.49:0/1217437924 shutdown_connections 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 -- 172.21.15.49:0/1217437924 wait complete. 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 Processor -- start 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 -- start start 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda301006e0 0x7fda3010b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda301018d0 0x7fda301096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda30102290 0x7fda30109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fda3010d4e0 con 0x7fda30102290 2024-03-18T00:50:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fda3010d360 con 0x7fda301006e0 2024-03-18T00:50:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda37ab5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fda3010d660 con 0x7fda301018d0 2024-03-18T00:50:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda36052640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda30102290 0x7fda30109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda36052640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda30102290 0x7fda30109be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58258/0 (socket says 172.21.15.49:58258) 2024-03-18T00:50:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda36052640 1 -- 172.21.15.49:0/559245882 learned_addr learned my addr 172.21.15.49:0/559245882 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:09.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda35851640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda301006e0 0x7fda3010b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.135+0000 7fda35050640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda301018d0 0x7fda301096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda36052640 1 -- 172.21.15.49:0/559245882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda301018d0 msgr2=0x7fda301096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda36052640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda301018d0 0x7fda301096a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda36052640 1 -- 172.21.15.49:0/559245882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda301006e0 msgr2=0x7fda3010b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda36052640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda301006e0 0x7fda3010b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda36052640 1 -- 172.21.15.49:0/559245882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3010a470 con 0x7fda30102290 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda35851640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda301006e0 0x7fda3010b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda36052640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda30102290 0x7fda30109be0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fda2800e970 tx=0x7fda2800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda2effd640 1 -- 172.21.15.49:0/559245882 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2800ccc0 con 0x7fda30102290 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda301a1c40 con 0x7fda30102290 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda2effd640 1 -- 172.21.15.49:0/559245882 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda2800ce60 con 0x7fda30102290 2024-03-18T00:50:09.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda2effd640 1 -- 172.21.15.49:0/559245882 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda28010490 con 0x7fda30102290 2024-03-18T00:50:09.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.136+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda301a2110 con 0x7fda30102290 2024-03-18T00:50:09.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.137+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda30068a20 con 0x7fda30102290 2024-03-18T00:50:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.138+0000 7fda2effd640 1 -- 172.21.15.49:0/559245882 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fda28005000 con 0x7fda30102290 2024-03-18T00:50:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.139+0000 7fda2effd640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda180a2800 0x7fda180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.139+0000 7fda35851640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda180a2800 0x7fda180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.139+0000 7fda2effd640 1 -- 172.21.15.49:0/559245882 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fda280cf120 con 0x7fda30102290 2024-03-18T00:50:09.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.139+0000 7fda35851640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda180a2800 0x7fda180a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fda200099a0 tx=0x7fda20009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:09.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.142+0000 7fda2effd640 1 -- 172.21.15.49:0/559245882 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fda28053740 con 0x7fda30102290 2024-03-18T00:50:09.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.326+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fda3002d1e0 con 0x7fda30102290 2024-03-18T00:50:09.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.326+0000 7fda2effd640 1 -- 172.21.15.49:0/559245882 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7fda2809c1a0 con 0x7fda30102290 2024-03-18T00:50:09.326 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 32 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.328+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda180a2800 msgr2=0x7fda180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.328+0000 7fda37ab5640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda180a2800 0x7fda180a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fda200099a0 tx=0x7fda20009500 comp rx=0 tx=0).stop 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda30102290 msgr2=0x7fda30109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda30102290 0x7fda30109be0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fda2800e970 tx=0x7fda2800ed70 comp rx=0 tx=0).stop 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 shutdown_connections 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda180a2800 0x7fda180a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fda30102290 0x7fda30109be0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fda301018d0 0x7fda301096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 --2- 172.21.15.49:0/559245882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fda301006e0 0x7fda3010b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 >> 172.21.15.49:0/559245882 conn(0x7fda300fbc70 msgr2=0x7fda30104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 shutdown_connections 2024-03-18T00:50:09.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.329+0000 7fda37ab5640 1 -- 172.21.15.49:0/559245882 wait complete. 2024-03-18T00:50:09.340 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 -- 172.21.15.49:0/608703780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40041006e0 msgr2=0x7f4004100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 --2- 172.21.15.49:0/608703780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40041006e0 0x7f4004100b00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009900 tx=0x7f3ffc02bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 -- 172.21.15.49:0/608703780 shutdown_connections 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 --2- 172.21.15.49:0/608703780 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40041018d0 0x7f4004101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 --2- 172.21.15.49:0/608703780 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4004102290 0x7f4004108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 --2- 172.21.15.49:0/608703780 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40041006e0 0x7f4004100b00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 -- 172.21.15.49:0/608703780 >> 172.21.15.49:0/608703780 conn(0x7f40040fbc70 msgr2=0x7f40040fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 -- 172.21.15.49:0/608703780 shutdown_connections 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 -- 172.21.15.49:0/608703780 wait complete. 2024-03-18T00:50:09.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 Processor -- start 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.452+0000 7f40093e6640 1 -- start start 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40093e6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40041006e0 0x7f400410b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40093e6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40041018d0 0x7f40041096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40093e6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4004102290 0x7f4004109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40093e6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f400410d480 con 0x7f40041006e0 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40093e6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f400410d300 con 0x7f40041018d0 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40093e6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f400410d600 con 0x7f4004102290 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f4002ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40041006e0 0x7f400410b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40027fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40041018d0 0x7f40041096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40027fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40041018d0 0x7f40041096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:42040/0 (socket says 172.21.15.49:42040) 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40027fc640 1 -- 172.21.15.49:0/4263539815 learned_addr learned my addr 172.21.15.49:0/4263539815 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f4002ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40041006e0 0x7f400410b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58264/0 (socket says 172.21.15.49:58264) 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40037fe640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4004102290 0x7f4004109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40027fc640 1 -- 172.21.15.49:0/4263539815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4004102290 msgr2=0x7f4004109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40027fc640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4004102290 0x7f4004109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40027fc640 1 -- 172.21.15.49:0/4263539815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40041006e0 msgr2=0x7f400410b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40027fc640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40041006e0 0x7f400410b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40027fc640 1 -- 172.21.15.49:0/4263539815 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400410a470 con 0x7f40041018d0 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f40027fc640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40041018d0 0x7f40041096a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00e9f0 tx=0x7f3fec00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f3ffbfff640 1 -- 172.21.15.49:0/4263539815 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec00cd80 con 0x7f40041018d0 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f3ffbfff640 1 -- 172.21.15.49:0/4263539815 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fec005500 con 0x7f40041018d0 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.453+0000 7f3ffbfff640 1 -- 172.21.15.49:0/4263539815 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec010690 con 0x7f40041018d0 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.454+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40041a1ca0 con 0x7f40041018d0 2024-03-18T00:50:09.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.454+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40041a21a0 con 0x7f40041018d0 2024-03-18T00:50:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.454+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4004107b20 con 0x7f40041018d0 2024-03-18T00:50:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.456+0000 7f3ffbfff640 1 -- 172.21.15.49:0/4263539815 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3fec0046c0 con 0x7f40041018d0 2024-03-18T00:50:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.456+0000 7f3ffbfff640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3ff00a2800 0x7f3ff00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.456+0000 7f4002ffd640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3ff00a2800 0x7f3ff00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.456+0000 7f3ffbfff640 1 -- 172.21.15.49:0/4263539815 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3fec0ceef0 con 0x7f40041018d0 2024-03-18T00:50:09.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.457+0000 7f4002ffd640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3ff00a2800 0x7f3ff00a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc000c00 tx=0x7f3ffc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:09.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.460+0000 7f3ffbfff640 1 -- 172.21.15.49:0/4263539815 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3fec09bf70 con 0x7f40041018d0 2024-03-18T00:50:09.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:09 smithi049 ceph-mon[32130]: from='client.25279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:50:09.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:09 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2005758306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:09.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:09 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2554772140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:50:09.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.644+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f400402d340 con 0x7f40041018d0 2024-03-18T00:50:09.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.645+0000 7f3ffbfff640 1 -- 172.21.15.49:0/4263539815 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f3fec09b870 con 0x7f40041018d0 2024-03-18T00:50:09.645 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 64 2024-03-18T00:50:09.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3ff00a2800 msgr2=0x7f3ff00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3ff00a2800 0x7f3ff00a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc000c00 tx=0x7f3ffc005fd0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40041018d0 msgr2=0x7f40041096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40041018d0 0x7f40041096a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00e9f0 tx=0x7f3fec00edf0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 shutdown_connections 2024-03-18T00:50:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3ff00a2800 0x7f3ff00a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40041018d0 0x7f40041096a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4004102290 0x7f4004109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 --2- 172.21.15.49:0/4263539815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40041006e0 0x7f400410b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 >> 172.21.15.49:0/4263539815 conn(0x7f40040fbc70 msgr2=0x7f4004104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 shutdown_connections 2024-03-18T00:50:09.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.647+0000 7f40093e6640 1 -- 172.21.15.49:0/4263539815 wait complete. 2024-03-18T00:50:09.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:50:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:09 smithi107 ceph-mon[36270]: from='client.25279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:50:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:09 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2005758306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:09 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2554772140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:50:09.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:09 smithi106 ceph-mon[35899]: from='client.25279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T00:50:09.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:09 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2005758306' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:09.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:09 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2554772140' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T00:50:09.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.768+0000 7f9feb259640 1 -- 172.21.15.49:0/2458828835 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 msgr2=0x7f9fe4101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.768+0000 7f9feb259640 1 --2- 172.21.15.49:0/2458828835 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 0x7f9fe4101c20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009900 tx=0x7f9fd402bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.768+0000 7f9feb259640 1 -- 172.21.15.49:0/2458828835 shutdown_connections 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.768+0000 7f9feb259640 1 --2- 172.21.15.49:0/2458828835 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe41029f0 0x7f9fe4102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.768+0000 7f9feb259640 1 --2- 172.21.15.49:0/2458828835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fe41033b0 0x7f9fe4109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.768+0000 7f9feb259640 1 --2- 172.21.15.49:0/2458828835 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 0x7f9fe4101c20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.768+0000 7f9feb259640 1 -- 172.21.15.49:0/2458828835 >> 172.21.15.49:0/2458828835 conn(0x7f9fe40fcd90 msgr2=0x7f9fe40ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.768+0000 7f9feb259640 1 -- 172.21.15.49:0/2458828835 shutdown_connections 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.768+0000 7f9feb259640 1 -- 172.21.15.49:0/2458828835 wait complete. 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9feb259640 1 Processor -- start 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9feb259640 1 -- start start 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9feb259640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 0x7f9fe4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9feb259640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fe41029f0 0x7f9fe4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9feb259640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe41033b0 0x7f9fe419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9feb259640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe410d480 con 0x7f9fe4101800 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9feb259640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe410d300 con 0x7f9fe41033b0 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9feb259640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe410d600 con 0x7f9fe41029f0 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9fe8ff5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 0x7f9fe4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9fe8ff5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 0x7f9fe4197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58284/0 (socket says 172.21.15.49:58284) 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9fe8ff5640 1 -- 172.21.15.49:0/1200403119 learned_addr learned my addr 172.21.15.49:0/1200403119 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9fe3fff640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fe41029f0 0x7f9fe4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.769+0000 7f9fe97f6640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe41033b0 0x7f9fe419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe8ff5640 1 -- 172.21.15.49:0/1200403119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fe41029f0 msgr2=0x7f9fe4198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe8ff5640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fe41029f0 0x7f9fe4198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe8ff5640 1 -- 172.21.15.49:0/1200403119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe41033b0 msgr2=0x7f9fe419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe8ff5640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe41033b0 0x7f9fe419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe8ff5640 1 -- 172.21.15.49:0/1200403119 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe419f710 con 0x7f9fe4101800 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe3fff640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fe41029f0 0x7f9fe4198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:09.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe8ff5640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 0x7f9fe4197af0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4000c00 tx=0x7f9fd402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe1ffb640 1 -- 172.21.15.49:0/1200403119 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4041070 con 0x7f9fe4101800 2024-03-18T00:50:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe419f9a0 con 0x7f9fe4101800 2024-03-18T00:50:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe1ffb640 1 -- 172.21.15.49:0/1200403119 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd4004450 con 0x7f9fe4101800 2024-03-18T00:50:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9fe1ffb640 1 -- 172.21.15.49:0/1200403119 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4034440 con 0x7f9fe4101800 2024-03-18T00:50:09.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.770+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe419fe90 con 0x7f9fe4101800 2024-03-18T00:50:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.771+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe4108b20 con 0x7f9fe4101800 2024-03-18T00:50:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.772+0000 7f9fe1ffb640 1 -- 172.21.15.49:0/1200403119 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9fd4044070 con 0x7f9fe4101800 2024-03-18T00:50:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.773+0000 7f9fe1ffb640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9fcc0a2800 0x7f9fcc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.773+0000 7f9fe1ffb640 1 -- 172.21.15.49:0/1200403119 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9fd40f0540 con 0x7f9fe4101800 2024-03-18T00:50:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.773+0000 7f9fe3fff640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9fcc0a2800 0x7f9fcc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:09.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.773+0000 7f9fe3fff640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9fcc0a2800 0x7f9fcc0a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4199000 tx=0x7f9fd8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:09.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.777+0000 7f9fe1ffb640 1 -- 172.21.15.49:0/1200403119 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fd40bd000 con 0x7f9fe4101800 2024-03-18T00:50:09.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.998+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fe402d1b0 con 0x7f9fe4101800 2024-03-18T00:50:09.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:09.999+0000 7f9fe1ffb640 1 -- 172.21.15.49:0/1200403119 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11932 (secure 0 0 0) 0x7f9fd40c10a0 con 0x7f9fe4101800 2024-03-18T00:50:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:09.999 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:50:03.914926+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24463,24470,24481,24499,24506]},"id":1}]} 2024-03-18T00:50:10.000 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 25 2024-03-18T00:50:10.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.001+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9fcc0a2800 msgr2=0x7f9fcc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9fcc0a2800 0x7f9fcc0a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4199000 tx=0x7f9fd8005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 msgr2=0x7f9fe4197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 0x7f9fe4197af0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4000c00 tx=0x7f9fd402c640 comp rx=0 tx=0).stop 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 shutdown_connections 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9fcc0a2800 0x7f9fcc0a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9fe41033b0 0x7f9fe419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9fe41029f0 0x7f9fe4198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 --2- 172.21.15.49:0/1200403119 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9fe4101800 0x7f9fe4197af0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 >> 172.21.15.49:0/1200403119 conn(0x7f9fe40fcd90 msgr2=0x7f9fe4105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 shutdown_connections 2024-03-18T00:50:10.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.002+0000 7f9feb259640 1 -- 172.21.15.49:0/1200403119 wait complete. 2024-03-18T00:50:10.013 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T00:50:10.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 -- 172.21.15.49:0/1920008062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 msgr2=0x7f3d20108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 --2- 172.21.15.49:0/1920008062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 0x7f3d20108b60 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009ca0 tx=0x7f3d1802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 -- 172.21.15.49:0/1920008062 shutdown_connections 2024-03-18T00:50:10.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 --2- 172.21.15.49:0/1920008062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d20101910 0x7f3d20101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 --2- 172.21.15.49:0/1920008062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 0x7f3d20108b60 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 --2- 172.21.15.49:0/1920008062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d20100720 0x7f3d20100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 -- 172.21.15.49:0/1920008062 >> 172.21.15.49:0/1920008062 conn(0x7f3d200fbc90 msgr2=0x7f3d200fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 -- 172.21.15.49:0/1920008062 shutdown_connections 2024-03-18T00:50:10.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 -- 172.21.15.49:0/1920008062 wait complete. 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.122+0000 7f3d24e27640 1 Processor -- start 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d24e27640 1 -- start start 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d24e27640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d20100720 0x7f3d20197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d24e27640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d20101910 0x7f3d20198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d24e27640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 0x7f3d2019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d24e27640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2010c3a0 con 0x7f3d201022d0 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d24e27640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2010c220 con 0x7f3d20101910 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d24e27640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2010c520 con 0x7f3d20100720 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 0x7f3d2019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 0x7f3d2019f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58298/0 (socket says 172.21.15.49:58298) 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1ed9d640 1 -- 172.21.15.49:0/2084114639 learned_addr learned my addr 172.21.15.49:0/2084114639 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1dd9b640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d20101910 0x7f3d20198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1e59c640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d20100720 0x7f3d20197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1ed9d640 1 -- 172.21.15.49:0/2084114639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d20100720 msgr2=0x7f3d20197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1ed9d640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d20100720 0x7f3d20197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1ed9d640 1 -- 172.21.15.49:0/2084114639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d20101910 msgr2=0x7f3d20198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1ed9d640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d20101910 0x7f3d20198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1ed9d640 1 -- 172.21.15.49:0/2084114639 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2019f710 con 0x7f3d201022d0 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d1ed9d640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 0x7f3d2019f0a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3d18037040 tx=0x7f3d18032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d177fe640 1 -- 172.21.15.49:0/2084114639 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18041020 con 0x7f3d201022d0 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d177fe640 1 -- 172.21.15.49:0/2084114639 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d18034cd0 con 0x7f3d201022d0 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.123+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2019f9a0 con 0x7f3d201022d0 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.124+0000 7f3d177fe640 1 -- 172.21.15.49:0/2084114639 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18039040 con 0x7f3d201022d0 2024-03-18T00:50:10.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.124+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2019fe90 con 0x7f3d201022d0 2024-03-18T00:50:10.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.124+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d201a00c0 con 0x7f3d201022d0 2024-03-18T00:50:10.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.126+0000 7f3d177fe640 1 -- 172.21.15.49:0/2084114639 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3d18044020 con 0x7f3d201022d0 2024-03-18T00:50:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.126+0000 7f3d177fe640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.126+0000 7f3d1e59c640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.126+0000 7f3d177fe640 1 -- 172.21.15.49:0/2084114639 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3d180f08a0 con 0x7f3d201022d0 2024-03-18T00:50:10.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.127+0000 7f3d1e59c640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3d20101770 tx=0x7f3d10005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.131+0000 7f3d177fe640 1 -- 172.21.15.49:0/2084114639 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3d1804b020 con 0x7f3d201022d0 2024-03-18T00:50:10.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.318+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d201a0830 con 0x7f3d201022d0 2024-03-18T00:50:10.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.320+0000 7f3d177fe640 1 -- 172.21.15.49:0/2084114639 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7f3d180b7e20 con 0x7f3d201022d0 2024-03-18T00:50:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:10.320 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T00:50:10.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d0c0a2800 msgr2=0x7f3d0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3d20101770 tx=0x7f3d10005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:10.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 msgr2=0x7f3d2019f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 0x7f3d2019f0a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3d18037040 tx=0x7f3d18032f40 comp rx=0 tx=0).stop 2024-03-18T00:50:10.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 shutdown_connections 2024-03-18T00:50:10.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d0c0a2800 0x7f3d0c0a4cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d20101910 0x7f3d20198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d201022d0 0x7f3d2019f0a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 --2- 172.21.15.49:0/2084114639 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d20100720 0x7f3d20197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 >> 172.21.15.49:0/2084114639 conn(0x7f3d200fbc90 msgr2=0x7f3d20069e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 shutdown_connections 2024-03-18T00:50:10.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.322+0000 7f3d24e27640 1 -- 172.21.15.49:0/2084114639 wait complete. 2024-03-18T00:50:10.334 DEBUG:tasks.fwd_scrub:Yielding 2024-03-18T00:50:10.334 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-18T00:50:10.345 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi049.front.sepia.ceph.com 2024-03-18T00:50:10.345 DEBUG:teuthology.orchestra.run.smithi049:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-18T00:50:10.346 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-18T00:50:10.346 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:50:10.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:50:10.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:50:10.349 INFO:teuthology.run_tasks:Running task workunit... 2024-03-18T00:50:10.393 INFO:tasks.workunit:Pulling workunits from ref c382027d5c66ef4e0aa1131ea3e8080c658d3184 2024-03-18T00:50:10.394 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-18T00:50:10.394 INFO:tasks.workunit:timeout=3h 2024-03-18T00:50:10.394 INFO:tasks.workunit:cleanup=True 2024-03-18T00:50:10.394 DEBUG:teuthology.orchestra.run.smithi049:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-18T00:50:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-18T00:50:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-18T00:50:10.411 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-18T00:50:10.412 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T00:50:10.412 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-18T00:50:10.412 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-03-18 00:48:11.449498574 +0000 2024-03-18T00:50:10.412 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-03-18 00:48:11.449498574 +0000 2024-03-18T00:50:10.412 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-03-18 00:50:03.210824940 +0000 2024-03-18T00:50:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-03-18 00:48:11.449498574 +0000 2024-03-18T00:50:10.412 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-18T00:50:10.412 DEBUG:teuthology.orchestra.run.smithi049:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-18T00:50:10.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.461+0000 7f184fa85640 1 -- 172.21.15.49:0/995310336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18481018d0 msgr2=0x7f1848101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.461+0000 7f184fa85640 1 --2- 172.21.15.49:0/995310336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18481018d0 0x7f1848101d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1838009900 tx=0x7f183802bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:10.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.461+0000 7f184fa85640 1 -- 172.21.15.49:0/995310336 shutdown_connections 2024-03-18T00:50:10.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.461+0000 7f184fa85640 1 --2- 172.21.15.49:0/995310336 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1848102290 0x7f1848108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.461+0000 7f184fa85640 1 --2- 172.21.15.49:0/995310336 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f18481018d0 0x7f1848101d50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.461+0000 7f184fa85640 1 --2- 172.21.15.49:0/995310336 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18481006e0 0x7f1848100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.461+0000 7f184fa85640 1 -- 172.21.15.49:0/995310336 >> 172.21.15.49:0/995310336 conn(0x7f18480fbc70 msgr2=0x7f18480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.461+0000 7f184fa85640 1 -- 172.21.15.49:0/995310336 shutdown_connections 2024-03-18T00:50:10.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.461+0000 7f184fa85640 1 -- 172.21.15.49:0/995310336 wait complete. 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184fa85640 1 Processor -- start 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184fa85640 1 -- start start 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184fa85640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18481006e0 0x7f1848104860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184fa85640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18481018d0 0x7f1848102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184fa85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1848102290 0x7f18481033e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184fa85640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f184810c360 con 0x7f18481018d0 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184fa85640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f184810c1e0 con 0x7f18481006e0 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184fa85640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f184810c4e0 con 0x7f1848102290 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184d020640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18481018d0 0x7f1848102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184d821640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18481006e0 0x7f1848104860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184d020640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18481018d0 0x7f1848102ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58320/0 (socket says 172.21.15.49:58320) 2024-03-18T00:50:10.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184d020640 1 -- 172.21.15.49:0/1205325799 learned_addr learned my addr 172.21.15.49:0/1205325799 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184e022640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1848102290 0x7f18481033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184d821640 1 -- 172.21.15.49:0/1205325799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1848102290 msgr2=0x7f18481033e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184d821640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1848102290 0x7f18481033e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184d821640 1 -- 172.21.15.49:0/1205325799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18481018d0 msgr2=0x7f1848102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184d821640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18481018d0 0x7f1848102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184d821640 1 -- 172.21.15.49:0/1205325799 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1848103c70 con 0x7f18481006e0 2024-03-18T00:50:10.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.462+0000 7f184e022640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1848102290 0x7f18481033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.463+0000 7f184d821640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18481006e0 0x7f1848104860 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f184400e9c0 tx=0x7f184400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.463+0000 7f183effd640 1 -- 172.21.15.49:0/1205325799 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184400ccc0 con 0x7f18481006e0 2024-03-18T00:50:10.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.463+0000 7f184fa85640 1 -- 172.21.15.49:0/1205325799 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18481a1c50 con 0x7f18481006e0 2024-03-18T00:50:10.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.463+0000 7f183effd640 1 -- 172.21.15.49:0/1205325799 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18440035f0 con 0x7f18481006e0 2024-03-18T00:50:10.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.463+0000 7f183effd640 1 -- 172.21.15.49:0/1205325799 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184400ce60 con 0x7f18481006e0 2024-03-18T00:50:10.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.463+0000 7f184fa85640 1 -- 172.21.15.49:0/1205325799 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18481a2120 con 0x7f18481006e0 2024-03-18T00:50:10.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.465+0000 7f183effd640 1 -- 172.21.15.49:0/1205325799 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f18440037a0 con 0x7f18481006e0 2024-03-18T00:50:10.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f185267a740 1 Processor -- start 2024-03-18T00:50:10.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f185267a740 1 -- start start 2024-03-18T00:50:10.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f185267a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cdd990 0x5604a5cdddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f185267a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f185267a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5ce5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f185267a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5604a5c420d0 con 0x5604a5ce34f0 2024-03-18T00:50:10.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f185267a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5604a5beac70 con 0x5604a5cde380 2024-03-18T00:50:10.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f185267a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5604a5c3e170 con 0x5604a5cdd990 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f183effd640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18340a2800 0x7f18340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f184d020640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5ce5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f184d821640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f184e022640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cdd990 0x5604a5cdddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f184d020640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5ce5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58326/0 (socket says 172.21.15.49:58326) 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f184d020640 1 -- 172.21.15.49:0/3208424799 learned_addr learned my addr 172.21.15.49:0/3208424799 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f184e022640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18340a2800 0x7f18340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.466+0000 7f183effd640 1 -- 172.21.15.49:0/1205325799 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f18440cedd0 con 0x7f18481006e0 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f184e022640 1 -- 172.21.15.49:0/3208424799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 msgr2=0x5604a5ce2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f184e022640 1 --2- 172.21.15.49:0/3208424799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f184e022640 1 -- 172.21.15.49:0/3208424799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 msgr2=0x5604a5ce5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f184e022640 1 --2- 172.21.15.49:0/3208424799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5ce5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f184e022640 1 -- 172.21.15.49:0/3208424799 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604a5a1e7e0 con 0x5604a5cdd990 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f184d821640 1 --2- 172.21.15.49:0/3208424799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f184d020640 1 --2- 172.21.15.49:0/3208424799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5ce5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f184e022640 1 --2- 172.21.15.49:0/3208424799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cdd990 0x5604a5cdddb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f18400099a0 tx=0x7f1840011f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf68b9d2eb5f8185 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f184e022640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18340a2800 0x7f18340a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1840005fd0 tx=0x7f184001e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f1833fff640 1 -- 172.21.15.49:0/3208424799 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1840022070 con 0x5604a5cdd990 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f1833fff640 1 -- 172.21.15.49:0/3208424799 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1840011570 con 0x5604a5cdd990 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.467+0000 7f1833fff640 1 -- 172.21.15.49:0/3208424799 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1840004310 con 0x5604a5cdd990 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.468+0000 7f185267a740 1 -- 172.21.15.49:0/3208424799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cdd990 msgr2=0x5604a5cdddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.468+0000 7f185267a740 1 --2- 172.21.15.49:0/3208424799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cdd990 0x5604a5cdddb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f18400099a0 tx=0x7f1840011f90 comp rx=0 tx=0).stop 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.468+0000 7f185267a740 1 -- 172.21.15.49:0/3208424799 shutdown_connections 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.468+0000 7f185267a740 1 --2- 172.21.15.49:0/3208424799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.468+0000 7f185267a740 1 --2- 172.21.15.49:0/3208424799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5ce5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.468+0000 7f185267a740 1 --2- 172.21.15.49:0/3208424799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cdd990 0x5604a5cdddb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.468+0000 7f185267a740 1 -- 172.21.15.49:0/3208424799 >> 172.21.15.49:0/3208424799 conn(0x5604a5ca5c50 msgr2=0x5604a5cd9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.468+0000 7f185267a740 1 -- 172.21.15.49:0/3208424799 shutdown_connections 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.468+0000 7f185267a740 1 -- 172.21.15.49:0/3208424799 wait complete. 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f185267a740 1 -- start start 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f185267a740 1 Processor -- start 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f185267a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5ce7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f185267a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f185267a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5ce34f0 0x5604a5cef430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f185267a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5604a5ce63e0 con 0x5604a5cdd990 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f185267a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5604a5bfe0d0 con 0x5604a5cde380 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f185267a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5604a5ce6560 con 0x5604a5ce34f0 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f185267a740 10 client.0 _ll_register_callbacks cb 0x5604a5bc7760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f184d821640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5ce7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f184d821640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5ce7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58340/0 (socket says 172.21.15.49:58340) 2024-03-18T00:50:10.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f184d821640 1 -- 172.21.15.49:0/3693916161 learned_addr learned my addr 172.21.15.49:0/3693916161 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f184d020640 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f184e022640 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5ce34f0 0x5604a5cef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f184d821640 1 -- 172.21.15.49:0/3693916161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5ce34f0 msgr2=0x5604a5cef430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f184d821640 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5ce34f0 0x5604a5cef430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f184d821640 1 -- 172.21.15.49:0/3693916161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 msgr2=0x5604a5ce83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.469+0000 7f184d821640 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f184d821640 1 -- 172.21.15.49:0/3693916161 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604a5a7ef70 con 0x5604a5cdd990 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f184d020640 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f184e022640 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5ce34f0 0x5604a5cef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f184d821640 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5ce7eb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f18440cc900 tx=0x7f184401dee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87aea7b3988cb37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f18317fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f18317fa640 1 -- 172.21.15.49:0/3693916161 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184409be50 con 0x5604a5cdd990 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f18317fa640 1 -- 172.21.15.49:0/3693916161 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f1844003b50 con 0x5604a5cdd990 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f185267a740 1 -- 172.21.15.49:0/3693916161 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1840011570 con 0x5604a5cdd990 2024-03-18T00:50:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f18317fa640 1 -- 172.21.15.49:0/3693916161 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f184409b750 con 0x5604a5cdd990 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f185267a740 10 client.15459 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f185267a740 1 -- 172.21.15.49:0/3693916161 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5604a5916170 con 0x5604a5cdd990 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f18317fa640 1 -- 172.21.15.49:0/3693916161 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1840011570 con 0x5604a5cdd990 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f18317fa640 1 -- 172.21.15.49:0/3693916161 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f1844084ce0 con 0x5604a5cdd990 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f185267a740 10 client.15459 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f185267a740 10 client.15459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.470+0000 7f185267a740 20 client.15459 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.471+0000 7f185267a740 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5604a5cf38f0 0x5604a5cf5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.471+0000 7f185267a740 4 client.15459 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.471+0000 7f185267a740 1 -- 172.21.15.49:0/3693916161 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604a5c4f610 con 0x5604a5cf38f0 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.471+0000 7f184d020640 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5604a5cf38f0 0x5604a5cf5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.471+0000 7f184d020640 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5604a5cf38f0 0x5604a5cf5d60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e369c94adeae94f server_cookie=884c81fc0ca3de32 in_seq=0 out_seq=0 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.471+0000 7f18317fa640 10 client.15459 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f18317fa640 1 -- 172.21.15.49:0/3693916161 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5604a5c4f610 con 0x5604a5cf38f0 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f18317fa640 10 client.15459 handle_command_reply: tid=0 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 client.15459 shutdown 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 -- 172.21.15.49:0/3693916161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 msgr2=0x5604a5ce7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5ce7eb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f18440cc900 tx=0x7f184401dee0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 -- 172.21.15.49:0/3693916161 shutdown_connections 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5604a5cf38f0 0x5604a5cf5d60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cde380 0x5604a5ce83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5ce34f0 0x5604a5cef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f18317fa640 0 client.15459 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 --2- 172.21.15.49:0/3693916161 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5ce7eb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 -- 172.21.15.49:0/3693916161 >> 172.21.15.49:0/3693916161 conn(0x5604a5ca5c50 msgr2=0x5604a5cd84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 -- 172.21.15.49:0/3693916161 shutdown_connections 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 1 -- 172.21.15.49:0/3693916161 wait complete. 2024-03-18T00:50:10.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.472+0000 7f185267a740 20 client.15459 trim_cache size 0 max 16384 2024-03-18T00:50:10.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.490+0000 7f185267a740 1 Processor -- start 2024-03-18T00:50:10.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.490+0000 7f185267a740 1 -- start start 2024-03-18T00:50:10.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.490+0000 7f185267a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5cdcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f185267a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5cef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f185267a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5ce34f0 0x5604a5ce7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f185267a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f18481aafb0 con 0x5604a5cdd990 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f185267a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f18481acaf0 con 0x5604a5ce34f0 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f185267a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18481ab680 con 0x5604a5cde380 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d821640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5cdcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d020640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5cef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184e022640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5ce34f0 0x5604a5ce7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d821640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5cdcc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58350/0 (socket says 172.21.15.49:58350) 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d821640 1 -- 172.21.15.49:0/1561283469 learned_addr learned my addr 172.21.15.49:0/1561283469 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d020640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5cef3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:41720/0 (socket says 172.21.15.49:41720) 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d020640 1 -- 172.21.15.49:0/1561283469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5ce34f0 msgr2=0x5604a5ce7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d020640 1 --2- 172.21.15.49:0/1561283469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5ce34f0 0x5604a5ce7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d020640 1 -- 172.21.15.49:0/1561283469 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 msgr2=0x5604a5cdcc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d020640 1 --2- 172.21.15.49:0/1561283469 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5cdcc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d020640 1 -- 172.21.15.49:0/1561283469 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604a5cdd5d0 con 0x5604a5cde380 2024-03-18T00:50:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d821640 1 --2- 172.21.15.49:0/1561283469 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5cdcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f184d020640 1 --2- 172.21.15.49:0/1561283469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5cef3f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f183802cfd0 tx=0x7f183802c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2b8f3114ddafc13 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f18337fe640 1 -- 172.21.15.49:0/1561283469 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1838048d20 con 0x5604a5cde380 2024-03-18T00:50:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f18337fe640 1 -- 172.21.15.49:0/1561283469 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18380046c0 con 0x5604a5cde380 2024-03-18T00:50:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.491+0000 7f18337fe640 1 -- 172.21.15.49:0/1561283469 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18380051c0 con 0x5604a5cde380 2024-03-18T00:50:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 -- 172.21.15.49:0/1561283469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 msgr2=0x5604a5cef3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 --2- 172.21.15.49:0/1561283469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5cef3f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f183802cfd0 tx=0x7f183802c880 comp rx=0 tx=0).stop 2024-03-18T00:50:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 -- 172.21.15.49:0/1561283469 shutdown_connections 2024-03-18T00:50:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 --2- 172.21.15.49:0/1561283469 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5ce34f0 0x5604a5ce7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 --2- 172.21.15.49:0/1561283469 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5cef3f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 --2- 172.21.15.49:0/1561283469 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5cdd990 0x5604a5cdcc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 -- 172.21.15.49:0/1561283469 >> 172.21.15.49:0/1561283469 conn(0x5604a5ca5c50 msgr2=0x5604a5cdc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 -- 172.21.15.49:0/1561283469 shutdown_connections 2024-03-18T00:50:10.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 -- 172.21.15.49:0/1561283469 wait complete. 2024-03-18T00:50:10.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.492+0000 7f185267a740 1 -- start start 2024-03-18T00:50:10.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f185267a740 1 Processor -- start 2024-03-18T00:50:10.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f185267a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cdd990 0x5604a5ced0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f185267a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5ced620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f185267a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5cee630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f185267a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f18481ac420 con 0x5604a5ce34f0 2024-03-18T00:50:10.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f185267a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f18481abd50 con 0x5604a5cdd990 2024-03-18T00:50:10.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f185267a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f18481a8f90 con 0x5604a5cde380 2024-03-18T00:50:10.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f185267a740 10 client.0 _ll_register_callbacks cb 0x7f18440960d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:10.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184e022640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5cee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184e022640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5cee630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58356/0 (socket says 172.21.15.49:58356) 2024-03-18T00:50:10.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184e022640 1 -- 172.21.15.49:0/686906205 learned_addr learned my addr 172.21.15.49:0/686906205 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184d821640 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cdd990 0x5604a5ced0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184d020640 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5ced620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184e022640 1 -- 172.21.15.49:0/686906205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 msgr2=0x5604a5ced620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184e022640 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5ced620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184e022640 1 -- 172.21.15.49:0/686906205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cdd990 msgr2=0x5604a5ced0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184e022640 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cdd990 0x5604a5ced0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.493+0000 7f184e022640 1 -- 172.21.15.49:0/686906205 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604a5cdd190 con 0x5604a5ce34f0 2024-03-18T00:50:10.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f184e022640 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5cee630 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x5604a5cdebe0 tx=0x7f1840003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dac10bc204d51cf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f18317fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:50:10.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f18317fa640 1 -- 172.21.15.49:0/686906205 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1840034070 con 0x5604a5ce34f0 2024-03-18T00:50:10.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f18317fa640 1 -- 172.21.15.49:0/686906205 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f1840011510 con 0x5604a5ce34f0 2024-03-18T00:50:10.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f185267a740 1 -- 172.21.15.49:0/686906205 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18380046c0 con 0x5604a5ce34f0 2024-03-18T00:50:10.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f18317fa640 1 -- 172.21.15.49:0/686906205 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f184002a040 con 0x5604a5ce34f0 2024-03-18T00:50:10.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f185267a740 10 client.15465 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:10.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f185267a740 1 -- 172.21.15.49:0/686906205 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5604a5d5a190 con 0x5604a5ce34f0 2024-03-18T00:50:10.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f18317fa640 1 -- 172.21.15.49:0/686906205 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18380046c0 con 0x5604a5ce34f0 2024-03-18T00:50:10.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f18317fa640 1 -- 172.21.15.49:0/686906205 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f1840043540 con 0x5604a5ce34f0 2024-03-18T00:50:10.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f185267a740 10 client.15465 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:10.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f185267a740 10 client.15465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:10.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f185267a740 20 client.15465 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:10.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f185267a740 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5604a5d5aab0 0x5604a5d5ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f185267a740 4 client.15465 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:50:10.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.494+0000 7f185267a740 1 -- 172.21.15.49:0/686906205 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5604a5cdf460 con 0x5604a5d5aab0 2024-03-18T00:50:10.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.495+0000 7f184d821640 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5604a5d5aab0 0x5604a5d5ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.495+0000 7f184d821640 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5604a5d5aab0 0x5604a5d5ced0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f10cc808b811f559 server_cookie=94d3527f284f39e9 in_seq=0 out_seq=0 2024-03-18T00:50:10.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.495+0000 7f18317fa640 10 client.15465 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:10.510 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c382027d5c66ef4e0aa1131ea3e8080c658d3184 2024-03-18T00:50:10.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f18317fa640 1 -- 172.21.15.49:0/686906205 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5604a5cdf460 con 0x5604a5d5aab0 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f18317fa640 10 client.15465 handle_command_reply: tid=0 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 client.15465 shutdown 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 -- 172.21.15.49:0/686906205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 msgr2=0x5604a5cee630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5cee630 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x5604a5cdebe0 tx=0x7f1840003f20 comp rx=0 tx=0).stop 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 -- 172.21.15.49:0/686906205 shutdown_connections 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5604a5d5aab0 0x5604a5d5ced0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5604a5ce34f0 0x5604a5cee630 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5604a5cde380 0x5604a5ced620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f18317fa640 0 client.15465 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 --2- 172.21.15.49:0/686906205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5604a5cdd990 0x5604a5ced0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 -- 172.21.15.49:0/686906205 >> 172.21.15.49:0/686906205 conn(0x5604a5ca5c50 msgr2=0x5604a5ce7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 -- 172.21.15.49:0/686906205 shutdown_connections 2024-03-18T00:50:10.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 1 -- 172.21.15.49:0/686906205 wait complete. 2024-03-18T00:50:10.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.496+0000 7f185267a740 20 client.15465 trim_cache size 0 max 16384 2024-03-18T00:50:10.514 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:50:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:50:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "275c30df-7f7a-4b25-9a70-d479df6ca9af", 2024-03-18T00:50:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:50:10.514 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:50:10.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 -- 172.21.15.49:0/1205325799 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18340a2800 msgr2=0x7f18340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18340a2800 0x7f18340a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1840005fd0 tx=0x7f184001e000 comp rx=0 tx=0).stop 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 -- 172.21.15.49:0/1205325799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18481006e0 msgr2=0x7f1848104860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18481006e0 0x7f1848104860 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f184400e9c0 tx=0x7f184400edc0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 -- 172.21.15.49:0/1205325799 shutdown_connections 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f18340a2800 0x7f18340a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1848102290 0x7f18481033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f18481018d0 0x7f1848102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 --2- 172.21.15.49:0/1205325799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f18481006e0 0x7f1848104860 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 -- 172.21.15.49:0/1205325799 >> 172.21.15.49:0/1205325799 conn(0x7f18480fbc70 msgr2=0x7f1848068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 -- 172.21.15.49:0/1205325799 shutdown_connections 2024-03-18T00:50:10.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.497+0000 7f184fa85640 1 -- 172.21.15.49:0/1205325799 wait complete. 2024-03-18T00:50:10.517 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:50:10.517 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:50:10.567 INFO:tasks.workunit.client.0.smithi049.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-18T00:50:10.578 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:10 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/559245882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:50:10.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:10 smithi049 ceph-mon[32130]: pgmap v352: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T00:50:10.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:10 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/4263539815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:50:10.579 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:10 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1200403119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:50:10.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.673+0000 7efd45edc640 1 -- 172.21.15.49:0/3912884635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd40100700 msgr2=0x7efd40100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.673+0000 7efd45edc640 1 --2- 172.21.15.49:0/3912884635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd40100700 0x7efd40100b20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7efd34009900 tx=0x7efd3402bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:10.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.673+0000 7efd45edc640 1 -- 172.21.15.49:0/3912884635 shutdown_connections 2024-03-18T00:50:10.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.673+0000 7efd45edc640 1 --2- 172.21.15.49:0/3912884635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd401018f0 0x7efd40101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.673+0000 7efd45edc640 1 --2- 172.21.15.49:0/3912884635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd401022b0 0x7efd40108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.673+0000 7efd45edc640 1 --2- 172.21.15.49:0/3912884635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd40100700 0x7efd40100b20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.673+0000 7efd45edc640 1 -- 172.21.15.49:0/3912884635 >> 172.21.15.49:0/3912884635 conn(0x7efd400fbc90 msgr2=0x7efd400fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.673+0000 7efd45edc640 1 -- 172.21.15.49:0/3912884635 shutdown_connections 2024-03-18T00:50:10.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.673+0000 7efd45edc640 1 -- 172.21.15.49:0/3912884635 wait complete. 2024-03-18T00:50:10.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd45edc640 1 Processor -- start 2024-03-18T00:50:10.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd45edc640 1 -- start start 2024-03-18T00:50:10.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd45edc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd40100700 0x7efd4006a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd45edc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd401018f0 0x7efd4019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd45edc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd401022b0 0x7efd4019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd45edc640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efd4010c350 con 0x7efd401022b0 2024-03-18T00:50:10.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd45edc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efd4010c1d0 con 0x7efd401018f0 2024-03-18T00:50:10.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd45edc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efd4010c4d0 con 0x7efd40100700 2024-03-18T00:50:10.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd3ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd401022b0 0x7efd4019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd3f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd40100700 0x7efd4006a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd3ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd401022b0 0x7efd4019eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58376/0 (socket says 172.21.15.49:58376) 2024-03-18T00:50:10.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd3ffff640 1 -- 172.21.15.49:0/15226001 learned_addr learned my addr 172.21.15.49:0/15226001 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd3f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd40100700 0x7efd4006a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:41740/0 (socket says 172.21.15.49:41740) 2024-03-18T00:50:10.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.674+0000 7efd3effd640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd401018f0 0x7efd4019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3ffff640 1 -- 172.21.15.49:0/15226001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd40100700 msgr2=0x7efd4006a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3ffff640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd40100700 0x7efd4006a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3ffff640 1 -- 172.21.15.49:0/15226001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd401018f0 msgr2=0x7efd4019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3ffff640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd401018f0 0x7efd4019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3ffff640 1 -- 172.21.15.49:0/15226001 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd4019f780 con 0x7efd401022b0 2024-03-18T00:50:10.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3effd640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd401018f0 0x7efd4019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd40100700 0x7efd4006a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3ffff640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd401022b0 0x7efd4019eec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efd3800efd0 tx=0x7efd3800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3cff9640 1 -- 172.21.15.49:0/15226001 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd38003b70 con 0x7efd401022b0 2024-03-18T00:50:10.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3cff9640 1 -- 172.21.15.49:0/15226001 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd38003d10 con 0x7efd401022b0 2024-03-18T00:50:10.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd45edc640 1 -- 172.21.15.49:0/15226001 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd4019fa70 con 0x7efd401022b0 2024-03-18T00:50:10.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd3cff9640 1 -- 172.21.15.49:0/15226001 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd380109f0 con 0x7efd401022b0 2024-03-18T00:50:10.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.675+0000 7efd45edc640 1 -- 172.21.15.49:0/15226001 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd4019ff70 con 0x7efd401022b0 2024-03-18T00:50:10.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:10 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/559245882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:50:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:10 smithi107 ceph-mon[36270]: pgmap v352: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T00:50:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:10 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/4263539815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:50:10.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:10 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1200403119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.677+0000 7efd3cff9640 1 -- 172.21.15.49:0/15226001 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efd38004020 con 0x7efd401022b0 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.677+0000 7efd48ba1740 1 Processor -- start 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.677+0000 7efd48ba1740 1 -- start start 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb124170 0x5633fb124590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb129660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb129cd0 0x5633fb12c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd48ba1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5633fb089500 con 0x5633fb124170 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd48ba1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5633fb031be0 con 0x5633fb124b60 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd48ba1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5633fb0855a0 con 0x5633fb129cd0 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3cff9640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd280a2800 0x7efd280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb124170 0x5633fb124590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb124170 0x5633fb124590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58392/0 (socket says 172.21.15.49:58392) 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 -- 172.21.15.49:0/2966594001 learned_addr learned my addr 172.21.15.49:0/2966594001 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb129660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb129cd0 0x5633fb12c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3ffff640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd280a2800 0x7efd280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3cff9640 1 -- 172.21.15.49:0/15226001 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7efd380cee80 con 0x7efd401022b0 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 -- 172.21.15.49:0/2966594001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb129cd0 msgr2=0x5633fb12c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb129cd0 0x5633fb12c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 -- 172.21.15.49:0/2966594001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 msgr2=0x5633fb129660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb129660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 -- 172.21.15.49:0/2966594001 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633fb0454b0 con 0x5633fb124170 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb129660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3effd640 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb124170 0x5633fb124590 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7efd30013c20 tx=0x7efd30009b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63172214fb2246f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd25ffb640 1 -- 172.21.15.49:0/2966594001 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd30013d10 con 0x5633fb124170 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.678+0000 7efd3ffff640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd280a2800 0x7efd280a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efd38005c40 tx=0x7efd3808d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd25ffb640 1 -- 172.21.15.49:0/2966594001 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd30011c70 con 0x5633fb124170 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd25ffb640 1 -- 172.21.15.49:0/2966594001 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3000e3b0 con 0x5633fb124170 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd48ba1740 1 -- 172.21.15.49:0/2966594001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb124170 msgr2=0x5633fb124590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb124170 0x5633fb124590 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7efd30013c20 tx=0x7efd30009b70 comp rx=0 tx=0).stop 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd48ba1740 1 -- 172.21.15.49:0/2966594001 shutdown_connections 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb129660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb129cd0 0x5633fb12c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2966594001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb124170 0x5633fb124590 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd48ba1740 1 -- 172.21.15.49:0/2966594001 >> 172.21.15.49:0/2966594001 conn(0x5633fb11f6e0 msgr2=0x5633fb11fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd48ba1740 1 -- 172.21.15.49:0/2966594001 shutdown_connections 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.679+0000 7efd48ba1740 1 -- 172.21.15.49:0/2966594001 wait complete. 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd48ba1740 1 -- start start 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd48ba1740 1 Processor -- start 2024-03-18T00:50:10.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124170 0x5633fb12e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb12eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd48ba1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5633fb12cc20 con 0x5633fb129cd0 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd48ba1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5633fb045160 con 0x5633fb124b60 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd48ba1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5633fb12cda0 con 0x5633fb124170 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd48ba1740 10 client.0 _ll_register_callbacks cb 0x5633fb029150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb135ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb135ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58408/0 (socket says 172.21.15.49:58408) 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3effd640 1 -- 172.21.15.49:0/2988018424 learned_addr learned my addr 172.21.15.49:0/2988018424 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb12eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3effd640 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124170 0x5633fb12e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3f7fe640 1 -- 172.21.15.49:0/2988018424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124170 msgr2=0x5633fb12e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124170 0x5633fb12e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3f7fe640 1 -- 172.21.15.49:0/2988018424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 msgr2=0x5633fb135ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb135ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3f7fe640 1 -- 172.21.15.49:0/2988018424 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633fae677d0 con 0x5633fb124b60 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.680+0000 7efd3effd640 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb12eaa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efd3402c170 tx=0x7efd34005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb2588816db71acd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd1effd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd1effd640 1 -- 172.21.15.49:0/2988018424 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd34005d50 con 0x5633fb124b60 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd1effd640 1 -- 172.21.15.49:0/2988018424 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7efd340046c0 con 0x5633fb124b60 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd48ba1740 1 -- 172.21.15.49:0/2988018424 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd30011c70 con 0x5633fb124b60 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd1effd640 1 -- 172.21.15.49:0/2988018424 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efd340048c0 con 0x5633fb124b60 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd48ba1740 10 client.25316 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd48ba1740 1 -- 172.21.15.49:0/2988018424 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5633fad9d3f0 con 0x5633fb124b60 2024-03-18T00:50:10.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd1effd640 1 -- 172.21.15.49:0/2988018424 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd30011c70 con 0x5633fb124b60 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd1effd640 1 -- 172.21.15.49:0/2988018424 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7efd34003c20 con 0x5633fb124b60 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd48ba1740 10 client.25316 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd48ba1740 10 client.25316 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd48ba1740 20 client.25316 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5633fb13a020 0x5633fb13c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd48ba1740 4 client.25316 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.681+0000 7efd48ba1740 1 -- 172.21.15.49:0/2988018424 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633fb096a40 con 0x5633fb13a020 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.682+0000 7efd3effd640 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5633fb13a020 0x5633fb13c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.682+0000 7efd3effd640 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5633fb13a020 0x5633fb13c460 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32ee8a0e499a0b3c server_cookie=fe4e2b9202e82531 in_seq=0 out_seq=0 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.682+0000 7efd1effd640 10 client.25316 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd1effd640 1 -- 172.21.15.49:0/2988018424 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633fb096a40 con 0x5633fb13a020 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd1effd640 10 client.25316 handle_command_reply: tid=0 2024-03-18T00:50:10.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 client.25316 shutdown 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 -- 172.21.15.49:0/2988018424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 msgr2=0x5633fb12eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb12eaa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efd3402c170 tx=0x7efd34005270 comp rx=0 tx=0).stop 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 -- 172.21.15.49:0/2988018424 shutdown_connections 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5633fb13a020 0x5633fb13c460 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124b60 0x5633fb12eaa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb135ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 --2- 172.21.15.49:0/2988018424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124170 0x5633fb12e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd1effd640 0 client.25316 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 -- 172.21.15.49:0/2988018424 >> 172.21.15.49:0/2988018424 conn(0x5633fb11f6e0 msgr2=0x5633fb11e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 -- 172.21.15.49:0/2988018424 shutdown_connections 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 1 -- 172.21.15.49:0/2988018424 wait complete. 2024-03-18T00:50:10.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.683+0000 7efd48ba1740 20 client.25316 trim_cache size 0 max 16384 2024-03-18T00:50:10.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:10 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/559245882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T00:50:10.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:10 smithi106 ceph-mon[35899]: pgmap v352: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T00:50:10.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:10 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/4263539815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T00:50:10.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:10 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1200403119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd48ba1740 1 Processor -- start 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd48ba1740 1 -- start start 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 0x5633fb11a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 0x5633fb135aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb125f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd48ba1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efd401a8ce0 con 0x5633fb129cd0 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd48ba1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efd401aa820 con 0x5633fb124170 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd48ba1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efd401a93b0 con 0x5633fb124b60 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb125f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb125f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58418/0 (socket says 172.21.15.49:58418) 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 -- 172.21.15.49:0/3081987324 learned_addr learned my addr 172.21.15.49:0/3081987324 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 --2- 172.21.15.49:0/3081987324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 0x5633fb11a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/3081987324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 0x5633fb135aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 -- 172.21.15.49:0/3081987324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 msgr2=0x5633fb135aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 --2- 172.21.15.49:0/3081987324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 0x5633fb135aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 -- 172.21.15.49:0/3081987324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 msgr2=0x5633fb11a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 --2- 172.21.15.49:0/3081987324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 0x5633fb11a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 -- 172.21.15.49:0/3081987324 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633fad5d0f0 con 0x5633fb129cd0 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.701+0000 7efd3effd640 1 --2- 172.21.15.49:0/3081987324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb125f80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7efd30013bc0 tx=0x7efd30018040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41682f933878cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd257fa640 1 -- 172.21.15.49:0/3081987324 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd30025040 con 0x5633fb129cd0 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd257fa640 1 -- 172.21.15.49:0/3081987324 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd30005520 con 0x5633fb129cd0 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd257fa640 1 -- 172.21.15.49:0/3081987324 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd300320c0 con 0x5633fb129cd0 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd48ba1740 1 -- 172.21.15.49:0/3081987324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 msgr2=0x5633fb125f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3081987324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb125f80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7efd30013bc0 tx=0x7efd30018040 comp rx=0 tx=0).stop 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd48ba1740 1 -- 172.21.15.49:0/3081987324 shutdown_connections 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3081987324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb125f80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3081987324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 0x5633fb135aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3081987324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 0x5633fb11a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd48ba1740 1 -- 172.21.15.49:0/3081987324 >> 172.21.15.49:0/3081987324 conn(0x5633fb11f6e0 msgr2=0x5633fb1253c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd48ba1740 1 -- 172.21.15.49:0/3081987324 shutdown_connections 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.702+0000 7efd48ba1740 1 -- 172.21.15.49:0/3081987324 wait complete. 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd48ba1740 1 -- start start 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd48ba1740 1 Processor -- start 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 0x5633fb132e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 0x5633fb133340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd48ba1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb134350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd48ba1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efd401aa150 con 0x5633fb129cd0 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd48ba1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efd401a9a80 con 0x5633fb124170 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd48ba1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efd401a6cc0 con 0x5633fb124b60 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd48ba1740 10 client.0 _ll_register_callbacks cb 0x7efd34009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:10.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58432/0 (socket says 172.21.15.49:58432) 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3effd640 1 -- 172.21.15.49:0/3570816497 learned_addr learned my addr 172.21.15.49:0/3570816497 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3effd640 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 0x5633fb132e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 0x5633fb133340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3effd640 1 -- 172.21.15.49:0/3570816497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 msgr2=0x5633fb133340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3effd640 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 0x5633fb133340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3effd640 1 -- 172.21.15.49:0/3570816497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 msgr2=0x5633fb132e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3effd640 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 0x5633fb132e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.703+0000 7efd3effd640 1 -- 172.21.15.49:0/3570816497 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633fb0509d0 con 0x5633fb129cd0 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd3effd640 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb134350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efd30013710 tx=0x7efd30031040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8b7a088bc5036e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd1effd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd1effd640 1 -- 172.21.15.49:0/3570816497 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd3001faa0 con 0x5633fb129cd0 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd1effd640 1 -- 172.21.15.49:0/3570816497 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7efd3001f7b0 con 0x5633fb129cd0 2024-03-18T00:50:10.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd48ba1740 1 -- 172.21.15.49:0/3570816497 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd30005520 con 0x5633fb129cd0 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd1effd640 1 -- 172.21.15.49:0/3570816497 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7efd30032120 con 0x5633fb129cd0 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd48ba1740 10 client.15483 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd48ba1740 1 -- 172.21.15.49:0/3570816497 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5633fb135a60 con 0x5633fb129cd0 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd1effd640 1 -- 172.21.15.49:0/3570816497 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd30005520 con 0x5633fb129cd0 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd1effd640 1 -- 172.21.15.49:0/3570816497 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7efd3002b040 con 0x5633fb129cd0 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd48ba1740 10 client.15483 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd48ba1740 10 client.15483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd48ba1740 20 client.15483 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5633fb1a0650 0x5633fb1a2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd48ba1740 4 client.15483 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.704+0000 7efd48ba1740 1 -- 172.21.15.49:0/3570816497 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5633fb090d90 con 0x5633fb1a0650 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5633fb1a0650 0x5633fb1a2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd3f7fe640 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5633fb1a0650 0x5633fb1a2a40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef5026ba31002760 server_cookie=cf00c942bd3a998b in_seq=0 out_seq=0 2024-03-18T00:50:10.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd1effd640 10 client.15483 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd1effd640 1 -- 172.21.15.49:0/3570816497 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5633fb090d90 con 0x5633fb1a0650 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd1effd640 10 client.15483 handle_command_reply: tid=0 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 client.15483 shutdown 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 -- 172.21.15.49:0/3570816497 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 msgr2=0x5633fb134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb134350 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7efd30013710 tx=0x7efd30031040 comp rx=0 tx=0).stop 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 -- 172.21.15.49:0/3570816497 shutdown_connections 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5633fb1a0650 0x5633fb1a2a40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5633fb129cd0 0x5633fb134350 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5633fb124b60 0x5633fb133340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 --2- 172.21.15.49:0/3570816497 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5633fb124170 0x5633fb132e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 -- 172.21.15.49:0/3570816497 >> 172.21.15.49:0/3570816497 conn(0x5633fb11f6e0 msgr2=0x5633fb12e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd1effd640 0 client.15483 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 -- 172.21.15.49:0/3570816497 shutdown_connections 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 1 -- 172.21.15.49:0/3570816497 wait complete. 2024-03-18T00:50:10.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.705+0000 7efd48ba1740 20 client.15483 trim_cache size 0 max 16384 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "275c30df-7f7a-4b25-9a70-d479df6ca9af": { 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "275c30df-7f7a-4b25-9a70-d479df6ca9af", 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:50:10.712 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:50:10.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 -- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd280a2800 msgr2=0x7efd280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd280a2800 0x7efd280a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efd38005c40 tx=0x7efd3808d040 comp rx=0 tx=0).stop 2024-03-18T00:50:10.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 -- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd401022b0 msgr2=0x7efd4019eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:10.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd401022b0 0x7efd4019eec0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efd3800efd0 tx=0x7efd3800c6a0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 -- 172.21.15.49:0/15226001 shutdown_connections 2024-03-18T00:50:10.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd280a2800 0x7efd280a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efd401018f0 0x7efd4019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efd401022b0 0x7efd4019eec0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 --2- 172.21.15.49:0/15226001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efd40100700 0x7efd4006a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:10.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 -- 172.21.15.49:0/15226001 >> 172.21.15.49:0/15226001 conn(0x7efd400fbc90 msgr2=0x7efd40104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:10.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 -- 172.21.15.49:0/15226001 shutdown_connections 2024-03-18T00:50:10.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:10.706+0000 7efd45edc640 1 -- 172.21.15.49:0/15226001 wait complete. 2024-03-18T00:50:10.717 INFO:tasks.cephfs.filesystem:scrub status for tag:275c30df-7f7a-4b25-9a70-d479df6ca9af - {'path': '/', 'tag': '275c30df-7f7a-4b25-9a70-d479df6ca9af', 'options': 'recursive,force'} 2024-03-18T00:50:11.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:11 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2084114639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:11.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:11 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:50:11.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:11 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:11.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:11 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:50:11.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:11 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:11 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2084114639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:11 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:50:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:11 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:11 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:50:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:11 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:11 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2084114639' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T00:50:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:11 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:50:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:11 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:11.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:11 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:50:11.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:11 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:12.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:12 smithi049 ceph-mon[32130]: pgmap v353: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T00:50:12.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:12 smithi107 ceph-mon[36270]: pgmap v353: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T00:50:12.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:12 smithi106 ceph-mon[35899]: pgmap v353: 97 pgs: 97 active+clean; 648 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T00:50:14.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:14 smithi049 ceph-mon[32130]: pgmap v354: 97 pgs: 97 active+clean; 649 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-18T00:50:14.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:14 smithi107 ceph-mon[36270]: pgmap v354: 97 pgs: 97 active+clean; 649 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-18T00:50:14.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:14 smithi106 ceph-mon[35899]: pgmap v354: 97 pgs: 97 active+clean; 649 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-18T00:50:15.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:15 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:50:15.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:15 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:50:15.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:15 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:50:16.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:16 smithi049 ceph-mon[32130]: pgmap v355: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-18T00:50:16.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:16 smithi107 ceph-mon[36270]: pgmap v355: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-18T00:50:16.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:16 smithi106 ceph-mon[35899]: pgmap v355: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-18T00:50:18.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:18 smithi049 ceph-mon[32130]: pgmap v356: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 11 op/s 2024-03-18T00:50:18.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:18 smithi107 ceph-mon[36270]: pgmap v356: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 11 op/s 2024-03-18T00:50:18.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:18 smithi106 ceph-mon[35899]: pgmap v356: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1023 B/s wr, 11 op/s 2024-03-18T00:50:20.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:20 smithi049 ceph-mon[32130]: pgmap v357: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-18T00:50:20.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:20 smithi107 ceph-mon[36270]: pgmap v357: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-18T00:50:20.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:20 smithi106 ceph-mon[35899]: pgmap v357: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-18T00:50:21.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:21 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:50:21.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:21 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:50:21.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:21 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:50:22.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:22 smithi049 ceph-mon[32130]: pgmap v358: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-18T00:50:22.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:22 smithi107 ceph-mon[36270]: pgmap v358: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-18T00:50:22.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:22 smithi106 ceph-mon[35899]: pgmap v358: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-18T00:50:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.502+0000 7fc49726a640 1 -- 172.21.15.49:0/3147067518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc490100700 msgr2=0x7fc490100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.502+0000 7fc49726a640 1 --2- 172.21.15.49:0/3147067518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc490100700 0x7fc490100b20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009900 tx=0x7fc48c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.503+0000 7fc49726a640 1 -- 172.21.15.49:0/3147067518 shutdown_connections 2024-03-18T00:50:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.503+0000 7fc49726a640 1 --2- 172.21.15.49:0/3147067518 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4901018f0 0x7fc490101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.503+0000 7fc49726a640 1 --2- 172.21.15.49:0/3147067518 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4901022b0 0x7fc490108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.503+0000 7fc49726a640 1 --2- 172.21.15.49:0/3147067518 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc490100700 0x7fc490100b20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.503+0000 7fc49726a640 1 -- 172.21.15.49:0/3147067518 >> 172.21.15.49:0/3147067518 conn(0x7fc4900fbc70 msgr2=0x7fc4900fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.503+0000 7fc49726a640 1 -- 172.21.15.49:0/3147067518 shutdown_connections 2024-03-18T00:50:23.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.503+0000 7fc49726a640 1 -- 172.21.15.49:0/3147067518 wait complete. 2024-03-18T00:50:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.503+0000 7fc49726a640 1 Processor -- start 2024-03-18T00:50:23.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc49726a640 1 -- start start 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc49726a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc490100700 0x7fc490197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc49726a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4901018f0 0x7fc490197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc49726a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4901022b0 0x7fc49019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc49726a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc49010c350 con 0x7fc4901018f0 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc49726a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc49010c1d0 con 0x7fc4901022b0 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc49726a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc49010c4d0 con 0x7fc490100700 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc494805640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4901018f0 0x7fc490197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495006640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc490100700 0x7fc490197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495807640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4901022b0 0x7fc49019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc494805640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4901018f0 0x7fc490197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48192/0 (socket says 172.21.15.49:48192) 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495807640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4901022b0 0x7fc49019f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:38530/0 (socket says 172.21.15.49:38530) 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495006640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc490100700 0x7fc490197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:32878/0 (socket says 172.21.15.49:32878) 2024-03-18T00:50:23.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495807640 1 -- 172.21.15.49:0/4000914420 learned_addr learned my addr 172.21.15.49:0/4000914420 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495807640 1 -- 172.21.15.49:0/4000914420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc490100700 msgr2=0x7fc490197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495807640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc490100700 0x7fc490197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495807640 1 -- 172.21.15.49:0/4000914420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4901018f0 msgr2=0x7fc490197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495807640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4901018f0 0x7fc490197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495807640 1 -- 172.21.15.49:0/4000914420 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49019f6a0 con 0x7fc4901022b0 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495006640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc490100700 0x7fc490197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.504+0000 7fc495807640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4901022b0 0x7fc49019f030 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc48800eb10 tx=0x7fc48800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.505+0000 7fc4867fc640 1 -- 172.21.15.49:0/4000914420 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488004b20 con 0x7fc4901022b0 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.505+0000 7fc4867fc640 1 -- 172.21.15.49:0/4000914420 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc488004cc0 con 0x7fc4901022b0 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.505+0000 7fc49726a640 1 -- 172.21.15.49:0/4000914420 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49019f990 con 0x7fc4901022b0 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.505+0000 7fc4867fc640 1 -- 172.21.15.49:0/4000914420 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc488010760 con 0x7fc4901022b0 2024-03-18T00:50:23.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.505+0000 7fc49726a640 1 -- 172.21.15.49:0/4000914420 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc49019fe90 con 0x7fc4901022b0 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.507+0000 7fc4867fc640 1 -- 172.21.15.49:0/4000914420 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc488005020 con 0x7fc4901022b0 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.507+0000 7fc499e4d740 1 Processor -- start 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.507+0000 7fc499e4d740 1 -- start start 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d68760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d6d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 0x5558a5d70330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc499e4d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5558a5ccda40 con 0x5558a5d6dea0 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc499e4d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5558a5c76920 con 0x5558a5d68340 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc499e4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5558a5cc9ae0 con 0x5558a5d68d30 2024-03-18T00:50:23.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc4867fc640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc47c0a28a0 0x7fc47c0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d68760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc495006640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d6d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d68760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:38538/0 (socket says 172.21.15.49:38538) 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 -- 172.21.15.49:0/1155026151 learned_addr learned my addr 172.21.15.49:0/1155026151 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc495006640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d6d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:32888/0 (socket says 172.21.15.49:32888) 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 --2- 172.21.15.49:0/1155026151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 0x5558a5d70330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc495807640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc47c0a28a0 0x7fc47c0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc4867fc640 1 -- 172.21.15.49:0/4000914420 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc4880ceff0 con 0x7fc4901022b0 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 -- 172.21.15.49:0/1155026151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 msgr2=0x5558a5d6d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 --2- 172.21.15.49:0/1155026151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d6d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 -- 172.21.15.49:0/1155026151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 msgr2=0x5558a5d70330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 --2- 172.21.15.49:0/1155026151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 0x5558a5d70330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 -- 172.21.15.49:0/1155026151 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558a5c8e460 con 0x5558a5d68340 2024-03-18T00:50:23.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc494805640 1 --2- 172.21.15.49:0/1155026151 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d68760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc4800058c0 tx=0x7fc480005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b01ceb69b4d7132 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.508+0000 7fc495807640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc47c0a28a0 0x7fc47c0a4d60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc48809b900 tx=0x7fc48801d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc47b7fe640 1 -- 172.21.15.49:0/1155026151 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480004030 con 0x5558a5d68340 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc47b7fe640 1 -- 172.21.15.49:0/1155026151 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc480005500 con 0x5558a5d68340 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc47b7fe640 1 -- 172.21.15.49:0/1155026151 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480004c70 con 0x5558a5d68340 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc499e4d740 1 -- 172.21.15.49:0/1155026151 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 msgr2=0x5558a5d68760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1155026151 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d68760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc4800058c0 tx=0x7fc480005850 comp rx=0 tx=0).stop 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc499e4d740 1 -- 172.21.15.49:0/1155026151 shutdown_connections 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1155026151 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 0x5558a5d70330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1155026151 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d6d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1155026151 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d68760 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc499e4d740 1 -- 172.21.15.49:0/1155026151 >> 172.21.15.49:0/1155026151 conn(0x5558a5d638b0 msgr2=0x5558a5d63cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.509+0000 7fc499e4d740 1 -- 172.21.15.49:0/1155026151 shutdown_connections 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 1 -- 172.21.15.49:0/1155026151 wait complete. 2024-03-18T00:50:23.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 1 -- start start 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 1 Processor -- start 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68340 0x5558a5d72610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d72b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d6dea0 0x5558a5d79b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5558a5d70e50 con 0x5558a5d68340 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5558a5c89d70 con 0x5558a5d6dea0 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5558a5d70fd0 con 0x5558a5d68d30 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc499e4d740 10 client.0 _ll_register_callbacks cb 0x5558a59e1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc494805640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68340 0x5558a5d72610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc494805640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68340 0x5558a5d72610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48204/0 (socket says 172.21.15.49:48204) 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc494805640 1 -- 172.21.15.49:0/1365722877 learned_addr learned my addr 172.21.15.49:0/1365722877 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc495006640 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d72b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.510+0000 7fc494805640 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d6dea0 0x5558a5d79b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc495006640 1 -- 172.21.15.49:0/1365722877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d6dea0 msgr2=0x5558a5d79b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc495006640 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d6dea0 0x5558a5d79b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc495006640 1 -- 172.21.15.49:0/1365722877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68340 msgr2=0x5558a5d72610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc495006640 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68340 0x5558a5d72610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc495006640 1 -- 172.21.15.49:0/1365722877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558a59e2210 con 0x5558a5d68d30 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc494805640 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68340 0x5558a5d72610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc495006640 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d72b50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc48c02c170 tx=0x7fc48c005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84f2ba1e23697a96 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc478ff9640 1 -- 172.21.15.49:0/1365722877 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c005d50 con 0x5558a5d68d30 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc499e4d740 1 -- 172.21.15.49:0/1365722877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc480005500 con 0x5558a5d68d30 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc478ff9640 1 -- 172.21.15.49:0/1365722877 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc48c0048c0 con 0x5558a5d68d30 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc478ff9640 1 -- 172.21.15.49:0/1365722877 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc48c004ac0 con 0x5558a5d68d30 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc499e4d740 10 client.25363 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:23.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.511+0000 7fc499e4d740 1 -- 172.21.15.49:0/1365722877 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5558a5aae620 con 0x5558a5d68d30 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.512+0000 7fc478ff9640 1 -- 172.21.15.49:0/1365722877 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc480005500 con 0x5558a5d68d30 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.512+0000 7fc478ff9640 1 -- 172.21.15.49:0/1365722877 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc48c044070 con 0x5558a5d68d30 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.512+0000 7fc499e4d740 10 client.25363 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.512+0000 7fc499e4d740 10 client.25363 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.512+0000 7fc499e4d740 20 client.25363 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.512+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a5d7e420 0x5558a5d80870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.512+0000 7fc499e4d740 4 client.25363 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.512+0000 7fc499e4d740 1 -- 172.21.15.49:0/1365722877 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558a5cdaf80 con 0x5558a5d7e420 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.512+0000 7fc494805640 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a5d7e420 0x5558a5d80870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc494805640 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a5d7e420 0x5558a5d80870 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef9bd33f802856c5 server_cookie=8d6cc9358c6ae693 in_seq=0 out_seq=0 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc478ff9640 10 client.25363 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc478ff9640 1 -- 172.21.15.49:0/1365722877 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5558a5cdaf80 con 0x5558a5d7e420 2024-03-18T00:50:23.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc478ff9640 10 client.25363 handle_command_reply: tid=0 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc499e4d740 1 client.25363 shutdown 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc499e4d740 1 -- 172.21.15.49:0/1365722877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 msgr2=0x5558a5d72b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d72b50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc48c02c170 tx=0x7fc48c005270 comp rx=0 tx=0).stop 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc499e4d740 1 -- 172.21.15.49:0/1365722877 shutdown_connections 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a5d7e420 0x5558a5d80870 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d6dea0 0x5558a5d79b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d72b50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc478ff9640 0 client.25363 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc499e4d740 1 --2- 172.21.15.49:0/1365722877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68340 0x5558a5d72610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.513+0000 7fc499e4d740 1 -- 172.21.15.49:0/1365722877 >> 172.21.15.49:0/1365722877 conn(0x5558a5d638b0 msgr2=0x5558a5d62bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.514+0000 7fc499e4d740 1 -- 172.21.15.49:0/1365722877 shutdown_connections 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.514+0000 7fc499e4d740 1 -- 172.21.15.49:0/1365722877 wait complete. 2024-03-18T00:50:23.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.514+0000 7fc499e4d740 20 client.25363 trim_cache size 0 max 16384 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc499e4d740 1 Processor -- start 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc499e4d740 1 -- start start 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d62320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d79b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 0x5558a5d62860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc499e4d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901a8d00 con 0x5558a5d6dea0 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc499e4d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901aa840 con 0x5558a5d68340 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc499e4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901a93d0 con 0x5558a5d68d30 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc495006640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d79b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc494805640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 0x5558a5d62860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc495006640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d79b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:32918/0 (socket says 172.21.15.49:32918) 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc495006640 1 -- 172.21.15.49:0/3846462041 learned_addr learned my addr 172.21.15.49:0/3846462041 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc494805640 1 --2- 172.21.15.49:0/3846462041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d62320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc494805640 1 -- 172.21.15.49:0/3846462041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 msgr2=0x5558a5d79b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc494805640 1 --2- 172.21.15.49:0/3846462041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d79b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc494805640 1 -- 172.21.15.49:0/3846462041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 msgr2=0x5558a5d62320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc494805640 1 --2- 172.21.15.49:0/3846462041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d62320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc494805640 1 -- 172.21.15.49:0/3846462041 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558a5d716a0 con 0x5558a5d6dea0 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.533+0000 7fc495006640 1 --2- 172.21.15.49:0/3846462041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d79b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.534+0000 7fc494805640 1 --2- 172.21.15.49:0/3846462041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 0x5558a5d62860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x5558a5d810f0 tx=0x7fc480024040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd4e24ce557828b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.534+0000 7fc47affd640 1 -- 172.21.15.49:0/3846462041 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480004690 con 0x5558a5d6dea0 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.534+0000 7fc47affd640 1 -- 172.21.15.49:0/3846462041 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc480030bc0 con 0x5558a5d6dea0 2024-03-18T00:50:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.534+0000 7fc47affd640 1 -- 172.21.15.49:0/3846462041 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc480011420 con 0x5558a5d6dea0 2024-03-18T00:50:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.535+0000 7fc499e4d740 1 -- 172.21.15.49:0/3846462041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 msgr2=0x5558a5d62860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.535+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3846462041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 0x5558a5d62860 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x5558a5d810f0 tx=0x7fc480024040 comp rx=0 tx=0).stop 2024-03-18T00:50:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.535+0000 7fc499e4d740 1 -- 172.21.15.49:0/3846462041 shutdown_connections 2024-03-18T00:50:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.535+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3846462041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d6dea0 0x5558a5d62860 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.535+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3846462041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d68d30 0x5558a5d79b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.535+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3846462041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d62320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.535+0000 7fc499e4d740 1 -- 172.21.15.49:0/3846462041 >> 172.21.15.49:0/3846462041 conn(0x5558a5d638b0 msgr2=0x5558a5d63c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.535+0000 7fc499e4d740 1 -- 172.21.15.49:0/3846462041 shutdown_connections 2024-03-18T00:50:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.535+0000 7fc499e4d740 1 -- 172.21.15.49:0/3846462041 wait complete. 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc499e4d740 1 -- start start 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc499e4d740 1 Processor -- start 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d77000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68d30 0x5558a5d77540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc499e4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d6dea0 0x5558a5d78550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc499e4d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901aa170 con 0x5558a5d68d30 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc499e4d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901a9aa0 con 0x5558a5d68340 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc499e4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc4901a6ce0 con 0x5558a5d6dea0 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc499e4d740 10 client.0 _ll_register_callbacks cb 0x7fc48c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc495006640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68d30 0x5558a5d77540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc494805640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d77000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc495006640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68d30 0x5558a5d77540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48220/0 (socket says 172.21.15.49:48220) 2024-03-18T00:50:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.536+0000 7fc495006640 1 -- 172.21.15.49:0/3674526968 learned_addr learned my addr 172.21.15.49:0/3674526968 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.537+0000 7fc494805640 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d6dea0 0x5558a5d78550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.537+0000 7fc495006640 1 -- 172.21.15.49:0/3674526968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d6dea0 msgr2=0x5558a5d78550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.537+0000 7fc495006640 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d6dea0 0x5558a5d78550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.537+0000 7fc495006640 1 -- 172.21.15.49:0/3674526968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 msgr2=0x5558a5d77000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.537+0000 7fc495006640 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d77000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.537+0000 7fc495006640 1 -- 172.21.15.49:0/3674526968 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558a5d78b40 con 0x5558a5d68d30 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.537+0000 7fc494805640 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d77000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.537+0000 7fc495006640 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68d30 0x5558a5d77540 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc48c000c00 tx=0x7fc48c005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cf0836950cec4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.537+0000 7fc478ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc478ff9640 1 -- 172.21.15.49:0/3674526968 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c005d50 con 0x5558a5d68d30 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc478ff9640 1 -- 172.21.15.49:0/3674526968 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc48c0046c0 con 0x5558a5d68d30 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc499e4d740 1 -- 172.21.15.49:0/3674526968 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc480030bc0 con 0x5558a5d68d30 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc478ff9640 1 -- 172.21.15.49:0/3674526968 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc48c003ea0 con 0x5558a5d68d30 2024-03-18T00:50:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc499e4d740 10 client.15501 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc499e4d740 1 -- 172.21.15.49:0/3674526968 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5558a5de4800 con 0x5558a5d68d30 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc478ff9640 1 -- 172.21.15.49:0/3674526968 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc480030bc0 con 0x5558a5d68d30 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc478ff9640 1 -- 172.21.15.49:0/3674526968 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc48c044070 con 0x5558a5d68d30 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc499e4d740 10 client.15501 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc499e4d740 10 client.15501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc499e4d740 20 client.15501 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a5d7e420 0x5558a5de70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc499e4d740 4 client.15501 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.538+0000 7fc499e4d740 1 -- 172.21.15.49:0/3674526968 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5558a5cd52d0 con 0x5558a5d7e420 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.539+0000 7fc494805640 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a5d7e420 0x5558a5de70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.539+0000 7fc494805640 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a5d7e420 0x5558a5de70b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63ec2339384a0837 server_cookie=9b0464b98ed274ac in_seq=0 out_seq=0 2024-03-18T00:50:23.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.539+0000 7fc478ff9640 10 client.15501 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc478ff9640 1 -- 172.21.15.49:0/3674526968 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5558a5cd52d0 con 0x5558a5d7e420 2024-03-18T00:50:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc478ff9640 10 client.15501 handle_command_reply: tid=0 2024-03-18T00:50:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc499e4d740 1 client.15501 shutdown 2024-03-18T00:50:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc499e4d740 1 -- 172.21.15.49:0/3674526968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68d30 msgr2=0x5558a5d77540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68d30 0x5558a5d77540 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc48c000c00 tx=0x7fc48c005bc0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc499e4d740 1 -- 172.21.15.49:0/3674526968 shutdown_connections 2024-03-18T00:50:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a5d7e420 0x5558a5de70b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a5d6dea0 0x5558a5d78550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a5d68d30 0x5558a5d77540 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc499e4d740 1 --2- 172.21.15.49:0/3674526968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a5d68340 0x5558a5d77000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.551+0000 7fc499e4d740 1 -- 172.21.15.49:0/3674526968 >> 172.21.15.49:0/3674526968 conn(0x5558a5d638b0 msgr2=0x5558a5d6d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.552+0000 7fc478ff9640 0 client.15501 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.552+0000 7fc499e4d740 1 -- 172.21.15.49:0/3674526968 shutdown_connections 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.552+0000 7fc499e4d740 1 -- 172.21.15.49:0/3674526968 wait complete. 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.552+0000 7fc499e4d740 20 client.15501 trim_cache size 0 max 16384 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:50:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:50:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.552+0000 7fc49726a640 1 -- 172.21.15.49:0/4000914420 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc47c0a28a0 msgr2=0x7fc47c0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.552+0000 7fc49726a640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc47c0a28a0 0x7fc47c0a4d60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc48809b900 tx=0x7fc48801d000 comp rx=0 tx=0).stop 2024-03-18T00:50:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.552+0000 7fc49726a640 1 -- 172.21.15.49:0/4000914420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4901022b0 msgr2=0x7fc49019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.552+0000 7fc49726a640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4901022b0 0x7fc49019f030 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc48800eb10 tx=0x7fc48800c2d0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.553+0000 7fc49726a640 1 -- 172.21.15.49:0/4000914420 shutdown_connections 2024-03-18T00:50:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.553+0000 7fc49726a640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc47c0a28a0 0x7fc47c0a4d60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.553+0000 7fc49726a640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4901022b0 0x7fc49019f030 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.553+0000 7fc49726a640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4901018f0 0x7fc490197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.553+0000 7fc49726a640 1 --2- 172.21.15.49:0/4000914420 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc490100700 0x7fc490197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.553+0000 7fc49726a640 1 -- 172.21.15.49:0/4000914420 >> 172.21.15.49:0/4000914420 conn(0x7fc4900fbc70 msgr2=0x7fc4900fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.553+0000 7fc49726a640 1 -- 172.21.15.49:0/4000914420 shutdown_connections 2024-03-18T00:50:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:23.553+0000 7fc49726a640 1 -- 172.21.15.49:0/4000914420 wait complete. 2024-03-18T00:50:24.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:50:24.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:50:24.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:23 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:50:24.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:50:24.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:23 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:24.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:50:24.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:50:24.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:23 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:50:25.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:24 smithi049 ceph-mon[32130]: pgmap v359: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-18T00:50:25.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:24 smithi107 ceph-mon[36270]: pgmap v359: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-18T00:50:25.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:24 smithi106 ceph-mon[35899]: pgmap v359: 97 pgs: 97 active+clean; 654 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-18T00:50:27.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:26 smithi049 ceph-mon[32130]: pgmap v360: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-18T00:50:27.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:26 smithi107 ceph-mon[36270]: pgmap v360: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-18T00:50:27.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:26 smithi106 ceph-mon[35899]: pgmap v360: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 KiB/s wr, 12 op/s 2024-03-18T00:50:29.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:28 smithi049 ceph-mon[32130]: pgmap v361: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-18T00:50:29.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:28 smithi107 ceph-mon[36270]: pgmap v361: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-18T00:50:29.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:28 smithi106 ceph-mon[35899]: pgmap v361: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-18T00:50:31.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:30 smithi049 ceph-mon[32130]: pgmap v362: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-18T00:50:31.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:30 smithi107 ceph-mon[36270]: pgmap v362: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-18T00:50:31.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:30 smithi106 ceph-mon[35899]: pgmap v362: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-18T00:50:33.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:32 smithi049 ceph-mon[32130]: pgmap v363: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-18T00:50:33.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:32 smithi107 ceph-mon[36270]: pgmap v363: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-18T00:50:33.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:32 smithi106 ceph-mon[35899]: pgmap v363: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-18T00:50:35.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:34 smithi049 ceph-mon[32130]: pgmap v364: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-18T00:50:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:34 smithi107 ceph-mon[36270]: pgmap v364: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-18T00:50:35.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:34 smithi106 ceph-mon[35899]: pgmap v364: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-18T00:50:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.692+0000 7fc4c39cd640 1 -- 172.21.15.49:0/268628351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1022b0 msgr2=0x7fc4bc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.692+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/268628351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1022b0 0x7fc4bc108b40 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4009ca0 tx=0x7fc4b402c100 comp rx=0 tx=0).stop 2024-03-18T00:50:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.692+0000 7fc4c39cd640 1 -- 172.21.15.49:0/268628351 shutdown_connections 2024-03-18T00:50:36.698 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.692+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/268628351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc1018f0 0x7fc4bc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.692+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/268628351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1022b0 0x7fc4bc108b40 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.692+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/268628351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4bc100700 0x7fc4bc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.692+0000 7fc4c39cd640 1 -- 172.21.15.49:0/268628351 >> 172.21.15.49:0/268628351 conn(0x7fc4bc0fbc70 msgr2=0x7fc4bc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.692+0000 7fc4c39cd640 1 -- 172.21.15.49:0/268628351 shutdown_connections 2024-03-18T00:50:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.692+0000 7fc4c39cd640 1 -- 172.21.15.49:0/268628351 wait complete. 2024-03-18T00:50:36.699 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c39cd640 1 Processor -- start 2024-03-18T00:50:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c39cd640 1 -- start start 2024-03-18T00:50:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c39cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4bc100700 0x7fc4bc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c39cd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1018f0 0x7fc4bc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c39cd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc1022b0 0x7fc4bc19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c39cd640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc10c360 con 0x7fc4bc1018f0 2024-03-18T00:50:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c39cd640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc10c1e0 con 0x7fc4bc1022b0 2024-03-18T00:50:36.700 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c39cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc10c4e0 con 0x7fc4bc100700 2024-03-18T00:50:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c1769640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4bc100700 0x7fc4bc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c1769640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4bc100700 0x7fc4bc197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56168/0 (socket says 172.21.15.49:56168) 2024-03-18T00:50:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c1769640 1 -- 172.21.15.49:0/4144748383 learned_addr learned my addr 172.21.15.49:0/4144748383 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c1f6a640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc1022b0 0x7fc4bc19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.693+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1018f0 0x7fc4bc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4c1769640 1 -- 172.21.15.49:0/4144748383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc1022b0 msgr2=0x7fc4bc19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:36.701 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4c1769640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc1022b0 0x7fc4bc19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4c1769640 1 -- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1018f0 msgr2=0x7fc4bc198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4c1769640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1018f0 0x7fc4bc198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4c1769640 1 -- 172.21.15.49:0/4144748383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4bc19f6e0 con 0x7fc4bc100700 2024-03-18T00:50:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4c1769640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4bc100700 0x7fc4bc197ac0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800e9c0 tx=0x7fc4b800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4b27fc640 1 -- 172.21.15.49:0/4144748383 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b800cde0 con 0x7fc4bc100700 2024-03-18T00:50:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4c39cd640 1 -- 172.21.15.49:0/4144748383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4bc19f9d0 con 0x7fc4bc100700 2024-03-18T00:50:36.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4b27fc640 1 -- 172.21.15.49:0/4144748383 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4b8010910 con 0x7fc4bc100700 2024-03-18T00:50:36.703 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4b27fc640 1 -- 172.21.15.49:0/4144748383 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8018a10 con 0x7fc4bc100700 2024-03-18T00:50:36.703 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.694+0000 7fc4c39cd640 1 -- 172.21.15.49:0/4144748383 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4bc19ff70 con 0x7fc4bc100700 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4b27fc640 1 -- 172.21.15.49:0/4144748383 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc4b801f020 con 0x7fc4bc100700 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c65b0740 1 Processor -- start 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c65b0740 1 -- start start 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63d7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63dc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 0x5651c63df330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5651c633ca40 con 0x5651c63d7340 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5651c62e5920 con 0x5651c63d7d30 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5651c6338ae0 con 0x5651c63dcea0 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c1769640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63dc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c1769640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63dc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:33114/0 (socket says 172.21.15.49:33114) 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c1769640 1 -- 172.21.15.49:0/487760321 learned_addr learned my addr 172.21.15.49:0/487760321 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:36.711 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 0x5651c63df330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.698+0000 7fc4c1f6a640 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63d7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4b27fc640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc4a80a28a0 0x7fc4a80a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4c0f68640 1 -- 172.21.15.49:0/487760321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 msgr2=0x5651c63dc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63dc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4c0f68640 1 -- 172.21.15.49:0/487760321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 msgr2=0x5651c63d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63d7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4c0f68640 1 -- 172.21.15.49:0/487760321 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c62fd460 con 0x5651c63dcea0 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4c1769640 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63dc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4c1f6a640 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63d7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4b27fc640 1 -- 172.21.15.49:0/4144748383 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc4b80cec00 con 0x7fc4bc100700 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4c1f6a640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc4a80a28a0 0x7fc4a80a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 0x5651c63df330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009bf0 tx=0x7fc4ac011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c0fe4949e926ba3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4a77fe640 1 -- 172.21.15.49:0/487760321 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac01e070 con 0x5651c63dcea0 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4a77fe640 1 -- 172.21.15.49:0/487760321 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4ac009da0 con 0x5651c63dcea0 2024-03-18T00:50:36.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.699+0000 7fc4a77fe640 1 -- 172.21.15.49:0/487760321 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac018840 con 0x5651c63dcea0 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.700+0000 7fc4c65b0740 1 -- 172.21.15.49:0/487760321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 msgr2=0x5651c63df330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.700+0000 7fc4c1f6a640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc4a80a28a0 0x7fc4a80a4d60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005e30 tx=0x7fc4b402e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.700+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 0x5651c63df330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009bf0 tx=0x7fc4ac011810 comp rx=0 tx=0).stop 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.700+0000 7fc4c65b0740 1 -- 172.21.15.49:0/487760321 shutdown_connections 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.700+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63dc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.700+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 0x5651c63df330 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.700+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/487760321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63d7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.700+0000 7fc4c65b0740 1 -- 172.21.15.49:0/487760321 >> 172.21.15.49:0/487760321 conn(0x5651c63d28b0 msgr2=0x5651c63d2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.701+0000 7fc4c65b0740 1 -- 172.21.15.49:0/487760321 shutdown_connections 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.701+0000 7fc4c65b0740 1 -- 172.21.15.49:0/487760321 wait complete. 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.701+0000 7fc4c65b0740 1 -- start start 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.701+0000 7fc4c65b0740 1 Processor -- start 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.701+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63e16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.701+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63e1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.701+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 0x5651c63e8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5651c63dfe50 con 0x5651c63d7340 2024-03-18T00:50:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5651c62f8d70 con 0x5651c63d7d30 2024-03-18T00:50:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5651c63dffd0 con 0x5651c63dcea0 2024-03-18T00:50:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c0f68640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63e1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c0f68640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63e1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:33124/0 (socket says 172.21.15.49:33124) 2024-03-18T00:50:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c0f68640 1 -- 172.21.15.49:0/2558725250 learned_addr learned my addr 172.21.15.49:0/2558725250 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c1769640 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63e16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c65b0740 10 client.0 _ll_register_callbacks cb 0x5651c6050a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:36.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c1f6a640 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 0x5651c63e8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c0f68640 1 -- 172.21.15.49:0/2558725250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 msgr2=0x5651c63e8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 0x5651c63e8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c0f68640 1 -- 172.21.15.49:0/2558725250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 msgr2=0x5651c63e16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63e16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c0f68640 1 -- 172.21.15.49:0/2558725250 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651c6051210 con 0x5651c63d7d30 2024-03-18T00:50:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c1769640 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63e16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63e1bf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0093a0 tx=0x7fc4ac004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf4f00df5a4c043e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:50:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.702+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/2558725250 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac01e070 con 0x5651c63d7d30 2024-03-18T00:50:36.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.703+0000 7fc4c65b0740 1 -- 172.21.15.49:0/2558725250 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4ac009da0 con 0x5651c63d7d30 2024-03-18T00:50:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.703+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/2558725250 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc4ac019a00 con 0x5651c63d7d30 2024-03-18T00:50:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.703+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/2558725250 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc4ac019c00 con 0x5651c63d7d30 2024-03-18T00:50:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.703+0000 7fc4c65b0740 10 client.25352 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.703+0000 7fc4c65b0740 1 -- 172.21.15.49:0/2558725250 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5651c611d620 con 0x5651c63d7d30 2024-03-18T00:50:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.703+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/2558725250 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4ac009da0 con 0x5651c63d7d30 2024-03-18T00:50:36.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.704+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/2558725250 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc4ac004a90 con 0x5651c63d7d30 2024-03-18T00:50:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.704+0000 7fc4c65b0740 10 client.25352 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.704+0000 7fc4c65b0740 10 client.25352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.704+0000 7fc4c65b0740 20 client.25352 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.705+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5651c63ed5d0 0x5651c63efa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.705+0000 7fc4c65b0740 4 client.25352 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:36.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.705+0000 7fc4c65b0740 1 -- 172.21.15.49:0/2558725250 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651c6349f80 con 0x5651c63ed5d0 2024-03-18T00:50:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.705+0000 7fc4c1769640 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5651c63ed5d0 0x5651c63efa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.705+0000 7fc4a4ff9640 10 client.25352 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.705+0000 7fc4c1769640 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5651c63ed5d0 0x5651c63efa70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bcdf52bedfec0e4 server_cookie=32a51d5f21c2d1b5 in_seq=0 out_seq=0 2024-03-18T00:50:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.706+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/2558725250 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5651c6349f80 con 0x5651c63ed5d0 2024-03-18T00:50:36.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.706+0000 7fc4a4ff9640 10 client.25352 handle_command_reply: tid=0 2024-03-18T00:50:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.706+0000 7fc4c65b0740 1 client.25352 shutdown 2024-03-18T00:50:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.707+0000 7fc4c65b0740 1 -- 172.21.15.49:0/2558725250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 msgr2=0x5651c63e1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.707+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63e1bf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac0093a0 tx=0x7fc4ac004610 comp rx=0 tx=0).stop 2024-03-18T00:50:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4c65b0740 1 -- 172.21.15.49:0/2558725250 shutdown_connections 2024-03-18T00:50:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5651c63ed5d0 0x5651c63efa70 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7d30 0x5651c63e1bf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63dcea0 0x5651c63e8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/2558725250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63d7340 0x5651c63e16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4c65b0740 1 -- 172.21.15.49:0/2558725250 >> 172.21.15.49:0/2558725250 conn(0x5651c63d28b0 msgr2=0x5651c63d1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4a4ff9640 0 client.25352 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4c65b0740 1 -- 172.21.15.49:0/2558725250 shutdown_connections 2024-03-18T00:50:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4c65b0740 1 -- 172.21.15.49:0/2558725250 wait complete. 2024-03-18T00:50:36.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.708+0000 7fc4c65b0740 20 client.25352 trim_cache size 0 max 16384 2024-03-18T00:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.737+0000 7fc4c65b0740 1 Processor -- start 2024-03-18T00:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.737+0000 7fc4c65b0740 1 -- start start 2024-03-18T00:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 0x5651c63e0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1a8bd0 con 0x5651c63dcea0 2024-03-18T00:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c0f68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1aa710 con 0x5651c63d7340 2024-03-18T00:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c0f68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56200/0 (socket says 172.21.15.49:56200) 2024-03-18T00:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c0f68640 1 -- 172.21.15.49:0/3437556781 learned_addr learned my addr 172.21.15.49:0/3437556781 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c1f6a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c1769640 1 --2- 172.21.15.49:0/3437556781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 0x5651c63e0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3437556781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1a92a0 con 0x5651c63d7d30 2024-03-18T00:50:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c0f68640 1 -- 172.21.15.49:0/3437556781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 msgr2=0x5651c63e0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/3437556781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 0x5651c63e0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c0f68640 1 -- 172.21.15.49:0/3437556781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 msgr2=0x5651c63e16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/3437556781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c0f68640 1 -- 172.21.15.49:0/3437556781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651c63dc360 con 0x5651c63d7d30 2024-03-18T00:50:36.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.738+0000 7fc4c1f6a640 1 --2- 172.21.15.49:0/3437556781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.739+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/3437556781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e8b80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009b40 tx=0x7fc4ac011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d3aa9bf97d2c18c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.739+0000 7fc4a6ffd640 1 -- 172.21.15.49:0/3437556781 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac01e040 con 0x5651c63d7d30 2024-03-18T00:50:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.739+0000 7fc4a6ffd640 1 -- 172.21.15.49:0/3437556781 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4ac028070 con 0x5651c63d7d30 2024-03-18T00:50:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.739+0000 7fc4a6ffd640 1 -- 172.21.15.49:0/3437556781 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac003df0 con 0x5651c63d7d30 2024-03-18T00:50:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3437556781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 msgr2=0x5651c63e8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3437556781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e8b80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac009b40 tx=0x7fc4ac011810 comp rx=0 tx=0).stop 2024-03-18T00:50:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3437556781 shutdown_connections 2024-03-18T00:50:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3437556781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3437556781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e8b80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3437556781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 0x5651c63e0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3437556781 >> 172.21.15.49:0/3437556781 conn(0x5651c63d28b0 msgr2=0x5651c63d2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3437556781 shutdown_connections 2024-03-18T00:50:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3437556781 wait complete. 2024-03-18T00:50:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.740+0000 7fc4c65b0740 1 -- start start 2024-03-18T00:50:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c65b0740 1 Processor -- start 2024-03-18T00:50:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 0x5651c63e8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c65b0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1aa040 con 0x5651c63dcea0 2024-03-18T00:50:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1a9970 con 0x5651c63d7340 2024-03-18T00:50:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c65b0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc4bc1a6c60 con 0x5651c63d7d30 2024-03-18T00:50:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c65b0740 10 client.0 _ll_register_callbacks cb 0x7fc4ac01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c1f6a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c0f68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c1769640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 0x5651c63e8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c1f6a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47176/0 (socket says 172.21.15.49:47176) 2024-03-18T00:50:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c0f68640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56206/0 (socket says 172.21.15.49:56206) 2024-03-18T00:50:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c1f6a640 1 -- 172.21.15.49:0/3205660009 learned_addr learned my addr 172.21.15.49:0/3205660009 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c1769640 1 -- 172.21.15.49:0/3205660009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 msgr2=0x5651c63e5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c1769640 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c1769640 1 -- 172.21.15.49:0/3205660009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 msgr2=0x5651c63e56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c1769640 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.741+0000 7fc4c1769640 1 -- 172.21.15.49:0/3205660009 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651c63e8c80 con 0x5651c63d7340 2024-03-18T00:50:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c1769640 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 0x5651c63e8650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc4b809b510 tx=0x7fc4b8086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d60b0a1c8560edc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:50:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/3205660009 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b809b350 con 0x5651c63d7340 2024-03-18T00:50:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/3205660009 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc4b8018bb0 con 0x5651c63d7340 2024-03-18T00:50:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3205660009 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4ac028070 con 0x5651c63d7340 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/3205660009 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc4b809aee0 con 0x5651c63d7340 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c65b0740 10 client.25358 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3205660009 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5651c6456d10 con 0x5651c63d7340 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/3205660009 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4b809bc80 con 0x5651c63d7340 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/3205660009 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fc4b80d7040 con 0x5651c63d7340 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c65b0740 10 client.25358 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c65b0740 10 client.25358 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c65b0740 20 client.25358 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5651c6457640 0x5651c6459a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c65b0740 4 client.25358 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:50:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.742+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3205660009 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fc4ac028070 con 0x5651c6457640 2024-03-18T00:50:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.743+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5651c6457640 0x5651c6459a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.743+0000 7fc4a4ff9640 10 client.25358 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.743+0000 7fc4c0f68640 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5651c6457640 0x5651c6459a60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ccccc8f9fe33e02 server_cookie=c396e8d5c2c7e754 in_seq=0 out_seq=0 2024-03-18T00:50:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4a4ff9640 1 -- 172.21.15.49:0/3205660009 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fc4ac028070 con 0x5651c6457640 2024-03-18T00:50:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4a4ff9640 10 client.25358 handle_command_reply: tid=0 2024-03-18T00:50:36.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 client.25358 shutdown 2024-03-18T00:50:36.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3205660009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 msgr2=0x5651c63e8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:36.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 0x5651c63e8650 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc4b809b510 tx=0x7fc4b8086040 comp rx=0 tx=0).stop 2024-03-18T00:50:36.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3205660009 shutdown_connections 2024-03-18T00:50:36.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5651c6457640 0x5651c6459a60 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5651c63dcea0 0x5651c63e56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5651c63d7d30 0x5651c63e5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4a4ff9640 0 client.25358 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:36.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 --2- 172.21.15.49:0/3205660009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5651c63d7340 0x5651c63e8650 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3205660009 >> 172.21.15.49:0/3205660009 conn(0x5651c63d28b0 msgr2=0x5651c63d9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:36.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3205660009 shutdown_connections 2024-03-18T00:50:36.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 1 -- 172.21.15.49:0/3205660009 wait complete. 2024-03-18T00:50:36.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.746+0000 7fc4c65b0740 20 client.25358 trim_cache size 0 max 16384 2024-03-18T00:50:36.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:50:36.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:50:36.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:50:36.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:50:36.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 -- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc4a80a28a0 msgr2=0x7fc4a80a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:36.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc4a80a28a0 0x7fc4a80a4d60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4005e30 tx=0x7fc4b402e040 comp rx=0 tx=0).stop 2024-03-18T00:50:36.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 -- 172.21.15.49:0/4144748383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4bc100700 msgr2=0x7fc4bc197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:36.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4bc100700 0x7fc4bc197ac0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800e9c0 tx=0x7fc4b800ee80 comp rx=0 tx=0).stop 2024-03-18T00:50:36.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 -- 172.21.15.49:0/4144748383 shutdown_connections 2024-03-18T00:50:36.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc4a80a28a0 0x7fc4a80a4d60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc4bc1022b0 0x7fc4bc19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc4bc1018f0 0x7fc4bc198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 --2- 172.21.15.49:0/4144748383 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4bc100700 0x7fc4bc197ac0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:36.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 -- 172.21.15.49:0/4144748383 >> 172.21.15.49:0/4144748383 conn(0x7fc4bc0fbc70 msgr2=0x7fc4bc0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:36.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 -- 172.21.15.49:0/4144748383 shutdown_connections 2024-03-18T00:50:36.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:36.747+0000 7fc4c39cd640 1 -- 172.21.15.49:0/4144748383 wait complete. 2024-03-18T00:50:37.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:36 smithi049 ceph-mon[32130]: pgmap v365: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-18T00:50:37.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:36 smithi107 ceph-mon[36270]: pgmap v365: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-18T00:50:37.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:36 smithi106 ceph-mon[35899]: pgmap v365: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-18T00:50:39.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:38 smithi049 ceph-mon[32130]: pgmap v366: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-18T00:50:39.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:38 smithi107 ceph-mon[36270]: pgmap v366: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-18T00:50:39.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:38 smithi106 ceph-mon[35899]: pgmap v366: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-18T00:50:40.718 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:50:40.718 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:50:40.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 -- 172.21.15.49:0/3478272085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f81022c0 msgr2=0x7f08f81046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:40.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 --2- 172.21.15.49:0/3478272085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f81022c0 0x7f08f81046b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009900 tx=0x7f08ec02bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:40.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 -- 172.21.15.49:0/3478272085 shutdown_connections 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 --2- 172.21.15.49:0/3478272085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08f8107520 0x7f08f8109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 --2- 172.21.15.49:0/3478272085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08f8104bf0 0x7f08f8106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 --2- 172.21.15.49:0/3478272085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f81022c0 0x7f08f81046b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 -- 172.21.15.49:0/3478272085 >> 172.21.15.49:0/3478272085 conn(0x7f08f80fbc70 msgr2=0x7f08f80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 -- 172.21.15.49:0/3478272085 shutdown_connections 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 -- 172.21.15.49:0/3478272085 wait complete. 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 Processor -- start 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.838+0000 7f08fee19640 1 -- start start 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fee19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f81022c0 0x7f08f8197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fee19640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08f8104bf0 0x7f08f81980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fee19640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08f8107520 0x7f08f819f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fee19640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f08f810d480 con 0x7f08f8104bf0 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fee19640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f08f810d300 con 0x7f08f8107520 2024-03-18T00:50:40.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fee19640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08f810d600 con 0x7f08f81022c0 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fcbb5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f81022c0 0x7f08f8197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08f8104bf0 0x7f08f81980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fcbb5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f81022c0 0x7f08f8197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59804/0 (socket says 172.21.15.49:59804) 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08f8104bf0 0x7f08f81980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46312/0 (socket says 172.21.15.49:46312) 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fd3b6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08f8107520 0x7f08f819f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08f7fff640 1 -- 172.21.15.49:0/813316158 learned_addr learned my addr 172.21.15.49:0/813316158 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fd3b6640 1 -- 172.21.15.49:0/813316158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f81022c0 msgr2=0x7f08f8197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fd3b6640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f81022c0 0x7f08f8197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fd3b6640 1 -- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08f8104bf0 msgr2=0x7f08f81980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fd3b6640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08f8104bf0 0x7f08f81980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fd3b6640 1 -- 172.21.15.49:0/813316158 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f819f7a0 con 0x7f08f8107520 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08f7fff640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08f8104bf0 0x7f08f81980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:40.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.839+0000 7f08fd3b6640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08f8107520 0x7f08f819f130 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009fd0 tx=0x7f08f000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:40.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.840+0000 7f08f5ffb640 1 -- 172.21.15.49:0/813316158 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f000ce20 con 0x7f08f8107520 2024-03-18T00:50:40.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.840+0000 7f08fee19640 1 -- 172.21.15.49:0/813316158 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f819fa90 con 0x7f08f8107520 2024-03-18T00:50:40.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.840+0000 7f08f5ffb640 1 -- 172.21.15.49:0/813316158 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f0005020 con 0x7f08f8107520 2024-03-18T00:50:40.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.840+0000 7f08f5ffb640 1 -- 172.21.15.49:0/813316158 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f0010720 con 0x7f08f8107520 2024-03-18T00:50:40.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.840+0000 7f08fee19640 1 -- 172.21.15.49:0/813316158 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f819ffe0 con 0x7f08f8107520 2024-03-18T00:50:40.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.842+0000 7f0901a07740 1 Processor -- start 2024-03-18T00:50:40.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f08f5ffb640 1 -- 172.21.15.49:0/813316158 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f08f0005540 con 0x7f08f8107520 2024-03-18T00:50:40.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f0901a07740 1 -- start start 2024-03-18T00:50:40.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f0901a07740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70dd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f0901a07740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70e2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f0901a07740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70e2cd0 0x5609c70e5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f0901a07740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5609c7042500 con 0x5609c70dd170 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f0901a07740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5609c6feabe0 con 0x5609c70e2cd0 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f0901a07740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5609c703e5a0 con 0x5609c70ddb60 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70dd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70dd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46328/0 (socket says 172.21.15.49:46328) 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f08f7fff640 1 -- 172.21.15.49:0/375590915 learned_addr learned my addr 172.21.15.49:0/375590915 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f08fcbb5640 1 --2- 172.21.15.49:0/375590915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70e2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.843+0000 7f08f5ffb640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e80a2880 0x7f08e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08f7fff640 1 -- 172.21.15.49:0/375590915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 msgr2=0x5609c70e2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:40.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08f7fff640 1 --2- 172.21.15.49:0/375590915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70e2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08f7fff640 1 -- 172.21.15.49:0/375590915 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70e2cd0 msgr2=0x5609c70e5160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08f7fff640 1 --2- 172.21.15.49:0/375590915 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70e2cd0 0x5609c70e5160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08f7fff640 1 -- 172.21.15.49:0/375590915 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609c6ffe4b0 con 0x5609c70dd170 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08f5ffb640 1 -- 172.21.15.49:0/813316158 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f08f00cf100 con 0x7f08f8107520 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08fd3b6640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e80a2880 0x7f08e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08f7fff640 1 --2- 172.21.15.49:0/375590915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70dd590 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f08e4011590 tx=0x7f08e4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723c364f55341ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08e2ffd640 1 -- 172.21.15.49:0/375590915 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e401e070 con 0x5609c70dd170 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08fd3b6640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e80a2880 0x7f08e80a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f08f0005ba0 tx=0x7f08f0005940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08e2ffd640 1 -- 172.21.15.49:0/375590915 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e400fac0 con 0x5609c70dd170 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.844+0000 7f08e2ffd640 1 -- 172.21.15.49:0/375590915 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4019bf0 con 0x5609c70dd170 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.845+0000 7f0901a07740 1 -- 172.21.15.49:0/375590915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 msgr2=0x5609c70dd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.845+0000 7f0901a07740 1 --2- 172.21.15.49:0/375590915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70dd590 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f08e4011590 tx=0x7f08e4009f90 comp rx=0 tx=0).stop 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.845+0000 7f0901a07740 1 -- 172.21.15.49:0/375590915 shutdown_connections 2024-03-18T00:50:40.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.845+0000 7f0901a07740 1 --2- 172.21.15.49:0/375590915 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70e2cd0 0x5609c70e5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.845+0000 7f0901a07740 1 --2- 172.21.15.49:0/375590915 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70e2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.845+0000 7f0901a07740 1 --2- 172.21.15.49:0/375590915 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70dd590 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.845+0000 7f0901a07740 1 -- 172.21.15.49:0/375590915 >> 172.21.15.49:0/375590915 conn(0x5609c70d86e0 msgr2=0x5609c70d8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.845+0000 7f0901a07740 1 -- 172.21.15.49:0/375590915 shutdown_connections 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.845+0000 7f0901a07740 1 -- 172.21.15.49:0/375590915 wait complete. 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f0901a07740 1 -- start start 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f0901a07740 1 Processor -- start 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f0901a07740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70e76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f0901a07740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70e7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f0901a07740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70e2cd0 0x5609c70eec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f0901a07740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5609c70e5c80 con 0x5609c70dd170 2024-03-18T00:50:40.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f0901a07740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5609c6ffe160 con 0x5609c70e2cd0 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f0901a07740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5609c70e5e00 con 0x5609c70ddb60 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f0901a07740 10 client.0 _ll_register_callbacks cb 0x5609c6fe2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70e76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70e76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46332/0 (socket says 172.21.15.49:46332) 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f08fcbb5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70e7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.846+0000 7f08f7fff640 1 -- 172.21.15.49:0/1784326339 learned_addr learned my addr 172.21.15.49:0/1784326339 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08fcbb5640 1 -- 172.21.15.49:0/1784326339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70e2cd0 msgr2=0x5609c70eec40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08fcbb5640 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70e2cd0 0x5609c70eec40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08fcbb5640 1 -- 172.21.15.49:0/1784326339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 msgr2=0x5609c70e76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08fcbb5640 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70e76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08fcbb5640 1 -- 172.21.15.49:0/1784326339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609c6e207d0 con 0x5609c70ddb60 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08f7fff640 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70e76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08fcbb5640 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70e7c00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009230 tx=0x7f08ec004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32627166707cf21c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:40.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08d7fff640 1 -- 172.21.15.49:0/1784326339 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec005d50 con 0x5609c70ddb60 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f0901a07740 1 -- 172.21.15.49:0/1784326339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08e400fac0 con 0x5609c70ddb60 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08d7fff640 1 -- 172.21.15.49:0/1784326339 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f08ec005230 con 0x5609c70ddb60 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f08d7fff640 1 -- 172.21.15.49:0/1784326339 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f08ec005430 con 0x5609c70ddb60 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f0901a07740 10 client.25399 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.847+0000 7f0901a07740 1 -- 172.21.15.49:0/1784326339 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5609c6d563f0 con 0x5609c70ddb60 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f08d7fff640 1 -- 172.21.15.49:0/1784326339 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e400fac0 con 0x5609c70ddb60 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f08d7fff640 1 -- 172.21.15.49:0/1784326339 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f08ec003970 con 0x5609c70ddb60 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f0901a07740 10 client.25399 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f0901a07740 10 client.25399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f0901a07740 20 client.25399 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:40.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f0901a07740 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5609c70f75f0 0x5609c70f9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f0901a07740 4 client.25399 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:40.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f0901a07740 1 -- 172.21.15.49:0/1784326339 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609c704fa40 con 0x5609c70f75f0 2024-03-18T00:50:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f08f7fff640 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5609c70f75f0 0x5609c70f9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f08d7fff640 10 client.25399 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.848+0000 7f08f7fff640 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5609c70f75f0 0x5609c70f9a50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=268da47ea01ef2e9 server_cookie=f71d144fcfd09ebd in_seq=0 out_seq=0 2024-03-18T00:50:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f08d7fff640 1 -- 172.21.15.49:0/1784326339 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5609c704fa40 con 0x5609c70f75f0 2024-03-18T00:50:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f08d7fff640 10 client.25399 handle_command_reply: tid=0 2024-03-18T00:50:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 client.25399 shutdown 2024-03-18T00:50:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 -- 172.21.15.49:0/1784326339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 msgr2=0x5609c70e7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70e7c00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009230 tx=0x7f08ec004270 comp rx=0 tx=0).stop 2024-03-18T00:50:40.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 -- 172.21.15.49:0/1784326339 shutdown_connections 2024-03-18T00:50:40.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5609c70f75f0 0x5609c70f9a50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f08d7fff640 0 client.25399 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:40.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70e2cd0 0x5609c70eec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70e7c00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 --2- 172.21.15.49:0/1784326339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70dd170 0x5609c70e76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 -- 172.21.15.49:0/1784326339 >> 172.21.15.49:0/1784326339 conn(0x5609c70d86e0 msgr2=0x5609c70d7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:40.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 -- 172.21.15.49:0/1784326339 shutdown_connections 2024-03-18T00:50:40.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.849+0000 7f0901a07740 1 -- 172.21.15.49:0/1784326339 wait complete. 2024-03-18T00:50:40.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.850+0000 7f0901a07740 20 client.25399 trim_cache size 0 max 16384 2024-03-18T00:50:40.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.884+0000 7f0901a07740 1 Processor -- start 2024-03-18T00:50:40.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.884+0000 7f0901a07740 1 -- start start 2024-03-18T00:50:40.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.884+0000 7f0901a07740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70dc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.884+0000 7f0901a07740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70ddb60 0x5609c70eec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f0901a07740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70e2cd0 0x5609c70fe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f0901a07740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f08f81a6bd0 con 0x5609c70ddb60 2024-03-18T00:50:40.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f0901a07740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f08f81a8710 con 0x5609c70dd170 2024-03-18T00:50:40.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f0901a07740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08f81a72a0 con 0x5609c70e2cd0 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08fcbb5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70ddb60 0x5609c70eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70dc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08fcbb5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70ddb60 0x5609c70eec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46348/0 (socket says 172.21.15.49:46348) 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08fcbb5640 1 -- 172.21.15.49:0/1509046583 learned_addr learned my addr 172.21.15.49:0/1509046583 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08f7fff640 1 --2- 172.21.15.49:0/1509046583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70e2cd0 0x5609c70fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08f7fff640 1 -- 172.21.15.49:0/1509046583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70e2cd0 msgr2=0x5609c70fe310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08f7fff640 1 --2- 172.21.15.49:0/1509046583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70e2cd0 0x5609c70fe310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08f7fff640 1 -- 172.21.15.49:0/1509046583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70ddb60 msgr2=0x5609c70eec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08f7fff640 1 --2- 172.21.15.49:0/1509046583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70ddb60 0x5609c70eec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08f7fff640 1 -- 172.21.15.49:0/1509046583 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609c6d160f0 con 0x5609c70dd170 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08f7fff640 1 --2- 172.21.15.49:0/1509046583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70dc050 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x5609c70fa2c0 tx=0x7f08e4009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c7c5f8380d59e09 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08e27fc640 1 -- 172.21.15.49:0/1509046583 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e401e040 con 0x5609c70dd170 2024-03-18T00:50:40.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.885+0000 7f08e27fc640 1 -- 172.21.15.49:0/1509046583 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e4005520 con 0x5609c70dd170 2024-03-18T00:50:40.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f08e27fc640 1 -- 172.21.15.49:0/1509046583 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08e4019cc0 con 0x5609c70dd170 2024-03-18T00:50:40.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f0901a07740 1 -- 172.21.15.49:0/1509046583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 msgr2=0x5609c70dc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:40.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f0901a07740 1 --2- 172.21.15.49:0/1509046583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70dc050 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x5609c70fa2c0 tx=0x7f08e4009f90 comp rx=0 tx=0).stop 2024-03-18T00:50:40.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f0901a07740 1 -- 172.21.15.49:0/1509046583 shutdown_connections 2024-03-18T00:50:40.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f0901a07740 1 --2- 172.21.15.49:0/1509046583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70e2cd0 0x5609c70fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f0901a07740 1 --2- 172.21.15.49:0/1509046583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70ddb60 0x5609c70eec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f0901a07740 1 --2- 172.21.15.49:0/1509046583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70dc050 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f0901a07740 1 -- 172.21.15.49:0/1509046583 >> 172.21.15.49:0/1509046583 conn(0x5609c70d86e0 msgr2=0x5609c70db940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f0901a07740 1 -- 172.21.15.49:0/1509046583 shutdown_connections 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.886+0000 7f0901a07740 1 -- 172.21.15.49:0/1509046583 wait complete. 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f0901a07740 1 -- start start 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f0901a07740 1 Processor -- start 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f0901a07740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70ec100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f0901a07740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70ec640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f0901a07740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70e2cd0 0x5609c70ed650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f0901a07740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f08f81a8040 con 0x5609c70e2cd0 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f0901a07740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f08f81a7970 con 0x5609c70dd170 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f0901a07740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08f81a4bb0 con 0x5609c70ddb60 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f0901a07740 10 client.0 _ll_register_callbacks cb 0x7f08ec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:40.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70ec100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f08fcbb5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70ec640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f08f7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70ec100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36530/0 (socket says 172.21.15.49:36530) 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f08f7fff640 1 -- 172.21.15.49:0/2718266978 learned_addr learned my addr 172.21.15.49:0/2718266978 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.887+0000 7f08f7fff640 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70e2cd0 0x5609c70ed650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08fcbb5640 1 -- 172.21.15.49:0/2718266978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 msgr2=0x5609c70ec100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08fcbb5640 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70ec100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08fcbb5640 1 -- 172.21.15.49:0/2718266978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70e2cd0 msgr2=0x5609c70ed650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08fcbb5640 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70e2cd0 0x5609c70ed650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08fcbb5640 1 -- 172.21.15.49:0/2718266978 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609c70099d0 con 0x5609c70ddb60 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08f7fff640 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70ec100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08fcbb5640 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70ec640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f08ec02c170 tx=0x7f08ec02c6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0e0c458a576daf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08d7fff640 1 -- 172.21.15.49:0/2718266978 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec009520 con 0x5609c70ddb60 2024-03-18T00:50:40.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f0901a07740 1 -- 172.21.15.49:0/2718266978 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08e4005520 con 0x5609c70ddb60 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08d7fff640 1 -- 172.21.15.49:0/2718266978 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f08ec0035f0 con 0x5609c70ddb60 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f08d7fff640 1 -- 172.21.15.49:0/2718266978 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f08ec0037f0 con 0x5609c70ddb60 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f0901a07740 10 client.25405 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.888+0000 7f0901a07740 1 -- 172.21.15.49:0/2718266978 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5609c70eedf0 con 0x5609c70ddb60 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f08d7fff640 1 -- 172.21.15.49:0/2718266978 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08e4005520 con 0x5609c70ddb60 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f08d7fff640 1 -- 172.21.15.49:0/2718266978 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f08ec058030 con 0x5609c70ddb60 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f0901a07740 10 client.25405 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f0901a07740 10 client.25405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f0901a07740 20 client.25405 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f0901a07740 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5609c715dfa0 0x5609c71603c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f0901a07740 4 client.25405 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f0901a07740 1 -- 172.21.15.49:0/2718266978 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5609c7049d90 con 0x5609c715dfa0 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f08f7fff640 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5609c715dfa0 0x5609c71603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:40.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f08d7fff640 10 client.25405 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.889+0000 7f08f7fff640 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5609c715dfa0 0x5609c71603c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33fa466b1861d48b server_cookie=d52696f95c6cbbc0 in_seq=0 out_seq=0 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f08d7fff640 1 -- 172.21.15.49:0/2718266978 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5609c7049d90 con 0x5609c715dfa0 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f08d7fff640 10 client.25405 handle_command_reply: tid=0 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 client.25405 shutdown 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 -- 172.21.15.49:0/2718266978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 msgr2=0x5609c70ec640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70ec640 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f08ec02c170 tx=0x7f08ec02c6d0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 -- 172.21.15.49:0/2718266978 shutdown_connections 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5609c715dfa0 0x5609c71603c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f08d7fff640 0 client.25405 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5609c70e2cd0 0x5609c70ed650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5609c70ddb60 0x5609c70ec640 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 --2- 172.21.15.49:0/2718266978 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5609c70dd170 0x5609c70ec100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 -- 172.21.15.49:0/2718266978 >> 172.21.15.49:0/2718266978 conn(0x5609c70d86e0 msgr2=0x5609c70dc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 -- 172.21.15.49:0/2718266978 shutdown_connections 2024-03-18T00:50:40.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 1 -- 172.21.15.49:0/2718266978 wait complete. 2024-03-18T00:50:40.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.890+0000 7f0901a07740 20 client.25405 trim_cache size 0 max 16384 2024-03-18T00:50:40.896 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:50:40.896 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:50:40.896 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:50:40.896 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:50:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 -- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e80a2880 msgr2=0x7f08e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e80a2880 0x7f08e80a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f08f0005ba0 tx=0x7f08f0005940 comp rx=0 tx=0).stop 2024-03-18T00:50:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 -- 172.21.15.49:0/813316158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08f8107520 msgr2=0x7f08f819f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08f8107520 0x7f08f819f130 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08f0009fd0 tx=0x7f08f000ef10 comp rx=0 tx=0).stop 2024-03-18T00:50:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 -- 172.21.15.49:0/813316158 shutdown_connections 2024-03-18T00:50:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e80a2880 0x7f08e80a4d40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08f8107520 0x7f08f819f130 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08f8104bf0 0x7f08f81980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 --2- 172.21.15.49:0/813316158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f81022c0 0x7f08f8197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:40.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 -- 172.21.15.49:0/813316158 >> 172.21.15.49:0/813316158 conn(0x7f08f80fbc70 msgr2=0x7f08f8102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:40.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 -- 172.21.15.49:0/813316158 shutdown_connections 2024-03-18T00:50:40.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:40.891+0000 7f08fee19640 1 -- 172.21.15.49:0/813316158 wait complete. 2024-03-18T00:50:40.902 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:50:40.902 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:50:41.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.012+0000 7fc086362640 1 -- 172.21.15.49:0/1861644116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0801013b0 msgr2=0x7fc080101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.012+0000 7fc086362640 1 --2- 172.21.15.49:0/1861644116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0801013b0 0x7fc080101830 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc078009ca0 tx=0x7fc07802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.012+0000 7fc086362640 1 -- 172.21.15.49:0/1861644116 shutdown_connections 2024-03-18T00:50:41.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.012+0000 7fc086362640 1 --2- 172.21.15.49:0/1861644116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc080100a10 0x7fc080100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.012+0000 7fc086362640 1 --2- 172.21.15.49:0/1861644116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0801013b0 0x7fc080101830 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.012+0000 7fc086362640 1 --2- 172.21.15.49:0/1861644116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0801000b0 0x7fc0801004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.013+0000 7fc086362640 1 -- 172.21.15.49:0/1861644116 >> 172.21.15.49:0/1861644116 conn(0x7fc0800fbc70 msgr2=0x7fc0800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.013+0000 7fc086362640 1 -- 172.21.15.49:0/1861644116 shutdown_connections 2024-03-18T00:50:41.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.013+0000 7fc086362640 1 -- 172.21.15.49:0/1861644116 wait complete. 2024-03-18T00:50:41.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.013+0000 7fc086362640 1 Processor -- start 2024-03-18T00:50:41.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.013+0000 7fc086362640 1 -- start start 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc086362640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0801000b0 0x7fc080199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc086362640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc080100a10 0x7fc08019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc086362640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0801013b0 0x7fc0801a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc086362640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc08010f670 con 0x7fc080100a10 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc086362640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc08010f4f0 con 0x7fc0801000b0 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc086362640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc08010f7f0 con 0x7fc0801013b0 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc07f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc080100a10 0x7fc08019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc07ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0801000b0 0x7fc080199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc07f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc080100a10 0x7fc08019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46378/0 (socket says 172.21.15.49:46378) 2024-03-18T00:50:41.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc07ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0801000b0 0x7fc080199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36538/0 (socket says 172.21.15.49:36538) 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc07f7fe640 1 -- 172.21.15.49:0/3851362808 learned_addr learned my addr 172.21.15.49:0/3851362808 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc0848ff640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0801013b0 0x7fc0801a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc0848ff640 1 -- 172.21.15.49:0/3851362808 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0801000b0 msgr2=0x7fc080199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc0848ff640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0801000b0 0x7fc080199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc0848ff640 1 -- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc080100a10 msgr2=0x7fc08019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc0848ff640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc080100a10 0x7fc08019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc0848ff640 1 -- 172.21.15.49:0/3851362808 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0801a1910 con 0x7fc0801013b0 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc07f7fe640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc080100a10 0x7fc08019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.014+0000 7fc07ffff640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0801000b0 0x7fc080199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.015+0000 7fc0848ff640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0801013b0 0x7fc0801a1270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc078037040 tx=0x7fc078032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.015+0000 7fc07d7fa640 1 -- 172.21.15.49:0/3851362808 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc078041020 con 0x7fc0801013b0 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.015+0000 7fc07d7fa640 1 -- 172.21.15.49:0/3851362808 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc078034cd0 con 0x7fc0801013b0 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.015+0000 7fc07d7fa640 1 -- 172.21.15.49:0/3851362808 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc078039040 con 0x7fc0801013b0 2024-03-18T00:50:41.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.015+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0801a1ba0 con 0x7fc0801013b0 2024-03-18T00:50:41.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.015+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0801a20c0 con 0x7fc0801013b0 2024-03-18T00:50:41.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.016+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0800685d0 con 0x7fc0801013b0 2024-03-18T00:50:41.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.018+0000 7fc07d7fa640 1 -- 172.21.15.49:0/3851362808 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc078044020 con 0x7fc0801013b0 2024-03-18T00:50:41.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.018+0000 7fc07d7fa640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0680a2880 0x7fc0680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.019+0000 7fc07ffff640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0680a2880 0x7fc0680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.019+0000 7fc07d7fa640 1 -- 172.21.15.49:0/3851362808 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc0780f0880 con 0x7fc0801013b0 2024-03-18T00:50:41.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.019+0000 7fc07ffff640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0680a2880 0x7fc0680a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc08019b1d0 tx=0x7fc070005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.023+0000 7fc07d7fa640 1 -- 172.21.15.49:0/3851362808 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc07804b020 con 0x7fc0801013b0 2024-03-18T00:50:41.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:40 smithi049 ceph-mon[32130]: pgmap v367: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T00:50:41.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.281+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0800623e0 con 0x7fc0801013b0 2024-03-18T00:50:41.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.282+0000 7fc07d7fa640 1 -- 172.21.15.49:0/3851362808 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11932 (secure 0 0 0) 0x7fc0780bd900 con 0x7fc0801013b0 2024-03-18T00:50:41.282 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:50:41.282 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:50:03.914926+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24463,24470,24481,24499,24506]},"id":1}]} 2024-03-18T00:50:41.282 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 25 2024-03-18T00:50:41.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0680a2880 msgr2=0x7fc0680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0680a2880 0x7fc0680a4d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc08019b1d0 tx=0x7fc070005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:41.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0801013b0 msgr2=0x7fc0801a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0801013b0 0x7fc0801a1270 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc078037040 tx=0x7fc078032ec0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 shutdown_connections 2024-03-18T00:50:41.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0680a2880 0x7fc0680a4d40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0801013b0 0x7fc0801a1270 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc080100a10 0x7fc08019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 --2- 172.21.15.49:0/3851362808 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0801000b0 0x7fc080199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 >> 172.21.15.49:0/3851362808 conn(0x7fc0800fbc70 msgr2=0x7fc0800fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 shutdown_connections 2024-03-18T00:50:41.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.284+0000 7fc086362640 1 -- 172.21.15.49:0/3851362808 wait complete. 2024-03-18T00:50:41.296 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:50:41.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.409+0000 7f9df7710640 1 -- 172.21.15.49:0/4723288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df01018d0 msgr2=0x7f9df0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.409+0000 7f9df7710640 1 --2- 172.21.15.49:0/4723288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df01018d0 0x7f9df0101d50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009900 tx=0x7f9de002bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:41.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.410+0000 7f9df7710640 1 -- 172.21.15.49:0/4723288 shutdown_connections 2024-03-18T00:50:41.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.410+0000 7f9df7710640 1 --2- 172.21.15.49:0/4723288 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df0102290 0x7f9df0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.410+0000 7f9df7710640 1 --2- 172.21.15.49:0/4723288 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df01018d0 0x7f9df0101d50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.410+0000 7f9df7710640 1 --2- 172.21.15.49:0/4723288 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df01006e0 0x7f9df0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.410+0000 7f9df7710640 1 -- 172.21.15.49:0/4723288 >> 172.21.15.49:0/4723288 conn(0x7f9df00fbc70 msgr2=0x7f9df00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.410+0000 7f9df7710640 1 -- 172.21.15.49:0/4723288 shutdown_connections 2024-03-18T00:50:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.410+0000 7f9df7710640 1 -- 172.21.15.49:0/4723288 wait complete. 2024-03-18T00:50:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.410+0000 7f9df7710640 1 Processor -- start 2024-03-18T00:50:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df7710640 1 -- start start 2024-03-18T00:50:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df7710640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df01006e0 0x7f9df010b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df7710640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df01018d0 0x7f9df01096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df7710640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df0102290 0x7f9df0109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df7710640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9df010d480 con 0x7f9df01018d0 2024-03-18T00:50:41.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df7710640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9df010d300 con 0x7f9df0102290 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df7710640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9df010d600 con 0x7f9df01006e0 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df4cab640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df01018d0 0x7f9df01096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df54ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df01006e0 0x7f9df010b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df4cab640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df01018d0 0x7f9df01096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46396/0 (socket says 172.21.15.49:46396) 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df4cab640 1 -- 172.21.15.49:0/2801085152 learned_addr learned my addr 172.21.15.49:0/2801085152 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df5cad640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df0102290 0x7f9df0109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df54ac640 1 -- 172.21.15.49:0/2801085152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df0102290 msgr2=0x7f9df0109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df54ac640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df0102290 0x7f9df0109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df54ac640 1 -- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df01018d0 msgr2=0x7f9df01096a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df54ac640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df01018d0 0x7f9df01096a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df54ac640 1 -- 172.21.15.49:0/2801085152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df010a470 con 0x7f9df01006e0 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.411+0000 7f9df4cab640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df01018d0 0x7f9df01096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.412+0000 7f9df54ac640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df01006e0 0x7f9df010b060 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00e9c0 tx=0x7f9dec00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.412+0000 7f9de67fc640 1 -- 172.21.15.49:0/2801085152 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00ccc0 con 0x7f9df01006e0 2024-03-18T00:50:41.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.412+0000 7f9df7710640 1 -- 172.21.15.49:0/2801085152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df01a1c00 con 0x7f9df01006e0 2024-03-18T00:50:41.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.412+0000 7f9de67fc640 1 -- 172.21.15.49:0/2801085152 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dec0035f0 con 0x7f9df01006e0 2024-03-18T00:50:41.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.412+0000 7f9de67fc640 1 -- 172.21.15.49:0/2801085152 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec00ce60 con 0x7f9df01006e0 2024-03-18T00:50:41.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.412+0000 7f9df7710640 1 -- 172.21.15.49:0/2801085152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df01a2120 con 0x7f9df01006e0 2024-03-18T00:50:41.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.414+0000 7f9dfa3dc740 1 Processor -- start 2024-03-18T00:50:41.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.415+0000 7f9de67fc640 1 -- 172.21.15.49:0/2801085152 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9dec040ba0 con 0x7f9df01006e0 2024-03-18T00:50:41.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.415+0000 7f9dfa3dc740 1 -- start start 2024-03-18T00:50:41.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.415+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c1f850 0x559ef4c1fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.415+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c24d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.415+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c253b0 0x559ef4c27840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.415+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559ef4b84490 con 0x559ef4c1f850 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.415+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559ef4b2cc90 con 0x559ef4c253b0 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.415+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ef4b80530 con 0x559ef4c20240 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.415+0000 7f9df4cab640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c253b0 0x559ef4c27840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df4cab640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c253b0 0x559ef4c27840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36562/0 (socket says 172.21.15.49:36562) 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df4cab640 1 -- 172.21.15.49:0/748932689 learned_addr learned my addr 172.21.15.49:0/748932689 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df5cad640 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c1f850 0x559ef4c1fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df54ac640 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c24d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9de67fc640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9ddc0a2800 0x7f9ddc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9de67fc640 1 -- 172.21.15.49:0/2801085152 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9dec0ce920 con 0x7f9df01006e0 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df5cad640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9ddc0a2800 0x7f9ddc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df5cad640 1 -- 172.21.15.49:0/748932689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 msgr2=0x559ef4c24d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df5cad640 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c24d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df5cad640 1 -- 172.21.15.49:0/748932689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c253b0 msgr2=0x559ef4c27840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df5cad640 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c253b0 0x559ef4c27840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df5cad640 1 -- 172.21.15.49:0/748932689 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ef4be7500 con 0x559ef4c1f850 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df4cab640 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c253b0 0x559ef4c27840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.416+0000 7f9df5cad640 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c1f850 0x559ef4c1fc70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005fd0 tx=0x7f9de8013ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7038ab3670001c9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.417+0000 7f9ddb7fe640 1 -- 172.21.15.49:0/748932689 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8023030 con 0x559ef4c1f850 2024-03-18T00:50:41.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.417+0000 7f9df5cad640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9ddc0a2800 0x7f9ddc0a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9de8023000 tx=0x7f9de8022000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.417+0000 7f9ddb7fe640 1 -- 172.21.15.49:0/748932689 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de8003e50 con 0x559ef4c1f850 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.417+0000 7f9ddb7fe640 1 -- 172.21.15.49:0/748932689 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8004030 con 0x559ef4c1f850 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/748932689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c1f850 msgr2=0x559ef4c1fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c1f850 0x559ef4c1fc70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9de8005fd0 tx=0x7f9de8013ef0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/748932689 shutdown_connections 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c253b0 0x559ef4c27840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c24d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/748932689 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c1f850 0x559ef4c1fc70 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/748932689 >> 172.21.15.49:0/748932689 conn(0x559ef4c1adc0 msgr2=0x559ef4c1b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/748932689 shutdown_connections 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/748932689 wait complete. 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 -- start start 2024-03-18T00:50:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.418+0000 7f9dfa3dc740 1 Processor -- start 2024-03-18T00:50:41.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c29e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 0x559ef4c31340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559ef4c28360 con 0x559ef4c253b0 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559ef4b40230 con 0x559ef4c1f850 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ef4c284e0 con 0x559ef4c20240 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9dfa3dc740 10 client.0 _ll_register_callbacks cb 0x559ef4b7c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df54ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c29e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df4cab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df54ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c29e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36578/0 (socket says 172.21.15.49:36578) 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df54ac640 1 -- 172.21.15.49:0/1137294181 learned_addr learned my addr 172.21.15.49:0/1137294181 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df5cad640 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 0x559ef4c31340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df54ac640 1 -- 172.21.15.49:0/1137294181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 msgr2=0x559ef4c2a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df54ac640 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df54ac640 1 -- 172.21.15.49:0/1137294181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 msgr2=0x559ef4c31340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df54ac640 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 0x559ef4c31340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.419+0000 7f9df54ac640 1 -- 172.21.15.49:0/1137294181 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ef4b4f100 con 0x559ef4c1f850 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9df54ac640 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c29e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9dec09afa0 tx=0x7f9dec0037a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=618535dbdb129039 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dd8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/1137294181 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec09f3f0 con 0x559ef4c1f850 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1137294181 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9de8003e50 con 0x559ef4c1f850 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/1137294181 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9dec003b10 con 0x559ef4c1f850 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/1137294181 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9dec0a3a40 con 0x559ef4c1f850 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dfa3dc740 10 client.25388 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1137294181 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559ef48983f0 con 0x559ef4c1f850 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/1137294181 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de8003e50 con 0x559ef4c1f850 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/1137294181 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f9dec09a890 con 0x559ef4c1f850 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dfa3dc740 10 client.25388 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dfa3dc740 10 client.25388 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:50:41.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dfa3dc740 20 client.25388 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x559ef4c35b60 0x559ef4c37fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dfa3dc740 4 client.25388 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.420+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1137294181 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ef4b919d0 con 0x559ef4c35b60 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.421+0000 7f9df4cab640 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x559ef4c35b60 0x559ef4c37fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.421+0000 7f9dd8ff9640 10 client.25388 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.421+0000 7f9df4cab640 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x559ef4c35b60 0x559ef4c37fb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2a78859873345672 server_cookie=3a368949bc054c67 in_seq=0 out_seq=0 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/1137294181 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559ef4b919d0 con 0x559ef4c35b60 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dd8ff9640 10 client.25388 handle_command_reply: tid=0 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 client.25388 shutdown 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1137294181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 msgr2=0x559ef4c29e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c29e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9dec09afa0 tx=0x7f9dec0037a0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1137294181 shutdown_connections 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x559ef4c35b60 0x559ef4c37fb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dd8ff9640 0 client.25388 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:50:41.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 0x559ef4c31340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1137294181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c29e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1137294181 >> 172.21.15.49:0/1137294181 conn(0x559ef4c1adc0 msgr2=0x559ef4c19fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1137294181 shutdown_connections 2024-03-18T00:50:41.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.422+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1137294181 wait complete. 2024-03-18T00:50:41.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.423+0000 7f9dfa3dc740 20 client.25388 trim_cache size 0 max 16384 2024-03-18T00:50:41.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:40 smithi107 ceph-mon[36270]: pgmap v367: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T00:50:41.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9dfa3dc740 1 Processor -- start 2024-03-18T00:50:41.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9dfa3dc740 1 -- start start 2024-03-18T00:50:41.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c20240 0x559ef4c31300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c253b0 0x559ef4c1e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01aaf90 con 0x559ef4c20240 2024-03-18T00:50:41.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01acad0 con 0x559ef4c1f850 2024-03-18T00:50:41.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01ab660 con 0x559ef4c253b0 2024-03-18T00:50:41.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9df4cab640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c20240 0x559ef4c31300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9df54ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9df4cab640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c20240 0x559ef4c31300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46442/0 (socket says 172.21.15.49:46442) 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9df4cab640 1 -- 172.21.15.49:0/1714176154 learned_addr learned my addr 172.21.15.49:0/1714176154 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.436+0000 7f9df5cad640 1 --2- 172.21.15.49:0/1714176154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c253b0 0x559ef4c1e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.437+0000 7f9df54ac640 1 -- 172.21.15.49:0/1714176154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c253b0 msgr2=0x559ef4c1e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.437+0000 7f9df54ac640 1 --2- 172.21.15.49:0/1714176154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c253b0 0x559ef4c1e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.437+0000 7f9df54ac640 1 -- 172.21.15.49:0/1714176154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c20240 msgr2=0x559ef4c31300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.437+0000 7f9df54ac640 1 --2- 172.21.15.49:0/1714176154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c20240 0x559ef4c31300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.437+0000 7f9df54ac640 1 -- 172.21.15.49:0/1714176154 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ef49627d0 con 0x559ef4c1f850 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.437+0000 7f9df54ac640 1 --2- 172.21.15.49:0/1714176154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2a2d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9dec097700 tx=0x7f9dec09b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6da405d8a0a42df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.437+0000 7f9ddaffd640 1 -- 172.21.15.49:0/1714176154 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec097260 con 0x559ef4c1f850 2024-03-18T00:50:41.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.437+0000 7f9ddaffd640 1 -- 172.21.15.49:0/1714176154 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dec09b9a0 con 0x559ef4c1f850 2024-03-18T00:50:41.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.437+0000 7f9ddaffd640 1 -- 172.21.15.49:0/1714176154 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec09a6f0 con 0x559ef4c1f850 2024-03-18T00:50:41.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1714176154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 msgr2=0x559ef4c2a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1714176154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2a2d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9dec097700 tx=0x7f9dec09b420 comp rx=0 tx=0).stop 2024-03-18T00:50:41.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1714176154 shutdown_connections 2024-03-18T00:50:41.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1714176154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c253b0 0x559ef4c1e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1714176154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c20240 0x559ef4c31300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/1714176154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2a2d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1714176154 >> 172.21.15.49:0/1714176154 conn(0x559ef4c1adc0 msgr2=0x559ef4c1e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1714176154 shutdown_connections 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/1714176154 wait complete. 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 -- start start 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.438+0000 7f9dfa3dc740 1 Processor -- start 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9dfa3dc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 0x559ef4c2fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01ac400 con 0x559ef4c253b0 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01abd30 con 0x559ef4c1f850 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9dfa3dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9df01a8f70 con 0x559ef4c20240 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9dfa3dc740 10 client.0 _ll_register_callbacks cb 0x7f9dec097560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:41.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df54ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df4cab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df54ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36604/0 (socket says 172.21.15.49:36604) 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df4cab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59906/0 (socket says 172.21.15.49:59906) 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df54ac640 1 -- 172.21.15.49:0/3931085550 learned_addr learned my addr 172.21.15.49:0/3931085550 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df5cad640 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 0x559ef4c2fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df4cab640 1 -- 172.21.15.49:0/3931085550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 msgr2=0x559ef4c2e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df4cab640 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df4cab640 1 -- 172.21.15.49:0/3931085550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 msgr2=0x559ef4c2fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df4cab640 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 0x559ef4c2fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df4cab640 1 -- 172.21.15.49:0/3931085550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ef4c301a0 con 0x559ef4c20240 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9df4cab640 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2ec10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9de0005ab0 tx=0x7f9de0049d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ca04fe80f3cdaca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.439+0000 7f9dd8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/3931085550 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0049820 con 0x559ef4c20240 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/3931085550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9dec09b9a0 con 0x559ef4c20240 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/3931085550 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9de00040c0 con 0x559ef4c20240 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/3931085550 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9de0057070 con 0x559ef4c20240 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dfa3dc740 10 client.25429 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/3931085550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559ef4c31510 con 0x559ef4c20240 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/3931085550 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9dec09b9a0 con 0x559ef4c20240 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/3931085550 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f9de0005070 con 0x559ef4c20240 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dfa3dc740 10 client.25429 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dfa3dc740 10 client.25429 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:50:41.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dfa3dc740 20 client.25429 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x559ef4c9c4c0 0x559ef4c9e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dfa3dc740 4 client.25429 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.440+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/3931085550 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559ef4b8bd20 con 0x559ef4c9c4c0 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9df54ac640 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x559ef4c9c4c0 0x559ef4c9e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dd8ff9640 10 client.25429 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9df54ac640 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x559ef4c9c4c0 0x559ef4c9e8e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=13bab5dabf07f832 server_cookie=ad12341a57ef5de1 in_seq=0 out_seq=0 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dd8ff9640 1 -- 172.21.15.49:0/3931085550 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559ef4b8bd20 con 0x559ef4c9c4c0 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dd8ff9640 10 client.25429 handle_command_reply: tid=0 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 client.25429 shutdown 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/3931085550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 msgr2=0x559ef4c2ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2ec10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9de0005ab0 tx=0x7f9de0049d70 comp rx=0 tx=0).stop 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/3931085550 shutdown_connections 2024-03-18T00:50:41.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x559ef4c9c4c0 0x559ef4c9e8e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ef4c253b0 0x559ef4c2fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ef4c20240 0x559ef4c2ec10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dd8ff9640 0 client.25429 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:50:41.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 --2- 172.21.15.49:0/3931085550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ef4c1f850 0x559ef4c2e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/3931085550 >> 172.21.15.49:0/3931085550 conn(0x559ef4c1adc0 msgr2=0x559ef4c1e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/3931085550 shutdown_connections 2024-03-18T00:50:41.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.441+0000 7f9dfa3dc740 1 -- 172.21.15.49:0/3931085550 wait complete. 2024-03-18T00:50:41.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9dfa3dc740 20 client.25429 trim_cache size 0 max 16384 2024-03-18T00:50:41.448 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:50:41.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 -- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9ddc0a2800 msgr2=0x7f9ddc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9ddc0a2800 0x7f9ddc0a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9de8023000 tx=0x7f9de8022000 comp rx=0 tx=0).stop 2024-03-18T00:50:41.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 -- 172.21.15.49:0/2801085152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df01006e0 msgr2=0x7f9df010b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df01006e0 0x7f9df010b060 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00e9c0 tx=0x7f9dec00ee80 comp rx=0 tx=0).stop 2024-03-18T00:50:41.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 -- 172.21.15.49:0/2801085152 shutdown_connections 2024-03-18T00:50:41.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9ddc0a2800 0x7f9ddc0a4cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9df0102290 0x7f9df0109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9df01018d0 0x7f9df01096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 --2- 172.21.15.49:0/2801085152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9df01006e0 0x7f9df010b060 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 -- 172.21.15.49:0/2801085152 >> 172.21.15.49:0/2801085152 conn(0x7f9df00fbc70 msgr2=0x7f9df0104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 -- 172.21.15.49:0/2801085152 shutdown_connections 2024-03-18T00:50:41.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.442+0000 7f9df7710640 1 -- 172.21.15.49:0/2801085152 wait complete. 2024-03-18T00:50:41.451 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:40 smithi106 ceph-mon[35899]: pgmap v367: 97 pgs: 97 active+clean; 655 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T00:50:41.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:50:41.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.577+0000 7f3f02aac640 1 -- 172.21.15.49:0/540816445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3efc1018d0 msgr2=0x7f3efc101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.577+0000 7f3f02aac640 1 --2- 172.21.15.49:0/540816445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3efc1018d0 0x7f3efc101d50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009900 tx=0x7f3ee802bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:41.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.577+0000 7f3f02aac640 1 -- 172.21.15.49:0/540816445 shutdown_connections 2024-03-18T00:50:41.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.577+0000 7f3f02aac640 1 --2- 172.21.15.49:0/540816445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3efc1018d0 0x7f3efc101d50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.577+0000 7f3f02aac640 1 --2- 172.21.15.49:0/540816445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3efc102290 0x7f3efc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.577+0000 7f3f02aac640 1 --2- 172.21.15.49:0/540816445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3efc1006e0 0x7f3efc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.577+0000 7f3f02aac640 1 -- 172.21.15.49:0/540816445 >> 172.21.15.49:0/540816445 conn(0x7f3efc0fbc70 msgr2=0x7f3efc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.577+0000 7f3f02aac640 1 -- 172.21.15.49:0/540816445 shutdown_connections 2024-03-18T00:50:41.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.577+0000 7f3f02aac640 1 -- 172.21.15.49:0/540816445 wait complete. 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f02aac640 1 Processor -- start 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f02aac640 1 -- start start 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f02aac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3efc1006e0 0x7f3efc0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f02aac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3efc1018d0 0x7f3efc19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f02aac640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3efc102290 0x7f3efc19ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f02aac640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc10c3b0 con 0x7f3efc102290 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f02aac640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc10c230 con 0x7f3efc1006e0 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f02aac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc10c530 con 0x7f3efc1018d0 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f00848640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3efc1006e0 0x7f3efc0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3efbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3efc1018d0 0x7f3efc19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f00848640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3efc1006e0 0x7f3efc0694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36618/0 (socket says 172.21.15.49:36618) 2024-03-18T00:50:41.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f00848640 1 -- 172.21.15.49:0/1420317388 learned_addr learned my addr 172.21.15.49:0/1420317388 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.578+0000 7f3f01049640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3efc102290 0x7f3efc19ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3efbfff640 1 -- 172.21.15.49:0/1420317388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3efc1006e0 msgr2=0x7f3efc0694b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3efbfff640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3efc1006e0 0x7f3efc0694b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3efbfff640 1 -- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3efc102290 msgr2=0x7f3efc19ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3efbfff640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3efc102290 0x7f3efc19ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3efbfff640 1 -- 172.21.15.49:0/1420317388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3efc19f780 con 0x7f3efc1018d0 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3f01049640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3efc102290 0x7f3efc19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3efbfff640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3efc1018d0 0x7f3efc19c850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8036040 tx=0x7f3ee802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3ef9ffb640 1 -- 172.21.15.49:0/1420317388 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8043020 con 0x7f3efc1018d0 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3ef9ffb640 1 -- 172.21.15.49:0/1420317388 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ee80054e0 con 0x7f3efc1018d0 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3ef9ffb640 1 -- 172.21.15.49:0/1420317388 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee8003c30 con 0x7f3efc1018d0 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.579+0000 7f3f02aac640 1 -- 172.21.15.49:0/1420317388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3efc19f9b0 con 0x7f3efc1018d0 2024-03-18T00:50:41.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.580+0000 7f3f02aac640 1 -- 172.21.15.49:0/1420317388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3efc19fe80 con 0x7f3efc1018d0 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.582+0000 7f3f05771740 1 Processor -- start 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.582+0000 7f3f05771740 1 -- start start 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.582+0000 7f3f05771740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb6850 0x55b2d7bb6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.582+0000 7f3f05771740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb7240 0x55b2d7bbbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.582+0000 7f3f05771740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bbe840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.582+0000 7f3f05771740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b2d7b1b490 con 0x55b2d7bbc3b0 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.582+0000 7f3f05771740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b2d7ac3c90 con 0x55b2d7bb6850 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3f05771740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b2d7b17530 con 0x55b2d7bb7240 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3f00848640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb7240 0x55b2d7bbbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3f01049640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb6850 0x55b2d7bb6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3f00848640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb7240 0x55b2d7bbbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59930/0 (socket says 172.21.15.49:59930) 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3f01049640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb6850 0x55b2d7bb6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36620/0 (socket says 172.21.15.49:36620) 2024-03-18T00:50:41.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3f00848640 1 -- 172.21.15.49:0/707125660 learned_addr learned my addr 172.21.15.49:0/707125660 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3ef9ffb640 1 -- 172.21.15.49:0/1420317388 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3ee8046020 con 0x7f3efc1018d0 2024-03-18T00:50:41.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3f01049640 1 -- 172.21.15.49:0/707125660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb7240 msgr2=0x55b2d7bbbd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3f01049640 1 --2- 172.21.15.49:0/707125660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb7240 0x55b2d7bbbd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.583+0000 7f3f01049640 1 -- 172.21.15.49:0/707125660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 msgr2=0x55b2d7bbe840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.584+0000 7f3ef9ffb640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3eec0a2900 0x7f3eec0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.584+0000 7f3f00848640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3eec0a2900 0x7f3eec0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.584+0000 7f3ef9ffb640 1 -- 172.21.15.49:0/1420317388 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3ee80f09d0 con 0x7f3efc1018d0 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.584+0000 7f3efbfff640 1 --2- 172.21.15.49:0/707125660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bbe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.584+0000 7f3f01049640 1 --2- 172.21.15.49:0/707125660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bbe840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.584+0000 7f3f01049640 1 -- 172.21.15.49:0/707125660 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2d7b7e500 con 0x55b2d7bb6850 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.584+0000 7f3efbfff640 1 --2- 172.21.15.49:0/707125660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bbe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.585+0000 7f3f00848640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3eec0a2900 0x7f3eec0a4dc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55b2d7bb7640 tx=0x7f3ef0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.585+0000 7f3f01049640 1 --2- 172.21.15.49:0/707125660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb6850 0x55b2d7bb6c70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4005fd0 tx=0x7f3ef4011bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a190bce5c4a7ecd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.585+0000 7f3ee6ffd640 1 -- 172.21.15.49:0/707125660 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4004d20 con 0x55b2d7bb6850 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.585+0000 7f3ee6ffd640 1 -- 172.21.15.49:0/707125660 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef400f9a0 con 0x55b2d7bb6850 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3ee6ffd640 1 -- 172.21.15.49:0/707125660 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4004490 con 0x55b2d7bb6850 2024-03-18T00:50:41.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 -- 172.21.15.49:0/707125660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb6850 msgr2=0x55b2d7bb6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 --2- 172.21.15.49:0/707125660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb6850 0x55b2d7bb6c70 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4005fd0 tx=0x7f3ef4011bf0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 -- 172.21.15.49:0/707125660 shutdown_connections 2024-03-18T00:50:41.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 --2- 172.21.15.49:0/707125660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bbe840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 --2- 172.21.15.49:0/707125660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb7240 0x55b2d7bbbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 --2- 172.21.15.49:0/707125660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb6850 0x55b2d7bb6c70 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 -- 172.21.15.49:0/707125660 >> 172.21.15.49:0/707125660 conn(0x55b2d7bb1dc0 msgr2=0x55b2d7bb21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 -- 172.21.15.49:0/707125660 shutdown_connections 2024-03-18T00:50:41.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 -- 172.21.15.49:0/707125660 wait complete. 2024-03-18T00:50:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.586+0000 7f3f05771740 1 -- start start 2024-03-18T00:50:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.587+0000 7f3f05771740 1 Processor -- start 2024-03-18T00:50:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.587+0000 7f3f05771740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.587+0000 7f3f05771740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.587+0000 7f3f05771740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bcc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.587+0000 7f3f00848640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.587+0000 7f3f00848640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59940/0 (socket says 172.21.15.49:59940) 2024-03-18T00:50:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f00848640 1 -- 172.21.15.49:0/2825921797 learned_addr learned my addr 172.21.15.49:0/2825921797 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b2d7bc37c0 con 0x55b2d7bb7240 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3efbfff640 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f01049640 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bcc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b2d7ad7230 con 0x55b2d7bbc3b0 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b2d7bc3940 con 0x55b2d7bb6850 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f05771740 10 client.0 _ll_register_callbacks cb 0x55b2d7b13010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f00848640 1 -- 172.21.15.49:0/2825921797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bbc3b0 msgr2=0x55b2d7bcc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f00848640 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bcc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f00848640 1 -- 172.21.15.49:0/2825921797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bb7240 msgr2=0x55b2d7bc56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f00848640 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f00848640 1 -- 172.21.15.49:0/2825921797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2d7ae6100 con 0x55b2d7bb6850 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3efbfff640 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ef400f9a0 con 0x55b2d7bb6850 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3f00848640 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc5160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00045f0 tx=0x7f3ef0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=265f239246b4a8dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.588+0000 7f3edbfff640 10 client.25441 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3edbfff640 1 -- 172.21.15.49:0/2825921797 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef001f700 con 0x55b2d7bb6850 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3edbfff640 1 -- 172.21.15.49:0/2825921797 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3ef001f8a0 con 0x55b2d7bb6850 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3edbfff640 1 -- 172.21.15.49:0/2825921797 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3ef001faa0 con 0x55b2d7bb6850 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3f05771740 10 client.25441 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55b2d782f3f0 con 0x55b2d7bb6850 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3edbfff640 1 -- 172.21.15.49:0/2825921797 <== mon.1 v2:172.21.15.107:3300/0 4 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f3ef0027070 con 0x55b2d7bb6850 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3f05771740 10 client.25441 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3f05771740 10 client.25441 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3f05771740 20 client.25441 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3f05771740 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b2d7bd0e00 0x55b2d7bd3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3f05771740 4 client.25441 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.589+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2d7b289d0 con 0x55b2d7bd0e00 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.590+0000 7f3efbfff640 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b2d7bd0e00 0x55b2d7bd3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.590+0000 7f3edbfff640 10 client.25441 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:50:41.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.590+0000 7f3efbfff640 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b2d7bd0e00 0x55b2d7bd3270 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7eaa4c434242602f server_cookie=80f15da448b24d36 in_seq=0 out_seq=0 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3edbfff640 1 -- 172.21.15.49:0/2825921797 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b2d7b289d0 con 0x55b2d7bd0e00 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3edbfff640 10 client.25441 handle_command_reply: tid=0 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 client.25441 shutdown 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 msgr2=0x55b2d7bc5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc5160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00045f0 tx=0x7f3ef0004730 comp rx=0 tx=0).stop 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 shutdown_connections 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b2d7bd0e00 0x55b2d7bd3270 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bcc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3edbfff640 0 client.25441 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 --2- 172.21.15.49:0/2825921797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc5160 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 >> 172.21.15.49:0/2825921797 conn(0x55b2d7bb1dc0 msgr2=0x55b2d7bb0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 shutdown_connections 2024-03-18T00:50:41.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 1 -- 172.21.15.49:0/2825921797 wait complete. 2024-03-18T00:50:41.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.591+0000 7f3f05771740 20 client.25441 trim_cache size 0 max 16384 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f05771740 1 Processor -- start 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f05771740 1 -- start start 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f05771740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f05771740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bcc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f05771740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bc52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f05771740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1a6ad0 con 0x55b2d7bbc3b0 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f05771740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1a8610 con 0x55b2d7bb7240 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f05771740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1a71a0 con 0x55b2d7bb6850 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f01049640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bc52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f00848640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f00848640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59950/0 (socket says 172.21.15.49:59950) 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f01049640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bc52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46494/0 (socket says 172.21.15.49:46494) 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3f00848640 1 -- 172.21.15.49:0/150502672 learned_addr learned my addr 172.21.15.49:0/150502672 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.618+0000 7f3efbfff640 1 --2- 172.21.15.49:0/150502672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bcc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.619+0000 7f3f00848640 1 -- 172.21.15.49:0/150502672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 msgr2=0x55b2d7bcc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.619+0000 7f3f00848640 1 --2- 172.21.15.49:0/150502672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bcc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.619+0000 7f3f00848640 1 -- 172.21.15.49:0/150502672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 msgr2=0x55b2d7bc52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.619+0000 7f3f00848640 1 --2- 172.21.15.49:0/150502672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bc52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.619+0000 7f3f00848640 1 -- 172.21.15.49:0/150502672 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2d78f97d0 con 0x55b2d7bb6850 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.619+0000 7f3f00848640 1 --2- 172.21.15.49:0/150502672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc4d70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00058c0 tx=0x7f3ef0023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f077d6a4557e01d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.619+0000 7f3ee67fc640 1 -- 172.21.15.49:0/150502672 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef001f900 con 0x55b2d7bb6850 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.619+0000 7f3ee67fc640 1 -- 172.21.15.49:0/150502672 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef400f9a0 con 0x55b2d7bb6850 2024-03-18T00:50:41.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.619+0000 7f3ee67fc640 1 -- 172.21.15.49:0/150502672 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0004c60 con 0x55b2d7bb6850 2024-03-18T00:50:41.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 -- 172.21.15.49:0/150502672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 msgr2=0x55b2d7bc4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 --2- 172.21.15.49:0/150502672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc4d70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3ef00058c0 tx=0x7f3ef0023000 comp rx=0 tx=0).stop 2024-03-18T00:50:41.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 -- 172.21.15.49:0/150502672 shutdown_connections 2024-03-18T00:50:41.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 --2- 172.21.15.49:0/150502672 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bcc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 --2- 172.21.15.49:0/150502672 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bc52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 --2- 172.21.15.49:0/150502672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc4d70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 -- 172.21.15.49:0/150502672 >> 172.21.15.49:0/150502672 conn(0x55b2d7bb1dc0 msgr2=0x55b2d7bb21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 -- 172.21.15.49:0/150502672 shutdown_connections 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 -- 172.21.15.49:0/150502672 wait complete. 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 -- start start 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.620+0000 7f3f05771740 1 Processor -- start 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f05771740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f05771740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f05771740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bcaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f05771740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1a7f40 con 0x55b2d7bbc3b0 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f05771740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1a7870 con 0x55b2d7bb7240 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f05771740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3efc1a4ab0 con 0x55b2d7bb6850 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f05771740 10 client.0 _ll_register_callbacks cb 0x7f3ef0009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3efbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3efbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36636/0 (socket says 172.21.15.49:36636) 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3efbfff640 1 -- 172.21.15.49:0/311179450 learned_addr learned my addr 172.21.15.49:0/311179450 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f00848640 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f01049640 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bcaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3efbfff640 1 -- 172.21.15.49:0/311179450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 msgr2=0x55b2d7bc9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3efbfff640 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3efbfff640 1 -- 172.21.15.49:0/311179450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 msgr2=0x55b2d7bcaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3efbfff640 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bcaef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3efbfff640 1 -- 172.21.15.49:0/311179450 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2d7bcb4e0 con 0x55b2d7bb7240 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.621+0000 7f3f00848640 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.622+0000 7f3efbfff640 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc9f50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80b9c70 tx=0x7f3ee80dd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6fe670394435a68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.622+0000 7f3edbfff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:50:41.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.622+0000 7f3edbfff640 1 -- 172.21.15.49:0/311179450 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee80b7e60 con 0x55b2d7bb7240 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.622+0000 7f3edbfff640 1 -- 172.21.15.49:0/311179450 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3ee80bb390 con 0x55b2d7bb7240 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.622+0000 7f3f05771740 1 -- 172.21.15.49:0/311179450 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ef400f9a0 con 0x55b2d7bb7240 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.622+0000 7f3edbfff640 1 -- 172.21.15.49:0/311179450 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3ee80bd2f0 con 0x55b2d7bb7240 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.622+0000 7f3f05771740 10 client.25412 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.622+0000 7f3f05771740 1 -- 172.21.15.49:0/311179450 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b2d7bcc850 con 0x55b2d7bb7240 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.622+0000 7f3edbfff640 1 -- 172.21.15.49:0/311179450 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ee80c1440 con 0x55b2d7bb7240 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3edbfff640 1 -- 172.21.15.49:0/311179450 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f3ee80dd0b0 con 0x55b2d7bb7240 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3f05771740 10 client.25412 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3f05771740 10 client.25412 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3f05771740 20 client.25412 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3f05771740 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b2d7c375c0 0x55b2d7c399b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3f05771740 4 client.25412 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:50:41.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3f05771740 1 -- 172.21.15.49:0/311179450 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f3ef400f9a0 con 0x55b2d7c375c0 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3f00848640 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b2d7c375c0 0x55b2d7c399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3edbfff640 10 client.25412 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3f00848640 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b2d7c375c0 0x55b2d7c399b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=52c3c41099ec79c3 server_cookie=9baf7a67cc5f054d in_seq=0 out_seq=0 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3edbfff640 1 -- 172.21.15.49:0/311179450 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f3ef400f9a0 con 0x55b2d7c375c0 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3edbfff640 10 client.25412 handle_command_reply: tid=0 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.623+0000 7f3f05771740 1 client.25412 shutdown 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 -- 172.21.15.49:0/311179450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 msgr2=0x55b2d7bc9f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc9f50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80b9c70 tx=0x7f3ee80dd040 comp rx=0 tx=0).stop 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 -- 172.21.15.49:0/311179450 shutdown_connections 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55b2d7c375c0 0x55b2d7c399b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3edbfff640 0 client.25412 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b2d7bb7240 0x55b2d7bc9f50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b2d7bbc3b0 0x55b2d7bcaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 --2- 172.21.15.49:0/311179450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b2d7bb6850 0x55b2d7bc9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 -- 172.21.15.49:0/311179450 >> 172.21.15.49:0/311179450 conn(0x55b2d7bb1dc0 msgr2=0x55b2d7bc4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 -- 172.21.15.49:0/311179450 shutdown_connections 2024-03-18T00:50:41.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 1 -- 172.21.15.49:0/311179450 wait complete. 2024-03-18T00:50:41.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f05771740 20 client.25412 trim_cache size 0 max 16384 2024-03-18T00:50:41.629 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:50:41.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.624+0000 7f3f02aac640 1 -- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3eec0a2900 msgr2=0x7f3eec0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3eec0a2900 0x7f3eec0a4dc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55b2d7bb7640 tx=0x7f3ef0005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 -- 172.21.15.49:0/1420317388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3efc1018d0 msgr2=0x7f3efc19c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3efc1018d0 0x7f3efc19c850 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8036040 tx=0x7f3ee802c650 comp rx=0 tx=0).stop 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 -- 172.21.15.49:0/1420317388 shutdown_connections 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3eec0a2900 0x7f3eec0a4dc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3efc102290 0x7f3efc19ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3efc1018d0 0x7f3efc19c850 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 --2- 172.21.15.49:0/1420317388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3efc1006e0 0x7f3efc0694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 -- 172.21.15.49:0/1420317388 >> 172.21.15.49:0/1420317388 conn(0x7f3efc0fbc70 msgr2=0x7f3efc104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 -- 172.21.15.49:0/1420317388 shutdown_connections 2024-03-18T00:50:41.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.625+0000 7f3f02aac640 1 -- 172.21.15.49:0/1420317388 wait complete. 2024-03-18T00:50:41.637 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:50:41.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.750+0000 7fe529759640 1 -- 172.21.15.49:0/4135956771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5241017f0 msgr2=0x7fe524101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.750+0000 7fe529759640 1 --2- 172.21.15.49:0/4135956771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5241017f0 0x7fe524101c10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe514009900 tx=0x7fe51402bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:41.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.750+0000 7fe529759640 1 -- 172.21.15.49:0/4135956771 shutdown_connections 2024-03-18T00:50:41.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.750+0000 7fe529759640 1 --2- 172.21.15.49:0/4135956771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5241033a0 0x7fe524109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.750+0000 7fe529759640 1 --2- 172.21.15.49:0/4135956771 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5241029e0 0x7fe524102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.750+0000 7fe529759640 1 --2- 172.21.15.49:0/4135956771 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5241017f0 0x7fe524101c10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.750+0000 7fe529759640 1 -- 172.21.15.49:0/4135956771 >> 172.21.15.49:0/4135956771 conn(0x7fe5240fcd40 msgr2=0x7fe5240ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.750+0000 7fe529759640 1 -- 172.21.15.49:0/4135956771 shutdown_connections 2024-03-18T00:50:41.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.750+0000 7fe529759640 1 -- 172.21.15.49:0/4135956771 wait complete. 2024-03-18T00:50:41.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe529759640 1 Processor -- start 2024-03-18T00:50:41.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe529759640 1 -- start start 2024-03-18T00:50:41.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe529759640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5241017f0 0x7fe524197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe529759640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5241029e0 0x7fe524198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe529759640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5241033a0 0x7fe52419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe529759640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe52410d470 con 0x7fe5241017f0 2024-03-18T00:50:41.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5241017f0 0x7fe524197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe529759640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe52410d2f0 con 0x7fe5241029e0 2024-03-18T00:50:41.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5241017f0 0x7fe524197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46520/0 (socket says 172.21.15.49:46520) 2024-03-18T00:50:41.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe522ffd640 1 -- 172.21.15.49:0/530314656 learned_addr learned my addr 172.21.15.49:0/530314656 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe529759640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe52410d5f0 con 0x7fe5241033a0 2024-03-18T00:50:41.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe5227fc640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5241029e0 0x7fe524198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.751+0000 7fe5237fe640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5241033a0 0x7fe52419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe5227fc640 1 -- 172.21.15.49:0/530314656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5241033a0 msgr2=0x7fe52419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe5227fc640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5241033a0 0x7fe52419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe5227fc640 1 -- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5241017f0 msgr2=0x7fe524197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe5227fc640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5241017f0 0x7fe524197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe5227fc640 1 -- 172.21.15.49:0/530314656 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52419f750 con 0x7fe5241029e0 2024-03-18T00:50:41.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe5237fe640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5241033a0 0x7fe52419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe522ffd640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5241017f0 0x7fe524197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe5227fc640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5241029e0 0x7fe524198070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe51800e9f0 tx=0x7fe51800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe50bfff640 1 -- 172.21.15.49:0/530314656 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51800ccc0 con 0x7fe5241029e0 2024-03-18T00:50:41.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe50bfff640 1 -- 172.21.15.49:0/530314656 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe518010910 con 0x7fe5241029e0 2024-03-18T00:50:41.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe50bfff640 1 -- 172.21.15.49:0/530314656 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518018960 con 0x7fe5241029e0 2024-03-18T00:50:41.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe529759640 1 -- 172.21.15.49:0/530314656 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe52419fa40 con 0x7fe5241029e0 2024-03-18T00:50:41.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.752+0000 7fe529759640 1 -- 172.21.15.49:0/530314656 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe52419ff40 con 0x7fe5241029e0 2024-03-18T00:50:41.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.755+0000 7fe50bfff640 1 -- 172.21.15.49:0/530314656 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe518010ac0 con 0x7fe5241029e0 2024-03-18T00:50:41.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.755+0000 7fe50bfff640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe50c0a2800 0x7fe50c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.755+0000 7fe50bfff640 1 -- 172.21.15.49:0/530314656 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe5180cee30 con 0x7fe5241029e0 2024-03-18T00:50:41.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.755+0000 7fe522ffd640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe50c0a2800 0x7fe50c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe52c412740 1 Processor -- start 2024-03-18T00:50:41.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe522ffd640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe50c0a2800 0x7fe50c0a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe51402c8e0 tx=0x7fe514005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe52c412740 1 -- start start 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe52c412740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e89cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe52c412740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89d240 0x56495e8a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe52c412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8a4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe52c412740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56495e801490 con 0x56495e89c850 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe52c412740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56495e7a9c90 con 0x56495e89d240 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe52c412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56495e7fd530 con 0x56495e8a23b0 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e89cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e89cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46528/0 (socket says 172.21.15.49:46528) 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe522ffd640 1 -- 172.21.15.49:0/3817485589 learned_addr learned my addr 172.21.15.49:0/3817485589 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe5227fc640 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89d240 0x56495e8a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.756+0000 7fe5237fe640 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe5227fc640 1 -- 172.21.15.49:0/3817485589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 msgr2=0x56495e8a4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe5227fc640 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8a4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe5227fc640 1 -- 172.21.15.49:0/3817485589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 msgr2=0x56495e89cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe5227fc640 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e89cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe5227fc640 1 -- 172.21.15.49:0/3817485589 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56495e864500 con 0x56495e89d240 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe522ffd640 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e89cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe5227fc640 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89d240 0x56495e8a1d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe51800c6a0 tx=0x7fe51809b4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7c3b5fd44b7240f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe5097fa640 1 -- 172.21.15.49:0/3817485589 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518099900 con 0x56495e89d240 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe5097fa640 1 -- 172.21.15.49:0/3817485589 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe518099b60 con 0x56495e89d240 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.757+0000 7fe5097fa640 1 -- 172.21.15.49:0/3817485589 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51809af10 con 0x56495e89d240 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.758+0000 7fe52c412740 1 -- 172.21.15.49:0/3817485589 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89d240 msgr2=0x56495e8a1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.758+0000 7fe52c412740 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89d240 0x56495e8a1d40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe51800c6a0 tx=0x7fe51809b4d0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.758+0000 7fe52c412740 1 -- 172.21.15.49:0/3817485589 shutdown_connections 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.758+0000 7fe52c412740 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89d240 0x56495e8a1d40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.758+0000 7fe52c412740 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8a4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.758+0000 7fe52c412740 1 --2- 172.21.15.49:0/3817485589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e89cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.758+0000 7fe52c412740 1 -- 172.21.15.49:0/3817485589 >> 172.21.15.49:0/3817485589 conn(0x56495e897dc0 msgr2=0x56495e8981c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.758+0000 7fe52c412740 1 -- 172.21.15.49:0/3817485589 shutdown_connections 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.758+0000 7fe52c412740 1 -- 172.21.15.49:0/3817485589 wait complete. 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe52c412740 1 -- start start 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe52c412740 1 Processor -- start 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe52c412740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8a6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe52c412740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 0x56495e8a7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe52c412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8ae270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe52c412740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56495e8a5300 con 0x56495e89d240 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe52c412740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56495e7bd230 con 0x56495e89c850 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe52c412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56495e8a5480 con 0x56495e8a23b0 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe5227fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 0x56495e8a7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8a6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8a6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36688/0 (socket says 172.21.15.49:36688) 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe522ffd640 1 -- 172.21.15.49:0/872734384 learned_addr learned my addr 172.21.15.49:0/872734384 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe52c412740 10 client.0 _ll_register_callbacks cb 0x56495e7f9010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.759+0000 7fe5237fe640 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe5227fc640 1 -- 172.21.15.49:0/872734384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 msgr2=0x56495e8ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe5227fc640 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe5227fc640 1 -- 172.21.15.49:0/872734384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 msgr2=0x56495e8a6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe5227fc640 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8a6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe5227fc640 1 -- 172.21.15.49:0/872734384 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56495e7cc100 con 0x56495e89d240 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe5237fe640 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe5227fc640 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 0x56495e8a7270 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe51800e9c0 tx=0x7fe51809b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8836d4cd6ec1032b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe506ff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe506ff5640 1 -- 172.21.15.49:0/872734384 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180afd90 con 0x56495e89d240 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe506ff5640 1 -- 172.21.15.49:0/872734384 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe51809acb0 con 0x56495e89d240 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe52c412740 1 -- 172.21.15.49:0/872734384 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe518099b60 con 0x56495e89d240 2024-03-18T00:50:41.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe506ff5640 1 -- 172.21.15.49:0/872734384 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe518080070 con 0x56495e89d240 2024-03-18T00:50:41.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe52c412740 10 client.15588 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:41.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.760+0000 7fe52c412740 1 -- 172.21.15.49:0/872734384 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56495e5153f0 con 0x56495e89d240 2024-03-18T00:50:41.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe506ff5640 1 -- 172.21.15.49:0/872734384 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe518099b60 con 0x56495e89d240 2024-03-18T00:50:41.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe506ff5640 1 -- 172.21.15.49:0/872734384 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fe51809cae0 con 0x56495e89d240 2024-03-18T00:50:41.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe52c412740 10 client.15588 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:41.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe52c412740 10 client.15588 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:50:41.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe52c412740 20 client.15588 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:41.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe52c412740 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x56495e8b2780 0x56495e8b4b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe52c412740 4 client.15588 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:41.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe52c412740 1 -- 172.21.15.49:0/872734384 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56495e80e9d0 con 0x56495e8b2780 2024-03-18T00:50:41.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe522ffd640 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x56495e8b2780 0x56495e8b4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe522ffd640 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x56495e8b2780 0x56495e8b4b70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4648521bd2ed1d23 server_cookie=e670698aa284200b in_seq=0 out_seq=0 2024-03-18T00:50:41.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.761+0000 7fe506ff5640 10 client.15588 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:50:41.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.762+0000 7fe506ff5640 1 -- 172.21.15.49:0/872734384 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56495e80e9d0 con 0x56495e8b2780 2024-03-18T00:50:41.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.762+0000 7fe506ff5640 10 client.15588 handle_command_reply: tid=0 2024-03-18T00:50:41.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 client.15588 shutdown 2024-03-18T00:50:41.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 -- 172.21.15.49:0/872734384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 msgr2=0x56495e8a7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 0x56495e8a7270 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe51800e9c0 tx=0x7fe51809b9e0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 -- 172.21.15.49:0/872734384 shutdown_connections 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x56495e8b2780 0x56495e8b4b70 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe506ff5640 0 client.15588 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 0x56495e8a7270 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 --2- 172.21.15.49:0/872734384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8a6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 -- 172.21.15.49:0/872734384 >> 172.21.15.49:0/872734384 conn(0x56495e897dc0 msgr2=0x56495e896fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 -- 172.21.15.49:0/872734384 shutdown_connections 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 1 -- 172.21.15.49:0/872734384 wait complete. 2024-03-18T00:50:41.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.763+0000 7fe52c412740 20 client.15588 trim_cache size 0 max 16384 2024-03-18T00:50:41.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe52c412740 1 Processor -- start 2024-03-18T00:50:41.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe52c412740 1 -- start start 2024-03-18T00:50:41.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe52c412740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e896750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe52c412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e89d240 0x56495e8ae230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe52c412740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e8a23b0 0x56495e896c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe52c412740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe5241a6cd0 con 0x56495e89c850 2024-03-18T00:50:41.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe52c412740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe5241a8810 con 0x56495e8a23b0 2024-03-18T00:50:41.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe52c412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe5241a73a0 con 0x56495e89d240 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe5227fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e89d240 0x56495e8ae230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e896750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe5227fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e89d240 0x56495e8ae230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60016/0 (socket says 172.21.15.49:60016) 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe5227fc640 1 -- 172.21.15.49:0/1363603254 learned_addr learned my addr 172.21.15.49:0/1363603254 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.789+0000 7fe5237fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e8a23b0 0x56495e896c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe5227fc640 1 -- 172.21.15.49:0/1363603254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e8a23b0 msgr2=0x56495e896c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe5227fc640 1 --2- 172.21.15.49:0/1363603254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e8a23b0 0x56495e896c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe5227fc640 1 -- 172.21.15.49:0/1363603254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 msgr2=0x56495e896750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe5227fc640 1 --2- 172.21.15.49:0/1363603254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e896750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe5227fc640 1 -- 172.21.15.49:0/1363603254 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56495e5df7d0 con 0x56495e89d240 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe522ffd640 1 --2- 172.21.15.49:0/1363603254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e896750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe5227fc640 1 --2- 172.21.15.49:0/1363603254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e89d240 0x56495e8ae230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe51800c6a0 tx=0x7fe5180cc980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2c987338e73d817 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe508ff9640 1 -- 172.21.15.49:0/1363603254 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51809cae0 con 0x56495e89d240 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe508ff9640 1 -- 172.21.15.49:0/1363603254 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51809cc80 con 0x56495e89d240 2024-03-18T00:50:41.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.790+0000 7fe508ff9640 1 -- 172.21.15.49:0/1363603254 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518018b00 con 0x56495e89d240 2024-03-18T00:50:41.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.791+0000 7fe52c412740 1 -- 172.21.15.49:0/1363603254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e89d240 msgr2=0x56495e8ae230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.791+0000 7fe52c412740 1 --2- 172.21.15.49:0/1363603254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e89d240 0x56495e8ae230 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe51800c6a0 tx=0x7fe5180cc980 comp rx=0 tx=0).stop 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.791+0000 7fe52c412740 1 -- 172.21.15.49:0/1363603254 shutdown_connections 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.791+0000 7fe52c412740 1 --2- 172.21.15.49:0/1363603254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e8a23b0 0x56495e896c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.791+0000 7fe52c412740 1 --2- 172.21.15.49:0/1363603254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e89d240 0x56495e8ae230 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.791+0000 7fe52c412740 1 --2- 172.21.15.49:0/1363603254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89c850 0x56495e896750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.791+0000 7fe52c412740 1 -- 172.21.15.49:0/1363603254 >> 172.21.15.49:0/1363603254 conn(0x56495e897dc0 msgr2=0x56495e89e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.791+0000 7fe52c412740 1 -- 172.21.15.49:0/1363603254 shutdown_connections 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.791+0000 7fe52c412740 1 -- 172.21.15.49:0/1363603254 wait complete. 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe52c412740 1 -- start start 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe52c412740 1 Processor -- start 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe52c412740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe52c412740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 0x56495e8abee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe52c412740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8acef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe52c412740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe5241a8140 con 0x56495e89d240 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe52c412740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe5241a7a70 con 0x56495e89c850 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe52c412740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe5241a4cb0 con 0x56495e8a23b0 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe52c412740 10 client.0 _ll_register_callbacks cb 0x7fe514042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8ab9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe5237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8ab9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36714/0 (socket says 172.21.15.49:36714) 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe522ffd640 1 -- 172.21.15.49:0/1087466874 learned_addr learned my addr 172.21.15.49:0/1087466874 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe5227fc640 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 0x56495e8abee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe522ffd640 1 -- 172.21.15.49:0/1087466874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 msgr2=0x56495e8acef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe522ffd640 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8acef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe522ffd640 1 -- 172.21.15.49:0/1087466874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 msgr2=0x56495e8abee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe522ffd640 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 0x56495e8abee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.792+0000 7fe522ffd640 1 -- 172.21.15.49:0/1087466874 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56495e8ad4e0 con 0x56495e89c850 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe522ffd640 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8ab9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe51402cfd0 tx=0x7fe51402bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76bf86a05542cdf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe506ff5640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe506ff5640 1 -- 172.21.15.49:0/1087466874 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5140046c0 con 0x56495e89c850 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe52c412740 1 -- 172.21.15.49:0/1087466874 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe51809cc80 con 0x56495e89c850 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe506ff5640 1 -- 172.21.15.49:0/1087466874 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe514004860 con 0x56495e89c850 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe506ff5640 1 -- 172.21.15.49:0/1087466874 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe51406d050 con 0x56495e89c850 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe52c412740 10 client.25430 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe52c412740 1 -- 172.21.15.49:0/1087466874 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56495e9189c0 con 0x56495e89c850 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe506ff5640 1 -- 172.21.15.49:0/1087466874 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51809cc80 con 0x56495e89c850 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.793+0000 7fe506ff5640 1 -- 172.21.15.49:0/1087466874 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fe51405d030 con 0x56495e89c850 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe52c412740 10 client.25430 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe52c412740 10 client.25430 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe52c412740 20 client.25430 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:41.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe52c412740 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x56495e8b2780 0x56495e91b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe52c412740 4 client.25430 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe52c412740 1 -- 172.21.15.49:0/1087466874 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56495e808d20 con 0x56495e8b2780 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe5227fc640 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x56495e8b2780 0x56495e91b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe506ff5640 10 client.25430 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe5227fc640 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x56495e8b2780 0x56495e91b330 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c56b312eeb2cacd server_cookie=453621feb50c1c in_seq=0 out_seq=0 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe506ff5640 1 -- 172.21.15.49:0/1087466874 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56495e808d20 con 0x56495e8b2780 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe506ff5640 10 client.25430 handle_command_reply: tid=0 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.794+0000 7fe52c412740 1 client.25430 shutdown 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 -- 172.21.15.49:0/1087466874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 msgr2=0x56495e8ab9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8ab9a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe51402cfd0 tx=0x7fe51402bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 -- 172.21.15.49:0/1087466874 shutdown_connections 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x56495e8b2780 0x56495e91b330 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe506ff5640 0 client.25430 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56495e8a23b0 0x56495e8acef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56495e89d240 0x56495e8abee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 --2- 172.21.15.49:0/1087466874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56495e89c850 0x56495e8ab9a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 -- 172.21.15.49:0/1087466874 >> 172.21.15.49:0/1087466874 conn(0x56495e897dc0 msgr2=0x56495e89b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 -- 172.21.15.49:0/1087466874 shutdown_connections 2024-03-18T00:50:41.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 1 -- 172.21.15.49:0/1087466874 wait complete. 2024-03-18T00:50:41.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.795+0000 7fe52c412740 20 client.25430 trim_cache size 0 max 16384 2024-03-18T00:50:41.800 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:50:41.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 -- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe50c0a2800 msgr2=0x7fe50c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe50c0a2800 0x7fe50c0a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe51402c8e0 tx=0x7fe514005b50 comp rx=0 tx=0).stop 2024-03-18T00:50:41.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 -- 172.21.15.49:0/530314656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5241029e0 msgr2=0x7fe524198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5241029e0 0x7fe524198070 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe51800e9f0 tx=0x7fe51800eeb0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 -- 172.21.15.49:0/530314656 shutdown_connections 2024-03-18T00:50:41.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe50c0a2800 0x7fe50c0a4cc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe5241029e0 0x7fe524198070 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5241033a0 0x7fe52419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 --2- 172.21.15.49:0/530314656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe5241017f0 0x7fe524197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 -- 172.21.15.49:0/530314656 >> 172.21.15.49:0/530314656 conn(0x7fe5240fcd40 msgr2=0x7fe524105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 -- 172.21.15.49:0/530314656 shutdown_connections 2024-03-18T00:50:41.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.796+0000 7fe529759640 1 -- 172.21.15.49:0/530314656 wait complete. 2024-03-18T00:50:41.810 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:50:41.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.923+0000 7f300ac7e640 1 -- 172.21.15.49:0/874028317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30041018d0 msgr2=0x7f3004101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.923+0000 7f300ac7e640 1 --2- 172.21.15.49:0/874028317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30041018d0 0x7f3004101d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009900 tx=0x7f2ff002bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:41.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.923+0000 7f300ac7e640 1 -- 172.21.15.49:0/874028317 shutdown_connections 2024-03-18T00:50:41.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.923+0000 7f300ac7e640 1 --2- 172.21.15.49:0/874028317 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3004102290 0x7f3004108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.923+0000 7f300ac7e640 1 --2- 172.21.15.49:0/874028317 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30041018d0 0x7f3004101d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.923+0000 7f300ac7e640 1 --2- 172.21.15.49:0/874028317 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30041006e0 0x7f3004100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.923+0000 7f300ac7e640 1 -- 172.21.15.49:0/874028317 >> 172.21.15.49:0/874028317 conn(0x7f30040fbc70 msgr2=0x7f30040fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.923+0000 7f300ac7e640 1 -- 172.21.15.49:0/874028317 shutdown_connections 2024-03-18T00:50:41.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.923+0000 7f300ac7e640 1 -- 172.21.15.49:0/874028317 wait complete. 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f300ac7e640 1 Processor -- start 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f300ac7e640 1 -- start start 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f300ac7e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30041006e0 0x7f3004069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f300ac7e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30041018d0 0x7f300419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f300ac7e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3004102290 0x7f300419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f300ac7e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f300410c340 con 0x7f3004102290 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f300ac7e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f300410c1c0 con 0x7f30041006e0 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f300ac7e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f300410c4c0 con 0x7f30041018d0 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f3008a1a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30041006e0 0x7f3004069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f300921b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3004102290 0x7f300419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f3008a1a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30041006e0 0x7f3004069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36730/0 (socket says 172.21.15.49:36730) 2024-03-18T00:50:41.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f3003fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30041018d0 0x7f300419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.924+0000 7f3008a1a640 1 -- 172.21.15.49:0/2832020509 learned_addr learned my addr 172.21.15.49:0/2832020509 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f3008a1a640 1 -- 172.21.15.49:0/2832020509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30041018d0 msgr2=0x7f300419c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f3008a1a640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30041018d0 0x7f300419c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f3008a1a640 1 -- 172.21.15.49:0/2832020509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3004102290 msgr2=0x7f300419eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f3008a1a640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3004102290 0x7f300419eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f3008a1a640 1 -- 172.21.15.49:0/2832020509 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f300419f760 con 0x7f30041006e0 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f3008a1a640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30041006e0 0x7f3004069510 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800d950 tx=0x7f2ff800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f3001ffb640 1 -- 172.21.15.49:0/2832020509 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8017020 con 0x7f30041006e0 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f3001ffb640 1 -- 172.21.15.49:0/2832020509 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ff8009c30 con 0x7f30041006e0 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f300ac7e640 1 -- 172.21.15.49:0/2832020509 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f300419fa50 con 0x7f30041006e0 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f3001ffb640 1 -- 172.21.15.49:0/2832020509 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff800f740 con 0x7f30041006e0 2024-03-18T00:50:41.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.925+0000 7f300ac7e640 1 -- 172.21.15.49:0/2832020509 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f300419ff20 con 0x7f30041006e0 2024-03-18T00:50:41.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.928+0000 7f3001ffb640 1 -- 172.21.15.49:0/2832020509 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2ff801a020 con 0x7f30041006e0 2024-03-18T00:50:41.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300d861740 1 Processor -- start 2024-03-18T00:50:41.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300d861740 1 -- start start 2024-03-18T00:50:41.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f3001ffb640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300d861740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300d861740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2ddd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300d861740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2de3b0 0x55c33c2e0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300d861740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c33c23d490 con 0x55c33c2d8850 2024-03-18T00:50:41.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300d861740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c33c1e5c90 con 0x55c33c2de3b0 2024-03-18T00:50:41.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300d861740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c33c239530 con 0x55c33c2d9240 2024-03-18T00:50:41.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300921b640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f3001ffb640 1 -- 172.21.15.49:0/2832020509 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2ff80ce420 con 0x7f30041006e0 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f3003fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2ddd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300921b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2de3b0 0x55c33c2e0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f3008a1a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300921b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2de3b0 0x55c33c2e0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36744/0 (socket says 172.21.15.49:36744) 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.929+0000 7f300921b640 1 -- 172.21.15.49:0/3303689085 learned_addr learned my addr 172.21.15.49:0/3303689085 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f3003fff640 1 -- 172.21.15.49:0/3303689085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2de3b0 msgr2=0x55c33c2e0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f3003fff640 1 --2- 172.21.15.49:0/3303689085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2de3b0 0x55c33c2e0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f3003fff640 1 -- 172.21.15.49:0/3303689085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 msgr2=0x55c33c2d8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f3003fff640 1 --2- 172.21.15.49:0/3303689085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f300921b640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc013c70 tx=0x7f2ffc009e90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f3003fff640 1 -- 172.21.15.49:0/3303689085 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c33c2a0500 con 0x55c33c2d9240 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f3008a1a640 1 --2- 172.21.15.49:0/3303689085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f3003fff640 1 --2- 172.21.15.49:0/3303689085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2ddd40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009230 tx=0x7f2ff0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71906a7dba3eff28 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f2feeffd640 1 -- 172.21.15.49:0/3303689085 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff002cb80 con 0x55c33c2d9240 2024-03-18T00:50:41.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f2feeffd640 1 -- 172.21.15.49:0/3303689085 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ff0005040 con 0x55c33c2d9240 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.930+0000 7f2feeffd640 1 -- 172.21.15.49:0/3303689085 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0004920 con 0x55c33c2d9240 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.931+0000 7f300d861740 1 -- 172.21.15.49:0/3303689085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 msgr2=0x55c33c2ddd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.931+0000 7f300d861740 1 --2- 172.21.15.49:0/3303689085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2ddd40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0009230 tx=0x7f2ff0009560 comp rx=0 tx=0).stop 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.931+0000 7f300d861740 1 -- 172.21.15.49:0/3303689085 shutdown_connections 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.931+0000 7f300d861740 1 --2- 172.21.15.49:0/3303689085 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2de3b0 0x55c33c2e0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.931+0000 7f300d861740 1 --2- 172.21.15.49:0/3303689085 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2ddd40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.931+0000 7f300d861740 1 --2- 172.21.15.49:0/3303689085 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.931+0000 7f300d861740 1 -- 172.21.15.49:0/3303689085 >> 172.21.15.49:0/3303689085 conn(0x55c33c2d3dc0 msgr2=0x55c33c2d41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.931+0000 7f300d861740 1 -- 172.21.15.49:0/3303689085 shutdown_connections 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.931+0000 7f300d861740 1 -- 172.21.15.49:0/3303689085 wait complete. 2024-03-18T00:50:41.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.932+0000 7f300d861740 1 -- start start 2024-03-18T00:50:41.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.932+0000 7f300d861740 1 Processor -- start 2024-03-18T00:50:41.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.932+0000 7f300d861740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d8850 0x55c33c2e7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.932+0000 7f300d861740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2e76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.932+0000 7f300d861740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.932+0000 7f300d861740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c33c2e5710 con 0x55c33c2de3b0 2024-03-18T00:50:41.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.932+0000 7f300d861740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c33c1f9230 con 0x55c33c2d8850 2024-03-18T00:50:41.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.932+0000 7f300d861740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c33c2e5890 con 0x55c33c2d9240 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300921b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300921b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ee6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46600/0 (socket says 172.21.15.49:46600) 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300921b640 1 -- 172.21.15.49:0/4134260370 learned_addr learned my addr 172.21.15.49:0/4134260370 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f3008a1a640 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d8850 0x55c33c2e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f3003fff640 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2e76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300d861740 10 client.0 _ll_register_callbacks cb 0x55c33c235010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300921b640 1 -- 172.21.15.49:0/4134260370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 msgr2=0x55c33c2e76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300921b640 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2e76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300921b640 1 -- 172.21.15.49:0/4134260370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d8850 msgr2=0x55c33c2e7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300921b640 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d8850 0x55c33c2e7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300921b640 1 -- 172.21.15.49:0/4134260370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c33c208100 con 0x55c33c2de3b0 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f3008a1a640 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d8850 0x55c33c2e7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:50:41.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f3003fff640 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2e76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f300921b640 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ee6f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009e60 tx=0x7f2ffc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dacc270ab7015b1a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.933+0000 7f2fe3fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4134260370 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc021410 con 0x55c33c2de3b0 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f300d861740 1 -- 172.21.15.49:0/4134260370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ff0005040 con 0x55c33c2de3b0 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4134260370 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2ffc0215b0 con 0x55c33c2de3b0 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4134260370 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2ffc033050 con 0x55c33c2de3b0 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f300d861740 10 client.15606 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f300d861740 1 -- 172.21.15.49:0/4134260370 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c33bf513f0 con 0x55c33c2de3b0 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4134260370 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ff0005040 con 0x55c33c2de3b0 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4134260370 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f2ffc020050 con 0x55c33c2de3b0 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f300d861740 10 client.15606 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f300d861740 10 client.15606 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:50:41.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f300d861740 20 client.15606 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f300d861740 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55c33c2f2fd0 0x55c33c2f5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f300d861740 4 client.15606 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.934+0000 7f300d861740 1 -- 172.21.15.49:0/4134260370 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c33c24a9d0 con 0x55c33c2f2fd0 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.935+0000 7f3008a1a640 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55c33c2f2fd0 0x55c33c2f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.935+0000 7f2fe3fff640 10 client.15606 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.935+0000 7f3008a1a640 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55c33c2f2fd0 0x55c33c2f5450 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69846a14c290dbe7 server_cookie=4fb697a6fa63a3a3 in_seq=0 out_seq=0 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4134260370 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c33c24a9d0 con 0x55c33c2f2fd0 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f2fe3fff640 10 client.15606 handle_command_reply: tid=0 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 client.15606 shutdown 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 -- 172.21.15.49:0/4134260370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 msgr2=0x55c33c2ee6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ee6f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc009e60 tx=0x7f2ffc004ab0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 -- 172.21.15.49:0/4134260370 shutdown_connections 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55c33c2f2fd0 0x55c33c2f5450 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ee6f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2e76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 --2- 172.21.15.49:0/4134260370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d8850 0x55c33c2e7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 -- 172.21.15.49:0/4134260370 >> 172.21.15.49:0/4134260370 conn(0x55c33c2d3dc0 msgr2=0x55c33c2d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f2fe3fff640 0 client.15606 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:50:41.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 -- 172.21.15.49:0/4134260370 shutdown_connections 2024-03-18T00:50:41.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 1 -- 172.21.15.49:0/4134260370 wait complete. 2024-03-18T00:50:41.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.936+0000 7f300d861740 20 client.15606 trim_cache size 0 max 16384 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f300d861740 1 Processor -- start 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f300d861740 1 -- start start 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f300d861740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f300d861740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d9240 0x55c33c2ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f300d861740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2de3b0 0x55c33c2d2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f300d861740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f30041a6b20 con 0x55c33c2d8850 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f300d861740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f30041a8660 con 0x55c33c2d9240 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f300d861740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f30041a71f0 con 0x55c33c2de3b0 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f3008a1a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f3008a1a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46612/0 (socket says 172.21.15.49:46612) 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f3008a1a640 1 -- 172.21.15.49:0/1862222028 learned_addr learned my addr 172.21.15.49:0/1862222028 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f3003fff640 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d9240 0x55c33c2ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.963+0000 7f300921b640 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2de3b0 0x55c33c2d2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f3008a1a640 1 -- 172.21.15.49:0/1862222028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2de3b0 msgr2=0x55c33c2d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f3008a1a640 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2de3b0 0x55c33c2d2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f3008a1a640 1 -- 172.21.15.49:0/1862222028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d9240 msgr2=0x55c33c2ee6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f3008a1a640 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d9240 0x55c33c2ee6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f3008a1a640 1 -- 172.21.15.49:0/1862222028 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c33c2cf640 con 0x55c33c2d8850 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f3003fff640 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d9240 0x55c33c2ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f3008a1a640 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d2750 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8097770 tx=0x7f2ff8097040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896f15b8123bc7b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f2fee7fc640 1 -- 172.21.15.49:0/1862222028 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff809e070 con 0x55c33c2d8850 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f2fee7fc640 1 -- 172.21.15.49:0/1862222028 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ff80b0070 con 0x55c33c2d8850 2024-03-18T00:50:41.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.964+0000 7f2fee7fc640 1 -- 172.21.15.49:0/1862222028 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80a30b0 con 0x55c33c2d8850 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 -- 172.21.15.49:0/1862222028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 msgr2=0x55c33c2d2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d2750 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8097770 tx=0x7f2ff8097040 comp rx=0 tx=0).stop 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 -- 172.21.15.49:0/1862222028 shutdown_connections 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2d9240 0x55c33c2ee6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2de3b0 0x55c33c2d2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 --2- 172.21.15.49:0/1862222028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2d8850 0x55c33c2d2750 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 -- 172.21.15.49:0/1862222028 >> 172.21.15.49:0/1862222028 conn(0x55c33c2d3dc0 msgr2=0x55c33c2e5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 -- 172.21.15.49:0/1862222028 shutdown_connections 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 -- 172.21.15.49:0/1862222028 wait complete. 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 -- start start 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.965+0000 7f300d861740 1 Processor -- start 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300d861740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2eb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300d861740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300d861740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2ecd80 0x55c33c2ed200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300d861740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f30041a7f90 con 0x55c33c2de3b0 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300d861740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f30041a78c0 con 0x55c33c2ecd80 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300d861740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f30041a4b00 con 0x55c33c2d9240 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f3003fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300d861740 10 client.0 _ll_register_callbacks cb 0x7f2ffc009db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f3003fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ebd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46624/0 (socket says 172.21.15.49:46624) 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f3003fff640 1 -- 172.21.15.49:0/4148930378 learned_addr learned my addr 172.21.15.49:0/4148930378 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300921b640 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2ecd80 0x55c33c2ed200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f3008a1a640 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2eb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300921b640 1 -- 172.21.15.49:0/4148930378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 msgr2=0x55c33c2eb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300921b640 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2eb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300921b640 1 -- 172.21.15.49:0/4148930378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 msgr2=0x55c33c2ebd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:41.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300921b640 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ebd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.966+0000 7f300921b640 1 -- 172.21.15.49:0/4148930378 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c33c01b7d0 con 0x55c33c2ecd80 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.967+0000 7f3003fff640 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ebd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.967+0000 7f300921b640 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2ecd80 0x55c33c2ed200 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc011ce0 tx=0x7f2ffc0044b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29c6fe70e7ba0dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.967+0000 7f2fe3fff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.967+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4148930378 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc03c060 con 0x55c33c2ecd80 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.967+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4148930378 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2ffc01f3e0 con 0x55c33c2ecd80 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.967+0000 7f300d861740 1 -- 172.21.15.49:0/4148930378 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ff80b0070 con 0x55c33c2ecd80 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.967+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4148930378 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2ffc01f620 con 0x55c33c2ecd80 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.967+0000 7f300d861740 10 client.25445 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.967+0000 7f300d861740 1 -- 172.21.15.49:0/4148930378 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c33c2ee870 con 0x55c33c2ecd80 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4148930378 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ff80b0070 con 0x55c33c2ecd80 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4148930378 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f2ffc021410 con 0x55c33c2ecd80 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f300d861740 10 client.25445 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:41.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f300d861740 10 client.25445 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f300d861740 20 client.25445 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f300d861740 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55c33c359970 0x55c33c35bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f300d861740 4 client.25445 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f300d861740 1 -- 172.21.15.49:0/4148930378 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c33c244d20 con 0x55c33c359970 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f3008a1a640 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55c33c359970 0x55c33c35bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f2fe3fff640 10 client.25445 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.968+0000 7f3008a1a640 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55c33c359970 0x55c33c35bd90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d9c74e5370b4ce83 server_cookie=5b66eb10160d9ad in_seq=0 out_seq=0 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f2fe3fff640 1 -- 172.21.15.49:0/4148930378 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c33c244d20 con 0x55c33c359970 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f2fe3fff640 10 client.25445 handle_command_reply: tid=0 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 client.25445 shutdown 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 -- 172.21.15.49:0/4148930378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2ecd80 msgr2=0x55c33c2ed200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2ecd80 0x55c33c2ed200 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc011ce0 tx=0x7f2ffc0044b0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 -- 172.21.15.49:0/4148930378 shutdown_connections 2024-03-18T00:50:41.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55c33c359970 0x55c33c35bd90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c33c2ecd80 0x55c33c2ed200 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c33c2de3b0 0x55c33c2ebd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f2fe3fff640 0 client.25445 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:50:41.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 --2- 172.21.15.49:0/4148930378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c33c2d9240 0x55c33c2eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 -- 172.21.15.49:0/4148930378 >> 172.21.15.49:0/4148930378 conn(0x55c33c2d3dc0 msgr2=0x55c33c2d7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 -- 172.21.15.49:0/4148930378 shutdown_connections 2024-03-18T00:50:41.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 1 -- 172.21.15.49:0/4148930378 wait complete. 2024-03-18T00:50:41.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.969+0000 7f300d861740 20 client.25445 trim_cache size 0 max 16384 2024-03-18T00:50:41.974 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:50:41.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 -- 172.21.15.49:0/2832020509 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ff40a2800 msgr2=0x7f2ff40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc013c70 tx=0x7f2ffc009e90 comp rx=0 tx=0).stop 2024-03-18T00:50:41.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 -- 172.21.15.49:0/2832020509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30041006e0 msgr2=0x7f3004069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30041006e0 0x7f3004069510 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800d950 tx=0x7f2ff800de10 comp rx=0 tx=0).stop 2024-03-18T00:50:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 -- 172.21.15.49:0/2832020509 shutdown_connections 2024-03-18T00:50:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ff40a2800 0x7f2ff40a4cc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3004102290 0x7f300419eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f30041018d0 0x7f300419c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 --2- 172.21.15.49:0/2832020509 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f30041006e0 0x7f3004069510 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 -- 172.21.15.49:0/2832020509 >> 172.21.15.49:0/2832020509 conn(0x7f30040fbc70 msgr2=0x7f3004104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 -- 172.21.15.49:0/2832020509 shutdown_connections 2024-03-18T00:50:41.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:41.970+0000 7f300ac7e640 1 -- 172.21.15.49:0/2832020509 wait complete. 2024-03-18T00:50:41.981 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:41 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3851362808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:50:41.986 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:50:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.098+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/3484668396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 msgr2=0x7fb1f4100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.098+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/3484668396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 0x7fb1f4100b00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009900 tx=0x7fb1f002bc30 comp rx=0 tx=0).stop 2024-03-18T00:50:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.098+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/3484668396 shutdown_connections 2024-03-18T00:50:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.098+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/3484668396 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1f41018d0 0x7fb1f4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.098+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/3484668396 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f4102290 0x7fb1f4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.098+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/3484668396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 0x7fb1f4100b00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.098+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/3484668396 >> 172.21.15.49:0/3484668396 conn(0x7fb1f40fbc70 msgr2=0x7fb1f40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.098+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/3484668396 shutdown_connections 2024-03-18T00:50:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.098+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/3484668396 wait complete. 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1fb3e6640 1 Processor -- start 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1fb3e6640 1 -- start start 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1fb3e6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 0x7fb1f4197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1fb3e6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f41018d0 0x7fb1f4198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f8981640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f41018d0 0x7fb1f4198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f8981640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f41018d0 0x7fb1f4198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46638/0 (socket says 172.21.15.49:46638) 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1fb3e6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1f4102290 0x7fb1f419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f9182640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 0x7fb1f4197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f9182640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 0x7fb1f4197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60084/0 (socket says 172.21.15.49:60084) 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f8981640 1 -- 172.21.15.49:0/2174852220 learned_addr learned my addr 172.21.15.49:0/2174852220 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f410d4e0 con 0x7fb1f41018d0 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f410d360 con 0x7fb1f4102290 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f410d660 con 0x7fb1f41006e0 2024-03-18T00:50:42.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f9983640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1f4102290 0x7fb1f419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f8981640 1 -- 172.21.15.49:0/2174852220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 msgr2=0x7fb1f4197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f8981640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 0x7fb1f4197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f8981640 1 -- 172.21.15.49:0/2174852220 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1f4102290 msgr2=0x7fb1f419f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f8981640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1f4102290 0x7fb1f419f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f8981640 1 -- 172.21.15.49:0/2174852220 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f419f760 con 0x7fb1f41018d0 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.099+0000 7fb1f9182640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 0x7fb1f4197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.100+0000 7fb1f8981640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f41018d0 0x7fb1f4198080 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009c40 tx=0x7fb1e400b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.100+0000 7fb1ea7fc640 1 -- 172.21.15.49:0/2174852220 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4005020 con 0x7fb1f41018d0 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.100+0000 7fb1ea7fc640 1 -- 172.21.15.49:0/2174852220 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1e40056a0 con 0x7fb1f41018d0 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.100+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f4109540 con 0x7fb1f41018d0 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.100+0000 7fb1ea7fc640 1 -- 172.21.15.49:0/2174852220 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4019670 con 0x7fb1f41018d0 2024-03-18T00:50:42.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.100+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f4109a40 con 0x7fb1f41018d0 2024-03-18T00:50:42.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.102+0000 7fb1ea7fc640 1 -- 172.21.15.49:0/2174852220 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb1e40051c0 con 0x7fb1f41018d0 2024-03-18T00:50:42.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.103+0000 7fb1ea7fc640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e00a2800 0x7fb1e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1fdfce740 1 Processor -- start 2024-03-18T00:50:42.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1ea7fc640 1 -- 172.21.15.49:0/2174852220 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb1e40d2470 con 0x7fb1f41018d0 2024-03-18T00:50:42.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e00a2800 0x7fb1e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1fdfce740 1 -- start start 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 0x560b17d90d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d913b0 0x560b17d93840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1fdfce740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x560b17cf0490 con 0x560b17d913b0 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1fdfce740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560b17c98c90 con 0x560b17d8c240 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1fdfce740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560b17cec530 con 0x560b17d8b850 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9983640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d913b0 0x560b17d93840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60098/0 (socket says 172.21.15.49:60098) 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 -- 172.21.15.49:0/1126171578 learned_addr learned my addr 172.21.15.49:0/1126171578 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9983640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d913b0 0x560b17d93840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46654/0 (socket says 172.21.15.49:46654) 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f8981640 1 --2- 172.21.15.49:0/1126171578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 0x560b17d90d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e00a2800 0x7fb1e00a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb1f002c170 tx=0x7fb1f002df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 -- 172.21.15.49:0/1126171578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 msgr2=0x560b17d90d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:42.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 --2- 172.21.15.49:0/1126171578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 0x560b17d90d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 -- 172.21.15.49:0/1126171578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d913b0 msgr2=0x560b17d93840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:42.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 --2- 172.21.15.49:0/1126171578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d913b0 0x560b17d93840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.104+0000 7fb1f9182640 1 -- 172.21.15.49:0/1126171578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b17d53500 con 0x560b17d8b850 2024-03-18T00:50:42.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.105+0000 7fb1f9182640 1 --2- 172.21.15.49:0/1126171578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8bc70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb1f002c710 tx=0x7fb1f0047000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4be6a0d4f4f8d106 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:42.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.105+0000 7fb1df7fe640 1 -- 172.21.15.49:0/1126171578 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f004f070 con 0x560b17d8b850 2024-03-18T00:50:42.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.105+0000 7fb1df7fe640 1 -- 172.21.15.49:0/1126171578 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1f0004ba0 con 0x560b17d8b850 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.105+0000 7fb1df7fe640 1 -- 172.21.15.49:0/1126171578 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0005600 con 0x560b17d8b850 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 -- 172.21.15.49:0/1126171578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 msgr2=0x560b17d8bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/1126171578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8bc70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb1f002c710 tx=0x7fb1f0047000 comp rx=0 tx=0).stop 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 -- 172.21.15.49:0/1126171578 shutdown_connections 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/1126171578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 0x560b17d90d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/1126171578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d913b0 0x560b17d93840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/1126171578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8bc70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 -- 172.21.15.49:0/1126171578 >> 172.21.15.49:0/1126171578 conn(0x560b17d86dc0 msgr2=0x560b17d871c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 -- 172.21.15.49:0/1126171578 shutdown_connections 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 -- 172.21.15.49:0/1126171578 wait complete. 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 -- start start 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.106+0000 7fb1fdfce740 1 Processor -- start 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8b850 0x560b17d95cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 0x560b17d96230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d913b0 0x560b17d9d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1f9182640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8b850 0x560b17d95cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1fdfce740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x560b17d94300 con 0x560b17d8b850 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1f9182640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8b850 0x560b17d95cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46666/0 (socket says 172.21.15.49:46666) 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1f9182640 1 -- 172.21.15.49:0/3763125230 learned_addr learned my addr 172.21.15.49:0/3763125230 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1f8981640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 0x560b17d96230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560b17cac230 con 0x560b17d8c240 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560b17d94480 con 0x560b17d913b0 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1f9983640 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d913b0 0x560b17d9d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.107+0000 7fb1fdfce740 10 client.0 _ll_register_callbacks cb 0x560b17ce8010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1f8981640 1 -- 172.21.15.49:0/3763125230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d913b0 msgr2=0x560b17d9d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1f8981640 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d913b0 0x560b17d9d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1f8981640 1 -- 172.21.15.49:0/3763125230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8b850 msgr2=0x560b17d95cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:42.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1f8981640 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8b850 0x560b17d95cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1f8981640 1 -- 172.21.15.49:0/3763125230 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b17cbb100 con 0x560b17d8c240 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1f0004ba0 con 0x560b17d8c240 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1f8981640 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 0x560b17d96230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40cff50 tx=0x7fb1e409eaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc4eb9faf7f0a606 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1dcff9640 10 client.25457 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3763125230 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e40a2f40 con 0x560b17d8c240 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3763125230 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb1e40048c0 con 0x560b17d8c240 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3763125230 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb1e40a7590 con 0x560b17d8c240 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1fdfce740 10 client.25457 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.108+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({fsmap=25}) -- 0x560b17a043f0 con 0x560b17d8c240 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3763125230 <== mon.2 v2:172.21.15.106:3300/0 4 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fb1e40d2110 con 0x560b17d8c240 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1fdfce740 10 client.25457 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1fdfce740 10 client.25457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1fdfce740 20 client.25457 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:42.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560b17da1990 0x560b17da3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1fdfce740 4 client.25457 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b17cfd9d0 con 0x560b17da1990 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1f9182640 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560b17da1990 0x560b17da3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1f9182640 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560b17da1990 0x560b17da3e20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f854e8e3bbbbcec5 server_cookie=9c2549f6b7a3321c in_seq=0 out_seq=0 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.109+0000 7fb1dcff9640 10 client.25457 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3763125230 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560b17cfd9d0 con 0x560b17da1990 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1dcff9640 10 client.25457 handle_command_reply: tid=0 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 client.25457 shutdown 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 msgr2=0x560b17d96230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 0x560b17d96230 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb1e40cff50 tx=0x7fb1e409eaa0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 shutdown_connections 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560b17da1990 0x560b17da3e20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d8c240 0x560b17d96230 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1dcff9640 0 client.25457 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:42.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d913b0 0x560b17d9d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3763125230 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8b850 0x560b17d95cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 >> 172.21.15.49:0/3763125230 conn(0x560b17d86dc0 msgr2=0x560b17d85fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:42.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.110+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 shutdown_connections 2024-03-18T00:50:42.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.111+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3763125230 wait complete. 2024-03-18T00:50:42.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.111+0000 7fb1fdfce740 20 client.25457 trim_cache size 0 max 16384 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1fdfce740 1 Processor -- start 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1fdfce740 1 -- start start 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17da86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1fdfce740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41a6a90 con 0x560b17d8c240 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1fdfce740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41a85d0 con 0x560b17d913b0 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1fdfce740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41a7160 con 0x560b17d8b850 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1f9182640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1f8981640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1f9182640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60112/0 (socket says 172.21.15.49:60112) 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1f9182640 1 -- 172.21.15.49:0/3977112296 learned_addr learned my addr 172.21.15.49:0/3977112296 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.124+0000 7fb1f9983640 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17da86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1f9182640 1 -- 172.21.15.49:0/3977112296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 msgr2=0x560b17da86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1f9182640 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17da86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1f9182640 1 -- 172.21.15.49:0/3977112296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 msgr2=0x560b17d9d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1f9182640 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1f9182640 1 -- 172.21.15.49:0/3977112296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b17ace7d0 con 0x560b17d8b850 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1f8981640 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1f9983640 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17da86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1f9182640 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8a730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1f002c710 tx=0x7fb1f0048000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c685acc8ca11e794 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1deffd640 1 -- 172.21.15.49:0/3977112296 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f005c070 con 0x560b17d8b850 2024-03-18T00:50:42.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1deffd640 1 -- 172.21.15.49:0/3977112296 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1f0005850 con 0x560b17d8b850 2024-03-18T00:50:42.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.125+0000 7fb1deffd640 1 -- 172.21.15.49:0/3977112296 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f004f040 con 0x560b17d8b850 2024-03-18T00:50:42.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3977112296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 msgr2=0x560b17d8a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:42.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8a730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1f002c710 tx=0x7fb1f0048000 comp rx=0 tx=0).stop 2024-03-18T00:50:42.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3977112296 shutdown_connections 2024-03-18T00:50:42.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17da86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3977112296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d8a730 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3977112296 >> 172.21.15.49:0/3977112296 conn(0x560b17d86dc0 msgr2=0x560b17d8a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:42.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3977112296 shutdown_connections 2024-03-18T00:50:42.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3977112296 wait complete. 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 -- start start 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.126+0000 7fb1fdfce740 1 Processor -- start 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d9a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1fdfce740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f8981640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f9182640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d9a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f8981640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46684/0 (socket says 172.21.15.49:46684) 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f8981640 1 -- 172.21.15.49:0/3297828591 learned_addr learned my addr 172.21.15.49:0/3297828591 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17d9ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41a7f00 con 0x560b17d8c240 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41a7830 con 0x560b17d913b0 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f41a4a70 con 0x560b17d8b850 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1fdfce740 10 client.0 _ll_register_callbacks cb 0x7fb1e4004bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:42.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f9983640 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17d9ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f9182640 1 -- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 msgr2=0x560b17d9ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f9182640 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17d9ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f9182640 1 -- 172.21.15.49:0/3297828591 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 msgr2=0x560b17d9aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f9182640 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f9182640 1 -- 172.21.15.49:0/3297828591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b17d9c050 con 0x560b17d8b850 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f8981640 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.127+0000 7fb1f9983640 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17d9ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1f9182640 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d9a510 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb1f002c7b0 tx=0x7fb1f0005110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfdedee84280ded5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3297828591 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0062060 con 0x560b17d8b850 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3297828591 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb1f00046c0 con 0x560b17d8b850 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1f0005850 con 0x560b17d8b850 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3297828591 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fb1f004f040 con 0x560b17d8b850 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1fdfce740 10 client.25498 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:42.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560b17d9d330 con 0x560b17d8b850 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3297828591 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1f0005850 con 0x560b17d8b850 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.128+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3297828591 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7fb1f005d070 con 0x560b17d8b850 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1fdfce740 10 client.25498 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1fdfce740 10 client.25498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1fdfce740 20 client.25498 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560b17da1990 0x560b17e09e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1fdfce740 4 client.25498 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560b17cf7d20 con 0x560b17da1990 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1f8981640 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560b17da1990 0x560b17e09e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1dcff9640 10 client.25498 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1f8981640 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560b17da1990 0x560b17e09e40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d331b23f7aba0c9 server_cookie=315d8afca32a186c in_seq=0 out_seq=0 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1dcff9640 1 -- 172.21.15.49:0/3297828591 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560b17cf7d20 con 0x560b17da1990 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1dcff9640 10 client.25498 handle_command_reply: tid=0 2024-03-18T00:50:42.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1fdfce740 1 client.25498 shutdown 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 msgr2=0x560b17d9a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.129+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d9a510 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb1f002c7b0 tx=0x7fb1f0005110 comp rx=0 tx=0).stop 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 shutdown_connections 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560b17da1990 0x560b17e09e40 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1dcff9640 0 client.25498 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560b17d913b0 0x560b17d9ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560b17d8c240 0x560b17d9aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fdfce740 1 --2- 172.21.15.49:0/3297828591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560b17d8b850 0x560b17d9a510 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 >> 172.21.15.49:0/3297828591 conn(0x560b17d86dc0 msgr2=0x560b17d8a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 shutdown_connections 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fdfce740 1 -- 172.21.15.49:0/3297828591 wait complete. 2024-03-18T00:50:42.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fdfce740 20 client.25498 trim_cache size 0 max 16384 2024-03-18T00:50:42.136 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:50:42.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e00a2800 msgr2=0x7fb1e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:42.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e00a2800 0x7fb1e00a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb1f002c170 tx=0x7fb1f002df90 comp rx=0 tx=0).stop 2024-03-18T00:50:42.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f41018d0 msgr2=0x7fb1f4198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:42.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.130+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f41018d0 0x7fb1f4198080 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4009c40 tx=0x7fb1e400b690 comp rx=0 tx=0).stop 2024-03-18T00:50:42.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.131+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 shutdown_connections 2024-03-18T00:50:42.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.131+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e00a2800 0x7fb1e00a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.131+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1f4102290 0x7fb1f419f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.131+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1f41018d0 0x7fb1f4198080 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.131+0000 7fb1fb3e6640 1 --2- 172.21.15.49:0/2174852220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1f41006e0 0x7fb1f4197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:42.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.131+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 >> 172.21.15.49:0/2174852220 conn(0x7fb1f40fbc70 msgr2=0x7fb1f4104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:42.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.131+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 shutdown_connections 2024-03-18T00:50:42.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:42.131+0000 7fb1fb3e6640 1 -- 172.21.15.49:0/2174852220 wait complete. 2024-03-18T00:50:42.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:41 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3851362808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:50:42.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:41 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3851362808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:50:43.145 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:50:43.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:50:43.145 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:50:43.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.278+0000 7f86a33da640 1 -- 172.21.15.49:0/3910335551 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f869c102290 msgr2=0x7f869c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.278+0000 7f86a33da640 1 --2- 172.21.15.49:0/3910335551 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f869c102290 0x7f869c108b20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8694009ca0 tx=0x7f869402c0f0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.278+0000 7f86a33da640 1 -- 172.21.15.49:0/3910335551 shutdown_connections 2024-03-18T00:50:43.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.278+0000 7f86a33da640 1 --2- 172.21.15.49:0/3910335551 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f869c102290 0x7f869c108b20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.278+0000 7f86a33da640 1 --2- 172.21.15.49:0/3910335551 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f869c1018d0 0x7f869c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.278+0000 7f86a33da640 1 --2- 172.21.15.49:0/3910335551 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f869c1006e0 0x7f869c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.278+0000 7f86a33da640 1 -- 172.21.15.49:0/3910335551 >> 172.21.15.49:0/3910335551 conn(0x7f869c0fbc70 msgr2=0x7f869c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.278+0000 7f86a33da640 1 -- 172.21.15.49:0/3910335551 shutdown_connections 2024-03-18T00:50:43.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.278+0000 7f86a33da640 1 -- 172.21.15.49:0/3910335551 wait complete. 2024-03-18T00:50:43.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a33da640 1 Processor -- start 2024-03-18T00:50:43.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a33da640 1 -- start start 2024-03-18T00:50:43.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a33da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f869c1006e0 0x7f869c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a33da640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f869c1018d0 0x7f869c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a33da640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f869c102290 0x7f869c19f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a33da640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f869c10d480 con 0x7f869c102290 2024-03-18T00:50:43.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a33da640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f869c10d300 con 0x7f869c1018d0 2024-03-18T00:50:43.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a33da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f869c10d600 con 0x7f869c1006e0 2024-03-18T00:50:43.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a1176640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f869c1006e0 0x7f869c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a1977640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f869c102290 0x7f869c19f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a1176640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f869c1006e0 0x7f869c197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60152/0 (socket says 172.21.15.49:60152) 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a1176640 1 -- 172.21.15.49:0/795461992 learned_addr learned my addr 172.21.15.49:0/795461992 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.279+0000 7f86a0975640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f869c1018d0 0x7f869c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86a1176640 1 -- 172.21.15.49:0/795461992 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f869c1018d0 msgr2=0x7f869c197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86a1176640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f869c1018d0 0x7f869c197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86a1176640 1 -- 172.21.15.49:0/795461992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f869c102290 msgr2=0x7f869c19f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86a1176640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f869c102290 0x7f869c19f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86a1176640 1 -- 172.21.15.49:0/795461992 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f869c19f6d0 con 0x7f869c1006e0 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86a1176640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f869c1006e0 0x7f869c197ab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f869800e9c0 tx=0x7f869800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86927fc640 1 -- 172.21.15.49:0/795461992 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f869800cd80 con 0x7f869c1006e0 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86927fc640 1 -- 172.21.15.49:0/795461992 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86980035f0 con 0x7f869c1006e0 2024-03-18T00:50:43.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86a33da640 1 -- 172.21.15.49:0/795461992 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f869c109540 con 0x7f869c1006e0 2024-03-18T00:50:43.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86a33da640 1 -- 172.21.15.49:0/795461992 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f869c109a90 con 0x7f869c1006e0 2024-03-18T00:50:43.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.280+0000 7f86927fc640 1 -- 172.21.15.49:0/795461992 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698010690 con 0x7f869c1006e0 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.283+0000 7f86927fc640 1 -- 172.21.15.49:0/795461992 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f86980037a0 con 0x7f869c1006e0 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86927fc640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f86880a2800 0x7f86880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a5fbd740 1 Processor -- start 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a5fbd740 1 -- start start 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86927fc640 1 -- 172.21.15.49:0/795461992 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f86980cee90 con 0x7f869c1006e0 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a0975640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f86880a2800 0x7f86880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126505990 0x55d126505db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126506380 0x55d12650ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d12650b4f0 0x55d12650d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a5fbd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d12646a0d0 con 0x55d126506380 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a5fbd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d126412c70 con 0x55d126505990 2024-03-18T00:50:43.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a5fbd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d126466170 con 0x55d12650b4f0 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a1977640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d12650b4f0 0x55d12650d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a1176640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126505990 0x55d126505db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a1176640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126505990 0x55d126505db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36854/0 (socket says 172.21.15.49:36854) 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a1176640 1 -- 172.21.15.49:0/183046550 learned_addr learned my addr 172.21.15.49:0/183046550 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.284+0000 7f86a1977640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d12650b4f0 0x55d12650d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60156/0 (socket says 172.21.15.49:60156) 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86a1176640 1 -- 172.21.15.49:0/183046550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d12650b4f0 msgr2=0x55d12650d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86a0975640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f86880a2800 0x7f86880a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f869c198fc0 tx=0x7f868c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86a1176640 1 --2- 172.21.15.49:0/183046550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d12650b4f0 0x55d12650d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86a1176640 1 -- 172.21.15.49:0/183046550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126506380 msgr2=0x55d12650ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86a1176640 1 --2- 172.21.15.49:0/183046550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126506380 0x55d12650ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86a1176640 1 -- 172.21.15.49:0/183046550 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1262467e0 con 0x55d126505990 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86a1176640 1 --2- 172.21.15.49:0/183046550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126505990 0x55d126505db0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f86980a3a20 tx=0x7f869809beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22ee2fb4fb441073 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86877fe640 1 -- 172.21.15.49:0/183046550 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86980a3f50 con 0x55d126505990 2024-03-18T00:50:43.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86877fe640 1 -- 172.21.15.49:0/183046550 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8698099ab0 con 0x55d126505990 2024-03-18T00:50:43.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.285+0000 7f86877fe640 1 -- 172.21.15.49:0/183046550 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86980afb90 con 0x55d126505990 2024-03-18T00:50:43.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.286+0000 7f86a5fbd740 1 -- 172.21.15.49:0/183046550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126505990 msgr2=0x55d126505db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.286+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/183046550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126505990 0x55d126505db0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f86980a3a20 tx=0x7f869809beb0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.286+0000 7f86a5fbd740 1 -- 172.21.15.49:0/183046550 shutdown_connections 2024-03-18T00:50:43.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.286+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/183046550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d12650b4f0 0x55d12650d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.286+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/183046550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126506380 0x55d12650ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.286+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/183046550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126505990 0x55d126505db0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.286+0000 7f86a5fbd740 1 -- 172.21.15.49:0/183046550 >> 172.21.15.49:0/183046550 conn(0x55d1264cdc50 msgr2=0x55d126501300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.286+0000 7f86a5fbd740 1 -- 172.21.15.49:0/183046550 shutdown_connections 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.286+0000 7f86a5fbd740 1 -- 172.21.15.49:0/183046550 wait complete. 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.287+0000 7f86a5fbd740 1 -- start start 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.287+0000 7f86a5fbd740 1 Processor -- start 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.287+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.287+0000 7f86a1176640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.287+0000 7f86a1176640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46716/0 (socket says 172.21.15.49:46716) 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.287+0000 7f86a1176640 1 -- 172.21.15.49:0/649801034 learned_addr learned my addr 172.21.15.49:0/649801034 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.287+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d1265103c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 0x55d126517400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d12650e440 con 0x55d126505990 2024-03-18T00:50:43.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d1264260d0 con 0x55d12650b4f0 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d12650e5c0 con 0x55d126506380 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a5fbd740 10 client.0 _ll_register_callbacks cb 0x55d1263ef760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a0975640 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d1265103c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a1977640 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 0x55d126517400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a1977640 1 -- 172.21.15.49:0/649801034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 msgr2=0x55d1265103c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a1977640 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d1265103c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a1977640 1 -- 172.21.15.49:0/649801034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 msgr2=0x55d12650fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a1977640 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a1977640 1 -- 172.21.15.49:0/649801034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1262a6f70 con 0x55d12650b4f0 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a0975640 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d1265103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a1176640 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f86a1977640 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 0x55d126517400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f869402c630 tx=0x7f8694009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a136a2d80f224cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f8684ff9640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:50:43.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.288+0000 7f8684ff9640 1 -- 172.21.15.49:0/649801034 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694044070 con 0x55d12650b4f0 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f8684ff9640 1 -- 172.21.15.49:0/649801034 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8694032890 con 0x55d12650b4f0 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8698099ab0 con 0x55d12650b4f0 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f8684ff9640 1 -- 172.21.15.49:0/649801034 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8694039040 con 0x55d12650b4f0 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f86a5fbd740 10 client.25475 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d12613e170 con 0x55d12650b4f0 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f8684ff9640 1 -- 172.21.15.49:0/649801034 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8698099ab0 con 0x55d12650b4f0 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f8684ff9640 1 -- 172.21.15.49:0/649801034 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f8694004490 con 0x55d12650b4f0 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f86a5fbd740 10 client.25475 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f86a5fbd740 10 client.25475 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f86a5fbd740 20 client.25475 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d12651b890 0x55d12651dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f86a5fbd740 4 client.25475 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.289+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d126477610 con 0x55d12651b890 2024-03-18T00:50:43.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.290+0000 7f86a1176640 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d12651b890 0x55d12651dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.290+0000 7f86a1176640 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d12651b890 0x55d12651dd10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ca03b7ae4c2da39 server_cookie=3aae29a515a879d5 in_seq=0 out_seq=0 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.290+0000 7f8684ff9640 10 client.25475 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f8684ff9640 1 -- 172.21.15.49:0/649801034 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d126477610 con 0x55d12651b890 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f8684ff9640 10 client.25475 handle_command_reply: tid=0 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 client.25475 shutdown 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 msgr2=0x55d126517400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 0x55d126517400 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f869402c630 tx=0x7f8694009800 comp rx=0 tx=0).stop 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 shutdown_connections 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d12651b890 0x55d12651dd10 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 0x55d126517400 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d1265103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/649801034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 >> 172.21.15.49:0/649801034 conn(0x55d1264cdc50 msgr2=0x55d1265004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f8684ff9640 0 client.25475 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:43.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 shutdown_connections 2024-03-18T00:50:43.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 1 -- 172.21.15.49:0/649801034 wait complete. 2024-03-18T00:50:43.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.291+0000 7f86a5fbd740 20 client.25475 trim_cache size 0 max 16384 2024-03-18T00:50:43.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a5fbd740 1 Processor -- start 2024-03-18T00:50:43.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a5fbd740 1 -- start start 2024-03-18T00:50:43.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126506380 0x55d1265173c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d12650b4f0 0x55d1265103c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a5fbd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f869c1a6be0 con 0x55d12650b4f0 2024-03-18T00:50:43.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a5fbd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f869c1a8720 con 0x55d126506380 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a5fbd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f869c1a72b0 con 0x55d126505990 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a0975640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126506380 0x55d1265173c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a1176640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a1977640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d12650b4f0 0x55d1265103c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a1176640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60166/0 (socket says 172.21.15.49:60166) 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a0975640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126506380 0x55d1265173c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36864/0 (socket says 172.21.15.49:36864) 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.309+0000 7f86a1176640 1 -- 172.21.15.49:0/3920826167 learned_addr learned my addr 172.21.15.49:0/3920826167 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f86a0975640 1 -- 172.21.15.49:0/3920826167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126505990 msgr2=0x55d12650fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f86a0975640 1 --2- 172.21.15.49:0/3920826167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f86a0975640 1 -- 172.21.15.49:0/3920826167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d12650b4f0 msgr2=0x55d1265103c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f86a0975640 1 --2- 172.21.15.49:0/3920826167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d12650b4f0 0x55d1265103c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f86a0975640 1 -- 172.21.15.49:0/3920826167 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1265016d0 con 0x55d126506380 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f86a1977640 1 --2- 172.21.15.49:0/3920826167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d12650b4f0 0x55d1265103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f86a0975640 1 --2- 172.21.15.49:0/3920826167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126506380 0x55d1265173c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f868c009fd0 tx=0x7f868c022000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b060498c419b7ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f8686ffd640 1 -- 172.21.15.49:0/3920826167 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868c033070 con 0x55d126506380 2024-03-18T00:50:43.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f8686ffd640 1 -- 172.21.15.49:0/3920826167 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f868c0249c0 con 0x55d126506380 2024-03-18T00:50:43.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.310+0000 7f8686ffd640 1 -- 172.21.15.49:0/3920826167 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868c02e9b0 con 0x55d126506380 2024-03-18T00:50:43.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 -- 172.21.15.49:0/3920826167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126506380 msgr2=0x55d1265173c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/3920826167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126506380 0x55d1265173c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f868c009fd0 tx=0x7f868c022000 comp rx=0 tx=0).stop 2024-03-18T00:50:43.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 -- 172.21.15.49:0/3920826167 shutdown_connections 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/3920826167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d126506380 0x55d1265173c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/3920826167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d12650b4f0 0x55d1265103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/3920826167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126505990 0x55d12650fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 -- 172.21.15.49:0/3920826167 >> 172.21.15.49:0/3920826167 conn(0x55d1264cdc50 msgr2=0x55d126507920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 -- 172.21.15.49:0/3920826167 shutdown_connections 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 -- 172.21.15.49:0/3920826167 wait complete. 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.311+0000 7f86a5fbd740 1 -- start start 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a5fbd740 1 Processor -- start 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d126514600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d126514b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a5fbd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 0x55d126515ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a5fbd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f869c1a8050 con 0x55d126505990 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a1176640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d126514600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a1176640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d126514600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46736/0 (socket says 172.21.15.49:46736) 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a5fbd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f869c1a7980 con 0x55d12650b4f0 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a5fbd740 1 -- 172.21.15.49:0/2452232159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f869c1a4bc0 con 0x55d126506380 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a1176640 1 -- 172.21.15.49:0/2452232159 learned_addr learned my addr 172.21.15.49:0/2452232159 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a0975640 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d126514b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.312+0000 7f86a1977640 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 0x55d126515ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a5fbd740 10 client.0 _ll_register_callbacks cb 0x7f868c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a0975640 1 -- 172.21.15.49:0/2452232159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 msgr2=0x55d126515ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a0975640 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 0x55d126515ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a0975640 1 -- 172.21.15.49:0/2452232159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 msgr2=0x55d126514600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a0975640 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d126514600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a0975640 1 -- 172.21.15.49:0/2452232159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1265160d0 con 0x55d126506380 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a1176640 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d126514600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:43.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a0975640 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d126514b40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f868c0106b0 tx=0x7f868c024fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b934c9b78ca74b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f8684ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f8684ff9640 1 -- 172.21.15.49:0/2452232159 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f868c03c060 con 0x55d126506380 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a5fbd740 1 -- 172.21.15.49:0/2452232159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f868c0249c0 con 0x55d126506380 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f8684ff9640 1 -- 172.21.15.49:0/2452232159 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f868c0246f0 con 0x55d126506380 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f8684ff9640 1 -- 172.21.15.49:0/2452232159 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f868c033070 con 0x55d126506380 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a5fbd740 10 client.25516 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.313+0000 7f86a5fbd740 1 -- 172.21.15.49:0/2452232159 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d126517400 con 0x55d126506380 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f8684ff9640 1 -- 172.21.15.49:0/2452232159 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f868c0249c0 con 0x55d126506380 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f8684ff9640 1 -- 172.21.15.49:0/2452232159 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f868c037810 con 0x55d126506380 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f86a5fbd740 10 client.25516 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f86a5fbd740 10 client.25516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f86a5fbd740 20 client.25516 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:43.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d1265820a0 0x55d1265844c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f86a5fbd740 4 client.25516 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f86a5fbd740 1 -- 172.21.15.49:0/2452232159 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d126471960 con 0x55d1265820a0 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f86a1176640 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d1265820a0 0x55d1265844c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f86a1176640 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d1265820a0 0x55d1265844c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6913736b53e356c server_cookie=5e58fcf74f2d2d6b in_seq=0 out_seq=0 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.314+0000 7f8684ff9640 10 client.25516 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f8684ff9640 1 -- 172.21.15.49:0/2452232159 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d126471960 con 0x55d1265820a0 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f8684ff9640 10 client.25516 handle_command_reply: tid=0 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 client.25516 shutdown 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 -- 172.21.15.49:0/2452232159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 msgr2=0x55d126514b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d126514b40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f868c0106b0 tx=0x7f868c024fd0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 -- 172.21.15.49:0/2452232159 shutdown_connections 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d1265820a0 0x55d1265844c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f8684ff9640 0 client.25516 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:43.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d12650b4f0 0x55d126515ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d126506380 0x55d126514b40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 --2- 172.21.15.49:0/2452232159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d126505990 0x55d126514600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 -- 172.21.15.49:0/2452232159 >> 172.21.15.49:0/2452232159 conn(0x55d1264cdc50 msgr2=0x55d12650fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 -- 172.21.15.49:0/2452232159 shutdown_connections 2024-03-18T00:50:43.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.316+0000 7f86a5fbd740 1 -- 172.21.15.49:0/2452232159 wait complete. 2024-03-18T00:50:43.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a5fbd740 20 client.25516 trim_cache size 0 max 16384 2024-03-18T00:50:43.321 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:50:43.321 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:50:43.321 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "57944234-21d3-48b9-8709-6b3e9f9930ef", 2024-03-18T00:50:43.322 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:50:43.322 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:50:43.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 -- 172.21.15.49:0/795461992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f86880a2800 msgr2=0x7f86880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f86880a2800 0x7f86880a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f869c198fc0 tx=0x7f868c005f90 comp rx=0 tx=0).stop 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 -- 172.21.15.49:0/795461992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f869c1006e0 msgr2=0x7f869c197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f869c1006e0 0x7f869c197ab0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f869800e9c0 tx=0x7f869800ee80 comp rx=0 tx=0).stop 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 -- 172.21.15.49:0/795461992 shutdown_connections 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f86880a2800 0x7f86880a4cc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f869c1018d0 0x7f869c197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f869c102290 0x7f869c19f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 --2- 172.21.15.49:0/795461992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f869c1006e0 0x7f869c197ab0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 -- 172.21.15.49:0/795461992 >> 172.21.15.49:0/795461992 conn(0x7f869c0fbc70 msgr2=0x7f869c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 -- 172.21.15.49:0/795461992 shutdown_connections 2024-03-18T00:50:43.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.317+0000 7f86a33da640 1 -- 172.21.15.49:0/795461992 wait complete. 2024-03-18T00:50:43.327 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:42 smithi049 ceph-mon[32130]: pgmap v368: 97 pgs: 97 active+clean; 656 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T00:50:43.328 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:50:43.328 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:50:43.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:42 smithi107 ceph-mon[36270]: pgmap v368: 97 pgs: 97 active+clean; 656 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T00:50:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 -- 172.21.15.49:0/1903404917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741006e0 msgr2=0x7f9374100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 --2- 172.21.15.49:0/1903404917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741006e0 0x7f9374100b00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f937000fb60 tx=0x7f937002fcf0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 -- 172.21.15.49:0/1903404917 shutdown_connections 2024-03-18T00:50:43.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 --2- 172.21.15.49:0/1903404917 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9374102290 0x7f9374108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 --2- 172.21.15.49:0/1903404917 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f93741018d0 0x7f9374101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 --2- 172.21.15.49:0/1903404917 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741006e0 0x7f9374100b00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 -- 172.21.15.49:0/1903404917 >> 172.21.15.49:0/1903404917 conn(0x7f93740fbc70 msgr2=0x7f93740fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 -- 172.21.15.49:0/1903404917 shutdown_connections 2024-03-18T00:50:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 -- 172.21.15.49:0/1903404917 wait complete. 2024-03-18T00:50:43.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 Processor -- start 2024-03-18T00:50:43.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.440+0000 7f937b64b640 1 -- start start 2024-03-18T00:50:43.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f937b64b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93741006e0 0x7f9374195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f937b64b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741018d0 0x7f9374195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f937b64b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9374102290 0x7f937419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f937b64b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f937410b260 con 0x7f9374102290 2024-03-18T00:50:43.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f937b64b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f937410b0e0 con 0x7f93741006e0 2024-03-18T00:50:43.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f937b64b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f937410b3e0 con 0x7f93741018d0 2024-03-18T00:50:43.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f93793e7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93741006e0 0x7f9374195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f9378be6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741018d0 0x7f9374195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f9379be8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9374102290 0x7f937419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f93793e7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93741006e0 0x7f9374195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36900/0 (socket says 172.21.15.49:36900) 2024-03-18T00:50:43.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f9378be6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741018d0 0x7f9374195eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60200/0 (socket says 172.21.15.49:60200) 2024-03-18T00:50:43.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f93793e7640 1 -- 172.21.15.49:0/2289614504 learned_addr learned my addr 172.21.15.49:0/2289614504 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f9378be6640 1 -- 172.21.15.49:0/2289614504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93741006e0 msgr2=0x7f9374195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f9378be6640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93741006e0 0x7f9374195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f9378be6640 1 -- 172.21.15.49:0/2289614504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9374102290 msgr2=0x7f937419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f9378be6640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9374102290 0x7f937419cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f9378be6640 1 -- 172.21.15.49:0/2289614504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f937419d590 con 0x7f93741018d0 2024-03-18T00:50:43.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f93793e7640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93741006e0 0x7f9374195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:43.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.441+0000 7f9378be6640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741018d0 0x7f9374195eb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9364009a50 tx=0x7f9364009e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.442+0000 7f936a7fc640 1 -- 172.21.15.49:0/2289614504 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364004020 con 0x7f93741018d0 2024-03-18T00:50:43.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.442+0000 7f936a7fc640 1 -- 172.21.15.49:0/2289614504 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93640041c0 con 0x7f93741018d0 2024-03-18T00:50:43.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.442+0000 7f936a7fc640 1 -- 172.21.15.49:0/2289614504 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364003d30 con 0x7f93741018d0 2024-03-18T00:50:43.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.442+0000 7f937b64b640 1 -- 172.21.15.49:0/2289614504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f937419d880 con 0x7f93741018d0 2024-03-18T00:50:43.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.442+0000 7f937b64b640 1 -- 172.21.15.49:0/2289614504 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f937419dd80 con 0x7f93741018d0 2024-03-18T00:50:43.452 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:42 smithi106 ceph-mon[35899]: pgmap v368: 97 pgs: 97 active+clean; 656 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.445+0000 7f936a7fc640 1 -- 172.21.15.49:0/2289614504 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f936400c9e0 con 0x7f93741018d0 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.445+0000 7f937e235740 1 Processor -- start 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.445+0000 7f937e235740 1 -- start start 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.445+0000 7f937e235740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a54590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f937e235740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a59660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f937e235740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a5c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f937e235740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5637459b9500 con 0x563745a54b60 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f937e235740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563745961be0 con 0x563745a59cd0 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f937e235740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5637459b55a0 con 0x563745a54170 2024-03-18T00:50:43.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f93793e7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a59660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f936a7fc640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93600a2880 0x7f93600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f9379be8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a54590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f93793e7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a59660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46748/0 (socket says 172.21.15.49:46748) 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f93793e7640 1 -- 172.21.15.49:0/4177655696 learned_addr learned my addr 172.21.15.49:0/4177655696 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f9378be6640 1 --2- 172.21.15.49:0/4177655696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a5c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f9379be8640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93600a2880 0x7f93600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f936a7fc640 1 -- 172.21.15.49:0/2289614504 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f93640cde40 con 0x7f93741018d0 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f9378be6640 1 -- 172.21.15.49:0/4177655696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 msgr2=0x563745a54590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f9378be6640 1 --2- 172.21.15.49:0/4177655696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a54590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f9378be6640 1 -- 172.21.15.49:0/4177655696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 msgr2=0x563745a59660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f9378be6640 1 --2- 172.21.15.49:0/4177655696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a59660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f9378be6640 1 -- 172.21.15.49:0/4177655696 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5637459754b0 con 0x563745a59cd0 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f93793e7640 1 --2- 172.21.15.49:0/4177655696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a59660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:43.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.446+0000 7f9378be6640 1 --2- 172.21.15.49:0/4177655696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a5c160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f93640cb970 tx=0x7f936400df20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b763f1432778050 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.447+0000 7f935f7fe640 1 -- 172.21.15.49:0/4177655696 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936409aec0 con 0x563745a59cd0 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.447+0000 7f935f7fe640 1 -- 172.21.15.49:0/4177655696 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93640a2f60 con 0x563745a59cd0 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.447+0000 7f935f7fe640 1 -- 172.21.15.49:0/4177655696 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9364004490 con 0x563745a59cd0 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 -- 172.21.15.49:0/4177655696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 msgr2=0x563745a5c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f9379be8640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93600a2880 0x7f93600a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x563745a553c0 tx=0x7f936c015040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 --2- 172.21.15.49:0/4177655696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a5c160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f93640cb970 tx=0x7f936400df20 comp rx=0 tx=0).stop 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 -- 172.21.15.49:0/4177655696 shutdown_connections 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 --2- 172.21.15.49:0/4177655696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a5c160 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 --2- 172.21.15.49:0/4177655696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a59660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 --2- 172.21.15.49:0/4177655696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a54590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 -- 172.21.15.49:0/4177655696 >> 172.21.15.49:0/4177655696 conn(0x563745a4f6e0 msgr2=0x563745a4fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 -- 172.21.15.49:0/4177655696 shutdown_connections 2024-03-18T00:50:43.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 -- 172.21.15.49:0/4177655696 wait complete. 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 -- start start 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.448+0000 7f937e235740 1 Processor -- start 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f937e235740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a5e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f937e235740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54b60 0x563745a5eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f937e235740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a65bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f93793e7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a5e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f937e235740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563745a5cbc0 con 0x563745a54170 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f937e235740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563745975160 con 0x563745a59cd0 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f937e235740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563745a5cd40 con 0x563745a54b60 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f93793e7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a5e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46752/0 (socket says 172.21.15.49:46752) 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f93793e7640 1 -- 172.21.15.49:0/372748398 learned_addr learned my addr 172.21.15.49:0/372748398 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f9378be6640 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54b60 0x563745a5eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f937e235740 10 client.0 _ll_register_callbacks cb 0x563745959150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:43.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f9379be8640 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a65bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f93793e7640 1 -- 172.21.15.49:0/372748398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54b60 msgr2=0x563745a5eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f93793e7640 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54b60 0x563745a5eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f93793e7640 1 -- 172.21.15.49:0/372748398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 msgr2=0x563745a65bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f93793e7640 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a65bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f93793e7640 1 -- 172.21.15.49:0/372748398 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637457977d0 con 0x563745a54170 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f93793e7640 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a5e650 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f93700057c0 tx=0x7f93700389f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedc806001710cb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.449+0000 7f935cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f935cff9640 1 -- 172.21.15.49:0/372748398 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9370047070 con 0x563745a54170 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f935cff9640 1 -- 172.21.15.49:0/372748398 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9370030690 con 0x563745a54170 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f937e235740 1 -- 172.21.15.49:0/372748398 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93640a2f60 con 0x563745a54170 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f935cff9640 1 -- 172.21.15.49:0/372748398 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9370053050 con 0x563745a54170 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f937e235740 10 client.15657 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f937e235740 1 -- 172.21.15.49:0/372748398 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5637456cd3f0 con 0x563745a54170 2024-03-18T00:50:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f935cff9640 1 -- 172.21.15.49:0/372748398 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93640a2f60 con 0x563745a54170 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f935cff9640 1 -- 172.21.15.49:0/372748398 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f9370038ce0 con 0x563745a54170 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f937e235740 10 client.15657 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.450+0000 7f937e235740 10 client.15657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.451+0000 7f937e235740 20 client.15657 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.451+0000 7f937e235740 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563745a6a0c0 0x563745a6c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.451+0000 7f937e235740 4 client.15657 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.451+0000 7f937e235740 1 -- 172.21.15.49:0/372748398 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5637459c6a40 con 0x563745a6a0c0 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.451+0000 7f9378be6640 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563745a6a0c0 0x563745a6c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.451+0000 7f9378be6640 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563745a6a0c0 0x563745a6c550 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffad5a4e9d56603c server_cookie=710ab1d7261a2958 in_seq=0 out_seq=0 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.451+0000 7f935cff9640 10 client.15657 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.452+0000 7f935cff9640 1 -- 172.21.15.49:0/372748398 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5637459c6a40 con 0x563745a6a0c0 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.452+0000 7f935cff9640 10 client.15657 handle_command_reply: tid=0 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.452+0000 7f937e235740 1 client.15657 shutdown 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.452+0000 7f937e235740 1 -- 172.21.15.49:0/372748398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 msgr2=0x563745a5e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.452+0000 7f937e235740 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a5e650 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f93700057c0 tx=0x7f93700389f0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.452+0000 7f937e235740 1 -- 172.21.15.49:0/372748398 shutdown_connections 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.452+0000 7f937e235740 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563745a6a0c0 0x563745a6c550 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.452+0000 7f935cff9640 0 client.15657 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.453+0000 7f937e235740 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a65bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.453+0000 7f937e235740 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54b60 0x563745a5eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.453+0000 7f937e235740 1 --2- 172.21.15.49:0/372748398 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a5e650 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.453+0000 7f937e235740 1 -- 172.21.15.49:0/372748398 >> 172.21.15.49:0/372748398 conn(0x563745a4f6e0 msgr2=0x563745a4e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.453+0000 7f937e235740 1 -- 172.21.15.49:0/372748398 shutdown_connections 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.453+0000 7f937e235740 1 -- 172.21.15.49:0/372748398 wait complete. 2024-03-18T00:50:43.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.453+0000 7f937e235740 20 client.15657 trim_cache size 0 max 16384 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.487+0000 7f937e235740 1 Processor -- start 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.487+0000 7f937e235740 1 -- start start 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f937e235740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a4e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f937e235740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a65b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f937e235740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a4e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f937e235740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f93741a6b70 con 0x563745a54b60 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f937e235740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f93741a86b0 con 0x563745a59cd0 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f937e235740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f93741a7240 con 0x563745a54170 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f93793e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a4e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f9378be6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a65b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f93793e7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a4e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60224/0 (socket says 172.21.15.49:60224) 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f93793e7640 1 -- 172.21.15.49:0/170156357 learned_addr learned my addr 172.21.15.49:0/170156357 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f9379be8640 1 --2- 172.21.15.49:0/170156357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a4e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f93793e7640 1 -- 172.21.15.49:0/170156357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 msgr2=0x563745a4e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f93793e7640 1 --2- 172.21.15.49:0/170156357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a4e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f93793e7640 1 -- 172.21.15.49:0/170156357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 msgr2=0x563745a65b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f93793e7640 1 --2- 172.21.15.49:0/170156357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a65b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f93793e7640 1 -- 172.21.15.49:0/170156357 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56374568d0f0 con 0x563745a54170 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.488+0000 7f93793e7640 1 --2- 172.21.15.49:0/170156357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a4e070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9370036570 tx=0x7f937000f8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dbbeac281752cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.489+0000 7f935effd640 1 -- 172.21.15.49:0/170156357 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9370047040 con 0x563745a54170 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.489+0000 7f935effd640 1 -- 172.21.15.49:0/170156357 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937005c070 con 0x563745a54170 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.489+0000 7f935effd640 1 -- 172.21.15.49:0/170156357 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937003d040 con 0x563745a54170 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.489+0000 7f937e235740 1 -- 172.21.15.49:0/170156357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 msgr2=0x563745a4e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.489+0000 7f937e235740 1 --2- 172.21.15.49:0/170156357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a4e070 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9370036570 tx=0x7f937000f8b0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.490+0000 7f937e235740 1 -- 172.21.15.49:0/170156357 shutdown_connections 2024-03-18T00:50:43.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.490+0000 7f937e235740 1 --2- 172.21.15.49:0/170156357 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a59cd0 0x563745a4e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.490+0000 7f937e235740 1 --2- 172.21.15.49:0/170156357 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54b60 0x563745a65b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.490+0000 7f937e235740 1 --2- 172.21.15.49:0/170156357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a54170 0x563745a4e070 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.490+0000 7f937e235740 1 -- 172.21.15.49:0/170156357 >> 172.21.15.49:0/170156357 conn(0x563745a4f6e0 msgr2=0x563745a5d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.490+0000 7f937e235740 1 -- 172.21.15.49:0/170156357 shutdown_connections 2024-03-18T00:50:43.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.490+0000 7f937e235740 1 -- 172.21.15.49:0/170156357 wait complete. 2024-03-18T00:50:43.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.490+0000 7f937e235740 1 -- start start 2024-03-18T00:50:43.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.490+0000 7f937e235740 1 Processor -- start 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f937e235740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a62e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f937e235740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a54b60 0x563745a633b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f937e235740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a59cd0 0x563745a643c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f937e235740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f93741a7fe0 con 0x563745a54170 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f937e235740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f93741a7910 con 0x563745a54b60 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f937e235740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f93741a4b50 con 0x563745a59cd0 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f93793e7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a62e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f937e235740 10 client.0 _ll_register_callbacks cb 0x7f936c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f93793e7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a62e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46760/0 (socket says 172.21.15.49:46760) 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f93793e7640 1 -- 172.21.15.49:0/3003180459 learned_addr learned my addr 172.21.15.49:0/3003180459 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:43.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f9378be6640 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a54b60 0x563745a633b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f9379be8640 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a59cd0 0x563745a643c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f93793e7640 1 -- 172.21.15.49:0/3003180459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a59cd0 msgr2=0x563745a643c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f93793e7640 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a59cd0 0x563745a643c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f93793e7640 1 -- 172.21.15.49:0/3003180459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a54b60 msgr2=0x563745a633b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f93793e7640 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a54b60 0x563745a633b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f93793e7640 1 -- 172.21.15.49:0/3003180459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5637459809d0 con 0x563745a54170 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f9378be6640 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a54b60 0x563745a633b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f93793e7640 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a62e70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f937002fd30 tx=0x7f9370038be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21481a348441161d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.491+0000 7f935cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f935cff9640 1 -- 172.21.15.49:0/3003180459 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937003d040 con 0x563745a54170 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f935cff9640 1 -- 172.21.15.49:0/3003180459 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9370003bd0 con 0x563745a54170 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f937e235740 1 -- 172.21.15.49:0/3003180459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f937005c070 con 0x563745a54170 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f935cff9640 1 -- 172.21.15.49:0/3003180459 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9370047040 con 0x563745a54170 2024-03-18T00:50:43.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f937e235740 10 client.15663 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f937e235740 1 -- 172.21.15.49:0/3003180459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563745a65af0 con 0x563745a54170 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f935cff9640 1 -- 172.21.15.49:0/3003180459 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f937005c070 con 0x563745a54170 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f935cff9640 1 -- 172.21.15.49:0/3003180459 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f9370038ce0 con 0x563745a54170 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f937e235740 10 client.15663 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f937e235740 10 client.15663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f937e235740 20 client.15663 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f937e235740 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563745a6a0c0 0x563745ad2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f937e235740 4 client.15663 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.492+0000 7f937e235740 1 -- 172.21.15.49:0/3003180459 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5637459c0d90 con 0x563745a6a0c0 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f9378be6640 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563745a6a0c0 0x563745ad2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f935cff9640 10 client.15663 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f9378be6640 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563745a6a0c0 0x563745ad2760 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cd7e1cc2ef504fb server_cookie=c9739eaae5a0bd8f in_seq=0 out_seq=0 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f935cff9640 1 -- 172.21.15.49:0/3003180459 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5637459c0d90 con 0x563745a6a0c0 2024-03-18T00:50:43.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f935cff9640 10 client.15663 handle_command_reply: tid=0 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 client.15663 shutdown 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 -- 172.21.15.49:0/3003180459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 msgr2=0x563745a62e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a62e70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f937002fd30 tx=0x7f9370038be0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 -- 172.21.15.49:0/3003180459 shutdown_connections 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563745a6a0c0 0x563745ad2760 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563745a54b60 0x563745a633b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563745a59cd0 0x563745a643c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 --2- 172.21.15.49:0/3003180459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563745a54170 0x563745a62e70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 -- 172.21.15.49:0/3003180459 >> 172.21.15.49:0/3003180459 conn(0x563745a4f6e0 msgr2=0x563745a52940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f935cff9640 0 client.15663 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 -- 172.21.15.49:0/3003180459 shutdown_connections 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.493+0000 7f937e235740 1 -- 172.21.15.49:0/3003180459 wait complete. 2024-03-18T00:50:43.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937e235740 20 client.15663 trim_cache size 0 max 16384 2024-03-18T00:50:43.500 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:50:43.500 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T00:50:43.500 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:50:43.500 INFO:teuthology.orchestra.run.smithi049.stdout: "57944234-21d3-48b9-8709-6b3e9f9930ef": { 2024-03-18T00:50:43.500 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:50:43.500 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "57944234-21d3-48b9-8709-6b3e9f9930ef", 2024-03-18T00:50:43.500 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:50:43.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:50:43.500 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:50:43.501 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:50:43.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 -- 172.21.15.49:0/2289614504 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93600a2880 msgr2=0x7f93600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93600a2880 0x7f93600a4d40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x563745a553c0 tx=0x7f936c015040 comp rx=0 tx=0).stop 2024-03-18T00:50:43.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 -- 172.21.15.49:0/2289614504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741018d0 msgr2=0x7f9374195eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:43.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741018d0 0x7f9374195eb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9364009a50 tx=0x7f9364009e50 comp rx=0 tx=0).stop 2024-03-18T00:50:43.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 -- 172.21.15.49:0/2289614504 shutdown_connections 2024-03-18T00:50:43.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93600a2880 0x7f93600a4d40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9374102290 0x7f937419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93741018d0 0x7f9374195eb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 --2- 172.21.15.49:0/2289614504 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f93741006e0 0x7f9374195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:43.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 -- 172.21.15.49:0/2289614504 >> 172.21.15.49:0/2289614504 conn(0x7f93740fbc70 msgr2=0x7f9374104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:43.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 -- 172.21.15.49:0/2289614504 shutdown_connections 2024-03-18T00:50:43.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:50:43.494+0000 7f937b64b640 1 -- 172.21.15.49:0/2289614504 wait complete. 2024-03-18T00:50:43.505 INFO:tasks.cephfs.filesystem:scrub status for tag:57944234-21d3-48b9-8709-6b3e9f9930ef - {'path': '/', 'tag': '57944234-21d3-48b9-8709-6b3e9f9930ef', 'options': 'recursive,force'} 2024-03-18T00:50:44.331 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:43 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:50:44.331 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:43 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:44.331 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:43 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:50:44.331 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:43 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:44.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:43 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:50:44.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:43 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:44.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:43 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:50:44.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:43 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:44.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:43 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:50:44.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:43 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:44.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:43 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:50:44.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:43 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:50:45.340 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:44 smithi107 ceph-mon[36270]: pgmap v369: 97 pgs: 97 active+clean; 656 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T00:50:45.340 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:44 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:50:45.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:44 smithi049 ceph-mon[32130]: pgmap v369: 97 pgs: 97 active+clean; 656 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T00:50:45.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:44 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:50:45.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:44 smithi106 ceph-mon[35899]: pgmap v369: 97 pgs: 97 active+clean; 656 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T00:50:45.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:44 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:50:47.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:46 smithi049 ceph-mon[32130]: pgmap v370: 97 pgs: 97 active+clean; 665 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T00:50:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:46 smithi107 ceph-mon[36270]: pgmap v370: 97 pgs: 97 active+clean; 665 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T00:50:47.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:46 smithi106 ceph-mon[35899]: pgmap v370: 97 pgs: 97 active+clean; 665 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T00:50:49.247 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:48 smithi106 ceph-mon[35899]: pgmap v371: 97 pgs: 97 active+clean; 665 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T00:50:49.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:48 smithi049 ceph-mon[32130]: pgmap v371: 97 pgs: 97 active+clean; 665 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T00:50:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:48 smithi107 ceph-mon[36270]: pgmap v371: 97 pgs: 97 active+clean; 665 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T00:50:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 -- 172.21.15.49:0/1772595321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27d01022e0 msgr2=0x7f27d01046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 --2- 172.21.15.49:0/1772595321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27d01022e0 0x7f27d01046d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009860 tx=0x7f27c402bb70 comp rx=0 tx=0).stop 2024-03-18T00:50:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 -- 172.21.15.49:0/1772595321 shutdown_connections 2024-03-18T00:50:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 --2- 172.21.15.49:0/1772595321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27d0104c10 0x7f27d0107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 --2- 172.21.15.49:0/1772595321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27d0107540 0x7f27d0109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 --2- 172.21.15.49:0/1772595321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27d01022e0 0x7f27d01046d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 -- 172.21.15.49:0/1772595321 >> 172.21.15.49:0/1772595321 conn(0x7f27d00fbc70 msgr2=0x7f27d00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 -- 172.21.15.49:0/1772595321 shutdown_connections 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 -- 172.21.15.49:0/1772595321 wait complete. 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 Processor -- start 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 -- start start 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.882+0000 7f27d7176640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27d01022e0 0x7f27d0197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d7176640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27d0104c10 0x7f27d0198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d7176640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27d0107540 0x7f27d019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d7176640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f27d010d470 con 0x7f27d01022e0 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d7176640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f27d010d2f0 con 0x7f27d0104c10 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d7176640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27d010d5f0 con 0x7f27d0107540 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d4f12640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27d01022e0 0x7f27d0197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27cffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27d0104c10 0x7f27d0198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27cffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27d0104c10 0x7f27d0198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:56386/0 (socket says 172.21.15.49:56386) 2024-03-18T00:50:49.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d4f12640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27d01022e0 0x7f27d0197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55652/0 (socket says 172.21.15.49:55652) 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27cffff640 1 -- 172.21.15.49:0/2960944832 learned_addr learned my addr 172.21.15.49:0/2960944832 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27cffff640 1 -- 172.21.15.49:0/2960944832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27d0107540 msgr2=0x7f27d019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d5713640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27d0107540 0x7f27d019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27cffff640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27d0107540 0x7f27d019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27cffff640 1 -- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27d01022e0 msgr2=0x7f27d0197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27cffff640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27d01022e0 0x7f27d0197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27cffff640 1 -- 172.21.15.49:0/2960944832 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d019f700 con 0x7f27d0104c10 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d5713640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27d0107540 0x7f27d019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27d4f12640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27d01022e0 0x7f27d0197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.883+0000 7f27cffff640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27d0104c10 0x7f27d0198020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00e9a0 tx=0x7f27bc00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.884+0000 7f27cdffb640 1 -- 172.21.15.49:0/2960944832 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00cd80 con 0x7f27d0104c10 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.884+0000 7f27d7176640 1 -- 172.21.15.49:0/2960944832 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d019f9f0 con 0x7f27d0104c10 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.884+0000 7f27cdffb640 1 -- 172.21.15.49:0/2960944832 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27bc0035f0 con 0x7f27d0104c10 2024-03-18T00:50:49.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.884+0000 7f27cdffb640 1 -- 172.21.15.49:0/2960944832 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc010670 con 0x7f27d0104c10 2024-03-18T00:50:49.887 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.884+0000 7f27d7176640 1 -- 172.21.15.49:0/2960944832 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d019fef0 con 0x7f27d0104c10 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.886+0000 7f27d9e34740 1 Processor -- start 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.886+0000 7f27d9e34740 1 -- start start 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2b340 0x564bbfb2b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 0x564bbfb30830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb30ea0 0x564bbfb33330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d9e34740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564bbfa90a40 con 0x564bbfb30ea0 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d9e34740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x564bbfa39920 con 0x564bbfb2b340 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d9e34740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564bbfa8cae0 con 0x564bbfb2bd30 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27cdffb640 1 -- 172.21.15.49:0/2960944832 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f27bc0037a0 con 0x7f27d0104c10 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27cffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb30ea0 0x564bbfb33330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d5713640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2b340 0x564bbfb2b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d5713640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2b340 0x564bbfb2b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:56400/0 (socket says 172.21.15.49:56400) 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27cffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb30ea0 0x564bbfb33330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55658/0 (socket says 172.21.15.49:55658) 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d5713640 1 -- 172.21.15.49:0/3486546480 learned_addr learned my addr 172.21.15.49:0/3486546480 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d4f12640 1 --2- 172.21.15.49:0/3486546480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 0x564bbfb30830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d5713640 1 -- 172.21.15.49:0/3486546480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 msgr2=0x564bbfb30830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d5713640 1 --2- 172.21.15.49:0/3486546480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 0x564bbfb30830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d5713640 1 -- 172.21.15.49:0/3486546480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb30ea0 msgr2=0x564bbfb33330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d5713640 1 --2- 172.21.15.49:0/3486546480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb30ea0 0x564bbfb33330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.887+0000 7f27d5713640 1 -- 172.21.15.49:0/3486546480 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bbfa51460 con 0x564bbfb2b340 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.888+0000 7f27d5713640 1 --2- 172.21.15.49:0/3486546480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2b340 0x564bbfb2b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009330 tx=0x7f27c80108a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa36a08fa7f693c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.888+0000 7f27cdffb640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27c00a28a0 0x7f27c00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.888+0000 7f27baffd640 1 -- 172.21.15.49:0/3486546480 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8004b20 con 0x564bbfb2b340 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.888+0000 7f27baffd640 1 -- 172.21.15.49:0/3486546480 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c8004cc0 con 0x564bbfb2b340 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.888+0000 7f27d4f12640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27c00a28a0 0x7f27c00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.888+0000 7f27cdffb640 1 -- 172.21.15.49:0/2960944832 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f27bc0ceda0 con 0x7f27d0104c10 2024-03-18T00:50:49.890 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.888+0000 7f27baffd640 1 -- 172.21.15.49:0/3486546480 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8005520 con 0x564bbfb2b340 2024-03-18T00:50:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 -- 172.21.15.49:0/3486546480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2b340 msgr2=0x564bbfb2b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d4f12640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27c00a28a0 0x7f27c00a4d60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f27c402c180 tx=0x7f27c402e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3486546480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2b340 0x564bbfb2b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009330 tx=0x7f27c80108a0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 -- 172.21.15.49:0/3486546480 shutdown_connections 2024-03-18T00:50:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3486546480 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb30ea0 0x564bbfb33330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3486546480 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 0x564bbfb30830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3486546480 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2b340 0x564bbfb2b760 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.893 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 -- 172.21.15.49:0/3486546480 >> 172.21.15.49:0/3486546480 conn(0x564bbfb268b0 msgr2=0x564bbfb26cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 -- 172.21.15.49:0/3486546480 shutdown_connections 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 -- 172.21.15.49:0/3486546480 wait complete. 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 -- start start 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.889+0000 7f27d9e34740 1 Processor -- start 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb35730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb35c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb3cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d9e34740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564bbfb33e50 con 0x564bbfb2b340 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d9e34740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x564bbfa4cd70 con 0x564bbfb2bd30 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d9e34740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564bbfb33fd0 con 0x564bbfb30ea0 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d9e34740 10 client.0 _ll_register_callbacks cb 0x564bbf7a4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d4f12640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb35730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.894 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d5713640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb3cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d5713640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb3cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:35576/0 (socket says 172.21.15.49:35576) 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d5713640 1 -- 172.21.15.49:0/3940046328 learned_addr learned my addr 172.21.15.49:0/3940046328 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27cffff640 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb35c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d5713640 1 -- 172.21.15.49:0/3940046328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 msgr2=0x564bbfb35c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d5713640 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb35c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d5713640 1 -- 172.21.15.49:0/3940046328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 msgr2=0x564bbfb35730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d5713640 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb35730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.890+0000 7f27d5713640 1 -- 172.21.15.49:0/3940046328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bbf7a5210 con 0x564bbfb30ea0 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27d5713640 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb3cc40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009a60 tx=0x7f27c8005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a251eda6d8506b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27affff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27affff640 1 -- 172.21.15.49:0/3940046328 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8027020 con 0x564bbfb30ea0 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27affff640 1 -- 172.21.15.49:0/3940046328 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f27c8004ac0 con 0x564bbfb30ea0 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27d9e34740 1 -- 172.21.15.49:0/3940046328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c8004cc0 con 0x564bbfb30ea0 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27affff640 1 -- 172.21.15.49:0/3940046328 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f27c801c040 con 0x564bbfb30ea0 2024-03-18T00:50:49.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27d9e34740 10 client.25543 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27d9e34740 1 -- 172.21.15.49:0/3940046328 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564bbf871620 con 0x564bbfb30ea0 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27affff640 1 -- 172.21.15.49:0/3940046328 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c8004cc0 con 0x564bbfb30ea0 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.891+0000 7f27affff640 1 -- 172.21.15.49:0/3940046328 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f27c800b040 con 0x564bbfb30ea0 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.892+0000 7f27d9e34740 10 client.25543 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.892+0000 7f27d9e34740 10 client.25543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.892+0000 7f27d9e34740 20 client.25543 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.892+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564bbfb41300 0x564bbfb436f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.892+0000 7f27d9e34740 4 client.25543 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.892+0000 7f27d9e34740 1 -- 172.21.15.49:0/3940046328 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bbfa9df80 con 0x564bbfb41300 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.892+0000 7f27d4f12640 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564bbfb41300 0x564bbfb436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.892+0000 7f27affff640 10 client.25543 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.892+0000 7f27d4f12640 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564bbfb41300 0x564bbfb436f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89eb14b0a98ca588 server_cookie=79a78d8767dd04b3 in_seq=0 out_seq=0 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27affff640 1 -- 172.21.15.49:0/3940046328 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564bbfa9df80 con 0x564bbfb41300 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27affff640 10 client.25543 handle_command_reply: tid=0 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 client.25543 shutdown 2024-03-18T00:50:49.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 -- 172.21.15.49:0/3940046328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 msgr2=0x564bbfb3cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb3cc40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009a60 tx=0x7f27c8005a90 comp rx=0 tx=0).stop 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 -- 172.21.15.49:0/3940046328 shutdown_connections 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564bbfb41300 0x564bbfb436f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27affff640 0 client.25543 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb35c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb3cc40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3940046328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb35730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 -- 172.21.15.49:0/3940046328 >> 172.21.15.49:0/3940046328 conn(0x564bbfb268b0 msgr2=0x564bbfb25bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 -- 172.21.15.49:0/3940046328 shutdown_connections 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.893+0000 7f27d9e34740 1 -- 172.21.15.49:0/3940046328 wait complete. 2024-03-18T00:50:49.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.894+0000 7f27d9e34740 20 client.25543 trim_cache size 0 max 16384 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d9e34740 1 Processor -- start 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d9e34740 1 -- start start 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb25320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb3cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb25860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d9e34740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b1610 con 0x564bbfb2b340 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d9e34740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b3150 con 0x564bbfb2bd30 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d9e34740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b1ce0 con 0x564bbfb30ea0 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27cffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb3cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d4f12640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb25320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d5713640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb25860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d4f12640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb25320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55668/0 (socket says 172.21.15.49:55668) 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27cffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb3cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:56426/0 (socket says 172.21.15.49:56426) 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.922+0000 7f27d4f12640 1 -- 172.21.15.49:0/2237919376 learned_addr learned my addr 172.21.15.49:0/2237919376 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27d5713640 1 -- 172.21.15.49:0/2237919376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 msgr2=0x564bbfb3cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27d5713640 1 --2- 172.21.15.49:0/2237919376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb3cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27d5713640 1 -- 172.21.15.49:0/2237919376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 msgr2=0x564bbfb25320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27d5713640 1 --2- 172.21.15.49:0/2237919376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb25320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27d5713640 1 -- 172.21.15.49:0/2237919376 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bbfb30bc0 con 0x564bbfb30ea0 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27cffff640 1 --2- 172.21.15.49:0/2237919376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb3cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27d5713640 1 --2- 172.21.15.49:0/2237919376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb25860 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f27c8000f80 tx=0x7f27c80108a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38a142eae3dc5651 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27ba7fc640 1 -- 172.21.15.49:0/2237919376 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8027070 con 0x564bbfb30ea0 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27ba7fc640 1 -- 172.21.15.49:0/2237919376 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c8005090 con 0x564bbfb30ea0 2024-03-18T00:50:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.923+0000 7f27ba7fc640 1 -- 172.21.15.49:0/2237919376 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c8032030 con 0x564bbfb30ea0 2024-03-18T00:50:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 -- 172.21.15.49:0/2237919376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 msgr2=0x564bbfb25860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 --2- 172.21.15.49:0/2237919376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb25860 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f27c8000f80 tx=0x7f27c80108a0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 -- 172.21.15.49:0/2237919376 shutdown_connections 2024-03-18T00:50:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 --2- 172.21.15.49:0/2237919376 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb2bd30 0x564bbfb3cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 --2- 172.21.15.49:0/2237919376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb30ea0 0x564bbfb25860 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 --2- 172.21.15.49:0/2237919376 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb25320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.927 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 -- 172.21.15.49:0/2237919376 >> 172.21.15.49:0/2237919376 conn(0x564bbfb268b0 msgr2=0x564bbfb35730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 -- 172.21.15.49:0/2237919376 shutdown_connections 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 -- 172.21.15.49:0/2237919376 wait complete. 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 -- start start 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.924+0000 7f27d9e34740 1 Processor -- start 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb3a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 0x564bbfb3a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d9e34740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb30ea0 0x564bbfb3b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d9e34740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b2a80 con 0x564bbfb2b340 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d9e34740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f27d01b23b0 con 0x564bbfb30ea0 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d9e34740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27d0071e60 con 0x564bbfb2bd30 2024-03-18T00:50:49.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d9e34740 10 client.0 _ll_register_callbacks cb 0x7f27bc0d2750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d4f12640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb3a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d5713640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb30ea0 0x564bbfb3b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d4f12640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb3a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55674/0 (socket says 172.21.15.49:55674) 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d4f12640 1 -- 172.21.15.49:0/3372651416 learned_addr learned my addr 172.21.15.49:0/3372651416 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27cffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 0x564bbfb3a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d4f12640 1 -- 172.21.15.49:0/3372651416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 msgr2=0x564bbfb3a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d4f12640 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 0x564bbfb3a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d4f12640 1 -- 172.21.15.49:0/3372651416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb30ea0 msgr2=0x564bbfb3b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d4f12640 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb30ea0 0x564bbfb3b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d4f12640 1 -- 172.21.15.49:0/3372651416 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bbfb3bbf0 con 0x564bbfb2b340 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.925+0000 7f27d5713640 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb30ea0 0x564bbfb3b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27d4f12640 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb3a0b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f27c40349e0 tx=0x7f27c402bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683b819c94883fe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27affff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27affff640 1 -- 172.21.15.49:0/3372651416 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4059070 con 0x564bbfb2b340 2024-03-18T00:50:49.929 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27affff640 1 -- 172.21.15.49:0/3372651416 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f27c40543d0 con 0x564bbfb2b340 2024-03-18T00:50:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27d9e34740 1 -- 172.21.15.49:0/3372651416 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c8005090 con 0x564bbfb2b340 2024-03-18T00:50:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27affff640 1 -- 172.21.15.49:0/3372651416 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f27c4039040 con 0x564bbfb2b340 2024-03-18T00:50:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27d9e34740 10 client.15681 fetch_fsmap learned FSMap version 25 2024-03-18T00:50:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27d9e34740 1 -- 172.21.15.49:0/3372651416 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564bbfb3cf70 con 0x564bbfb2b340 2024-03-18T00:50:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27affff640 1 -- 172.21.15.49:0/3372651416 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27c8005090 con 0x564bbfb2b340 2024-03-18T00:50:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.926+0000 7f27affff640 1 -- 172.21.15.49:0/3372651416 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f27c404b040 con 0x564bbfb2b340 2024-03-18T00:50:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.927+0000 7f27d9e34740 10 client.15681 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:50:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.927+0000 7f27d9e34740 10 client.15681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:50:49.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.927+0000 7f27d9e34740 20 client.15681 populate_metadata read hostname 'smithi049' 2024-03-18T00:50:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.927+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564bbfb41300 0x564bbfba9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:50:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.927+0000 7f27d9e34740 4 client.15681 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:50:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.927+0000 7f27d9e34740 1 -- 172.21.15.49:0/3372651416 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564bbfa982d0 con 0x564bbfb41300 2024-03-18T00:50:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.927+0000 7f27cffff640 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564bbfb41300 0x564bbfba9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:50:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.927+0000 7f27affff640 10 client.15681 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:49.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.927+0000 7f27cffff640 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564bbfb41300 0x564bbfba9d60 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=500e82c3111cee15 server_cookie=8d7dc9cec4a35dde in_seq=0 out_seq=0 2024-03-18T00:50:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27affff640 1 -- 172.21.15.49:0/3372651416 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564bbfa982d0 con 0x564bbfb41300 2024-03-18T00:50:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27affff640 10 client.15681 handle_command_reply: tid=0 2024-03-18T00:50:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 client.15681 shutdown 2024-03-18T00:50:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 -- 172.21.15.49:0/3372651416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 msgr2=0x564bbfb3a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb3a0b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f27c40349e0 tx=0x7f27c402bb70 comp rx=0 tx=0).stop 2024-03-18T00:50:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 -- 172.21.15.49:0/3372651416 shutdown_connections 2024-03-18T00:50:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564bbfb41300 0x564bbfba9d60 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bbfb30ea0 0x564bbfb3b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bbfb2bd30 0x564bbfb3a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27affff640 0 client.15681 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 --2- 172.21.15.49:0/3372651416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bbfb2b340 0x564bbfb3a0b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 -- 172.21.15.49:0/3372651416 >> 172.21.15.49:0/3372651416 conn(0x564bbfb268b0 msgr2=0x564bbfb29b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 -- 172.21.15.49:0/3372651416 shutdown_connections 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 1 -- 172.21.15.49:0/3372651416 wait complete. 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.934+0000 7f27d9e34740 20 client.15681 trim_cache size 0 max 16384 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:50:49.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:50:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 -- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27c00a28a0 msgr2=0x7f27c00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27c00a28a0 0x7f27c00a4d60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f27c402c180 tx=0x7f27c402e040 comp rx=0 tx=0).stop 2024-03-18T00:50:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 -- 172.21.15.49:0/2960944832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27d0104c10 msgr2=0x7f27d0198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:50:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27d0104c10 0x7f27d0198020 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00e9a0 tx=0x7f27bc00ee60 comp rx=0 tx=0).stop 2024-03-18T00:50:49.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 -- 172.21.15.49:0/2960944832 shutdown_connections 2024-03-18T00:50:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27c00a28a0 0x7f27c00a4d60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27d0104c10 0x7f27d0198020 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27d0107540 0x7f27d019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 --2- 172.21.15.49:0/2960944832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27d01022e0 0x7f27d0197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:50:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 -- 172.21.15.49:0/2960944832 >> 172.21.15.49:0/2960944832 conn(0x7f27d00fbc70 msgr2=0x7f27d0102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:50:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 -- 172.21.15.49:0/2960944832 shutdown_connections 2024-03-18T00:50:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:50:49.935+0000 7f27d7176640 1 -- 172.21.15.49:0/2960944832 wait complete. 2024-03-18T00:50:51.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:50 smithi049 ceph-mon[32130]: pgmap v372: 97 pgs: 97 active+clean; 665 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T00:50:51.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:50 smithi107 ceph-mon[36270]: pgmap v372: 97 pgs: 97 active+clean; 665 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T00:50:51.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:50 smithi106 ceph-mon[35899]: pgmap v372: 97 pgs: 97 active+clean; 665 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-18T00:50:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:52 smithi049 ceph-mon[32130]: pgmap v373: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-18T00:50:53.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:52 smithi107 ceph-mon[36270]: pgmap v373: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-18T00:50:53.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:52 smithi106 ceph-mon[35899]: pgmap v373: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-03-18T00:50:55.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:54 smithi049 ceph-mon[32130]: pgmap v374: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-18T00:50:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:54 smithi107 ceph-mon[36270]: pgmap v374: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-18T00:50:55.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:54 smithi106 ceph-mon[35899]: pgmap v374: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-18T00:50:57.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:57 smithi107 ceph-mon[36270]: pgmap v375: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-18T00:50:57.346 INFO:tasks.workunit.client.0.smithi049.stderr:Note: switching to 'c382027d5c66ef4e0aa1131ea3e8080c658d3184'. 2024-03-18T00:50:57.346 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:57.346 INFO:tasks.workunit.client.0.smithi049.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T00:50:57.346 INFO:tasks.workunit.client.0.smithi049.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr: git switch -c 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr:Or undo this operation with: 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr: git switch - 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:57.347 INFO:tasks.workunit.client.0.smithi049.stderr:HEAD is now at c382027d5c6 ignorelist mon down 2024-03-18T00:50:57.351 DEBUG:teuthology.orchestra.run.smithi049:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-18T00:50:57.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:57 smithi049 ceph-mon[32130]: pgmap v375: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-18T00:50:57.375 INFO:tasks.workunit.client.0.smithi049.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T00:50:57.377 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T00:50:57.377 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T00:50:57.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:57 smithi106 ceph-mon[35899]: pgmap v375: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-18T00:50:58.150 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T00:50:58.220 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T00:50:58.264 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T00:50:58.265 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T00:50:58.265 INFO:tasks.workunit.client.0.smithi049.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T00:50:58.306 INFO:tasks.workunit.client.0.smithi049.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T00:50:58.310 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T00:50:58.310 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-18T00:50:58.367 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-03-18T00:50:58.368 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-03-18T00:50:58.369 DEBUG:teuthology.orchestra.run.smithi049:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c382027d5c66ef4e0aa1131ea3e8080c658d3184 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-03-18T00:50:58.429 INFO:tasks.workunit.client.0.smithi049.stdout:getting iogen 2024-03-18T00:50:58.429 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'getting iogen' 2024-03-18T00:50:58.429 INFO:tasks.workunit.client.0.smithi049.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-18T00:50:58.451 INFO:tasks.workunit.client.0.smithi049.stderr:--2024-03-18 00:50:58-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-03-18T00:50:58.452 INFO:tasks.workunit.client.0.smithi049.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-18T00:50:58.473 INFO:tasks.workunit.client.0.smithi049.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-18T00:50:58.494 INFO:tasks.workunit.client.0.smithi049.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-18T00:50:58.494 INFO:tasks.workunit.client.0.smithi049.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-03-18T00:50:58.494 INFO:tasks.workunit.client.0.smithi049.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-03-18T00:50:58.497 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:58.497 INFO:tasks.workunit.client.0.smithi049.stderr: 0K ....... 100% 1.28G=0s 2024-03-18T00:50:58.497 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:58.497 INFO:tasks.workunit.client.0.smithi049.stderr:2024-03-18 00:50:58 (1.28 GB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-03-18T00:50:58.497 INFO:tasks.workunit.client.0.smithi049.stderr: 2024-03-18T00:50:58.497 INFO:tasks.workunit.client.0.smithi049.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-03-18T00:50:58.511 INFO:tasks.workunit.client.0.smithi049.stdout:iogen_3.1p0 2024-03-18T00:50:58.512 INFO:tasks.workunit.client.0.smithi049.stdout:iogen_3.1p0/src 2024-03-18T00:50:58.514 INFO:tasks.workunit.client.0.smithi049.stdout:iogen_3.1p0/src/iogen.8 2024-03-18T00:50:58.515 INFO:tasks.workunit.client.0.smithi049.stdout:iogen_3.1p0/src/iogen.c 2024-03-18T00:50:58.516 INFO:tasks.workunit.client.0.smithi049.stdout:iogen_3.1p0/README 2024-03-18T00:50:58.518 INFO:tasks.workunit.client.0.smithi049.stdout:iogen_3.1p0/Makefile.bsd 2024-03-18T00:50:58.519 INFO:tasks.workunit.client.0.smithi049.stdout:iogen_3.1p0/Makefile 2024-03-18T00:50:58.521 INFO:tasks.workunit.client.0.smithi049.stderr:+ cd iogen_3.1p0 2024-03-18T00:50:58.521 INFO:tasks.workunit.client.0.smithi049.stdout:making iogen 2024-03-18T00:50:58.522 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'making iogen' 2024-03-18T00:50:58.522 INFO:tasks.workunit.client.0.smithi049.stderr:+ make 2024-03-18T00:50:58.529 INFO:tasks.workunit.client.0.smithi049.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-03-18T00:50:58.755 INFO:tasks.workunit.client.0.smithi049.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-03-18T00:50:58.841 INFO:tasks.workunit.client.0.smithi049.stdout:running iogen 2024-03-18T00:50:58.842 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'running iogen' 2024-03-18T00:50:58.842 INFO:tasks.workunit.client.0.smithi049.stderr:+ ./iogen -n 5 -s 2g 2024-03-18T00:50:58.842 INFO:tasks.workunit.client.0.smithi049.stderr:child spawned 86258 2024-03-18T00:50:58.842 INFO:tasks.workunit.client.0.smithi049.stderr:child spawned 86259 2024-03-18T00:50:58.842 INFO:tasks.workunit.client.0.smithi049.stderr:child spawned 86260 2024-03-18T00:50:58.842 INFO:tasks.workunit.client.0.smithi049.stderr:child spawned 86261 2024-03-18T00:50:58.842 INFO:tasks.workunit.client.0.smithi049.stderr:child spawned 86262 2024-03-18T00:50:58.842 INFO:tasks.workunit.client.0.smithi049.stdout:sleep for 10 min 2024-03-18T00:50:58.843 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'sleep for 10 min' 2024-03-18T00:50:58.843 INFO:tasks.workunit.client.0.smithi049.stderr:+ sleep 600 2024-03-18T00:50:59.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:50:59 smithi049 ceph-mon[32130]: pgmap v376: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-18T00:50:59.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:50:59 smithi107 ceph-mon[36270]: pgmap v376: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-18T00:50:59.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:50:59 smithi106 ceph-mon[35899]: pgmap v376: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 12 op/s 2024-03-18T00:51:01.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:01 smithi049 ceph-mon[32130]: pgmap v377: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-18T00:51:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:01 smithi107 ceph-mon[36270]: pgmap v377: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-18T00:51:01.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:01 smithi106 ceph-mon[35899]: pgmap v377: 97 pgs: 97 active+clean; 666 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-18T00:51:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.102+0000 7f27cbc81640 1 -- 172.21.15.49:0/1515276395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4105410 msgr2=0x7f27c410bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.102+0000 7f27cbc81640 1 --2- 172.21.15.49:0/1515276395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4105410 0x7f27c410bca0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00acb0 tx=0x7f27bc02c1c0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.102+0000 7f27cbc81640 1 -- 172.21.15.49:0/1515276395 shutdown_connections 2024-03-18T00:51:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.102+0000 7f27cbc81640 1 --2- 172.21.15.49:0/1515276395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27c4104a50 0x7f27c4104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.102+0000 7f27cbc81640 1 --2- 172.21.15.49:0/1515276395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4105410 0x7f27c410bca0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.102+0000 7f27cbc81640 1 --2- 172.21.15.49:0/1515276395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27c4103860 0x7f27c4103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.102+0000 7f27cbc81640 1 -- 172.21.15.49:0/1515276395 >> 172.21.15.49:0/1515276395 conn(0x7f27c40fedf0 msgr2=0x7f27c4101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.103+0000 7f27cbc81640 1 -- 172.21.15.49:0/1515276395 shutdown_connections 2024-03-18T00:51:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.103+0000 7f27cbc81640 1 -- 172.21.15.49:0/1515276395 wait complete. 2024-03-18T00:51:03.105 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.104+0000 7f27cbc81640 1 Processor -- start 2024-03-18T00:51:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.104+0000 7f27cbc81640 1 -- start start 2024-03-18T00:51:03.106 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.105+0000 7f27cbc81640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27c4103860 0x7f27c4199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.105+0000 7f27cbc81640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4104a50 0x7f27c4199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.105+0000 7f27cbc81640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27c4105410 0x7f27c41a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.105+0000 7f27cbc81640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f27c410f640 con 0x7f27c4104a50 2024-03-18T00:51:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.105+0000 7f27cbc81640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f27c410f4c0 con 0x7f27c4105410 2024-03-18T00:51:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.105+0000 7f27cbc81640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27c410f7c0 con 0x7f27c4103860 2024-03-18T00:51:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.106+0000 7f27c921c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4104a50 0x7f27c4199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.106+0000 7f27c921c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4104a50 0x7f27c4199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56554/0 (socket says 172.21.15.49:56554) 2024-03-18T00:51:03.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.106+0000 7f27c921c640 1 -- 172.21.15.49:0/2367493815 learned_addr learned my addr 172.21.15.49:0/2367493815 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.107+0000 7f27c921c640 1 -- 172.21.15.49:0/2367493815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27c4103860 msgr2=0x7f27c4199a70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:51:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.107+0000 7f27c921c640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27c4103860 0x7f27c4199a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.109 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.107+0000 7f27c921c640 1 -- 172.21.15.49:0/2367493815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27c4105410 msgr2=0x7f27c41a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.107+0000 7f27c921c640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27c4105410 0x7f27c41a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.107+0000 7f27c921c640 1 -- 172.21.15.49:0/2367493815 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27c41a1690 con 0x7f27c4104a50 2024-03-18T00:51:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.107+0000 7f27c921c640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4104a50 0x7f27c4199fb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f27b400b450 tx=0x7f27b400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.107+0000 7f27baffd640 1 -- 172.21.15.49:0/2367493815 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b4015020 con 0x7f27c4104a50 2024-03-18T00:51:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.107+0000 7f27baffd640 1 -- 172.21.15.49:0/2367493815 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27b4011d70 con 0x7f27c4104a50 2024-03-18T00:51:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.107+0000 7f27cbc81640 1 -- 172.21.15.49:0/2367493815 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27c41a1980 con 0x7f27c4104a50 2024-03-18T00:51:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.108+0000 7f27baffd640 1 -- 172.21.15.49:0/2367493815 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b400f340 con 0x7f27c4104a50 2024-03-18T00:51:03.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.108+0000 7f27cbc81640 1 -- 172.21.15.49:0/2367493815 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27c41a1fa0 con 0x7f27c4104a50 2024-03-18T00:51:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.111+0000 7f27ce86b740 1 Processor -- start 2024-03-18T00:51:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.111+0000 7f27ce86b740 1 -- start start 2024-03-18T00:51:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.111+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x55935811c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.111+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 0x5593581219f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.111+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x5593581244f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.113 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.111+0000 7f27ce86b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559358081a40 con 0x559358122060 2024-03-18T00:51:03.114 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.111+0000 7f27ce86b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55935802a920 con 0x55935811c500 2024-03-18T00:51:03.114 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.111+0000 7f27ce86b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55935807dae0 con 0x55935811cef0 2024-03-18T00:51:03.114 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.111+0000 7f27baffd640 1 -- 172.21.15.49:0/2367493815 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f27b4018020 con 0x7f27c4104a50 2024-03-18T00:51:03.114 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.112+0000 7f27c921c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x5593581244f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:03.114 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.112+0000 7f27c921c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x5593581244f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56564/0 (socket says 172.21.15.49:56564) 2024-03-18T00:51:03.114 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.112+0000 7f27c921c640 1 -- 172.21.15.49:0/3985105850 learned_addr learned my addr 172.21.15.49:0/3985105850 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.112+0000 7f27c921c640 1 -- 172.21.15.49:0/3985105850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 msgr2=0x5593581219f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.112+0000 7f27c921c640 1 --2- 172.21.15.49:0/3985105850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 0x5593581219f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.112+0000 7f27c921c640 1 -- 172.21.15.49:0/3985105850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 msgr2=0x55935811c920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.112+0000 7f27c921c640 1 --2- 172.21.15.49:0/3985105850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x55935811c920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.112+0000 7f27c921c640 1 -- 172.21.15.49:0/3985105850 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559358042460 con 0x559358122060 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.113+0000 7f27baffd640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27b00a2870 0x7f27b00a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.113+0000 7f27ca21e640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27b00a2870 0x7f27b00a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.113+0000 7f27c921c640 1 --2- 172.21.15.49:0/3985105850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x5593581244f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f27b40cbf60 tx=0x7f27b4097160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993f9b5dd8e66be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.113+0000 7f27baffd640 1 -- 172.21.15.49:0/2367493815 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f27b40ce540 con 0x7f27c4104a50 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.114+0000 7f27affff640 1 -- 172.21.15.49:0/3985105850 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b40b2040 con 0x559358122060 2024-03-18T00:51:03.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.114+0000 7f27affff640 1 -- 172.21.15.49:0/3985105850 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27b409b5c0 con 0x559358122060 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.114+0000 7f27affff640 1 -- 172.21.15.49:0/3985105850 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b409aa00 con 0x559358122060 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.114+0000 7f27ce86b740 1 -- 172.21.15.49:0/3985105850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 msgr2=0x5593581244f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.114+0000 7f27ce86b740 1 --2- 172.21.15.49:0/3985105850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x5593581244f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f27b40cbf60 tx=0x7f27b4097160 comp rx=0 tx=0).stop 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.114+0000 7f27ca21e640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27b00a2870 0x7f27b00a4d30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f27bc001440 tx=0x7f27bc000b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.115+0000 7f27ce86b740 1 -- 172.21.15.49:0/3985105850 shutdown_connections 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.115+0000 7f27ce86b740 1 --2- 172.21.15.49:0/3985105850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x5593581244f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.115+0000 7f27ce86b740 1 --2- 172.21.15.49:0/3985105850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 0x5593581219f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.115+0000 7f27ce86b740 1 --2- 172.21.15.49:0/3985105850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x55935811c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.115+0000 7f27ce86b740 1 -- 172.21.15.49:0/3985105850 >> 172.21.15.49:0/3985105850 conn(0x559358117a70 msgr2=0x559358117e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.115+0000 7f27ce86b740 1 -- 172.21.15.49:0/3985105850 shutdown_connections 2024-03-18T00:51:03.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.115+0000 7f27ce86b740 1 -- 172.21.15.49:0/3985105850 wait complete. 2024-03-18T00:51:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.115+0000 7f27ce86b740 1 -- start start 2024-03-18T00:51:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.115+0000 7f27ce86b740 1 Processor -- start 2024-03-18T00:51:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x55935812f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 0x55935812f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358136630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ce86b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559358125010 con 0x559358122060 2024-03-18T00:51:03.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ce86b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55935803dd70 con 0x55935811c500 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ce86b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559358125190 con 0x55935811cef0 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ca21e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358136630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ce86b740 10 client.0 _ll_register_callbacks cb 0x559357d95a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ca21e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358136630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56574/0 (socket says 172.21.15.49:56574) 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.116+0000 7f27ca21e640 1 -- 172.21.15.49:0/2657374699 learned_addr learned my addr 172.21.15.49:0/2657374699 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.117+0000 7f27ca21e640 1 -- 172.21.15.49:0/2657374699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 msgr2=0x55935812f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.117+0000 7f27ca21e640 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 0x55935812f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.117+0000 7f27ca21e640 1 -- 172.21.15.49:0/2657374699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 msgr2=0x55935812f0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.117+0000 7f27ca21e640 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x55935812f0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.117+0000 7f27ca21e640 1 -- 172.21.15.49:0/2657374699 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559357d96210 con 0x559358122060 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.117+0000 7f27ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.117+0000 7f27ca21e640 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358136630 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00ac80 tx=0x7f27bc03d850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b85d15aaed90a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:03.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.117+0000 7f27ad7fa640 1 -- 172.21.15.49:0/2657374699 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc04e060 con 0x559358122060 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ad7fa640 1 -- 172.21.15.49:0/2657374699 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f27bc03e800 con 0x559358122060 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ce86b740 1 -- 172.21.15.49:0/2657374699 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27b409b5c0 con 0x559358122060 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ad7fa640 1 -- 172.21.15.49:0/2657374699 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f27bc03ea00 con 0x559358122060 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ce86b740 10 client.15693 fetch_fsmap learned FSMap version 25 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ce86b740 1 -- 172.21.15.49:0/2657374699 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559357e62620 con 0x559358122060 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ad7fa640 1 -- 172.21.15.49:0/2657374699 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27b409b5c0 con 0x559358122060 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ad7fa640 1 -- 172.21.15.49:0/2657374699 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f27bc00a030 con 0x559358122060 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ce86b740 10 client.15693 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ce86b740 10 client.15693 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:03.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ce86b740 20 client.15693 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ce86b740 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55935813af80 0x55935813d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.118+0000 7f27ce86b740 4 client.15693 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.119+0000 7f27ce86b740 1 -- 172.21.15.49:0/2657374699 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55935808ef80 con 0x55935813af80 2024-03-18T00:51:03.124 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.121+0000 7f27c9a1d640 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55935813af80 0x55935813d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.128+0000 7f27c9a1d640 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55935813af80 0x55935813d410 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6737f0cf76677c9a server_cookie=a3408b42b7edfd5e in_seq=0 out_seq=0 2024-03-18T00:51:03.128 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.128+0000 7f27ad7fa640 10 client.15693 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:03.134 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.133+0000 7f27ad7fa640 1 -- 172.21.15.49:0/2657374699 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55935808ef80 con 0x55935813af80 2024-03-18T00:51:03.134 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.133+0000 7f27ad7fa640 10 client.15693 handle_command_reply: tid=0 2024-03-18T00:51:03.134 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.133+0000 7f27ce86b740 1 client.15693 shutdown 2024-03-18T00:51:03.134 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.133+0000 7f27ce86b740 1 -- 172.21.15.49:0/2657374699 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 msgr2=0x559358136630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:03.135 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.133+0000 7f27ce86b740 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358136630 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00ac80 tx=0x7f27bc03d850 comp rx=0 tx=0).stop 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ce86b740 1 -- 172.21.15.49:0/2657374699 shutdown_connections 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ce86b740 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55935813af80 0x55935813d410 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ce86b740 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358136630 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ce86b740 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 0x55935812f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ce86b740 1 --2- 172.21.15.49:0/2657374699 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x55935812f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ce86b740 1 -- 172.21.15.49:0/2657374699 >> 172.21.15.49:0/2657374699 conn(0x559358117a70 msgr2=0x559358116c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ad7fa640 0 client.15693 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ce86b740 1 -- 172.21.15.49:0/2657374699 shutdown_connections 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ce86b740 1 -- 172.21.15.49:0/2657374699 wait complete. 2024-03-18T00:51:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.134+0000 7f27ce86b740 20 client.15693 trim_cache size 0 max 16384 2024-03-18T00:51:03.158 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ce86b740 1 Processor -- start 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ce86b740 1 -- start start 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811c500 0x55935812e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811cef0 0x5593581365f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x55935811b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ce86b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41a69e0 con 0x559358122060 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ce86b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41a8520 con 0x55935811cef0 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ce86b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41a70b0 con 0x55935811c500 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ca21e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x55935811b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ca21e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x55935811b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56582/0 (socket says 172.21.15.49:56582) 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.157+0000 7f27ca21e640 1 -- 172.21.15.49:0/256868684 learned_addr learned my addr 172.21.15.49:0/256868684 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.158+0000 7f27ca21e640 1 -- 172.21.15.49:0/256868684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811c500 msgr2=0x55935812e6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.158+0000 7f27ca21e640 1 --2- 172.21.15.49:0/256868684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811c500 0x55935812e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.158+0000 7f27ca21e640 1 -- 172.21.15.49:0/256868684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811cef0 msgr2=0x5593581365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.158+0000 7f27ca21e640 1 --2- 172.21.15.49:0/256868684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811cef0 0x5593581365f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.158+0000 7f27ca21e640 1 -- 172.21.15.49:0/256868684 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55935813da90 con 0x559358122060 2024-03-18T00:51:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.158+0000 7f27ca21e640 1 --2- 172.21.15.49:0/256868684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x55935811b3e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00aa40 tx=0x7f27bc03df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a720a23402ec49 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.158+0000 7f27af7fe640 1 -- 172.21.15.49:0/256868684 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc04e070 con 0x559358122060 2024-03-18T00:51:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.158+0000 7f27af7fe640 1 -- 172.21.15.49:0/256868684 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27bc030050 con 0x559358122060 2024-03-18T00:51:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.158+0000 7f27af7fe640 1 -- 172.21.15.49:0/256868684 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc054400 con 0x559358122060 2024-03-18T00:51:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.159+0000 7f27ce86b740 1 -- 172.21.15.49:0/256868684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 msgr2=0x55935811b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.159+0000 7f27ce86b740 1 --2- 172.21.15.49:0/256868684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x55935811b3e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f27bc00aa40 tx=0x7f27bc03df90 comp rx=0 tx=0).stop 2024-03-18T00:51:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.159+0000 7f27ce86b740 1 -- 172.21.15.49:0/256868684 shutdown_connections 2024-03-18T00:51:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.159+0000 7f27ce86b740 1 --2- 172.21.15.49:0/256868684 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811cef0 0x5593581365f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.159+0000 7f27ce86b740 1 --2- 172.21.15.49:0/256868684 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x55935811b3e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.159+0000 7f27ce86b740 1 --2- 172.21.15.49:0/256868684 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811c500 0x55935812e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.159+0000 7f27ce86b740 1 -- 172.21.15.49:0/256868684 >> 172.21.15.49:0/256868684 conn(0x559358117a70 msgr2=0x55935811acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.159+0000 7f27ce86b740 1 -- 172.21.15.49:0/256868684 shutdown_connections 2024-03-18T00:51:03.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.159+0000 7f27ce86b740 1 -- 172.21.15.49:0/256868684 wait complete. 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ce86b740 1 -- start start 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ce86b740 1 Processor -- start 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x5593581360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 0x559358134720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ce86b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358134c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ce86b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41a7e50 con 0x559358122060 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ce86b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41a7780 con 0x55935811c500 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ce86b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f27c41a49c0 con 0x55935811cef0 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ce86b740 10 client.0 _ll_register_callbacks cb 0x7f27bc00ab00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ca21e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ca21e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358134c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56584/0 (socket says 172.21.15.49:56584) 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.160+0000 7f27ca21e640 1 -- 172.21.15.49:0/910480746 learned_addr learned my addr 172.21.15.49:0/910480746 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27c9a1d640 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x5593581360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:03.165 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ca21e640 1 -- 172.21.15.49:0/910480746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 msgr2=0x559358134720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ca21e640 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 0x559358134720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ca21e640 1 -- 172.21.15.49:0/910480746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 msgr2=0x5593581360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ca21e640 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x5593581360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ca21e640 1 -- 172.21.15.49:0/910480746 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593581351a0 con 0x559358122060 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ca21e640 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358134c60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f27bc031640 tx=0x7f27bc031cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87be9e85c839f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ad7fa640 1 -- 172.21.15.49:0/910480746 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc04e070 con 0x559358122060 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ce86b740 1 -- 172.21.15.49:0/910480746 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27bc030050 con 0x559358122060 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.161+0000 7f27ad7fa640 1 -- 172.21.15.49:0/910480746 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f27bc054950 con 0x559358122060 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ad7fa640 1 -- 172.21.15.49:0/910480746 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f27bc06b070 con 0x559358122060 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ce86b740 10 client.15699 fetch_fsmap learned FSMap version 25 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ce86b740 1 -- 172.21.15.49:0/910480746 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5593581a1f80 con 0x559358122060 2024-03-18T00:51:03.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ad7fa640 1 -- 172.21.15.49:0/910480746 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27bc030050 con 0x559358122060 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ad7fa640 1 -- 172.21.15.49:0/910480746 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7f27bc05d630 con 0x559358122060 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ce86b740 10 client.15699 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ce86b740 10 client.15699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ce86b740 20 client.15699 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ce86b740 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5593581a2870 0x5593581a4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ce86b740 4 client.15699 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.162+0000 7f27ce86b740 1 -- 172.21.15.49:0/910480746 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55935812eea0 con 0x5593581a2870 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.163+0000 7f27c9a1d640 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5593581a2870 0x5593581a4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.164+0000 7f27ad7fa640 10 client.15699 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:03.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.164+0000 7f27c9a1d640 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5593581a2870 0x5593581a4c90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48544bfa0b40350e server_cookie=3ce520c2da02106 in_seq=0 out_seq=0 2024-03-18T00:51:03.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:03 smithi049 ceph-mon[32130]: pgmap v378: 97 pgs: 97 active+clean; 434 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 36 MiB/s wr, 49 op/s 2024-03-18T00:51:03.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:03 smithi107 ceph-mon[36270]: pgmap v378: 97 pgs: 97 active+clean; 434 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 36 MiB/s wr, 49 op/s 2024-03-18T00:51:03.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:03 smithi106 ceph-mon[35899]: pgmap v378: 97 pgs: 97 active+clean; 434 MiB data, 414 MiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 36 MiB/s wr, 49 op/s 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.511+0000 7f27ad7fa640 1 -- 172.21.15.49:0/910480746 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55935812eea0 con 0x5593581a2870 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.511+0000 7f27ad7fa640 10 client.15699 handle_command_reply: tid=0 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 client.15699 shutdown 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 -- 172.21.15.49:0/910480746 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 msgr2=0x559358134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358134c60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f27bc031640 tx=0x7f27bc031cc0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 -- 172.21.15.49:0/910480746 shutdown_connections 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5593581a2870 0x5593581a4c90 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559358122060 0x559358134c60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55935811cef0 0x559358134720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 --2- 172.21.15.49:0/910480746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55935811c500 0x5593581360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 -- 172.21.15.49:0/910480746 >> 172.21.15.49:0/910480746 conn(0x559358117a70 msgr2=0x55935811acd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:03.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ad7fa640 0 client.15699 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:03.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 -- 172.21.15.49:0/910480746 shutdown_connections 2024-03-18T00:51:03.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.512+0000 7f27ce86b740 1 -- 172.21.15.49:0/910480746 wait complete. 2024-03-18T00:51:03.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.513+0000 7f27ce86b740 20 client.15699 trim_cache size 0 max 16384 2024-03-18T00:51:03.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:51:03.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:51:03.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:51:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:51:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.515+0000 7f27cbc81640 1 -- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27b00a2870 msgr2=0x7f27b00a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.515+0000 7f27cbc81640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27b00a2870 0x7f27b00a4d30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f27bc001440 tx=0x7f27bc000b60 comp rx=0 tx=0).stop 2024-03-18T00:51:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.515+0000 7f27cbc81640 1 -- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4104a50 msgr2=0x7f27c4199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:03.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.515+0000 7f27cbc81640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4104a50 0x7f27c4199fb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f27b400b450 tx=0x7f27b400b910 comp rx=0 tx=0).stop 2024-03-18T00:51:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.516+0000 7f27cbc81640 1 -- 172.21.15.49:0/2367493815 shutdown_connections 2024-03-18T00:51:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.516+0000 7f27cbc81640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f27b00a2870 0x7f27b00a4d30 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.516+0000 7f27cbc81640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f27c4105410 0x7f27c41a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.516+0000 7f27cbc81640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f27c4104a50 0x7f27c4199fb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.516+0000 7f27cbc81640 1 --2- 172.21.15.49:0/2367493815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f27c4103860 0x7f27c4199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.516+0000 7f27cbc81640 1 -- 172.21.15.49:0/2367493815 >> 172.21.15.49:0/2367493815 conn(0x7f27c40fedf0 msgr2=0x7f27c4107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.516+0000 7f27cbc81640 1 -- 172.21.15.49:0/2367493815 shutdown_connections 2024-03-18T00:51:03.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:03.516+0000 7f27cbc81640 1 -- 172.21.15.49:0/2367493815 wait complete. 2024-03-18T00:51:05.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:05 smithi049 ceph-mon[32130]: pgmap v379: 97 pgs: 97 active+clean; 2.4 GiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 205 MiB/s wr, 91 op/s 2024-03-18T00:51:05.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:05 smithi107 ceph-mon[36270]: pgmap v379: 97 pgs: 97 active+clean; 2.4 GiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 205 MiB/s wr, 91 op/s 2024-03-18T00:51:05.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:05 smithi106 ceph-mon[35899]: pgmap v379: 97 pgs: 97 active+clean; 2.4 GiB data, 721 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 205 MiB/s wr, 91 op/s 2024-03-18T00:51:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:07 smithi049 ceph-mon[32130]: pgmap v380: 97 pgs: 97 active+clean; 2.9 GiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 248 MiB/s wr, 140 op/s 2024-03-18T00:51:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:07 smithi107 ceph-mon[36270]: pgmap v380: 97 pgs: 97 active+clean; 2.9 GiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 248 MiB/s wr, 140 op/s 2024-03-18T00:51:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:07 smithi106 ceph-mon[35899]: pgmap v380: 97 pgs: 97 active+clean; 2.9 GiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 248 MiB/s wr, 140 op/s 2024-03-18T00:51:09.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:09 smithi049 ceph-mon[32130]: pgmap v381: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 426 MiB/s wr, 304 op/s 2024-03-18T00:51:09.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:09 smithi107 ceph-mon[36270]: pgmap v381: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 426 MiB/s wr, 304 op/s 2024-03-18T00:51:09.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:09 smithi106 ceph-mon[35899]: pgmap v381: 97 pgs: 97 active+clean; 5.0 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 426 MiB/s wr, 304 op/s 2024-03-18T00:51:11.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:11 smithi049 ceph-mon[32130]: pgmap v382: 97 pgs: 97 active+clean; 6.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 595 MiB/s wr, 703 op/s 2024-03-18T00:51:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:11 smithi107 ceph-mon[36270]: pgmap v382: 97 pgs: 97 active+clean; 6.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 595 MiB/s wr, 703 op/s 2024-03-18T00:51:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:11 smithi106 ceph-mon[35899]: pgmap v382: 97 pgs: 97 active+clean; 6.7 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 595 MiB/s wr, 703 op/s 2024-03-18T00:51:13.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:13 smithi049 ceph-mon[32130]: pgmap v383: 97 pgs: 97 active+clean; 7.7 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 705 MiB/s wr, 1.05k op/s 2024-03-18T00:51:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:13 smithi107 ceph-mon[36270]: pgmap v383: 97 pgs: 97 active+clean; 7.7 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 705 MiB/s wr, 1.05k op/s 2024-03-18T00:51:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:13 smithi106 ceph-mon[35899]: pgmap v383: 97 pgs: 97 active+clean; 7.7 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 705 MiB/s wr, 1.05k op/s 2024-03-18T00:51:13.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:51:13.507 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:51:13.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.926+0000 7ff37b59e640 1 -- 172.21.15.49:0/1958837834 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c10b2f0 msgr2=0x7ff37c10d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:13.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.926+0000 7ff37b59e640 1 --2- 172.21.15.49:0/1958837834 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c10b2f0 0x7ff37c10d6e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff36c009860 tx=0x7ff36c02bb90 comp rx=0 tx=0).stop 2024-03-18T00:51:13.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.929+0000 7ff37b59e640 1 -- 172.21.15.49:0/1958837834 shutdown_connections 2024-03-18T00:51:13.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.929+0000 7ff37b59e640 1 --2- 172.21.15.49:0/1958837834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37c10dd70 0x7ff37c110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.929+0000 7ff37b59e640 1 --2- 172.21.15.49:0/1958837834 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c10b2f0 0x7ff37c10d6e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.929+0000 7ff37b59e640 1 --2- 172.21.15.49:0/1958837834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff37c1089c0 0x7ff37c10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.929+0000 7ff37b59e640 1 -- 172.21.15.49:0/1958837834 >> 172.21.15.49:0/1958837834 conn(0x7ff37c06e6c0 msgr2=0x7ff37c070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:13.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.930+0000 7ff37b59e640 1 -- 172.21.15.49:0/1958837834 shutdown_connections 2024-03-18T00:51:13.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.930+0000 7ff37b59e640 1 -- 172.21.15.49:0/1958837834 wait complete. 2024-03-18T00:51:13.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.932+0000 7ff37b59e640 1 Processor -- start 2024-03-18T00:51:13.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.932+0000 7ff37b59e640 1 -- start start 2024-03-18T00:51:13.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.933+0000 7ff37b59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c1089c0 0x7ff37c1192f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.933+0000 7ff37b59e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff37c10b2f0 0x7ff37c11e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.933+0000 7ff37b59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37c10dd70 0x7ff37c119830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.933+0000 7ff37b59e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c113030 con 0x7ff37c1089c0 2024-03-18T00:51:13.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.933+0000 7ff37b59e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c112eb0 con 0x7ff37c10b2f0 2024-03-18T00:51:13.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.933+0000 7ff37b59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c1131b0 con 0x7ff37c10dd70 2024-03-18T00:51:13.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.935+0000 7ff37a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c1089c0 0x7ff37c1192f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:13.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.935+0000 7ff37a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c1089c0 0x7ff37c1192f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59926/0 (socket says 172.21.15.49:59926) 2024-03-18T00:51:13.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.935+0000 7ff37a59c640 1 -- 172.21.15.49:0/2906213634 learned_addr learned my addr 172.21.15.49:0/2906213634 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:13.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.935+0000 7ff373fff640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff37c10b2f0 0x7ff37c11e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:13.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.936+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37c10dd70 0x7ff37c119830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:13.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.936+0000 7ff37a59c640 1 -- 172.21.15.49:0/2906213634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37c10dd70 msgr2=0x7ff37c119830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:13.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.936+0000 7ff37a59c640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37c10dd70 0x7ff37c119830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.936+0000 7ff37a59c640 1 -- 172.21.15.49:0/2906213634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff37c10b2f0 msgr2=0x7ff37c11e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:13.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.936+0000 7ff37a59c640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff37c10b2f0 0x7ff37c11e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.937+0000 7ff37a59c640 1 -- 172.21.15.49:0/2906213634 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff37c1b6d00 con 0x7ff37c1089c0 2024-03-18T00:51:13.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.938+0000 7ff37a59c640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c1089c0 0x7ff37c1192f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff36400d950 tx=0x7ff36400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:13.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.939+0000 7ff3737fe640 1 -- 172.21.15.49:0/2906213634 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff364017070 con 0x7ff37c1089c0 2024-03-18T00:51:13.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.939+0000 7ff3737fe640 1 -- 172.21.15.49:0/2906213634 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff364009c30 con 0x7ff37c1089c0 2024-03-18T00:51:13.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.939+0000 7ff37b59e640 1 -- 172.21.15.49:0/2906213634 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff37c1b6f90 con 0x7ff37c1089c0 2024-03-18T00:51:13.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.940+0000 7ff3737fe640 1 -- 172.21.15.49:0/2906213634 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff36400f6b0 con 0x7ff37c1089c0 2024-03-18T00:51:13.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.940+0000 7ff37b59e640 1 -- 172.21.15.49:0/2906213634 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff37c1b73f0 con 0x7ff37c1089c0 2024-03-18T00:51:13.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.945+0000 7ff3837fb740 1 Processor -- start 2024-03-18T00:51:13.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.946+0000 7ff3837fb740 1 -- start start 2024-03-18T00:51:13.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.946+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948c0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.946+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.946+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c5ec0 0x5560948c8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.946+0000 7ff3837fb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x556094825500 con 0x5560948c0360 2024-03-18T00:51:13.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.947+0000 7ff3837fb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5560947cdbe0 con 0x5560948c5ec0 2024-03-18T00:51:13.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.947+0000 7ff3837fb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5560948215a0 con 0x5560948c0d50 2024-03-18T00:51:13.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.948+0000 7ff37ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948c0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:13.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.948+0000 7ff37ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948c0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59930/0 (socket says 172.21.15.49:59930) 2024-03-18T00:51:13.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.948+0000 7ff37ad9d640 1 -- 172.21.15.49:0/766732021 learned_addr learned my addr 172.21.15.49:0/766732021 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:13.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.950+0000 7ff37ad9d640 1 -- 172.21.15.49:0/766732021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 msgr2=0x5560948c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:13.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.950+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/766732021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.950+0000 7ff37ad9d640 1 -- 172.21.15.49:0/766732021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c5ec0 msgr2=0x5560948c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:13.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.950+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/766732021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c5ec0 0x5560948c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.950+0000 7ff37ad9d640 1 -- 172.21.15.49:0/766732021 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560947e14b0 con 0x5560948c0360 2024-03-18T00:51:13.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.952+0000 7ff3737fe640 1 -- 172.21.15.49:0/2906213634 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff3640109d0 con 0x7ff37c1089c0 2024-03-18T00:51:13.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.952+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/766732021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948c0780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff37400aad0 tx=0x7ff3740086a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f3d22c250aa45c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:13.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.954+0000 7ff3737fe640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3680a28e0 0x7ff3680a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.954+0000 7ff3737fe640 1 -- 172.21.15.49:0/2906213634 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff3640ce3a0 con 0x7ff37c1089c0 2024-03-18T00:51:13.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.955+0000 7ff37a59c640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3680a28e0 0x7ff3680a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:13.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.957+0000 7ff370ff9640 1 -- 172.21.15.49:0/766732021 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374003970 con 0x5560948c0360 2024-03-18T00:51:13.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.957+0000 7ff370ff9640 1 -- 172.21.15.49:0/766732021 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff374004020 con 0x5560948c0360 2024-03-18T00:51:13.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.957+0000 7ff370ff9640 1 -- 172.21.15.49:0/766732021 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374014830 con 0x5560948c0360 2024-03-18T00:51:13.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.956+0000 7ff37a59c640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3680a28e0 0x7ff3680a4da0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff364009610 tx=0x7ff364005880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:13.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.959+0000 7ff3837fb740 1 -- 172.21.15.49:0/766732021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 msgr2=0x5560948c0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:13.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.959+0000 7ff3837fb740 1 --2- 172.21.15.49:0/766732021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948c0780 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7ff37400aad0 tx=0x7ff3740086a0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.964+0000 7ff3837fb740 1 -- 172.21.15.49:0/766732021 shutdown_connections 2024-03-18T00:51:13.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.964+0000 7ff3837fb740 1 --2- 172.21.15.49:0/766732021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c5ec0 0x5560948c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.964+0000 7ff3837fb740 1 --2- 172.21.15.49:0/766732021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.964+0000 7ff3837fb740 1 --2- 172.21.15.49:0/766732021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948c0780 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.964+0000 7ff3837fb740 1 -- 172.21.15.49:0/766732021 >> 172.21.15.49:0/766732021 conn(0x5560948bb8d0 msgr2=0x5560948bbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:13.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.971+0000 7ff3837fb740 1 -- 172.21.15.49:0/766732021 shutdown_connections 2024-03-18T00:51:13.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.972+0000 7ff3837fb740 1 -- 172.21.15.49:0/766732021 wait complete. 2024-03-18T00:51:13.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.973+0000 7ff3837fb740 1 -- start start 2024-03-18T00:51:13.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.973+0000 7ff3837fb740 1 Processor -- start 2024-03-18T00:51:13.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.975+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.975+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c0d50 0x5560948cae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.975+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c5ec0 0x5560948d1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.975+0000 7ff3837fb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5560948c8e70 con 0x5560948c0360 2024-03-18T00:51:13.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.976+0000 7ff3837fb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5560947e1160 con 0x5560948c0d50 2024-03-18T00:51:13.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.976+0000 7ff3837fb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5560948c8ff0 con 0x5560948c5ec0 2024-03-18T00:51:13.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.976+0000 7ff3837fb740 10 client.0 _ll_register_callbacks cb 0x5560947c5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:13.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.976+0000 7ff37ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:13.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.976+0000 7ff37ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59934/0 (socket says 172.21.15.49:59934) 2024-03-18T00:51:13.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.976+0000 7ff37ad9d640 1 -- 172.21.15.49:0/4170595343 learned_addr learned my addr 172.21.15.49:0/4170595343 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:13.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.978+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c5ec0 0x5560948d1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:13.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.978+0000 7ff37ad9d640 1 -- 172.21.15.49:0/4170595343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c5ec0 msgr2=0x5560948d1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:13.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.978+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c5ec0 0x5560948d1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.978+0000 7ff37ad9d640 1 -- 172.21.15.49:0/4170595343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c0d50 msgr2=0x5560948cae70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:13.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.978+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c0d50 0x5560948cae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:13.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.978+0000 7ff37ad9d640 1 -- 172.21.15.49:0/4170595343 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560946037d0 con 0x5560948c0360 2024-03-18T00:51:13.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.979+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948ca930 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff374006010 tx=0x7ff374004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73c74ca674131c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:13.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.979+0000 7ff361ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:13.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.980+0000 7ff361ffb640 1 -- 172.21.15.49:0/4170595343 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff374004a40 con 0x5560948c0360 2024-03-18T00:51:13.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.981+0000 7ff361ffb640 1 -- 172.21.15.49:0/4170595343 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff374004be0 con 0x5560948c0360 2024-03-18T00:51:13.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.981+0000 7ff3837fb740 1 -- 172.21.15.49:0/4170595343 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff374004020 con 0x5560948c0360 2024-03-18T00:51:13.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.982+0000 7ff361ffb640 1 -- 172.21.15.49:0/4170595343 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff374004de0 con 0x5560948c0360 2024-03-18T00:51:13.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.983+0000 7ff3837fb740 10 client.15711 fetch_fsmap learned FSMap version 25 2024-03-18T00:51:13.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.983+0000 7ff3837fb740 1 -- 172.21.15.49:0/4170595343 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5560945393f0 con 0x5560948c0360 2024-03-18T00:51:13.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.986+0000 7ff361ffb640 1 -- 172.21.15.49:0/4170595343 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff374004020 con 0x5560948c0360 2024-03-18T00:51:13.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.986+0000 7ff361ffb640 1 -- 172.21.15.49:0/4170595343 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7ff374005170 con 0x5560948c0360 2024-03-18T00:51:13.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.988+0000 7ff3837fb740 10 client.15711 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:51:13.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.988+0000 7ff3837fb740 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:13.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.988+0000 7ff3837fb740 20 client.15711 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:13.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.988+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5560948d6170 0x5560948d8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:13.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.988+0000 7ff3837fb740 4 client.15711 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:13.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.988+0000 7ff3837fb740 1 -- 172.21.15.49:0/4170595343 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556094832a40 con 0x5560948d6170 2024-03-18T00:51:13.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.991+0000 7ff373fff640 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5560948d6170 0x5560948d8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:13.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.994+0000 7ff361ffb640 10 client.15711 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:13.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.994+0000 7ff373fff640 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5560948d6170 0x5560948d8560 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a75cf9ccf4c6bb6 server_cookie=f2d3a6377f403dc2 in_seq=0 out_seq=0 2024-03-18T00:51:14.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.999+0000 7ff361ffb640 1 -- 172.21.15.49:0/4170595343 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556094832a40 con 0x5560948d6170 2024-03-18T00:51:14.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:13.999+0000 7ff361ffb640 10 client.15711 handle_command_reply: tid=0 2024-03-18T00:51:14.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.000+0000 7ff3837fb740 1 client.15711 shutdown 2024-03-18T00:51:14.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.000+0000 7ff3837fb740 1 -- 172.21.15.49:0/4170595343 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 msgr2=0x5560948ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:14.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.000+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948ca930 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7ff374006010 tx=0x7ff374004730 comp rx=0 tx=0).stop 2024-03-18T00:51:14.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff3837fb740 1 -- 172.21.15.49:0/4170595343 shutdown_connections 2024-03-18T00:51:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5560948d6170 0x5560948d8560 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c0d50 0x5560948cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c5ec0 0x5560948d1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4170595343 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948ca930 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff361ffb640 0 client.15711 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:14.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff3837fb740 1 -- 172.21.15.49:0/4170595343 >> 172.21.15.49:0/4170595343 conn(0x5560948bb8d0 msgr2=0x5560948bab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:14.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff3837fb740 1 -- 172.21.15.49:0/4170595343 shutdown_connections 2024-03-18T00:51:14.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff3837fb740 1 -- 172.21.15.49:0/4170595343 wait complete. 2024-03-18T00:51:14.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.001+0000 7ff3837fb740 20 client.15711 trim_cache size 0 max 16384 2024-03-18T00:51:14.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.054+0000 7ff3837fb740 1 Processor -- start 2024-03-18T00:51:14.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.054+0000 7ff3837fb740 1 -- start start 2024-03-18T00:51:14.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.055+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c0360 0x5560948ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.055+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948d1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.055+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c5ec0 0x5560948caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.056+0000 7ff3837fb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c1c0380 con 0x5560948c5ec0 2024-03-18T00:51:14.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.056+0000 7ff3837fb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c1c1ec0 con 0x5560948c0360 2024-03-18T00:51:14.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.056+0000 7ff3837fb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c1c0a50 con 0x5560948c0d50 2024-03-18T00:51:14.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.056+0000 7ff373fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948d1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:14.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.056+0000 7ff373fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948d1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60214/0 (socket says 172.21.15.49:60214) 2024-03-18T00:51:14.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.056+0000 7ff373fff640 1 -- 172.21.15.49:0/1126994409 learned_addr learned my addr 172.21.15.49:0/1126994409 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:14.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.056+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/1126994409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c0360 0x5560948ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:14.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.056+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/1126994409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c5ec0 0x5560948caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:14.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.057+0000 7ff37ad9d640 1 -- 172.21.15.49:0/1126994409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 msgr2=0x5560948d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:14.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.057+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/1126994409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948d1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.057+0000 7ff37ad9d640 1 -- 172.21.15.49:0/1126994409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c0360 msgr2=0x5560948ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:14.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.057+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/1126994409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c0360 0x5560948ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.057+0000 7ff37ad9d640 1 -- 172.21.15.49:0/1126994409 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560948d88d0 con 0x5560948c5ec0 2024-03-18T00:51:14.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.058+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/1126994409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c5ec0 0x5560948caa80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff37400aad0 tx=0x7ff374000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e9a54a391fa3b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:14.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.059+0000 7ff363fff640 1 -- 172.21.15.49:0/1126994409 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3740047a0 con 0x5560948c5ec0 2024-03-18T00:51:14.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.059+0000 7ff363fff640 1 -- 172.21.15.49:0/1126994409 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff374003f40 con 0x5560948c5ec0 2024-03-18T00:51:14.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.059+0000 7ff363fff640 1 -- 172.21.15.49:0/1126994409 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37401e940 con 0x5560948c5ec0 2024-03-18T00:51:14.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.061+0000 7ff3837fb740 1 -- 172.21.15.49:0/1126994409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c5ec0 msgr2=0x5560948caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:14.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.061+0000 7ff3837fb740 1 --2- 172.21.15.49:0/1126994409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c5ec0 0x5560948caa80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff37400aad0 tx=0x7ff374000f40 comp rx=0 tx=0).stop 2024-03-18T00:51:14.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.061+0000 7ff3837fb740 1 -- 172.21.15.49:0/1126994409 shutdown_connections 2024-03-18T00:51:14.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.061+0000 7ff3837fb740 1 --2- 172.21.15.49:0/1126994409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c5ec0 0x5560948caa80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.061+0000 7ff3837fb740 1 --2- 172.21.15.49:0/1126994409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948d1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.061+0000 7ff3837fb740 1 --2- 172.21.15.49:0/1126994409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c0360 0x5560948ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.061+0000 7ff3837fb740 1 -- 172.21.15.49:0/1126994409 >> 172.21.15.49:0/1126994409 conn(0x5560948bb8d0 msgr2=0x5560948bbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:14.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.062+0000 7ff3837fb740 1 -- 172.21.15.49:0/1126994409 shutdown_connections 2024-03-18T00:51:14.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.062+0000 7ff3837fb740 1 -- 172.21.15.49:0/1126994409 wait complete. 2024-03-18T00:51:14.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.062+0000 7ff3837fb740 1 -- start start 2024-03-18T00:51:14.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.062+0000 7ff3837fb740 1 Processor -- start 2024-03-18T00:51:14.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.063+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.064+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.064+0000 7ff3837fb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c5ec0 0x5560948d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.064+0000 7ff37ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.064+0000 7ff37ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948cf4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59946/0 (socket says 172.21.15.49:59946) 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.064+0000 7ff37ad9d640 1 -- 172.21.15.49:0/4178942071 learned_addr learned my addr 172.21.15.49:0/4178942071 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.064+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c1c17f0 con 0x5560948c0360 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.064+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c1c1120 con 0x5560948c5ec0 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.065+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff37c1be360 con 0x5560948c0d50 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.065+0000 7ff3837fb740 10 client.0 _ll_register_callbacks cb 0x7ff37400a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.066+0000 7ff37ad9d640 1 -- 172.21.15.49:0/4178942071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 msgr2=0x5560948cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.066+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.066+0000 7ff37ad9d640 1 -- 172.21.15.49:0/4178942071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c5ec0 msgr2=0x5560948d0a40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.066+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c5ec0 0x5560948d0a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.066+0000 7ff37ad9d640 1 -- 172.21.15.49:0/4178942071 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560944f90f0 con 0x5560948c0360 2024-03-18T00:51:14.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.066+0000 7ff37ad9d640 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948cf4f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff374005860 tx=0x7ff374004c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10202fd6a1c8e9b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:14.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.066+0000 7ff361ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.067+0000 7ff361ffb640 1 -- 172.21.15.49:0/4178942071 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3740058e0 con 0x5560948c0360 2024-03-18T00:51:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.067+0000 7ff361ffb640 1 -- 172.21.15.49:0/4178942071 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff374004f80 con 0x5560948c0360 2024-03-18T00:51:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.068+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff374003f40 con 0x5560948c0360 2024-03-18T00:51:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.068+0000 7ff361ffb640 1 -- 172.21.15.49:0/4178942071 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7ff374005180 con 0x5560948c0360 2024-03-18T00:51:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.068+0000 7ff3837fb740 10 client.15717 fetch_fsmap learned FSMap version 25 2024-03-18T00:51:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.068+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5560947ec9d0 con 0x5560948c0360 2024-03-18T00:51:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.068+0000 7ff361ffb640 1 -- 172.21.15.49:0/4178942071 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff374003f40 con 0x5560948c0360 2024-03-18T00:51:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.069+0000 7ff361ffb640 1 -- 172.21.15.49:0/4178942071 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 25) ==== 8354+0+0 (secure 0 0 0) 0x7ff374037040 con 0x5560948c0360 2024-03-18T00:51:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.069+0000 7ff3837fb740 10 client.15717 fetch_fsmap finished waiting for FSMap version 25 2024-03-18T00:51:14.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.069+0000 7ff3837fb740 10 client.15717 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:14.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.069+0000 7ff3837fb740 20 client.15717 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:14.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.069+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55609493c750 0x55609493eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.069+0000 7ff3837fb740 4 client.15717 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:51:14.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.069+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55609482cd90 con 0x55609493c750 2024-03-18T00:51:14.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.072+0000 7ff373fff640 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55609493c750 0x55609493eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:14.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.076+0000 7ff373fff640 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55609493c750 0x55609493eb70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dbef095ee4f82f server_cookie=625bc76f65972038 in_seq=0 out_seq=0 2024-03-18T00:51:14.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.076+0000 7ff361ffb640 10 client.15717 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:14.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.078+0000 7ff361ffb640 1 -- 172.21.15.49:0/4178942071 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55609482cd90 con 0x55609493c750 2024-03-18T00:51:14.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.078+0000 7ff361ffb640 10 client.15717 handle_command_reply: tid=0 2024-03-18T00:51:14.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.078+0000 7ff3837fb740 1 client.15717 shutdown 2024-03-18T00:51:14.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.079+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 msgr2=0x5560948cf4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:14.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.079+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948cf4f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff374005860 tx=0x7ff374004c30 comp rx=0 tx=0).stop 2024-03-18T00:51:14.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.080+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 shutdown_connections 2024-03-18T00:51:14.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.081+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55609493c750 0x55609493eb70 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.081+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5560948c5ec0 0x5560948d0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.081+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5560948c0d50 0x5560948cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.081+0000 7ff3837fb740 1 --2- 172.21.15.49:0/4178942071 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5560948c0360 0x5560948cf4f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.082+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 >> 172.21.15.49:0/4178942071 conn(0x5560948bb8d0 msgr2=0x5560948ca540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.082+0000 7ff361ffb640 0 client.15717 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.082+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 shutdown_connections 2024-03-18T00:51:14.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.082+0000 7ff3837fb740 1 -- 172.21.15.49:0/4178942071 wait complete. 2024-03-18T00:51:14.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.083+0000 7ff3837fb740 20 client.15717 trim_cache size 0 max 16384 2024-03-18T00:51:14.087 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:51:14.087 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:51:14.088 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:51:14.088 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:51:14.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.087+0000 7ff37b59e640 1 -- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3680a28e0 msgr2=0x7ff3680a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:14.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.087+0000 7ff37b59e640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3680a28e0 0x7ff3680a4da0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff364009610 tx=0x7ff364005880 comp rx=0 tx=0).stop 2024-03-18T00:51:14.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.087+0000 7ff37b59e640 1 -- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c1089c0 msgr2=0x7ff37c1192f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:14.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.087+0000 7ff37b59e640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c1089c0 0x7ff37c1192f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff36400d950 tx=0x7ff36400de10 comp rx=0 tx=0).stop 2024-03-18T00:51:14.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.088+0000 7ff37b59e640 1 -- 172.21.15.49:0/2906213634 shutdown_connections 2024-03-18T00:51:14.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.088+0000 7ff37b59e640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff3680a28e0 0x7ff3680a4da0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.088+0000 7ff37b59e640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff37c10b2f0 0x7ff37c11e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.088+0000 7ff37b59e640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff37c10dd70 0x7ff37c119830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.088+0000 7ff37b59e640 1 --2- 172.21.15.49:0/2906213634 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff37c1089c0 0x7ff37c1192f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.088+0000 7ff37b59e640 1 -- 172.21.15.49:0/2906213634 >> 172.21.15.49:0/2906213634 conn(0x7ff37c06e6c0 msgr2=0x7ff37c10b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:14.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.088+0000 7ff37b59e640 1 -- 172.21.15.49:0/2906213634 shutdown_connections 2024-03-18T00:51:14.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.088+0000 7ff37b59e640 1 -- 172.21.15.49:0/2906213634 wait complete. 2024-03-18T00:51:14.143 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:51:14.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:51:14.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.587+0000 7f256a397640 1 -- 172.21.15.49:0/2946722882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f256410db20 msgr2=0x7f256410dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:14.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.587+0000 7f256a397640 1 --2- 172.21.15.49:0/2946722882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f256410db20 0x7f256410dfd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2554009860 tx=0x7f255402bbb0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.589+0000 7f256a397640 1 -- 172.21.15.49:0/2946722882 shutdown_connections 2024-03-18T00:51:14.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.589+0000 7f256a397640 1 --2- 172.21.15.49:0/2946722882 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f256410b150 0x7f256410d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.589+0000 7f256a397640 1 --2- 172.21.15.49:0/2946722882 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f256410db20 0x7f256410dfd0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.589+0000 7f256a397640 1 --2- 172.21.15.49:0/2946722882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25641087e0 0x7f256410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.589+0000 7f256a397640 1 -- 172.21.15.49:0/2946722882 >> 172.21.15.49:0/2946722882 conn(0x7f256406e650 msgr2=0x7f2564070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:14.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.594+0000 7f256a397640 1 -- 172.21.15.49:0/2946722882 shutdown_connections 2024-03-18T00:51:14.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.598+0000 7f256a397640 1 -- 172.21.15.49:0/2946722882 wait complete. 2024-03-18T00:51:14.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.599+0000 7f256a397640 1 Processor -- start 2024-03-18T00:51:14.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.599+0000 7f256a397640 1 -- start start 2024-03-18T00:51:14.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.600+0000 7f256a397640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25641087e0 0x7f2564114e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.600+0000 7f256a397640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256410b150 0x7f25641153a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.600+0000 7f256a397640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f256410db20 0x7f25641b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.600+0000 7f256a397640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f256411c360 con 0x7f25641087e0 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.600+0000 7f256a397640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f256411c1e0 con 0x7f256410db20 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.600+0000 7f256a397640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f256411c4e0 con 0x7f256410b150 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.601+0000 7f2568934640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f256410db20 0x7f25641b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.601+0000 7f2568934640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f256410db20 0x7f25641b6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:42996/0 (socket says 172.21.15.49:42996) 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.601+0000 7f2568934640 1 -- 172.21.15.49:0/347626573 learned_addr learned my addr 172.21.15.49:0/347626573 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.602+0000 7f2568934640 1 -- 172.21.15.49:0/347626573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256410b150 msgr2=0x7f25641153a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.602+0000 7f2568934640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256410b150 0x7f25641153a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.602+0000 7f2568934640 1 -- 172.21.15.49:0/347626573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25641087e0 msgr2=0x7f2564114e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.602+0000 7f2568934640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25641087e0 0x7f2564114e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:14.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.602+0000 7f2568934640 1 -- 172.21.15.49:0/347626573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25641184d0 con 0x7f256410db20 2024-03-18T00:51:14.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.602+0000 7f2568934640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f256410db20 0x7f25641b6a70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2554034be0 tx=0x7f25540045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:14.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.604+0000 7f25617fa640 1 -- 172.21.15.49:0/347626573 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2554039040 con 0x7f256410db20 2024-03-18T00:51:14.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.604+0000 7f25617fa640 1 -- 172.21.15.49:0/347626573 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2554042070 con 0x7f256410db20 2024-03-18T00:51:14.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.604+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25641b7010 con 0x7f256410db20 2024-03-18T00:51:14.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.605+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25641b7550 con 0x7f256410db20 2024-03-18T00:51:14.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.605+0000 7f25617fa640 1 -- 172.21.15.49:0/347626573 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2554004b60 con 0x7f256410db20 2024-03-18T00:51:14.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.612+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2544005290 con 0x7f256410db20 2024-03-18T00:51:14.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.613+0000 7f25617fa640 1 -- 172.21.15.49:0/347626573 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2554004d00 con 0x7f256410db20 2024-03-18T00:51:14.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.615+0000 7f25617fa640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25500a2940 0x7f25500a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:14.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.616+0000 7f25617fa640 1 -- 172.21.15.49:0/347626573 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f25540f0370 con 0x7f256410db20 2024-03-18T00:51:14.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.619+0000 7f2563fff640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25500a2940 0x7f25500a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:14.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.627+0000 7f25617fa640 1 -- 172.21.15.49:0/347626573 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f25540bd890 con 0x7f256410db20 2024-03-18T00:51:14.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:14.629+0000 7f2563fff640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25500a2940 0x7f25500a4e00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f25640728d0 tx=0x7f254c006bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:15.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.144+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2544005580 con 0x7f256410db20 2024-03-18T00:51:15.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.147+0000 7f25617fa640 1 -- 172.21.15.49:0/347626573 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11933 (secure 0 0 0) 0x7f2554009440 con 0x7f256410db20 2024-03-18T00:51:15.151 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:51:15.151 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:51:14.980346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24506,24499,24481,24463,24470]},"id":1}]} 2024-03-18T00:51:15.152 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 26 2024-03-18T00:51:15.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.157+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25500a2940 msgr2=0x7f25500a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.157+0000 7f256a397640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25500a2940 0x7f25500a4e00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f25640728d0 tx=0x7f254c006bd0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.157+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f256410db20 msgr2=0x7f25641b6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.157+0000 7f256a397640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f256410db20 0x7f25641b6a70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2554034be0 tx=0x7f25540045e0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.158+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 shutdown_connections 2024-03-18T00:51:15.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.158+0000 7f256a397640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f25500a2940 0x7f25500a4e00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.158+0000 7f256a397640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f256410db20 0x7f25641b6a70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.158+0000 7f256a397640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f256410b150 0x7f25641153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.158+0000 7f256a397640 1 --2- 172.21.15.49:0/347626573 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f25641087e0 0x7f2564114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.158+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 >> 172.21.15.49:0/347626573 conn(0x7f256406e650 msgr2=0x7f256410a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:15.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.159+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 shutdown_connections 2024-03-18T00:51:15.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.159+0000 7f256a397640 1 -- 172.21.15.49:0/347626573 wait complete. 2024-03-18T00:51:15.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:51:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:15 smithi049 ceph-mon[32130]: pgmap v384: 97 pgs: 97 active+clean; 9.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 919 MiB/s wr, 1.78k op/s 2024-03-18T00:51:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:15 smithi107 ceph-mon[36270]: pgmap v384: 97 pgs: 97 active+clean; 9.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 919 MiB/s wr, 1.78k op/s 2024-03-18T00:51:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:15 smithi106 ceph-mon[35899]: pgmap v384: 97 pgs: 97 active+clean; 9.4 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 919 MiB/s wr, 1.78k op/s 2024-03-18T00:51:15.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.621+0000 7fa41d863640 1 -- 172.21.15.49:0/3636041168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa41810db00 msgr2=0x7fa41810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.621+0000 7fa41d863640 1 --2- 172.21.15.49:0/3636041168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa41810db00 0x7fa41810df80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa410009ca0 tx=0x7fa41002c140 comp rx=0 tx=0).stop 2024-03-18T00:51:15.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.628+0000 7fa41d863640 1 -- 172.21.15.49:0/3636041168 shutdown_connections 2024-03-18T00:51:15.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.628+0000 7fa41d863640 1 --2- 172.21.15.49:0/3636041168 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa41810db00 0x7fa41810df80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.628+0000 7fa41d863640 1 --2- 172.21.15.49:0/3636041168 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41810b150 0x7fa41810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.628+0000 7fa41d863640 1 --2- 172.21.15.49:0/3636041168 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa4181087e0 0x7fa41810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.628+0000 7fa41d863640 1 -- 172.21.15.49:0/3636041168 >> 172.21.15.49:0/3636041168 conn(0x7fa41806e640 msgr2=0x7fa418070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:15.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.635+0000 7fa41d863640 1 -- 172.21.15.49:0/3636041168 shutdown_connections 2024-03-18T00:51:15.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.635+0000 7fa41d863640 1 -- 172.21.15.49:0/3636041168 wait complete. 2024-03-18T00:51:15.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.637+0000 7fa41d863640 1 Processor -- start 2024-03-18T00:51:15.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.638+0000 7fa41d863640 1 -- start start 2024-03-18T00:51:15.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.639+0000 7fa41d863640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa4181087e0 0x7fa418119ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.640+0000 7fa41d863640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa41810b150 0x7fa41811a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.640+0000 7fa416ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa4181087e0 0x7fa418119ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.640+0000 7fa416ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa4181087e0 0x7fa418119ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59968/0 (socket says 172.21.15.49:59968) 2024-03-18T00:51:15.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.640+0000 7fa41d863640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41810db00 0x7fa418118170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.640+0000 7fa416ffd640 1 -- 172.21.15.49:0/3709591197 learned_addr learned my addr 172.21.15.49:0/3709591197 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:15.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.640+0000 7fa41d863640 1 -- 172.21.15.49:0/3709591197 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa418110960 con 0x7fa4181087e0 2024-03-18T00:51:15.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.640+0000 7fa41d863640 1 -- 172.21.15.49:0/3709591197 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181107e0 con 0x7fa41810b150 2024-03-18T00:51:15.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.641+0000 7fa41d863640 1 -- 172.21.15.49:0/3709591197 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa418110ae0 con 0x7fa41810db00 2024-03-18T00:51:15.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.642+0000 7fa4177fe640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41810db00 0x7fa418118170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.643+0000 7fa4167fc640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa41810b150 0x7fa41811a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.644+0000 7fa416ffd640 1 -- 172.21.15.49:0/3709591197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41810db00 msgr2=0x7fa418118170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.644+0000 7fa416ffd640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41810db00 0x7fa418118170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.644+0000 7fa416ffd640 1 -- 172.21.15.49:0/3709591197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa41810b150 msgr2=0x7fa41811a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.644+0000 7fa416ffd640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa41810b150 0x7fa41811a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.644+0000 7fa416ffd640 1 -- 172.21.15.49:0/3709591197 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa418118990 con 0x7fa4181087e0 2024-03-18T00:51:15.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.644+0000 7fa416ffd640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa4181087e0 0x7fa418119ae0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00d950 tx=0x7fa40c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:15.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.655+0000 7fa41c861640 1 -- 172.21.15.49:0/3709591197 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c017070 con 0x7fa4181087e0 2024-03-18T00:51:15.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.656+0000 7fa41c861640 1 -- 172.21.15.49:0/3709591197 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa40c009c30 con 0x7fa4181087e0 2024-03-18T00:51:15.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.656+0000 7fa41d863640 1 -- 172.21.15.49:0/3709591197 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa418118c80 con 0x7fa4181087e0 2024-03-18T00:51:15.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.656+0000 7fa41d863640 1 -- 172.21.15.49:0/3709591197 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4181b7650 con 0x7fa4181087e0 2024-03-18T00:51:15.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.663+0000 7fa420446740 1 Processor -- start 2024-03-18T00:51:15.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.663+0000 7fa420446740 1 -- start start 2024-03-18T00:51:15.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.663+0000 7fa41c861640 1 -- 172.21.15.49:0/3709591197 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c00f620 con 0x7fa4181087e0 2024-03-18T00:51:15.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.665+0000 7fa420446740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcc1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.665+0000 7fa420446740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcc7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.665+0000 7fa420446740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcc9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.665+0000 7fa420446740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55bdbbc26490 con 0x55bdbbcc7650 2024-03-18T00:51:15.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.665+0000 7fa420446740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55bdbbbcec90 con 0x55bdbbcc24f0 2024-03-18T00:51:15.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.665+0000 7fa420446740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55bdbbc22530 con 0x55bdbbcc1b90 2024-03-18T00:51:15.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.666+0000 7fa4167fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcc9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.666+0000 7fa4167fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcc9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:59980/0 (socket says 172.21.15.49:59980) 2024-03-18T00:51:15.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.666+0000 7fa4167fc640 1 -- 172.21.15.49:0/4243290828 learned_addr learned my addr 172.21.15.49:0/4243290828 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:15.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.666+0000 7fa4177fe640 1 --2- 172.21.15.49:0/4243290828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcc1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.669+0000 7fa4167fc640 1 -- 172.21.15.49:0/4243290828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 msgr2=0x55bdbbcc1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:15.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.669+0000 7fa4167fc640 1 --2- 172.21.15.49:0/4243290828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcc1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.669+0000 7fa4167fc640 1 -- 172.21.15.49:0/4243290828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 msgr2=0x55bdbbcc7080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:51:15.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.669+0000 7fa4167fc640 1 --2- 172.21.15.49:0/4243290828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcc7080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.669+0000 7fa4167fc640 1 -- 172.21.15.49:0/4243290828 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdbbc89500 con 0x55bdbbcc7650 2024-03-18T00:51:15.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.671+0000 7fa41c861640 1 -- 172.21.15.49:0/3709591197 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa40c010460 con 0x7fa4181087e0 2024-03-18T00:51:15.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.671+0000 7fa4167fc640 1 --2- 172.21.15.49:0/4243290828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcc9a60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa40800b7f0 tx=0x7fa40800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21379d028f6c4f98 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:15.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.673+0000 7fa41c861640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4000a2a00 0x7fa4000a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.673+0000 7fa41c861640 1 -- 172.21.15.49:0/3709591197 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa40c0ce2f0 con 0x7fa4181087e0 2024-03-18T00:51:15.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.672+0000 7fa3fd7fa640 1 -- 172.21.15.49:0/4243290828 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40800d950 con 0x55bdbbcc7650 2024-03-18T00:51:15.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.673+0000 7fa3fd7fa640 1 -- 172.21.15.49:0/4243290828 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa408018070 con 0x55bdbbcc7650 2024-03-18T00:51:15.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.673+0000 7fa3fd7fa640 1 -- 172.21.15.49:0/4243290828 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4080123e0 con 0x55bdbbcc7650 2024-03-18T00:51:15.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.675+0000 7fa420446740 1 -- 172.21.15.49:0/4243290828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 msgr2=0x55bdbbcc9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.675+0000 7fa420446740 1 --2- 172.21.15.49:0/4243290828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcc9a60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa40800b7f0 tx=0x7fa40800bcb0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.676+0000 7fa4177fe640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4000a2a00 0x7fa4000a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.684+0000 7fa4177fe640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4000a2a00 0x7fa4000a4ec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa410037040 tx=0x7fa41002e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:15.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.685+0000 7fa420446740 1 -- 172.21.15.49:0/4243290828 shutdown_connections 2024-03-18T00:51:15.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.685+0000 7fa420446740 1 --2- 172.21.15.49:0/4243290828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcc7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.685+0000 7fa420446740 1 --2- 172.21.15.49:0/4243290828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcc9a60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.685+0000 7fa420446740 1 --2- 172.21.15.49:0/4243290828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcc1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.685+0000 7fa420446740 1 -- 172.21.15.49:0/4243290828 >> 172.21.15.49:0/4243290828 conn(0x55bdbbcbd100 msgr2=0x55bdbbcbd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:15.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.685+0000 7fa420446740 1 -- 172.21.15.49:0/4243290828 shutdown_connections 2024-03-18T00:51:15.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.686+0000 7fa420446740 1 -- 172.21.15.49:0/4243290828 wait complete. 2024-03-18T00:51:15.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.687+0000 7fa420446740 1 -- start start 2024-03-18T00:51:15.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.687+0000 7fa420446740 1 Processor -- start 2024-03-18T00:51:15.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.688+0000 7fa420446740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbccbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.688+0000 7fa420446740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbccc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.688+0000 7fa420446740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcd33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.688+0000 7fa420446740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55bdbbcca580 con 0x55bdbbcc7650 2024-03-18T00:51:15.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.688+0000 7fa420446740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55bdbbbe2230 con 0x55bdbbcc24f0 2024-03-18T00:51:15.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.688+0000 7fa420446740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55bdbbcca700 con 0x55bdbbcc1b90 2024-03-18T00:51:15.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.688+0000 7fa420446740 10 client.0 _ll_register_callbacks cb 0x55bdbbc1e010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:15.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.690+0000 7fa416ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbccbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.690+0000 7fa416ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbccbe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60282/0 (socket says 172.21.15.49:60282) 2024-03-18T00:51:15.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.690+0000 7fa416ffd640 1 -- 172.21.15.49:0/2128915129 learned_addr learned my addr 172.21.15.49:0/2128915129 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.690+0000 7fa4167fc640 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbccc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.691+0000 7fa4177fe640 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcd33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.692+0000 7fa4167fc640 1 -- 172.21.15.49:0/2128915129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 msgr2=0x55bdbbccbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:15.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.692+0000 7fa4167fc640 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbccbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.692+0000 7fa4167fc640 1 -- 172.21.15.49:0/2128915129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 msgr2=0x55bdbbcd33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:15.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.692+0000 7fa4167fc640 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcd33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.692+0000 7fa4167fc640 1 -- 172.21.15.49:0/2128915129 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdbbbf1100 con 0x55bdbbcc24f0 2024-03-18T00:51:15.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.693+0000 7fa4167fc640 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbccc3b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa408006fd0 tx=0x7fa408014f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e682976f75419350 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:15.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.694+0000 7fa3fb7f6640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:51:15.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.694+0000 7fa3fb7f6640 1 -- 172.21.15.49:0/2128915129 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40800db50 con 0x55bdbbcc24f0 2024-03-18T00:51:15.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.694+0000 7fa420446740 1 -- 172.21.15.49:0/2128915129 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa408018070 con 0x55bdbbcc24f0 2024-03-18T00:51:15.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.694+0000 7fa3fb7f6640 1 -- 172.21.15.49:0/2128915129 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa40800dcf0 con 0x55bdbbcc24f0 2024-03-18T00:51:15.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.695+0000 7fa3fb7f6640 1 -- 172.21.15.49:0/2128915129 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa408013710 con 0x55bdbbcc24f0 2024-03-18T00:51:15.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.695+0000 7fa420446740 10 client.25544 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:15.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.695+0000 7fa420446740 1 -- 172.21.15.49:0/2128915129 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bdbb93a3f0 con 0x55bdbbcc24f0 2024-03-18T00:51:15.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.698+0000 7fa3fb7f6640 1 -- 172.21.15.49:0/2128915129 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa408018070 con 0x55bdbbcc24f0 2024-03-18T00:51:15.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.698+0000 7fa3fb7f6640 1 -- 172.21.15.49:0/2128915129 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fa40801c3f0 con 0x55bdbbcc24f0 2024-03-18T00:51:15.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.698+0000 7fa420446740 10 client.25544 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:15.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.698+0000 7fa420446740 10 client.25544 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:51:15.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.698+0000 7fa420446740 20 client.25544 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:15.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.698+0000 7fa420446740 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55bdbbcd7fe0 0x55bdbbcda400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.698+0000 7fa420446740 4 client.25544 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:15.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.698+0000 7fa420446740 1 -- 172.21.15.49:0/2128915129 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bdbbc339d0 con 0x55bdbbcd7fe0 2024-03-18T00:51:15.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.702+0000 7fa416ffd640 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55bdbbcd7fe0 0x55bdbbcda400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.709+0000 7fa416ffd640 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55bdbbcd7fe0 0x55bdbbcda400 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c9e5164f16a63f6b server_cookie=a4062ca992a65aef in_seq=0 out_seq=0 2024-03-18T00:51:15.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.709+0000 7fa3fb7f6640 10 client.25544 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:51:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.711+0000 7fa3fb7f6640 1 -- 172.21.15.49:0/2128915129 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bdbbc339d0 con 0x55bdbbcd7fe0 2024-03-18T00:51:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.712+0000 7fa3fb7f6640 10 client.25544 handle_command_reply: tid=0 2024-03-18T00:51:15.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.712+0000 7fa420446740 1 client.25544 shutdown 2024-03-18T00:51:15.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.716+0000 7fa420446740 1 -- 172.21.15.49:0/2128915129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 msgr2=0x55bdbbccc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.716+0000 7fa420446740 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbccc3b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa408006fd0 tx=0x7fa408014f30 comp rx=0 tx=0).stop 2024-03-18T00:51:15.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.717+0000 7fa420446740 1 -- 172.21.15.49:0/2128915129 shutdown_connections 2024-03-18T00:51:15.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.717+0000 7fa420446740 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55bdbbcd7fe0 0x55bdbbcda400 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.717+0000 7fa420446740 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbccc3b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.717+0000 7fa420446740 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcc7650 0x55bdbbcd33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.717+0000 7fa420446740 1 --2- 172.21.15.49:0/2128915129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbccbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.717+0000 7fa420446740 1 -- 172.21.15.49:0/2128915129 >> 172.21.15.49:0/2128915129 conn(0x55bdbbcbd100 msgr2=0x55bdbbcbc360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:15.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.717+0000 7fa3fb7f6640 0 client.25544 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:51:15.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.718+0000 7fa420446740 1 -- 172.21.15.49:0/2128915129 shutdown_connections 2024-03-18T00:51:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.718+0000 7fa420446740 1 -- 172.21.15.49:0/2128915129 wait complete. 2024-03-18T00:51:15.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.718+0000 7fa420446740 20 client.25544 trim_cache size 0 max 16384 2024-03-18T00:51:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.762+0000 7fa420446740 1 Processor -- start 2024-03-18T00:51:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa420446740 1 -- start start 2024-03-18T00:51:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa420446740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcc0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa420446740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcd33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa420446740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcdece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa420446740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181c0570 con 0x55bdbbcbd100 2024-03-18T00:51:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa420446740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181c20e0 con 0x55bdbbcc24f0 2024-03-18T00:51:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa420446740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181c0c40 con 0x55bdbbcc1b90 2024-03-18T00:51:15.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa416ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcc0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa416ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcc0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60010/0 (socket says 172.21.15.49:60010) 2024-03-18T00:51:15.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.763+0000 7fa416ffd640 1 -- 172.21.15.49:0/1081388027 learned_addr learned my addr 172.21.15.49:0/1081388027 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:15.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.764+0000 7fa4177fe640 1 --2- 172.21.15.49:0/1081388027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcdece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.764+0000 7fa416ffd640 1 -- 172.21.15.49:0/1081388027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 msgr2=0x55bdbbcd33b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.764+0000 7fa416ffd640 1 --2- 172.21.15.49:0/1081388027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcd33b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.764+0000 7fa416ffd640 1 -- 172.21.15.49:0/1081388027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 msgr2=0x55bdbbcdece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.764+0000 7fa416ffd640 1 --2- 172.21.15.49:0/1081388027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcdece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.764+0000 7fa416ffd640 1 -- 172.21.15.49:0/1081388027 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bdbba047d0 con 0x55bdbbcbd100 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.764+0000 7fa4177fe640 1 --2- 172.21.15.49:0/1081388027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcdece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.764+0000 7fa416ffd640 1 --2- 172.21.15.49:0/1081388027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcc0a70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0058c0 tx=0x7fa40c010c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994849f14cbe5d15 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.765+0000 7fa3fcff9640 1 -- 172.21.15.49:0/1081388027 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0b3050 con 0x55bdbbcbd100 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.765+0000 7fa3fcff9640 1 -- 172.21.15.49:0/1081388027 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa40c09ed60 con 0x55bdbbcbd100 2024-03-18T00:51:15.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.765+0000 7fa3fcff9640 1 -- 172.21.15.49:0/1081388027 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c020040 con 0x55bdbbcbd100 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.766+0000 7fa420446740 1 -- 172.21.15.49:0/1081388027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 msgr2=0x55bdbbcc0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.766+0000 7fa420446740 1 --2- 172.21.15.49:0/1081388027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcc0a70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0058c0 tx=0x7fa40c010c00 comp rx=0 tx=0).stop 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.766+0000 7fa420446740 1 -- 172.21.15.49:0/1081388027 shutdown_connections 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.766+0000 7fa420446740 1 --2- 172.21.15.49:0/1081388027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcdece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.766+0000 7fa420446740 1 --2- 172.21.15.49:0/1081388027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcd33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.766+0000 7fa420446740 1 --2- 172.21.15.49:0/1081388027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcc0a70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.766+0000 7fa420446740 1 -- 172.21.15.49:0/1081388027 >> 172.21.15.49:0/1081388027 conn(0x55bdbbccba30 msgr2=0x55bdbbcc0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.766+0000 7fa420446740 1 -- 172.21.15.49:0/1081388027 shutdown_connections 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.766+0000 7fa420446740 1 -- 172.21.15.49:0/1081388027 wait complete. 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.767+0000 7fa420446740 1 -- start start 2024-03-18T00:51:15.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.767+0000 7fa420446740 1 Processor -- start 2024-03-18T00:51:15.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.767+0000 7fa420446740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcd0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.767+0000 7fa420446740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcd0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.767+0000 7fa420446740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcd1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.767+0000 7fa420446740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181c19e0 con 0x55bdbbcbd100 2024-03-18T00:51:15.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.767+0000 7fa420446740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181c1310 con 0x55bdbbcc1b90 2024-03-18T00:51:15.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.767+0000 7fa420446740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181be550 con 0x55bdbbcc24f0 2024-03-18T00:51:15.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.767+0000 7fa420446740 10 client.0 _ll_register_callbacks cb 0x7fa40800b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa416ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcd0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa416ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcd0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60016/0 (socket says 172.21.15.49:60016) 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa416ffd640 1 -- 172.21.15.49:0/1677527653 learned_addr learned my addr 172.21.15.49:0/1677527653 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa4167fc640 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcd0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa416ffd640 1 -- 172.21.15.49:0/1677527653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc24f0 msgr2=0x55bdbbcd1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa416ffd640 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcd1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa416ffd640 1 -- 172.21.15.49:0/1677527653 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc1b90 msgr2=0x55bdbbcd0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa416ffd640 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcd0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa416ffd640 1 -- 172.21.15.49:0/1677527653 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bdbbcd22c0 con 0x55bdbbcbd100 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa416ffd640 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcd0780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0990f0 tx=0x7fa40c0974b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c26ef466fcf44a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.768+0000 7fa3faff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.769+0000 7fa3faff5640 1 -- 172.21.15.49:0/1677527653 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c020070 con 0x55bdbbcbd100 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.769+0000 7fa3faff5640 1 -- 172.21.15.49:0/1677527653 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa40c0b92e0 con 0x55bdbbcbd100 2024-03-18T00:51:15.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.769+0000 7fa420446740 1 -- 172.21.15.49:0/1677527653 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa40c09ed60 con 0x55bdbbcbd100 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.769+0000 7fa3faff5640 1 -- 172.21.15.49:0/1677527653 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa40c0b3040 con 0x55bdbbcbd100 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.770+0000 7fa420446740 10 client.15738 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.770+0000 7fa420446740 1 -- 172.21.15.49:0/1677527653 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55bdbbcd35a0 con 0x55bdbbcbd100 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.770+0000 7fa3faff5640 1 -- 172.21.15.49:0/1677527653 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa40c09ed60 con 0x55bdbbcbd100 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.770+0000 7fa3faff5640 1 -- 172.21.15.49:0/1677527653 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fa40c023060 con 0x55bdbbcbd100 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.770+0000 7fa420446740 10 client.15738 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.771+0000 7fa420446740 10 client.15738 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.771+0000 7fa420446740 20 client.15738 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.771+0000 7fa420446740 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55bdbbcc7650 0x55bdbbd40900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.771+0000 7fa420446740 4 client.15738 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.771+0000 7fa420446740 1 -- 172.21.15.49:0/1677527653 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bdbbc2dd20 con 0x55bdbbcc7650 2024-03-18T00:51:15.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.772+0000 7fa4167fc640 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55bdbbcc7650 0x55bdbbd40900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.773+0000 7fa4167fc640 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55bdbbcc7650 0x55bdbbd40900 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b544b5bb8ede41d3 server_cookie=6b36f0e39b27c27b in_seq=0 out_seq=0 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.773+0000 7fa3faff5640 10 client.15738 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.773+0000 7fa3faff5640 1 -- 172.21.15.49:0/1677527653 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bdbbc2dd20 con 0x55bdbbcc7650 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.773+0000 7fa3faff5640 10 client.15738 handle_command_reply: tid=0 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.774+0000 7fa420446740 1 client.15738 shutdown 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.774+0000 7fa420446740 1 -- 172.21.15.49:0/1677527653 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 msgr2=0x55bdbbcd0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.774+0000 7fa420446740 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcd0780 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa40c0990f0 tx=0x7fa40c0974b0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.774+0000 7fa420446740 1 -- 172.21.15.49:0/1677527653 shutdown_connections 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.774+0000 7fa420446740 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55bdbbcc7650 0x55bdbbd40900 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.774+0000 7fa3faff5640 0 client.15738 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:51:15.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.774+0000 7fa420446740 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55bdbbcc1b90 0x55bdbbcd0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.774+0000 7fa420446740 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55bdbbcc24f0 0x55bdbbcd1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.775+0000 7fa420446740 1 --2- 172.21.15.49:0/1677527653 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55bdbbcbd100 0x55bdbbcd0780 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.775+0000 7fa420446740 1 -- 172.21.15.49:0/1677527653 >> 172.21.15.49:0/1677527653 conn(0x55bdbbccba30 msgr2=0x55bdbbcc0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:15.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.776+0000 7fa420446740 1 -- 172.21.15.49:0/1677527653 shutdown_connections 2024-03-18T00:51:15.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.776+0000 7fa420446740 1 -- 172.21.15.49:0/1677527653 wait complete. 2024-03-18T00:51:15.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.776+0000 7fa420446740 20 client.15738 trim_cache size 0 max 16384 2024-03-18T00:51:15.777 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:51:15.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 -- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4000a2a00 msgr2=0x7fa4000a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4000a2a00 0x7fa4000a4ec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa410037040 tx=0x7fa41002e040 comp rx=0 tx=0).stop 2024-03-18T00:51:15.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 -- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa4181087e0 msgr2=0x7fa418119ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:15.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa4181087e0 0x7fa418119ae0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00d950 tx=0x7fa40c00de10 comp rx=0 tx=0).stop 2024-03-18T00:51:15.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 -- 172.21.15.49:0/3709591197 shutdown_connections 2024-03-18T00:51:15.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4000a2a00 0x7fa4000a4ec0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa41810b150 0x7fa41811a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa41810db00 0x7fa418118170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 --2- 172.21.15.49:0/3709591197 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa4181087e0 0x7fa418119ae0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:15.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.778+0000 7fa3fe7fc640 1 -- 172.21.15.49:0/3709591197 >> 172.21.15.49:0/3709591197 conn(0x7fa41806e640 msgr2=0x7fa418109010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:15.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.779+0000 7fa3fe7fc640 1 -- 172.21.15.49:0/3709591197 shutdown_connections 2024-03-18T00:51:15.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:15.779+0000 7fa3fe7fc640 1 -- 172.21.15.49:0/3709591197 wait complete. 2024-03-18T00:51:15.813 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:51:16.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.221+0000 7f2e8e551640 1 -- 172.21.15.49:0/3567583321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e881005c0 msgr2=0x7f2e881009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.221+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3567583321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e881005c0 0x7f2e881009e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2e84009900 tx=0x7f2e8402bc60 comp rx=0 tx=0).stop 2024-03-18T00:51:16.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.221+0000 7f2e8e551640 1 -- 172.21.15.49:0/3567583321 shutdown_connections 2024-03-18T00:51:16.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.221+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3567583321 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e881017b0 0x7f2e88101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.221+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3567583321 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e88102170 0x7f2e88108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.221+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3567583321 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e881005c0 0x7f2e881009e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.221+0000 7f2e8e551640 1 -- 172.21.15.49:0/3567583321 >> 172.21.15.49:0/3567583321 conn(0x7f2e880fbb50 msgr2=0x7f2e880fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.222+0000 7f2e8e551640 1 -- 172.21.15.49:0/3567583321 shutdown_connections 2024-03-18T00:51:16.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.222+0000 7f2e8e551640 1 -- 172.21.15.49:0/3567583321 wait complete. 2024-03-18T00:51:16.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.223+0000 7f2e8e551640 1 Processor -- start 2024-03-18T00:51:16.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.223+0000 7f2e8e551640 1 -- start start 2024-03-18T00:51:16.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8e551640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e881005c0 0x7f2e880694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8e551640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e881017b0 0x7f2e8819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8e551640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e88102170 0x7f2e881a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8e551640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8810c310 con 0x7f2e881017b0 2024-03-18T00:51:16.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8e551640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8810c190 con 0x7f2e881005c0 2024-03-18T00:51:16.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8e551640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8810c490 con 0x7f2e88102170 2024-03-18T00:51:16.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8d54f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e881005c0 0x7f2e880694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8d54f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e881005c0 0x7f2e880694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:43068/0 (socket says 172.21.15.49:43068) 2024-03-18T00:51:16.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8d54f640 1 -- 172.21.15.49:0/3028323691 learned_addr learned my addr 172.21.15.49:0/3028323691 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8dd50640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e88102170 0x7f2e881a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.224+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e881017b0 0x7f2e8819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.225+0000 7f2e8d54f640 1 -- 172.21.15.49:0/3028323691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e88102170 msgr2=0x7f2e881a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.225+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e88102170 0x7f2e881a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.225+0000 7f2e8d54f640 1 -- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e881017b0 msgr2=0x7f2e8819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.225+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e881017b0 0x7f2e8819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.225+0000 7f2e8d54f640 1 -- 172.21.15.49:0/3028323691 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e881a18e0 con 0x7f2e881005c0 2024-03-18T00:51:16.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.226+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e881005c0 0x7f2e880694b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2e8402efd0 tx=0x7f2e84005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.227+0000 7f2e7e7fc640 1 -- 172.21.15.49:0/3028323691 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e84041070 con 0x7f2e881005c0 2024-03-18T00:51:16.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.227+0000 7f2e7e7fc640 1 -- 172.21.15.49:0/3028323691 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e840054e0 con 0x7f2e881005c0 2024-03-18T00:51:16.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.227+0000 7f2e7e7fc640 1 -- 172.21.15.49:0/3028323691 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840346a0 con 0x7f2e881005c0 2024-03-18T00:51:16.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.229+0000 7f2e8e551640 1 -- 172.21.15.49:0/3028323691 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e881a1b10 con 0x7f2e881005c0 2024-03-18T00:51:16.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.230+0000 7f2e8e551640 1 -- 172.21.15.49:0/3028323691 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e881a1ec0 con 0x7f2e881005c0 2024-03-18T00:51:16.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.235+0000 7f2e92397740 1 Processor -- start 2024-03-18T00:51:16.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.236+0000 7f2e92397740 1 -- start start 2024-03-18T00:51:16.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.236+0000 7f2e7e7fc640 1 -- 172.21.15.49:0/3028323691 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2e84044070 con 0x7f2e881005c0 2024-03-18T00:51:16.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.236+0000 7f2e92397740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c4fa40 0x563d45c4fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.236+0000 7f2e92397740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c54f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.237+0000 7f2e92397740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c57a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.237+0000 7f2e7e7fc640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2e740a2a00 0x7f2e740a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.237+0000 7f2e8d54f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c54f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.237+0000 7f2e8d54f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c54f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60050/0 (socket says 172.21.15.49:60050) 2024-03-18T00:51:16.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.237+0000 7f2e8d54f640 1 -- 172.21.15.49:0/3993018226 learned_addr learned my addr 172.21.15.49:0/3993018226 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.237+0000 7f2e92397740 1 -- 172.21.15.49:0/3993018226 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563d45bb4490 con 0x563d45c50430 2024-03-18T00:51:16.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.237+0000 7f2e7e7fc640 1 -- 172.21.15.49:0/3028323691 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2e840f0520 con 0x7f2e881005c0 2024-03-18T00:51:16.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.237+0000 7f2e92397740 1 -- 172.21.15.49:0/3993018226 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563d45b5cc90 con 0x563d45c4fa40 2024-03-18T00:51:16.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.237+0000 7f2e92397740 1 -- 172.21.15.49:0/3993018226 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563d45bb0530 con 0x563d45c555a0 2024-03-18T00:51:16.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.238+0000 7f2e8dd50640 1 --2- 172.21.15.49:0/3993018226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c4fa40 0x563d45c4fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.238+0000 7f2e8dd50640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2e740a2a00 0x7f2e740a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.238+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/3993018226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c57a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.238+0000 7f2e8dd50640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2e740a2a00 0x7f2e740a4ec0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009270 tx=0x7f2e800098a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.239+0000 7f2e8d54f640 1 -- 172.21.15.49:0/3993018226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 msgr2=0x563d45c57a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.239+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/3993018226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c57a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.239+0000 7f2e8d54f640 1 -- 172.21.15.49:0/3993018226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c4fa40 msgr2=0x563d45c4fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.239+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/3993018226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c4fa40 0x563d45c4fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.239+0000 7f2e8d54f640 1 -- 172.21.15.49:0/3993018226 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d45c17500 con 0x563d45c50430 2024-03-18T00:51:16.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.240+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/3993018226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c54f30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2e840b9810 tx=0x7f2e8408fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e826469e313703ef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.241+0000 7f2e737fe640 1 -- 172.21.15.49:0/3993018226 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840bf070 con 0x563d45c50430 2024-03-18T00:51:16.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.241+0000 7f2e737fe640 1 -- 172.21.15.49:0/3993018226 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e84004c80 con 0x563d45c50430 2024-03-18T00:51:16.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.241+0000 7f2e737fe640 1 -- 172.21.15.49:0/3993018226 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840c0f90 con 0x563d45c50430 2024-03-18T00:51:16.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.242+0000 7f2e92397740 1 -- 172.21.15.49:0/3993018226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 msgr2=0x563d45c54f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.242+0000 7f2e92397740 1 --2- 172.21.15.49:0/3993018226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c54f30 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2e840b9810 tx=0x7f2e8408fbf0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.243+0000 7f2e92397740 1 -- 172.21.15.49:0/3993018226 shutdown_connections 2024-03-18T00:51:16.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.243+0000 7f2e92397740 1 --2- 172.21.15.49:0/3993018226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c57a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.243+0000 7f2e92397740 1 --2- 172.21.15.49:0/3993018226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c54f30 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.243+0000 7f2e92397740 1 --2- 172.21.15.49:0/3993018226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c4fa40 0x563d45c4fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.243+0000 7f2e92397740 1 -- 172.21.15.49:0/3993018226 >> 172.21.15.49:0/3993018226 conn(0x563d45c4afb0 msgr2=0x563d45c4b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.243+0000 7f2e92397740 1 -- 172.21.15.49:0/3993018226 shutdown_connections 2024-03-18T00:51:16.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.244+0000 7f2e92397740 1 -- 172.21.15.49:0/3993018226 wait complete. 2024-03-18T00:51:16.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.244+0000 7f2e92397740 1 -- start start 2024-03-18T00:51:16.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.244+0000 7f2e92397740 1 Processor -- start 2024-03-18T00:51:16.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.245+0000 7f2e92397740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 0x563d45c59b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.245+0000 7f2e92397740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c50430 0x563d45c5a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.245+0000 7f2e92397740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c610b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.245+0000 7f2e92397740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563d45bb4490 con 0x563d45c4fa40 2024-03-18T00:51:16.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.245+0000 7f2e92397740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563d45b70230 con 0x563d45c50430 2024-03-18T00:51:16.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.245+0000 7f2e92397740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563d45c58310 con 0x563d45c555a0 2024-03-18T00:51:16.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.245+0000 7f2e92397740 10 client.0 _ll_register_callbacks cb 0x563d45bac010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:16.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.246+0000 7f2e8d54f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 0x563d45c59b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.246+0000 7f2e8dd50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.246+0000 7f2e8dd50640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c610b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60350/0 (socket says 172.21.15.49:60350) 2024-03-18T00:51:16.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.246+0000 7f2e8dd50640 1 -- 172.21.15.49:0/1928725394 learned_addr learned my addr 172.21.15.49:0/1928725394 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.246+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c50430 0x563d45c5a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.246+0000 7f2e8d54f640 1 -- 172.21.15.49:0/1928725394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 msgr2=0x563d45c610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.246+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c610b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.246+0000 7f2e8d54f640 1 -- 172.21.15.49:0/1928725394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c50430 msgr2=0x563d45c5a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.247+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c50430 0x563d45c5a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.247+0000 7f2e8d54f640 1 -- 172.21.15.49:0/1928725394 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d45b7f100 con 0x563d45c4fa40 2024-03-18T00:51:16.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.247+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 0x563d45c59b30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2e8402cee0 tx=0x7f2e840bca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2b838bfee4e729b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.247+0000 7f2e70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.247+0000 7f2e70ff9640 1 -- 172.21.15.49:0/1928725394 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e840bf070 con 0x563d45c4fa40 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.247+0000 7f2e70ff9640 1 -- 172.21.15.49:0/1928725394 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2e840038a0 con 0x563d45c4fa40 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.247+0000 7f2e92397740 1 -- 172.21.15.49:0/1928725394 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e84004c80 con 0x563d45c4fa40 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e70ff9640 1 -- 172.21.15.49:0/1928725394 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e84005690 con 0x563d45c4fa40 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e92397740 10 client.15750 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e92397740 1 -- 172.21.15.49:0/1928725394 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563d458c83f0 con 0x563d45c4fa40 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e70ff9640 1 -- 172.21.15.49:0/1928725394 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e84004c80 con 0x563d45c4fa40 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e70ff9640 1 -- 172.21.15.49:0/1928725394 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f2e840bc290 con 0x563d45c4fa40 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e92397740 10 client.15750 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:16.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e92397740 10 client.15750 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:51:16.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e92397740 20 client.15750 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:16.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e92397740 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x563d45c65790 0x563d45c67bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e92397740 4 client.15750 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:16.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.248+0000 7f2e92397740 1 -- 172.21.15.49:0/1928725394 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d45bc19d0 con 0x563d45c65790 2024-03-18T00:51:16.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.253+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x563d45c65790 0x563d45c67bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.255+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x563d45c65790 0x563d45c67bc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=45192b08da4d176e server_cookie=626368064ef8d0e6 in_seq=0 out_seq=0 2024-03-18T00:51:16.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.255+0000 7f2e70ff9640 10 client.15750 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:51:16.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.256+0000 7f2e70ff9640 1 -- 172.21.15.49:0/1928725394 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563d45bc19d0 con 0x563d45c65790 2024-03-18T00:51:16.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.256+0000 7f2e70ff9640 10 client.15750 handle_command_reply: tid=0 2024-03-18T00:51:16.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.256+0000 7f2e92397740 1 client.15750 shutdown 2024-03-18T00:51:16.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.257+0000 7f2e92397740 1 -- 172.21.15.49:0/1928725394 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 msgr2=0x563d45c59b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.257+0000 7f2e92397740 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 0x563d45c59b30 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f2e8402cee0 tx=0x7f2e840bca30 comp rx=0 tx=0).stop 2024-03-18T00:51:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.257+0000 7f2e92397740 1 -- 172.21.15.49:0/1928725394 shutdown_connections 2024-03-18T00:51:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.257+0000 7f2e92397740 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x563d45c65790 0x563d45c67bc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.257+0000 7f2e70ff9640 0 client.15750 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:51:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.257+0000 7f2e92397740 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c50430 0x563d45c5a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.257+0000 7f2e92397740 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.257+0000 7f2e92397740 1 --2- 172.21.15.49:0/1928725394 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 0x563d45c59b30 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.257+0000 7f2e92397740 1 -- 172.21.15.49:0/1928725394 >> 172.21.15.49:0/1928725394 conn(0x563d45c4afb0 msgr2=0x563d45c4a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.258+0000 7f2e92397740 1 -- 172.21.15.49:0/1928725394 shutdown_connections 2024-03-18T00:51:16.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.258+0000 7f2e92397740 1 -- 172.21.15.49:0/1928725394 wait complete. 2024-03-18T00:51:16.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.258+0000 7f2e92397740 20 client.15750 trim_cache size 0 max 16384 2024-03-18T00:51:16.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.286+0000 7f2e92397740 1 Processor -- start 2024-03-18T00:51:16.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.288+0000 7f2e92397740 1 -- start start 2024-03-18T00:51:16.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.288+0000 7f2e92397740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 0x563d45c59fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.288+0000 7f2e92397740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c50430 0x563d45c61040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.288+0000 7f2e92397740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c555a0 0x563d45c49fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.288+0000 7f2e92397740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2e881aadb0 con 0x563d45c4fa40 2024-03-18T00:51:16.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.288+0000 7f2e92397740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2e881ac920 con 0x563d45c555a0 2024-03-18T00:51:16.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.288+0000 7f2e92397740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2e881ab480 con 0x563d45c50430 2024-03-18T00:51:16.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.289+0000 7f2e8cd4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c50430 0x563d45c61040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.289+0000 7f2e8cd4e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c50430 0x563d45c61040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60360/0 (socket says 172.21.15.49:60360) 2024-03-18T00:51:16.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.289+0000 7f2e8cd4e640 1 -- 172.21.15.49:0/2631699291 learned_addr learned my addr 172.21.15.49:0/2631699291 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.291+0000 7f2e8cd4e640 1 -- 172.21.15.49:0/2631699291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c555a0 msgr2=0x563d45c49fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:16.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.291+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/2631699291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c555a0 0x563d45c49fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.291+0000 7f2e8cd4e640 1 -- 172.21.15.49:0/2631699291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 msgr2=0x563d45c59fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.291+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/2631699291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 0x563d45c59fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.291+0000 7f2e8cd4e640 1 -- 172.21.15.49:0/2631699291 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d459927d0 con 0x563d45c50430 2024-03-18T00:51:16.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.299+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/2631699291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c50430 0x563d45c61040 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006fd0 tx=0x7f2e78016430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21b0c5e6325e8c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.317+0000 7f2e72ffd640 1 -- 172.21.15.49:0/2631699291 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78016b30 con 0x563d45c50430 2024-03-18T00:51:16.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.317+0000 7f2e72ffd640 1 -- 172.21.15.49:0/2631699291 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e78016cd0 con 0x563d45c50430 2024-03-18T00:51:16.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.317+0000 7f2e72ffd640 1 -- 172.21.15.49:0/2631699291 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78022620 con 0x563d45c50430 2024-03-18T00:51:16.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.322+0000 7f2e92397740 1 -- 172.21.15.49:0/2631699291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c50430 msgr2=0x563d45c61040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.322+0000 7f2e92397740 1 --2- 172.21.15.49:0/2631699291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c50430 0x563d45c61040 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006fd0 tx=0x7f2e78016430 comp rx=0 tx=0).stop 2024-03-18T00:51:16.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.327+0000 7f2e92397740 1 -- 172.21.15.49:0/2631699291 shutdown_connections 2024-03-18T00:51:16.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.327+0000 7f2e92397740 1 --2- 172.21.15.49:0/2631699291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c555a0 0x563d45c49fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.327+0000 7f2e92397740 1 --2- 172.21.15.49:0/2631699291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c50430 0x563d45c61040 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.327+0000 7f2e92397740 1 --2- 172.21.15.49:0/2631699291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c4fa40 0x563d45c59fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.327+0000 7f2e92397740 1 -- 172.21.15.49:0/2631699291 >> 172.21.15.49:0/2631699291 conn(0x563d45c4afb0 msgr2=0x563d45c498b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.331+0000 7f2e92397740 1 -- 172.21.15.49:0/2631699291 shutdown_connections 2024-03-18T00:51:16.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.338+0000 7f2e92397740 1 -- 172.21.15.49:0/2631699291 wait complete. 2024-03-18T00:51:16.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.338+0000 7f2e92397740 1 -- start start 2024-03-18T00:51:16.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.338+0000 7f2e92397740 1 Processor -- start 2024-03-18T00:51:16.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.344+0000 7f2e92397740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c4fa40 0x563d45c5e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.344+0000 7f2e92397740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c5e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.344+0000 7f2e92397740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c5f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.344+0000 7f2e92397740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2e881ac220 con 0x563d45c50430 2024-03-18T00:51:16.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.344+0000 7f2e92397740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2e881abb50 con 0x563d45c4fa40 2024-03-18T00:51:16.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.344+0000 7f2e92397740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2e881a8d90 con 0x563d45c555a0 2024-03-18T00:51:16.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.345+0000 7f2e92397740 10 client.0 _ll_register_callbacks cb 0x7f2e840bc980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.346+0000 7f2e8cd4e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c5e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.346+0000 7f2e8cd4e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c5e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60064/0 (socket says 172.21.15.49:60064) 2024-03-18T00:51:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.346+0000 7f2e8cd4e640 1 -- 172.21.15.49:0/825313152 learned_addr learned my addr 172.21.15.49:0/825313152 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.347+0000 7f2e8cd4e640 1 -- 172.21.15.49:0/825313152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 msgr2=0x563d45c5f9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.347+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c5f9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.347+0000 7f2e8cd4e640 1 -- 172.21.15.49:0/825313152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c4fa40 msgr2=0x563d45c5e460 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.347+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c4fa40 0x563d45c5e460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.347+0000 7f2e8cd4e640 1 -- 172.21.15.49:0/825313152 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d45c5ffa0 con 0x563d45c50430 2024-03-18T00:51:16.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.347+0000 7f2e8cd4e640 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c5e9a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2e78008010 tx=0x7f2e7800d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bcba0e25c8adb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.350+0000 7f2e70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:16.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.350+0000 7f2e70ff9640 1 -- 172.21.15.49:0/825313152 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78014470 con 0x563d45c50430 2024-03-18T00:51:16.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.350+0000 7f2e92397740 1 -- 172.21.15.49:0/825313152 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2e78016cd0 con 0x563d45c50430 2024-03-18T00:51:16.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.351+0000 7f2e70ff9640 1 -- 172.21.15.49:0/825313152 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2e78014610 con 0x563d45c50430 2024-03-18T00:51:16.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.354+0000 7f2e70ff9640 1 -- 172.21.15.49:0/825313152 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2e78014810 con 0x563d45c50430 2024-03-18T00:51:16.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.355+0000 7f2e92397740 10 client.15753 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:16.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.355+0000 7f2e92397740 1 -- 172.21.15.49:0/825313152 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563d45c612d0 con 0x563d45c50430 2024-03-18T00:51:16.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.364+0000 7f2e70ff9640 1 -- 172.21.15.49:0/825313152 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e78016cd0 con 0x563d45c50430 2024-03-18T00:51:16.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.367+0000 7f2e70ff9640 1 -- 172.21.15.49:0/825313152 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f2e78024430 con 0x563d45c50430 2024-03-18T00:51:16.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.367+0000 7f2e92397740 10 client.15753 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:16.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.367+0000 7f2e92397740 10 client.15753 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:51:16.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.367+0000 7f2e92397740 20 client.15753 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:16.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.367+0000 7f2e92397740 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x563d45c65790 0x563d45cce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.367+0000 7f2e92397740 4 client.15753 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:51:16.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.367+0000 7f2e92397740 1 -- 172.21.15.49:0/825313152 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563d45bbbd20 con 0x563d45c65790 2024-03-18T00:51:16.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.371+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x563d45c65790 0x563d45cce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.395+0000 7f2e8d54f640 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x563d45c65790 0x563d45cce030 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e0e1c71c0a2a9ae7 server_cookie=c89422304853d01b in_seq=0 out_seq=0 2024-03-18T00:51:16.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.395+0000 7f2e70ff9640 10 client.15753 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:51:16.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.397+0000 7f2e70ff9640 1 -- 172.21.15.49:0/825313152 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563d45bbbd20 con 0x563d45c65790 2024-03-18T00:51:16.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.397+0000 7f2e70ff9640 10 client.15753 handle_command_reply: tid=0 2024-03-18T00:51:16.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.398+0000 7f2e92397740 1 client.15753 shutdown 2024-03-18T00:51:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.401+0000 7f2e92397740 1 -- 172.21.15.49:0/825313152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 msgr2=0x563d45c5e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.401+0000 7f2e92397740 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c5e9a0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2e78008010 tx=0x7f2e7800d3c0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.404+0000 7f2e92397740 1 -- 172.21.15.49:0/825313152 shutdown_connections 2024-03-18T00:51:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.404+0000 7f2e92397740 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x563d45c65790 0x563d45cce030 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.404+0000 7f2e92397740 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563d45c555a0 0x563d45c5f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.404+0000 7f2e92397740 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563d45c50430 0x563d45c5e9a0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.404+0000 7f2e92397740 1 --2- 172.21.15.49:0/825313152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563d45c4fa40 0x563d45c5e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.404+0000 7f2e92397740 1 -- 172.21.15.49:0/825313152 >> 172.21.15.49:0/825313152 conn(0x563d45c4afb0 msgr2=0x563d45c4a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.404+0000 7f2e70ff9640 0 client.15753 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:51:16.407 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:51:16.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.405+0000 7f2e92397740 1 -- 172.21.15.49:0/825313152 shutdown_connections 2024-03-18T00:51:16.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.405+0000 7f2e92397740 1 -- 172.21.15.49:0/825313152 wait complete. 2024-03-18T00:51:16.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.405+0000 7f2e92397740 20 client.15753 trim_cache size 0 max 16384 2024-03-18T00:51:16.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.406+0000 7f2e8e551640 1 -- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2e740a2a00 msgr2=0x7f2e740a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.406+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2e740a2a00 0x7f2e740a4ec0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009270 tx=0x7f2e800098a0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.406+0000 7f2e8e551640 1 -- 172.21.15.49:0/3028323691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e881005c0 msgr2=0x7f2e880694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.406+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e881005c0 0x7f2e880694b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2e8402efd0 tx=0x7f2e84005e40 comp rx=0 tx=0).stop 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.407+0000 7f2e8e551640 1 -- 172.21.15.49:0/3028323691 shutdown_connections 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.407+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2e740a2a00 0x7f2e740a4ec0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.407+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2e88102170 0x7f2e881a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.407+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2e881017b0 0x7f2e8819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.407+0000 7f2e8e551640 1 --2- 172.21.15.49:0/3028323691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2e881005c0 0x7f2e880694b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.407+0000 7f2e8e551640 1 -- 172.21.15.49:0/3028323691 >> 172.21.15.49:0/3028323691 conn(0x7f2e880fbb50 msgr2=0x7f2e88104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.408+0000 7f2e8e551640 1 -- 172.21.15.49:0/3028323691 shutdown_connections 2024-03-18T00:51:16.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.408+0000 7f2e8e551640 1 -- 172.21.15.49:0/3028323691 wait complete. 2024-03-18T00:51:16.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:16 smithi107 ceph-mon[36270]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:51:16.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:16 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/347626573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:51:16.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:51:16.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:16 smithi049 ceph-mon[32130]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:51:16.442 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:16 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/347626573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:51:16.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:16 smithi106 ceph-mon[35899]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:51:16.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:16 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/347626573' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:51:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f79695ab640 1 -- 172.21.15.49:0/3374418942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79640ff8b0 msgr2=0x7f79640ffd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f79695ab640 1 --2- 172.21.15.49:0/3374418942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79640ff8b0 0x7f79640ffd30 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7958009900 tx=0x7f795802bc80 comp rx=0 tx=0).stop 2024-03-18T00:51:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f79695ab640 1 -- 172.21.15.49:0/3374418942 shutdown_connections 2024-03-18T00:51:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f79695ab640 1 --2- 172.21.15.49:0/3374418942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7964100270 0x7f7964100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f79695ab640 1 --2- 172.21.15.49:0/3374418942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79640ff8b0 0x7f79640ffd30 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f79695ab640 1 --2- 172.21.15.49:0/3374418942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7964101dc0 0x7f79641021e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f79695ab640 1 -- 172.21.15.49:0/3374418942 >> 172.21.15.49:0/3374418942 conn(0x7f79640fb410 msgr2=0x7f79640fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f269b468640 1 -- 172.21.15.49:0/2858178163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2694100720 msgr2=0x7f2694100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f269b468640 1 --2- 172.21.15.49:0/2858178163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2694100720 0x7f2694100b40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2690009980 tx=0x7f269002de90 comp rx=0 tx=0).stop 2024-03-18T00:51:16.786 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f79695ab640 1 -- 172.21.15.49:0/3374418942 shutdown_connections 2024-03-18T00:51:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.785+0000 7f79695ab640 1 -- 172.21.15.49:0/3374418942 wait complete. 2024-03-18T00:51:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.786+0000 7f79695ab640 1 Processor -- start 2024-03-18T00:51:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79695ab640 1 -- start start 2024-03-18T00:51:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79695ab640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79640ff8b0 0x7f7964199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79695ab640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7964100270 0x7f796419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.787 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79695ab640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7964101dc0 0x7f79641a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79695ab640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f796410f770 con 0x7f7964101dc0 2024-03-18T00:51:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79695ab640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f796410f5f0 con 0x7f7964100270 2024-03-18T00:51:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79695ab640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f796410f8f0 con 0x7f79640ff8b0 2024-03-18T00:51:16.788 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79637fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7964101dc0 0x7f79641a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79637fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7964101dc0 0x7f79641a12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60082/0 (socket says 172.21.15.49:60082) 2024-03-18T00:51:16.789 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.787+0000 7f79637fe640 1 -- 172.21.15.49:0/1816965648 learned_addr learned my addr 172.21.15.49:0/1816965648 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f269b468640 1 -- 172.21.15.49:0/2858178163 shutdown_connections 2024-03-18T00:51:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f269b468640 1 --2- 172.21.15.49:0/2858178163 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2694101910 0x7f2694101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f269b468640 1 --2- 172.21.15.49:0/2858178163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f26941022d0 0x7f2694108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f269b468640 1 --2- 172.21.15.49:0/2858178163 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2694100720 0x7f2694100b40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f269b468640 1 -- 172.21.15.49:0/2858178163 >> 172.21.15.49:0/2858178163 conn(0x7f26940fbc90 msgr2=0x7f26940fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f7962ffd640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79640ff8b0 0x7f7964199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.790 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f79637fe640 1 -- 172.21.15.49:0/1816965648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79640ff8b0 msgr2=0x7f7964199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f79637fe640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79640ff8b0 0x7f7964199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f79637fe640 1 -- 172.21.15.49:0/1816965648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7964100270 msgr2=0x7f796419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f79637fe640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7964100270 0x7f796419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f79637fe640 1 -- 172.21.15.49:0/1816965648 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79641a1960 con 0x7f7964101dc0 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.788+0000 7f79637fe640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7964101dc0 0x7f79641a12c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f795c00eb10 tx=0x7f795c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.789+0000 7f794bfff640 1 -- 172.21.15.49:0/1816965648 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c004b20 con 0x7f7964101dc0 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.789+0000 7f79695ab640 1 -- 172.21.15.49:0/1816965648 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79641a1c50 con 0x7f7964101dc0 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.789+0000 7f794bfff640 1 -- 172.21.15.49:0/1816965648 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f795c004cc0 con 0x7f7964101dc0 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.789+0000 7f794bfff640 1 -- 172.21.15.49:0/1816965648 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795c010690 con 0x7f7964101dc0 2024-03-18T00:51:16.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.789+0000 7f79695ab640 1 -- 172.21.15.49:0/1816965648 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79641a2170 con 0x7f7964101dc0 2024-03-18T00:51:16.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.790+0000 7f269b468640 1 -- 172.21.15.49:0/2858178163 shutdown_connections 2024-03-18T00:51:16.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.790+0000 7f269b468640 1 -- 172.21.15.49:0/2858178163 wait complete. 2024-03-18T00:51:16.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.791+0000 7f269b468640 1 Processor -- start 2024-03-18T00:51:16.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.791+0000 7f269b468640 1 -- start start 2024-03-18T00:51:16.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.791+0000 7f269b468640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2694100720 0x7f26940701f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.791+0000 7f269b468640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2694101910 0x7f2694070730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.792+0000 7f269b468640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26941022d0 0x7f26941ae630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.792+0000 7f269b468640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f269410b320 con 0x7f26941022d0 2024-03-18T00:51:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.792+0000 7f269b468640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f269410b1a0 con 0x7f2694100720 2024-03-18T00:51:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.792+0000 7f269b468640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f269410b4a0 con 0x7f2694101910 2024-03-18T00:51:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.792+0000 7f2699a05640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26941022d0 0x7f26941ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.792+0000 7f2699a05640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26941022d0 0x7f26941ae630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60098/0 (socket says 172.21.15.49:60098) 2024-03-18T00:51:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.792+0000 7f2699a05640 1 -- 172.21.15.49:0/1582041061 learned_addr learned my addr 172.21.15.49:0/1582041061 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.792+0000 7f2699a05640 1 -- 172.21.15.49:0/1582041061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2694101910 msgr2=0x7f2694070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.793+0000 7f2699a05640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2694101910 0x7f2694070730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.793+0000 7f2699a05640 1 -- 172.21.15.49:0/1582041061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2694100720 msgr2=0x7f26940701f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:51:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.793+0000 7f2699a05640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2694100720 0x7f26940701f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.793+0000 7f2699a05640 1 -- 172.21.15.49:0/1582041061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26941a5a30 con 0x7f26941022d0 2024-03-18T00:51:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.793+0000 7f2699a05640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26941022d0 0x7f26941ae630 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f268c00eaa0 tx=0x7f268c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.793+0000 7f268a7fc640 1 -- 172.21.15.49:0/1582041061 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c00cdb0 con 0x7f26941022d0 2024-03-18T00:51:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.794+0000 7f268a7fc640 1 -- 172.21.15.49:0/1582041061 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f268c005020 con 0x7f26941022d0 2024-03-18T00:51:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.794+0000 7f268a7fc640 1 -- 172.21.15.49:0/1582041061 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c0116d0 con 0x7f26941022d0 2024-03-18T00:51:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.794+0000 7f269b468640 1 -- 172.21.15.49:0/1582041061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26941aeb70 con 0x7f26941022d0 2024-03-18T00:51:16.795 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.794+0000 7f796c18b740 1 Processor -- start 2024-03-18T00:51:16.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.794+0000 7f269b468640 1 -- 172.21.15.49:0/1582041061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26941aef20 con 0x7f26941022d0 2024-03-18T00:51:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.795+0000 7f794bfff640 1 -- 172.21.15.49:0/1816965648 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f795c005000 con 0x7f7964101dc0 2024-03-18T00:51:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.795+0000 7f796c18b740 1 -- start start 2024-03-18T00:51:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.795+0000 7f796c18b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6ef760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.795+0000 7f796c18b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.795+0000 7f796c18b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 0x5639ca6f7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.795+0000 7f796c18b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5639ca654a40 con 0x5639ca6efd30 2024-03-18T00:51:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f796c18b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5639ca5fd920 con 0x5639ca6ef340 2024-03-18T00:51:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f796c18b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5639ca650ae0 con 0x5639ca6f4ea0 2024-03-18T00:51:16.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f7962ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f7962ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60110/0 (socket says 172.21.15.49:60110) 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f7962ffd640 1 -- 172.21.15.49:0/1884786686 learned_addr learned my addr 172.21.15.49:0/1884786686 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f794bfff640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f794c0a2920 0x7f794c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f79627fc640 1 --2- 172.21.15.49:0/1884786686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 0x5639ca6f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f7962ffd640 1 -- 172.21.15.49:0/1884786686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 msgr2=0x5639ca6f7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f7962ffd640 1 --2- 172.21.15.49:0/1884786686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 0x5639ca6f7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f7962ffd640 1 -- 172.21.15.49:0/1884786686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 msgr2=0x5639ca6ef760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f7962ffd640 1 --2- 172.21.15.49:0/1884786686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6ef760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.796+0000 7f7962ffd640 1 -- 172.21.15.49:0/1884786686 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639ca615460 con 0x5639ca6efd30 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.797+0000 7f79627fc640 1 --2- 172.21.15.49:0/1884786686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 0x5639ca6f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.797+0000 7f7962ffd640 1 --2- 172.21.15.49:0/1884786686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f4830 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f795400b7f0 tx=0x7f795400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4151c1c0b7c8ccbe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.797+0000 7f794bfff640 1 -- 172.21.15.49:0/1816965648 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f795c0cf1f0 con 0x7f7964101dc0 2024-03-18T00:51:16.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.797+0000 7f79637fe640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f794c0a2920 0x7f794c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.798+0000 7f79637fe640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f794c0a2920 0x7f794c0a4de0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f795c0049d0 tx=0x7f795c0058a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.798+0000 7f79497fa640 1 -- 172.21.15.49:0/1884786686 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795400d950 con 0x5639ca6efd30 2024-03-18T00:51:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.798+0000 7f79497fa640 1 -- 172.21.15.49:0/1884786686 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7954018070 con 0x5639ca6efd30 2024-03-18T00:51:16.799 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.798+0000 7f79497fa640 1 -- 172.21.15.49:0/1884786686 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79540123e0 con 0x5639ca6efd30 2024-03-18T00:51:16.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.798+0000 7f269e050740 1 Processor -- start 2024-03-18T00:51:16.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.799+0000 7f796c18b740 1 -- 172.21.15.49:0/1884786686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 msgr2=0x5639ca6f4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.799+0000 7f796c18b740 1 --2- 172.21.15.49:0/1884786686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f4830 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f795400b7f0 tx=0x7f795400bcb0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.799+0000 7f796c18b740 1 -- 172.21.15.49:0/1884786686 shutdown_connections 2024-03-18T00:51:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.799+0000 7f796c18b740 1 --2- 172.21.15.49:0/1884786686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 0x5639ca6f7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.799+0000 7f796c18b740 1 --2- 172.21.15.49:0/1884786686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f4830 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.799+0000 7f796c18b740 1 --2- 172.21.15.49:0/1884786686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6ef760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.799+0000 7f796c18b740 1 -- 172.21.15.49:0/1884786686 >> 172.21.15.49:0/1884786686 conn(0x5639ca6ea8b0 msgr2=0x5639ca6eacb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.799+0000 7f796c18b740 1 -- 172.21.15.49:0/1884786686 shutdown_connections 2024-03-18T00:51:16.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.799+0000 7f796c18b740 1 -- 172.21.15.49:0/1884786686 wait complete. 2024-03-18T00:51:16.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f268a7fc640 1 -- 172.21.15.49:0/1582041061 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f268c0051d0 con 0x7f26941022d0 2024-03-18T00:51:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f796c18b740 1 -- start start 2024-03-18T00:51:16.802 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f796c18b740 1 Processor -- start 2024-03-18T00:51:16.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f269e050740 1 -- start start 2024-03-18T00:51:16.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f269e050740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fd850 0x5615d09fdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f269e050740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fe240 0x5615d0a02d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f269e050740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d0a033b0 0x5615d0a05840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f269e050740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5615d0962490 con 0x5615d09fd850 2024-03-18T00:51:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f269e050740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5615d090ac90 con 0x5615d09fe240 2024-03-18T00:51:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.800+0000 7f269e050740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5615d095e530 con 0x5615d0a033b0 2024-03-18T00:51:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f796c18b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6ef340 0x5639ca6f96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.803 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f796c18b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f796c18b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6f4ea0 0x5639ca700c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f796c18b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5639ca6f7e50 con 0x5639ca6efd30 2024-03-18T00:51:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f796c18b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5639ca610d70 con 0x5639ca6f4ea0 2024-03-18T00:51:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f796c18b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5639ca6f7fd0 con 0x5639ca6ef340 2024-03-18T00:51:16.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f796c18b740 10 client.0 _ll_register_callbacks cb 0x5639ca368a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:16.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f2698a03640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fd850 0x5615d09fdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f2698a03640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fd850 0x5615d09fdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60126/0 (socket says 172.21.15.49:60126) 2024-03-18T00:51:16.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f2698a03640 1 -- 172.21.15.49:0/1999491133 learned_addr learned my addr 172.21.15.49:0/1999491133 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f7962ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f7962ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60134/0 (socket says 172.21.15.49:60134) 2024-03-18T00:51:16.805 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f7962ffd640 1 -- 172.21.15.49:0/3361809235 learned_addr learned my addr 172.21.15.49:0/3361809235 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f2698a03640 1 -- 172.21.15.49:0/1999491133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d0a033b0 msgr2=0x5615d0a05840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f2698a03640 1 --2- 172.21.15.49:0/1999491133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d0a033b0 0x5615d0a05840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f2698a03640 1 -- 172.21.15.49:0/1999491133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fe240 msgr2=0x5615d0a02d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.801+0000 7f2698a03640 1 --2- 172.21.15.49:0/1999491133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fe240 0x5615d0a02d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f7962ffd640 1 -- 172.21.15.49:0/3361809235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6ef340 msgr2=0x5639ca6f96d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f7962ffd640 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6ef340 0x5639ca6f96d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f7962ffd640 1 -- 172.21.15.49:0/3361809235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6f4ea0 msgr2=0x5639ca700c50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f7962ffd640 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6f4ea0 0x5639ca700c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f7962ffd640 1 -- 172.21.15.49:0/3361809235 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639ca369210 con 0x5639ca6efd30 2024-03-18T00:51:16.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f268a7fc640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26800a2880 0x7f26800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f2698a03640 1 -- 172.21.15.49:0/1999491133 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615d09c5500 con 0x5615d09fd850 2024-03-18T00:51:16.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f268a7fc640 1 -- 172.21.15.49:0/1582041061 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f268c0ced00 con 0x7f26941022d0 2024-03-18T00:51:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f7962ffd640 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f9c10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7954006e10 tx=0x7f79540137f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb62d08bbb894d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f2698a03640 1 --2- 172.21.15.49:0/1999491133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fd850 0x5615d09fdc70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2684009940 tx=0x7f2684009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b8b917a41e36da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f7946ff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.802+0000 7f7946ff5640 1 -- 172.21.15.49:0/3361809235 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954023070 con 0x5639ca6efd30 2024-03-18T00:51:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f7946ff5640 1 -- 172.21.15.49:0/3361809235 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f795400dd60 con 0x5639ca6efd30 2024-03-18T00:51:16.809 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f796c18b740 1 -- 172.21.15.49:0/3361809235 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7954018070 con 0x5639ca6efd30 2024-03-18T00:51:16.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f2698a03640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26800a2880 0x7f26800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f267f7fe640 1 -- 172.21.15.49:0/1999491133 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684015020 con 0x5615d09fd850 2024-03-18T00:51:16.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f267f7fe640 1 -- 172.21.15.49:0/1999491133 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f268401b9b0 con 0x5615d09fd850 2024-03-18T00:51:16.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f267f7fe640 1 -- 172.21.15.49:0/1999491133 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684011cf0 con 0x5615d09fd850 2024-03-18T00:51:16.810 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f7946ff5640 1 -- 172.21.15.49:0/3361809235 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f795402f050 con 0x5639ca6efd30 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f796c18b740 10 client.15774 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f796c18b740 1 -- 172.21.15.49:0/3361809235 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5639ca435620 con 0x5639ca6efd30 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.803+0000 7f7946ff5640 1 -- 172.21.15.49:0/3361809235 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7954018070 con 0x5639ca6efd30 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.804+0000 7f7946ff5640 1 -- 172.21.15.49:0/3361809235 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f795401b380 con 0x5639ca6efd30 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.804+0000 7f796c18b740 10 client.15774 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.804+0000 7f796c18b740 10 client.15774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.804+0000 7f796c18b740 20 client.15774 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.804+0000 7f796c18b740 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5639ca705170 0x5639ca707590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.804+0000 7f796c18b740 4 client.15774 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:16.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.804+0000 7f796c18b740 1 -- 172.21.15.49:0/3361809235 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639ca661f80 con 0x5639ca705170 2024-03-18T00:51:16.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 -- 172.21.15.49:0/1999491133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fd850 msgr2=0x5615d09fdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f2698a03640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26800a2880 0x7f26800a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2684016000 tx=0x7f26840198d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 --2- 172.21.15.49:0/1999491133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fd850 0x5615d09fdc70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2684009940 tx=0x7f2684009e00 comp rx=0 tx=0).stop 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 -- 172.21.15.49:0/1999491133 shutdown_connections 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 --2- 172.21.15.49:0/1999491133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fe240 0x5615d0a02d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 --2- 172.21.15.49:0/1999491133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d0a033b0 0x5615d0a05840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 --2- 172.21.15.49:0/1999491133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fd850 0x5615d09fdc70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 -- 172.21.15.49:0/1999491133 >> 172.21.15.49:0/1999491133 conn(0x5615d09f8dc0 msgr2=0x5615d09f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 -- 172.21.15.49:0/1999491133 shutdown_connections 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 -- 172.21.15.49:0/1999491133 wait complete. 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.805+0000 7f269e050740 1 -- start start 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.806+0000 7f269e050740 1 Processor -- start 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.808+0000 7f269e050740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 0x5615d0a07d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.808+0000 7f269e050740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a082b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.814 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.809+0000 7f79627fc640 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5639ca705170 0x5639ca707590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.810+0000 7f269e050740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 0x5615d0a0f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.810+0000 7f269e050740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5615d0a06360 con 0x5615d09fe240 2024-03-18T00:51:16.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.810+0000 7f269e050740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5615d091e230 con 0x5615d0a033b0 2024-03-18T00:51:16.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.810+0000 7f269e050740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5615d0a064e0 con 0x5615d09fd850 2024-03-18T00:51:16.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.810+0000 7f269e050740 10 client.0 _ll_register_callbacks cb 0x5615d095a010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:16.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2699204640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 0x5615d0a07d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2699204640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 0x5615d0a07d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60430/0 (socket says 172.21.15.49:60430) 2024-03-18T00:51:16.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2699204640 1 -- 172.21.15.49:0/3242187208 learned_addr learned my addr 172.21.15.49:0/3242187208 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2698a03640 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a082b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.815 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f7946ff5640 10 client.15774 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:16.815 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f79627fc640 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5639ca705170 0x5639ca707590 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ee74d430b66a3d6 server_cookie=3640cad2d68554f1 in_seq=0 out_seq=0 2024-03-18T00:51:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2699a05640 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 0x5615d0a0f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2698a03640 1 -- 172.21.15.49:0/3242187208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 msgr2=0x5615d0a07d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2698a03640 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 0x5615d0a07d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2698a03640 1 -- 172.21.15.49:0/3242187208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 msgr2=0x5615d0a0f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2698a03640 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 0x5615d0a0f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.811+0000 7f2698a03640 1 -- 172.21.15.49:0/3242187208 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615d092d100 con 0x5615d09fe240 2024-03-18T00:51:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.812+0000 7f2698a03640 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a082b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f268400b4b0 tx=0x7f2684014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26be87beb2bbe1d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.813+0000 7f267cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.813+0000 7f267cff9640 1 -- 172.21.15.49:0/3242187208 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684015040 con 0x5615d09fe240 2024-03-18T00:51:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.813+0000 7f267cff9640 1 -- 172.21.15.49:0/3242187208 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f268400bb30 con 0x5615d09fe240 2024-03-18T00:51:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.813+0000 7f269e050740 1 -- 172.21.15.49:0/3242187208 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f268401b9b0 con 0x5615d09fe240 2024-03-18T00:51:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.814+0000 7f267cff9640 1 -- 172.21.15.49:0/3242187208 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2684036070 con 0x5615d09fe240 2024-03-18T00:51:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.814+0000 7f269e050740 10 client.15777 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.814+0000 7f269e050740 1 -- 172.21.15.49:0/3242187208 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5615d06763f0 con 0x5615d09fe240 2024-03-18T00:51:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f7946ff5640 1 -- 172.21.15.49:0/3361809235 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5639ca661f80 con 0x5639ca705170 2024-03-18T00:51:16.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f7946ff5640 10 client.15774 handle_command_reply: tid=0 2024-03-18T00:51:16.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f267cff9640 1 -- 172.21.15.49:0/3242187208 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f268401b9b0 con 0x5615d09fe240 2024-03-18T00:51:16.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f796c18b740 1 client.15774 shutdown 2024-03-18T00:51:16.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f267cff9640 1 -- 172.21.15.49:0/3242187208 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f268401b050 con 0x5615d09fe240 2024-03-18T00:51:16.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f269e050740 10 client.15777 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:16.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f269e050740 10 client.15777 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:51:16.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f269e050740 20 client.15777 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:16.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f269e050740 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5615d0a13970 0x5615d0a15dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f269e050740 4 client.15777 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:16.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.815+0000 7f269e050740 1 -- 172.21.15.49:0/3242187208 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615d096f9d0 con 0x5615d0a13970 2024-03-18T00:51:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 -- 172.21.15.49:0/3361809235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 msgr2=0x5639ca6f9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f9c10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7954006e10 tx=0x7f79540137f0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 -- 172.21.15.49:0/3361809235 shutdown_connections 2024-03-18T00:51:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5639ca705170 0x5639ca707590 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6f4ea0 0x5639ca700c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f9c10 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 --2- 172.21.15.49:0/3361809235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6ef340 0x5639ca6f96d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 -- 172.21.15.49:0/3361809235 >> 172.21.15.49:0/3361809235 conn(0x5639ca6ea8b0 msgr2=0x5639ca6e9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f7946ff5640 0 client.15774 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 -- 172.21.15.49:0/3361809235 shutdown_connections 2024-03-18T00:51:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 1 -- 172.21.15.49:0/3361809235 wait complete. 2024-03-18T00:51:16.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.816+0000 7f796c18b740 20 client.15774 trim_cache size 0 max 16384 2024-03-18T00:51:16.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.818+0000 7f2699204640 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5615d0a13970 0x5615d0a15dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.824+0000 7f2699204640 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5615d0a13970 0x5615d0a15dc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60930fff2bb3c804 server_cookie=6d8c06b8d1ec5517 in_seq=0 out_seq=0 2024-03-18T00:51:16.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.824+0000 7f267cff9640 10 client.15777 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:51:16.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.825+0000 7f267cff9640 1 -- 172.21.15.49:0/3242187208 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5615d096f9d0 con 0x5615d0a13970 2024-03-18T00:51:16.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.826+0000 7f267cff9640 10 client.15777 handle_command_reply: tid=0 2024-03-18T00:51:16.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.826+0000 7f269e050740 1 client.15777 shutdown 2024-03-18T00:51:16.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.826+0000 7f269e050740 1 -- 172.21.15.49:0/3242187208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 msgr2=0x5615d0a082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.826+0000 7f269e050740 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a082b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f268400b4b0 tx=0x7f2684014000 comp rx=0 tx=0).stop 2024-03-18T00:51:16.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.827+0000 7f269e050740 1 -- 172.21.15.49:0/3242187208 shutdown_connections 2024-03-18T00:51:16.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.827+0000 7f269e050740 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5615d0a13970 0x5615d0a15dc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.827+0000 7f269e050740 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 0x5615d0a0f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.827+0000 7f269e050740 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a082b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.827+0000 7f269e050740 1 --2- 172.21.15.49:0/3242187208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 0x5615d0a07d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.827+0000 7f269e050740 1 -- 172.21.15.49:0/3242187208 >> 172.21.15.49:0/3242187208 conn(0x5615d09f8dc0 msgr2=0x5615d09f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.827+0000 7f267cff9640 0 client.15777 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:51:16.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.828+0000 7f269e050740 1 -- 172.21.15.49:0/3242187208 shutdown_connections 2024-03-18T00:51:16.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.828+0000 7f269e050740 1 -- 172.21.15.49:0/3242187208 wait complete. 2024-03-18T00:51:16.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.828+0000 7f269e050740 20 client.15777 trim_cache size 0 max 16384 2024-03-18T00:51:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.854+0000 7f796c18b740 1 Processor -- start 2024-03-18T00:51:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.854+0000 7f796c18b740 1 -- start start 2024-03-18T00:51:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.854+0000 7f796c18b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6f1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.854+0000 7f796c18b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.855+0000 7f796c18b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 0x5639ca700c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.855+0000 7f796c18b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f79641ab0b0 con 0x5639ca6efd30 2024-03-18T00:51:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.855+0000 7f796c18b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f79641acc20 con 0x5639ca6ef340 2024-03-18T00:51:16.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.855+0000 7f796c18b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f79641ab780 con 0x5639ca6f4ea0 2024-03-18T00:51:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.855+0000 7f79627fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6f1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.855+0000 7f79627fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6f1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:43202/0 (socket says 172.21.15.49:43202) 2024-03-18T00:51:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.855+0000 7f79627fc640 1 -- 172.21.15.49:0/2368995114 learned_addr learned my addr 172.21.15.49:0/2368995114 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.856 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.855+0000 7f7962ffd640 1 --2- 172.21.15.49:0/2368995114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.856+0000 7f7962ffd640 1 -- 172.21.15.49:0/2368995114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 msgr2=0x5639ca700c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.856+0000 7f7962ffd640 1 --2- 172.21.15.49:0/2368995114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 0x5639ca700c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.856+0000 7f7962ffd640 1 -- 172.21.15.49:0/2368995114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 msgr2=0x5639ca6f1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.856+0000 7f7962ffd640 1 --2- 172.21.15.49:0/2368995114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6f1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.856+0000 7f7962ffd640 1 -- 172.21.15.49:0/2368995114 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639ca6f8130 con 0x5639ca6efd30 2024-03-18T00:51:16.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.856+0000 7f79627fc640 1 --2- 172.21.15.49:0/2368995114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6f1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:51:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.857+0000 7f7962ffd640 1 --2- 172.21.15.49:0/2368995114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f15d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7954004810 tx=0x7f795401bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab27976cbc0c780 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.857+0000 7f7948ff9640 1 -- 172.21.15.49:0/2368995114 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954023040 con 0x5639ca6efd30 2024-03-18T00:51:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.857+0000 7f7948ff9640 1 -- 172.21.15.49:0/2368995114 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7954018050 con 0x5639ca6efd30 2024-03-18T00:51:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.857+0000 7f7948ff9640 1 -- 172.21.15.49:0/2368995114 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7954013860 con 0x5639ca6efd30 2024-03-18T00:51:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.858+0000 7f796c18b740 1 -- 172.21.15.49:0/2368995114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 msgr2=0x5639ca6f15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.858+0000 7f796c18b740 1 --2- 172.21.15.49:0/2368995114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f15d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7954004810 tx=0x7f795401bc80 comp rx=0 tx=0).stop 2024-03-18T00:51:16.859 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.859+0000 7f796c18b740 1 -- 172.21.15.49:0/2368995114 shutdown_connections 2024-03-18T00:51:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.859+0000 7f796c18b740 1 --2- 172.21.15.49:0/2368995114 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6f4ea0 0x5639ca700c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.859+0000 7f796c18b740 1 --2- 172.21.15.49:0/2368995114 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6efd30 0x5639ca6f15d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.859+0000 7f796c18b740 1 --2- 172.21.15.49:0/2368995114 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6f1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.859+0000 7f796c18b740 1 -- 172.21.15.49:0/2368995114 >> 172.21.15.49:0/2368995114 conn(0x5639ca6ea8b0 msgr2=0x5639ca6eac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.859+0000 7f796c18b740 1 -- 172.21.15.49:0/2368995114 shutdown_connections 2024-03-18T00:51:16.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.860+0000 7f796c18b740 1 -- 172.21.15.49:0/2368995114 wait complete. 2024-03-18T00:51:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.860+0000 7f796c18b740 1 -- start start 2024-03-18T00:51:16.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f796c18b740 1 Processor -- start 2024-03-18T00:51:16.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f269e050740 1 Processor -- start 2024-03-18T00:51:16.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f269e050740 1 -- start start 2024-03-18T00:51:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f796c18b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6fe010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f796c18b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6efd30 0x5639ca6fe550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f796c18b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6f4ea0 0x5639ca6ff560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f796c18b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f79641ac520 con 0x5639ca6f4ea0 2024-03-18T00:51:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f796c18b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f79641abe50 con 0x5639ca6ef340 2024-03-18T00:51:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f796c18b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f79641a9090 con 0x5639ca6efd30 2024-03-18T00:51:16.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.861+0000 7f796c18b740 10 client.0 _ll_register_callbacks cb 0x7f795400b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:16.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.862+0000 7f269e050740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fd850 0x5615d09ff740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.862+0000 7f79627fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6f4ea0 0x5639ca6ff560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.862+0000 7f7962ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6efd30 0x5639ca6fe550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.862+0000 7f269e050740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fe240 0x5615d0a0f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.862+0000 7f79627fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6f4ea0 0x5639ca6ff560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60160/0 (socket says 172.21.15.49:60160) 2024-03-18T00:51:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.862+0000 7f79627fc640 1 -- 172.21.15.49:0/125304901 learned_addr learned my addr 172.21.15.49:0/125304901 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.862+0000 7f269e050740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d0a033b0 0x5615d09ffc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f269e050740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f26941b7ef0 con 0x5615d0a033b0 2024-03-18T00:51:16.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f269e050740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f26941b9a60 con 0x5615d09fd850 2024-03-18T00:51:16.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f269e050740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f26941b85c0 con 0x5615d09fe240 2024-03-18T00:51:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f79627fc640 1 -- 172.21.15.49:0/125304901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6efd30 msgr2=0x5639ca6fe550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f79627fc640 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6efd30 0x5639ca6fe550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f79627fc640 1 -- 172.21.15.49:0/125304901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 msgr2=0x5639ca6fe010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f79627fc640 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6fe010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f79627fc640 1 -- 172.21.15.49:0/125304901 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639ca6ffb50 con 0x5639ca6f4ea0 2024-03-18T00:51:16.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f2699a05640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d0a033b0 0x5615d09ffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f2699a05640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d0a033b0 0x5615d09ffc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60162/0 (socket says 172.21.15.49:60162) 2024-03-18T00:51:16.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f2699a05640 1 -- 172.21.15.49:0/3707963192 learned_addr learned my addr 172.21.15.49:0/3707963192 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f2699204640 1 --2- 172.21.15.49:0/3707963192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fd850 0x5615d09ff740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f2699a05640 1 -- 172.21.15.49:0/3707963192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fe240 msgr2=0x5615d0a0f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f2699a05640 1 --2- 172.21.15.49:0/3707963192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fe240 0x5615d0a0f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f2699a05640 1 -- 172.21.15.49:0/3707963192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fd850 msgr2=0x5615d09ff740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f79627fc640 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6f4ea0 0x5639ca6ff560 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7958009230 tx=0x7f7958004aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737936aae8050d1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f2699a05640 1 --2- 172.21.15.49:0/3707963192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fd850 0x5615d09ff740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f2699a05640 1 -- 172.21.15.49:0/3707963192 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615d07407d0 con 0x5615d0a033b0 2024-03-18T00:51:16.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.864+0000 7f2699204640 1 --2- 172.21.15.49:0/3707963192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fd850 0x5615d09ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:51:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.863+0000 7f7946ff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.864+0000 7f7946ff5640 1 -- 172.21.15.49:0/125304901 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958004180 con 0x5639ca6f4ea0 2024-03-18T00:51:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.864+0000 7f7946ff5640 1 -- 172.21.15.49:0/125304901 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7958004320 con 0x5639ca6f4ea0 2024-03-18T00:51:16.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.864+0000 7f796c18b740 1 -- 172.21.15.49:0/125304901 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7954018050 con 0x5639ca6f4ea0 2024-03-18T00:51:16.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.864+0000 7f2699a05640 1 --2- 172.21.15.49:0/3707963192 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d0a033b0 0x5615d09ffc80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f268c00efd0 tx=0x7f268c086200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ef80602fefea5b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.870 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.864+0000 7f7946ff5640 1 -- 172.21.15.49:0/125304901 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7958042040 con 0x5639ca6f4ea0 2024-03-18T00:51:16.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.864+0000 7f267effd640 1 -- 172.21.15.49:0/3707963192 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c01f040 con 0x5615d0a033b0 2024-03-18T00:51:16.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.864+0000 7f796c18b740 10 client.15783 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:16.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.864+0000 7f796c18b740 1 -- 172.21.15.49:0/125304901 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5639ca700ed0 con 0x5639ca6f4ea0 2024-03-18T00:51:16.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.865+0000 7f267effd640 1 -- 172.21.15.49:0/3707963192 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f268c0bb050 con 0x5615d0a033b0 2024-03-18T00:51:16.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.865+0000 7f267effd640 1 -- 172.21.15.49:0/3707963192 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f268c09aad0 con 0x5615d0a033b0 2024-03-18T00:51:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.865+0000 7f7946ff5640 1 -- 172.21.15.49:0/125304901 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7954018050 con 0x5639ca6f4ea0 2024-03-18T00:51:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.865+0000 7f7946ff5640 1 -- 172.21.15.49:0/125304901 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f795802caa0 con 0x5639ca6f4ea0 2024-03-18T00:51:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.865+0000 7f796c18b740 10 client.15783 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.865+0000 7f796c18b740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.865+0000 7f796c18b740 20 client.15783 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.865+0000 7f796c18b740 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5639ca77cbe0 0x5639ca77f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.866+0000 7f796c18b740 4 client.15783 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:51:16.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.866+0000 7f796c18b740 1 -- 172.21.15.49:0/125304901 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5639ca65c2d0 con 0x5639ca77cbe0 2024-03-18T00:51:16.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.866+0000 7f269e050740 1 -- 172.21.15.49:0/3707963192 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d0a033b0 msgr2=0x5615d09ffc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.866+0000 7f269e050740 1 --2- 172.21.15.49:0/3707963192 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d0a033b0 0x5615d09ffc80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f268c00efd0 tx=0x7f268c086200 comp rx=0 tx=0).stop 2024-03-18T00:51:16.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.867+0000 7f269e050740 1 -- 172.21.15.49:0/3707963192 shutdown_connections 2024-03-18T00:51:16.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.867+0000 7f269e050740 1 --2- 172.21.15.49:0/3707963192 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d0a033b0 0x5615d09ffc80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.867+0000 7f269e050740 1 --2- 172.21.15.49:0/3707963192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fe240 0x5615d0a0f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.867+0000 7f269e050740 1 --2- 172.21.15.49:0/3707963192 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d09fd850 0x5615d09ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.867+0000 7f269e050740 1 -- 172.21.15.49:0/3707963192 >> 172.21.15.49:0/3707963192 conn(0x5615d09f8dc0 msgr2=0x5615d0a07d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.867+0000 7f269e050740 1 -- 172.21.15.49:0/3707963192 shutdown_connections 2024-03-18T00:51:16.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.867+0000 7f269e050740 1 -- 172.21.15.49:0/3707963192 wait complete. 2024-03-18T00:51:16.874 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.868+0000 7f7962ffd640 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5639ca77cbe0 0x5639ca77f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.868+0000 7f269e050740 1 -- start start 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.868+0000 7f269e050740 1 Processor -- start 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.869+0000 7f269e050740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 0x5615d0a0efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.869+0000 7f269e050740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a0baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.869+0000 7f2698a03640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a0baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.869+0000 7f269e050740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 0x5615d0a0bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.869+0000 7f2698a03640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a0baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60172/0 (socket says 172.21.15.49:60172) 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.869+0000 7f2698a03640 1 -- 172.21.15.49:0/143506257 learned_addr learned my addr 172.21.15.49:0/143506257 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.869+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f26941b9360 con 0x5615d09fe240 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.869+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f26941b8c90 con 0x5615d0a033b0 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.869+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f26941b5ed0 con 0x5615d09fd850 2024-03-18T00:51:16.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f269e050740 10 client.0 _ll_register_callbacks cb 0x7f26840097f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:16.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f2699204640 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 0x5615d0a0efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f2699a05640 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 0x5615d0a0bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f2698a03640 1 -- 172.21.15.49:0/143506257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 msgr2=0x5615d0a0efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f2698a03640 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 0x5615d0a0efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f2698a03640 1 -- 172.21.15.49:0/143506257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 msgr2=0x5615d0a0bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:16.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f2698a03640 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 0x5615d0a0bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f2698a03640 1 -- 172.21.15.49:0/143506257 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615d0a0c5d0 con 0x5615d09fe240 2024-03-18T00:51:16.877 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f7962ffd640 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5639ca77cbe0 0x5639ca77f000 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad7705ed8c72fe46 server_cookie=6bbccfb542df6013 in_seq=0 out_seq=0 2024-03-18T00:51:16.877 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:16.870+0000 7f7946ff5640 10 client.15783 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:16.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.871+0000 7f2698a03640 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a0baa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2684022040 tx=0x7f268400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da2639709e864e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:16.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.871+0000 7f267cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:16.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.871+0000 7f267cff9640 1 -- 172.21.15.49:0/143506257 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2684040060 con 0x5615d09fe240 2024-03-18T00:51:16.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.871+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f268c0bb050 con 0x5615d09fe240 2024-03-18T00:51:16.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.871+0000 7f267cff9640 1 -- 172.21.15.49:0/143506257 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f268401b840 con 0x5615d09fe240 2024-03-18T00:51:16.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.872+0000 7f267cff9640 1 -- 172.21.15.49:0/143506257 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2684015040 con 0x5615d09fe240 2024-03-18T00:51:16.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.872+0000 7f269e050740 10 client.15789 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:16.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.872+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5615d0a78cb0 con 0x5615d09fe240 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.872+0000 7f267cff9640 1 -- 172.21.15.49:0/143506257 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f268c0bb050 con 0x5615d09fe240 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.872+0000 7f267cff9640 1 -- 172.21.15.49:0/143506257 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f268403b480 con 0x5615d09fe240 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.872+0000 7f269e050740 10 client.15789 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.872+0000 7f269e050740 10 client.15789 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.872+0000 7f269e050740 20 client.15789 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.873+0000 7f269e050740 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5615d0a795a0 0x5615d0a7b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.873+0000 7f269e050740 4 client.15789 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.873+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615d0a07b60 con 0x5615d0a795a0 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.873+0000 7f2699204640 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5615d0a795a0 0x5615d0a7b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.877+0000 7f2699204640 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5615d0a795a0 0x5615d0a7b9c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3574f3efcca30d82 server_cookie=2306661b6619fc07 in_seq=0 out_seq=0 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.877+0000 7f267cff9640 10 client.15789 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.878+0000 7f267cff9640 1 -- 172.21.15.49:0/143506257 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615d0a07b60 con 0x5615d0a795a0 2024-03-18T00:51:16.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.878+0000 7f267cff9640 10 client.15789 handle_command_reply: tid=0 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.878+0000 7f269e050740 1 client.15789 shutdown 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 msgr2=0x5615d0a0baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f269e050740 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a0baa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f2684022040 tx=0x7f268400bf10 comp rx=0 tx=0).stop 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 shutdown_connections 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f269e050740 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5615d0a795a0 0x5615d0a7b9c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f269e050740 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5615d0a033b0 0x5615d0a0bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f269e050740 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5615d09fe240 0x5615d0a0baa0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f269e050740 1 --2- 172.21.15.49:0/143506257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5615d09fd850 0x5615d0a0efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 >> 172.21.15.49:0/143506257 conn(0x5615d09f8dc0 msgr2=0x5615d0a068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f267cff9640 0 client.15789 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.879+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 shutdown_connections 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.880+0000 7f269e050740 1 -- 172.21.15.49:0/143506257 wait complete. 2024-03-18T00:51:16.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.880+0000 7f269e050740 20 client.15789 trim_cache size 0 max 16384 2024-03-18T00:51:16.883 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:51:16.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.881+0000 7f269b468640 1 -- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26800a2880 msgr2=0x7f26800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.881+0000 7f269b468640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26800a2880 0x7f26800a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2684016000 tx=0x7f26840198d0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.881+0000 7f269b468640 1 -- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26941022d0 msgr2=0x7f26941ae630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.881+0000 7f269b468640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26941022d0 0x7f26941ae630 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f268c00eaa0 tx=0x7f268c00ef60 comp rx=0 tx=0).stop 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.882+0000 7f269b468640 1 -- 172.21.15.49:0/1582041061 shutdown_connections 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.882+0000 7f269b468640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f26800a2880 0x7f26800a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.882+0000 7f269b468640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f26941022d0 0x7f26941ae630 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.882+0000 7f269b468640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2694101910 0x7f2694070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.882+0000 7f269b468640 1 --2- 172.21.15.49:0/1582041061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2694100720 0x7f26940701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.882+0000 7f269b468640 1 -- 172.21.15.49:0/1582041061 >> 172.21.15.49:0/1582041061 conn(0x7f26940fbc90 msgr2=0x7f2694104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.882+0000 7f269b468640 1 -- 172.21.15.49:0/1582041061 shutdown_connections 2024-03-18T00:51:16.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:16.882+0000 7f269b468640 1 -- 172.21.15.49:0/1582041061 wait complete. 2024-03-18T00:51:16.909 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:51:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.351+0000 7f2ab77fe640 1 -- 172.21.15.49:0/3939413702 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa002cc80 con 0x7f2ab8101910 2024-03-18T00:51:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.353+0000 7f2abec12640 1 -- 172.21.15.49:0/3939413702 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 msgr2=0x7f2ab8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.353+0000 7f2abec12640 1 --2- 172.21.15.49:0/3939413702 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 0x7f2ab8101d90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0009900 tx=0x7f2aa002bc30 comp rx=0 tx=0).stop 2024-03-18T00:51:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.354+0000 7f2abec12640 1 -- 172.21.15.49:0/3939413702 shutdown_connections 2024-03-18T00:51:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.354+0000 7f2abec12640 1 --2- 172.21.15.49:0/3939413702 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab81022d0 0x7f2ab8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.354+0000 7f2abec12640 1 --2- 172.21.15.49:0/3939413702 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 0x7f2ab8101d90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.354+0000 7f2abec12640 1 --2- 172.21.15.49:0/3939413702 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab8100720 0x7f2ab8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.354+0000 7f2abec12640 1 -- 172.21.15.49:0/3939413702 >> 172.21.15.49:0/3939413702 conn(0x7f2ab80fbc90 msgr2=0x7f2ab80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.354+0000 7f2abec12640 1 -- 172.21.15.49:0/3939413702 shutdown_connections 2024-03-18T00:51:17.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.354+0000 7f2abec12640 1 -- 172.21.15.49:0/3939413702 wait complete. 2024-03-18T00:51:17.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.355+0000 7f2abec12640 1 Processor -- start 2024-03-18T00:51:17.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.355+0000 7f2abec12640 1 -- start start 2024-03-18T00:51:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.356+0000 7f2abec12640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab8100720 0x7f2ab819e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.356+0000 7f2abec12640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 0x7f2ab819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.356+0000 7f2abec12640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab81022d0 0x7f2ab81a5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.356+0000 7f2abec12640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab810c470 con 0x7f2ab8101910 2024-03-18T00:51:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.356+0000 7f2abec12640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab810c2f0 con 0x7f2ab8100720 2024-03-18T00:51:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.356+0000 7f2abec12640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab810c5f0 con 0x7f2ab81022d0 2024-03-18T00:51:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.356+0000 7f2ab7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 0x7f2ab819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.356+0000 7f2ab7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 0x7f2ab819eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60186/0 (socket says 172.21.15.49:60186) 2024-03-18T00:51:17.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.356+0000 7f2ab7fff640 1 -- 172.21.15.49:0/627972855 learned_addr learned my addr 172.21.15.49:0/627972855 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:17.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.357+0000 7f2ab7fff640 1 -- 172.21.15.49:0/627972855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab81022d0 msgr2=0x7f2ab81a5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.357+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab81022d0 0x7f2ab81a5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.357+0000 7f2ab7fff640 1 -- 172.21.15.49:0/627972855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab8100720 msgr2=0x7f2ab819e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.357+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab8100720 0x7f2ab819e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.357+0000 7f2ab7fff640 1 -- 172.21.15.49:0/627972855 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab81a5e80 con 0x7f2ab8101910 2024-03-18T00:51:17.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.358+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 0x7f2ab819eed0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2aa002c240 tx=0x7f2aa0004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.358+0000 7f2ab5ffb640 1 -- 172.21.15.49:0/627972855 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0043020 con 0x7f2ab8101910 2024-03-18T00:51:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.358+0000 7f2abec12640 1 -- 172.21.15.49:0/627972855 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab81a6110 con 0x7f2ab8101910 2024-03-18T00:51:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.358+0000 7f2ab5ffb640 1 -- 172.21.15.49:0/627972855 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa00054e0 con 0x7f2ab8101910 2024-03-18T00:51:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.358+0000 7f2ab5ffb640 1 -- 172.21.15.49:0/627972855 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0003d60 con 0x7f2ab8101910 2024-03-18T00:51:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.359+0000 7f2abec12640 1 -- 172.21.15.49:0/627972855 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab81a66b0 con 0x7f2ab8101910 2024-03-18T00:51:17.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.364+0000 7f2ab5ffb640 1 -- 172.21.15.49:0/627972855 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2aa0046020 con 0x7f2ab8101910 2024-03-18T00:51:17.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.365+0000 7f2ac1803740 1 Processor -- start 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.365+0000 7f2ac1803740 1 -- start start 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.365+0000 7f2ab5ffb640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2aa40a2850 0x7f2aa40a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.365+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eea9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.365+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeaed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.365+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaf3b0 0x558a1eeb1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.366+0000 7f2ac1803740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558a1ee0e490 con 0x558a1eeaa240 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.366+0000 7f2ac1803740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558a1edb6c90 con 0x558a1eeaf3b0 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.366+0000 7f2ac1803740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558a1ee0a530 con 0x558a1eea9850 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.366+0000 7f2abd1af640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2aa40a2850 0x7f2aa40a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.366+0000 7f2abc9ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eea9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.366+0000 7f2abc9ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eea9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60508/0 (socket says 172.21.15.49:60508) 2024-03-18T00:51:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.366+0000 7f2abc9ae640 1 -- 172.21.15.49:0/4184370798 learned_addr learned my addr 172.21.15.49:0/4184370798 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.366+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/4184370798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeaed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.367+0000 7f2abc9ae640 1 -- 172.21.15.49:0/4184370798 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaf3b0 msgr2=0x558a1eeb1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.367+0000 7f2abc9ae640 1 --2- 172.21.15.49:0/4184370798 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaf3b0 0x558a1eeb1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.367+0000 7f2ab5ffb640 1 -- 172.21.15.49:0/627972855 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2aa00f0d80 con 0x7f2ab8101910 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.367+0000 7f2abc9ae640 1 -- 172.21.15.49:0/4184370798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 msgr2=0x558a1eeaed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.367+0000 7f2abc9ae640 1 --2- 172.21.15.49:0/4184370798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeaed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.367+0000 7f2abc9ae640 1 -- 172.21.15.49:0/4184370798 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a1ee71500 con 0x558a1eea9850 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.367+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/4184370798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeaed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.367+0000 7f2abd1af640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2aa40a2850 0x7f2aa40a4d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000ab50 tx=0x7f2ab0009290 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.367+0000 7f2abc9ae640 1 --2- 172.21.15.49:0/4184370798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eea9c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800baf0 tx=0x7f2aa800d2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d603d6a9d3f87728 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.368+0000 7f2aaeffd640 1 -- 172.21.15.49:0/4184370798 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8016070 con 0x558a1eea9850 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.368+0000 7f2aaeffd640 1 -- 172.21.15.49:0/4184370798 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa800dd10 con 0x558a1eea9850 2024-03-18T00:51:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.368+0000 7f2aaeffd640 1 -- 172.21.15.49:0/4184370798 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800f840 con 0x558a1eea9850 2024-03-18T00:51:17.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.369+0000 7f2ac1803740 1 -- 172.21.15.49:0/4184370798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 msgr2=0x558a1eea9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.369+0000 7f2ac1803740 1 --2- 172.21.15.49:0/4184370798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eea9c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800baf0 tx=0x7f2aa800d2e0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.369+0000 7f2ac1803740 1 -- 172.21.15.49:0/4184370798 shutdown_connections 2024-03-18T00:51:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.369+0000 7f2ac1803740 1 --2- 172.21.15.49:0/4184370798 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaf3b0 0x558a1eeb1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.369+0000 7f2ac1803740 1 --2- 172.21.15.49:0/4184370798 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeaed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.369+0000 7f2ac1803740 1 --2- 172.21.15.49:0/4184370798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eea9c70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.369+0000 7f2ac1803740 1 -- 172.21.15.49:0/4184370798 >> 172.21.15.49:0/4184370798 conn(0x558a1eea4dc0 msgr2=0x558a1eea51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.369+0000 7f2ac1803740 1 -- 172.21.15.49:0/4184370798 shutdown_connections 2024-03-18T00:51:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.369+0000 7f2ac1803740 1 -- 172.21.15.49:0/4184370798 wait complete. 2024-03-18T00:51:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.370+0000 7f2ac1803740 1 -- start start 2024-03-18T00:51:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.370+0000 7f2ac1803740 1 Processor -- start 2024-03-18T00:51:17.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eeb3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeb42c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaf3b0 0x558a1eebb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ac1803740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558a1eeb2360 con 0x558a1eeaa240 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ac1803740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558a1edca230 con 0x558a1eeaf3b0 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ac1803740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558a1eeb24e0 con 0x558a1eea9850 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ac1803740 10 client.0 _ll_register_callbacks cb 0x558a1ee06010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ab7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeb42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ab7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeb42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60204/0 (socket says 172.21.15.49:60204) 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.371+0000 7f2ab7fff640 1 -- 172.21.15.49:0/3639168209 learned_addr learned my addr 172.21.15.49:0/3639168209 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.372+0000 7f2abd1af640 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaf3b0 0x558a1eebb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.372+0000 7f2ab7fff640 1 -- 172.21.15.49:0/3639168209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 msgr2=0x558a1eeb3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.372+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eeb3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.372+0000 7f2ab7fff640 1 -- 172.21.15.49:0/3639168209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaf3b0 msgr2=0x558a1eebb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.372+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaf3b0 0x558a1eebb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.372+0000 7f2ab7fff640 1 -- 172.21.15.49:0/3639168209 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a1edd9100 con 0x558a1eeaa240 2024-03-18T00:51:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.373+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeb42c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00bd400 tx=0x7f2aa00bd300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c285a32ca071d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.373+0000 7f2a97fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.374+0000 7f2a97fff640 1 -- 172.21.15.49:0/3639168209 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa009ab90 con 0x558a1eeaa240 2024-03-18T00:51:17.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.374+0000 7f2a97fff640 1 -- 172.21.15.49:0/3639168209 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2aa00bb7a0 con 0x558a1eeaa240 2024-03-18T00:51:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.374+0000 7f2ac1803740 1 -- 172.21.15.49:0/3639168209 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2aa800dd10 con 0x558a1eeaa240 2024-03-18T00:51:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.374+0000 7f2a97fff640 1 -- 172.21.15.49:0/3639168209 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2aa0005690 con 0x558a1eeaa240 2024-03-18T00:51:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.374+0000 7f2ac1803740 10 client.15801 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:17.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.374+0000 7f2ac1803740 1 -- 172.21.15.49:0/3639168209 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558a1eb223f0 con 0x558a1eeaa240 2024-03-18T00:51:17.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:17 smithi049 ceph-mon[32130]: pgmap v385: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 790 MiB/s wr, 1.84k op/s 2024-03-18T00:51:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.375+0000 7f2a97fff640 1 -- 172.21.15.49:0/3639168209 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa800dd10 con 0x558a1eeaa240 2024-03-18T00:51:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.375+0000 7f2a97fff640 1 -- 172.21.15.49:0/3639168209 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f2aa00b9d70 con 0x558a1eeaa240 2024-03-18T00:51:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.375+0000 7f2ac1803740 10 client.15801 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.375+0000 7f2ac1803740 10 client.15801 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:51:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.375+0000 7f2ac1803740 20 client.15801 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:17.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.375+0000 7f2ac1803740 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x558a1eebfb10 0x558a1eec1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.375+0000 7f2ac1803740 4 client.15801 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:17.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.375+0000 7f2ac1803740 1 -- 172.21.15.49:0/3639168209 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a1ee1b9d0 con 0x558a1eebfb10 2024-03-18T00:51:17.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.377+0000 7f2abc9ae640 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x558a1eebfb10 0x558a1eec1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.379+0000 7f2abc9ae640 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x558a1eebfb10 0x558a1eec1fb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b64d89d2dff3982e server_cookie=5f369c00f79f038d in_seq=0 out_seq=0 2024-03-18T00:51:17.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.379+0000 7f2a97fff640 10 client.15801 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:51:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.381+0000 7f2a97fff640 1 -- 172.21.15.49:0/3639168209 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558a1ee1b9d0 con 0x558a1eebfb10 2024-03-18T00:51:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.381+0000 7f2a97fff640 10 client.15801 handle_command_reply: tid=0 2024-03-18T00:51:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 client.15801 shutdown 2024-03-18T00:51:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 -- 172.21.15.49:0/3639168209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 msgr2=0x558a1eeb42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeb42c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00bd400 tx=0x7f2aa00bd300 comp rx=0 tx=0).stop 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 -- 172.21.15.49:0/3639168209 shutdown_connections 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x558a1eebfb10 0x558a1eec1fb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaf3b0 0x558a1eebb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaa240 0x558a1eeb42c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 --2- 172.21.15.49:0/3639168209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eeb3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 -- 172.21.15.49:0/3639168209 >> 172.21.15.49:0/3639168209 conn(0x558a1eea4dc0 msgr2=0x558a1eea3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2a97fff640 0 client.15801 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 -- 172.21.15.49:0/3639168209 shutdown_connections 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 1 -- 172.21.15.49:0/3639168209 wait complete. 2024-03-18T00:51:17.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.382+0000 7f2ac1803740 20 client.15801 trim_cache size 0 max 16384 2024-03-18T00:51:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.401+0000 7f2ac1803740 1 Processor -- start 2024-03-18T00:51:17.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.401+0000 7f2ac1803740 1 -- start start 2024-03-18T00:51:17.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.401+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eea9850 0x558a1eeab740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.402+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eeaa240 0x558a1eeabc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.402+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1eebb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.402+0000 7f2ac1803740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81af3e0 con 0x558a1eeaf3b0 2024-03-18T00:51:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.402+0000 7f2ac1803740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81b0f50 con 0x558a1eea9850 2024-03-18T00:51:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.402+0000 7f2ac1803740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81afab0 con 0x558a1eeaa240 2024-03-18T00:51:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.402+0000 7f2abd1af640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1eebb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.402+0000 7f2abd1af640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1eebb280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60208/0 (socket says 172.21.15.49:60208) 2024-03-18T00:51:17.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.402+0000 7f2abd1af640 1 -- 172.21.15.49:0/1753530041 learned_addr learned my addr 172.21.15.49:0/1753530041 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.402+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/1753530041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eeaa240 0x558a1eeabc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.403+0000 7f2ab7fff640 1 -- 172.21.15.49:0/1753530041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eea9850 msgr2=0x558a1eeab740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.403+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/1753530041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eea9850 0x558a1eeab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.403+0000 7f2ab7fff640 1 -- 172.21.15.49:0/1753530041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 msgr2=0x558a1eebb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.403+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/1753530041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1eebb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.403+0000 7f2ab7fff640 1 -- 172.21.15.49:0/1753530041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a1ebec7d0 con 0x558a1eeaa240 2024-03-18T00:51:17.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.404+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/1753530041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eeaa240 0x558a1eeabc80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00ba0d0 tx=0x7f2aa00bd370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=535a743f61a374f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.405+0000 7f2aae7fc640 1 -- 172.21.15.49:0/1753530041 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa009ae30 con 0x558a1eeaa240 2024-03-18T00:51:17.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.405+0000 7f2aae7fc640 1 -- 172.21.15.49:0/1753530041 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa00c1850 con 0x558a1eeaa240 2024-03-18T00:51:17.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.405+0000 7f2aae7fc640 1 -- 172.21.15.49:0/1753530041 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa00bb7a0 con 0x558a1eeaa240 2024-03-18T00:51:17.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.407+0000 7f2ac1803740 1 -- 172.21.15.49:0/1753530041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eeaa240 msgr2=0x558a1eeabc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.407+0000 7f2ac1803740 1 --2- 172.21.15.49:0/1753530041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eeaa240 0x558a1eeabc80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00ba0d0 tx=0x7f2aa00bd370 comp rx=0 tx=0).stop 2024-03-18T00:51:17.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.407+0000 7f2ac1803740 1 -- 172.21.15.49:0/1753530041 shutdown_connections 2024-03-18T00:51:17.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.408+0000 7f2ac1803740 1 --2- 172.21.15.49:0/1753530041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1eebb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.408+0000 7f2ac1803740 1 --2- 172.21.15.49:0/1753530041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eeaa240 0x558a1eeabc80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.408+0000 7f2ac1803740 1 --2- 172.21.15.49:0/1753530041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eea9850 0x558a1eeab740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.408+0000 7f2ac1803740 1 -- 172.21.15.49:0/1753530041 >> 172.21.15.49:0/1753530041 conn(0x558a1eea4dc0 msgr2=0x558a1eeb3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:17.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.408+0000 7f2ac1803740 1 -- 172.21.15.49:0/1753530041 shutdown_connections 2024-03-18T00:51:17.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.408+0000 7f2ac1803740 1 -- 172.21.15.49:0/1753530041 wait complete. 2024-03-18T00:51:17.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.409+0000 7f2ac1803740 1 -- start start 2024-03-18T00:51:17.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.409+0000 7f2ac1803740 1 Processor -- start 2024-03-18T00:51:17.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.410+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eeb8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.410+0000 7f2ac1803740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaa240 0x558a1eeb85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.411+0000 7f2abc9ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eeb8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.411+0000 7f2abc9ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eeb8060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60538/0 (socket says 172.21.15.49:60538) 2024-03-18T00:51:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.411+0000 7f2abc9ae640 1 -- 172.21.15.49:0/2784891759 learned_addr learned my addr 172.21.15.49:0/2784891759 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.411+0000 7f2ac1803740 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1ef23ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.411+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81b0850 con 0x558a1eeaf3b0 2024-03-18T00:51:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.411+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81b0180 con 0x558a1eeaa240 2024-03-18T00:51:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.411+0000 7f2ab7fff640 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaa240 0x558a1eeb85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.411+0000 7f2abd1af640 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1ef23ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.411+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab81ad3c0 con 0x558a1eea9850 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.412+0000 7f2ac1803740 10 client.0 _ll_register_callbacks cb 0x7f2aa0003f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.412+0000 7f2abd1af640 1 -- 172.21.15.49:0/2784891759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 msgr2=0x558a1eeb8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.412+0000 7f2abd1af640 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eeb8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.412+0000 7f2abd1af640 1 -- 172.21.15.49:0/2784891759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaa240 msgr2=0x558a1eeb85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.412+0000 7f2abd1af640 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaa240 0x558a1eeb85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.412+0000 7f2abd1af640 1 -- 172.21.15.49:0/2784891759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a1eeb8c40 con 0x558a1eeaf3b0 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.413+0000 7f2abd1af640 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1ef23ee0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000afd0 tx=0x7f2ab0004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd38d775c8b37ffd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.413+0000 7f2a97fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.413+0000 7f2a97fff640 1 -- 172.21.15.49:0/2784891759 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab001ab50 con 0x558a1eeaf3b0 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.413+0000 7f2a97fff640 1 -- 172.21.15.49:0/2784891759 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2ab001acf0 con 0x558a1eeaf3b0 2024-03-18T00:51:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.413+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558a1ee15d20 con 0x558a1eeaf3b0 2024-03-18T00:51:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.413+0000 7f2a97fff640 1 -- 172.21.15.49:0/2784891759 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2ab0024460 con 0x558a1eeaf3b0 2024-03-18T00:51:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.413+0000 7f2ac1803740 10 client.15807 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.414+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558a1ef255c0 con 0x558a1eeaf3b0 2024-03-18T00:51:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.414+0000 7f2a97fff640 1 -- 172.21.15.49:0/2784891759 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558a1ee15d20 con 0x558a1eeaf3b0 2024-03-18T00:51:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.414+0000 7f2a97fff640 1 -- 172.21.15.49:0/2784891759 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f2ab0018550 con 0x558a1eeaf3b0 2024-03-18T00:51:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.414+0000 7f2ac1803740 10 client.15807 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.414+0000 7f2ac1803740 10 client.15807 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:51:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.414+0000 7f2ac1803740 20 client.15807 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.414+0000 7f2ac1803740 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x558a1ef25fa0 0x558a1ef283c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.415+0000 7f2ac1803740 4 client.15807 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:51:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.415+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558a1ed936a0 con 0x558a1ef25fa0 2024-03-18T00:51:17.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.416+0000 7f2abc9ae640 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x558a1ef25fa0 0x558a1ef283c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.419+0000 7f2a97fff640 10 client.15807 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:51:17.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.418+0000 7f2abc9ae640 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x558a1ef25fa0 0x558a1ef283c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9ae97f55c3aa3b4e server_cookie=d432d82cdb54ee29 in_seq=0 out_seq=0 2024-03-18T00:51:17.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.419+0000 7f2a97fff640 1 -- 172.21.15.49:0/2784891759 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558a1ed936a0 con 0x558a1ef25fa0 2024-03-18T00:51:17.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.419+0000 7f2a97fff640 10 client.15807 handle_command_reply: tid=0 2024-03-18T00:51:17.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.419+0000 7f2ac1803740 1 client.15807 shutdown 2024-03-18T00:51:17.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.419+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 msgr2=0x558a1ef23ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.419+0000 7f2ac1803740 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1ef23ee0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000afd0 tx=0x7f2ab0004b50 comp rx=0 tx=0).stop 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.419+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 shutdown_connections 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.419+0000 7f2ac1803740 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x558a1ef25fa0 0x558a1ef283c0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.420+0000 7f2ac1803740 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558a1eeaa240 0x558a1eeb85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.420+0000 7f2ac1803740 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558a1eeaf3b0 0x558a1ef23ee0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.420+0000 7f2a97fff640 0 client.15807 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.420+0000 7f2ac1803740 1 --2- 172.21.15.49:0/2784891759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558a1eea9850 0x558a1eeb8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.420+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 >> 172.21.15.49:0/2784891759 conn(0x558a1eea4dc0 msgr2=0x558a1eeab740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.420+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 shutdown_connections 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.420+0000 7f2ac1803740 1 -- 172.21.15.49:0/2784891759 wait complete. 2024-03-18T00:51:17.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.420+0000 7f2ac1803740 20 client.15807 trim_cache size 0 max 16384 2024-03-18T00:51:17.424 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:51:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 -- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2aa40a2850 msgr2=0x7f2aa40a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2aa40a2850 0x7f2aa40a4d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000ab50 tx=0x7f2ab0009290 comp rx=0 tx=0).stop 2024-03-18T00:51:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 -- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 msgr2=0x7f2ab819eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 0x7f2ab819eed0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2aa002c240 tx=0x7f2aa0004980 comp rx=0 tx=0).stop 2024-03-18T00:51:17.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 -- 172.21.15.49:0/627972855 shutdown_connections 2024-03-18T00:51:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2aa40a2850 0x7f2aa40a4d10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ab81022d0 0x7f2ab81a5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ab8101910 0x7f2ab819eed0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 --2- 172.21.15.49:0/627972855 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ab8100720 0x7f2ab819e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 -- 172.21.15.49:0/627972855 >> 172.21.15.49:0/627972855 conn(0x7f2ab80fbc90 msgr2=0x7f2ab80fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 -- 172.21.15.49:0/627972855 shutdown_connections 2024-03-18T00:51:17.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.421+0000 7f2abec12640 1 -- 172.21.15.49:0/627972855 wait complete. 2024-03-18T00:51:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:17 smithi107 ceph-mon[36270]: pgmap v385: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 790 MiB/s wr, 1.84k op/s 2024-03-18T00:51:17.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:17 smithi106 ceph-mon[35899]: pgmap v385: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 790 MiB/s wr, 1.84k op/s 2024-03-18T00:51:17.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:51:17.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.811+0000 7f72b34ac640 1 -- 172.21.15.49:0/1395013366 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac101350 msgr2=0x7f72ac1017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.811+0000 7f72b34ac640 1 --2- 172.21.15.49:0/1395013366 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac101350 0x7f72ac1017d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009900 tx=0x7f72a802bc30 comp rx=0 tx=0).stop 2024-03-18T00:51:17.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.811+0000 7f72b34ac640 1 -- 172.21.15.49:0/1395013366 shutdown_connections 2024-03-18T00:51:17.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.811+0000 7f72b34ac640 1 --2- 172.21.15.49:0/1395013366 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72ac1009b0 0x7f72ac100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.811+0000 7f72b34ac640 1 --2- 172.21.15.49:0/1395013366 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac101350 0x7f72ac1017d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.811+0000 7f72b34ac640 1 --2- 172.21.15.49:0/1395013366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72ac100050 0x7f72ac100470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.811+0000 7f72b34ac640 1 -- 172.21.15.49:0/1395013366 >> 172.21.15.49:0/1395013366 conn(0x7f72ac0fbc70 msgr2=0x7f72ac0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:17.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.814+0000 7f72b34ac640 1 -- 172.21.15.49:0/1395013366 shutdown_connections 2024-03-18T00:51:17.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.815+0000 7f72b34ac640 1 -- 172.21.15.49:0/1395013366 wait complete. 2024-03-18T00:51:17.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.817+0000 7f72b34ac640 1 Processor -- start 2024-03-18T00:51:17.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.817+0000 7f72b34ac640 1 -- start start 2024-03-18T00:51:17.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.818+0000 7f72b34ac640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac100050 0x7f72ac199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.819+0000 7f72b1248640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac100050 0x7f72ac199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.819+0000 7f72b1248640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac100050 0x7f72ac199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52080/0 (socket says 172.21.15.49:52080) 2024-03-18T00:51:17.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.819+0000 7f72b1248640 1 -- 172.21.15.49:0/2324350224 learned_addr learned my addr 172.21.15.49:0/2324350224 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:17.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.819+0000 7f72b34ac640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72ac1009b0 0x7f72ac19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.819+0000 7f72b34ac640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72ac101350 0x7f72ac1a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.819+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac10f650 con 0x7f72ac100050 2024-03-18T00:51:17.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.819+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac10f4d0 con 0x7f72ac101350 2024-03-18T00:51:17.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.819+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac10f7d0 con 0x7f72ac1009b0 2024-03-18T00:51:17.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.820+0000 7f72b0a47640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72ac1009b0 0x7f72ac19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.820+0000 7f72b1a49640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72ac101350 0x7f72ac1a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.820+0000 7f72b0a47640 1 -- 172.21.15.49:0/2324350224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72ac101350 msgr2=0x7f72ac1a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.820+0000 7f72b0a47640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72ac101350 0x7f72ac1a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.820+0000 7f72b0a47640 1 -- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac100050 msgr2=0x7f72ac199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.820+0000 7f72b0a47640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac100050 0x7f72ac199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.820+0000 7f72b0a47640 1 -- 172.21.15.49:0/2324350224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72ac1a1900 con 0x7f72ac1009b0 2024-03-18T00:51:17.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.821+0000 7f72b1248640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac100050 0x7f72ac199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:51:17.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.822+0000 7f72b0a47640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72ac1009b0 0x7f72ac19a200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c970 tx=0x7f729c00ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.823+0000 7f72a27fc640 1 -- 172.21.15.49:0/2324350224 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c007bb0 con 0x7f72ac1009b0 2024-03-18T00:51:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.823+0000 7f72a27fc640 1 -- 172.21.15.49:0/2324350224 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c007d50 con 0x7f72ac1009b0 2024-03-18T00:51:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.823+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72ac1a1bf0 con 0x7f72ac1009b0 2024-03-18T00:51:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.823+0000 7f72a27fc640 1 -- 172.21.15.49:0/2324350224 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c007bb0 con 0x7f72ac1009b0 2024-03-18T00:51:17.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.824+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72ac1a2140 con 0x7f72ac1009b0 2024-03-18T00:51:17.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.830+0000 7f72b6172740 1 Processor -- start 2024-03-18T00:51:17.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.831+0000 7f72b6172740 1 -- start start 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72b6172740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba39060 0x55a2dba39480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72a27fc640 1 -- 172.21.15.49:0/2324350224 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f729c004c20 con 0x7f72ac1009b0 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72b6172740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39a50 0x55a2dba3e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72b6172740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 0x55a2dba41050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72b6172740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a2db995490 con 0x55a2dba39a50 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72b6172740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a2db93dc90 con 0x55a2dba39060 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72b6172740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a2db991530 con 0x55a2dba3ebc0 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72b1248640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39a50 0x55a2dba3e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72b1248640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39a50 0x55a2dba3e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52088/0 (socket says 172.21.15.49:52088) 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.832+0000 7f72b1248640 1 -- 172.21.15.49:0/440309301 learned_addr learned my addr 172.21.15.49:0/440309301 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72b1a49640 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba39060 0x55a2dba39480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72a27fc640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72980a2bf0 0x7f72980a50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72b0a47640 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 0x55a2dba41050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72b1248640 1 -- 172.21.15.49:0/440309301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 msgr2=0x55a2dba41050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72b1248640 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 0x55a2dba41050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72b1248640 1 -- 172.21.15.49:0/440309301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba39060 msgr2=0x55a2dba39480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72b1248640 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba39060 0x55a2dba39480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72b1248640 1 -- 172.21.15.49:0/440309301 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2db9f8500 con 0x55a2dba39a50 2024-03-18T00:51:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72b0a47640 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 0x55a2dba41050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:51:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.833+0000 7f72a27fc640 1 -- 172.21.15.49:0/2324350224 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f729c0079b0 con 0x7f72ac1009b0 2024-03-18T00:51:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.834+0000 7f72b1248640 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39a50 0x55a2dba3e550 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f72a4011850 tx=0x7f72a4011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3636b84bac4e8a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.834+0000 7f72b1a49640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72980a2bf0 0x7f72980a50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.834+0000 7f72977fe640 1 -- 172.21.15.49:0/440309301 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a400fb40 con 0x55a2dba39a50 2024-03-18T00:51:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.834+0000 7f72977fe640 1 -- 172.21.15.49:0/440309301 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a4016070 con 0x55a2dba39a50 2024-03-18T00:51:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.834+0000 7f72977fe640 1 -- 172.21.15.49:0/440309301 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a40126b0 con 0x55a2dba39a50 2024-03-18T00:51:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.835+0000 7f72b6172740 1 -- 172.21.15.49:0/440309301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39a50 msgr2=0x55a2dba3e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.835+0000 7f72b1a49640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72980a2bf0 0x7f72980a50b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009230 tx=0x7f72a8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.835+0000 7f72b6172740 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39a50 0x55a2dba3e550 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f72a4011850 tx=0x7f72a4011d10 comp rx=0 tx=0).stop 2024-03-18T00:51:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.835+0000 7f72b6172740 1 -- 172.21.15.49:0/440309301 shutdown_connections 2024-03-18T00:51:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.835+0000 7f72b6172740 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 0x55a2dba41050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.835+0000 7f72b6172740 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39a50 0x55a2dba3e550 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.835+0000 7f72b6172740 1 --2- 172.21.15.49:0/440309301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba39060 0x55a2dba39480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.835+0000 7f72b6172740 1 -- 172.21.15.49:0/440309301 >> 172.21.15.49:0/440309301 conn(0x55a2dba345d0 msgr2=0x55a2dba349d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:17.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.836+0000 7f72b6172740 1 -- 172.21.15.49:0/440309301 shutdown_connections 2024-03-18T00:51:17.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.836+0000 7f72b6172740 1 -- 172.21.15.49:0/440309301 wait complete. 2024-03-18T00:51:17.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.836+0000 7f72b6172740 1 -- start start 2024-03-18T00:51:17.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.836+0000 7f72b6172740 1 Processor -- start 2024-03-18T00:51:17.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b6172740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39060 0x55a2dba43730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b6172740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba3ebc0 0x55a2dba43c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b6172740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba48c30 0x55a2dba4b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b6172740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a2dba41b70 con 0x55a2dba39060 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b6172740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a2db951230 con 0x55a2dba3ebc0 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b6172740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a2dba41cf0 con 0x55a2dba48c30 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b6172740 10 client.0 _ll_register_callbacks cb 0x55a2db98d010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b1248640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39060 0x55a2dba43730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b1248640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39060 0x55a2dba43730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52090/0 (socket says 172.21.15.49:52090) 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.837+0000 7f72b1248640 1 -- 172.21.15.49:0/3155830850 learned_addr learned my addr 172.21.15.49:0/3155830850 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.838+0000 7f72b1248640 1 -- 172.21.15.49:0/3155830850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba48c30 msgr2=0x55a2dba4b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.838+0000 7f72b1248640 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba48c30 0x55a2dba4b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.838+0000 7f72b1248640 1 -- 172.21.15.49:0/3155830850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba3ebc0 msgr2=0x55a2dba43c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:17.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.838+0000 7f72b1248640 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba3ebc0 0x55a2dba43c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:17.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.838+0000 7f72b1248640 1 -- 172.21.15.49:0/3155830850 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2db960100 con 0x55a2dba39060 2024-03-18T00:51:17.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.838+0000 7f72b1248640 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39060 0x55a2dba43730 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f72a400f730 tx=0x7f72a4004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37bc07d3545c3a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:17.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.838+0000 7f7294ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.839+0000 7f7294ff9640 1 -- 172.21.15.49:0/3155830850 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a4028020 con 0x55a2dba39060 2024-03-18T00:51:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.839+0000 7f7294ff9640 1 -- 172.21.15.49:0/3155830850 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f72a400fd50 con 0x55a2dba39060 2024-03-18T00:51:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.839+0000 7f72b6172740 1 -- 172.21.15.49:0/3155830850 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72a4016070 con 0x55a2dba39060 2024-03-18T00:51:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.839+0000 7f7294ff9640 1 -- 172.21.15.49:0/3155830850 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f72a401b470 con 0x55a2dba39060 2024-03-18T00:51:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.839+0000 7f72b6172740 10 client.15819 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:17.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.839+0000 7f72b6172740 1 -- 172.21.15.49:0/3155830850 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a2db6a93f0 con 0x55a2dba39060 2024-03-18T00:51:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.839+0000 7f7294ff9640 1 -- 172.21.15.49:0/3155830850 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a4016070 con 0x55a2dba39060 2024-03-18T00:51:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.840+0000 7f7294ff9640 1 -- 172.21.15.49:0/3155830850 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f72a400f040 con 0x55a2dba39060 2024-03-18T00:51:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.840+0000 7f72b6172740 10 client.15819 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.840+0000 7f72b6172740 10 client.15819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.840+0000 7f72b6172740 20 client.15819 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.840+0000 7f72b6172740 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a2dba4f530 0x55a2dba51920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.840+0000 7f72b6172740 4 client.15819 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.840+0000 7f72b6172740 1 -- 172.21.15.49:0/3155830850 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2db9a29d0 con 0x55a2dba4f530 2024-03-18T00:51:17.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.841+0000 7f72b0a47640 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a2dba4f530 0x55a2dba51920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.843+0000 7f7294ff9640 10 client.15819 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:17.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:17.843+0000 7f72b0a47640 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a2dba4f530 0x55a2dba51920 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26824251c9978adb server_cookie=5630938e458df1b3 in_seq=0 out_seq=0 2024-03-18T00:51:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:19 smithi107 ceph-mon[36270]: pgmap v386: 97 pgs: 97 active+clean; 9.9 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 938 MiB/s wr, 2.10k op/s 2024-03-18T00:51:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:19 smithi106 ceph-mon[35899]: pgmap v386: 97 pgs: 97 active+clean; 9.9 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 938 MiB/s wr, 2.10k op/s 2024-03-18T00:51:19.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:19 smithi049 ceph-mon[32130]: pgmap v386: 97 pgs: 97 active+clean; 9.9 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 938 MiB/s wr, 2.10k op/s 2024-03-18T00:51:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:21 smithi107 ceph-mon[36270]: pgmap v387: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 918 MiB/s wr, 2.24k op/s 2024-03-18T00:51:21.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:21 smithi106 ceph-mon[35899]: pgmap v387: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 918 MiB/s wr, 2.24k op/s 2024-03-18T00:51:21.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:21 smithi049 ceph-mon[32130]: pgmap v387: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 918 MiB/s wr, 2.24k op/s 2024-03-18T00:51:23.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:23 smithi107 ceph-mon[36270]: pgmap v388: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 814 MiB/s wr, 1.98k op/s 2024-03-18T00:51:23.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:23 smithi106 ceph-mon[35899]: pgmap v388: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 814 MiB/s wr, 1.98k op/s 2024-03-18T00:51:23.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:23 smithi049 ceph-mon[32130]: pgmap v388: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 814 MiB/s wr, 1.98k op/s 2024-03-18T00:51:24.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:24 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:51:24.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:24 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:51:24.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:24 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:51:25.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.479+0000 7f7294ff9640 1 -- 172.21.15.49:0/3155830850 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a2db9a29d0 con 0x55a2dba4f530 2024-03-18T00:51:25.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.479+0000 7f7294ff9640 10 client.15819 handle_command_reply: tid=0 2024-03-18T00:51:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.479+0000 7f7946ff5640 1 -- 172.21.15.49:0/125304901 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5639ca65c2d0 con 0x5639ca77cbe0 2024-03-18T00:51:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.479+0000 7f7946ff5640 10 client.15783 handle_command_reply: tid=0 2024-03-18T00:51:25.480 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.479+0000 7f796c18b740 1 client.15783 shutdown 2024-03-18T00:51:25.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.479+0000 7f72b6172740 1 client.15819 shutdown 2024-03-18T00:51:25.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f72b6172740 1 -- 172.21.15.49:0/3155830850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39060 msgr2=0x55a2dba43730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:25.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f72b6172740 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39060 0x55a2dba43730 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f72a400f730 tx=0x7f72a4004db0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f796c18b740 1 -- 172.21.15.49:0/125304901 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6f4ea0 msgr2=0x5639ca6ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f796c18b740 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6f4ea0 0x5639ca6ff560 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7958009230 tx=0x7f7958004aa0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f796c18b740 1 -- 172.21.15.49:0/125304901 shutdown_connections 2024-03-18T00:51:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f796c18b740 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5639ca77cbe0 0x5639ca77f000 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f796c18b740 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5639ca6f4ea0 0x5639ca6ff560 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.481 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f796c18b740 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5639ca6efd30 0x5639ca6fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f796c18b740 1 --2- 172.21.15.49:0/125304901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5639ca6ef340 0x5639ca6fe010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f796c18b740 1 -- 172.21.15.49:0/125304901 >> 172.21.15.49:0/125304901 conn(0x5639ca6ea8b0 msgr2=0x5639ca6f42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:25.482 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f7946ff5640 0 client.15783 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:25.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f72b6172740 1 -- 172.21.15.49:0/3155830850 shutdown_connections 2024-03-18T00:51:25.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f72b6172740 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a2dba4f530 0x55a2dba51920 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f72b6172740 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba3ebc0 0x55a2dba43c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f72b6172740 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba48c30 0x55a2dba4b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f72b6172740 1 --2- 172.21.15.49:0/3155830850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba39060 0x55a2dba43730 secure :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f72a400f730 tx=0x7f72a4004db0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f72b6172740 1 -- 172.21.15.49:0/3155830850 >> 172.21.15.49:0/3155830850 conn(0x55a2dba345d0 msgr2=0x55a2dba2afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:25.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.480+0000 7f7294ff9640 0 client.15819 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.481+0000 7f796c18b740 1 -- 172.21.15.49:0/125304901 shutdown_connections 2024-03-18T00:51:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.481+0000 7f796c18b740 1 -- 172.21.15.49:0/125304901 wait complete. 2024-03-18T00:51:25.483 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.481+0000 7f796c18b740 20 client.15783 trim_cache size 0 max 16384 2024-03-18T00:51:25.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.481+0000 7f72b6172740 1 -- 172.21.15.49:0/3155830850 shutdown_connections 2024-03-18T00:51:25.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.482+0000 7f72b6172740 1 -- 172.21.15.49:0/3155830850 wait complete. 2024-03-18T00:51:25.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.482+0000 7f72b6172740 20 client.15819 trim_cache size 0 max 16384 2024-03-18T00:51:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:51:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:51:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:51:25.484 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:51:25.500 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.499+0000 7f79695ab640 1 -- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f794c0a2920 msgr2=0x7f794c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:25.500 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.499+0000 7f79695ab640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f794c0a2920 0x7f794c0a4de0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f795c0049d0 tx=0x7f795c0058a0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.500 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.499+0000 7f79695ab640 1 -- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7964101dc0 msgr2=0x7f79641a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:25.500 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.499+0000 7f79695ab640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7964101dc0 0x7f79641a12c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f795c00eb10 tx=0x7f795c00c2d0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.504+0000 7f79695ab640 1 -- 172.21.15.49:0/1816965648 shutdown_connections 2024-03-18T00:51:25.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.504+0000 7f79695ab640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f794c0a2920 0x7f794c0a4de0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.504+0000 7f79695ab640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7964100270 0x7f796419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.504+0000 7f79695ab640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7964101dc0 0x7f79641a12c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.504+0000 7f79695ab640 1 --2- 172.21.15.49:0/1816965648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79640ff8b0 0x7f7964199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.504+0000 7f79695ab640 1 -- 172.21.15.49:0/1816965648 >> 172.21.15.49:0/1816965648 conn(0x7f79640fb410 msgr2=0x7f79640fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.507+0000 7f79695ab640 1 -- 172.21.15.49:0/1816965648 shutdown_connections 2024-03-18T00:51:25.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:25.511+0000 7f79695ab640 1 -- 172.21.15.49:0/1816965648 wait complete. 2024-03-18T00:51:25.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.535+0000 7f72b6172740 1 Processor -- start 2024-03-18T00:51:25.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.535+0000 7f72b6172740 1 -- start start 2024-03-18T00:51:25.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.536+0000 7f72b6172740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 0x55a2dba26e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:25.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.536+0000 7f72b6172740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 0x55a2dba3b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:25.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.537+0000 7f72b6172740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba273b0 0x55a2dba3b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:25.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.537+0000 7f72b6172740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac1a6be0 con 0x55a2dba273b0 2024-03-18T00:51:25.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.537+0000 7f72b1a49640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba273b0 0x55a2dba3b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:25.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.537+0000 7f72b1a49640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba273b0 0x55a2dba3b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52104/0 (socket says 172.21.15.49:52104) 2024-03-18T00:51:25.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.537+0000 7f72b1a49640 1 -- 172.21.15.49:0/2544455210 learned_addr learned my addr 172.21.15.49:0/2544455210 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:25.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.537+0000 7f72b6172740 1 -- 172.21.15.49:0/2544455210 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac1a8750 con 0x55a2dba48c30 2024-03-18T00:51:25.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.538+0000 7f72b6172740 1 -- 172.21.15.49:0/2544455210 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac1a72b0 con 0x55a2dba3ebc0 2024-03-18T00:51:25.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.538+0000 7f72b0a47640 1 --2- 172.21.15.49:0/2544455210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 0x55a2dba3b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:25.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.538+0000 7f72b1a49640 1 -- 172.21.15.49:0/2544455210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 msgr2=0x55a2dba26e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:25.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.538+0000 7f72b1a49640 1 --2- 172.21.15.49:0/2544455210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 0x55a2dba26e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.538+0000 7f72b1a49640 1 -- 172.21.15.49:0/2544455210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 msgr2=0x55a2dba3b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:25.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.538+0000 7f72b1a49640 1 --2- 172.21.15.49:0/2544455210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 0x55a2dba3b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.539+0000 7f72b1a49640 1 -- 172.21.15.49:0/2544455210 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2db7737d0 con 0x55a2dba273b0 2024-03-18T00:51:25.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.539+0000 7f72b0a47640 1 --2- 172.21.15.49:0/2544455210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 0x55a2dba3b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:51:25.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.547+0000 7f72b1a49640 1 --2- 172.21.15.49:0/2544455210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba273b0 0x55a2dba3b670 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f72a8005ec0 tx=0x7f72a802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac87a893ca7cf34c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:25.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.559+0000 7f7296ffd640 1 -- 172.21.15.49:0/2544455210 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a804f020 con 0x55a2dba273b0 2024-03-18T00:51:25.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.560+0000 7f7296ffd640 1 -- 172.21.15.49:0/2544455210 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a8046070 con 0x55a2dba273b0 2024-03-18T00:51:25.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.562+0000 7f72b6172740 1 -- 172.21.15.49:0/2544455210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba273b0 msgr2=0x55a2dba3b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:25.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.562+0000 7f72b6172740 1 --2- 172.21.15.49:0/2544455210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba273b0 0x55a2dba3b670 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f72a8005ec0 tx=0x7f72a802bc30 comp rx=0 tx=0).stop 2024-03-18T00:51:25.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.563+0000 7f72b6172740 1 -- 172.21.15.49:0/2544455210 shutdown_connections 2024-03-18T00:51:25.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.563+0000 7f72b6172740 1 --2- 172.21.15.49:0/2544455210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 0x55a2dba3b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.563+0000 7f72b6172740 1 --2- 172.21.15.49:0/2544455210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba273b0 0x55a2dba3b670 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.563+0000 7f72b6172740 1 --2- 172.21.15.49:0/2544455210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba3ebc0 0x55a2dba26e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.563+0000 7f72b6172740 1 -- 172.21.15.49:0/2544455210 >> 172.21.15.49:0/2544455210 conn(0x55a2dba345d0 msgr2=0x55a2dba420c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:25.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.565+0000 7f72b6172740 1 -- 172.21.15.49:0/2544455210 shutdown_connections 2024-03-18T00:51:25.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.568+0000 7f72b6172740 1 -- 172.21.15.49:0/2544455210 wait complete. 2024-03-18T00:51:25.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.569+0000 7f72b6172740 1 -- start start 2024-03-18T00:51:25.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.569+0000 7f72b6172740 1 Processor -- start 2024-03-18T00:51:25.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.570+0000 7f72b6172740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba3ebc0 0x55a2dba5fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:25.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.570+0000 7f72b6172740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 0x55a2dba60320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:25.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.570+0000 7f72b6172740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba612d0 0x55a2dba61750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:25.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.570+0000 7f72b6172740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac1a8050 con 0x55a2dba3ebc0 2024-03-18T00:51:25.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.570+0000 7f72b6172740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac1a7980 con 0x55a2dba48c30 2024-03-18T00:51:25.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.570+0000 7f72b6172740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f72ac1a4bc0 con 0x55a2dba612d0 2024-03-18T00:51:25.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.570+0000 7f72b6172740 10 client.0 _ll_register_callbacks cb 0x7f72a40116a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:25.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.573+0000 7f72b0a47640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 0x55a2dba60320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:25.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.574+0000 7f72b1a49640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba612d0 0x55a2dba61750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:25.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.574+0000 7f72b1248640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba3ebc0 0x55a2dba5fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:25.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.574+0000 7f72b1248640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba3ebc0 0x55a2dba5fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52112/0 (socket says 172.21.15.49:52112) 2024-03-18T00:51:25.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.574+0000 7f72b1248640 1 -- 172.21.15.49:0/2639792127 learned_addr learned my addr 172.21.15.49:0/2639792127 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:25.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.578+0000 7f72b1248640 1 -- 172.21.15.49:0/2639792127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba612d0 msgr2=0x55a2dba61750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:25.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.578+0000 7f72b1248640 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba612d0 0x55a2dba61750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.578+0000 7f72b1248640 1 -- 172.21.15.49:0/2639792127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 msgr2=0x55a2dba60320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:25.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.578+0000 7f72b1248640 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 0x55a2dba60320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.578+0000 7f72b1248640 1 -- 172.21.15.49:0/2639792127 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2dba61c90 con 0x55a2dba3ebc0 2024-03-18T00:51:25.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.581+0000 7f72b1248640 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba3ebc0 0x55a2dba5fde0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f72a4015040 tx=0x7f72a4009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c030cd6487a933 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:25.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.581+0000 7f7294ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:25.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.584+0000 7f7294ff9640 1 -- 172.21.15.49:0/2639792127 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a403b070 con 0x55a2dba3ebc0 2024-03-18T00:51:25.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.584+0000 7f7294ff9640 1 -- 172.21.15.49:0/2639792127 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f72a4012740 con 0x55a2dba3ebc0 2024-03-18T00:51:25.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.589+0000 7f72b6172740 1 -- 172.21.15.49:0/2639792127 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72a8046070 con 0x55a2dba3ebc0 2024-03-18T00:51:25.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.590+0000 7f7294ff9640 1 -- 172.21.15.49:0/2639792127 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f72a4028040 con 0x55a2dba3ebc0 2024-03-18T00:51:25.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.592+0000 7f72b6172740 10 client.15825 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:25.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.592+0000 7f72b6172740 1 -- 172.21.15.49:0/2639792127 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a2dba2b900 con 0x55a2dba3ebc0 2024-03-18T00:51:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.597+0000 7f7294ff9640 1 -- 172.21.15.49:0/2639792127 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a8046070 con 0x55a2dba3ebc0 2024-03-18T00:51:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.599+0000 7f7294ff9640 1 -- 172.21.15.49:0/2639792127 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f72a403e020 con 0x55a2dba3ebc0 2024-03-18T00:51:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.602+0000 7f72b6172740 10 client.15825 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.602+0000 7f72b6172740 10 client.15825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.602+0000 7f72b6172740 20 client.15825 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.602+0000 7f72b6172740 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a2dba2c220 0x55a2dba2e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.602+0000 7f72b6172740 4 client.15825 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:51:25.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.602+0000 7f72b6172740 1 -- 172.21.15.49:0/2639792127 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a2db99cd20 con 0x55a2dba2c220 2024-03-18T00:51:25.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.604+0000 7f72b0a47640 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a2dba2c220 0x55a2dba2e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.617+0000 7f72b0a47640 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a2dba2c220 0x55a2dba2e640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f0c19e2aac1134e server_cookie=8c3011be9efbf7e8 in_seq=0 out_seq=0 2024-03-18T00:51:25.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.617+0000 7f7294ff9640 10 client.15825 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:25.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.624+0000 7f7294ff9640 1 -- 172.21.15.49:0/2639792127 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a2db99cd20 con 0x55a2dba2c220 2024-03-18T00:51:25.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.624+0000 7f7294ff9640 10 client.15825 handle_command_reply: tid=0 2024-03-18T00:51:25.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.624+0000 7f72b6172740 1 client.15825 shutdown 2024-03-18T00:51:25.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:25 smithi049 ceph-mon[32130]: pgmap v389: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1006 MiB/s wr, 2.25k op/s 2024-03-18T00:51:25.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.628+0000 7f72b6172740 1 -- 172.21.15.49:0/2639792127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba3ebc0 msgr2=0x55a2dba5fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:25.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.628+0000 7f72b6172740 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba3ebc0 0x55a2dba5fde0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f72a4015040 tx=0x7f72a4009850 comp rx=0 tx=0).stop 2024-03-18T00:51:25.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.629+0000 7f72b6172740 1 -- 172.21.15.49:0/2639792127 shutdown_connections 2024-03-18T00:51:25.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.629+0000 7f72b6172740 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a2dba2c220 0x55a2dba2e640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.629+0000 7f72b6172740 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a2dba48c30 0x55a2dba60320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.629+0000 7f72b6172740 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a2dba612d0 0x55a2dba61750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.629+0000 7f72b6172740 1 --2- 172.21.15.49:0/2639792127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a2dba3ebc0 0x55a2dba5fde0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.629+0000 7f7294ff9640 0 client.15825 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:25.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.629+0000 7f72b6172740 1 -- 172.21.15.49:0/2639792127 >> 172.21.15.49:0/2639792127 conn(0x55a2dba345d0 msgr2=0x55a2dba3b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:25.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.630+0000 7f72b6172740 1 -- 172.21.15.49:0/2639792127 shutdown_connections 2024-03-18T00:51:25.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.630+0000 7f72b6172740 1 -- 172.21.15.49:0/2639792127 wait complete. 2024-03-18T00:51:25.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.638+0000 7f72b6172740 20 client.15825 trim_cache size 0 max 16384 2024-03-18T00:51:25.642 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:51:25.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.647+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72980a2bf0 msgr2=0x7f72980a50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:25.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.647+0000 7f72b34ac640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72980a2bf0 0x7f72980a50b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f72a8009230 tx=0x7f72a8009260 comp rx=0 tx=0).stop 2024-03-18T00:51:25.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.647+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72ac1009b0 msgr2=0x7f72ac19a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:25.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.647+0000 7f72b34ac640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72ac1009b0 0x7f72ac19a200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f729c00c970 tx=0x7f729c00ce30 comp rx=0 tx=0).stop 2024-03-18T00:51:25.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.651+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 shutdown_connections 2024-03-18T00:51:25.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.651+0000 7f72b34ac640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f72980a2bf0 0x7f72980a50b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.651+0000 7f72b34ac640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72ac101350 0x7f72ac1a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.651+0000 7f72b34ac640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72ac1009b0 0x7f72ac19a200 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.651+0000 7f72b34ac640 1 --2- 172.21.15.49:0/2324350224 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72ac100050 0x7f72ac199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:25.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.651+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 >> 172.21.15.49:0/2324350224 conn(0x7f72ac0fbc70 msgr2=0x7f72ac0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:25.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.656+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 shutdown_connections 2024-03-18T00:51:25.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:25.656+0000 7f72b34ac640 1 -- 172.21.15.49:0/2324350224 wait complete. 2024-03-18T00:51:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:25 smithi107 ceph-mon[36270]: pgmap v389: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1006 MiB/s wr, 2.25k op/s 2024-03-18T00:51:25.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:25 smithi106 ceph-mon[35899]: pgmap v389: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1006 MiB/s wr, 2.25k op/s 2024-03-18T00:51:26.723 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:51:26.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:51:26.724 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:51:26.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:51:26.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:26 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:51:26.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:26 smithi049 ceph-mon[32130]: pgmap v390: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 794 MiB/s wr, 1.55k op/s 2024-03-18T00:51:26.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:51:26.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:26 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:51:26.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:26 smithi107 ceph-mon[36270]: pgmap v390: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 794 MiB/s wr, 1.55k op/s 2024-03-18T00:51:26.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:51:26.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:26 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:51:26.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:26 smithi106 ceph-mon[35899]: pgmap v390: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 794 MiB/s wr, 1.55k op/s 2024-03-18T00:51:27.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.171+0000 7f4f47115640 1 -- 172.21.15.49:0/1080013027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f40072bf0 msgr2=0x7f4f40073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:27.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.171+0000 7f4f47115640 1 --2- 172.21.15.49:0/1080013027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f40072bf0 0x7f4f40073010 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009900 tx=0x7f4f3002bc60 comp rx=0 tx=0).stop 2024-03-18T00:51:27.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.178+0000 7f4f47115640 1 -- 172.21.15.49:0/1080013027 shutdown_connections 2024-03-18T00:51:27.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.178+0000 7f4f47115640 1 --2- 172.21.15.49:0/1080013027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f40109870 0x7f4f4010e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.178+0000 7f4f47115640 1 --2- 172.21.15.49:0/1080013027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f40108eb0 0x7f4f40109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.178+0000 7f4f47115640 1 --2- 172.21.15.49:0/1080013027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f40072bf0 0x7f4f40073010 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.178+0000 7f4f47115640 1 -- 172.21.15.49:0/1080013027 >> 172.21.15.49:0/1080013027 conn(0x7f4f4006e6c0 msgr2=0x7f4f40070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:27.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.184+0000 7f4f47115640 1 -- 172.21.15.49:0/1080013027 shutdown_connections 2024-03-18T00:51:27.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.184+0000 7f4f47115640 1 -- 172.21.15.49:0/1080013027 wait complete. 2024-03-18T00:51:27.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.185+0000 7f4f47115640 1 Processor -- start 2024-03-18T00:51:27.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.185+0000 7f4f47115640 1 -- start start 2024-03-18T00:51:27.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f47115640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f40072bf0 0x7f4f401a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f47115640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f40108eb0 0x7f4f401a6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f47115640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f40109870 0x7f4f401ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f47115640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4011c1e0 con 0x7f4f40109870 2024-03-18T00:51:27.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f47115640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4011c060 con 0x7f4f40108eb0 2024-03-18T00:51:27.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f47115640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4f4011c360 con 0x7f4f40072bf0 2024-03-18T00:51:27.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f46113640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f40072bf0 0x7f4f401a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f45912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f40108eb0 0x7f4f401a6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f45912640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f40108eb0 0x7f4f401a6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:51370/0 (socket says 172.21.15.49:51370) 2024-03-18T00:51:27.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f45912640 1 -- 172.21.15.49:0/1799527827 learned_addr learned my addr 172.21.15.49:0/1799527827 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:27.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f45912640 1 -- 172.21.15.49:0/1799527827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f40072bf0 msgr2=0x7f4f401a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:27.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f45912640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f40072bf0 0x7f4f401a6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f45912640 1 -- 172.21.15.49:0/1799527827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f40109870 msgr2=0x7f4f401ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:27.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f45912640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f40109870 0x7f4f401ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.186+0000 7f4f45912640 1 -- 172.21.15.49:0/1799527827 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f401ae480 con 0x7f4f40108eb0 2024-03-18T00:51:27.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.187+0000 7f4f45912640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f40108eb0 0x7f4f401a6da0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00b4a0 tx=0x7f4f3c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:27.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.187+0000 7f4f377fe640 1 -- 172.21.15.49:0/1799527827 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c014070 con 0x7f4f40108eb0 2024-03-18T00:51:27.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.188+0000 7f4f377fe640 1 -- 172.21.15.49:0/1799527827 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f3c010af0 con 0x7f4f40108eb0 2024-03-18T00:51:27.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.188+0000 7f4f377fe640 1 -- 172.21.15.49:0/1799527827 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c017d10 con 0x7f4f40108eb0 2024-03-18T00:51:27.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.188+0000 7f4f47115640 1 -- 172.21.15.49:0/1799527827 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f401ae770 con 0x7f4f40108eb0 2024-03-18T00:51:27.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.189+0000 7f4f47115640 1 -- 172.21.15.49:0/1799527827 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f401aec10 con 0x7f4f40108eb0 2024-03-18T00:51:27.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.192+0000 7f4f377fe640 1 -- 172.21.15.49:0/1799527827 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4f3c017070 con 0x7f4f40108eb0 2024-03-18T00:51:27.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.201+0000 7f4f4b03a740 1 Processor -- start 2024-03-18T00:51:27.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.202+0000 7f4f4b03a740 1 -- start start 2024-03-18T00:51:27.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.202+0000 7f4f377fe640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f2c0a2980 0x7f4f2c0a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.203+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46ab50 0x558cbe46af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.203+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46b540 0x558cbe470040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.203+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe472b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.204+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558cbe3cf0d0 con 0x558cbe4706b0 2024-03-18T00:51:27.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.204+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558cbe377c70 con 0x558cbe46b540 2024-03-18T00:51:27.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.204+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558cbe3cb170 con 0x558cbe46ab50 2024-03-18T00:51:27.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.203+0000 7f4f377fe640 1 -- 172.21.15.49:0/1799527827 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4f3c0cdce0 con 0x7f4f40108eb0 2024-03-18T00:51:27.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.204+0000 7f4f46914640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe472b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.204+0000 7f4f46914640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe472b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52146/0 (socket says 172.21.15.49:52146) 2024-03-18T00:51:27.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.204+0000 7f4f46914640 1 -- 172.21.15.49:0/1354092027 learned_addr learned my addr 172.21.15.49:0/1354092027 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:27.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.204+0000 7f4f46914640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f2c0a2980 0x7f4f2c0a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.204+0000 7f4f46113640 1 --2- 172.21.15.49:0/1354092027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46ab50 0x558cbe46af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.205+0000 7f4f46113640 1 -- 172.21.15.49:0/1354092027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46b540 msgr2=0x558cbe470040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:27.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.205+0000 7f4f46113640 1 --2- 172.21.15.49:0/1354092027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46b540 0x558cbe470040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.205+0000 7f4f46113640 1 -- 172.21.15.49:0/1354092027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 msgr2=0x558cbe472b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:27.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.205+0000 7f4f46113640 1 --2- 172.21.15.49:0/1354092027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe472b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.205+0000 7f4f46113640 1 -- 172.21.15.49:0/1354092027 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cbe1ab7e0 con 0x558cbe46ab50 2024-03-18T00:51:27.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.206+0000 7f4f46113640 1 --2- 172.21.15.49:0/1354092027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46ab50 0x558cbe46af70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4f30036040 tx=0x7f4f3002c600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b7f6664244845 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:27.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.207+0000 7f4f46914640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f2c0a2980 0x7f4f2c0a4e40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009810 tx=0x7f4f38009bb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:27.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.209+0000 7f4f34bfe640 1 -- 172.21.15.49:0/1354092027 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30042070 con 0x558cbe46ab50 2024-03-18T00:51:27.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.209+0000 7f4f34bfe640 1 -- 172.21.15.49:0/1354092027 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f3002cbe0 con 0x558cbe46ab50 2024-03-18T00:51:27.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.209+0000 7f4f34bfe640 1 -- 172.21.15.49:0/1354092027 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30004920 con 0x558cbe46ab50 2024-03-18T00:51:27.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.210+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1354092027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46ab50 msgr2=0x558cbe46af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:27.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.210+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1354092027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46ab50 0x558cbe46af70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4f30036040 tx=0x7f4f3002c600 comp rx=0 tx=0).stop 2024-03-18T00:51:27.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.212+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1354092027 shutdown_connections 2024-03-18T00:51:27.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.212+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1354092027 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46b540 0x558cbe470040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.212+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1354092027 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe472b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.212+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1354092027 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46ab50 0x558cbe46af70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.212+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1354092027 >> 172.21.15.49:0/1354092027 conn(0x558cbe432c50 msgr2=0x558cbe466470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:27.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.213+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1354092027 shutdown_connections 2024-03-18T00:51:27.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.213+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1354092027 wait complete. 2024-03-18T00:51:27.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.213+0000 7f4f4b03a740 1 -- start start 2024-03-18T00:51:27.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.213+0000 7f4f4b03a740 1 Processor -- start 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.214+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 0x558cbe475160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.214+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46b540 0x558cbe4756a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.214+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.214+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558cbe473600 con 0x558cbe46b540 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.214+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558cbe38b0d0 con 0x558cbe46ab50 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.214+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558cbe473780 con 0x558cbe4706b0 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.216+0000 7f4f45912640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46b540 0x558cbe4756a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.216+0000 7f4f45912640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46b540 0x558cbe4756a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52154/0 (socket says 172.21.15.49:52154) 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.216+0000 7f4f45912640 1 -- 172.21.15.49:0/2166769768 learned_addr learned my addr 172.21.15.49:0/2166769768 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:27.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.214+0000 7f4f4b03a740 10 client.0 _ll_register_callbacks cb 0x558cbe354760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:27.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.216+0000 7f4f46113640 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 0x558cbe475160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.217+0000 7f4f46914640 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.217+0000 7f4f45912640 1 -- 172.21.15.49:0/2166769768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 msgr2=0x558cbe47c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:27.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.217+0000 7f4f45912640 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.217+0000 7f4f45912640 1 -- 172.21.15.49:0/2166769768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 msgr2=0x558cbe475160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.217+0000 7f4f45912640 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 0x558cbe475160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.217+0000 7f4f45912640 1 -- 172.21.15.49:0/2166769768 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cbe20bf70 con 0x558cbe46b540 2024-03-18T00:51:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.218+0000 7f4f45912640 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46b540 0x558cbe4756a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c006ed0 tx=0x7f4f3c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a311eaed4e7df1c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:27.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.218+0000 7f4f29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.219+0000 7f4f29ffb640 1 -- 172.21.15.49:0/2166769768 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c0b3060 con 0x558cbe46b540 2024-03-18T00:51:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.219+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2166769768 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f3002cbe0 con 0x558cbe46b540 2024-03-18T00:51:27.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.219+0000 7f4f29ffb640 1 -- 172.21.15.49:0/2166769768 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4f3c017360 con 0x558cbe46b540 2024-03-18T00:51:27.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.220+0000 7f4f29ffb640 1 -- 172.21.15.49:0/2166769768 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4f3c01e070 con 0x558cbe46b540 2024-03-18T00:51:27.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.220+0000 7f4f4b03a740 10 client.15831 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:27.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.220+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2166769768 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558cbe0a3170 con 0x558cbe46b540 2024-03-18T00:51:27.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.221+0000 7f4f29ffb640 1 -- 172.21.15.49:0/2166769768 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f3002cbe0 con 0x558cbe46b540 2024-03-18T00:51:27.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.221+0000 7f4f29ffb640 1 -- 172.21.15.49:0/2166769768 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f4f3c0b3200 con 0x558cbe46b540 2024-03-18T00:51:27.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.221+0000 7f4f4b03a740 10 client.15831 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:27.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.221+0000 7f4f4b03a740 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:27.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.221+0000 7f4f4b03a740 20 client.15831 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:27.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.222+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558cbe492090 0x558cbe4944d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.222+0000 7f4f4b03a740 4 client.15831 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:27.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.222+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2166769768 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cbe3dc610 con 0x558cbe492090 2024-03-18T00:51:27.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.228+0000 7f4f46113640 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558cbe492090 0x558cbe4944d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.233+0000 7f4f46113640 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558cbe492090 0x558cbe4944d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed039236d6de2e27 server_cookie=fc489cc41539b338 in_seq=0 out_seq=0 2024-03-18T00:51:27.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.233+0000 7f4f29ffb640 10 client.15831 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:27.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.246+0000 7f4f29ffb640 1 -- 172.21.15.49:0/2166769768 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558cbe3dc610 con 0x558cbe492090 2024-03-18T00:51:27.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.246+0000 7f4f29ffb640 10 client.15831 handle_command_reply: tid=0 2024-03-18T00:51:27.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.247+0000 7f4f4b03a740 1 client.15831 shutdown 2024-03-18T00:51:27.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.248+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2166769768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46b540 msgr2=0x558cbe4756a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:27.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.248+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46b540 0x558cbe4756a0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c006ed0 tx=0x7f4f3c0047d0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.248+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2166769768 shutdown_connections 2024-03-18T00:51:27.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.249+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558cbe492090 0x558cbe4944d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.249+0000 7f4f29ffb640 0 client.15831 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:27.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.249+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.249+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46b540 0x558cbe4756a0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.249+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2166769768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 0x558cbe475160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.249+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2166769768 >> 172.21.15.49:0/2166769768 conn(0x558cbe432c50 msgr2=0x558cbe4656e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:27.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.249+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2166769768 shutdown_connections 2024-03-18T00:51:27.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.249+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2166769768 wait complete. 2024-03-18T00:51:27.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.250+0000 7f4f4b03a740 20 client.15831 trim_cache size 0 max 16384 2024-03-18T00:51:27.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.320+0000 7f4f4b03a740 1 Processor -- start 2024-03-18T00:51:27.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.320+0000 7f4f4b03a740 1 -- start start 2024-03-18T00:51:27.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.320+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 0x558cbe46cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.320+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46b540 0x558cbe47c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.320+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe46d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.320+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401af2b0 con 0x558cbe4706b0 2024-03-18T00:51:27.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.320+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401b0e20 con 0x558cbe46ab50 2024-03-18T00:51:27.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.320+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401af980 con 0x558cbe46b540 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.320+0000 7f4f46914640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe46d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f46914640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe46d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52156/0 (socket says 172.21.15.49:52156) 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f46914640 1 -- 172.21.15.49:0/2492985331 learned_addr learned my addr 172.21.15.49:0/2492985331 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f45912640 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46b540 0x558cbe47c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f46113640 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 0x558cbe46cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f46113640 1 -- 172.21.15.49:0/2492985331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46b540 msgr2=0x558cbe47c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f46113640 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46b540 0x558cbe47c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f46113640 1 -- 172.21.15.49:0/2492985331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 msgr2=0x558cbe46d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f46113640 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe46d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f46113640 1 -- 172.21.15.49:0/2492985331 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cbe474be0 con 0x558cbe46ab50 2024-03-18T00:51:27.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f45912640 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46b540 0x558cbe47c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:51:27.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.321+0000 7f4f46113640 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 0x558cbe46cbc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4f3002c270 tx=0x7f4f30000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db439acaf9c4112 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:27.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.324+0000 7f4f2bfff640 1 -- 172.21.15.49:0/2492985331 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30059070 con 0x558cbe46ab50 2024-03-18T00:51:27.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.329+0000 7f4f2bfff640 1 -- 172.21.15.49:0/2492985331 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f3002cd40 con 0x558cbe46ab50 2024-03-18T00:51:27.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.329+0000 7f4f2bfff640 1 -- 172.21.15.49:0/2492985331 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30042040 con 0x558cbe46ab50 2024-03-18T00:51:27.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.330+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2492985331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 msgr2=0x558cbe46cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:27.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.330+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 0x558cbe46cbc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4f3002c270 tx=0x7f4f30000bc0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.331+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2492985331 shutdown_connections 2024-03-18T00:51:27.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.331+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe4706b0 0x558cbe46d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.331+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe46b540 0x558cbe47c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.331+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/2492985331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46ab50 0x558cbe46cbc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.331+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2492985331 >> 172.21.15.49:0/2492985331 conn(0x558cbe432c50 msgr2=0x558cbe465f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:27.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.345+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2492985331 shutdown_connections 2024-03-18T00:51:27.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.354+0000 7f4f4b03a740 1 -- 172.21.15.49:0/2492985331 wait complete. 2024-03-18T00:51:27.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.354+0000 7f4f4b03a740 1 -- start start 2024-03-18T00:51:27.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.354+0000 7f4f4b03a740 1 Processor -- start 2024-03-18T00:51:27.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.355+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46ab50 0x558cbe47a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.355+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46b540 0x558cbe47a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.355+0000 7f4f4b03a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.355+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401b0720 con 0x558cbe46ab50 2024-03-18T00:51:27.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.355+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4f401b0050 con 0x558cbe46b540 2024-03-18T00:51:27.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.355+0000 7f4f4b03a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4f40116b00 con 0x558cbe4706b0 2024-03-18T00:51:27.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.355+0000 7f4f4b03a740 10 client.0 _ll_register_callbacks cb 0x7f4f3c017630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:27.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.355+0000 7f4f46914640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.356+0000 7f4f46914640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46244/0 (socket says 172.21.15.49:46244) 2024-03-18T00:51:27.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.356+0000 7f4f46914640 1 -- 172.21.15.49:0/1983559199 learned_addr learned my addr 172.21.15.49:0/1983559199 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:27.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.356+0000 7f4f46914640 1 -- 172.21.15.49:0/1983559199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46b540 msgr2=0x558cbe47a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:27.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.356+0000 7f4f46914640 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46b540 0x558cbe47a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.356+0000 7f4f46914640 1 -- 172.21.15.49:0/1983559199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46ab50 msgr2=0x558cbe47a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:27.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.356+0000 7f4f46914640 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46ab50 0x558cbe47a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.356+0000 7f4f46914640 1 -- 172.21.15.49:0/1983559199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cbe47bb40 con 0x558cbe4706b0 2024-03-18T00:51:27.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.357+0000 7f4f46914640 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47b550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4f38010e80 tx=0x7f4f3801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdf02944cc8e138f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:27.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.369+0000 7f4f29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:51:27.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.369+0000 7f4f29ffb640 1 -- 172.21.15.49:0/1983559199 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f38003e00 con 0x558cbe4706b0 2024-03-18T00:51:27.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.369+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1983559199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f3002cd40 con 0x558cbe4706b0 2024-03-18T00:51:27.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.370+0000 7f4f29ffb640 1 -- 172.21.15.49:0/1983559199 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4f38009040 con 0x558cbe4706b0 2024-03-18T00:51:27.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.371+0000 7f4f29ffb640 1 -- 172.21.15.49:0/1983559199 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4f38004740 con 0x558cbe4706b0 2024-03-18T00:51:27.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.375+0000 7f4f4b03a740 10 client.25639 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:27.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.375+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1983559199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558cbe4f8720 con 0x558cbe4706b0 2024-03-18T00:51:27.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.379+0000 7f4f29ffb640 1 -- 172.21.15.49:0/1983559199 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f3002cd40 con 0x558cbe4706b0 2024-03-18T00:51:27.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.379+0000 7f4f29ffb640 1 -- 172.21.15.49:0/1983559199 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f4f3800f040 con 0x558cbe4706b0 2024-03-18T00:51:27.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.382+0000 7f4f4b03a740 10 client.25639 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:27.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.382+0000 7f4f4b03a740 10 client.25639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:27.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.382+0000 7f4f4b03a740 20 client.25639 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:27.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.382+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558cbe4f90a0 0x558cbe4fb520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:27.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.382+0000 7f4f4b03a740 4 client.25639 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:51:27.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.382+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1983559199 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558cbe474f50 con 0x558cbe4f90a0 2024-03-18T00:51:27.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.384+0000 7f4f46113640 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558cbe4f90a0 0x558cbe4fb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:27.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.394+0000 7f4f46113640 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558cbe4f90a0 0x558cbe4fb520 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=198af511fd93690e server_cookie=35eb194ee3e97871 in_seq=0 out_seq=0 2024-03-18T00:51:27.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.395+0000 7f4f29ffb640 10 client.25639 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:27.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.892+0000 7f4f29ffb640 1 -- 172.21.15.49:0/1983559199 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558cbe474f50 con 0x558cbe4f90a0 2024-03-18T00:51:27.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.892+0000 7f4f29ffb640 10 client.25639 handle_command_reply: tid=0 2024-03-18T00:51:27.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.893+0000 7f4f4b03a740 1 client.25639 shutdown 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.893+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1983559199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 msgr2=0x558cbe47b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.893+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47b550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4f38010e80 tx=0x7f4f3801a000 comp rx=0 tx=0).stop 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.893+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1983559199 shutdown_connections 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.893+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558cbe4f90a0 0x558cbe4fb520 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.894+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558cbe46b540 0x558cbe47a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.894+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558cbe4706b0 0x558cbe47b550 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.894+0000 7f4f4b03a740 1 --2- 172.21.15.49:0/1983559199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558cbe46ab50 0x558cbe47a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.894+0000 7f4f29ffb640 0 client.25639 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.894+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1983559199 >> 172.21.15.49:0/1983559199 conn(0x558cbe432c50 msgr2=0x558cbe469700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:27.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.894+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1983559199 shutdown_connections 2024-03-18T00:51:27.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.894+0000 7f4f4b03a740 1 -- 172.21.15.49:0/1983559199 wait complete. 2024-03-18T00:51:27.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.896+0000 7f4f4b03a740 20 client.25639 trim_cache size 0 max 16384 2024-03-18T00:51:27.899 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:51:27.899 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:51:27.899 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "9e2e80bf-0c02-4da1-8be6-1e2499b149cf", 2024-03-18T00:51:27.899 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:51:27.899 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:51:27.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.905+0000 7f4f47115640 1 -- 172.21.15.49:0/1799527827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f2c0a2980 msgr2=0x7f4f2c0a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:27.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.905+0000 7f4f47115640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f2c0a2980 0x7f4f2c0a4e40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4f38009810 tx=0x7f4f38009bb0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.906+0000 7f4f47115640 1 -- 172.21.15.49:0/1799527827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f40108eb0 msgr2=0x7f4f401a6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:27.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.906+0000 7f4f47115640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f40108eb0 0x7f4f401a6da0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c00b4a0 tx=0x7f4f3c00b960 comp rx=0 tx=0).stop 2024-03-18T00:51:27.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.906+0000 7f4f47115640 1 -- 172.21.15.49:0/1799527827 shutdown_connections 2024-03-18T00:51:27.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.906+0000 7f4f47115640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f2c0a2980 0x7f4f2c0a4e40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.906+0000 7f4f47115640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4f40108eb0 0x7f4f401a6da0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.906+0000 7f4f47115640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4f40109870 0x7f4f401ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.907+0000 7f4f47115640 1 --2- 172.21.15.49:0/1799527827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4f40072bf0 0x7f4f401a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:27.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.907+0000 7f4f47115640 1 -- 172.21.15.49:0/1799527827 >> 172.21.15.49:0/1799527827 conn(0x7f4f4006e6c0 msgr2=0x7f4f40070300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:27.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.914+0000 7f4f47115640 1 -- 172.21.15.49:0/1799527827 shutdown_connections 2024-03-18T00:51:27.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:27.914+0000 7f4f47115640 1 -- 172.21.15.49:0/1799527827 wait complete. 2024-03-18T00:51:27.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:51:27.995 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:51:28.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.395+0000 7f40d3236640 1 -- 172.21.15.49:0/836158022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40cc10b150 msgr2=0x7f40cc10d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:28.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.395+0000 7f40d3236640 1 --2- 172.21.15.49:0/836158022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40cc10b150 0x7f40cc10d5c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009900 tx=0x7f40c002bc80 comp rx=0 tx=0).stop 2024-03-18T00:51:28.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.400+0000 7f40d3236640 1 -- 172.21.15.49:0/836158022 shutdown_connections 2024-03-18T00:51:28.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.400+0000 7f40d3236640 1 --2- 172.21.15.49:0/836158022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40cc10db00 0x7f40cc10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.400+0000 7f40d3236640 1 --2- 172.21.15.49:0/836158022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40cc10b150 0x7f40cc10d5c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.400+0000 7f40d3236640 1 --2- 172.21.15.49:0/836158022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40cc1087e0 0x7f40cc10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.400+0000 7f40d3236640 1 -- 172.21.15.49:0/836158022 >> 172.21.15.49:0/836158022 conn(0x7f40cc06e640 msgr2=0x7f40cc070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:28.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.405+0000 7f40d3236640 1 -- 172.21.15.49:0/836158022 shutdown_connections 2024-03-18T00:51:28.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.407+0000 7f40d3236640 1 -- 172.21.15.49:0/836158022 wait complete. 2024-03-18T00:51:28.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.409+0000 7f40d3236640 1 Processor -- start 2024-03-18T00:51:28.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.409+0000 7f40d3236640 1 -- start start 2024-03-18T00:51:28.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.409+0000 7f40d3236640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40cc1087e0 0x7f40cc1a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.409+0000 7f40d3236640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40cc10b150 0x7f40cc1a6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.409+0000 7f40d3236640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40cc10db00 0x7f40cc1ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.410+0000 7f40d3236640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc11c3c0 con 0x7f40cc1087e0 2024-03-18T00:51:28.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.410+0000 7f40d3236640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc11c240 con 0x7f40cc10db00 2024-03-18T00:51:28.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.410+0000 7f40d3236640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40cc11c540 con 0x7f40cc10b150 2024-03-18T00:51:28.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.410+0000 7f40d17d3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40cc10db00 0x7f40cc1ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.410+0000 7f40d17d3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40cc10db00 0x7f40cc1ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:41586/0 (socket says 172.21.15.49:41586) 2024-03-18T00:51:28.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.410+0000 7f40d17d3640 1 -- 172.21.15.49:0/1854709389 learned_addr learned my addr 172.21.15.49:0/1854709389 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:28.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.411+0000 7f40cbfff640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40cc10b150 0x7f40cc1a6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.412+0000 7f40d17d3640 1 -- 172.21.15.49:0/1854709389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40cc10b150 msgr2=0x7f40cc1a6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:28.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.412+0000 7f40d17d3640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40cc10b150 0x7f40cc1a6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.412+0000 7f40d17d3640 1 -- 172.21.15.49:0/1854709389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40cc1087e0 msgr2=0x7f40cc1a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:28.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.412+0000 7f40d17d3640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40cc1087e0 0x7f40cc1a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.412+0000 7f40d17d3640 1 -- 172.21.15.49:0/1854709389 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40cc1ae6b0 con 0x7f40cc10db00 2024-03-18T00:51:28.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.413+0000 7f40d17d3640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40cc10db00 0x7f40cc1ae040 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00b3c0 tx=0x7f40bc00b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:28.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.414+0000 7f40c9ffb640 1 -- 172.21.15.49:0/1854709389 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc011830 con 0x7f40cc10db00 2024-03-18T00:51:28.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.414+0000 7f40d3236640 1 -- 172.21.15.49:0/1854709389 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40cc1ae9a0 con 0x7f40cc10db00 2024-03-18T00:51:28.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.414+0000 7f40d3236640 1 -- 172.21.15.49:0/1854709389 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40cc1aeec0 con 0x7f40cc10db00 2024-03-18T00:51:28.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.419+0000 7f40c9ffb640 1 -- 172.21.15.49:0/1854709389 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40bc016070 con 0x7f40cc10db00 2024-03-18T00:51:28.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.419+0000 7f40c9ffb640 1 -- 172.21.15.49:0/1854709389 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc00f2e0 con 0x7f40cc10db00 2024-03-18T00:51:28.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.420+0000 7f40d5e1e740 1 Processor -- start 2024-03-18T00:51:28.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.420+0000 7f40d5e1e740 1 -- start start 2024-03-18T00:51:28.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.422+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563803995170 0x563803995590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.422+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995b60 0x56380399a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.422+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56380399acd0 0x56380399d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.422+0000 7f40d5e1e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5638038fa500 con 0x563803995170 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.422+0000 7f40d5e1e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5638038a2be0 con 0x563803995b60 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.422+0000 7f40d5e1e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5638038f65a0 con 0x56380399acd0 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.422+0000 7f40c9ffb640 1 -- 172.21.15.49:0/1854709389 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f40bc00f540 con 0x7f40cc10db00 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.424+0000 7f40c9ffb640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f40b40a29d0 0x7f40b40a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.425+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563803995170 0x563803995590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.425+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563803995170 0x563803995590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56720/0 (socket says 172.21.15.49:56720) 2024-03-18T00:51:28.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.425+0000 7f40cbfff640 1 -- 172.21.15.49:0/2359801668 learned_addr learned my addr 172.21.15.49:0/2359801668 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:28.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.425+0000 7f40d17d3640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f40b40a29d0 0x7f40b40a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.425+0000 7f40c9ffb640 1 -- 172.21.15.49:0/1854709389 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f40bc0cf050 con 0x7f40cc10db00 2024-03-18T00:51:28.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.426+0000 7f40cbfff640 1 -- 172.21.15.49:0/2359801668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56380399acd0 msgr2=0x56380399d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:28.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.426+0000 7f40cbfff640 1 --2- 172.21.15.49:0/2359801668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56380399acd0 0x56380399d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.426+0000 7f40cbfff640 1 -- 172.21.15.49:0/2359801668 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995b60 msgr2=0x56380399a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:28.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.426+0000 7f40cbfff640 1 --2- 172.21.15.49:0/2359801668 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995b60 0x56380399a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.427+0000 7f40cbfff640 1 -- 172.21.15.49:0/2359801668 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638038b64b0 con 0x563803995170 2024-03-18T00:51:28.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.428+0000 7f40d17d3640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f40b40a29d0 0x7f40b40a4e90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f40bc010fd0 tx=0x7f40bc0cf2a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:28.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.433+0000 7f40cbfff640 1 --2- 172.21.15.49:0/2359801668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563803995170 0x563803995590 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f40c002cfd0 tx=0x7f40c0005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b49c107411a1751 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:28.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.447+0000 7f40b2ffd640 1 -- 172.21.15.49:0/2359801668 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c002cd90 con 0x563803995170 2024-03-18T00:51:28.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.447+0000 7f40b2ffd640 1 -- 172.21.15.49:0/2359801668 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c00046c0 con 0x563803995170 2024-03-18T00:51:28.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.449+0000 7f40d5e1e740 1 -- 172.21.15.49:0/2359801668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563803995170 msgr2=0x563803995590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:28.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.449+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/2359801668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563803995170 0x563803995590 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f40c002cfd0 tx=0x7f40c0005de0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.449+0000 7f40d5e1e740 1 -- 172.21.15.49:0/2359801668 shutdown_connections 2024-03-18T00:51:28.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.449+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/2359801668 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995b60 0x56380399a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.449+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/2359801668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56380399acd0 0x56380399d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.449+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/2359801668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563803995170 0x563803995590 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.450+0000 7f40d5e1e740 1 -- 172.21.15.49:0/2359801668 >> 172.21.15.49:0/2359801668 conn(0x5638039906e0 msgr2=0x563803990ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:28.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.450+0000 7f40d5e1e740 1 -- 172.21.15.49:0/2359801668 shutdown_connections 2024-03-18T00:51:28.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.451+0000 7f40d5e1e740 1 -- 172.21.15.49:0/2359801668 wait complete. 2024-03-18T00:51:28.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.452+0000 7f40d5e1e740 1 -- start start 2024-03-18T00:51:28.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.452+0000 7f40d5e1e740 1 Processor -- start 2024-03-18T00:51:28.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.453+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x56380399f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.453+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x56380399fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.453+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.453+0000 7f40d5e1e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56380399dc70 con 0x56380399acd0 2024-03-18T00:51:28.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.453+0000 7f40d5e1e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5638038b6160 con 0x563803995170 2024-03-18T00:51:28.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.453+0000 7f40d5e1e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56380399ddf0 con 0x563803995b60 2024-03-18T00:51:28.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.453+0000 7f40d5e1e740 10 client.0 _ll_register_callbacks cb 0x56380389a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:28.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.454+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.454+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56734/0 (socket says 172.21.15.49:56734) 2024-03-18T00:51:28.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.454+0000 7f40cbfff640 1 -- 172.21.15.49:0/711755544 learned_addr learned my addr 172.21.15.49:0/711755544 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:28.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.455+0000 7f40cbfff640 1 -- 172.21.15.49:0/711755544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 msgr2=0x56380399fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:28.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.455+0000 7f40cbfff640 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x56380399fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.455+0000 7f40cbfff640 1 -- 172.21.15.49:0/711755544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 msgr2=0x56380399f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:28.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.455+0000 7f40cbfff640 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x56380399f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.455+0000 7f40cbfff640 1 -- 172.21.15.49:0/711755544 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638036d87d0 con 0x56380399acd0 2024-03-18T00:51:28.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.457+0000 7f40cbfff640 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a6c90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f40c002c800 tx=0x7f40c002c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f24be4ebe018bae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:28.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.457+0000 7f40abfff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:28.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.459+0000 7f40abfff640 1 -- 172.21.15.49:0/711755544 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00035f0 con 0x56380399acd0 2024-03-18T00:51:28.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.459+0000 7f40d5e1e740 1 -- 172.21.15.49:0/711755544 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40c00046c0 con 0x56380399acd0 2024-03-18T00:51:28.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.461+0000 7f40abfff640 1 -- 172.21.15.49:0/711755544 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f40c0003790 con 0x56380399acd0 2024-03-18T00:51:28.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.461+0000 7f40abfff640 1 -- 172.21.15.49:0/711755544 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40c0003990 con 0x56380399acd0 2024-03-18T00:51:28.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.463+0000 7f40d5e1e740 10 client.15843 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:28.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.463+0000 7f40d5e1e740 1 -- 172.21.15.49:0/711755544 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56380360e3f0 con 0x56380399acd0 2024-03-18T00:51:28.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.464+0000 7f40abfff640 1 -- 172.21.15.49:0/711755544 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c00046c0 con 0x56380399acd0 2024-03-18T00:51:28.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.464+0000 7f40abfff640 1 -- 172.21.15.49:0/711755544 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f40c0034bc0 con 0x56380399acd0 2024-03-18T00:51:28.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.464+0000 7f40d5e1e740 10 client.15843 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:28.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.464+0000 7f40d5e1e740 10 client.15843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:28.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.464+0000 7f40d5e1e740 20 client.15843 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:28.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.464+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638039ab470 0x5638039ad890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.464+0000 7f40d5e1e740 4 client.15843 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:28.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.464+0000 7f40d5e1e740 1 -- 172.21.15.49:0/711755544 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563803907a40 con 0x5638039ab470 2024-03-18T00:51:28.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.467+0000 7f40d0fd2640 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638039ab470 0x5638039ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.484+0000 7f40d0fd2640 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638039ab470 0x5638039ad890 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbe697a9f04736c8 server_cookie=d1bdb526348c369d in_seq=0 out_seq=0 2024-03-18T00:51:28.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.484+0000 7f40abfff640 10 client.15843 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:28.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.491+0000 7f40abfff640 1 -- 172.21.15.49:0/711755544 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563803907a40 con 0x5638039ab470 2024-03-18T00:51:28.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.491+0000 7f40abfff640 10 client.15843 handle_command_reply: tid=0 2024-03-18T00:51:28.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.492+0000 7f40d5e1e740 1 client.15843 shutdown 2024-03-18T00:51:28.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.495+0000 7f40d5e1e740 1 -- 172.21.15.49:0/711755544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 msgr2=0x5638039a6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:28.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.495+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a6c90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f40c002c800 tx=0x7f40c002c420 comp rx=0 tx=0).stop 2024-03-18T00:51:28.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.496+0000 7f40d5e1e740 1 -- 172.21.15.49:0/711755544 shutdown_connections 2024-03-18T00:51:28.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.496+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638039ab470 0x5638039ad890 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.496+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a6c90 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.496+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x56380399fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.496+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/711755544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x56380399f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.496+0000 7f40d5e1e740 1 -- 172.21.15.49:0/711755544 >> 172.21.15.49:0/711755544 conn(0x5638039906e0 msgr2=0x56380398f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:28.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.506+0000 7f40abfff640 0 client.15843 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:28.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.506+0000 7f40d5e1e740 1 -- 172.21.15.49:0/711755544 shutdown_connections 2024-03-18T00:51:28.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.507+0000 7f40d5e1e740 1 -- 172.21.15.49:0/711755544 wait complete. 2024-03-18T00:51:28.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.507+0000 7f40d5e1e740 20 client.15843 trim_cache size 0 max 16384 2024-03-18T00:51:28.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.562+0000 7f40d5e1e740 1 Processor -- start 2024-03-18T00:51:28.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.563+0000 7f40d5e1e740 1 -- start start 2024-03-18T00:51:28.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.563+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638039906e0 0x56380399f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.564+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638039906e0 0x56380399f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.564+0000 7f40cbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638039906e0 0x56380399f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:56740/0 (socket says 172.21.15.49:56740) 2024-03-18T00:51:28.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.564+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x56380399fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.564+0000 7f40cbfff640 1 -- 172.21.15.49:0/4055570155 learned_addr learned my addr 172.21.15.49:0/4055570155 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:28.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.564+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/4055570155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x5638039a6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.565+0000 7f40d5e1e740 1 -- 172.21.15.49:0/4055570155 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f40b800e690 con 0x5638039906e0 2024-03-18T00:51:28.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.565+0000 7f40d5e1e740 1 -- 172.21.15.49:0/4055570155 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f40b8010200 con 0x563803995170 2024-03-18T00:51:28.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.565+0000 7f40d5e1e740 1 -- 172.21.15.49:0/4055570155 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40b800ed60 con 0x563803995b60 2024-03-18T00:51:28.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.565+0000 7f40d0fd2640 1 --2- 172.21.15.49:0/4055570155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x56380399fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.566+0000 7f40cbfff640 1 -- 172.21.15.49:0/4055570155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 msgr2=0x5638039a6bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:28.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.566+0000 7f40cbfff640 1 --2- 172.21.15.49:0/4055570155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x5638039a6bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.566+0000 7f40cbfff640 1 -- 172.21.15.49:0/4055570155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 msgr2=0x56380399fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:28.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.566+0000 7f40cbfff640 1 --2- 172.21.15.49:0/4055570155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x56380399fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.566+0000 7f40cbfff640 1 -- 172.21.15.49:0/4055570155 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638035ce0f0 con 0x5638039906e0 2024-03-18T00:51:28.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.568+0000 7f40cbfff640 1 --2- 172.21.15.49:0/4055570155 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638039906e0 0x56380399f710 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f40c002c1c0 tx=0x7f40c0004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f71772c86d0bea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:28.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.571+0000 7f40b27fc640 1 -- 172.21.15.49:0/4055570155 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0062070 con 0x5638039906e0 2024-03-18T00:51:28.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.573+0000 7f40b27fc640 1 -- 172.21.15.49:0/4055570155 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40c004e070 con 0x5638039906e0 2024-03-18T00:51:28.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.574+0000 7f40d5e1e740 1 -- 172.21.15.49:0/4055570155 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638039906e0 msgr2=0x56380399f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:28.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.574+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/4055570155 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638039906e0 0x56380399f710 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f40c002c1c0 tx=0x7f40c0004e30 comp rx=0 tx=0).stop 2024-03-18T00:51:28.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.575+0000 7f40d5e1e740 1 -- 172.21.15.49:0/4055570155 shutdown_connections 2024-03-18T00:51:28.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.575+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/4055570155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x56380399fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.576+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/4055570155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x5638039a6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.576+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/4055570155 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638039906e0 0x56380399f710 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.576+0000 7f40d5e1e740 1 -- 172.21.15.49:0/4055570155 >> 172.21.15.49:0/4055570155 conn(0x563803990e30 msgr2=0x5638039970f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:28.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.577+0000 7f40d5e1e740 1 -- 172.21.15.49:0/4055570155 shutdown_connections 2024-03-18T00:51:28.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.578+0000 7f40d5e1e740 1 -- 172.21.15.49:0/4055570155 wait complete. 2024-03-18T00:51:28.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.578+0000 7f40d5e1e740 1 -- start start 2024-03-18T00:51:28.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.579+0000 7f40d5e1e740 1 Processor -- start 2024-03-18T00:51:28.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.580+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x5638039a6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.581+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x5638039a35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.581+0000 7f40d5e1e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.581+0000 7f40d5e1e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f40b800fb00 con 0x56380399acd0 2024-03-18T00:51:28.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.581+0000 7f40d5e1e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f40b800f430 con 0x563803995170 2024-03-18T00:51:28.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.581+0000 7f40d5e1e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40b800c670 con 0x563803995b60 2024-03-18T00:51:28.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.582+0000 7f40d5e1e740 10 client.0 _ll_register_callbacks cb 0x7f40c0005d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:28.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.583+0000 7f40d0fd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x5638039a35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.583+0000 7f40d0fd2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x5638039a35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:36536/0 (socket says 172.21.15.49:36536) 2024-03-18T00:51:28.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.583+0000 7f40d0fd2640 1 -- 172.21.15.49:0/1610941986 learned_addr learned my addr 172.21.15.49:0/1610941986 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:28.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.583+0000 7f40cbfff640 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x5638039a6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.583+0000 7f40cbfff640 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.585+0000 7f40d0fd2640 1 -- 172.21.15.49:0/1610941986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 msgr2=0x5638039a6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:28.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.585+0000 7f40d0fd2640 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x5638039a6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.585+0000 7f40d0fd2640 1 -- 172.21.15.49:0/1610941986 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 msgr2=0x5638039a3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:28.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.585+0000 7f40d0fd2640 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.586+0000 7f40d0fd2640 1 -- 172.21.15.49:0/1610941986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638038c19d0 con 0x563803995b60 2024-03-18T00:51:28.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.588+0000 7f40d0fd2640 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x5638039a35a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x563803991650 tx=0x7f40c400d420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4810e3367927e1c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:28.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.589+0000 7f40abfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:51:28.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.591+0000 7f40abfff640 1 -- 172.21.15.49:0/1610941986 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c401b070 con 0x563803995b60 2024-03-18T00:51:28.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.591+0000 7f40abfff640 1 -- 172.21.15.49:0/1610941986 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f40c4017cb0 con 0x563803995b60 2024-03-18T00:51:28.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.594+0000 7f40d5e1e740 1 -- 172.21.15.49:0/1610941986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563803901d90 con 0x563803995b60 2024-03-18T00:51:28.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.610+0000 7f40abfff640 1 -- 172.21.15.49:0/1610941986 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f40c4025050 con 0x563803995b60 2024-03-18T00:51:28.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.610+0000 7f40d5e1e740 10 client.25645 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:28.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.610+0000 7f40d5e1e740 1 -- 172.21.15.49:0/1610941986 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563803a10880 con 0x563803995b60 2024-03-18T00:51:28.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.613+0000 7f40abfff640 1 -- 172.21.15.49:0/1610941986 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563803901d90 con 0x563803995b60 2024-03-18T00:51:28.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.613+0000 7f40abfff640 1 -- 172.21.15.49:0/1610941986 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f40c4020a60 con 0x563803995b60 2024-03-18T00:51:28.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.614+0000 7f40d5e1e740 10 client.25645 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:28.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.614+0000 7f40d5e1e740 10 client.25645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:28.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.614+0000 7f40d5e1e740 20 client.25645 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:28.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.614+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563803a11240 0x563803a13660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:28.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.614+0000 7f40d5e1e740 4 client.25645 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:51:28.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.614+0000 7f40d5e1e740 1 -- 172.21.15.49:0/1610941986 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56380387f540 con 0x563803a11240 2024-03-18T00:51:28.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.616+0000 7f40cbfff640 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563803a11240 0x563803a13660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:28.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.617+0000 7f40abfff640 10 client.25645 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:28.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.617+0000 7f40cbfff640 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563803a11240 0x563803a13660 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2afebf00c99be09 server_cookie=ffbfb5c0076fe39e in_seq=0 out_seq=0 2024-03-18T00:51:28.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.622+0000 7f40abfff640 1 -- 172.21.15.49:0/1610941986 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x56380387f540 con 0x563803a11240 2024-03-18T00:51:28.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.622+0000 7f40abfff640 10 client.25645 handle_command_reply: tid=0 2024-03-18T00:51:28.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 client.25645 shutdown 2024-03-18T00:51:28.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 -- 172.21.15.49:0/1610941986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 msgr2=0x5638039a35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:28.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x5638039a35a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x563803991650 tx=0x7f40c400d420 comp rx=0 tx=0).stop 2024-03-18T00:51:28.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 -- 172.21.15.49:0/1610941986 shutdown_connections 2024-03-18T00:51:28.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563803a11240 0x563803a13660 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56380399acd0 0x5638039a3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563803995b60 0x5638039a35a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 --2- 172.21.15.49:0/1610941986 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563803995170 0x5638039a6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 -- 172.21.15.49:0/1610941986 >> 172.21.15.49:0/1610941986 conn(0x563803990e30 msgr2=0x5638039970f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:28.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40abfff640 0 client.25645 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:28.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 -- 172.21.15.49:0/1610941986 shutdown_connections 2024-03-18T00:51:28.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.623+0000 7f40d5e1e740 1 -- 172.21.15.49:0/1610941986 wait complete. 2024-03-18T00:51:28.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.624+0000 7f40d5e1e740 20 client.25645 trim_cache size 0 max 16384 2024-03-18T00:51:28.625 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:51:28.625 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-18T00:51:28.625 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:51:28.625 INFO:teuthology.orchestra.run.smithi049.stdout: "9e2e80bf-0c02-4da1-8be6-1e2499b149cf": { 2024-03-18T00:51:28.626 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:51:28.626 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "9e2e80bf-0c02-4da1-8be6-1e2499b149cf", 2024-03-18T00:51:28.626 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:51:28.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:51:28.626 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:51:28.626 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:51:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.625+0000 7f40d3236640 1 -- 172.21.15.49:0/1854709389 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f40b40a29d0 msgr2=0x7f40b40a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.625+0000 7f40d3236640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f40b40a29d0 0x7f40b40a4e90 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f40bc010fd0 tx=0x7f40bc0cf2a0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.625+0000 7f40d3236640 1 -- 172.21.15.49:0/1854709389 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40cc10db00 msgr2=0x7f40cc1ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.625+0000 7f40d3236640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40cc10db00 0x7f40cc1ae040 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00b3c0 tx=0x7f40bc00b880 comp rx=0 tx=0).stop 2024-03-18T00:51:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.626+0000 7f40d3236640 1 -- 172.21.15.49:0/1854709389 shutdown_connections 2024-03-18T00:51:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.626+0000 7f40d3236640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f40b40a29d0 0x7f40b40a4e90 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.626+0000 7f40d3236640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f40cc10db00 0x7f40cc1ae040 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.626+0000 7f40d3236640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f40cc10b150 0x7f40cc1a6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.626+0000 7f40d3236640 1 --2- 172.21.15.49:0/1854709389 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f40cc1087e0 0x7f40cc1a6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:28.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.626+0000 7f40d3236640 1 -- 172.21.15.49:0/1854709389 >> 172.21.15.49:0/1854709389 conn(0x7f40cc06e640 msgr2=0x7f40cc108c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:28.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.626+0000 7f40d3236640 1 -- 172.21.15.49:0/1854709389 shutdown_connections 2024-03-18T00:51:28.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:28.626+0000 7f40d3236640 1 -- 172.21.15.49:0/1854709389 wait complete. 2024-03-18T00:51:28.652 INFO:tasks.cephfs.filesystem:scrub status for tag:9e2e80bf-0c02-4da1-8be6-1e2499b149cf - {'path': '/', 'tag': '9e2e80bf-0c02-4da1-8be6-1e2499b149cf', 'options': 'recursive,force'} 2024-03-18T00:51:28.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:28 smithi049 ceph-mon[32130]: pgmap v391: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 942 MiB/s wr, 1.85k op/s 2024-03-18T00:51:28.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:51:28.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:51:28.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:28 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:51:28.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:28 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:51:28.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:28 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:51:28.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:28 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:51:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:28 smithi107 ceph-mon[36270]: pgmap v391: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 942 MiB/s wr, 1.85k op/s 2024-03-18T00:51:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:51:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:51:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:28 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:51:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:28 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:51:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:28 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:51:28.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:28 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:51:28.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:28 smithi106 ceph-mon[35899]: pgmap v391: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 942 MiB/s wr, 1.85k op/s 2024-03-18T00:51:28.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:51:28.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:51:28.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:28 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:51:28.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:28 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:51:28.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:28 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:51:28.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:28 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:51:29.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:29 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:51:29.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:29 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:51:29.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:29 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:51:29.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:29 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:51:29.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:29 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:51:29.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:29 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:51:30.888 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:30 smithi049 ceph-mon[32130]: pgmap v392: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 912 MiB/s wr, 2.01k op/s 2024-03-18T00:51:30.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:30 smithi107 ceph-mon[36270]: pgmap v392: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 912 MiB/s wr, 2.01k op/s 2024-03-18T00:51:30.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:30 smithi106 ceph-mon[35899]: pgmap v392: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 912 MiB/s wr, 2.01k op/s 2024-03-18T00:51:32.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:32 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:51:32.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:32 smithi049 ceph-mon[32130]: pgmap v393: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 819 MiB/s wr, 1.88k op/s 2024-03-18T00:51:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:32 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:51:32.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:32 smithi107 ceph-mon[36270]: pgmap v393: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 819 MiB/s wr, 1.88k op/s 2024-03-18T00:51:32.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:32 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:51:32.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:32 smithi106 ceph-mon[35899]: pgmap v393: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 819 MiB/s wr, 1.88k op/s 2024-03-18T00:51:34.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:34 smithi107 ceph-mon[36270]: pgmap v394: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 GiB/s wr, 2.31k op/s 2024-03-18T00:51:34.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:34 smithi106 ceph-mon[35899]: pgmap v394: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 GiB/s wr, 2.31k op/s 2024-03-18T00:51:35.128 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:34 smithi049 ceph-mon[32130]: pgmap v394: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 GiB/s wr, 2.31k op/s 2024-03-18T00:51:35.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:35 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:51:35.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:35 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:51:35.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:35 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:51:35.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:35 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:51:36.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:35 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:51:36.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:35 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:51:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:36 smithi107 ceph-mon[36270]: pgmap v395: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 790 MiB/s wr, 1.75k op/s 2024-03-18T00:51:36.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:36 smithi106 ceph-mon[35899]: pgmap v395: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 790 MiB/s wr, 1.75k op/s 2024-03-18T00:51:37.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:36 smithi049 ceph-mon[32130]: pgmap v395: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 790 MiB/s wr, 1.75k op/s 2024-03-18T00:51:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad74113640 1 -- 172.21.15.49:0/384154296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c10deb0 msgr2=0x7fad6c110370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:38.901 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad70ead640 1 -- 172.21.15.49:0/384154296 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad68037020 con 0x7fad6c10deb0 2024-03-18T00:51:38.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad74113640 1 --2- 172.21.15.49:0/384154296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c10deb0 0x7fad6c110370 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fad6800b3f0 tx=0x7fad6802c4c0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad74113640 1 -- 172.21.15.49:0/384154296 shutdown_connections 2024-03-18T00:51:38.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad74113640 1 --2- 172.21.15.49:0/384154296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad6c10b410 0x7fad6c10d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad74113640 1 --2- 172.21.15.49:0/384154296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c10deb0 0x7fad6c110370 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad74113640 1 --2- 172.21.15.49:0/384154296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad6c108a50 0x7fad6c10ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad74113640 1 -- 172.21.15.49:0/384154296 >> 172.21.15.49:0/384154296 conn(0x7fad6c06e9e0 msgr2=0x7fad6c070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad74113640 1 -- 172.21.15.49:0/384154296 shutdown_connections 2024-03-18T00:51:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.901+0000 7fad74113640 1 -- 172.21.15.49:0/384154296 wait complete. 2024-03-18T00:51:38.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.902+0000 7fad74113640 1 Processor -- start 2024-03-18T00:51:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad74113640 1 -- start start 2024-03-18T00:51:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad74113640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c108a50 0x7fad6c19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad74113640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad6c10b410 0x7fad6c19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad74113640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad6c10deb0 0x7fad6c1a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad74113640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c113b40 con 0x7fad6c108a50 2024-03-18T00:51:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad74113640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c1139c0 con 0x7fad6c10b410 2024-03-18T00:51:38.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad74113640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad6c113cc0 con 0x7fad6c10deb0 2024-03-18T00:51:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad71eaf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c108a50 0x7fad6c19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad71eaf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c108a50 0x7fad6c19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33486/0 (socket says 172.21.15.49:33486) 2024-03-18T00:51:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.903+0000 7fad71eaf640 1 -- 172.21.15.49:0/2912203752 learned_addr learned my addr 172.21.15.49:0/2912203752 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.904+0000 7fad71eaf640 1 -- 172.21.15.49:0/2912203752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad6c10deb0 msgr2=0x7fad6c1a3580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:51:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.904+0000 7fad71eaf640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad6c10deb0 0x7fad6c1a3580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.904+0000 7fad71eaf640 1 -- 172.21.15.49:0/2912203752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad6c10b410 msgr2=0x7fad6c19e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.904+0000 7fad71eaf640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad6c10b410 0x7fad6c19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.904+0000 7fad71eaf640 1 -- 172.21.15.49:0/2912203752 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6c1a3bf0 con 0x7fad6c108a50 2024-03-18T00:51:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.904+0000 7fad71eaf640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c108a50 0x7fad6c19dfe0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fad6400ba50 tx=0x7fad6400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.904+0000 7fad62ffd640 1 -- 172.21.15.49:0/2912203752 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad64019070 con 0x7fad6c108a50 2024-03-18T00:51:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.905+0000 7fad74113640 1 -- 172.21.15.49:0/2912203752 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad6c1a3ee0 con 0x7fad6c108a50 2024-03-18T00:51:38.906 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.905+0000 7fad74113640 1 -- 172.21.15.49:0/2912203752 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad6c1a4430 con 0x7fad6c108a50 2024-03-18T00:51:38.907 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.905+0000 7fad62ffd640 1 -- 172.21.15.49:0/2912203752 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad6400cbe0 con 0x7fad6c108a50 2024-03-18T00:51:38.907 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.905+0000 7fad62ffd640 1 -- 172.21.15.49:0/2912203752 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad640076e0 con 0x7fad6c108a50 2024-03-18T00:51:38.908 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.908+0000 7fad76dcf740 1 Processor -- start 2024-03-18T00:51:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.908+0000 7fad76dcf740 1 -- start start 2024-03-18T00:51:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.908+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523b340 0x55d38523b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.908+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523bd30 0x55d385240830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d385240ea0 0x55d385243330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad76dcf740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d3851a0a40 con 0x55d385240ea0 2024-03-18T00:51:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad76dcf740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d385149920 con 0x55d38523bd30 2024-03-18T00:51:38.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad76dcf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d38519cae0 con 0x55d38523b340 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad716ae640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d385240ea0 0x55d385243330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad716ae640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d385240ea0 0x55d385243330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33488/0 (socket says 172.21.15.49:33488) 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad716ae640 1 -- 172.21.15.49:0/411141325 learned_addr learned my addr 172.21.15.49:0/411141325 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad716ae640 1 -- 172.21.15.49:0/411141325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523b340 msgr2=0x55d38523b760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad716ae640 1 --2- 172.21.15.49:0/411141325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523b340 0x55d38523b760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad716ae640 1 -- 172.21.15.49:0/411141325 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523bd30 msgr2=0x55d385240830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad716ae640 1 --2- 172.21.15.49:0/411141325 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523bd30 0x55d385240830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.909+0000 7fad716ae640 1 -- 172.21.15.49:0/411141325 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d385161460 con 0x55d385240ea0 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.910+0000 7fad62ffd640 1 -- 172.21.15.49:0/2912203752 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fad6401c070 con 0x7fad6c108a50 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.910+0000 7fad716ae640 1 --2- 172.21.15.49:0/411141325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d385240ea0 0x55d385243330 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00aa30 tx=0x7fad5c00aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec38da80e042f0bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.910+0000 7fad57fff640 1 -- 172.21.15.49:0/411141325 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c00ce40 con 0x55d385240ea0 2024-03-18T00:51:38.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.910+0000 7fad57fff640 1 -- 172.21.15.49:0/411141325 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad5c00f930 con 0x55d385240ea0 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.910+0000 7fad57fff640 1 -- 172.21.15.49:0/411141325 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c018c30 con 0x55d385240ea0 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad62ffd640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad580a2900 0x7fad580a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad76dcf740 1 -- 172.21.15.49:0/411141325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d385240ea0 msgr2=0x55d385243330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad76dcf740 1 --2- 172.21.15.49:0/411141325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d385240ea0 0x55d385243330 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00aa30 tx=0x7fad5c00aef0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad62ffd640 1 -- 172.21.15.49:0/2912203752 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fad640ce0c0 con 0x7fad6c108a50 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad726b0640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad580a2900 0x7fad580a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad76dcf740 1 -- 172.21.15.49:0/411141325 shutdown_connections 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad76dcf740 1 --2- 172.21.15.49:0/411141325 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523bd30 0x55d385240830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad76dcf740 1 --2- 172.21.15.49:0/411141325 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d385240ea0 0x55d385243330 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad76dcf740 1 --2- 172.21.15.49:0/411141325 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523b340 0x55d38523b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.911+0000 7fad76dcf740 1 -- 172.21.15.49:0/411141325 >> 172.21.15.49:0/411141325 conn(0x55d3852368b0 msgr2=0x55d385236cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.912+0000 7fad76dcf740 1 -- 172.21.15.49:0/411141325 shutdown_connections 2024-03-18T00:51:38.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.912+0000 7fad76dcf740 1 -- 172.21.15.49:0/411141325 wait complete. 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.912+0000 7fad726b0640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad580a2900 0x7fad580a4dc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55d38523c590 tx=0x7fad680334f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.912+0000 7fad76dcf740 1 -- start start 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.912+0000 7fad76dcf740 1 Processor -- start 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523b340 0x55d3852456d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523bd30 0x55d385245c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad71eaf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523b340 0x55d3852456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad71eaf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523b340 0x55d3852456d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33494/0 (socket says 172.21.15.49:33494) 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad71eaf640 1 -- 172.21.15.49:0/1789084156 learned_addr learned my addr 172.21.15.49:0/1789084156 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d385240ea0 0x55d38524cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d385243e50 con 0x55d38523b340 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d38515cd70 con 0x55d385240ea0 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d385243fd0 con 0x55d38523bd30 2024-03-18T00:51:38.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.913+0000 7fad76dcf740 10 client.0 _ll_register_callbacks cb 0x55d384eb4a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad71eaf640 1 -- 172.21.15.49:0/1789084156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523bd30 msgr2=0x55d385245c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad71eaf640 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523bd30 0x55d385245c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad71eaf640 1 -- 172.21.15.49:0/1789084156 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d385240ea0 msgr2=0x55d38524cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad71eaf640 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d385240ea0 0x55d38524cc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad71eaf640 1 -- 172.21.15.49:0/1789084156 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d384eb5210 con 0x55d38523b340 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad71eaf640 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523b340 0x55d3852456d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fad640a2cb0 tx=0x7fad6409ab00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77101ac7a860aa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad557fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad557fa640 1 -- 172.21.15.49:0/1789084156 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad640a2d50 con 0x55d38523b340 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad5c00f930 con 0x55d38523b340 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.914+0000 7fad557fa640 1 -- 172.21.15.49:0/1789084156 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fad640a2ef0 con 0x55d38523b340 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.915+0000 7fad557fa640 1 -- 172.21.15.49:0/1789084156 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fad64099e90 con 0x55d38523b340 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.915+0000 7fad76dcf740 10 client.15861 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:38.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.915+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d384f81620 con 0x55d38523b340 2024-03-18T00:51:38.921 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.915+0000 7fad557fa640 1 -- 172.21.15.49:0/1789084156 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad5c00f930 con 0x55d38523b340 2024-03-18T00:51:38.921 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.916+0000 7fad557fa640 1 -- 172.21.15.49:0/1789084156 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fad64096930 con 0x55d38523b340 2024-03-18T00:51:38.921 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.916+0000 7fad76dcf740 10 client.15861 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:38.921 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.916+0000 7fad76dcf740 10 client.15861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:38.921 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.916+0000 7fad76dcf740 20 client.15861 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:38.922 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.916+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d385251590 0x55d385253a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.922 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.916+0000 7fad76dcf740 4 client.15861 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:38.922 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.916+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3851adf80 con 0x55d385251590 2024-03-18T00:51:38.922 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.917+0000 7fad716ae640 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d385251590 0x55d385253a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.922 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.919+0000 7fad716ae640 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d385251590 0x55d385253a20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64ad924f6c0c6759 server_cookie=c96ae2966adbe219 in_seq=0 out_seq=0 2024-03-18T00:51:38.922 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.919+0000 7fad557fa640 10 client.15861 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.924+0000 7fad557fa640 1 -- 172.21.15.49:0/1789084156 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d3851adf80 con 0x55d385251590 2024-03-18T00:51:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.924+0000 7fad557fa640 10 client.15861 handle_command_reply: tid=0 2024-03-18T00:51:38.924 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.924+0000 7fad76dcf740 1 client.15861 shutdown 2024-03-18T00:51:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.924+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523b340 msgr2=0x55d3852456d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.924+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523b340 0x55d3852456d0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fad640a2cb0 tx=0x7fad6409ab00 comp rx=0 tx=0).stop 2024-03-18T00:51:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 shutdown_connections 2024-03-18T00:51:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d385251590 0x55d385253a20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d385240ea0 0x55d38524cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523bd30 0x55d385245c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1789084156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523b340 0x55d3852456d0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 >> 172.21.15.49:0/1789084156 conn(0x55d3852368b0 msgr2=0x55d385235bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:38.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad557fa640 0 client.15861 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:38.926 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 shutdown_connections 2024-03-18T00:51:38.926 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad76dcf740 1 -- 172.21.15.49:0/1789084156 wait complete. 2024-03-18T00:51:38.926 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.925+0000 7fad76dcf740 20 client.15861 trim_cache size 0 max 16384 2024-03-18T00:51:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:38 smithi107 ceph-mon[36270]: pgmap v396: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 955 MiB/s wr, 2.03k op/s 2024-03-18T00:51:38.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:38 smithi106 ceph-mon[35899]: pgmap v396: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 955 MiB/s wr, 2.03k op/s 2024-03-18T00:51:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.954+0000 7fad76dcf740 1 Processor -- start 2024-03-18T00:51:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.954+0000 7fad76dcf740 1 -- start start 2024-03-18T00:51:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.954+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d3852368b0 0x55d3852452e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.954+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523b340 0x55d385245820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.954+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523bd30 0x55d385258320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.954 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.954+0000 7fad76dcf740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fad5000e530 con 0x55d3852368b0 2024-03-18T00:51:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.954+0000 7fad76dcf740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fad500100a0 con 0x55d38523b340 2024-03-18T00:51:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.954+0000 7fad76dcf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad5000ec00 con 0x55d38523bd30 2024-03-18T00:51:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.955+0000 7fad716ae640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523b340 0x55d385245820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.955+0000 7fad716ae640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523b340 0x55d385245820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:34504/0 (socket says 172.21.15.49:34504) 2024-03-18T00:51:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.955+0000 7fad716ae640 1 -- 172.21.15.49:0/1180975527 learned_addr learned my addr 172.21.15.49:0/1180975527 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:38.955 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.955+0000 7fad71eaf640 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d3852368b0 0x55d3852452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.956+0000 7fad726b0640 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523bd30 0x55d385258320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.956+0000 7fad71eaf640 1 -- 172.21.15.49:0/1180975527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523bd30 msgr2=0x55d385258320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:38.956 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.956+0000 7fad71eaf640 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523bd30 0x55d385258320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.956+0000 7fad71eaf640 1 -- 172.21.15.49:0/1180975527 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523b340 msgr2=0x55d385245820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.957+0000 7fad71eaf640 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523b340 0x55d385245820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.957+0000 7fad71eaf640 1 -- 172.21.15.49:0/1180975527 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d385232310 con 0x55d3852368b0 2024-03-18T00:51:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.957+0000 7fad716ae640 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523b340 0x55d385245820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:51:38.957 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.957+0000 7fad71eaf640 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d3852368b0 0x55d3852452e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fad64097410 tx=0x7fad640a3360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8365bf858c6cff6a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.958+0000 7fad577fe640 1 -- 172.21.15.49:0/1180975527 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6409eb90 con 0x55d3852368b0 2024-03-18T00:51:38.958 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.959+0000 7fad577fe640 1 -- 172.21.15.49:0/1180975527 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad64099b90 con 0x55d3852368b0 2024-03-18T00:51:38.959 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.959+0000 7fad577fe640 1 -- 172.21.15.49:0/1180975527 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad640a2fb0 con 0x55d3852368b0 2024-03-18T00:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.960+0000 7fad76dcf740 1 -- 172.21.15.49:0/1180975527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d3852368b0 msgr2=0x55d3852452e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.960+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d3852368b0 0x55d3852452e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fad64097410 tx=0x7fad640a3360 comp rx=0 tx=0).stop 2024-03-18T00:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.960+0000 7fad76dcf740 1 -- 172.21.15.49:0/1180975527 shutdown_connections 2024-03-18T00:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.960+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d38523b340 0x55d385245820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.960+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523bd30 0x55d385258320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.960+0000 7fad76dcf740 1 --2- 172.21.15.49:0/1180975527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d3852368b0 0x55d3852452e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.960+0000 7fad76dcf740 1 -- 172.21.15.49:0/1180975527 >> 172.21.15.49:0/1180975527 conn(0x55d3852409e0 msgr2=0x55d38523d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.960+0000 7fad76dcf740 1 -- 172.21.15.49:0/1180975527 shutdown_connections 2024-03-18T00:51:38.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.960+0000 7fad76dcf740 1 -- 172.21.15.49:0/1180975527 wait complete. 2024-03-18T00:51:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.961+0000 7fad76dcf740 1 -- start start 2024-03-18T00:51:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.961+0000 7fad76dcf740 1 Processor -- start 2024-03-18T00:51:38.961 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.961+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d3852368b0 0x55d38524a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.961+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523b340 0x55d38524acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.961+0000 7fad76dcf740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523bd30 0x55d38524bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad76dcf740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fad5000f9a0 con 0x55d38523bd30 2024-03-18T00:51:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad76dcf740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fad5000f2d0 con 0x55d3852368b0 2024-03-18T00:51:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad76dcf740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad5000c510 con 0x55d38523b340 2024-03-18T00:51:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad76dcf740 10 client.0 _ll_register_callbacks cb 0x7fad64096ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad726b0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523bd30 0x55d38524bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad726b0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523bd30 0x55d38524bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33516/0 (socket says 172.21.15.49:33516) 2024-03-18T00:51:38.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad726b0640 1 -- 172.21.15.49:0/3247917277 learned_addr learned my addr 172.21.15.49:0/3247917277 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad716ae640 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523b340 0x55d38524acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad726b0640 1 -- 172.21.15.49:0/3247917277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523b340 msgr2=0x55d38524acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad726b0640 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523b340 0x55d38524acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad726b0640 1 -- 172.21.15.49:0/3247917277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d3852368b0 msgr2=0x55d38524a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:38.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad726b0640 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d3852368b0 0x55d38524a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.962+0000 7fad726b0640 1 -- 172.21.15.49:0/3247917277 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d38524c2b0 con 0x55d38523bd30 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.963+0000 7fad726b0640 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523bd30 0x55d38524bcc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fad680045e0 tx=0x7fad6802c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971cd5f06fd24f91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.963+0000 7fad557fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.963+0000 7fad557fa640 1 -- 172.21.15.49:0/3247917277 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6804e070 con 0x55d38523bd30 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.963+0000 7fad76dcf740 1 -- 172.21.15.49:0/3247917277 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad64099b90 con 0x55d38523bd30 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.963+0000 7fad557fa640 1 -- 172.21.15.49:0/3247917277 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fad68003a00 con 0x55d38523bd30 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.963+0000 7fad557fa640 1 -- 172.21.15.49:0/3247917277 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fad68037040 con 0x55d38523bd30 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.963+0000 7fad76dcf740 10 client.15867 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.963+0000 7fad76dcf740 1 -- 172.21.15.49:0/3247917277 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d3852b8170 con 0x55d38523bd30 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.964+0000 7fad557fa640 1 -- 172.21.15.49:0/3247917277 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad64099b90 con 0x55d38523bd30 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.964+0000 7fad557fa640 1 -- 172.21.15.49:0/3247917277 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fad6804a420 con 0x55d38523bd30 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.964+0000 7fad76dcf740 10 client.15867 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:38.965 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.964+0000 7fad76dcf740 10 client.15867 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.964+0000 7fad76dcf740 20 client.15867 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.964+0000 7fad76dcf740 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d385240ea0 0x55d3852babf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.964+0000 7fad76dcf740 4 client.15867 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:51:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.964+0000 7fad76dcf740 1 -- 172.21.15.49:0/3247917277 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d3851a82d0 con 0x55d385240ea0 2024-03-18T00:51:38.966 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.965+0000 7fad71eaf640 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d385240ea0 0x55d3852babf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.969+0000 7fad71eaf640 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d385240ea0 0x55d3852babf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d049a721e609f7 server_cookie=7134408ef62592b3 in_seq=0 out_seq=0 2024-03-18T00:51:38.969 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:38.969+0000 7fad557fa640 10 client.15867 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:39.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:38 smithi049 ceph-mon[32130]: pgmap v396: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 955 MiB/s wr, 2.03k op/s 2024-03-18T00:51:40.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:40 smithi106 ceph-mon[35899]: pgmap v397: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 923 MiB/s wr, 1.93k op/s 2024-03-18T00:51:41.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:40 smithi049 ceph-mon[32130]: pgmap v397: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 923 MiB/s wr, 1.93k op/s 2024-03-18T00:51:41.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:40 smithi107 ceph-mon[36270]: pgmap v397: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 923 MiB/s wr, 1.93k op/s 2024-03-18T00:51:42.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:42 smithi106 ceph-mon[35899]: pgmap v398: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 822 MiB/s wr, 1.56k op/s 2024-03-18T00:51:43.128 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:42 smithi049 ceph-mon[32130]: pgmap v398: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 822 MiB/s wr, 1.56k op/s 2024-03-18T00:51:43.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:42 smithi107 ceph-mon[36270]: pgmap v398: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 822 MiB/s wr, 1.56k op/s 2024-03-18T00:51:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad557fa640 1 -- 172.21.15.49:0/3247917277 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d3851a82d0 con 0x55d385240ea0 2024-03-18T00:51:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad557fa640 10 client.15867 handle_command_reply: tid=0 2024-03-18T00:51:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad76dcf740 1 client.15867 shutdown 2024-03-18T00:51:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad76dcf740 1 -- 172.21.15.49:0/3247917277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523bd30 msgr2=0x55d38524bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad76dcf740 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523bd30 0x55d38524bcc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fad680045e0 tx=0x7fad6802c4c0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad76dcf740 1 -- 172.21.15.49:0/3247917277 shutdown_connections 2024-03-18T00:51:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad76dcf740 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d385240ea0 0x55d3852babf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad76dcf740 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d38523bd30 0x55d38524bcc0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad76dcf740 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d38523b340 0x55d38524acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad76dcf740 1 --2- 172.21.15.49:0/3247917277 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d3852368b0 0x55d38524a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.933+0000 7fad76dcf740 1 -- 172.21.15.49:0/3247917277 >> 172.21.15.49:0/3247917277 conn(0x55d3852409e0 msgr2=0x55d3852452c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.934+0000 7fad557fa640 0 client.15867 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.934+0000 7fad76dcf740 1 -- 172.21.15.49:0/3247917277 shutdown_connections 2024-03-18T00:51:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.934+0000 7fad76dcf740 1 -- 172.21.15.49:0/3247917277 wait complete. 2024-03-18T00:51:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.934+0000 7fad76dcf740 20 client.15867 trim_cache size 0 max 16384 2024-03-18T00:51:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:51:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:51:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:51:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.934+0000 7fad74113640 1 -- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad580a2900 msgr2=0x7fad580a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.934+0000 7fad74113640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad580a2900 0x7fad580a4dc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x55d38523c590 tx=0x7fad680334f0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.935+0000 7fad74113640 1 -- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c108a50 msgr2=0x7fad6c19dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.935+0000 7fad74113640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c108a50 0x7fad6c19dfe0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fad6400ba50 tx=0x7fad6400bf10 comp rx=0 tx=0).stop 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.935+0000 7fad74113640 1 -- 172.21.15.49:0/2912203752 shutdown_connections 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.935+0000 7fad74113640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad580a2900 0x7fad580a4dc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.935+0000 7fad74113640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad6c10b410 0x7fad6c19e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.935+0000 7fad74113640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad6c10deb0 0x7fad6c1a3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.935+0000 7fad74113640 1 --2- 172.21.15.49:0/2912203752 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad6c108a50 0x7fad6c19dfe0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.935+0000 7fad74113640 1 -- 172.21.15.49:0/2912203752 >> 172.21.15.49:0/2912203752 conn(0x7fad6c06e9e0 msgr2=0x7fad6c109390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.935+0000 7fad74113640 1 -- 172.21.15.49:0/2912203752 shutdown_connections 2024-03-18T00:51:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:43.936+0000 7fad74113640 1 -- 172.21.15.49:0/2912203752 wait complete. 2024-03-18T00:51:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:44 smithi049 ceph-mon[32130]: pgmap v399: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 2.05k op/s 2024-03-18T00:51:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:44 smithi107 ceph-mon[36270]: pgmap v399: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 2.05k op/s 2024-03-18T00:51:45.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:44 smithi106 ceph-mon[35899]: pgmap v399: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 2.05k op/s 2024-03-18T00:51:46.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:46 smithi107 ceph-mon[36270]: pgmap v400: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 798 MiB/s wr, 1.59k op/s 2024-03-18T00:51:47.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:46 smithi049 ceph-mon[32130]: pgmap v400: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 798 MiB/s wr, 1.59k op/s 2024-03-18T00:51:47.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:46 smithi106 ceph-mon[35899]: pgmap v400: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 798 MiB/s wr, 1.59k op/s 2024-03-18T00:51:49.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:48 smithi049 ceph-mon[32130]: pgmap v401: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 948 MiB/s wr, 2.17k op/s 2024-03-18T00:51:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:48 smithi107 ceph-mon[36270]: pgmap v401: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 948 MiB/s wr, 2.17k op/s 2024-03-18T00:51:49.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:48 smithi106 ceph-mon[35899]: pgmap v401: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 948 MiB/s wr, 2.17k op/s 2024-03-18T00:51:50.997 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:50 smithi106 ceph-mon[35899]: pgmap v402: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 913 MiB/s wr, 2.41k op/s 2024-03-18T00:51:51.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:50 smithi049 ceph-mon[32130]: pgmap v402: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 913 MiB/s wr, 2.41k op/s 2024-03-18T00:51:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:50 smithi107 ceph-mon[36270]: pgmap v402: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 913 MiB/s wr, 2.41k op/s 2024-03-18T00:51:53.132 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:52 smithi049 ceph-mon[32130]: pgmap v403: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 820 MiB/s wr, 2.26k op/s 2024-03-18T00:51:53.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:52 smithi107 ceph-mon[36270]: pgmap v403: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 820 MiB/s wr, 2.26k op/s 2024-03-18T00:51:53.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:52 smithi106 ceph-mon[35899]: pgmap v403: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 820 MiB/s wr, 2.26k op/s 2024-03-18T00:51:55.130 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:54 smithi049 ceph-mon[32130]: pgmap v404: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 GiB/s wr, 2.66k op/s 2024-03-18T00:51:55.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:54 smithi107 ceph-mon[36270]: pgmap v404: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 GiB/s wr, 2.66k op/s 2024-03-18T00:51:55.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:54 smithi106 ceph-mon[35899]: pgmap v404: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 GiB/s wr, 2.66k op/s 2024-03-18T00:51:57.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:56 smithi049 ceph-mon[32130]: pgmap v405: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 795 MiB/s wr, 2.05k op/s 2024-03-18T00:51:57.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:56 smithi107 ceph-mon[36270]: pgmap v405: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 795 MiB/s wr, 2.05k op/s 2024-03-18T00:51:57.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:56 smithi106 ceph-mon[35899]: pgmap v405: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 795 MiB/s wr, 2.05k op/s 2024-03-18T00:51:57.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.306+0000 7f78aace8640 1 -- 172.21.15.49:0/1712324281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a410c220 msgr2=0x7f78a410e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:57.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.306+0000 7f78aace8640 1 --2- 172.21.15.49:0/1712324281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a410c220 0x7f78a410e610 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f788c009900 tx=0x7f788c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:51:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.308+0000 7f78aace8640 1 -- 172.21.15.49:0/1712324281 shutdown_connections 2024-03-18T00:51:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.308+0000 7f78aace8640 1 --2- 172.21.15.49:0/1712324281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a410eba0 0x7f78a4111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.308+0000 7f78aace8640 1 --2- 172.21.15.49:0/1712324281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a410c220 0x7f78a410e610 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.308 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.308+0000 7f78aace8640 1 --2- 172.21.15.49:0/1712324281 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78a41098f0 0x7f78a410bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.308+0000 7f78aace8640 1 -- 172.21.15.49:0/1712324281 >> 172.21.15.49:0/1712324281 conn(0x7f78a406e820 msgr2=0x7f78a4070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.311+0000 7f78aace8640 1 -- 172.21.15.49:0/1712324281 shutdown_connections 2024-03-18T00:51:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.311+0000 7f78aace8640 1 -- 172.21.15.49:0/1712324281 wait complete. 2024-03-18T00:51:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.313+0000 7f78aace8640 1 Processor -- start 2024-03-18T00:51:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.313+0000 7f78aace8640 1 -- start start 2024-03-18T00:51:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.313+0000 7f78aace8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a41098f0 0x7f78a4109360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.313+0000 7f78aace8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78a410c220 0x7f78a41a73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.313+0000 7f78aace8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a410eba0 0x7f78a41a9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.313+0000 7f78aace8640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4114c40 con 0x7f78a41098f0 2024-03-18T00:51:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.313+0000 7f78aace8640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4114ac0 con 0x7f78a410c220 2024-03-18T00:51:57.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.313+0000 7f78aace8640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4114dc0 con 0x7f78a410eba0 2024-03-18T00:51:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.314+0000 7f78a8a84640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a41098f0 0x7f78a4109360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.314+0000 7f78a8a84640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a41098f0 0x7f78a4109360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52092/0 (socket says 172.21.15.49:52092) 2024-03-18T00:51:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.314+0000 7f78a8a84640 1 -- 172.21.15.49:0/4040940068 learned_addr learned my addr 172.21.15.49:0/4040940068 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.314+0000 7f78a3fff640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78a410c220 0x7f78a41a73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.315+0000 7f78a8a84640 1 -- 172.21.15.49:0/4040940068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a410eba0 msgr2=0x7f78a41a9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.315+0000 7f78a8a84640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a410eba0 0x7f78a41a9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.315+0000 7f78a8a84640 1 -- 172.21.15.49:0/4040940068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78a410c220 msgr2=0x7f78a41a73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:57.315 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.315+0000 7f78a8a84640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78a410c220 0x7f78a41a73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.316 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.315+0000 7f78a8a84640 1 -- 172.21.15.49:0/4040940068 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a41aa0e0 con 0x7f78a41098f0 2024-03-18T00:51:57.316 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.315+0000 7f78a8a84640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a41098f0 0x7f78a4109360 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f789c00d950 tx=0x7f789c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.321+0000 7f78a1ffb640 1 -- 172.21.15.49:0/4040940068 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c017070 con 0x7f78a41098f0 2024-03-18T00:51:57.322 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.321+0000 7f78aace8640 1 -- 172.21.15.49:0/4040940068 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a41aa3d0 con 0x7f78a41098f0 2024-03-18T00:51:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.321+0000 7f78aace8640 1 -- 172.21.15.49:0/4040940068 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a41aa940 con 0x7f78a41098f0 2024-03-18T00:51:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.323+0000 7f78a1ffb640 1 -- 172.21.15.49:0/4040940068 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c009c30 con 0x7f78a41098f0 2024-03-18T00:51:57.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.323+0000 7f78a1ffb640 1 -- 172.21.15.49:0/4040940068 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c00f6b0 con 0x7f78a41098f0 2024-03-18T00:51:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.328+0000 7f78ad9ab740 1 Processor -- start 2024-03-18T00:51:57.331 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.331+0000 7f78ad9ab740 1 -- start start 2024-03-18T00:51:57.331 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.331+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0cb50 0x558ed5a0cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.331 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.331+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a12040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.331 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.331+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a14b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.331 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.331+0000 7f78ad9ab740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558ed5969a40 con 0x558ed5a126b0 2024-03-18T00:51:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.331+0000 7f78ad9ab740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558ed5912920 con 0x558ed5a0cb50 2024-03-18T00:51:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.331+0000 7f78ad9ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558ed5965ae0 con 0x558ed5a0d540 2024-03-18T00:51:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.332+0000 7f78a8a84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a12040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.332+0000 7f78a3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a14b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.332+0000 7f78a8a84640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a12040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44788/0 (socket says 172.21.15.49:44788) 2024-03-18T00:51:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.332+0000 7f78a8a84640 1 -- 172.21.15.49:0/3341788560 learned_addr learned my addr 172.21.15.49:0/3341788560 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.333+0000 7f78a8a84640 1 -- 172.21.15.49:0/3341788560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0cb50 msgr2=0x558ed5a0cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:51:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.333+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3341788560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0cb50 0x558ed5a0cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.333+0000 7f78a8a84640 1 -- 172.21.15.49:0/3341788560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 msgr2=0x558ed5a14b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.333+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3341788560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a14b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.333+0000 7f78a8a84640 1 -- 172.21.15.49:0/3341788560 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ed592a460 con 0x558ed5a0d540 2024-03-18T00:51:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.333+0000 7f78a3fff640 1 --2- 172.21.15.49:0/3341788560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a14b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:51:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.334+0000 7f78a1ffb640 1 -- 172.21.15.49:0/4040940068 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f789c0109d0 con 0x7f78a41098f0 2024-03-18T00:51:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.334+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3341788560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a12040 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f789c0d2bd0 tx=0x7f789c0d2cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe4b03cee4a5ddd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:57.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.337+0000 7f78a1ffb640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78900a29f0 0x7f78900a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.337 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.338+0000 7f78a9285640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78900a29f0 0x7f78900a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.338 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.338+0000 7f78a1ffb640 1 -- 172.21.15.49:0/4040940068 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f789c0cea60 con 0x7f78a41098f0 2024-03-18T00:51:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.341+0000 7f78a9285640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78900a29f0 0x7f78900a4eb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x558ed5a0dda0 tx=0x7f7894009250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:57.342 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.339+0000 7f789affd640 1 -- 172.21.15.49:0/3341788560 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c0e2070 con 0x558ed5a0d540 2024-03-18T00:51:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.341+0000 7f789affd640 1 -- 172.21.15.49:0/3341788560 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c0dd440 con 0x558ed5a0d540 2024-03-18T00:51:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.341+0000 7f789affd640 1 -- 172.21.15.49:0/3341788560 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c0dc600 con 0x558ed5a0d540 2024-03-18T00:51:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.343+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3341788560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 msgr2=0x558ed5a12040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.343+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3341788560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a12040 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f789c0d2bd0 tx=0x7f789c0d2cc0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.343+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3341788560 shutdown_connections 2024-03-18T00:51:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.343+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3341788560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a14b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.343+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3341788560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a12040 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.343+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3341788560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0cb50 0x558ed5a0cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.343+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3341788560 >> 172.21.15.49:0/3341788560 conn(0x558ed59ff8b0 msgr2=0x558ed59ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:57.345 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.345+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3341788560 shutdown_connections 2024-03-18T00:51:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.345+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3341788560 wait complete. 2024-03-18T00:51:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.346+0000 7f78ad9ab740 1 -- start start 2024-03-18T00:51:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.346+0000 7f78ad9ab740 1 Processor -- start 2024-03-18T00:51:57.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.347+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0cb50 0x558ed5a05d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.347+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0d540 0x558ed5a06290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.347+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a0b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.347+0000 7f78ad9ab740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558ed5a04440 con 0x558ed5a126b0 2024-03-18T00:51:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.347+0000 7f78ad9ab740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558ed5925d70 con 0x558ed5a0d540 2024-03-18T00:51:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.347+0000 7f78ad9ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558ed5a045c0 con 0x558ed5a0cb50 2024-03-18T00:51:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.347+0000 7f78ad9ab740 10 client.0 _ll_register_callbacks cb 0x558ed567da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.349+0000 7f78a3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0d540 0x558ed5a06290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.349+0000 7f78a9285640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a0b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.349+0000 7f78a9285640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a0b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52110/0 (socket says 172.21.15.49:52110) 2024-03-18T00:51:57.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.349+0000 7f78a9285640 1 -- 172.21.15.49:0/3863578062 learned_addr learned my addr 172.21.15.49:0/3863578062 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:57.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.351+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0cb50 0x558ed5a05d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.351+0000 7f78a3fff640 1 -- 172.21.15.49:0/3863578062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0cb50 msgr2=0x558ed5a05d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.351+0000 7f78a3fff640 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0cb50 0x558ed5a05d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.351+0000 7f78a3fff640 1 -- 172.21.15.49:0/3863578062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 msgr2=0x558ed5a0b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.351+0000 7f78a3fff640 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a0b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.352+0000 7f78a3fff640 1 -- 172.21.15.49:0/3863578062 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ed567e210 con 0x558ed5a0d540 2024-03-18T00:51:57.356 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.355+0000 7f78a3fff640 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0d540 0x558ed5a06290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f788c009850 tx=0x7f788c02cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2b550a550b17960 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:57.357 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.355+0000 7f7883fff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:51:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.358+0000 7f7883fff640 1 -- 172.21.15.49:0/3863578062 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c02cca0 con 0x558ed5a0d540 2024-03-18T00:51:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.358+0000 7f7883fff640 1 -- 172.21.15.49:0/3863578062 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f788c005210 con 0x558ed5a0d540 2024-03-18T00:51:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.359+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3863578062 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f789c0dd440 con 0x558ed5a0d540 2024-03-18T00:51:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.366+0000 7f7883fff640 1 -- 172.21.15.49:0/3863578062 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f788c005430 con 0x558ed5a0d540 2024-03-18T00:51:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.366+0000 7f78ad9ab740 10 client.25610 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.366+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3863578062 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558ed574a620 con 0x558ed5a0d540 2024-03-18T00:51:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.372+0000 7f7883fff640 1 -- 172.21.15.49:0/3863578062 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c0dd440 con 0x558ed5a0d540 2024-03-18T00:51:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.372+0000 7f7883fff640 1 -- 172.21.15.49:0/3863578062 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f788c0038a0 con 0x558ed5a0d540 2024-03-18T00:51:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.372+0000 7f78ad9ab740 10 client.25610 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.372+0000 7f78ad9ab740 10 client.25610 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.372+0000 7f78ad9ab740 20 client.25610 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.372+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558ed5a2b930 0x558ed5a2dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.372+0000 7f78ad9ab740 4 client.25610 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:57.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.372+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3863578062 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ed5976f80 con 0x558ed5a2b930 2024-03-18T00:51:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.379+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558ed5a2b930 0x558ed5a2dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.387+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558ed5a2b930 0x558ed5a2dd70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2135aea391e2cb20 server_cookie=1e389f54b5e6ee39 in_seq=0 out_seq=0 2024-03-18T00:51:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.387+0000 7f7883fff640 10 client.25610 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.392+0000 7f7883fff640 1 -- 172.21.15.49:0/3863578062 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558ed5976f80 con 0x558ed5a2b930 2024-03-18T00:51:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.392+0000 7f7883fff640 10 client.25610 handle_command_reply: tid=0 2024-03-18T00:51:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.392+0000 7f78ad9ab740 1 client.25610 shutdown 2024-03-18T00:51:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.392+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3863578062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0d540 msgr2=0x558ed5a06290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.392+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0d540 0x558ed5a06290 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f788c009850 tx=0x7f788c02cc30 comp rx=0 tx=0).stop 2024-03-18T00:51:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.394+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3863578062 shutdown_connections 2024-03-18T00:51:57.394 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.394+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558ed5a2b930 0x558ed5a2dd70 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.394+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0d540 0x558ed5a06290 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.394+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a126b0 0x558ed5a0b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.394+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3863578062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0cb50 0x558ed5a05d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.394+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3863578062 >> 172.21.15.49:0/3863578062 conn(0x558ed59ff8b0 msgr2=0x558ed59febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:57.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.394+0000 7f7883fff640 0 client.25610 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:57.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.394+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3863578062 shutdown_connections 2024-03-18T00:51:57.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.395+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3863578062 wait complete. 2024-03-18T00:51:57.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.395+0000 7f78ad9ab740 20 client.25610 trim_cache size 0 max 16384 2024-03-18T00:51:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.443+0000 7f78ad9ab740 1 Processor -- start 2024-03-18T00:51:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.443+0000 7f78ad9ab740 1 -- start start 2024-03-18T00:51:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.444+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0cb50 0x558ed5a0e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.444+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a0ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.444+0000 7f78a8a84640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0cb50 0x558ed5a0e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.444+0000 7f78a8a84640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0cb50 0x558ed5a0e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52122/0 (socket says 172.21.15.49:52122) 2024-03-18T00:51:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.444+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a126b0 0x558ed5a05020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.445+0000 7f78a8a84640 1 -- 172.21.15.49:0/3928907520 learned_addr learned my addr 172.21.15.49:0/3928907520 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.445+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3928907520 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f788400e650 con 0x558ed5a0cb50 2024-03-18T00:51:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.445+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3928907520 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f78840101c0 con 0x558ed5a126b0 2024-03-18T00:51:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.445+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3928907520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f788400ed20 con 0x558ed5a0d540 2024-03-18T00:51:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.445+0000 7f78a9285640 1 --2- 172.21.15.49:0/3928907520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a126b0 0x558ed5a05020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.446+0000 7f78a3fff640 1 --2- 172.21.15.49:0/3928907520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a0ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.446+0000 7f78a8a84640 1 -- 172.21.15.49:0/3928907520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 msgr2=0x558ed5a0ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:57.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.446+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3928907520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a0ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.446+0000 7f78a8a84640 1 -- 172.21.15.49:0/3928907520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a126b0 msgr2=0x558ed5a05020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:57.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.446+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3928907520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a126b0 0x558ed5a05020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.446+0000 7f78a8a84640 1 -- 172.21.15.49:0/3928907520 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ed5a00300 con 0x558ed5a0cb50 2024-03-18T00:51:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.447+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3928907520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0cb50 0x558ed5a0e8c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f789c09b080 tx=0x7f789c0ddf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69190dacb33eed69 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.447+0000 7f789a7fc640 1 -- 172.21.15.49:0/3928907520 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c020070 con 0x558ed5a0cb50 2024-03-18T00:51:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.448+0000 7f789a7fc640 1 -- 172.21.15.49:0/3928907520 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c009de0 con 0x558ed5a0cb50 2024-03-18T00:51:57.448 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.448+0000 7f789a7fc640 1 -- 172.21.15.49:0/3928907520 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c09b380 con 0x558ed5a0cb50 2024-03-18T00:51:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.450+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3928907520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0cb50 msgr2=0x558ed5a0e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.450+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3928907520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0cb50 0x558ed5a0e8c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f789c09b080 tx=0x7f789c0ddf90 comp rx=0 tx=0).stop 2024-03-18T00:51:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.450+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3928907520 shutdown_connections 2024-03-18T00:51:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.450+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3928907520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a126b0 0x558ed5a05020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.450+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3928907520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a0d540 0x558ed5a0ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.450+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3928907520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0cb50 0x558ed5a0e8c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.450+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3928907520 >> 172.21.15.49:0/3928907520 conn(0x558ed59ff8b0 msgr2=0x558ed59faa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:57.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.451+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3928907520 shutdown_connections 2024-03-18T00:51:57.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.452+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3928907520 wait complete. 2024-03-18T00:51:57.453 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.453+0000 7f78ad9ab740 1 -- start start 2024-03-18T00:51:57.453 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.454+0000 7f78ad9ab740 1 Processor -- start 2024-03-18T00:51:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.456+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0cb50 0x558ed5a3c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0d540 0x558ed5a3cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78ad9ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a126b0 0x558ed5a3deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78ad9ab740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f788400fac0 con 0x558ed5a0d540 2024-03-18T00:51:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78a3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0d540 0x558ed5a3cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78ad9ab740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f788400f3f0 con 0x558ed5a0cb50 2024-03-18T00:51:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78ad9ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f788400c630 con 0x558ed5a126b0 2024-03-18T00:51:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78a3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0d540 0x558ed5a3cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52134/0 (socket says 172.21.15.49:52134) 2024-03-18T00:51:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78a3fff640 1 -- 172.21.15.49:0/3128508056 learned_addr learned my addr 172.21.15.49:0/3128508056 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78ad9ab740 10 client.0 _ll_register_callbacks cb 0x7f788c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78a9285640 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a126b0 0x558ed5a3deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78a3fff640 1 -- 172.21.15.49:0/3128508056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a126b0 msgr2=0x558ed5a3deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78a3fff640 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a126b0 0x558ed5a3deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78a3fff640 1 -- 172.21.15.49:0/3128508056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0cb50 msgr2=0x558ed5a3c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78a3fff640 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0cb50 0x558ed5a3c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:57.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.457+0000 7f78a3fff640 1 -- 172.21.15.49:0/3128508056 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ed5a3e4a0 con 0x558ed5a0d540 2024-03-18T00:51:57.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.458+0000 7f78a3fff640 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0d540 0x558ed5a3cf10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f788c003c70 tx=0x7f788c02c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=486cfe33b5972646 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:57.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.458+0000 7f7883fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:57.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.459+0000 7f7883fff640 1 -- 172.21.15.49:0/3128508056 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c009520 con 0x558ed5a0d540 2024-03-18T00:51:57.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.459+0000 7f7883fff640 1 -- 172.21.15.49:0/3128508056 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f788c034440 con 0x558ed5a0d540 2024-03-18T00:51:57.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.459+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3128508056 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f789c009de0 con 0x558ed5a0d540 2024-03-18T00:51:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.460+0000 7f7883fff640 1 -- 172.21.15.49:0/3128508056 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f788c003ac0 con 0x558ed5a0d540 2024-03-18T00:51:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.460+0000 7f78ad9ab740 10 client.15885 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:57.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.460+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3128508056 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558ed5a91ac0 con 0x558ed5a0d540 2024-03-18T00:51:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.462+0000 7f7883fff640 1 -- 172.21.15.49:0/3128508056 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f789c009de0 con 0x558ed5a0d540 2024-03-18T00:51:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.462+0000 7f7883fff640 1 -- 172.21.15.49:0/3128508056 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f788c0346f0 con 0x558ed5a0d540 2024-03-18T00:51:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.462+0000 7f78ad9ab740 10 client.15885 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.462+0000 7f78ad9ab740 10 client.15885 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.462+0000 7f78ad9ab740 20 client.15885 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:57.462 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.462+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558ed5a92520 0x558ed5a94940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.462+0000 7f78ad9ab740 4 client.15885 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:51:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.462+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3128508056 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558ed59712d0 con 0x558ed5a92520 2024-03-18T00:51:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.463+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558ed5a92520 0x558ed5a94940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.465+0000 7f78a8a84640 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558ed5a92520 0x558ed5a94940 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22f0c7574f9bcb49 server_cookie=7600d6b98a20bf68 in_seq=0 out_seq=0 2024-03-18T00:51:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:51:57.465+0000 7f7883fff640 10 client.15885 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:58.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:51:58.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:51:59.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.021+0000 7faff2236640 1 -- 172.21.15.49:0/2036467909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafec108e20 msgr2=0x7fafec1092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:59.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.021+0000 7faff2236640 1 --2- 172.21.15.49:0/2036467909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafec108e20 0x7fafec1092a0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009900 tx=0x7fafd402bc80 comp rx=0 tx=0).stop 2024-03-18T00:51:59.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.026+0000 7faff2236640 1 -- 172.21.15.49:0/2036467909 shutdown_connections 2024-03-18T00:51:59.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.026+0000 7faff2236640 1 --2- 172.21.15.49:0/2036467909 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafec1097e0 0x7fafec10e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.026+0000 7faff2236640 1 --2- 172.21.15.49:0/2036467909 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafec108e20 0x7fafec1092a0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.026+0000 7faff2236640 1 --2- 172.21.15.49:0/2036467909 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafec072b50 0x7fafec072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.026+0000 7faff2236640 1 -- 172.21.15.49:0/2036467909 >> 172.21.15.49:0/2036467909 conn(0x7fafec06e600 msgr2=0x7fafec070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:59.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.028+0000 7faff2236640 1 -- 172.21.15.49:0/2036467909 shutdown_connections 2024-03-18T00:51:59.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.031+0000 7faff2236640 1 -- 172.21.15.49:0/2036467909 wait complete. 2024-03-18T00:51:59.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.032+0000 7faff2236640 1 Processor -- start 2024-03-18T00:51:59.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.036+0000 7faff2236640 1 -- start start 2024-03-18T00:51:59.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.037+0000 7faff2236640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafec072b50 0x7fafec1a67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.037+0000 7faff2236640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafec108e20 0x7fafec1a6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.037+0000 7faff2236640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafec1097e0 0x7fafec1add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.037+0000 7faff2236640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fafec11c190 con 0x7fafec072b50 2024-03-18T00:51:59.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.037+0000 7faff2236640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fafec11c010 con 0x7fafec1097e0 2024-03-18T00:51:59.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.037+0000 7faff2236640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fafec11c310 con 0x7fafec108e20 2024-03-18T00:51:59.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.040+0000 7fafebfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafec1097e0 0x7fafec1add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:59.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.040+0000 7fafebfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafec1097e0 0x7fafec1add90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:40720/0 (socket says 172.21.15.49:40720) 2024-03-18T00:51:59.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.040+0000 7fafebfff640 1 -- 172.21.15.49:0/4217093928 learned_addr learned my addr 172.21.15.49:0/4217093928 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:59.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.040+0000 7fafeaffd640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafec108e20 0x7fafec1a6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:59.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.041+0000 7fafebfff640 1 -- 172.21.15.49:0/4217093928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafec108e20 msgr2=0x7fafec1a6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:59.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.041+0000 7fafebfff640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafec108e20 0x7fafec1a6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.041+0000 7fafebfff640 1 -- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafec072b50 msgr2=0x7fafec1a67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:59.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.041+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafec072b50 0x7fafec1a67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:59.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.041+0000 7fafebfff640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafec072b50 0x7fafec1a67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.041+0000 7fafebfff640 1 -- 172.21.15.49:0/4217093928 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafec110040 con 0x7fafec1097e0 2024-03-18T00:51:59.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.042+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafec072b50 0x7fafec1a67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:51:59.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.044+0000 7fafebfff640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafec1097e0 0x7fafec1add90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00eaa0 tx=0x7fafdc00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:59.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.050+0000 7fafe8ff9640 1 -- 172.21.15.49:0/4217093928 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafdc00cdb0 con 0x7fafec1097e0 2024-03-18T00:51:59.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.050+0000 7fafe8ff9640 1 -- 172.21.15.49:0/4217093928 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafdc005020 con 0x7fafec1097e0 2024-03-18T00:51:59.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.050+0000 7fafe8ff9640 1 -- 172.21.15.49:0/4217093928 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafdc0116d0 con 0x7fafec1097e0 2024-03-18T00:51:59.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.051+0000 7faff2236640 1 -- 172.21.15.49:0/4217093928 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafec110330 con 0x7fafec1097e0 2024-03-18T00:51:59.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.051+0000 7faff2236640 1 -- 172.21.15.49:0/4217093928 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafec110800 con 0x7fafec1097e0 2024-03-18T00:51:59.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.058+0000 7faff4e29740 1 Processor -- start 2024-03-18T00:51:59.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.063+0000 7faff4e29740 1 -- start start 2024-03-18T00:51:59.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.063+0000 7faff4e29740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a94360 0x559643a94780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.063+0000 7faff4e29740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 0x559643a99850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.063+0000 7faff4e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a99ec0 0x559643a9c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.063+0000 7faff4e29740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5596439f9500 con 0x559643a94d50 2024-03-18T00:51:59.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.063+0000 7faff4e29740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5596439a1be0 con 0x559643a94360 2024-03-18T00:51:59.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.063+0000 7faff4e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5596439f55a0 con 0x559643a99ec0 2024-03-18T00:51:59.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.063+0000 7fafe8ff9640 1 -- 172.21.15.49:0/4217093928 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fafdc004a40 con 0x7fafec1097e0 2024-03-18T00:51:59.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.064+0000 7fafeaffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a94360 0x559643a94780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:59.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.064+0000 7fafeaffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a94360 0x559643a94780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:40736/0 (socket says 172.21.15.49:40736) 2024-03-18T00:51:59.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.064+0000 7fafeaffd640 1 -- 172.21.15.49:0/3543023520 learned_addr learned my addr 172.21.15.49:0/3543023520 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:59.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.064+0000 7fafeaffd640 1 --2- 172.21.15.49:0/3543023520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a99ec0 0x559643a9c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:59.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.065+0000 7fafeaffd640 1 -- 172.21.15.49:0/3543023520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a99ec0 msgr2=0x559643a9c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:59.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.065+0000 7fafeaffd640 1 --2- 172.21.15.49:0/3543023520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a99ec0 0x559643a9c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.065+0000 7fafeaffd640 1 -- 172.21.15.49:0/3543023520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 msgr2=0x559643a99850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:59.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.065+0000 7fafeaffd640 1 --2- 172.21.15.49:0/3543023520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 0x559643a99850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.065+0000 7fafeaffd640 1 -- 172.21.15.49:0/3543023520 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596439b54b0 con 0x559643a94360 2024-03-18T00:51:59.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.066+0000 7fafeaffd640 1 --2- 172.21.15.49:0/3543023520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a94360 0x559643a94780 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fafd4005da0 tx=0x7fafd4005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d125323c7d7d3e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:59.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.066+0000 7fafe8ff9640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd80a29d0 0x7fafd80a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.066+0000 7fafe8ff9640 1 -- 172.21.15.49:0/4217093928 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fafdc0ceb50 con 0x7fafec1097e0 2024-03-18T00:51:59.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.067+0000 7fafe1ffb640 1 -- 172.21.15.49:0/3543023520 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd402da00 con 0x559643a94360 2024-03-18T00:51:59.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.067+0000 7fafe1ffb640 1 -- 172.21.15.49:0/3543023520 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd402dba0 con 0x559643a94360 2024-03-18T00:51:59.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.067+0000 7fafe1ffb640 1 -- 172.21.15.49:0/3543023520 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd4034440 con 0x559643a94360 2024-03-18T00:51:59.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.067+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd80a29d0 0x7fafd80a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:59.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.068+0000 7faff4e29740 1 -- 172.21.15.49:0/3543023520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a94360 msgr2=0x559643a94780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:51:59.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.068+0000 7faff4e29740 1 --2- 172.21.15.49:0/3543023520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a94360 0x559643a94780 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fafd4005da0 tx=0x7fafd4005d10 comp rx=0 tx=0).stop 2024-03-18T00:51:59.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.069+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd80a29d0 0x7fafd80a4e90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x559643a95150 tx=0x7fafe4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:59.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.069+0000 7faff4e29740 1 -- 172.21.15.49:0/3543023520 shutdown_connections 2024-03-18T00:51:59.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.069+0000 7faff4e29740 1 --2- 172.21.15.49:0/3543023520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a99ec0 0x559643a9c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.069+0000 7faff4e29740 1 --2- 172.21.15.49:0/3543023520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 0x559643a99850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.069+0000 7faff4e29740 1 --2- 172.21.15.49:0/3543023520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a94360 0x559643a94780 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.069+0000 7faff4e29740 1 -- 172.21.15.49:0/3543023520 >> 172.21.15.49:0/3543023520 conn(0x559643a8f8d0 msgr2=0x559643a8fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:51:59.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.070+0000 7faff4e29740 1 -- 172.21.15.49:0/3543023520 shutdown_connections 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.071+0000 7faff4e29740 1 -- 172.21.15.49:0/3543023520 wait complete. 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.071+0000 7faff4e29740 1 -- start start 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.071+0000 7faff4e29740 1 Processor -- start 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.072+0000 7faff4e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a94360 0x559643a9e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.072+0000 7faff4e29740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 0x559643a9ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.072+0000 7faff4e29740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 0x559643aa5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.072+0000 7faff4e29740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559643a9cdb0 con 0x559643a94d50 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.072+0000 7faff4e29740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5596439b5160 con 0x559643a99ec0 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.072+0000 7faff4e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559643a9cf30 con 0x559643a94360 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.072+0000 7faff4e29740 10 client.0 _ll_register_callbacks cb 0x559643999150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.072+0000 7fafeaffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 0x559643a9ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:59.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.073+0000 7fafeaffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 0x559643a9ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49144/0 (socket says 172.21.15.49:49144) 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.073+0000 7fafeaffd640 1 -- 172.21.15.49:0/86116072 learned_addr learned my addr 172.21.15.49:0/86116072 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.073+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a94360 0x559643a9e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.073+0000 7fafeaffd640 1 -- 172.21.15.49:0/86116072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a94360 msgr2=0x559643a9e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.073+0000 7fafeaffd640 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a94360 0x559643a9e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.073+0000 7fafeaffd640 1 -- 172.21.15.49:0/86116072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 msgr2=0x559643aa5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.073+0000 7fafeaffd640 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 0x559643aa5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.073+0000 7fafeaffd640 1 -- 172.21.15.49:0/86116072 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596437d77d0 con 0x559643a94d50 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.074+0000 7fafeaffd640 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 0x559643a9ee70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fafd4004650 tx=0x7fafd4004990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8569a4039f7a0b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.074+0000 7fafcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:51:59.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.074+0000 7fafcaffd640 1 -- 172.21.15.49:0/86116072 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd402dd50 con 0x559643a94d50 2024-03-18T00:51:59.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.074+0000 7fafcaffd640 1 -- 172.21.15.49:0/86116072 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fafd4004d90 con 0x559643a94d50 2024-03-18T00:51:59.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.074+0000 7faff4e29740 1 -- 172.21.15.49:0/86116072 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafd402dba0 con 0x559643a94d50 2024-03-18T00:51:59.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.075+0000 7fafcaffd640 1 -- 172.21.15.49:0/86116072 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fafd4004f90 con 0x559643a94d50 2024-03-18T00:51:59.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.075+0000 7faff4e29740 10 client.15891 fetch_fsmap learned FSMap version 26 2024-03-18T00:51:59.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.075+0000 7faff4e29740 1 -- 172.21.15.49:0/86116072 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55964370d3f0 con 0x559643a94d50 2024-03-18T00:51:59.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.075+0000 7fafcaffd640 1 -- 172.21.15.49:0/86116072 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd402dba0 con 0x559643a94d50 2024-03-18T00:51:59.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.076+0000 7fafcaffd640 1 -- 172.21.15.49:0/86116072 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fafd403c070 con 0x559643a94d50 2024-03-18T00:51:59.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.077+0000 7faff4e29740 10 client.15891 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:51:59.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.077+0000 7faff4e29740 10 client.15891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:51:59.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.077+0000 7faff4e29740 20 client.15891 populate_metadata read hostname 'smithi049' 2024-03-18T00:51:59.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.077+0000 7faff4e29740 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559643aa9fa0 0x559643aac390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:51:59.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.077+0000 7faff4e29740 4 client.15891 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:51:59.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.077+0000 7faff4e29740 1 -- 172.21.15.49:0/86116072 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559643a06a40 con 0x559643aa9fa0 2024-03-18T00:51:59.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.079+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559643aa9fa0 0x559643aac390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:51:59.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.084+0000 7fafcaffd640 10 client.15891 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:51:59.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:51:59.084+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559643aa9fa0 0x559643aac390 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff7d683a0699e89e server_cookie=5fbdc7fcca33aef0 in_seq=0 out_seq=0 2024-03-18T00:51:59.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:51:58 smithi049 ceph-mon[32130]: pgmap v406: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 945 MiB/s wr, 2.25k op/s 2024-03-18T00:51:59.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:51:58 smithi107 ceph-mon[36270]: pgmap v406: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 945 MiB/s wr, 2.25k op/s 2024-03-18T00:51:59.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:51:58 smithi106 ceph-mon[35899]: pgmap v406: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 945 MiB/s wr, 2.25k op/s 2024-03-18T00:52:01.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:00 smithi049 ceph-mon[32130]: pgmap v407: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 919 MiB/s wr, 1.89k op/s 2024-03-18T00:52:01.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:00 smithi107 ceph-mon[36270]: pgmap v407: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 919 MiB/s wr, 1.89k op/s 2024-03-18T00:52:01.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:00 smithi106 ceph-mon[35899]: pgmap v407: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 919 MiB/s wr, 1.89k op/s 2024-03-18T00:52:03.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:02 smithi049 ceph-mon[32130]: pgmap v408: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 817 MiB/s wr, 1.39k op/s 2024-03-18T00:52:03.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:02 smithi107 ceph-mon[36270]: pgmap v408: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 817 MiB/s wr, 1.39k op/s 2024-03-18T00:52:03.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:02 smithi106 ceph-mon[35899]: pgmap v408: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 817 MiB/s wr, 1.39k op/s 2024-03-18T00:52:05.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:05 smithi049 ceph-mon[32130]: pgmap v409: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 GiB/s wr, 1.66k op/s 2024-03-18T00:52:05.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:05 smithi107 ceph-mon[36270]: pgmap v409: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 GiB/s wr, 1.66k op/s 2024-03-18T00:52:05.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:05 smithi106 ceph-mon[35899]: pgmap v409: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.0 GiB/s wr, 1.66k op/s 2024-03-18T00:52:07.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:07 smithi049 ceph-mon[32130]: pgmap v410: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 798 MiB/s wr, 1.26k op/s 2024-03-18T00:52:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:07 smithi107 ceph-mon[36270]: pgmap v410: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 798 MiB/s wr, 1.26k op/s 2024-03-18T00:52:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:07 smithi106 ceph-mon[35899]: pgmap v410: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 798 MiB/s wr, 1.26k op/s 2024-03-18T00:52:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.840+0000 7f7883fff640 1 -- 172.21.15.49:0/3128508056 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558ed59712d0 con 0x558ed5a92520 2024-03-18T00:52:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.840+0000 7f7883fff640 10 client.15885 handle_command_reply: tid=0 2024-03-18T00:52:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.840+0000 7f78ad9ab740 1 client.15885 shutdown 2024-03-18T00:52:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.840+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3128508056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0d540 msgr2=0x558ed5a3cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:07.840 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.840+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0d540 0x558ed5a3cf10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f788c003c70 tx=0x7f788c02c1a0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3128508056 shutdown_connections 2024-03-18T00:52:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558ed5a92520 0x558ed5a94940 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558ed5a126b0 0x558ed5a3deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f7883fff640 0 client.15885 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558ed5a0d540 0x558ed5a3cf10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f78ad9ab740 1 --2- 172.21.15.49:0/3128508056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558ed5a0cb50 0x558ed5a3c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3128508056 >> 172.21.15.49:0/3128508056 conn(0x558ed59ff8b0 msgr2=0x558ed5a02b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3128508056 shutdown_connections 2024-03-18T00:52:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f78ad9ab740 1 -- 172.21.15.49:0/3128508056 wait complete. 2024-03-18T00:52:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.841+0000 7f78ad9ab740 20 client.15885 trim_cache size 0 max 16384 2024-03-18T00:52:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:52:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:52:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:52:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:52:07.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.841+0000 7fafcaffd640 1 -- 172.21.15.49:0/86116072 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559643a06a40 con 0x559643aa9fa0 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.841+0000 7fafcaffd640 10 client.15891 handle_command_reply: tid=0 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7faff4e29740 1 client.15891 shutdown 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7faff4e29740 1 -- 172.21.15.49:0/86116072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 msgr2=0x559643a9ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7faff4e29740 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 0x559643a9ee70 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fafd4004650 tx=0x7fafd4004990 comp rx=0 tx=0).stop 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7faff4e29740 1 -- 172.21.15.49:0/86116072 shutdown_connections 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7faff4e29740 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559643aa9fa0 0x559643aac390 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7faff4e29740 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 0x559643aa5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7faff4e29740 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94d50 0x559643a9ee70 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7fafcaffd640 0 client.15891 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7faff4e29740 1 --2- 172.21.15.49:0/86116072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a94360 0x559643a9e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.842+0000 7faff4e29740 1 -- 172.21.15.49:0/86116072 >> 172.21.15.49:0/86116072 conn(0x559643a8f8d0 msgr2=0x559643a8eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.842+0000 7f78aace8640 1 -- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78900a29f0 msgr2=0x7f78900a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.842+0000 7f78aace8640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78900a29f0 0x7f78900a4eb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x558ed5a0dda0 tx=0x7f7894009250 comp rx=0 tx=0).stop 2024-03-18T00:52:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.842+0000 7f78aace8640 1 -- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a41098f0 msgr2=0x7f78a4109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.842+0000 7f78aace8640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a41098f0 0x7f78a4109360 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f789c00d950 tx=0x7f789c00de10 comp rx=0 tx=0).stop 2024-03-18T00:52:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.843+0000 7f78aace8640 1 -- 172.21.15.49:0/4040940068 shutdown_connections 2024-03-18T00:52:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.843+0000 7f78aace8640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f78900a29f0 0x7f78900a4eb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.843+0000 7f78aace8640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f78a410c220 0x7f78a41a73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.843+0000 7f78aace8640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78a410eba0 0x7f78a41a9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.843+0000 7f78aace8640 1 --2- 172.21.15.49:0/4040940068 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f78a41098f0 0x7f78a4109360 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.843+0000 7f78aace8640 1 -- 172.21.15.49:0/4040940068 >> 172.21.15.49:0/4040940068 conn(0x7f78a406e820 msgr2=0x7f78a410a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.843+0000 7f78aace8640 1 -- 172.21.15.49:0/4040940068 shutdown_connections 2024-03-18T00:52:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:07.843+0000 7f78aace8640 1 -- 172.21.15.49:0/4040940068 wait complete. 2024-03-18T00:52:07.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.843+0000 7faff4e29740 1 -- 172.21.15.49:0/86116072 shutdown_connections 2024-03-18T00:52:07.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.843+0000 7faff4e29740 1 -- 172.21.15.49:0/86116072 wait complete. 2024-03-18T00:52:07.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.843+0000 7faff4e29740 20 client.15891 trim_cache size 0 max 16384 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.878+0000 7faff4e29740 1 Processor -- start 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.878+0000 7faff4e29740 1 -- start start 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.878+0000 7faff4e29740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643a96320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.878+0000 7faff4e29740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 0x559643aa5e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.878+0000 7faff4e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a96860 0x559643a8e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.878+0000 7faff4e29740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc00e520 con 0x559643a94360 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.878+0000 7faff4e29740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc010090 con 0x559643a99ec0 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.878+0000 7faff4e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc00ebf0 con 0x559643a96860 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.879+0000 7fafeb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643a96320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.879+0000 7fafeb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643a96320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39884/0 (socket says 172.21.15.49:39884) 2024-03-18T00:52:07.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.879+0000 7fafeb7fe640 1 -- 172.21.15.49:0/3857092690 learned_addr learned my addr 172.21.15.49:0/3857092690 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:07.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.879+0000 7fafeb7fe640 1 -- 172.21.15.49:0/3857092690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a96860 msgr2=0x559643a8e280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:07.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.879+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/3857092690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a96860 0x559643a8e280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.879+0000 7fafeb7fe640 1 -- 172.21.15.49:0/3857092690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 msgr2=0x559643aa5e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:07.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.879+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/3857092690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 0x559643aa5e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.879+0000 7fafeb7fe640 1 -- 172.21.15.49:0/3857092690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596436cd0f0 con 0x559643a94360 2024-03-18T00:52:07.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.879+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/3857092690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643a96320 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x559643a90840 tx=0x7fafe4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62753706710b656 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:07.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.880+0000 7fafe17fa640 1 -- 172.21.15.49:0/3857092690 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4013c50 con 0x559643a94360 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.880+0000 7fafe17fa640 1 -- 172.21.15.49:0/3857092690 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafe4029070 con 0x559643a94360 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.880+0000 7fafe17fa640 1 -- 172.21.15.49:0/3857092690 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4020ae0 con 0x559643a94360 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.881+0000 7faff4e29740 1 -- 172.21.15.49:0/3857092690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 msgr2=0x559643a96320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.881+0000 7faff4e29740 1 --2- 172.21.15.49:0/3857092690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643a96320 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x559643a90840 tx=0x7fafe4012000 comp rx=0 tx=0).stop 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.881+0000 7faff4e29740 1 -- 172.21.15.49:0/3857092690 shutdown_connections 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.881+0000 7faff4e29740 1 --2- 172.21.15.49:0/3857092690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 0x559643aa5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.881+0000 7faff4e29740 1 --2- 172.21.15.49:0/3857092690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a96860 0x559643a8e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.881+0000 7faff4e29740 1 --2- 172.21.15.49:0/3857092690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643a96320 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.881+0000 7faff4e29740 1 -- 172.21.15.49:0/3857092690 >> 172.21.15.49:0/3857092690 conn(0x559643a8f8d0 msgr2=0x559643a9e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.881+0000 7faff4e29740 1 -- 172.21.15.49:0/3857092690 shutdown_connections 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.881+0000 7faff4e29740 1 -- 172.21.15.49:0/3857092690 wait complete. 2024-03-18T00:52:07.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.882+0000 7faff4e29740 1 -- start start 2024-03-18T00:52:07.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.882+0000 7faff4e29740 1 Processor -- start 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.883+0000 7faff4e29740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643aa3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.883+0000 7fafeb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643aa3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7fafeb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643aa3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39896/0 (socket says 172.21.15.49:39896) 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7faff4e29740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a96860 0x559643aa3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7faff4e29740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 0x559643aa4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7faff4e29740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc00f990 con 0x559643a94360 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7faff4e29740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc00f2c0 con 0x559643a99ec0 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7faff4e29740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fafcc00c500 con 0x559643a96860 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7fafeb7fe640 1 -- 172.21.15.49:0/3739061365 learned_addr learned my addr 172.21.15.49:0/3739061365 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:07.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7faff4e29740 10 client.0 _ll_register_callbacks cb 0x7fafd402d950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:07.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7fafeb7fe640 1 -- 172.21.15.49:0/3739061365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a96860 msgr2=0x559643aa3740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a96860 0x559643aa3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7fafeb7fe640 1 -- 172.21.15.49:0/3739061365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 msgr2=0x559643aa4750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 0x559643aa4750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.884+0000 7fafeb7fe640 1 -- 172.21.15.49:0/3739061365 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596439c09d0 con 0x559643a94360 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.885+0000 7fafeb7fe640 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643aa3200 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fafe4006bb0 tx=0x7fafe4013c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc3ad03701981ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.885+0000 7fafcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.885+0000 7fafcaffd640 1 -- 172.21.15.49:0/3739061365 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4020c30 con 0x559643a94360 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.885+0000 7faff4e29740 1 -- 172.21.15.49:0/3739061365 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafe4029070 con 0x559643a94360 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.885+0000 7fafcaffd640 1 -- 172.21.15.49:0/3739061365 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fafe4020dd0 con 0x559643a94360 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.886+0000 7fafcaffd640 1 -- 172.21.15.49:0/3739061365 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fafe403c070 con 0x559643a94360 2024-03-18T00:52:07.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.886+0000 7faff4e29740 10 client.15897 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:07.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.886+0000 7faff4e29740 1 -- 172.21.15.49:0/3739061365 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559643aa5e80 con 0x559643a94360 2024-03-18T00:52:07.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.887+0000 7fafcaffd640 1 -- 172.21.15.49:0/3739061365 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafe4031070 con 0x559643a94360 2024-03-18T00:52:07.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.887+0000 7fafcaffd640 1 -- 172.21.15.49:0/3739061365 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fafe4036a60 con 0x559643a94360 2024-03-18T00:52:07.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.887+0000 7faff4e29740 10 client.15897 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:07.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.887+0000 7faff4e29740 10 client.15897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:07.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.887+0000 7faff4e29740 20 client.15897 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:07.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.887+0000 7faff4e29740 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559643b10630 0x559643b12a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:07.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.887+0000 7faff4e29740 4 client.15897 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:52:07.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.887+0000 7faff4e29740 1 -- 172.21.15.49:0/3739061365 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fafe4029070 con 0x559643b10630 2024-03-18T00:52:07.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.888+0000 7fafeaffd640 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559643b10630 0x559643b12a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:07.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.892+0000 7fafcaffd640 10 client.15897 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:07.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.892+0000 7fafeaffd640 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559643b10630 0x559643b12a50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91082f4a1c92e9b3 server_cookie=708a4af24c557d48 in_seq=0 out_seq=0 2024-03-18T00:52:07.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.894+0000 7fafcaffd640 1 -- 172.21.15.49:0/3739061365 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fafe4029070 con 0x559643b10630 2024-03-18T00:52:07.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.894+0000 7fafcaffd640 10 client.15897 handle_command_reply: tid=0 2024-03-18T00:52:07.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.894+0000 7faff4e29740 1 client.15897 shutdown 2024-03-18T00:52:07.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 -- 172.21.15.49:0/3739061365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 msgr2=0x559643aa3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:07.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643aa3200 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fafe4006bb0 tx=0x7fafe4013c80 comp rx=0 tx=0).stop 2024-03-18T00:52:07.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 -- 172.21.15.49:0/3739061365 shutdown_connections 2024-03-18T00:52:07.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559643b10630 0x559643b12a50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559643a99ec0 0x559643aa4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559643a96860 0x559643aa3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 --2- 172.21.15.49:0/3739061365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559643a94360 0x559643aa3200 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 -- 172.21.15.49:0/3739061365 >> 172.21.15.49:0/3739061365 conn(0x559643a8f8d0 msgr2=0x559643a92b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7fafcaffd640 0 client.15897 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 -- 172.21.15.49:0/3739061365 shutdown_connections 2024-03-18T00:52:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 1 -- 172.21.15.49:0/3739061365 wait complete. 2024-03-18T00:52:07.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.895+0000 7faff4e29740 20 client.15897 trim_cache size 0 max 16384 2024-03-18T00:52:07.896 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:52:07.896 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:52:07.897 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:52:07.897 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:52:07.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 -- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd80a29d0 msgr2=0x7fafd80a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:07.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd80a29d0 0x7fafd80a4e90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x559643a95150 tx=0x7fafe4008040 comp rx=0 tx=0).stop 2024-03-18T00:52:07.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 -- 172.21.15.49:0/4217093928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafec1097e0 msgr2=0x7fafec1add90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:07.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafec1097e0 0x7fafec1add90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00eaa0 tx=0x7fafdc00ef60 comp rx=0 tx=0).stop 2024-03-18T00:52:07.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 -- 172.21.15.49:0/4217093928 shutdown_connections 2024-03-18T00:52:07.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd80a29d0 0x7fafd80a4e90 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafec1097e0 0x7fafec1add90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafec108e20 0x7fafec1a6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 --2- 172.21.15.49:0/4217093928 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafec072b50 0x7fafec1a67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:07.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.896+0000 7faff2236640 1 -- 172.21.15.49:0/4217093928 >> 172.21.15.49:0/4217093928 conn(0x7fafec06e600 msgr2=0x7fafec070200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:07.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.897+0000 7faff2236640 1 -- 172.21.15.49:0/4217093928 shutdown_connections 2024-03-18T00:52:07.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:07.897+0000 7faff2236640 1 -- 172.21.15.49:0/4217093928 wait complete. 2024-03-18T00:52:07.922 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:52:07.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:52:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.269+0000 7f50f4153640 1 -- 172.21.15.49:0/2123547436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec101820 msgr2=0x7f50ec101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.269+0000 7f50f4153640 1 --2- 172.21.15.49:0/2123547436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec101820 0x7f50ec101c40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f50e8009900 tx=0x7f50e802bc80 comp rx=0 tx=0).stop 2024-03-18T00:52:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.269+0000 7f50f4153640 1 -- 172.21.15.49:0/2123547436 shutdown_connections 2024-03-18T00:52:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.269+0000 7f50f4153640 1 --2- 172.21.15.49:0/2123547436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50ec102a10 0x7f50ec102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.269+0000 7f50f4153640 1 --2- 172.21.15.49:0/2123547436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50ec1033d0 0x7f50ec109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.269+0000 7f50f4153640 1 --2- 172.21.15.49:0/2123547436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec101820 0x7f50ec101c40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.269+0000 7f50f4153640 1 -- 172.21.15.49:0/2123547436 >> 172.21.15.49:0/2123547436 conn(0x7f50ec0fcd90 msgr2=0x7f50ec0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.269+0000 7f50f4153640 1 -- 172.21.15.49:0/2123547436 shutdown_connections 2024-03-18T00:52:08.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.269+0000 7f50f4153640 1 -- 172.21.15.49:0/2123547436 wait complete. 2024-03-18T00:52:08.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f4153640 1 Processor -- start 2024-03-18T00:52:08.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f4153640 1 -- start start 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f4153640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50ec101820 0x7f50ec197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f4153640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50ec102a10 0x7f50ec1981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f4153640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec1033d0 0x7f50ec19f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f4153640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec10d5b0 con 0x7f50ec1033d0 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f4153640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec10d430 con 0x7f50ec102a10 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f4153640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f50ec10d730 con 0x7f50ec101820 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f26f0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec1033d0 0x7f50ec19f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f26f0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec1033d0 0x7f50ec19f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39922/0 (socket says 172.21.15.49:39922) 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.270+0000 7f50f26f0640 1 -- 172.21.15.49:0/2809769188 learned_addr learned my addr 172.21.15.49:0/2809769188 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50f26f0640 1 -- 172.21.15.49:0/2809769188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50ec101820 msgr2=0x7f50ec197c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50f26f0640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50ec101820 0x7f50ec197c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50f26f0640 1 -- 172.21.15.49:0/2809769188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50ec102a10 msgr2=0x7f50ec1981d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50f26f0640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50ec102a10 0x7f50ec1981d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50f26f0640 1 -- 172.21.15.49:0/2809769188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ec19f8d0 con 0x7f50ec1033d0 2024-03-18T00:52:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50f26f0640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec1033d0 0x7f50ec19f230 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f50e400b990 tx=0x7f50e400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50e2ffd640 1 -- 172.21.15.49:0/2809769188 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e4019070 con 0x7f50ec1033d0 2024-03-18T00:52:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50e2ffd640 1 -- 172.21.15.49:0/2809769188 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50e400cba0 con 0x7f50ec1033d0 2024-03-18T00:52:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50ec073a70 con 0x7f50ec1033d0 2024-03-18T00:52:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50e2ffd640 1 -- 172.21.15.49:0/2809769188 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50e4007650 con 0x7f50ec1033d0 2024-03-18T00:52:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.271+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50ec073fc0 con 0x7f50ec1033d0 2024-03-18T00:52:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.274+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50ec108b60 con 0x7f50ec1033d0 2024-03-18T00:52:08.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.275+0000 7f50e2ffd640 1 -- 172.21.15.49:0/2809769188 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f50e401c070 con 0x7f50ec1033d0 2024-03-18T00:52:08.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.276+0000 7f50e2ffd640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50d80a2800 0x7f50d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:08.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.276+0000 7f50f1eef640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50d80a2800 0x7f50d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:08.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.276+0000 7f50e2ffd640 1 -- 172.21.15.49:0/2809769188 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f50e40cdff0 con 0x7f50ec1033d0 2024-03-18T00:52:08.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.277+0000 7f50f1eef640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50d80a2800 0x7f50d80a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f50e8036040 tx=0x7f50e8005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:08.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.284+0000 7f50e2ffd640 1 -- 172.21.15.49:0/2809769188 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f50e409b110 con 0x7f50ec1033d0 2024-03-18T00:52:08.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.895+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50ec02d570 con 0x7f50ec1033d0 2024-03-18T00:52:08.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.898+0000 7f50e2ffd640 1 -- 172.21.15.49:0/2809769188 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11933 (secure 0 0 0) 0x7f50e409aa10 con 0x7f50ec1033d0 2024-03-18T00:52:08.900 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:52:08.900 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:51:14.980346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24506,24499,24481,24463,24470]},"id":1}]} 2024-03-18T00:52:08.900 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 26 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.910+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50d80a2800 msgr2=0x7f50d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.910+0000 7f50f4153640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50d80a2800 0x7f50d80a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f50e8036040 tx=0x7f50e8005fd0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.910+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec1033d0 msgr2=0x7f50ec19f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.910+0000 7f50f4153640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec1033d0 0x7f50ec19f230 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f50e400b990 tx=0x7f50e400be50 comp rx=0 tx=0).stop 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.910+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 shutdown_connections 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.910+0000 7f50f4153640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50d80a2800 0x7f50d80a4cc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.910+0000 7f50f4153640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50ec102a10 0x7f50ec1981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.911+0000 7f50f4153640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50ec1033d0 0x7f50ec19f230 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.911+0000 7f50f4153640 1 --2- 172.21.15.49:0/2809769188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50ec101820 0x7f50ec197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.911+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 >> 172.21.15.49:0/2809769188 conn(0x7f50ec0fcd90 msgr2=0x7f50ec105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:08.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.911+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 shutdown_connections 2024-03-18T00:52:08.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:08.912+0000 7f50f4153640 1 -- 172.21.15.49:0/2809769188 wait complete. 2024-03-18T00:52:08.937 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:52:09.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.238+0000 7fbd764f1640 1 -- 172.21.15.49:0/1320511137 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd701005f0 msgr2=0x7fbd70100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.238+0000 7fbd764f1640 1 --2- 172.21.15.49:0/1320511137 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd701005f0 0x7fbd70100a10 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fbd68009ca0 tx=0x7fbd6802e350 comp rx=0 tx=0).stop 2024-03-18T00:52:09.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.241+0000 7fbd764f1640 1 -- 172.21.15.49:0/1320511137 shutdown_connections 2024-03-18T00:52:09.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.241+0000 7fbd764f1640 1 --2- 172.21.15.49:0/1320511137 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd701021a0 0x7fbd70108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.241+0000 7fbd764f1640 1 --2- 172.21.15.49:0/1320511137 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd701017e0 0x7fbd70101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.241+0000 7fbd764f1640 1 --2- 172.21.15.49:0/1320511137 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd701005f0 0x7fbd70100a10 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.241+0000 7fbd764f1640 1 -- 172.21.15.49:0/1320511137 >> 172.21.15.49:0/1320511137 conn(0x7fbd700fbb80 msgr2=0x7fbd700fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.249+0000 7fbd764f1640 1 -- 172.21.15.49:0/1320511137 shutdown_connections 2024-03-18T00:52:09.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.250+0000 7fbd764f1640 1 -- 172.21.15.49:0/1320511137 wait complete. 2024-03-18T00:52:09.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.251+0000 7fbd764f1640 1 Processor -- start 2024-03-18T00:52:09.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.252+0000 7fbd764f1640 1 -- start start 2024-03-18T00:52:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.252+0000 7fbd764f1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd701005f0 0x7fbd70195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.252+0000 7fbd764f1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd701017e0 0x7fbd70195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.252+0000 7fbd764f1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd701021a0 0x7fbd7019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.252+0000 7fbd764f1640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7010b2c0 con 0x7fbd701005f0 2024-03-18T00:52:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.252+0000 7fbd764f1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7010b140 con 0x7fbd701017e0 2024-03-18T00:52:09.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.252+0000 7fbd764f1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7010b440 con 0x7fbd701021a0 2024-03-18T00:52:09.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.253+0000 7fbd74a8e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd701021a0 0x7fbd7019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.253+0000 7fbd74a8e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd701021a0 0x7fbd7019ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34230/0 (socket says 172.21.15.49:34230) 2024-03-18T00:52:09.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.253+0000 7fbd74a8e640 1 -- 172.21.15.49:0/3390793875 learned_addr learned my addr 172.21.15.49:0/3390793875 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.256+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd701005f0 0x7fbd70195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.258+0000 7fbd74a8e640 1 -- 172.21.15.49:0/3390793875 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd701017e0 msgr2=0x7fbd70195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.258+0000 7fbd74a8e640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd701017e0 0x7fbd70195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.258+0000 7fbd74a8e640 1 -- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd701005f0 msgr2=0x7fbd70195850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.258+0000 7fbd74a8e640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd701005f0 0x7fbd70195850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.258+0000 7fbd74a8e640 1 -- 172.21.15.49:0/3390793875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd701a1880 con 0x7fbd701021a0 2024-03-18T00:52:09.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.258+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd701005f0 0x7fbd70195850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:52:09.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.263+0000 7fbd74a8e640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd701021a0 0x7fbd7019ce00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400b3f0 tx=0x7fbd6400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.265+0000 7fbd6d7fa640 1 -- 172.21.15.49:0/3390793875 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd64015020 con 0x7fbd701021a0 2024-03-18T00:52:09.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.265+0000 7fbd764f1640 1 -- 172.21.15.49:0/3390793875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd7019d3a0 con 0x7fbd701021a0 2024-03-18T00:52:09.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.265+0000 7fbd764f1640 1 -- 172.21.15.49:0/3390793875 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd7019d970 con 0x7fbd701021a0 2024-03-18T00:52:09.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.265+0000 7fbd6d7fa640 1 -- 172.21.15.49:0/3390793875 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd64011d10 con 0x7fbd701021a0 2024-03-18T00:52:09.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.265+0000 7fbd6d7fa640 1 -- 172.21.15.49:0/3390793875 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400f2e0 con 0x7fbd701021a0 2024-03-18T00:52:09.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.270+0000 7fbd791b8740 1 Processor -- start 2024-03-18T00:52:09.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.274+0000 7fbd791b8740 1 -- start start 2024-03-18T00:52:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.274+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d2a40 0x55b27b1d2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.274+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1d7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.274+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 0x55b27b1daa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.274+0000 7fbd791b8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b27b137490 con 0x55b27b1d3430 2024-03-18T00:52:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.274+0000 7fbd791b8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b27b0dfc90 con 0x55b27b1d2a40 2024-03-18T00:52:09.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.274+0000 7fbd791b8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b27b133530 con 0x55b27b1d85a0 2024-03-18T00:52:09.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.276+0000 7fbd6d7fa640 1 -- 172.21.15.49:0/3390793875 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbd64018020 con 0x7fbd701021a0 2024-03-18T00:52:09.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.277+0000 7fbd6ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1d7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.277+0000 7fbd6ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1d7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39948/0 (socket says 172.21.15.49:39948) 2024-03-18T00:52:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.277+0000 7fbd6ffff640 1 -- 172.21.15.49:0/1903497369 learned_addr learned my addr 172.21.15.49:0/1903497369 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.278+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d2a40 0x55b27b1d2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.278+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 0x55b27b1daa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.279 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.278+0000 7fbd6d7fa640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd580a2950 0x7fbd580a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.279+0000 7fbd6d7fa640 1 -- 172.21.15.49:0/3390793875 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbd640ce200 con 0x7fbd701021a0 2024-03-18T00:52:09.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.281+0000 7fbd74a8e640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd580a2950 0x7fbd580a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.282+0000 7fbd6ffff640 1 -- 172.21.15.49:0/1903497369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 msgr2=0x55b27b1daa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:09.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.282+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 0x55b27b1daa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.282+0000 7fbd6ffff640 1 -- 172.21.15.49:0/1903497369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d2a40 msgr2=0x55b27b1d2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:09.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.282+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d2a40 0x55b27b1d2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.282+0000 7fbd6ffff640 1 -- 172.21.15.49:0/1903497369 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b27b19a500 con 0x55b27b1d3430 2024-03-18T00:52:09.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.282+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 0x55b27b1daa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:52:09.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.282+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1d7f30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbd70101640 tx=0x7fbd68000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa059c7724b433a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.283+0000 7fbd74a8e640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd580a2950 0x7fbd580a4e10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400bfd0 tx=0x7fbd640ce450 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.283+0000 7fbd567fc640 1 -- 172.21.15.49:0/1903497369 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68044020 con 0x55b27b1d3430 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.283+0000 7fbd567fc640 1 -- 172.21.15.49:0/1903497369 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd6803c040 con 0x55b27b1d3430 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.283+0000 7fbd567fc640 1 -- 172.21.15.49:0/1903497369 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6803b070 con 0x55b27b1d3430 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.284+0000 7fbd791b8740 1 -- 172.21.15.49:0/1903497369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 msgr2=0x55b27b1d7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.284+0000 7fbd791b8740 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1d7f30 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbd70101640 tx=0x7fbd68000f40 comp rx=0 tx=0).stop 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.286+0000 7fbd791b8740 1 -- 172.21.15.49:0/1903497369 shutdown_connections 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.286+0000 7fbd791b8740 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 0x55b27b1daa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.286+0000 7fbd791b8740 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1d7f30 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.286+0000 7fbd791b8740 1 --2- 172.21.15.49:0/1903497369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d2a40 0x55b27b1d2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.286+0000 7fbd791b8740 1 -- 172.21.15.49:0/1903497369 >> 172.21.15.49:0/1903497369 conn(0x55b27b1cdfb0 msgr2=0x55b27b1ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.287+0000 7fbd791b8740 1 -- 172.21.15.49:0/1903497369 shutdown_connections 2024-03-18T00:52:09.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.288+0000 7fbd791b8740 1 -- 172.21.15.49:0/1903497369 wait complete. 2024-03-18T00:52:09.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.289+0000 7fbd791b8740 1 -- start start 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.289+0000 7fbd791b8740 1 Processor -- start 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.290+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d2a40 0x55b27b1dcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.290+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 0x55b27b1dd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.290+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d85a0 0x55b27b1e4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.290+0000 7fbd791b8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b27b1db4f0 con 0x55b27b1d85a0 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.290+0000 7fbd791b8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b27b0f3230 con 0x55b27b1d3430 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.290+0000 7fbd791b8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b27b1db670 con 0x55b27b1d2a40 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.290+0000 7fbd791b8740 10 client.0 _ll_register_callbacks cb 0x55b27b12f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.291+0000 7fbd6f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d85a0 0x55b27b1e4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.291+0000 7fbd6f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d85a0 0x55b27b1e4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39956/0 (socket says 172.21.15.49:39956) 2024-03-18T00:52:09.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.291+0000 7fbd6f7fe640 1 -- 172.21.15.49:0/2720885662 learned_addr learned my addr 172.21.15.49:0/2720885662 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.292+0000 7fbd6f7fe640 1 -- 172.21.15.49:0/2720885662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d2a40 msgr2=0x55b27b1dcfb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.292+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d2a40 0x55b27b1dcfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.292+0000 7fbd6f7fe640 1 -- 172.21.15.49:0/2720885662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 msgr2=0x55b27b1dd4f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.292+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 0x55b27b1dd4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.292+0000 7fbd6f7fe640 1 -- 172.21.15.49:0/2720885662 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b27b102100 con 0x55b27b1d85a0 2024-03-18T00:52:09.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.294+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d85a0 0x55b27b1e4530 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fbd60004d50 tx=0x7fbd60001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad62b91923440b44 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.295+0000 7fbd4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:09.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.296+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/2720885662 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd60018060 con 0x55b27b1d85a0 2024-03-18T00:52:09.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.297+0000 7fbd791b8740 1 -- 172.21.15.49:0/2720885662 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd6803c040 con 0x55b27b1d85a0 2024-03-18T00:52:09.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.297+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/2720885662 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbd60013600 con 0x55b27b1d85a0 2024-03-18T00:52:09.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.298+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/2720885662 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbd60013800 con 0x55b27b1d85a0 2024-03-18T00:52:09.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.298+0000 7fbd791b8740 10 client.15915 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:09.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.298+0000 7fbd791b8740 1 -- 172.21.15.49:0/2720885662 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b27ae4b3f0 con 0x55b27b1d85a0 2024-03-18T00:52:09.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.299+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/2720885662 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd6803c040 con 0x55b27b1d85a0 2024-03-18T00:52:09.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.300+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/2720885662 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fbd6001ebe0 con 0x55b27b1d85a0 2024-03-18T00:52:09.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.301+0000 7fbd791b8740 10 client.15915 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:09.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.301+0000 7fbd791b8740 10 client.15915 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:52:09.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.301+0000 7fbd791b8740 20 client.15915 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:09.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.301+0000 7fbd791b8740 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b27b1f9ea0 0x55b27b1fc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.301+0000 7fbd791b8740 4 client.15915 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:09.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.301+0000 7fbd791b8740 1 -- 172.21.15.49:0/2720885662 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b27b1449d0 con 0x55b27b1f9ea0 2024-03-18T00:52:09.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.308+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b27b1f9ea0 0x55b27b1fc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.331+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b27b1f9ea0 0x55b27b1fc340 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ae2b9cf429eefa60 server_cookie=d0d9c8f050ca8dbd in_seq=0 out_seq=0 2024-03-18T00:52:09.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.331+0000 7fbd4f7fe640 10 client.15915 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:52:09.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.338+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/2720885662 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b27b1449d0 con 0x55b27b1f9ea0 2024-03-18T00:52:09.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.338+0000 7fbd4f7fe640 10 client.15915 handle_command_reply: tid=0 2024-03-18T00:52:09.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.339+0000 7fbd791b8740 1 client.15915 shutdown 2024-03-18T00:52:09.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.342+0000 7fbd791b8740 1 -- 172.21.15.49:0/2720885662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d85a0 msgr2=0x55b27b1e4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.342+0000 7fbd791b8740 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d85a0 0x55b27b1e4530 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fbd60004d50 tx=0x7fbd60001650 comp rx=0 tx=0).stop 2024-03-18T00:52:09.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.342+0000 7fbd791b8740 1 -- 172.21.15.49:0/2720885662 shutdown_connections 2024-03-18T00:52:09.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.342+0000 7fbd791b8740 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b27b1f9ea0 0x55b27b1fc340 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.342+0000 7fbd791b8740 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 0x55b27b1dd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.342+0000 7fbd791b8740 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d85a0 0x55b27b1e4530 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.342+0000 7fbd791b8740 1 --2- 172.21.15.49:0/2720885662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d2a40 0x55b27b1dcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.342+0000 7fbd791b8740 1 -- 172.21.15.49:0/2720885662 >> 172.21.15.49:0/2720885662 conn(0x55b27b1cdfb0 msgr2=0x55b27b1cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.343+0000 7fbd4f7fe640 0 client.15915 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:52:09.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.343+0000 7fbd791b8740 1 -- 172.21.15.49:0/2720885662 shutdown_connections 2024-03-18T00:52:09.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.345+0000 7fbd791b8740 1 -- 172.21.15.49:0/2720885662 wait complete. 2024-03-18T00:52:09.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.346+0000 7fbd791b8740 20 client.15915 trim_cache size 0 max 16384 2024-03-18T00:52:09.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:09 smithi049 ceph-mon[32130]: pgmap v411: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 944 MiB/s wr, 1.67k op/s 2024-03-18T00:52:09.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:09 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2809769188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:52:09.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.389+0000 7fbd791b8740 1 Processor -- start 2024-03-18T00:52:09.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.389+0000 7fbd791b8740 1 -- start start 2024-03-18T00:52:09.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.390+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d2a40 0x55b27b1d4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.391+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1e44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.391+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d85a0 0x55b27b1d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.391+0000 7fbd791b8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c00e610 con 0x55b27b1d3430 2024-03-18T00:52:09.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.391+0000 7fbd791b8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c010180 con 0x55b27b1d85a0 2024-03-18T00:52:09.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.391+0000 7fbd791b8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c00ece0 con 0x55b27b1d2a40 2024-03-18T00:52:09.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.392+0000 7fbd6ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1e44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.392+0000 7fbd6ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1e44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:39972/0 (socket says 172.21.15.49:39972) 2024-03-18T00:52:09.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.392+0000 7fbd6ffff640 1 -- 172.21.15.49:0/3822833648 learned_addr learned my addr 172.21.15.49:0/3822833648 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.394+0000 7fbd6ffff640 1 -- 172.21.15.49:0/3822833648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d2a40 msgr2=0x55b27b1d4970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:09.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.394+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/3822833648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d2a40 0x55b27b1d4970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.394+0000 7fbd6ffff640 1 -- 172.21.15.49:0/3822833648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d85a0 msgr2=0x55b27b1d4eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:09.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.394+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/3822833648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d85a0 0x55b27b1d4eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.394+0000 7fbd6ffff640 1 -- 172.21.15.49:0/3822833648 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b27af157d0 con 0x55b27b1d3430 2024-03-18T00:52:09.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.396+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/3822833648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1e44f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbd68005fa0 tx=0x7fbd68000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b0afccf70c7a1a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.398+0000 7fbd55ffb640 1 -- 172.21.15.49:0/3822833648 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68044040 con 0x55b27b1d3430 2024-03-18T00:52:09.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.398+0000 7fbd55ffb640 1 -- 172.21.15.49:0/3822833648 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd68034c70 con 0x55b27b1d3430 2024-03-18T00:52:09.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.398+0000 7fbd55ffb640 1 -- 172.21.15.49:0/3822833648 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6803b040 con 0x55b27b1d3430 2024-03-18T00:52:09.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.400+0000 7fbd791b8740 1 -- 172.21.15.49:0/3822833648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 msgr2=0x55b27b1e44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.400+0000 7fbd791b8740 1 --2- 172.21.15.49:0/3822833648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1e44f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fbd68005fa0 tx=0x7fbd68000f40 comp rx=0 tx=0).stop 2024-03-18T00:52:09.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.401+0000 7fbd791b8740 1 -- 172.21.15.49:0/3822833648 shutdown_connections 2024-03-18T00:52:09.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.401+0000 7fbd791b8740 1 --2- 172.21.15.49:0/3822833648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d85a0 0x55b27b1d4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.402+0000 7fbd791b8740 1 --2- 172.21.15.49:0/3822833648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d3430 0x55b27b1e44f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.403+0000 7fbd791b8740 1 --2- 172.21.15.49:0/3822833648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d2a40 0x55b27b1d4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.403+0000 7fbd791b8740 1 -- 172.21.15.49:0/3822833648 >> 172.21.15.49:0/3822833648 conn(0x55b27b1cdfb0 msgr2=0x55b27b1dcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.404+0000 7fbd791b8740 1 -- 172.21.15.49:0/3822833648 shutdown_connections 2024-03-18T00:52:09.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.405+0000 7fbd791b8740 1 -- 172.21.15.49:0/3822833648 wait complete. 2024-03-18T00:52:09.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.405+0000 7fbd791b8740 1 -- start start 2024-03-18T00:52:09.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.406+0000 7fbd791b8740 1 Processor -- start 2024-03-18T00:52:09.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.406+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d2a40 0x55b27b1e1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.407+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 0x55b27b1e1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.407+0000 7fbd791b8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 0x55b27b25e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.407+0000 7fbd791b8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c00fa80 con 0x55b27b1d2a40 2024-03-18T00:52:09.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.407+0000 7fbd791b8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c00f3b0 con 0x55b27b1d3430 2024-03-18T00:52:09.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.407+0000 7fbd791b8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c00c5f0 con 0x55b27b1d85a0 2024-03-18T00:52:09.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.407+0000 7fbd6ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 0x55b27b1e1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.407+0000 7fbd6ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 0x55b27b1e1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47866/0 (socket says 172.21.15.49:47866) 2024-03-18T00:52:09.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.407+0000 7fbd6ffff640 1 -- 172.21.15.49:0/441026759 learned_addr learned my addr 172.21.15.49:0/441026759 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.408+0000 7fbd791b8740 10 client.0 _ll_register_callbacks cb 0x7fbd60009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:09.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.408+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 0x55b27b25e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.408+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d2a40 0x55b27b1e1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.409+0000 7fbd6f7fe640 1 -- 172.21.15.49:0/441026759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 msgr2=0x55b27b25e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.409+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 0x55b27b25e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.409+0000 7fbd6f7fe640 1 -- 172.21.15.49:0/441026759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 msgr2=0x55b27b1e1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.409+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 0x55b27b1e1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.409+0000 7fbd6f7fe640 1 -- 172.21.15.49:0/441026759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b27b1e4610 con 0x55b27b1d2a40 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.409+0000 7fbd6f7fe640 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d2a40 0x55b27b1e1000 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbd600071e0 tx=0x7fbd60001f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1b444a29d5f59f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.411+0000 7fbd4f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.411+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/441026759 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd60018030 con 0x55b27b1d2a40 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.411+0000 7fbd791b8740 1 -- 172.21.15.49:0/441026759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd68034c70 con 0x55b27b1d2a40 2024-03-18T00:52:09.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.412+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/441026759 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbd6001da20 con 0x55b27b1d2a40 2024-03-18T00:52:09.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.412+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/441026759 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbd60030050 con 0x55b27b1d2a40 2024-03-18T00:52:09.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.412+0000 7fbd791b8740 10 client.15921 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:09.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.413+0000 7fbd791b8740 1 -- 172.21.15.49:0/441026759 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b27b25f6c0 con 0x55b27b1d2a40 2024-03-18T00:52:09.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.413+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/441026759 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd68034c70 con 0x55b27b1d2a40 2024-03-18T00:52:09.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.414+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/441026759 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fbd60003690 con 0x55b27b1d2a40 2024-03-18T00:52:09.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.414+0000 7fbd791b8740 10 client.15921 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:09.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.414+0000 7fbd791b8740 10 client.15921 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:52:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.414+0000 7fbd791b8740 20 client.15921 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.414+0000 7fbd791b8740 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b27b260090 0x55b27b262480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.415+0000 7fbd791b8740 4 client.15921 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:09.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.415+0000 7fbd791b8740 1 -- 172.21.15.49:0/441026759 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b27b13ed20 con 0x55b27b260090 2024-03-18T00:52:09.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.415+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b27b260090 0x55b27b262480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.418+0000 7fbd6ffff640 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b27b260090 0x55b27b262480 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1cf3d179a2ac5a9e server_cookie=b0edd2315750d97c in_seq=0 out_seq=0 2024-03-18T00:52:09.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.418+0000 7fbd4f7fe640 10 client.15921 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:52:09.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.419+0000 7fbd4f7fe640 1 -- 172.21.15.49:0/441026759 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b27b13ed20 con 0x55b27b260090 2024-03-18T00:52:09.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.419+0000 7fbd4f7fe640 10 client.15921 handle_command_reply: tid=0 2024-03-18T00:52:09.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.419+0000 7fbd791b8740 1 client.15921 shutdown 2024-03-18T00:52:09.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.419+0000 7fbd791b8740 1 -- 172.21.15.49:0/441026759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d2a40 msgr2=0x55b27b1e1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.419+0000 7fbd791b8740 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d2a40 0x55b27b1e1000 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fbd600071e0 tx=0x7fbd60001f10 comp rx=0 tx=0).stop 2024-03-18T00:52:09.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd791b8740 1 -- 172.21.15.49:0/441026759 shutdown_connections 2024-03-18T00:52:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd791b8740 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b27b260090 0x55b27b262480 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd791b8740 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b27b1d3430 0x55b27b1e1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd791b8740 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b27b1d85a0 0x55b27b25e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd4f7fe640 0 client.15921 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:52:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd791b8740 1 --2- 172.21.15.49:0/441026759 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b27b1d2a40 0x55b27b1e1000 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd791b8740 1 -- 172.21.15.49:0/441026759 >> 172.21.15.49:0/441026759 conn(0x55b27b1cdfb0 msgr2=0x55b27b1dcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd791b8740 1 -- 172.21.15.49:0/441026759 shutdown_connections 2024-03-18T00:52:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd791b8740 1 -- 172.21.15.49:0/441026759 wait complete. 2024-03-18T00:52:09.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.420+0000 7fbd791b8740 20 client.15921 trim_cache size 0 max 16384 2024-03-18T00:52:09.422 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:09.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 -- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd580a2950 msgr2=0x7fbd580a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd580a2950 0x7fbd580a4e10 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400bfd0 tx=0x7fbd640ce450 comp rx=0 tx=0).stop 2024-03-18T00:52:09.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 -- 172.21.15.49:0/3390793875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd701021a0 msgr2=0x7fbd7019ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd701021a0 0x7fbd7019ce00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400b3f0 tx=0x7fbd6400b8b0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 -- 172.21.15.49:0/3390793875 shutdown_connections 2024-03-18T00:52:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbd580a2950 0x7fbd580a4e10 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbd701017e0 0x7fbd70195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbd701021a0 0x7fbd7019ce00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 --2- 172.21.15.49:0/3390793875 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbd701005f0 0x7fbd70195850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 -- 172.21.15.49:0/3390793875 >> 172.21.15.49:0/3390793875 conn(0x7fbd700fbb80 msgr2=0x7fbd70104a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 -- 172.21.15.49:0/3390793875 shutdown_connections 2024-03-18T00:52:09.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.421+0000 7fbd764f1640 1 -- 172.21.15.49:0/3390793875 wait complete. 2024-03-18T00:52:09.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:09 smithi107 ceph-mon[36270]: pgmap v411: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 944 MiB/s wr, 1.67k op/s 2024-03-18T00:52:09.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:09 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2809769188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:52:09.445 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:52:09.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:09 smithi106 ceph-mon[35899]: pgmap v411: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 944 MiB/s wr, 1.67k op/s 2024-03-18T00:52:09.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:09 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2809769188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:52:09.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.745+0000 7f9bd12cd640 1 -- 172.21.15.49:0/416148841 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bcc107d00 msgr2=0x7f9bcc108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.745+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/416148841 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bcc107d00 0x7f9bcc108120 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009900 tx=0x7f9bc002bc80 comp rx=0 tx=0).stop 2024-03-18T00:52:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.753+0000 7f9bd12cd640 1 -- 172.21.15.49:0/416148841 shutdown_connections 2024-03-18T00:52:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.753+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/416148841 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bcc108ef0 0x7f9bcc109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.753+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/416148841 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bcc1098b0 0x7f9bcc110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.753+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/416148841 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bcc107d00 0x7f9bcc108120 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.753+0000 7f9bd12cd640 1 -- 172.21.15.49:0/416148841 >> 172.21.15.49:0/416148841 conn(0x7f9bcc076cb0 msgr2=0x7f9bcc0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.753+0000 7f9bd12cd640 1 -- 172.21.15.49:0/416148841 shutdown_connections 2024-03-18T00:52:09.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.754+0000 7f9bd12cd640 1 -- 172.21.15.49:0/416148841 wait complete. 2024-03-18T00:52:09.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.755+0000 7f9bd12cd640 1 Processor -- start 2024-03-18T00:52:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.756+0000 7f9bd12cd640 1 -- start start 2024-03-18T00:52:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.757+0000 7f9bd12cd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bcc107d00 0x7f9bcc0701f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.757+0000 7f9bd12cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bcc108ef0 0x7f9bcc070730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.757+0000 7f9bd12cd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bcc1098b0 0x7f9bcc07df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.757+0000 7f9bd12cd640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc1139a0 con 0x7f9bcc107d00 2024-03-18T00:52:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.757+0000 7f9bd12cd640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc113820 con 0x7f9bcc1098b0 2024-03-18T00:52:09.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.757+0000 7f9bd12cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc113b20 con 0x7f9bcc108ef0 2024-03-18T00:52:09.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.759+0000 7f9bd0acc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bcc1098b0 0x7f9bcc07df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.759+0000 7f9bd0acc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bcc1098b0 0x7f9bcc07df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47894/0 (socket says 172.21.15.49:47894) 2024-03-18T00:52:09.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.760+0000 7f9bd0acc640 1 -- 172.21.15.49:0/1801908066 learned_addr learned my addr 172.21.15.49:0/1801908066 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.761+0000 7f9bd0acc640 1 -- 172.21.15.49:0/1801908066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bcc108ef0 msgr2=0x7f9bcc070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.761+0000 7f9bd0acc640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bcc108ef0 0x7f9bcc070730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.761+0000 7f9bd0acc640 1 -- 172.21.15.49:0/1801908066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bcc107d00 msgr2=0x7f9bcc0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.761+0000 7f9bd0acc640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bcc107d00 0x7f9bcc0701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.761+0000 7f9bd0acc640 1 -- 172.21.15.49:0/1801908066 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bcc07e4d0 con 0x7f9bcc1098b0 2024-03-18T00:52:09.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.762+0000 7f9bd0acc640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bcc1098b0 0x7f9bcc07df90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c970 tx=0x7f9bbc00ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.765+0000 7f9bc97fa640 1 -- 172.21.15.49:0/1801908066 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc007bb0 con 0x7f9bcc1098b0 2024-03-18T00:52:09.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.765+0000 7f9bc97fa640 1 -- 172.21.15.49:0/1801908066 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bbc007d50 con 0x7f9bcc1098b0 2024-03-18T00:52:09.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.765+0000 7f9bc97fa640 1 -- 172.21.15.49:0/1801908066 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc018810 con 0x7f9bcc1098b0 2024-03-18T00:52:09.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.767+0000 7f9bd12cd640 1 -- 172.21.15.49:0/1801908066 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bcc07e7c0 con 0x7f9bcc1098b0 2024-03-18T00:52:09.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.767+0000 7f9bd12cd640 1 -- 172.21.15.49:0/1801908066 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bcc07ecc0 con 0x7f9bcc1098b0 2024-03-18T00:52:09.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.777+0000 7f9bc97fa640 1 -- 172.21.15.49:0/1801908066 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9bbc00fac0 con 0x7f9bcc1098b0 2024-03-18T00:52:09.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.778+0000 7f9bc97fa640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bb40a2a00 0x7f9bb40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.779+0000 7f9bc97fa640 1 -- 172.21.15.49:0/1801908066 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9bbc0cf050 con 0x7f9bcc1098b0 2024-03-18T00:52:09.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.779+0000 7f9bd51f1740 1 Processor -- start 2024-03-18T00:52:09.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.782+0000 7f9bcbfff640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bb40a2a00 0x7f9bb40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.783+0000 7f9bd51f1740 1 -- start start 2024-03-18T00:52:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.789+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560894643a40 0x560894643e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.789+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x560894648f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.789+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5608946495a0 0x56089464ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.789+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5608945a8490 con 0x560894644430 2024-03-18T00:52:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.789+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560894550c90 con 0x560894643a40 2024-03-18T00:52:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.789+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5608945a4530 con 0x5608946495a0 2024-03-18T00:52:09.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.789+0000 7f9bcb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x560894648f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.789+0000 7f9bcb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x560894648f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40002/0 (socket says 172.21.15.49:40002) 2024-03-18T00:52:09.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.789+0000 7f9bcb7fe640 1 -- 172.21.15.49:0/1863039105 learned_addr learned my addr 172.21.15.49:0/1863039105 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.790+0000 7f9bcb7fe640 1 -- 172.21.15.49:0/1863039105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5608946495a0 msgr2=0x56089464ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.790+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/1863039105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5608946495a0 0x56089464ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.790+0000 7f9bcb7fe640 1 -- 172.21.15.49:0/1863039105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560894643a40 msgr2=0x560894643e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.790+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/1863039105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560894643a40 0x560894643e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.790+0000 7f9bcb7fe640 1 -- 172.21.15.49:0/1863039105 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56089460b500 con 0x560894644430 2024-03-18T00:52:09.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.793+0000 7f9bcbfff640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bb40a2a00 0x7f9bb40a4ec0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0036040 tx=0x7f9bc002cf90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.801+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/1863039105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x560894648f30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4007d50 tx=0x7f9bc400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7126010327bb3c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.803+0000 7f9bb27fc640 1 -- 172.21.15.49:0/1863039105 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4014070 con 0x560894644430 2024-03-18T00:52:09.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.803+0000 7f9bb27fc640 1 -- 172.21.15.49:0/1863039105 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bc400fb30 con 0x560894644430 2024-03-18T00:52:09.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.803+0000 7f9bb27fc640 1 -- 172.21.15.49:0/1863039105 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc400ec30 con 0x560894644430 2024-03-18T00:52:09.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.806+0000 7f9bd51f1740 1 -- 172.21.15.49:0/1863039105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 msgr2=0x560894648f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.806+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/1863039105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x560894648f30 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4007d50 tx=0x7f9bc400c6a0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.806+0000 7f9bd51f1740 1 -- 172.21.15.49:0/1863039105 shutdown_connections 2024-03-18T00:52:09.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.806+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/1863039105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5608946495a0 0x56089464ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.806+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/1863039105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x560894648f30 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.806+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/1863039105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560894643a40 0x560894643e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.806+0000 7f9bd51f1740 1 -- 172.21.15.49:0/1863039105 >> 172.21.15.49:0/1863039105 conn(0x56089463efb0 msgr2=0x56089463f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.812+0000 7f9bd51f1740 1 -- 172.21.15.49:0/1863039105 shutdown_connections 2024-03-18T00:52:09.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.815+0000 7f9bd51f1740 1 -- 172.21.15.49:0/1863039105 wait complete. 2024-03-18T00:52:09.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.816+0000 7f9bd51f1740 1 -- start start 2024-03-18T00:52:09.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.816+0000 7f9bd51f1740 1 Processor -- start 2024-03-18T00:52:09.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.817+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x56089464de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.817+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 0x56089464e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.817+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5608946495a0 0x560894655410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.817+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56089464c550 con 0x560894643a40 2024-03-18T00:52:09.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.817+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560894564230 con 0x5608946495a0 2024-03-18T00:52:09.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.817+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56089464c6d0 con 0x560894644430 2024-03-18T00:52:09.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.817+0000 7f9bd51f1740 10 client.0 _ll_register_callbacks cb 0x5608945a0010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:09.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.818+0000 7f9bcb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 0x56089464e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.818+0000 7f9bcb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 0x56089464e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34308/0 (socket says 172.21.15.49:34308) 2024-03-18T00:52:09.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.818+0000 7f9bcb7fe640 1 -- 172.21.15.49:0/3472474254 learned_addr learned my addr 172.21.15.49:0/3472474254 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.819+0000 7f9bcbfff640 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x56089464de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.820+0000 7f9bd0acc640 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5608946495a0 0x560894655410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.821+0000 7f9bcbfff640 1 -- 172.21.15.49:0/3472474254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 msgr2=0x56089464e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:09.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.821+0000 7f9bcbfff640 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 0x56089464e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.821+0000 7f9bcbfff640 1 -- 172.21.15.49:0/3472474254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5608946495a0 msgr2=0x560894655410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:09.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.821+0000 7f9bcbfff640 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5608946495a0 0x560894655410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.821+0000 7f9bcbfff640 1 -- 172.21.15.49:0/3472474254 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560894573100 con 0x560894643a40 2024-03-18T00:52:09.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.824+0000 7f9bcbfff640 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x56089464de90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9bc002c7c0 tx=0x7f9bc0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8a595e5fbd080c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.824+0000 7f9bab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:09.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.825+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3472474254 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0045040 con 0x560894643a40 2024-03-18T00:52:09.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.826+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3472474254 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bc400fb30 con 0x560894643a40 2024-03-18T00:52:09.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.826+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3472474254 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9bc00037f0 con 0x560894643a40 2024-03-18T00:52:09.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.827+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3472474254 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9bc0003a10 con 0x560894643a40 2024-03-18T00:52:09.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.832+0000 7f9bd51f1740 10 client.15927 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:09.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.832+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3472474254 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5608942bc3f0 con 0x560894643a40 2024-03-18T00:52:09.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.837+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3472474254 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bc400fb30 con 0x560894643a40 2024-03-18T00:52:09.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.837+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3472474254 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f9bc003fcb0 con 0x560894643a40 2024-03-18T00:52:09.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.837+0000 7f9bd51f1740 10 client.15927 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:09.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.837+0000 7f9bd51f1740 10 client.15927 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:52:09.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.837+0000 7f9bd51f1740 20 client.15927 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:09.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.837+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x56089465a0e0 0x56089465c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.837+0000 7f9bd51f1740 4 client.15927 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:09.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.837+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3472474254 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608945b59d0 con 0x56089465a0e0 2024-03-18T00:52:09.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.841+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x56089465a0e0 0x56089465c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.845+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x56089465a0e0 0x56089465c560 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=550eb1641b06d0c0 server_cookie=b7fe87346ec5bc99 in_seq=0 out_seq=0 2024-03-18T00:52:09.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.846+0000 7f9bab7fe640 10 client.15927 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:52:09.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.847+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3472474254 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5608945b59d0 con 0x56089465a0e0 2024-03-18T00:52:09.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.848+0000 7f9bab7fe640 10 client.15927 handle_command_reply: tid=0 2024-03-18T00:52:09.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.848+0000 7f9bd51f1740 1 client.15927 shutdown 2024-03-18T00:52:09.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.853+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3472474254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 msgr2=0x56089464de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.853+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x56089464de90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f9bc002c7c0 tx=0x7f9bc0009260 comp rx=0 tx=0).stop 2024-03-18T00:52:09.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.860+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3472474254 shutdown_connections 2024-03-18T00:52:09.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.860+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x56089465a0e0 0x56089465c560 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.860+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5608946495a0 0x560894655410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.860+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 0x56089464e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.860+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3472474254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x56089464de90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.860+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3472474254 >> 172.21.15.49:0/3472474254 conn(0x56089463efb0 msgr2=0x56089463e140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.861+0000 7f9bab7fe640 0 client.15927 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:52:09.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.861+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3472474254 shutdown_connections 2024-03-18T00:52:09.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.861+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3472474254 wait complete. 2024-03-18T00:52:09.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.861+0000 7f9bd51f1740 20 client.15927 trim_cache size 0 max 16384 2024-03-18T00:52:09.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.911+0000 7f9bd51f1740 1 Processor -- start 2024-03-18T00:52:09.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.913+0000 7f9bd51f1740 1 -- start start 2024-03-18T00:52:09.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.913+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560894643a40 0x56089464daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.913+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x5608946553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.913+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5608946495a0 0x56089464dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.913+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc1c0220 con 0x560894644430 2024-03-18T00:52:09.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.913+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc1c1d90 con 0x560894643a40 2024-03-18T00:52:09.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.913+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc1c08f0 con 0x5608946495a0 2024-03-18T00:52:09.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.914+0000 7f9bcb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x5608946553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.914+0000 7f9bcb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x5608946553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40016/0 (socket says 172.21.15.49:40016) 2024-03-18T00:52:09.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.914+0000 7f9bcb7fe640 1 -- 172.21.15.49:0/993823377 learned_addr learned my addr 172.21.15.49:0/993823377 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.914+0000 7f9bd0acc640 1 --2- 172.21.15.49:0/993823377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5608946495a0 0x56089464dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.915+0000 7f9bcb7fe640 1 -- 172.21.15.49:0/993823377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5608946495a0 msgr2=0x56089464dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:09.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.916+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/993823377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5608946495a0 0x56089464dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.916+0000 7f9bcb7fe640 1 -- 172.21.15.49:0/993823377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560894643a40 msgr2=0x56089464daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:09.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.916+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/993823377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560894643a40 0x56089464daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.916+0000 7f9bcb7fe640 1 -- 172.21.15.49:0/993823377 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608943867d0 con 0x560894644430 2024-03-18T00:52:09.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.916+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/993823377 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x5608946553d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4007d20 tx=0x7f9bc401b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98f3889ad58658b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.917+0000 7f9bb1ffb640 1 -- 172.21.15.49:0/993823377 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4014040 con 0x560894644430 2024-03-18T00:52:09.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.917+0000 7f9bb1ffb640 1 -- 172.21.15.49:0/993823377 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bc4007380 con 0x560894644430 2024-03-18T00:52:09.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.917+0000 7f9bb1ffb640 1 -- 172.21.15.49:0/993823377 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4005260 con 0x560894644430 2024-03-18T00:52:09.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.919+0000 7f9bd51f1740 1 -- 172.21.15.49:0/993823377 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 msgr2=0x5608946553d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.919+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/993823377 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x5608946553d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4007d20 tx=0x7f9bc401b040 comp rx=0 tx=0).stop 2024-03-18T00:52:09.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.919+0000 7f9bd51f1740 1 -- 172.21.15.49:0/993823377 shutdown_connections 2024-03-18T00:52:09.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.919+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/993823377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5608946495a0 0x56089464dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.919+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/993823377 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894644430 0x5608946553d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.919+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/993823377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560894643a40 0x56089464daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.919+0000 7f9bd51f1740 1 -- 172.21.15.49:0/993823377 >> 172.21.15.49:0/993823377 conn(0x56089463efb0 msgr2=0x560894645890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.919+0000 7f9bd51f1740 1 -- 172.21.15.49:0/993823377 shutdown_connections 2024-03-18T00:52:09.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.920+0000 7f9bd51f1740 1 -- 172.21.15.49:0/993823377 wait complete. 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.920+0000 7f9bd51f1740 1 -- start start 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.920+0000 7f9bd51f1740 1 Processor -- start 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x560894655060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 0x560894651b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bd51f1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5608946495a0 0x5608946520c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc1c1690 con 0x560894643a40 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc1c0fc0 con 0x5608946495a0 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bd51f1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc1be200 con 0x560894644430 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bd51f1740 10 client.0 _ll_register_callbacks cb 0x7f9bc0042040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bcbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x560894655060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bcbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x560894655060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40026/0 (socket says 172.21.15.49:40026) 2024-03-18T00:52:09.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bcbfff640 1 -- 172.21.15.49:0/3112582217 learned_addr learned my addr 172.21.15.49:0/3112582217 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:09.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bcbfff640 1 -- 172.21.15.49:0/3112582217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 msgr2=0x560894651b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bcbfff640 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 0x560894651b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bcbfff640 1 -- 172.21.15.49:0/3112582217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5608946495a0 msgr2=0x5608946520c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bcbfff640 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5608946495a0 0x5608946520c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.921+0000 7f9bcbfff640 1 -- 172.21.15.49:0/3112582217 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608946526b0 con 0x560894643a40 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.922+0000 7f9bcbfff640 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x560894655060 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0034560 tx=0x7f9bc0034a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac48e33699594f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.922+0000 7f9bab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.922+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3112582217 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0045040 con 0x560894643a40 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.922+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3112582217 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9bc0003bb0 con 0x560894643a40 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.922+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3112582217 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bc4007380 con 0x560894643a40 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.922+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3112582217 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9bc002cd30 con 0x560894643a40 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.923+0000 7f9bd51f1740 10 client.15933 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:09.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.923+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3112582217 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5608946bf510 con 0x560894643a40 2024-03-18T00:52:09.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.923+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3112582217 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bc4007380 con 0x560894643a40 2024-03-18T00:52:09.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.924+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3112582217 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f9bc003f950 con 0x560894643a40 2024-03-18T00:52:09.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.924+0000 7f9bd51f1740 10 client.15933 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:09.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.924+0000 7f9bd51f1740 10 client.15933 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:52:09.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.924+0000 7f9bd51f1740 20 client.15933 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:09.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.924+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x5608946bff40 0x5608946c2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:09.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.924+0000 7f9bd51f1740 4 client.15933 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:09.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.924+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3112582217 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5608945afd20 con 0x5608946bff40 2024-03-18T00:52:09.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.926+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x5608946bff40 0x5608946c2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:09.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.930+0000 7f9bab7fe640 10 client.15933 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:52:09.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.930+0000 7f9bcb7fe640 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x5608946bff40 0x5608946c2360 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5703a02d0204b3dc server_cookie=1e0500aa3cc9d13b in_seq=0 out_seq=0 2024-03-18T00:52:09.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.933+0000 7f9bab7fe640 1 -- 172.21.15.49:0/3112582217 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5608945afd20 con 0x5608946bff40 2024-03-18T00:52:09.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.933+0000 7f9bab7fe640 10 client.15933 handle_command_reply: tid=0 2024-03-18T00:52:09.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.934+0000 7f9bd51f1740 1 client.15933 shutdown 2024-03-18T00:52:09.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.934+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3112582217 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 msgr2=0x560894655060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.934+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x560894655060 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0034560 tx=0x7f9bc0034a00 comp rx=0 tx=0).stop 2024-03-18T00:52:09.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.934+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3112582217 shutdown_connections 2024-03-18T00:52:09.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.934+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x5608946bff40 0x5608946c2360 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.934+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5608946495a0 0x5608946520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.934+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560894644430 0x560894651b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.934+0000 7f9bd51f1740 1 --2- 172.21.15.49:0/3112582217 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560894643a40 0x560894655060 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.934+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3112582217 >> 172.21.15.49:0/3112582217 conn(0x56089463efb0 msgr2=0x56089464da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.935+0000 7f9bab7fe640 0 client.15933 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:52:09.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.935+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3112582217 shutdown_connections 2024-03-18T00:52:09.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.935+0000 7f9bd51f1740 1 -- 172.21.15.49:0/3112582217 wait complete. 2024-03-18T00:52:09.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.935+0000 7f9bd51f1740 20 client.15933 trim_cache size 0 max 16384 2024-03-18T00:52:09.937 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:09.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.936+0000 7f9bd12cd640 1 -- 172.21.15.49:0/1801908066 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bb40a2a00 msgr2=0x7f9bb40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.936+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bb40a2a00 0x7f9bb40a4ec0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0036040 tx=0x7f9bc002cf90 comp rx=0 tx=0).stop 2024-03-18T00:52:09.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.937+0000 7f9bd12cd640 1 -- 172.21.15.49:0/1801908066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bcc1098b0 msgr2=0x7f9bcc07df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:09.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.937+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bcc1098b0 0x7f9bcc07df90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c970 tx=0x7f9bbc00ce30 comp rx=0 tx=0).stop 2024-03-18T00:52:09.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.937+0000 7f9bd12cd640 1 -- 172.21.15.49:0/1801908066 shutdown_connections 2024-03-18T00:52:09.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.937+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9bb40a2a00 0x7f9bb40a4ec0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.937+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9bcc1098b0 0x7f9bcc07df90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.937+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9bcc108ef0 0x7f9bcc070730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.937+0000 7f9bd12cd640 1 --2- 172.21.15.49:0/1801908066 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9bcc107d00 0x7f9bcc0701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:09.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.937+0000 7f9bd12cd640 1 -- 172.21.15.49:0/1801908066 >> 172.21.15.49:0/1801908066 conn(0x7f9bcc076cb0 msgr2=0x7f9bcc10c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:09.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.937+0000 7f9bd12cd640 1 -- 172.21.15.49:0/1801908066 shutdown_connections 2024-03-18T00:52:09.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:09.938+0000 7f9bd12cd640 1 -- 172.21.15.49:0/1801908066 wait complete. 2024-03-18T00:52:09.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:52:10.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.338+0000 7f3c00ba9640 1 -- 172.21.15.49:0/3500413456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bfc108eb0 msgr2=0x7f3bfc109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:10.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.338+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/3500413456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bfc108eb0 0x7f3bfc109330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4009be0 tx=0x7f3bf402bf30 comp rx=0 tx=0).stop 2024-03-18T00:52:10.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.340+0000 7f3c00ba9640 1 -- 172.21.15.49:0/3500413456 shutdown_connections 2024-03-18T00:52:10.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.340+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/3500413456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bfc108eb0 0x7f3bfc109330 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.340+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/3500413456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bfc109870 0x7f3bfc10e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.340+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/3500413456 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3bfc072bf0 0x7f3bfc073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.340+0000 7f3c00ba9640 1 -- 172.21.15.49:0/3500413456 >> 172.21.15.49:0/3500413456 conn(0x7f3bfc06e6c0 msgr2=0x7f3bfc070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:10.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.346+0000 7f3c00ba9640 1 -- 172.21.15.49:0/3500413456 shutdown_connections 2024-03-18T00:52:10.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.350+0000 7f3c00ba9640 1 -- 172.21.15.49:0/3500413456 wait complete. 2024-03-18T00:52:10.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.351+0000 7f3c00ba9640 1 Processor -- start 2024-03-18T00:52:10.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.355+0000 7f3c00ba9640 1 -- start start 2024-03-18T00:52:10.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.355+0000 7f3c00ba9640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3bfc072bf0 0x7f3bfc19e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.355+0000 7f3c00ba9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bfc109870 0x7f3bfc19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.355+0000 7f3c00ba9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bfc1a36a0 0x7f3bfc1a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.355+0000 7f3c00ba9640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc111b10 con 0x7f3bfc072bf0 2024-03-18T00:52:10.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.355+0000 7f3c00ba9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc111990 con 0x7f3bfc1a36a0 2024-03-18T00:52:10.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.355+0000 7f3c00ba9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc111c90 con 0x7f3bfc109870 2024-03-18T00:52:10.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.357+0000 7f3bfb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3bfc072bf0 0x7f3bfc19e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.357+0000 7f3bfb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3bfc072bf0 0x7f3bfc19e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40046/0 (socket says 172.21.15.49:40046) 2024-03-18T00:52:10.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.357+0000 7f3bfb7fe640 1 -- 172.21.15.49:0/1461698804 learned_addr learned my addr 172.21.15.49:0/1461698804 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:10.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.362+0000 7f3bfb7fe640 1 -- 172.21.15.49:0/1461698804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bfc109870 msgr2=0x7f3bfc19e640 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:10.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.362+0000 7f3bfb7fe640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bfc109870 0x7f3bfc19e640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.362+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bfc1a36a0 0x7f3bfc1a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.362+0000 7f3bfb7fe640 1 -- 172.21.15.49:0/1461698804 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bfc1a36a0 msgr2=0x7f3bfc1a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:10.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.362+0000 7f3bfb7fe640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bfc1a36a0 0x7f3bfc1a5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.362+0000 7f3bfb7fe640 1 -- 172.21.15.49:0/1461698804 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bfc1a6100 con 0x7f3bfc072bf0 2024-03-18T00:52:10.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.364+0000 7f3bfb7fe640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3bfc072bf0 0x7f3bfc19e100 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3bf001fa70 tx=0x7f3bf001ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:10.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.373+0000 7f3bf8ff9640 1 -- 172.21.15.49:0/1461698804 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf001dc50 con 0x7f3bfc072bf0 2024-03-18T00:52:10.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.373+0000 7f3c00ba9640 1 -- 172.21.15.49:0/1461698804 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bfc1a63f0 con 0x7f3bfc072bf0 2024-03-18T00:52:10.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.376+0000 7f3c00ba9640 1 -- 172.21.15.49:0/1461698804 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bfc1a6940 con 0x7f3bfc072bf0 2024-03-18T00:52:10.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.376+0000 7f3bf8ff9640 1 -- 172.21.15.49:0/1461698804 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bf001ddf0 con 0x7f3bfc072bf0 2024-03-18T00:52:10.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.376+0000 7f3bf8ff9640 1 -- 172.21.15.49:0/1461698804 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00216a0 con 0x7f3bfc072bf0 2024-03-18T00:52:10.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.384+0000 7f3c049f6740 1 Processor -- start 2024-03-18T00:52:10.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.385+0000 7f3c049f6740 1 -- start start 2024-03-18T00:52:10.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.385+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f16a40 0x55a353f16e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.386+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f17430 0x55a353f1bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.386+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f1c5a0 0x55a353f1ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.386+0000 7f3c049f6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a353e7b490 con 0x55a353f1c5a0 2024-03-18T00:52:10.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.386+0000 7f3c049f6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a353e23c90 con 0x55a353f17430 2024-03-18T00:52:10.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.386+0000 7f3c049f6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a353e77530 con 0x55a353f16a40 2024-03-18T00:52:10.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.387+0000 7f3bfbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f16a40 0x55a353f16e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.388+0000 7f3bfbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f16a40 0x55a353f16e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34368/0 (socket says 172.21.15.49:34368) 2024-03-18T00:52:10.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.388+0000 7f3bfbfff640 1 -- 172.21.15.49:0/2891349414 learned_addr learned my addr 172.21.15.49:0/2891349414 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:10.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.392+0000 7f3bfbfff640 1 -- 172.21.15.49:0/2891349414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f17430 msgr2=0x55a353f1bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:10.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.392+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/2891349414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f17430 0x55a353f1bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.393+0000 7f3bfbfff640 1 -- 172.21.15.49:0/2891349414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f1c5a0 msgr2=0x55a353f1ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:10.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.393+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/2891349414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f1c5a0 0x55a353f1ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.393+0000 7f3bfbfff640 1 -- 172.21.15.49:0/2891349414 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a353ede500 con 0x55a353f16a40 2024-03-18T00:52:10.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.395+0000 7f3bf8ff9640 1 -- 172.21.15.49:0/1461698804 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3bf0003780 con 0x7f3bfc072bf0 2024-03-18T00:52:10.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.396+0000 7f3bf8ff9640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3be40a2990 0x7f3be40a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.397+0000 7f3bf8ff9640 1 -- 172.21.15.49:0/1461698804 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3bf00e1e40 con 0x7f3bfc072bf0 2024-03-18T00:52:10.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.401+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/2891349414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f16a40 0x55a353f16e60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009bd0 tx=0x7f3bec010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb192a09f5cd1897 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:10.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.408+0000 7f3bfb7fe640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3be40a2990 0x7f3be40a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.409+0000 7f3be1ffb640 1 -- 172.21.15.49:0/2891349414 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec01b480 con 0x55a353f16a40 2024-03-18T00:52:10.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.410+0000 7f3be1ffb640 1 -- 172.21.15.49:0/2891349414 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bec01bb00 con 0x55a353f16a40 2024-03-18T00:52:10.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.412+0000 7f3c049f6740 1 -- 172.21.15.49:0/2891349414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f16a40 msgr2=0x55a353f16e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:10.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.412+0000 7f3c049f6740 1 --2- 172.21.15.49:0/2891349414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f16a40 0x55a353f16e60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3bec009bd0 tx=0x7f3bec010740 comp rx=0 tx=0).stop 2024-03-18T00:52:10.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.415+0000 7f3c049f6740 1 -- 172.21.15.49:0/2891349414 shutdown_connections 2024-03-18T00:52:10.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.415+0000 7f3c049f6740 1 --2- 172.21.15.49:0/2891349414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f17430 0x55a353f1bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.415+0000 7f3c049f6740 1 --2- 172.21.15.49:0/2891349414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f1c5a0 0x55a353f1ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.415+0000 7f3c049f6740 1 --2- 172.21.15.49:0/2891349414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f16a40 0x55a353f16e60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.415+0000 7f3c049f6740 1 -- 172.21.15.49:0/2891349414 >> 172.21.15.49:0/2891349414 conn(0x55a353f11fb0 msgr2=0x55a353f123b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:10.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.416+0000 7f3c049f6740 1 -- 172.21.15.49:0/2891349414 shutdown_connections 2024-03-18T00:52:10.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.419+0000 7f3c049f6740 1 -- 172.21.15.49:0/2891349414 wait complete. 2024-03-18T00:52:10.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.419+0000 7f3c049f6740 1 -- start start 2024-03-18T00:52:10.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.420+0000 7f3c049f6740 1 Processor -- start 2024-03-18T00:52:10.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.420+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f16a40 0x55a353f211e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.421+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f17430 0x55a353f21720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.421+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f1c5a0 0x55a353f28760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.421+0000 7f3c049f6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a353f1f5a0 con 0x55a353f17430 2024-03-18T00:52:10.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.421+0000 7f3c049f6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a353e37230 con 0x55a353f16a40 2024-03-18T00:52:10.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.421+0000 7f3c049f6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a353f1f720 con 0x55a353f1c5a0 2024-03-18T00:52:10.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.421+0000 7f3c049f6740 10 client.0 _ll_register_callbacks cb 0x55a353e73010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:10.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.421+0000 7f3bfbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f16a40 0x55a353f211e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.422+0000 7f3bfbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f1c5a0 0x55a353f28760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.422+0000 7f3bfbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f1c5a0 0x55a353f28760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34382/0 (socket says 172.21.15.49:34382) 2024-03-18T00:52:10.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.422+0000 7f3bfbfff640 1 -- 172.21.15.49:0/1341189184 learned_addr learned my addr 172.21.15.49:0/1341189184 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:10.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.427+0000 7f3bfb7fe640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3be40a2990 0x7f3be40a4e50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55a353f17ea0 tx=0x7f3bf00035f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:10.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.427+0000 7f3bfaffd640 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f17430 0x55a353f21720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.429+0000 7f3bfbfff640 1 -- 172.21.15.49:0/1341189184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f16a40 msgr2=0x55a353f211e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:10.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.429+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f16a40 0x55a353f211e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.429+0000 7f3bfbfff640 1 -- 172.21.15.49:0/1341189184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f17430 msgr2=0x55a353f21720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:10.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.429+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f17430 0x55a353f21720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.429+0000 7f3bfbfff640 1 -- 172.21.15.49:0/1341189184 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a353e46100 con 0x55a353f1c5a0 2024-03-18T00:52:10.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.430+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f1c5a0 0x55a353f28760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0057c0 tx=0x7f3bec010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2297514dee58cfae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:10.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.430+0000 7f3bdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:52:10.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.431+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1341189184 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec022070 con 0x55a353f1c5a0 2024-03-18T00:52:10.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.431+0000 7f3c049f6740 1 -- 172.21.15.49:0/1341189184 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a353e889d0 con 0x55a353f1c5a0 2024-03-18T00:52:10.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.431+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1341189184 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3bec005250 con 0x55a353f1c5a0 2024-03-18T00:52:10.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.431+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1341189184 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3bec032050 con 0x55a353f1c5a0 2024-03-18T00:52:10.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.434+0000 7f3c049f6740 10 client.25696 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:10.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.434+0000 7f3c049f6740 1 -- 172.21.15.49:0/1341189184 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a353b8f3f0 con 0x55a353f1c5a0 2024-03-18T00:52:10.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.444+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1341189184 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a353e889d0 con 0x55a353f1c5a0 2024-03-18T00:52:10.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.444+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1341189184 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f3bec00f040 con 0x55a353f1c5a0 2024-03-18T00:52:10.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.444+0000 7f3c049f6740 10 client.25696 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:10.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.444+0000 7f3c049f6740 10 client.25696 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:52:10.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.444+0000 7f3c049f6740 20 client.25696 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:10.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.444+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a353f2d3f0 0x55a353f2f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.444+0000 7f3c049f6740 4 client.25696 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:10.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.444+0000 7f3c049f6740 1 -- 172.21.15.49:0/1341189184 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a353e82d20 con 0x55a353f2d3f0 2024-03-18T00:52:10.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.446+0000 7f3bfaffd640 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a353f2d3f0 0x55a353f2f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.455+0000 7f3bfaffd640 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a353f2d3f0 0x55a353f2f810 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f0cca1e2d201bdc1 server_cookie=15826cc2755b4f70 in_seq=0 out_seq=0 2024-03-18T00:52:10.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.457+0000 7f3bdaffd640 10 client.25696 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:52:10.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.461+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1341189184 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a353e82d20 con 0x55a353f2d3f0 2024-03-18T00:52:10.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.461+0000 7f3bdaffd640 10 client.25696 handle_command_reply: tid=0 2024-03-18T00:52:10.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.461+0000 7f3c049f6740 1 client.25696 shutdown 2024-03-18T00:52:10.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.464+0000 7f3c049f6740 1 -- 172.21.15.49:0/1341189184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f1c5a0 msgr2=0x55a353f28760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:10.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.464+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f1c5a0 0x55a353f28760 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0057c0 tx=0x7f3bec010740 comp rx=0 tx=0).stop 2024-03-18T00:52:10.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.465+0000 7f3c049f6740 1 -- 172.21.15.49:0/1341189184 shutdown_connections 2024-03-18T00:52:10.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.465+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a353f2d3f0 0x55a353f2f810 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.465+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f1c5a0 0x55a353f28760 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.465+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f17430 0x55a353f21720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.465+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1341189184 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f16a40 0x55a353f211e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.465+0000 7f3c049f6740 1 -- 172.21.15.49:0/1341189184 >> 172.21.15.49:0/1341189184 conn(0x55a353f11fb0 msgr2=0x55a353f11140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:10.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.470+0000 7f3bdaffd640 0 client.25696 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:52:10.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.476+0000 7f3c049f6740 1 -- 172.21.15.49:0/1341189184 shutdown_connections 2024-03-18T00:52:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.478+0000 7f3c049f6740 1 -- 172.21.15.49:0/1341189184 wait complete. 2024-03-18T00:52:10.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.478+0000 7f3c049f6740 20 client.25696 trim_cache size 0 max 16384 2024-03-18T00:52:10.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.545+0000 7f3c049f6740 1 Processor -- start 2024-03-18T00:52:10.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.546+0000 7f3c049f6740 1 -- start start 2024-03-18T00:52:10.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.546+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f0cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.546+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 0x55a353f28680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.546+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f1c5a0 0x55a353f211e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.546+0000 7f3c049f6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3be800e690 con 0x55a353f16a40 2024-03-18T00:52:10.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.546+0000 7f3c049f6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8010200 con 0x55a353f1c5a0 2024-03-18T00:52:10.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.546+0000 7f3c049f6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3be800ed60 con 0x55a353f17430 2024-03-18T00:52:10.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.547+0000 7f3bfbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f0cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.547+0000 7f3bfbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f0cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40066/0 (socket says 172.21.15.49:40066) 2024-03-18T00:52:10.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.547+0000 7f3bfbfff640 1 -- 172.21.15.49:0/4176341346 learned_addr learned my addr 172.21.15.49:0/4176341346 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:10.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.547+0000 7f3bfaffd640 1 --2- 172.21.15.49:0/4176341346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 0x55a353f28680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.547+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/4176341346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f1c5a0 0x55a353f211e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.548+0000 7f3bfbfff640 1 -- 172.21.15.49:0/4176341346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 msgr2=0x55a353f28680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:10.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.548+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/4176341346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 0x55a353f28680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.548+0000 7f3bfbfff640 1 -- 172.21.15.49:0/4176341346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 msgr2=0x55a353f0cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:10.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.548+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/4176341346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f0cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.548+0000 7f3bfbfff640 1 -- 172.21.15.49:0/4176341346 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a353c597d0 con 0x55a353f1c5a0 2024-03-18T00:52:10.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.550+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/4176341346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f1c5a0 0x55a353f211e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3bec01e040 tx=0x7f3bec005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55b27e6c1c4b2db8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:10.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.550+0000 7f3be17fa640 1 -- 172.21.15.49:0/4176341346 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec022040 con 0x55a353f1c5a0 2024-03-18T00:52:10.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.550+0000 7f3be17fa640 1 -- 172.21.15.49:0/4176341346 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3bec004950 con 0x55a353f1c5a0 2024-03-18T00:52:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.552+0000 7f3c049f6740 1 -- 172.21.15.49:0/4176341346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f1c5a0 msgr2=0x55a353f211e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.552+0000 7f3c049f6740 1 --2- 172.21.15.49:0/4176341346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f1c5a0 0x55a353f211e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3bec01e040 tx=0x7f3bec005750 comp rx=0 tx=0).stop 2024-03-18T00:52:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.552+0000 7f3c049f6740 1 -- 172.21.15.49:0/4176341346 shutdown_connections 2024-03-18T00:52:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.552+0000 7f3c049f6740 1 --2- 172.21.15.49:0/4176341346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f1c5a0 0x55a353f211e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.552+0000 7f3c049f6740 1 --2- 172.21.15.49:0/4176341346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 0x55a353f28680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.552+0000 7f3c049f6740 1 --2- 172.21.15.49:0/4176341346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f0cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.553+0000 7f3c049f6740 1 -- 172.21.15.49:0/4176341346 >> 172.21.15.49:0/4176341346 conn(0x55a353f11fb0 msgr2=0x55a353f2f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:10.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.553+0000 7f3c049f6740 1 -- 172.21.15.49:0/4176341346 shutdown_connections 2024-03-18T00:52:10.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.554+0000 7f3c049f6740 1 -- 172.21.15.49:0/4176341346 wait complete. 2024-03-18T00:52:10.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.555+0000 7f3c049f6740 1 -- start start 2024-03-18T00:52:10.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.556+0000 7f3c049f6740 1 Processor -- start 2024-03-18T00:52:10.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f268c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 0x55a353f26e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3bfbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f268c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3c049f6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f27340 0x55a353f277c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3c049f6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3be800fb00 con 0x55a353f16a40 2024-03-18T00:52:10.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3c049f6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3be800f430 con 0x55a353f27340 2024-03-18T00:52:10.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3c049f6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3be800c670 con 0x55a353f17430 2024-03-18T00:52:10.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3c049f6740 10 client.0 _ll_register_callbacks cb 0x7f3bec009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:10.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3bfbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f268c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40074/0 (socket says 172.21.15.49:40074) 2024-03-18T00:52:10.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.557+0000 7f3bfbfff640 1 -- 172.21.15.49:0/1877683602 learned_addr learned my addr 172.21.15.49:0/1877683602 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:10.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.558+0000 7f3bfaffd640 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 0x55a353f26e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.558+0000 7f3bfbfff640 1 -- 172.21.15.49:0/1877683602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 msgr2=0x55a353f26e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:10.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.558+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 0x55a353f26e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.558+0000 7f3bfbfff640 1 -- 172.21.15.49:0/1877683602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f27340 msgr2=0x55a353f277c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:10.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.558+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f27340 0x55a353f277c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.558+0000 7f3bfbfff640 1 -- 172.21.15.49:0/1877683602 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a353f27fc0 con 0x55a353f16a40 2024-03-18T00:52:10.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.559+0000 7f3bfbfff640 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f268c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3bec010780 tx=0x7f3bec005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941e941876a1bc0f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:10.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.559+0000 7f3bdaffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:10.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.559+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1877683602 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bec02f070 con 0x55a353f16a40 2024-03-18T00:52:10.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.560+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1877683602 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3bec009530 con 0x55a353f16a40 2024-03-18T00:52:10.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.560+0000 7f3c049f6740 1 -- 172.21.15.49:0/1877683602 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a353e006a0 con 0x55a353f16a40 2024-03-18T00:52:10.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.560+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1877683602 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3bec022040 con 0x55a353f16a40 2024-03-18T00:52:10.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.560+0000 7f3c049f6740 10 client.15948 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:10.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.560+0000 7f3c049f6740 1 -- 172.21.15.49:0/1877683602 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a353f91d40 con 0x55a353f16a40 2024-03-18T00:52:10.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.561+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1877683602 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a353e006a0 con 0x55a353f16a40 2024-03-18T00:52:10.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.561+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1877683602 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f3bec01cad0 con 0x55a353f16a40 2024-03-18T00:52:10.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.562+0000 7f3c049f6740 10 client.15948 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:10.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.562+0000 7f3c049f6740 10 client.15948 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:52:10.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.562+0000 7f3c049f6740 20 client.15948 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:10.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.562+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a353f92610 0x55a353f94a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:10.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.562+0000 7f3c049f6740 4 client.15948 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:10.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.562+0000 7f3c049f6740 1 -- 172.21.15.49:0/1877683602 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a353e4cdc0 con 0x55a353f92610 2024-03-18T00:52:10.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.564+0000 7f3bfaffd640 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a353f92610 0x55a353f94a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:10.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.569+0000 7f3bfaffd640 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a353f92610 0x55a353f94a30 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4feb2338ecffb20d server_cookie=cc46e459d35b720b in_seq=0 out_seq=0 2024-03-18T00:52:10.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.569+0000 7f3bdaffd640 10 client.15948 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:52:10.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.572+0000 7f3bdaffd640 1 -- 172.21.15.49:0/1877683602 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a353e4cdc0 con 0x55a353f92610 2024-03-18T00:52:10.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.572+0000 7f3bdaffd640 10 client.15948 handle_command_reply: tid=0 2024-03-18T00:52:10.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.572+0000 7f3c049f6740 1 client.15948 shutdown 2024-03-18T00:52:10.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.573+0000 7f3c049f6740 1 -- 172.21.15.49:0/1877683602 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 msgr2=0x55a353f268c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:10.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.573+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f268c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f3bec010780 tx=0x7f3bec005d60 comp rx=0 tx=0).stop 2024-03-18T00:52:10.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.574+0000 7f3c049f6740 1 -- 172.21.15.49:0/1877683602 shutdown_connections 2024-03-18T00:52:10.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.574+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a353f92610 0x55a353f94a30 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.574+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a353f27340 0x55a353f277c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.574+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a353f17430 0x55a353f26e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.574+0000 7f3bdaffd640 0 client.15948 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:52:10.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.574+0000 7f3c049f6740 1 --2- 172.21.15.49:0/1877683602 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a353f16a40 0x55a353f268c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.575+0000 7f3c049f6740 1 -- 172.21.15.49:0/1877683602 >> 172.21.15.49:0/1877683602 conn(0x55a353f11fb0 msgr2=0x55a353f0cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:10.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.575+0000 7f3c049f6740 1 -- 172.21.15.49:0/1877683602 shutdown_connections 2024-03-18T00:52:10.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.575+0000 7f3c049f6740 1 -- 172.21.15.49:0/1877683602 wait complete. 2024-03-18T00:52:10.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.575+0000 7f3c049f6740 20 client.15948 trim_cache size 0 max 16384 2024-03-18T00:52:10.577 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 -- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3be40a2990 msgr2=0x7f3be40a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3be40a2990 0x7f3be40a4e50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55a353f17ea0 tx=0x7f3bf00035f0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 -- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3bfc072bf0 msgr2=0x7f3bfc19e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3bfc072bf0 0x7f3bfc19e100 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f3bf001fa70 tx=0x7f3bf001ff30 comp rx=0 tx=0).stop 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 -- 172.21.15.49:0/1461698804 shutdown_connections 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3be40a2990 0x7f3be40a4e50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3bfc1a36a0 0x7f3bfc1a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3bfc109870 0x7f3bfc19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 --2- 172.21.15.49:0/1461698804 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3bfc072bf0 0x7f3bfc19e100 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 -- 172.21.15.49:0/1461698804 >> 172.21.15.49:0/1461698804 conn(0x7f3bfc06e6c0 msgr2=0x7f3bfc10c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:10.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 -- 172.21.15.49:0/1461698804 shutdown_connections 2024-03-18T00:52:10.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:10.577+0000 7f3c00ba9640 1 -- 172.21.15.49:0/1461698804 wait complete. 2024-03-18T00:52:10.604 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:52:11.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.011+0000 7f4115ab1640 1 -- 172.21.15.49:0/637297403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f411010eb50 msgr2=0x7f4110110f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.011+0000 7f4115ab1640 1 --2- 172.21.15.49:0/637297403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f411010eb50 0x7f4110110f40 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f4108009be0 tx=0x7f410802bef0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.011+0000 7f4115ab1640 1 -- 172.21.15.49:0/637297403 shutdown_connections 2024-03-18T00:52:11.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.011+0000 7f4115ab1640 1 --2- 172.21.15.49:0/637297403 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f411010c220 0x7f411010e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.011+0000 7f4115ab1640 1 --2- 172.21.15.49:0/637297403 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f411010eb50 0x7f4110110f40 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.012+0000 7f4115ab1640 1 --2- 172.21.15.49:0/637297403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f41101098f0 0x7f411010bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.012+0000 7f4115ab1640 1 -- 172.21.15.49:0/637297403 >> 172.21.15.49:0/637297403 conn(0x7f411006e6c0 msgr2=0x7f4110070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.012+0000 7f4115ab1640 1 -- 172.21.15.49:0/637297403 shutdown_connections 2024-03-18T00:52:11.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.012+0000 7f4115ab1640 1 -- 172.21.15.49:0/637297403 wait complete. 2024-03-18T00:52:11.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.014+0000 7f4115ab1640 1 Processor -- start 2024-03-18T00:52:11.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.014+0000 7f4115ab1640 1 -- start start 2024-03-18T00:52:11.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.015+0000 7f4115ab1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41101098f0 0x7f411019f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.016+0000 7f4115ab1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f411010c220 0x7f411019f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.016+0000 7f4115ab1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f411010eb50 0x7f41101a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.016+0000 7f4114aaf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41101098f0 0x7f411019f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.016+0000 7f4114aaf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41101098f0 0x7f411019f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47996/0 (socket says 172.21.15.49:47996) 2024-03-18T00:52:11.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.017+0000 7f4114aaf640 1 -- 172.21.15.49:0/2183265650 learned_addr learned my addr 172.21.15.49:0/2183265650 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.017+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4110114b20 con 0x7f411010c220 2024-03-18T00:52:11.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.017+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f41101149a0 con 0x7f41101098f0 2024-03-18T00:52:11.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.017+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4110114ca0 con 0x7f411010eb50 2024-03-18T00:52:11.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.018+0000 7f41152b0640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f411010eb50 0x7f41101a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.018+0000 7f410ffff640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f411010c220 0x7f411019f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.019+0000 7f410ffff640 1 -- 172.21.15.49:0/2183265650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f411010eb50 msgr2=0x7f41101a66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.019+0000 7f410ffff640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f411010eb50 0x7f41101a66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.019+0000 7f410ffff640 1 -- 172.21.15.49:0/2183265650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41101098f0 msgr2=0x7f411019f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.019+0000 7f410ffff640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41101098f0 0x7f411019f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.019+0000 7f410ffff640 1 -- 172.21.15.49:0/2183265650 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41101a6d20 con 0x7f411010c220 2024-03-18T00:52:11.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.020+0000 7f410ffff640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f411010c220 0x7f411019f640 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f410400d960 tx=0x7f410400de20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.020+0000 7f410dffb640 1 -- 172.21.15.49:0/2183265650 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410401b070 con 0x7f411010c220 2024-03-18T00:52:11.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.020+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4110108840 con 0x7f411010c220 2024-03-18T00:52:11.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.021+0000 7f410dffb640 1 -- 172.21.15.49:0/2183265650 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f410400ece0 con 0x7f411010c220 2024-03-18T00:52:11.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.021+0000 7f410dffb640 1 -- 172.21.15.49:0/2183265650 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41040147a0 con 0x7f411010c220 2024-03-18T00:52:11.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.021+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4110108e60 con 0x7f411010c220 2024-03-18T00:52:11.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.025+0000 7f41199d0740 1 Processor -- start 2024-03-18T00:52:11.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.026+0000 7f41199d0740 1 -- start start 2024-03-18T00:52:11.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.026+0000 7f41199d0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fed840 0x55f1f6fedc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.026+0000 7f41199d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6fee230 0x55f1f6ff27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.026+0000 7f41199d0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6ff2e30 0x55f1f6ff52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.026+0000 7f41199d0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f1f6f49490 con 0x55f1f6fed840 2024-03-18T00:52:11.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.026+0000 7f41199d0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f1f6ef1c90 con 0x55f1f6ff2e30 2024-03-18T00:52:11.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.026+0000 7f41199d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f1f6f45530 con 0x55f1f6fee230 2024-03-18T00:52:11.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.027+0000 7f4114aaf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6fee230 0x55f1f6ff27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.027+0000 7f41152b0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fed840 0x55f1f6fedc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.027+0000 7f41152b0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fed840 0x55f1f6fedc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40098/0 (socket says 172.21.15.49:40098) 2024-03-18T00:52:11.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.027+0000 7f41152b0640 1 -- 172.21.15.49:0/1333135761 learned_addr learned my addr 172.21.15.49:0/1333135761 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.028+0000 7f41152b0640 1 -- 172.21.15.49:0/1333135761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6fee230 msgr2=0x55f1f6ff27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:11.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.028+0000 7f41152b0640 1 --2- 172.21.15.49:0/1333135761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6fee230 0x55f1f6ff27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.028+0000 7f41152b0640 1 -- 172.21.15.49:0/1333135761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6ff2e30 msgr2=0x55f1f6ff52a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:52:11.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.028+0000 7f41152b0640 1 --2- 172.21.15.49:0/1333135761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6ff2e30 0x55f1f6ff52a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.028+0000 7f41152b0640 1 -- 172.21.15.49:0/1333135761 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f6fac500 con 0x55f1f6fed840 2024-03-18T00:52:11.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.029+0000 7f410dffb640 1 -- 172.21.15.49:0/2183265650 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f410401e070 con 0x7f411010c220 2024-03-18T00:52:11.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.029+0000 7f41152b0640 1 --2- 172.21.15.49:0/1333135761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fed840 0x55f1f6fedc60 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4108009be0 tx=0x7f4108033bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1d7b8faac2834c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.029+0000 7f40faffd640 1 -- 172.21.15.49:0/1333135761 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4108043030 con 0x55f1f6fed840 2024-03-18T00:52:11.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.029+0000 7f40faffd640 1 -- 172.21.15.49:0/1333135761 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4108033e20 con 0x55f1f6fed840 2024-03-18T00:52:11.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.029+0000 7f40faffd640 1 -- 172.21.15.49:0/1333135761 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410802cb60 con 0x55f1f6fed840 2024-03-18T00:52:11.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.030+0000 7f410dffb640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f41000a2900 0x7f41000a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.030+0000 7f410dffb640 1 -- 172.21.15.49:0/2183265650 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f41040cddb0 con 0x7f411010c220 2024-03-18T00:52:11.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.031+0000 7f41199d0740 1 -- 172.21.15.49:0/1333135761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fed840 msgr2=0x55f1f6fedc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.031+0000 7f41199d0740 1 --2- 172.21.15.49:0/1333135761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fed840 0x55f1f6fedc60 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4108009be0 tx=0x7f4108033bf0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.031+0000 7f41199d0740 1 -- 172.21.15.49:0/1333135761 shutdown_connections 2024-03-18T00:52:11.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.031+0000 7f41199d0740 1 --2- 172.21.15.49:0/1333135761 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6ff2e30 0x55f1f6ff52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.031+0000 7f41199d0740 1 --2- 172.21.15.49:0/1333135761 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6fee230 0x55f1f6ff27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.031+0000 7f41199d0740 1 --2- 172.21.15.49:0/1333135761 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fed840 0x55f1f6fedc60 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.031+0000 7f41199d0740 1 -- 172.21.15.49:0/1333135761 >> 172.21.15.49:0/1333135761 conn(0x55f1f6fdff30 msgr2=0x55f1f6fe0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.031+0000 7f41199d0740 1 -- 172.21.15.49:0/1333135761 shutdown_connections 2024-03-18T00:52:11.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.031+0000 7f41199d0740 1 -- 172.21.15.49:0/1333135761 wait complete. 2024-03-18T00:52:11.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.032+0000 7f4114aaf640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f41000a2900 0x7f41000a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.032+0000 7f41199d0740 1 -- start start 2024-03-18T00:52:11.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.032+0000 7f41199d0740 1 Processor -- start 2024-03-18T00:52:11.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.033+0000 7f41199d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6fed840 0x55f1f6ff77b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.033+0000 7f41199d0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ff7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.033+0000 7f41199d0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6ff2e30 0x55f1f6ffed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.033+0000 7f4114aaf640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f41000a2900 0x7f41000a4dc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55f1f6fe0ea0 tx=0x7f40fc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f41199d0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f1f6ff5d60 con 0x55f1f6fee230 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f41199d0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f1f6f05230 con 0x55f1f6ff2e30 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f41199d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f1f6ff5ee0 con 0x55f1f6fed840 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f41199d0740 10 client.0 _ll_register_callbacks cb 0x55f1f6f41010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f410ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ff7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f410ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ff7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40100/0 (socket says 172.21.15.49:40100) 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f410ffff640 1 -- 172.21.15.49:0/1900605012 learned_addr learned my addr 172.21.15.49:0/1900605012 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f410ffff640 1 -- 172.21.15.49:0/1900605012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6fed840 msgr2=0x55f1f6ff77b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f410ffff640 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6fed840 0x55f1f6ff77b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f410ffff640 1 -- 172.21.15.49:0/1900605012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6ff2e30 msgr2=0x55f1f6ffed90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f410ffff640 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6ff2e30 0x55f1f6ffed90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.034+0000 7f410ffff640 1 -- 172.21.15.49:0/1900605012 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f6f14100 con 0x55f1f6fee230 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.035+0000 7f410ffff640 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ff7cf0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f410400bfd0 tx=0x7f410409a380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189cdd39628cba05 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.035+0000 7f40effff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:11.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.035+0000 7f40effff640 1 -- 172.21.15.49:0/1900605012 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410409e820 con 0x55f1f6fee230 2024-03-18T00:52:11.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.035+0000 7f41199d0740 1 -- 172.21.15.49:0/1900605012 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4108033e20 con 0x55f1f6fee230 2024-03-18T00:52:11.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.035+0000 7f40effff640 1 -- 172.21.15.49:0/1900605012 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f41040850b0 con 0x55f1f6fee230 2024-03-18T00:52:11.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.035+0000 7f40effff640 1 -- 172.21.15.49:0/1900605012 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f41040a2e70 con 0x55f1f6fee230 2024-03-18T00:52:11.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.035+0000 7f41199d0740 10 client.15960 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:11.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.035+0000 7f41199d0740 1 -- 172.21.15.49:0/1900605012 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f1f6c5d3f0 con 0x55f1f6fee230 2024-03-18T00:52:11.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.036+0000 7f40effff640 1 -- 172.21.15.49:0/1900605012 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4108033e20 con 0x55f1f6fee230 2024-03-18T00:52:11.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.036+0000 7f40effff640 1 -- 172.21.15.49:0/1900605012 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f4104096040 con 0x55f1f6fee230 2024-03-18T00:52:11.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.036+0000 7f41199d0740 10 client.15960 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:11.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.036+0000 7f41199d0740 10 client.15960 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:52:11.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.036+0000 7f41199d0740 20 client.15960 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:11.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.036+0000 7f41199d0740 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f1f7003640 0x55f1f7005a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.037+0000 7f41199d0740 4 client.15960 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:11.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.037+0000 7f41199d0740 1 -- 172.21.15.49:0/1900605012 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1f6f569d0 con 0x55f1f7003640 2024-03-18T00:52:11.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.038+0000 7f4114aaf640 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f1f7003640 0x55f1f7005a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.040+0000 7f4114aaf640 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f1f7003640 0x55f1f7005a60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57fe643a9cd44944 server_cookie=34fcf5ecc2f81729 in_seq=0 out_seq=0 2024-03-18T00:52:11.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.040+0000 7f40effff640 10 client.15960 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:52:11.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.045+0000 7f40effff640 1 -- 172.21.15.49:0/1900605012 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1f6f569d0 con 0x55f1f7003640 2024-03-18T00:52:11.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.045+0000 7f40effff640 10 client.15960 handle_command_reply: tid=0 2024-03-18T00:52:11.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.045+0000 7f41199d0740 1 client.15960 shutdown 2024-03-18T00:52:11.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.045+0000 7f41199d0740 1 -- 172.21.15.49:0/1900605012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 msgr2=0x55f1f6ff7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.045+0000 7f41199d0740 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ff7cf0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f410400bfd0 tx=0x7f410409a380 comp rx=0 tx=0).stop 2024-03-18T00:52:11.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.047+0000 7f41199d0740 1 -- 172.21.15.49:0/1900605012 shutdown_connections 2024-03-18T00:52:11.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.048+0000 7f41199d0740 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f1f7003640 0x55f1f7005a60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.048+0000 7f41199d0740 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6ff2e30 0x55f1f6ffed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.048+0000 7f40effff640 0 client.15960 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:52:11.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.048+0000 7f41199d0740 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ff7cf0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.049+0000 7f41199d0740 1 --2- 172.21.15.49:0/1900605012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6fed840 0x55f1f6ff77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.049+0000 7f41199d0740 1 -- 172.21.15.49:0/1900605012 >> 172.21.15.49:0/1900605012 conn(0x55f1f6fdff30 msgr2=0x55f1f6feced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.049+0000 7f41199d0740 1 -- 172.21.15.49:0/1900605012 shutdown_connections 2024-03-18T00:52:11.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.050+0000 7f41199d0740 1 -- 172.21.15.49:0/1900605012 wait complete. 2024-03-18T00:52:11.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.050+0000 7f41199d0740 20 client.15960 trim_cache size 0 max 16384 2024-03-18T00:52:11.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.072+0000 7f41199d0740 1 Processor -- start 2024-03-18T00:52:11.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.073+0000 7f41199d0740 1 -- start start 2024-03-18T00:52:11.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.073+0000 7f41199d0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 0x55f1f6ff7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.073+0000 7f41199d0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6fdefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.074+0000 7f41199d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 0x55f1f6fdf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.074+0000 7f41199d0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f40f400e560 con 0x55f1f6fee230 2024-03-18T00:52:11.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.074+0000 7f41199d0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f40f40100d0 con 0x55f1f6fed840 2024-03-18T00:52:11.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.074+0000 7f41199d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40f400ec30 con 0x55f1f6ff2e30 2024-03-18T00:52:11.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.075+0000 7f41152b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 0x55f1f6fdf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.075+0000 7f410ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6fdefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.075+0000 7f4114aaf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 0x55f1f6ff7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.075+0000 7f4114aaf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 0x55f1f6ff7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48014/0 (socket says 172.21.15.49:48014) 2024-03-18T00:52:11.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.075+0000 7f4114aaf640 1 -- 172.21.15.49:0/4068241638 learned_addr learned my addr 172.21.15.49:0/4068241638 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.076+0000 7f4114aaf640 1 -- 172.21.15.49:0/4068241638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 msgr2=0x55f1f6fdf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:11.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.076+0000 7f4114aaf640 1 --2- 172.21.15.49:0/4068241638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 0x55f1f6fdf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.076+0000 7f4114aaf640 1 -- 172.21.15.49:0/4068241638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 msgr2=0x55f1f6fdefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:11.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.076+0000 7f4114aaf640 1 --2- 172.21.15.49:0/4068241638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6fdefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.076+0000 7f4114aaf640 1 -- 172.21.15.49:0/4068241638 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1f6d277d0 con 0x55f1f6fed840 2024-03-18T00:52:11.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.078+0000 7f4114aaf640 1 --2- 172.21.15.49:0/4068241638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 0x55f1f6ff7b50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00be80 tx=0x7f40fc019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9965a41d5a94cc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.079+0000 7f40fa7fc640 1 -- 172.21.15.49:0/4068241638 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc027330 con 0x55f1f6fed840 2024-03-18T00:52:11.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.079+0000 7f40fa7fc640 1 -- 172.21.15.49:0/4068241638 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40fc01f670 con 0x55f1f6fed840 2024-03-18T00:52:11.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.079+0000 7f40fa7fc640 1 -- 172.21.15.49:0/4068241638 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc01c8d0 con 0x55f1f6fed840 2024-03-18T00:52:11.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.080+0000 7f41199d0740 1 -- 172.21.15.49:0/4068241638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 msgr2=0x55f1f6ff7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.080+0000 7f41199d0740 1 --2- 172.21.15.49:0/4068241638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 0x55f1f6ff7b50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f40fc00be80 tx=0x7f40fc019000 comp rx=0 tx=0).stop 2024-03-18T00:52:11.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.081+0000 7f41199d0740 1 -- 172.21.15.49:0/4068241638 shutdown_connections 2024-03-18T00:52:11.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.081+0000 7f41199d0740 1 --2- 172.21.15.49:0/4068241638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 0x55f1f6fdf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.081+0000 7f41199d0740 1 --2- 172.21.15.49:0/4068241638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6fdefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.081+0000 7f41199d0740 1 --2- 172.21.15.49:0/4068241638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 0x55f1f6ff7b50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.081+0000 7f41199d0740 1 -- 172.21.15.49:0/4068241638 >> 172.21.15.49:0/4068241638 conn(0x55f1f6fdff30 msgr2=0x55f1f6fde8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.081+0000 7f41199d0740 1 -- 172.21.15.49:0/4068241638 shutdown_connections 2024-03-18T00:52:11.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.081+0000 7f41199d0740 1 -- 172.21.15.49:0/4068241638 wait complete. 2024-03-18T00:52:11.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.081+0000 7f41199d0740 1 -- start start 2024-03-18T00:52:11.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.082+0000 7f41199d0740 1 Processor -- start 2024-03-18T00:52:11.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.082+0000 7f41199d0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 0x55f1f6ffb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f41199d0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ffbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f41199d0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 0x55f1f7067930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f41199d0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f40f400f9d0 con 0x55f1f6fee230 2024-03-18T00:52:11.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f41199d0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f40f400f300 con 0x55f1f6fed840 2024-03-18T00:52:11.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f41199d0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f40f400c540 con 0x55f1f6ff2e30 2024-03-18T00:52:11.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f41199d0740 10 client.0 _ll_register_callbacks cb 0x7f4108009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:11.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f410ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ffbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f41152b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 0x55f1f7067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f410ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ffbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40126/0 (socket says 172.21.15.49:40126) 2024-03-18T00:52:11.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f410ffff640 1 -- 172.21.15.49:0/770951948 learned_addr learned my addr 172.21.15.49:0/770951948 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.083+0000 7f41152b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 0x55f1f7067930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34478/0 (socket says 172.21.15.49:34478) 2024-03-18T00:52:11.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.084+0000 7f410ffff640 1 -- 172.21.15.49:0/770951948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 msgr2=0x55f1f7067930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:11.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.084+0000 7f410ffff640 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 0x55f1f7067930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.084+0000 7f410ffff640 1 -- 172.21.15.49:0/770951948 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 msgr2=0x55f1f6ffb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:11.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.084+0000 7f410ffff640 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 0x55f1f6ffb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.084+0000 7f410ffff640 1 -- 172.21.15.49:0/770951948 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1f7067e70 con 0x55f1f6fee230 2024-03-18T00:52:11.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.084+0000 7f410ffff640 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ffbd80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f41040aeab0 tx=0x7f410409a3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbafdb19fdf71559 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.085+0000 7f40effff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:11.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.085+0000 7f40effff640 1 -- 172.21.15.49:0/770951948 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104096a20 con 0x55f1f6fee230 2024-03-18T00:52:11.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.085+0000 7f40effff640 1 -- 172.21.15.49:0/770951948 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4104099bd0 con 0x55f1f6fee230 2024-03-18T00:52:11.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.085+0000 7f41199d0740 1 -- 172.21.15.49:0/770951948 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40fc01f670 con 0x55f1f6fee230 2024-03-18T00:52:11.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.085+0000 7f40effff640 1 -- 172.21.15.49:0/770951948 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f410409e820 con 0x55f1f6fee230 2024-03-18T00:52:11.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.085+0000 7f41199d0740 10 client.15966 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:11.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.085+0000 7f41199d0740 1 -- 172.21.15.49:0/770951948 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f1f7068fa0 con 0x55f1f6fee230 2024-03-18T00:52:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.086+0000 7f40effff640 1 -- 172.21.15.49:0/770951948 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40fc01f670 con 0x55f1f6fee230 2024-03-18T00:52:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.086+0000 7f40effff640 1 -- 172.21.15.49:0/770951948 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f4104096040 con 0x55f1f6fee230 2024-03-18T00:52:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.086+0000 7f41199d0740 10 client.15966 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.086+0000 7f41199d0740 10 client.15966 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:52:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.086+0000 7f41199d0740 20 client.15966 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.086+0000 7f41199d0740 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f1f7003640 0x55f1f706b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.086+0000 7f41199d0740 4 client.15966 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.086+0000 7f41199d0740 1 -- 172.21.15.49:0/770951948 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f1f6f50d20 con 0x55f1f7003640 2024-03-18T00:52:11.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.087+0000 7f4114aaf640 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f1f7003640 0x55f1f706b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.089+0000 7f40effff640 10 client.15966 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.089+0000 7f4114aaf640 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f1f7003640 0x55f1f706b820 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c24438a710b92d08 server_cookie=d4d4205f91e5a4e7 in_seq=0 out_seq=0 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f40effff640 1 -- 172.21.15.49:0/770951948 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f1f6f50d20 con 0x55f1f7003640 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f40effff640 10 client.15966 handle_command_reply: tid=0 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f41199d0740 1 client.15966 shutdown 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f41199d0740 1 -- 172.21.15.49:0/770951948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 msgr2=0x55f1f6ffbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f41199d0740 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ffbd80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f41040aeab0 tx=0x7f410409a3f0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f41199d0740 1 -- 172.21.15.49:0/770951948 shutdown_connections 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f41199d0740 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f1f7003640 0x55f1f706b820 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f41199d0740 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f1f6ff2e30 0x55f1f7067930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f41199d0740 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f1f6fee230 0x55f1f6ffbd80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f41199d0740 1 --2- 172.21.15.49:0/770951948 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f1f6fed840 0x55f1f6ffb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f40effff640 0 client.15966 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:52:11.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.090+0000 7f41199d0740 1 -- 172.21.15.49:0/770951948 >> 172.21.15.49:0/770951948 conn(0x55f1f6fdff30 msgr2=0x55f1f6fdf140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.091+0000 7f41199d0740 1 -- 172.21.15.49:0/770951948 shutdown_connections 2024-03-18T00:52:11.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.091+0000 7f41199d0740 1 -- 172.21.15.49:0/770951948 wait complete. 2024-03-18T00:52:11.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.091+0000 7f41199d0740 20 client.15966 trim_cache size 0 max 16384 2024-03-18T00:52:11.092 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:11.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f41000a2900 msgr2=0x7f41000a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f41000a2900 0x7f41000a4dc0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55f1f6fe0ea0 tx=0x7f40fc00b410 comp rx=0 tx=0).stop 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f411010c220 msgr2=0x7f411019f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f411010c220 0x7f411019f640 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f410400d960 tx=0x7f410400de20 comp rx=0 tx=0).stop 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 shutdown_connections 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f41000a2900 0x7f41000a4dc0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f411010eb50 0x7f41101a66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f411010c220 0x7f411019f640 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 --2- 172.21.15.49:0/2183265650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f41101098f0 0x7f411019f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.092+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 >> 172.21.15.49:0/2183265650 conn(0x7f411006e6c0 msgr2=0x7f411010ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.093+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 shutdown_connections 2024-03-18T00:52:11.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.093+0000 7f4115ab1640 1 -- 172.21.15.49:0/2183265650 wait complete. 2024-03-18T00:52:11.143 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:52:11.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:11 smithi049 ceph-mon[32130]: pgmap v412: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 920 MiB/s wr, 1.77k op/s 2024-03-18T00:52:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:11 smithi107 ceph-mon[36270]: pgmap v412: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 920 MiB/s wr, 1.77k op/s 2024-03-18T00:52:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:11 smithi106 ceph-mon[35899]: pgmap v412: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 920 MiB/s wr, 1.77k op/s 2024-03-18T00:52:11.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.549+0000 7f57052d0640 1 -- 172.21.15.49:0/1729488816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57001017b0 msgr2=0x7f5700101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.549+0000 7f57052d0640 1 --2- 172.21.15.49:0/1729488816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57001017b0 0x7f5700101c30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f56f000fb10 tx=0x7f56f002fdf0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.549+0000 7f57052d0640 1 -- 172.21.15.49:0/1729488816 shutdown_connections 2024-03-18T00:52:11.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.549+0000 7f57052d0640 1 --2- 172.21.15.49:0/1729488816 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5700102170 0x7f5700108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.549+0000 7f57052d0640 1 --2- 172.21.15.49:0/1729488816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57001017b0 0x7f5700101c30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.549+0000 7f57052d0640 1 --2- 172.21.15.49:0/1729488816 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f57001005c0 0x7f57001009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.550+0000 7f57052d0640 1 -- 172.21.15.49:0/1729488816 >> 172.21.15.49:0/1729488816 conn(0x7f57000fbb50 msgr2=0x7f57000fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.552+0000 7f57052d0640 1 -- 172.21.15.49:0/1729488816 shutdown_connections 2024-03-18T00:52:11.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.553+0000 7f57052d0640 1 -- 172.21.15.49:0/1729488816 wait complete. 2024-03-18T00:52:11.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.554+0000 7f57052d0640 1 Processor -- start 2024-03-18T00:52:11.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.554+0000 7f57052d0640 1 -- start start 2024-03-18T00:52:11.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.555+0000 7f57052d0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57001005c0 0x7f57001118a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.555+0000 7f57052d0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5700102170 0x7f570010e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.555+0000 7f57052d0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f570010e850 0x7f57001ae070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.555+0000 7f57052d0640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f57001139a0 con 0x7f5700102170 2024-03-18T00:52:11.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.555+0000 7f57052d0640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5700113820 con 0x7f570010e850 2024-03-18T00:52:11.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.555+0000 7f57052d0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5700113b20 con 0x7f57001005c0 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.556+0000 7f56ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5700102170 0x7f570010e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.556+0000 7f56ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5700102170 0x7f570010e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40136/0 (socket says 172.21.15.49:40136) 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.556+0000 7f56ff7fe640 1 -- 172.21.15.49:0/1577816351 learned_addr learned my addr 172.21.15.49:0/1577816351 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.556+0000 7f5704acf640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f570010e850 0x7f57001ae070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.556+0000 7f56ff7fe640 1 -- 172.21.15.49:0/1577816351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57001005c0 msgr2=0x7f57001118a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.556+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57001005c0 0x7f57001118a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.556+0000 7f56ff7fe640 1 -- 172.21.15.49:0/1577816351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f570010e850 msgr2=0x7f57001ae070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.556+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f570010e850 0x7f57001ae070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.556+0000 7f56ff7fe640 1 -- 172.21.15.49:0/1577816351 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57001ae5b0 con 0x7f5700102170 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.557+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5700102170 0x7f570010e310 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f56f000fa60 tx=0x7f56f000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.558+0000 7f56fd7fa640 1 -- 172.21.15.49:0/1577816351 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f000f250 con 0x7f5700102170 2024-03-18T00:52:11.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.558+0000 7f56fd7fa640 1 -- 172.21.15.49:0/1577816351 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56f0034040 con 0x7f5700102170 2024-03-18T00:52:11.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.558+0000 7f56fd7fa640 1 -- 172.21.15.49:0/1577816351 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f0041e60 con 0x7f5700102170 2024-03-18T00:52:11.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.559+0000 7f57052d0640 1 -- 172.21.15.49:0/1577816351 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57001ae780 con 0x7f5700102170 2024-03-18T00:52:11.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.559+0000 7f57052d0640 1 -- 172.21.15.49:0/1577816351 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57001aec20 con 0x7f5700102170 2024-03-18T00:52:11.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.566+0000 7f56fd7fa640 1 -- 172.21.15.49:0/1577816351 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f56f000c040 con 0x7f5700102170 2024-03-18T00:52:11.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.567+0000 7f56fd7fa640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f56e80a2a10 0x7f56e80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.567+0000 7f56fd7fa640 1 -- 172.21.15.49:0/1577816351 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f56f00f1340 con 0x7f5700102170 2024-03-18T00:52:11.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.572+0000 7f56fffff640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f56e80a2a10 0x7f56e80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.572+0000 7f57091f3740 1 Processor -- start 2024-03-18T00:52:11.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.579+0000 7f56fffff640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f56e80a2a10 0x7f56e80a4ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5700101610 tx=0x7f56f4005ba0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.581+0000 7f57091f3740 1 -- start start 2024-03-18T00:52:11.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.581+0000 7f57091f3740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f662ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.581+0000 7f57091f3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f662f430 0x55d7f6633f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.581+0000 7f57091f3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f66345a0 0x55d7f6636a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.581+0000 7f57091f3740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6593490 con 0x55d7f662ea40 2024-03-18T00:52:11.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.581+0000 7f57091f3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d7f653bc90 con 0x55d7f66345a0 2024-03-18T00:52:11.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.581+0000 7f57091f3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d7f658f530 con 0x55d7f662f430 2024-03-18T00:52:11.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.585+0000 7f5704acf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f66345a0 0x55d7f6636a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.586+0000 7f56fffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f662ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.586+0000 7f56fffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f662ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40142/0 (socket says 172.21.15.49:40142) 2024-03-18T00:52:11.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.586+0000 7f56fffff640 1 -- 172.21.15.49:0/1857593603 learned_addr learned my addr 172.21.15.49:0/1857593603 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.586+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/1857593603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f662f430 0x55d7f6633f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.586+0000 7f56fffff640 1 -- 172.21.15.49:0/1857593603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f662f430 msgr2=0x55d7f6633f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:11.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.586+0000 7f56fffff640 1 --2- 172.21.15.49:0/1857593603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f662f430 0x55d7f6633f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.586+0000 7f56fffff640 1 -- 172.21.15.49:0/1857593603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f66345a0 msgr2=0x55d7f6636a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:11.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.586+0000 7f56fffff640 1 --2- 172.21.15.49:0/1857593603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f66345a0 0x55d7f6636a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.586+0000 7f56fffff640 1 -- 172.21.15.49:0/1857593603 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f65f6500 con 0x55d7f662ea40 2024-03-18T00:52:11.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.588+0000 7f56fffff640 1 --2- 172.21.15.49:0/1857593603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f662ee60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f56f400cc70 tx=0x7f56f400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb869975b572ddb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.589+0000 7f56e67fc640 1 -- 172.21.15.49:0/1857593603 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4020070 con 0x55d7f662ea40 2024-03-18T00:52:11.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.589+0000 7f56e67fc640 1 -- 172.21.15.49:0/1857593603 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56f4016070 con 0x55d7f662ea40 2024-03-18T00:52:11.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.589+0000 7f56e67fc640 1 -- 172.21.15.49:0/1857593603 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f400e750 con 0x55d7f662ea40 2024-03-18T00:52:11.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.593+0000 7f57091f3740 1 -- 172.21.15.49:0/1857593603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 msgr2=0x55d7f662ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.593+0000 7f57091f3740 1 --2- 172.21.15.49:0/1857593603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f662ee60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f56f400cc70 tx=0x7f56f400f000 comp rx=0 tx=0).stop 2024-03-18T00:52:11.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.594+0000 7f57091f3740 1 -- 172.21.15.49:0/1857593603 shutdown_connections 2024-03-18T00:52:11.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.594+0000 7f57091f3740 1 --2- 172.21.15.49:0/1857593603 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f66345a0 0x55d7f6636a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.594+0000 7f57091f3740 1 --2- 172.21.15.49:0/1857593603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f662f430 0x55d7f6633f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.594+0000 7f57091f3740 1 --2- 172.21.15.49:0/1857593603 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f662ee60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.594+0000 7f57091f3740 1 -- 172.21.15.49:0/1857593603 >> 172.21.15.49:0/1857593603 conn(0x55d7f6629fb0 msgr2=0x55d7f662a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.597+0000 7f57091f3740 1 -- 172.21.15.49:0/1857593603 shutdown_connections 2024-03-18T00:52:11.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.600+0000 7f57091f3740 1 -- 172.21.15.49:0/1857593603 wait complete. 2024-03-18T00:52:11.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.601+0000 7f57091f3740 1 -- start start 2024-03-18T00:52:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.602+0000 7f57091f3740 1 Processor -- start 2024-03-18T00:52:11.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.603+0000 7f57091f3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f662ea40 0x55d7f6638fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.603+0000 7f57091f3740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662f430 0x55d7f66394f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.603+0000 7f57091f3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f66345a0 0x55d7f6640530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f57091f3740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6637550 con 0x55d7f662f430 2024-03-18T00:52:11.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f57091f3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d7f654f230 con 0x55d7f66345a0 2024-03-18T00:52:11.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f57091f3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d7f66376d0 con 0x55d7f662ea40 2024-03-18T00:52:11.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f56ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662f430 0x55d7f66394f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f56ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662f430 0x55d7f66394f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40154/0 (socket says 172.21.15.49:40154) 2024-03-18T00:52:11.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f56ff7fe640 1 -- 172.21.15.49:0/791596211 learned_addr learned my addr 172.21.15.49:0/791596211 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f57091f3740 10 client.0 _ll_register_callbacks cb 0x55d7f658b010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:11.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f56ff7fe640 1 -- 172.21.15.49:0/791596211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f662ea40 msgr2=0x55d7f6638fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:11.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f662ea40 0x55d7f6638fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.604+0000 7f56ff7fe640 1 -- 172.21.15.49:0/791596211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f66345a0 msgr2=0x55d7f6640530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:11.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.605+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f66345a0 0x55d7f6640530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.605+0000 7f56ff7fe640 1 -- 172.21.15.49:0/791596211 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f655e100 con 0x55d7f662f430 2024-03-18T00:52:11.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.605+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662f430 0x55d7f66394f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f56f0006010 tx=0x7f56f00be360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3cbb860b83fb4a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.605+0000 7f56df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:11.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.610+0000 7f56df7fe640 1 -- 172.21.15.49:0/791596211 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f00c6400 con 0x55d7f662f430 2024-03-18T00:52:11.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.610+0000 7f56df7fe640 1 -- 172.21.15.49:0/791596211 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f56f000f3f0 con 0x55d7f662f430 2024-03-18T00:52:11.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.610+0000 7f57091f3740 1 -- 172.21.15.49:0/791596211 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56f4016070 con 0x55d7f662f430 2024-03-18T00:52:11.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.612+0000 7f56df7fe640 1 -- 172.21.15.49:0/791596211 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f56f00dc280 con 0x55d7f662f430 2024-03-18T00:52:11.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.613+0000 7f57091f3740 10 client.15978 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:11.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.613+0000 7f57091f3740 1 -- 172.21.15.49:0/791596211 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d7f62a73f0 con 0x55d7f662f430 2024-03-18T00:52:11.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.614+0000 7f56df7fe640 1 -- 172.21.15.49:0/791596211 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56f00d5030 con 0x55d7f662f430 2024-03-18T00:52:11.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.614+0000 7f56df7fe640 1 -- 172.21.15.49:0/791596211 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f56f0049030 con 0x55d7f662f430 2024-03-18T00:52:11.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.615+0000 7f57091f3740 10 client.15978 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:11.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.615+0000 7f57091f3740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:11.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.615+0000 7f57091f3740 20 client.15978 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:11.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.615+0000 7f57091f3740 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d7f6644dc0 0x55d7f6647220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.615+0000 7f57091f3740 4 client.15978 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:11.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.615+0000 7f57091f3740 1 -- 172.21.15.49:0/791596211 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f56f4016070 con 0x55d7f6644dc0 2024-03-18T00:52:11.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.620+0000 7f56fffff640 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d7f6644dc0 0x55d7f6647220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.635+0000 7f56fffff640 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d7f6644dc0 0x55d7f6647220 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f490e7315834f5cf server_cookie=59c25b92670bd284 in_seq=0 out_seq=0 2024-03-18T00:52:11.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.635+0000 7f56df7fe640 10 client.15978 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:11.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.642+0000 7f56df7fe640 1 -- 172.21.15.49:0/791596211 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f56f4016070 con 0x55d7f6644dc0 2024-03-18T00:52:11.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.642+0000 7f56df7fe640 10 client.15978 handle_command_reply: tid=0 2024-03-18T00:52:11.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.642+0000 7f57091f3740 1 client.15978 shutdown 2024-03-18T00:52:11.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.645+0000 7f57091f3740 1 -- 172.21.15.49:0/791596211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662f430 msgr2=0x55d7f66394f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.645+0000 7f57091f3740 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662f430 0x55d7f66394f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f56f0006010 tx=0x7f56f00be360 comp rx=0 tx=0).stop 2024-03-18T00:52:11.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.645+0000 7f57091f3740 1 -- 172.21.15.49:0/791596211 shutdown_connections 2024-03-18T00:52:11.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.645+0000 7f57091f3740 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d7f6644dc0 0x55d7f6647220 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.645+0000 7f57091f3740 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f66345a0 0x55d7f6640530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.646+0000 7f57091f3740 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662f430 0x55d7f66394f0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.646+0000 7f57091f3740 1 --2- 172.21.15.49:0/791596211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f662ea40 0x55d7f6638fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.646+0000 7f57091f3740 1 -- 172.21.15.49:0/791596211 >> 172.21.15.49:0/791596211 conn(0x55d7f6629fb0 msgr2=0x55d7f6629140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.647+0000 7f56df7fe640 0 client.15978 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:11.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.648+0000 7f57091f3740 1 -- 172.21.15.49:0/791596211 shutdown_connections 2024-03-18T00:52:11.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.649+0000 7f57091f3740 1 -- 172.21.15.49:0/791596211 wait complete. 2024-03-18T00:52:11.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.649+0000 7f57091f3740 20 client.15978 trim_cache size 0 max 16384 2024-03-18T00:52:11.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.677+0000 7f57091f3740 1 Processor -- start 2024-03-18T00:52:11.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f57091f3740 1 -- start start 2024-03-18T00:52:11.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f57091f3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f6629fb0 0x55d7f6638990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f57091f3740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f66404f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f57091f3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f662f430 0x55d7f66288b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f57091f3740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec00e600 con 0x55d7f662ea40 2024-03-18T00:52:11.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f57091f3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec010170 con 0x55d7f662f430 2024-03-18T00:52:11.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f57091f3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec00ecd0 con 0x55d7f6629fb0 2024-03-18T00:52:11.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f56ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f66404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f56ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f66404f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40170/0 (socket says 172.21.15.49:40170) 2024-03-18T00:52:11.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.678+0000 7f56ff7fe640 1 -- 172.21.15.49:0/1132562641 learned_addr learned my addr 172.21.15.49:0/1132562641 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.679+0000 7f56ff7fe640 1 -- 172.21.15.49:0/1132562641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f6629fb0 msgr2=0x55d7f6638990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:11.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.679+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/1132562641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f6629fb0 0x55d7f6638990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.679+0000 7f56ff7fe640 1 -- 172.21.15.49:0/1132562641 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f662f430 msgr2=0x55d7f66288b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:11.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.679+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/1132562641 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f662f430 0x55d7f66288b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.679+0000 7f56ff7fe640 1 -- 172.21.15.49:0/1132562641 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f63717d0 con 0x55d7f662ea40 2024-03-18T00:52:11.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.679+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/1132562641 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f66404f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f56f00bdbf0 tx=0x7f56f0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee9d2d9567f1064 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.679+0000 7f56e5ffb640 1 -- 172.21.15.49:0/1132562641 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f00cb080 con 0x55d7f662ea40 2024-03-18T00:52:11.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.679+0000 7f56e5ffb640 1 -- 172.21.15.49:0/1132562641 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56f00d5040 con 0x55d7f662ea40 2024-03-18T00:52:11.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.680+0000 7f56e5ffb640 1 -- 172.21.15.49:0/1132562641 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f000f3f0 con 0x55d7f662ea40 2024-03-18T00:52:11.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.681+0000 7f57091f3740 1 -- 172.21.15.49:0/1132562641 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 msgr2=0x55d7f66404f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.681+0000 7f57091f3740 1 --2- 172.21.15.49:0/1132562641 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f66404f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f56f00bdbf0 tx=0x7f56f0005c00 comp rx=0 tx=0).stop 2024-03-18T00:52:11.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.681+0000 7f57091f3740 1 -- 172.21.15.49:0/1132562641 shutdown_connections 2024-03-18T00:52:11.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.681+0000 7f57091f3740 1 --2- 172.21.15.49:0/1132562641 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f662f430 0x55d7f66288b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.681+0000 7f57091f3740 1 --2- 172.21.15.49:0/1132562641 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f66404f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.681+0000 7f57091f3740 1 --2- 172.21.15.49:0/1132562641 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f6629fb0 0x55d7f6638990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.681+0000 7f57091f3740 1 -- 172.21.15.49:0/1132562641 >> 172.21.15.49:0/1132562641 conn(0x55d7f6630210 msgr2=0x55d7f6638280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.682+0000 7f57091f3740 1 -- 172.21.15.49:0/1132562641 shutdown_connections 2024-03-18T00:52:11.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.682+0000 7f57091f3740 1 -- 172.21.15.49:0/1132562641 wait complete. 2024-03-18T00:52:11.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.683+0000 7f57091f3740 1 -- start start 2024-03-18T00:52:11.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.683+0000 7f57091f3740 1 Processor -- start 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f57091f3740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f6629fb0 0x55d7f663e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f57091f3740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f663e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f57091f3740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f662f430 0x55d7f663f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f57091f3740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec00fa70 con 0x55d7f662ea40 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f57091f3740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec00f3a0 con 0x55d7f662f430 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f57091f3740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f56ec00c5e0 con 0x55d7f6629fb0 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f57091f3740 10 client.0 _ll_register_callbacks cb 0x7f56f0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f56ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f663e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f56ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f663e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40176/0 (socket says 172.21.15.49:40176) 2024-03-18T00:52:11.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.684+0000 7f56ff7fe640 1 -- 172.21.15.49:0/2034105547 learned_addr learned my addr 172.21.15.49:0/2034105547 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:11.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f56ff7fe640 1 -- 172.21.15.49:0/2034105547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f6629fb0 msgr2=0x55d7f663e080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:11.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f6629fb0 0x55d7f663e080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f56ff7fe640 1 -- 172.21.15.49:0/2034105547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f662f430 msgr2=0x55d7f663f560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:11.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f662f430 0x55d7f663f560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f56ff7fe640 1 -- 172.21.15.49:0/2034105547 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f663fb50 con 0x55d7f662ea40 2024-03-18T00:52:11.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f56ff7fe640 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f663e5c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f56f00ba5e0 tx=0x7f56f00bc170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93bbb6c5c30222e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:11.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f56df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:11.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f56df7fe640 1 -- 172.21.15.49:0/2034105547 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f00cb060 con 0x55d7f662ea40 2024-03-18T00:52:11.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f57091f3740 1 -- 172.21.15.49:0/2034105547 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56f00d5040 con 0x55d7f662ea40 2024-03-18T00:52:11.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.685+0000 7f56df7fe640 1 -- 172.21.15.49:0/2034105547 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f56f00b9bc0 con 0x55d7f662ea40 2024-03-18T00:52:11.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.686+0000 7f56df7fe640 1 -- 172.21.15.49:0/2034105547 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f56f00c2cd0 con 0x55d7f662ea40 2024-03-18T00:52:11.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.686+0000 7f57091f3740 10 client.15984 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:11.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.686+0000 7f57091f3740 1 -- 172.21.15.49:0/2034105547 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d7f663fb50 con 0x55d7f662ea40 2024-03-18T00:52:11.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.686+0000 7f56df7fe640 1 -- 172.21.15.49:0/2034105547 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56f00d5040 con 0x55d7f662ea40 2024-03-18T00:52:11.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.687+0000 7f56df7fe640 1 -- 172.21.15.49:0/2034105547 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f56f0049040 con 0x55d7f662ea40 2024-03-18T00:52:11.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.687+0000 7f57091f3740 10 client.15984 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:11.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.687+0000 7f57091f3740 10 client.15984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:11.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.687+0000 7f57091f3740 20 client.15984 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:11.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.687+0000 7f57091f3740 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d7f66345a0 0x55d7f66adba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:11.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.687+0000 7f57091f3740 4 client.15984 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:11.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.687+0000 7f57091f3740 1 -- 172.21.15.49:0/2034105547 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d7f65a09d0 con 0x55d7f66345a0 2024-03-18T00:52:11.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.690+0000 7f56fffff640 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d7f66345a0 0x55d7f66adba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:11.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.692+0000 7f56df7fe640 10 client.15984 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:11.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.692+0000 7f56fffff640 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d7f66345a0 0x55d7f66adba0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb49906ee82fee19 server_cookie=d641789defa923ef in_seq=0 out_seq=0 2024-03-18T00:52:11.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f56df7fe640 1 -- 172.21.15.49:0/2034105547 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d7f65a09d0 con 0x55d7f66345a0 2024-03-18T00:52:11.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f56df7fe640 10 client.15984 handle_command_reply: tid=0 2024-03-18T00:52:11.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f57091f3740 1 client.15984 shutdown 2024-03-18T00:52:11.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f57091f3740 1 -- 172.21.15.49:0/2034105547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 msgr2=0x55d7f663e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f57091f3740 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f663e5c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f56f00ba5e0 tx=0x7f56f00bc170 comp rx=0 tx=0).stop 2024-03-18T00:52:11.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f57091f3740 1 -- 172.21.15.49:0/2034105547 shutdown_connections 2024-03-18T00:52:11.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f57091f3740 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d7f66345a0 0x55d7f66adba0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f56df7fe640 0 client.15984 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:11.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f57091f3740 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d7f662f430 0x55d7f663f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f57091f3740 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d7f662ea40 0x55d7f663e5c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.696+0000 7f57091f3740 1 --2- 172.21.15.49:0/2034105547 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d7f6629fb0 0x55d7f663e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.697+0000 7f57091f3740 1 -- 172.21.15.49:0/2034105547 >> 172.21.15.49:0/2034105547 conn(0x55d7f6630210 msgr2=0x55d7f66288b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.697+0000 7f57091f3740 1 -- 172.21.15.49:0/2034105547 shutdown_connections 2024-03-18T00:52:11.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.697+0000 7f57091f3740 1 -- 172.21.15.49:0/2034105547 wait complete. 2024-03-18T00:52:11.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.698+0000 7f57091f3740 20 client.15984 trim_cache size 0 max 16384 2024-03-18T00:52:11.698 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:11.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 -- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f56e80a2a10 msgr2=0x7f56e80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f56e80a2a10 0x7f56e80a4ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5700101610 tx=0x7f56f4005ba0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 -- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5700102170 msgr2=0x7f570010e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:11.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5700102170 0x7f570010e310 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f56f000fa60 tx=0x7f56f000eee0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 -- 172.21.15.49:0/1577816351 shutdown_connections 2024-03-18T00:52:11.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f56e80a2a10 0x7f56e80a4ed0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f570010e850 0x7f57001ae070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5700102170 0x7f570010e310 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 --2- 172.21.15.49:0/1577816351 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57001005c0 0x7f57001118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:11.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 -- 172.21.15.49:0/1577816351 >> 172.21.15.49:0/1577816351 conn(0x7f57000fbb50 msgr2=0x7f57000fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:11.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 -- 172.21.15.49:0/1577816351 shutdown_connections 2024-03-18T00:52:11.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:11.699+0000 7f57052d0640 1 -- 172.21.15.49:0/1577816351 wait complete. 2024-03-18T00:52:12.726 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:52:12.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:52:12.726 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:52:13.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.265+0000 7fc22f59e640 1 -- 172.21.15.49:0/1600917120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc230072bf0 msgr2=0x7fc230073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:13.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.265+0000 7fc22f59e640 1 --2- 172.21.15.49:0/1600917120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc230072bf0 0x7fc230073010 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc228009ca0 tx=0x7fc22802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.267+0000 7fc22f59e640 1 -- 172.21.15.49:0/1600917120 shutdown_connections 2024-03-18T00:52:13.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.267+0000 7fc22f59e640 1 --2- 172.21.15.49:0/1600917120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc230108eb0 0x7fc230109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.267+0000 7fc22f59e640 1 --2- 172.21.15.49:0/1600917120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc230109870 0x7fc23010e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.267+0000 7fc22f59e640 1 --2- 172.21.15.49:0/1600917120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc230072bf0 0x7fc230073010 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc228009ca0 tx=0x7fc22802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.267+0000 7fc22f59e640 1 -- 172.21.15.49:0/1600917120 >> 172.21.15.49:0/1600917120 conn(0x7fc23006e6c0 msgr2=0x7fc230070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:13.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.271+0000 7fc22f59e640 1 -- 172.21.15.49:0/1600917120 shutdown_connections 2024-03-18T00:52:13.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.271+0000 7fc22f59e640 1 -- 172.21.15.49:0/1600917120 wait complete. 2024-03-18T00:52:13.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.273+0000 7fc22f59e640 1 Processor -- start 2024-03-18T00:52:13.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.274+0000 7fc22f59e640 1 -- start start 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22f59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc230108eb0 0x7fc230115040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22f59e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc230109870 0x7fc230115580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22f59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301b4860 0x7fc2301b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22f59e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc23011c180 con 0x7fc230108eb0 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22f59e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc23011c000 con 0x7fc230109870 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22f59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc23011c300 con 0x7fc2301b4860 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc230108eb0 0x7fc230115040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc230109870 0x7fc230115580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc230109870 0x7fc230115580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48100/0 (socket says 172.21.15.49:48100) 2024-03-18T00:52:13.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22dd9b640 1 -- 172.21.15.49:0/2754602460 learned_addr learned my addr 172.21.15.49:0/2754602460 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:13.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.275+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc230108eb0 0x7fc230115040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40200/0 (socket says 172.21.15.49:40200) 2024-03-18T00:52:13.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.276+0000 7fc22dd9b640 1 -- 172.21.15.49:0/2754602460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301b4860 msgr2=0x7fc2301b6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:13.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.276+0000 7fc22dd9b640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301b4860 0x7fc2301b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.277+0000 7fc22dd9b640 1 -- 172.21.15.49:0/2754602460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc230108eb0 msgr2=0x7fc230115040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:13.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.277+0000 7fc22dd9b640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc230108eb0 0x7fc230115040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.277+0000 7fc22dd9b640 1 -- 172.21.15.49:0/2754602460 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2301b7190 con 0x7fc230109870 2024-03-18T00:52:13.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.293+0000 7fc22dd9b640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc230109870 0x7fc230115580 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc22400d950 tx=0x7fc22400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:13.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.299+0000 7fc21f7fe640 1 -- 172.21.15.49:0/2754602460 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224017070 con 0x7fc230109870 2024-03-18T00:52:13.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.299+0000 7fc21f7fe640 1 -- 172.21.15.49:0/2754602460 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc224009c30 con 0x7fc230109870 2024-03-18T00:52:13.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.299+0000 7fc21f7fe640 1 -- 172.21.15.49:0/2754602460 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22400f6b0 con 0x7fc230109870 2024-03-18T00:52:13.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.301+0000 7fc22f59e640 1 -- 172.21.15.49:0/2754602460 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2301b7480 con 0x7fc230109870 2024-03-18T00:52:13.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.301+0000 7fc22f59e640 1 -- 172.21.15.49:0/2754602460 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2301b7980 con 0x7fc230109870 2024-03-18T00:52:13.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.307+0000 7fc237a7a740 1 Processor -- start 2024-03-18T00:52:13.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.310+0000 7fc237a7a740 1 -- start start 2024-03-18T00:52:13.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.311+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb8f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.311+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 0x555d32bbe040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.311+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 0x555d32bc0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.311+0000 7fc237a7a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555d32b1d0d0 con 0x555d32bb9540 2024-03-18T00:52:13.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.311+0000 7fc237a7a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555d32ac5c70 con 0x555d32bb8b50 2024-03-18T00:52:13.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.311+0000 7fc237a7a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555d32b19170 con 0x555d32bbe6b0 2024-03-18T00:52:13.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.315+0000 7fc22ed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.316+0000 7fc22dd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 0x555d32bc0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.318+0000 7fc22ed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48102/0 (socket says 172.21.15.49:48102) 2024-03-18T00:52:13.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.318+0000 7fc22ed9d640 1 -- 172.21.15.49:0/1399091934 learned_addr learned my addr 172.21.15.49:0/1399091934 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:13.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.322+0000 7fc21f7fe640 1 -- 172.21.15.49:0/2754602460 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc224005040 con 0x7fc230109870 2024-03-18T00:52:13.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.324+0000 7fc22e59c640 1 --2- 172.21.15.49:0/1399091934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 0x555d32bbe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.324+0000 7fc22ed9d640 1 -- 172.21.15.49:0/1399091934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 msgr2=0x555d32bc0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:13.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.324+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/1399091934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 0x555d32bc0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.324+0000 7fc22ed9d640 1 -- 172.21.15.49:0/1399091934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 msgr2=0x555d32bbe040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:13.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.324+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/1399091934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 0x555d32bbe040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.324+0000 7fc22ed9d640 1 -- 172.21.15.49:0/1399091934 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d328f97e0 con 0x555d32bb8b50 2024-03-18T00:52:13.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.324+0000 7fc21f7fe640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2180a2a10 0x7fc2180a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.325+0000 7fc21f7fe640 1 -- 172.21.15.49:0/2754602460 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc2240d19a0 con 0x7fc230109870 2024-03-18T00:52:13.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.325+0000 7fc22e59c640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2180a2a10 0x7fc2180a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.325+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/1399091934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb8f70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc22000a9f0 tx=0x7fc22000aeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d4a8dcf9e20db03 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:13.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.326+0000 7fc21cff9640 1 -- 172.21.15.49:0/1399091934 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22000cde0 con 0x555d32bb8b50 2024-03-18T00:52:13.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.327+0000 7fc21cff9640 1 -- 172.21.15.49:0/1399091934 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc22000f930 con 0x555d32bb8b50 2024-03-18T00:52:13.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.328+0000 7fc237a7a740 1 -- 172.21.15.49:0/1399091934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 msgr2=0x555d32bb8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:13.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.329+0000 7fc237a7a740 1 --2- 172.21.15.49:0/1399091934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb8f70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc22000a9f0 tx=0x7fc22000aeb0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.330+0000 7fc22e59c640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2180a2a10 0x7fc2180a4ed0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc2280324c0 tx=0x7fc228000f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:13.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.336+0000 7fc237a7a740 1 -- 172.21.15.49:0/1399091934 shutdown_connections 2024-03-18T00:52:13.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.336+0000 7fc237a7a740 1 --2- 172.21.15.49:0/1399091934 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 0x555d32bc0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.336+0000 7fc237a7a740 1 --2- 172.21.15.49:0/1399091934 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 0x555d32bbe040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.336+0000 7fc237a7a740 1 --2- 172.21.15.49:0/1399091934 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb8f70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.336+0000 7fc237a7a740 1 -- 172.21.15.49:0/1399091934 >> 172.21.15.49:0/1399091934 conn(0x555d32b80c50 msgr2=0x555d32bb4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:13.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.337+0000 7fc237a7a740 1 -- 172.21.15.49:0/1399091934 shutdown_connections 2024-03-18T00:52:13.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.340+0000 7fc237a7a740 1 -- 172.21.15.49:0/1399091934 wait complete. 2024-03-18T00:52:13.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.341+0000 7fc237a7a740 1 -- start start 2024-03-18T00:52:13.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.342+0000 7fc237a7a740 1 Processor -- start 2024-03-18T00:52:13.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.343+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bb8b50 0x555d32bc2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.343+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb9540 0x555d32bc3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.343+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bbe6b0 0x555d32bca470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.344+0000 7fc237a7a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555d32bc16b0 con 0x555d32bbe6b0 2024-03-18T00:52:13.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.344+0000 7fc237a7a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555d32ad90d0 con 0x555d32bb9540 2024-03-18T00:52:13.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.344+0000 7fc237a7a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555d32bc1830 con 0x555d32bb8b50 2024-03-18T00:52:13.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.345+0000 7fc237a7a740 10 client.0 _ll_register_callbacks cb 0x555d32aa2760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:13.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.346+0000 7fc22ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bbe6b0 0x555d32bca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.346+0000 7fc22ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bbe6b0 0x555d32bca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40214/0 (socket says 172.21.15.49:40214) 2024-03-18T00:52:13.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.346+0000 7fc22ed9d640 1 -- 172.21.15.49:0/79264129 learned_addr learned my addr 172.21.15.49:0/79264129 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:13.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.347+0000 7fc22dd9b640 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb9540 0x555d32bc3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.347+0000 7fc22e59c640 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bb8b50 0x555d32bc2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.350+0000 7fc22ed9d640 1 -- 172.21.15.49:0/79264129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bb8b50 msgr2=0x555d32bc2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:13.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.350+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bb8b50 0x555d32bc2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.350+0000 7fc22ed9d640 1 -- 172.21.15.49:0/79264129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb9540 msgr2=0x555d32bc3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:13.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.350+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb9540 0x555d32bc3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.350+0000 7fc22ed9d640 1 -- 172.21.15.49:0/79264129 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d32959f70 con 0x555d32bbe6b0 2024-03-18T00:52:13.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.350+0000 7fc22dd9b640 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb9540 0x555d32bc3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:52:13.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.359+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bbe6b0 0x555d32bca470 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc22000f660 tx=0x7fc22000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766aa3964c8b0eb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:13.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.362+0000 7fc215ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:13.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.363+0000 7fc215ffb640 1 -- 172.21.15.49:0/79264129 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220026060 con 0x555d32bbe6b0 2024-03-18T00:52:13.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.363+0000 7fc237a7a740 1 -- 172.21.15.49:0/79264129 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc22000f930 con 0x555d32bbe6b0 2024-03-18T00:52:13.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.363+0000 7fc215ffb640 1 -- 172.21.15.49:0/79264129 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc22000fcf0 con 0x555d32bbe6b0 2024-03-18T00:52:13.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.367+0000 7fc215ffb640 1 -- 172.21.15.49:0/79264129 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc220004040 con 0x555d32bbe6b0 2024-03-18T00:52:13.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.368+0000 7fc237a7a740 10 client.15993 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:13.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.368+0000 7fc237a7a740 1 -- 172.21.15.49:0/79264129 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555d327f1170 con 0x555d32bbe6b0 2024-03-18T00:52:13.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.371+0000 7fc215ffb640 1 -- 172.21.15.49:0/79264129 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc22000f930 con 0x555d32bbe6b0 2024-03-18T00:52:13.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.373+0000 7fc215ffb640 1 -- 172.21.15.49:0/79264129 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fc22002c400 con 0x555d32bbe6b0 2024-03-18T00:52:13.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.373+0000 7fc237a7a740 10 client.15993 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:13.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.373+0000 7fc237a7a740 10 client.15993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:13.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.374+0000 7fc237a7a740 20 client.15993 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:13.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.374+0000 7fc237a7a740 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d32bce5d0 0x555d32bd09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.374+0000 7fc237a7a740 4 client.15993 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:13.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.374+0000 7fc237a7a740 1 -- 172.21.15.49:0/79264129 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d32b2a610 con 0x555d32bce5d0 2024-03-18T00:52:13.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:13 smithi049 ceph-mon[32130]: pgmap v413: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 820 MiB/s wr, 1.57k op/s 2024-03-18T00:52:13.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.378+0000 7fc22e59c640 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d32bce5d0 0x555d32bd09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.387+0000 7fc215ffb640 10 client.15993 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:13.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.387+0000 7fc22e59c640 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d32bce5d0 0x555d32bd09c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55e998735e8a8a9c server_cookie=d8f6538d43f8fe71 in_seq=0 out_seq=0 2024-03-18T00:52:13.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.391+0000 7fc215ffb640 1 -- 172.21.15.49:0/79264129 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555d32b2a610 con 0x555d32bce5d0 2024-03-18T00:52:13.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.391+0000 7fc215ffb640 10 client.15993 handle_command_reply: tid=0 2024-03-18T00:52:13.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.391+0000 7fc237a7a740 1 client.15993 shutdown 2024-03-18T00:52:13.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.394+0000 7fc237a7a740 1 -- 172.21.15.49:0/79264129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bbe6b0 msgr2=0x555d32bca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:13.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.394+0000 7fc237a7a740 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bbe6b0 0x555d32bca470 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc22000f660 tx=0x7fc22000f690 comp rx=0 tx=0).stop 2024-03-18T00:52:13.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.394+0000 7fc237a7a740 1 -- 172.21.15.49:0/79264129 shutdown_connections 2024-03-18T00:52:13.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.394+0000 7fc237a7a740 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d32bce5d0 0x555d32bd09c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.394+0000 7fc237a7a740 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb9540 0x555d32bc3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.394+0000 7fc237a7a740 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bbe6b0 0x555d32bca470 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.394+0000 7fc237a7a740 1 --2- 172.21.15.49:0/79264129 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bb8b50 0x555d32bc2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.394+0000 7fc237a7a740 1 -- 172.21.15.49:0/79264129 >> 172.21.15.49:0/79264129 conn(0x555d32b80c50 msgr2=0x555d32bb36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:13.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.394+0000 7fc215ffb640 0 client.15993 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:13.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.405+0000 7fc237a7a740 1 -- 172.21.15.49:0/79264129 shutdown_connections 2024-03-18T00:52:13.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.409+0000 7fc237a7a740 1 -- 172.21.15.49:0/79264129 wait complete. 2024-03-18T00:52:13.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.410+0000 7fc237a7a740 20 client.15993 trim_cache size 0 max 16384 2024-03-18T00:52:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:13 smithi107 ceph-mon[36270]: pgmap v413: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 820 MiB/s wr, 1.57k op/s 2024-03-18T00:52:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:13 smithi106 ceph-mon[35899]: pgmap v413: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 820 MiB/s wr, 1.57k op/s 2024-03-18T00:52:13.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.465+0000 7fc237a7a740 1 Processor -- start 2024-03-18T00:52:13.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.466+0000 7fc237a7a740 1 -- start start 2024-03-18T00:52:13.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.466+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.466+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 0x555d32bca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.467+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 0x555d32bb44d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.467+0000 7fc237a7a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000e500 con 0x555d32bb9540 2024-03-18T00:52:13.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.467+0000 7fc237a7a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc210010070 con 0x555d32bb8b50 2024-03-18T00:52:13.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.467+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.467+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48128/0 (socket says 172.21.15.49:48128) 2024-03-18T00:52:13.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.467+0000 7fc22e59c640 1 -- 172.21.15.49:0/2476218656 learned_addr learned my addr 172.21.15.49:0/2476218656 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:13.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.467+0000 7fc237a7a740 1 -- 172.21.15.49:0/2476218656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000ebd0 con 0x555d32bbe6b0 2024-03-18T00:52:13.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.467+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/2476218656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 0x555d32bb44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.468+0000 7fc22dd9b640 1 --2- 172.21.15.49:0/2476218656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 0x555d32bca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.469+0000 7fc22ed9d640 1 -- 172.21.15.49:0/2476218656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 msgr2=0x555d32bb3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:13.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.469+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/2476218656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.469+0000 7fc22ed9d640 1 -- 172.21.15.49:0/2476218656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 msgr2=0x555d32bca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:13.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.469+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/2476218656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 0x555d32bca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.469+0000 7fc22ed9d640 1 -- 172.21.15.49:0/2476218656 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d32bd0cc0 con 0x555d32bbe6b0 2024-03-18T00:52:13.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.472+0000 7fc22ed9d640 1 --2- 172.21.15.49:0/2476218656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 0x555d32bb44d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc2200185e0 tx=0x7fc220018cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e23c4b44401034f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:13.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.476+0000 7fc217fff640 1 -- 172.21.15.49:0/2476218656 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220026050 con 0x555d32bbe6b0 2024-03-18T00:52:13.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.476+0000 7fc217fff640 1 -- 172.21.15.49:0/2476218656 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc220005df0 con 0x555d32bbe6b0 2024-03-18T00:52:13.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.476+0000 7fc217fff640 1 -- 172.21.15.49:0/2476218656 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220005020 con 0x555d32bbe6b0 2024-03-18T00:52:13.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.477+0000 7fc237a7a740 1 -- 172.21.15.49:0/2476218656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 msgr2=0x555d32bb44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:13.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.477+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2476218656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 0x555d32bb44d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc2200185e0 tx=0x7fc220018cf0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.478+0000 7fc237a7a740 1 -- 172.21.15.49:0/2476218656 shutdown_connections 2024-03-18T00:52:13.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.478+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2476218656 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bbe6b0 0x555d32bb44d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.478+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2476218656 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb9540 0x555d32bca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.478+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2476218656 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bb8b50 0x555d32bb3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.478+0000 7fc237a7a740 1 -- 172.21.15.49:0/2476218656 >> 172.21.15.49:0/2476218656 conn(0x555d32b80c50 msgr2=0x555d32bc2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:13.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.478+0000 7fc237a7a740 1 -- 172.21.15.49:0/2476218656 shutdown_connections 2024-03-18T00:52:13.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.478+0000 7fc237a7a740 1 -- 172.21.15.49:0/2476218656 wait complete. 2024-03-18T00:52:13.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.479+0000 7fc237a7a740 1 -- start start 2024-03-18T00:52:13.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.479+0000 7fc237a7a740 1 Processor -- start 2024-03-18T00:52:13.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb8b50 0x555d32bc7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bb9540 0x555d32bc8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc237a7a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bbe6b0 0x555d32bc9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc237a7a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000f970 con 0x555d32bb8b50 2024-03-18T00:52:13.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc237a7a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000f2a0 con 0x555d32bbe6b0 2024-03-18T00:52:13.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc237a7a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc21000c590 con 0x555d32bb9540 2024-03-18T00:52:13.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc237a7a740 10 client.0 _ll_register_callbacks cb 0x7fc2240d41e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb8b50 0x555d32bc7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc22e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb8b50 0x555d32bc7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40234/0 (socket says 172.21.15.49:40234) 2024-03-18T00:52:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.480+0000 7fc22e59c640 1 -- 172.21.15.49:0/2416812848 learned_addr learned my addr 172.21.15.49:0/2416812848 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.481+0000 7fc22e59c640 1 -- 172.21.15.49:0/2416812848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bb9540 msgr2=0x555d32bc8200 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.481+0000 7fc22e59c640 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bb9540 0x555d32bc8200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.481+0000 7fc22e59c640 1 -- 172.21.15.49:0/2416812848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bbe6b0 msgr2=0x555d32bc9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.481+0000 7fc22e59c640 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bbe6b0 0x555d32bc9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:13.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.481+0000 7fc22e59c640 1 -- 172.21.15.49:0/2416812848 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d32bc9800 con 0x555d32bb8b50 2024-03-18T00:52:13.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.482+0000 7fc22e59c640 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb8b50 0x555d32bc7cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc230073940 tx=0x7fc2280095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0524df875b5309 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:13.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.482+0000 7fc215ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:13.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.483+0000 7fc215ffb640 1 -- 172.21.15.49:0/2416812848 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc228055070 con 0x555d32bb8b50 2024-03-18T00:52:13.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.483+0000 7fc215ffb640 1 -- 172.21.15.49:0/2416812848 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc228003cf0 con 0x555d32bb8b50 2024-03-18T00:52:13.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.483+0000 7fc237a7a740 1 -- 172.21.15.49:0/2416812848 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc220005df0 con 0x555d32bb8b50 2024-03-18T00:52:13.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.484+0000 7fc215ffb640 1 -- 172.21.15.49:0/2416812848 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc22805e050 con 0x555d32bb8b50 2024-03-18T00:52:13.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.484+0000 7fc237a7a740 10 client.15999 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:13.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.484+0000 7fc237a7a740 1 -- 172.21.15.49:0/2416812848 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555d32c349c0 con 0x555d32bb8b50 2024-03-18T00:52:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.485+0000 7fc215ffb640 1 -- 172.21.15.49:0/2416812848 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc220005df0 con 0x555d32bb8b50 2024-03-18T00:52:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.485+0000 7fc215ffb640 1 -- 172.21.15.49:0/2416812848 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fc228004510 con 0x555d32bb8b50 2024-03-18T00:52:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.485+0000 7fc237a7a740 10 client.15999 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.485+0000 7fc237a7a740 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.485+0000 7fc237a7a740 20 client.15999 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.485+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d32c35450 0x555d32c378b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.485+0000 7fc237a7a740 4 client.15999 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:52:13.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.485+0000 7fc237a7a740 1 -- 172.21.15.49:0/2416812848 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555d32b24960 con 0x555d32c35450 2024-03-18T00:52:13.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.487+0000 7fc22dd9b640 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d32c35450 0x555d32c378b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:13.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.494+0000 7fc22dd9b640 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d32c35450 0x555d32c378b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb278b20d5a3ee4c server_cookie=fa9cd8fc3463c1b1 in_seq=0 out_seq=0 2024-03-18T00:52:13.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:13.494+0000 7fc215ffb640 10 client.15999 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:14.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.423+0000 7fc215ffb640 1 -- 172.21.15.49:0/2416812848 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555d32b24960 con 0x555d32c35450 2024-03-18T00:52:14.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.423+0000 7fc215ffb640 10 client.15999 handle_command_reply: tid=0 2024-03-18T00:52:14.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.423+0000 7fc237a7a740 1 client.15999 shutdown 2024-03-18T00:52:14.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 -- 172.21.15.49:0/2416812848 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb8b50 msgr2=0x555d32bc7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb8b50 0x555d32bc7cc0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc230073940 tx=0x7fc2280095c0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 -- 172.21.15.49:0/2416812848 shutdown_connections 2024-03-18T00:52:14.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d32c35450 0x555d32c378b0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d32bbe6b0 0x555d32bc9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc215ffb640 0 client.15999 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d32bb9540 0x555d32bc8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 --2- 172.21.15.49:0/2416812848 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d32bb8b50 0x555d32bc7cc0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 -- 172.21.15.49:0/2416812848 >> 172.21.15.49:0/2416812848 conn(0x555d32b80c50 msgr2=0x555d32bb7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 -- 172.21.15.49:0/2416812848 shutdown_connections 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 1 -- 172.21.15.49:0/2416812848 wait complete. 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.424+0000 7fc237a7a740 20 client.15999 trim_cache size 0 max 16384 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:52:14.427 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "2f6cef41-4cfb-4150-bbd7-005032be35ac", 2024-03-18T00:52:14.428 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:52:14.428 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:52:14.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 -- 172.21.15.49:0/2754602460 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2180a2a10 msgr2=0x7fc2180a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2180a2a10 0x7fc2180a4ed0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc2280324c0 tx=0x7fc228000f40 comp rx=0 tx=0).stop 2024-03-18T00:52:14.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 -- 172.21.15.49:0/2754602460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc230109870 msgr2=0x7fc230115580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc230109870 0x7fc230115580 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc22400d950 tx=0x7fc22400de10 comp rx=0 tx=0).stop 2024-03-18T00:52:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 -- 172.21.15.49:0/2754602460 shutdown_connections 2024-03-18T00:52:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2180a2a10 0x7fc2180a4ed0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc230109870 0x7fc230115580 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc2301b4860 0x7fc2301b6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 --2- 172.21.15.49:0/2754602460 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc230108eb0 0x7fc230115040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 -- 172.21.15.49:0/2754602460 >> 172.21.15.49:0/2754602460 conn(0x7fc23006e6c0 msgr2=0x7fc230070360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 -- 172.21.15.49:0/2754602460 shutdown_connections 2024-03-18T00:52:14.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.425+0000 7fc22f59e640 1 -- 172.21.15.49:0/2754602460 wait complete. 2024-03-18T00:52:14.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:52:14.447 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:52:14.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.783+0000 7f2861497640 1 -- 172.21.15.49:0/1430554809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f285c107e00 msgr2=0x7f285c108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.783+0000 7f2861497640 1 --2- 172.21.15.49:0/1430554809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f285c107e00 0x7f285c108220 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2854009ca0 tx=0x7f285402c140 comp rx=0 tx=0).stop 2024-03-18T00:52:14.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.783+0000 7f2861497640 1 -- 172.21.15.49:0/1430554809 shutdown_connections 2024-03-18T00:52:14.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.783+0000 7f2861497640 1 --2- 172.21.15.49:0/1430554809 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f285c108ff0 0x7f285c109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.784+0000 7f2861497640 1 --2- 172.21.15.49:0/1430554809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f285c1099b0 0x7f285c110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.784+0000 7f2861497640 1 --2- 172.21.15.49:0/1430554809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f285c107e00 0x7f285c108220 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.784+0000 7f2861497640 1 -- 172.21.15.49:0/1430554809 >> 172.21.15.49:0/1430554809 conn(0x7f285c076d90 msgr2=0x7f285c0791d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:14.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.785+0000 7f2861497640 1 -- 172.21.15.49:0/1430554809 shutdown_connections 2024-03-18T00:52:14.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.791+0000 7f2861497640 1 -- 172.21.15.49:0/1430554809 wait complete. 2024-03-18T00:52:14.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.792+0000 7f2861497640 1 Processor -- start 2024-03-18T00:52:14.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.795+0000 7f2861497640 1 -- start start 2024-03-18T00:52:14.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.795+0000 7f2861497640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f285c107e00 0x7f285c0702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.796+0000 7f2861497640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f285c108ff0 0x7f285c0707f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.796+0000 7f2861497640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f285c1099b0 0x7f285c1ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.796+0000 7f2861497640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f285c113be0 con 0x7f285c108ff0 2024-03-18T00:52:14.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.796+0000 7f2861497640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f285c113a60 con 0x7f285c107e00 2024-03-18T00:52:14.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.796+0000 7f2861497640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f285c113d60 con 0x7f285c1099b0 2024-03-18T00:52:14.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.799+0000 7f285affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f285c107e00 0x7f285c0702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.800+0000 7f285affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f285c107e00 0x7f285c0702b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48146/0 (socket says 172.21.15.49:48146) 2024-03-18T00:52:14.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.800+0000 7f285affd640 1 -- 172.21.15.49:0/1704296098 learned_addr learned my addr 172.21.15.49:0/1704296098 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:14.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.801+0000 7f285a7fc640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f285c108ff0 0x7f285c0707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.801+0000 7f285b7fe640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f285c1099b0 0x7f285c1ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.802+0000 7f285affd640 1 -- 172.21.15.49:0/1704296098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f285c1099b0 msgr2=0x7f285c1ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.802+0000 7f285affd640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f285c1099b0 0x7f285c1ae2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.802+0000 7f285affd640 1 -- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f285c108ff0 msgr2=0x7f285c0707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.802+0000 7f285affd640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f285c108ff0 0x7f285c0707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.802+0000 7f285affd640 1 -- 172.21.15.49:0/1704296098 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c07a030 con 0x7f285c107e00 2024-03-18T00:52:14.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.805+0000 7f285affd640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f285c107e00 0x7f285c0702b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f285402c750 tx=0x7f2854035d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:14.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.809+0000 7f2843fff640 1 -- 172.21.15.49:0/1704296098 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854039040 con 0x7f285c107e00 2024-03-18T00:52:14.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.809+0000 7f2843fff640 1 -- 172.21.15.49:0/1704296098 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2854044070 con 0x7f285c107e00 2024-03-18T00:52:14.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.809+0000 7f2861497640 1 -- 172.21.15.49:0/1704296098 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285c07a2c0 con 0x7f285c107e00 2024-03-18T00:52:14.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.809+0000 7f2861497640 1 -- 172.21.15.49:0/1704296098 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f285c07a830 con 0x7f285c107e00 2024-03-18T00:52:14.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.809+0000 7f2843fff640 1 -- 172.21.15.49:0/1704296098 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854004640 con 0x7f285c107e00 2024-03-18T00:52:14.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.816+0000 7f2864083740 1 Processor -- start 2024-03-18T00:52:14.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.817+0000 7f2864083740 1 -- start start 2024-03-18T00:52:14.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.817+0000 7f2864083740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03170 0x55b7b8a03590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.817+0000 7f2864083740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03b60 0x55b7b8a08660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.817+0000 7f2864083740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a0b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.817+0000 7f2864083740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b7b8968500 con 0x55b7b8a03170 2024-03-18T00:52:14.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.817+0000 7f2864083740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b7b8910be0 con 0x55b7b8a08cd0 2024-03-18T00:52:14.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.817+0000 7f2864083740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b7b89645a0 con 0x55b7b8a03b60 2024-03-18T00:52:14.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.818+0000 7f285b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03170 0x55b7b8a03590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.818+0000 7f285b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03170 0x55b7b8a03590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40266/0 (socket says 172.21.15.49:40266) 2024-03-18T00:52:14.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.818+0000 7f285b7fe640 1 -- 172.21.15.49:0/3738053629 learned_addr learned my addr 172.21.15.49:0/3738053629 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:14.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.820+0000 7f285b7fe640 1 -- 172.21.15.49:0/3738053629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03b60 msgr2=0x55b7b8a08660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:52:14.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.820+0000 7f285b7fe640 1 --2- 172.21.15.49:0/3738053629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03b60 0x55b7b8a08660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.820+0000 7f285b7fe640 1 -- 172.21.15.49:0/3738053629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a08cd0 msgr2=0x55b7b8a0b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:14.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.820+0000 7f285b7fe640 1 --2- 172.21.15.49:0/3738053629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a0b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.820+0000 7f285b7fe640 1 -- 172.21.15.49:0/3738053629 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7b89244b0 con 0x55b7b8a03170 2024-03-18T00:52:14.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.821+0000 7f285b7fe640 1 --2- 172.21.15.49:0/3738053629 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03170 0x55b7b8a03590 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f284c00a790 tx=0x7f284c00ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffeadf742a410dd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:14.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.821+0000 7f2843fff640 1 -- 172.21.15.49:0/1704296098 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f28540328d0 con 0x7f285c107e00 2024-03-18T00:52:14.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.823+0000 7f2843fff640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f28440a2980 0x7f28440a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.824+0000 7f2843fff640 1 -- 172.21.15.49:0/1704296098 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f28540f0990 con 0x7f285c107e00 2024-03-18T00:52:14.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.824+0000 7f285affd640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f28440a2980 0x7f28440a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.825+0000 7f285affd640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f28440a2980 0x7f28440a4e40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f28540b9800 tx=0x7f2854004140 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:14.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.827+0000 7f28417fa640 1 -- 172.21.15.49:0/3738053629 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c015070 con 0x55b7b8a03170 2024-03-18T00:52:14.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.827+0000 7f28417fa640 1 -- 172.21.15.49:0/3738053629 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f284c00cb40 con 0x55b7b8a03170 2024-03-18T00:52:14.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.829+0000 7f2864083740 1 -- 172.21.15.49:0/3738053629 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03170 msgr2=0x55b7b8a03590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.829+0000 7f2864083740 1 --2- 172.21.15.49:0/3738053629 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03170 0x55b7b8a03590 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f284c00a790 tx=0x7f284c00ac50 comp rx=0 tx=0).stop 2024-03-18T00:52:14.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.832+0000 7f2864083740 1 -- 172.21.15.49:0/3738053629 shutdown_connections 2024-03-18T00:52:14.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.832+0000 7f2864083740 1 --2- 172.21.15.49:0/3738053629 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a0b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.832+0000 7f2864083740 1 --2- 172.21.15.49:0/3738053629 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03b60 0x55b7b8a08660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.832+0000 7f2864083740 1 --2- 172.21.15.49:0/3738053629 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03170 0x55b7b8a03590 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.832+0000 7f2864083740 1 -- 172.21.15.49:0/3738053629 >> 172.21.15.49:0/3738053629 conn(0x55b7b89fe6e0 msgr2=0x55b7b89feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:14.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.833+0000 7f2864083740 1 -- 172.21.15.49:0/3738053629 shutdown_connections 2024-03-18T00:52:14.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.835+0000 7f2864083740 1 -- 172.21.15.49:0/3738053629 wait complete. 2024-03-18T00:52:14.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.836+0000 7f2864083740 1 -- start start 2024-03-18T00:52:14.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.837+0000 7f2864083740 1 Processor -- start 2024-03-18T00:52:14.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.837+0000 7f2864083740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03170 0x55b7b8a0d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.838+0000 7f2864083740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a0de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.838+0000 7f2864083740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a14ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.838+0000 7f2864083740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b7b8a0bcd0 con 0x55b7b8a03b60 2024-03-18T00:52:14.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.838+0000 7f2864083740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b7b8924160 con 0x55b7b8a03170 2024-03-18T00:52:14.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.838+0000 7f2864083740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b7b8a0be50 con 0x55b7b8a08cd0 2024-03-18T00:52:14.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.838+0000 7f2864083740 10 client.0 _ll_register_callbacks cb 0x55b7b8908150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:14.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.842+0000 7f285b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03170 0x55b7b8a0d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.842+0000 7f285b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a14ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.844+0000 7f285b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03170 0x55b7b8a0d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48160/0 (socket says 172.21.15.49:48160) 2024-03-18T00:52:14.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.844+0000 7f285b7fe640 1 -- 172.21.15.49:0/406360320 learned_addr learned my addr 172.21.15.49:0/406360320 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:14.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.845+0000 7f285a7fc640 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a0de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.849+0000 7f285b7fe640 1 -- 172.21.15.49:0/406360320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03170 msgr2=0x55b7b8a0d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:14.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.849+0000 7f285b7fe640 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03170 0x55b7b8a0d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.849+0000 7f285b7fe640 1 -- 172.21.15.49:0/406360320 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 msgr2=0x55b7b8a0de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:14.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.849+0000 7f285b7fe640 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a0de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.849+0000 7f285b7fe640 1 -- 172.21.15.49:0/406360320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7b87467d0 con 0x55b7b8a08cd0 2024-03-18T00:52:14.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.855+0000 7f285b7fe640 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a14ea0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f284c00a760 tx=0x7f284c008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b6fd28b871192cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:14.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.856+0000 7f283eff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:52:14.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.858+0000 7f283eff5640 1 -- 172.21.15.49:0/406360320 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c015070 con 0x55b7b8a08cd0 2024-03-18T00:52:14.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.858+0000 7f283eff5640 1 -- 172.21.15.49:0/406360320 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f284c018d90 con 0x55b7b8a08cd0 2024-03-18T00:52:14.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.859+0000 7f2864083740 1 -- 172.21.15.49:0/406360320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b7b8975a40 con 0x55b7b8a08cd0 2024-03-18T00:52:14.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.861+0000 7f283eff5640 1 -- 172.21.15.49:0/406360320 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f284c023040 con 0x55b7b8a08cd0 2024-03-18T00:52:14.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.862+0000 7f2864083740 10 client.25735 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:14.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.862+0000 7f2864083740 1 -- 172.21.15.49:0/406360320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b7b867c3f0 con 0x55b7b8a08cd0 2024-03-18T00:52:14.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.868+0000 7f283eff5640 1 -- 172.21.15.49:0/406360320 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b7b8975a40 con 0x55b7b8a08cd0 2024-03-18T00:52:14.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.868+0000 7f283eff5640 1 -- 172.21.15.49:0/406360320 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f284c031020 con 0x55b7b8a08cd0 2024-03-18T00:52:14.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.868+0000 7f2864083740 10 client.25735 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:14.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.868+0000 7f2864083740 10 client.25735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:14.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.868+0000 7f2864083740 20 client.25735 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:14.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.868+0000 7f2864083740 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b7b8a19230 0x55b7b8a1b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.868+0000 7f2864083740 4 client.25735 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:14.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.868+0000 7f2864083740 1 -- 172.21.15.49:0/406360320 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7b896fd90 con 0x55b7b8a19230 2024-03-18T00:52:14.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.878+0000 7f285a7fc640 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b7b8a19230 0x55b7b8a1b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.894+0000 7f285a7fc640 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b7b8a19230 0x55b7b8a1b620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ebc322890139729 server_cookie=ae8c8c1960544c1b in_seq=0 out_seq=0 2024-03-18T00:52:14.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.894+0000 7f283eff5640 10 client.25735 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:14.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.899+0000 7f283eff5640 1 -- 172.21.15.49:0/406360320 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b7b896fd90 con 0x55b7b8a19230 2024-03-18T00:52:14.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.899+0000 7f283eff5640 10 client.25735 handle_command_reply: tid=0 2024-03-18T00:52:14.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.899+0000 7f2864083740 1 client.25735 shutdown 2024-03-18T00:52:14.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 -- 172.21.15.49:0/406360320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a08cd0 msgr2=0x55b7b8a14ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a14ea0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f284c00a760 tx=0x7f284c008040 comp rx=0 tx=0).stop 2024-03-18T00:52:14.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 -- 172.21.15.49:0/406360320 shutdown_connections 2024-03-18T00:52:14.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b7b8a19230 0x55b7b8a1b620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a14ea0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a0de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 --2- 172.21.15.49:0/406360320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03170 0x55b7b8a0d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 -- 172.21.15.49:0/406360320 >> 172.21.15.49:0/406360320 conn(0x55b7b89fe6e0 msgr2=0x55b7b89fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:14.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f283eff5640 0 client.25735 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:14.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 -- 172.21.15.49:0/406360320 shutdown_connections 2024-03-18T00:52:14.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.901+0000 7f2864083740 1 -- 172.21.15.49:0/406360320 wait complete. 2024-03-18T00:52:14.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.906+0000 7f2864083740 20 client.25735 trim_cache size 0 max 16384 2024-03-18T00:52:14.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.948+0000 7f2864083740 1 Processor -- start 2024-03-18T00:52:14.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.948+0000 7f2864083740 1 -- start start 2024-03-18T00:52:14.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f2864083740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 0x55b7b8a04660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f2864083740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a14dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f2864083740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a05240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f2864083740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f284800e640 con 0x55b7b8a03b60 2024-03-18T00:52:14.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f2864083740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f28480101b0 con 0x55b7b8a08cd0 2024-03-18T00:52:14.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f2864083740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f284800ed10 con 0x55b7b8a03170 2024-03-18T00:52:14.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f285a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a14dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f285a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a14dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40288/0 (socket says 172.21.15.49:40288) 2024-03-18T00:52:14.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f285a7fc640 1 -- 172.21.15.49:0/3742742305 learned_addr learned my addr 172.21.15.49:0/3742742305 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:14.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.949+0000 7f285b7fe640 1 --2- 172.21.15.49:0/3742742305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 0x55b7b8a04660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.950+0000 7f285a7fc640 1 -- 172.21.15.49:0/3742742305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 msgr2=0x55b7b8a04660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:14.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.950+0000 7f285a7fc640 1 --2- 172.21.15.49:0/3742742305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 0x55b7b8a04660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.950+0000 7f285a7fc640 1 -- 172.21.15.49:0/3742742305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a08cd0 msgr2=0x55b7b8a05240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:14.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.950+0000 7f285a7fc640 1 --2- 172.21.15.49:0/3742742305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a05240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.950+0000 7f285a7fc640 1 -- 172.21.15.49:0/3742742305 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7b863c0f0 con 0x55b7b8a03b60 2024-03-18T00:52:14.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.950+0000 7f285a7fc640 1 --2- 172.21.15.49:0/3742742305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a14dc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2850004790 tx=0x7f285000b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2097a1eb03d9bb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:14.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.951+0000 7f2840ff9640 1 -- 172.21.15.49:0/3742742305 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28500192d0 con 0x55b7b8a03b60 2024-03-18T00:52:14.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.951+0000 7f2840ff9640 1 -- 172.21.15.49:0/3742742305 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f285001d030 con 0x55b7b8a03b60 2024-03-18T00:52:14.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.951+0000 7f2840ff9640 1 -- 172.21.15.49:0/3742742305 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850014440 con 0x55b7b8a03b60 2024-03-18T00:52:14.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 -- 172.21.15.49:0/3742742305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 msgr2=0x55b7b8a14dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 --2- 172.21.15.49:0/3742742305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a14dc0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2850004790 tx=0x7f285000b850 comp rx=0 tx=0).stop 2024-03-18T00:52:14.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 -- 172.21.15.49:0/3742742305 shutdown_connections 2024-03-18T00:52:14.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 --2- 172.21.15.49:0/3742742305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a05240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 --2- 172.21.15.49:0/3742742305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a03b60 0x55b7b8a14dc0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 --2- 172.21.15.49:0/3742742305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 0x55b7b8a04660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 -- 172.21.15.49:0/3742742305 >> 172.21.15.49:0/3742742305 conn(0x55b7b89fe6e0 msgr2=0x55b7b8a1b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 -- 172.21.15.49:0/3742742305 shutdown_connections 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 -- 172.21.15.49:0/3742742305 wait complete. 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.952+0000 7f2864083740 1 -- start start 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.953+0000 7f2864083740 1 Processor -- start 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.953+0000 7f2864083740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 0x55b7b8a14e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.953+0000 7f2864083740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03b60 0x55b7b8a11970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.953+0000 7f2864083740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a11eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.953+0000 7f2864083740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f284800fab0 con 0x55b7b8a08cd0 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.953+0000 7f2864083740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f284800f3e0 con 0x55b7b8a03b60 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.953+0000 7f2864083740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f284800c620 con 0x55b7b8a03170 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.953+0000 7f2864083740 10 client.0 _ll_register_callbacks cb 0x7f284c00a5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:14.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.954+0000 7f285a7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03b60 0x55b7b8a11970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.954+0000 7f285a7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03b60 0x55b7b8a11970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48168/0 (socket says 172.21.15.49:48168) 2024-03-18T00:52:14.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.954+0000 7f285a7fc640 1 -- 172.21.15.49:0/1993780714 learned_addr learned my addr 172.21.15.49:0/1993780714 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:14.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.954+0000 7f285b7fe640 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 0x55b7b8a14e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.955+0000 7f285a7fc640 1 -- 172.21.15.49:0/1993780714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 msgr2=0x55b7b8a14e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:14.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.955+0000 7f285a7fc640 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 0x55b7b8a14e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.955+0000 7f285a7fc640 1 -- 172.21.15.49:0/1993780714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a08cd0 msgr2=0x55b7b8a11eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:14.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.955+0000 7f285a7fc640 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a11eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.955+0000 7f285a7fc640 1 -- 172.21.15.49:0/1993780714 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7b892f9d0 con 0x55b7b8a03b60 2024-03-18T00:52:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.956+0000 7f285a7fc640 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03b60 0x55b7b8a11970 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55b7b8a17f10 tx=0x7f2850016f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb28c0ba74a25d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.956+0000 7f283eff5640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:52:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.956+0000 7f283eff5640 1 -- 172.21.15.49:0/1993780714 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850004020 con 0x55b7b8a03b60 2024-03-18T00:52:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.956+0000 7f2864083740 1 -- 172.21.15.49:0/1993780714 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f285001d030 con 0x55b7b8a03b60 2024-03-18T00:52:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.957+0000 7f283eff5640 1 -- 172.21.15.49:0/1993780714 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f28500041c0 con 0x55b7b8a03b60 2024-03-18T00:52:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.957+0000 7f283eff5640 1 -- 172.21.15.49:0/1993780714 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2850004420 con 0x55b7b8a03b60 2024-03-18T00:52:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.957+0000 7f2864083740 10 client.25673 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:14.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.957+0000 7f2864083740 1 -- 172.21.15.49:0/1993780714 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b7b8a7e4e0 con 0x55b7b8a03b60 2024-03-18T00:52:14.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.958+0000 7f283eff5640 1 -- 172.21.15.49:0/1993780714 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f285001d030 con 0x55b7b8a03b60 2024-03-18T00:52:14.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.960+0000 7f283eff5640 1 -- 172.21.15.49:0/1993780714 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f2850035080 con 0x55b7b8a03b60 2024-03-18T00:52:14.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.960+0000 7f2864083740 10 client.25673 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:14.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.960+0000 7f2864083740 10 client.25673 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:14.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.960+0000 7f2864083740 20 client.25673 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:14.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.960+0000 7f2864083740 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b7b8a7eeb0 0x55b7b8a812d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:14.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.960+0000 7f2864083740 4 client.25673 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:52:14.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.960+0000 7f2864083740 1 -- 172.21.15.49:0/1993780714 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b7b88ed540 con 0x55b7b8a7eeb0 2024-03-18T00:52:14.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.962+0000 7f285b7fe640 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b7b8a7eeb0 0x55b7b8a812d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:14.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.964+0000 7f283eff5640 10 client.25673 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:14.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.964+0000 7f285b7fe640 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b7b8a7eeb0 0x55b7b8a812d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c237f49cc681518 server_cookie=202c33f66b5e8b68 in_seq=0 out_seq=0 2024-03-18T00:52:14.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f283eff5640 1 -- 172.21.15.49:0/1993780714 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55b7b88ed540 con 0x55b7b8a7eeb0 2024-03-18T00:52:14.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f283eff5640 10 client.25673 handle_command_reply: tid=0 2024-03-18T00:52:14.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f2864083740 1 client.25673 shutdown 2024-03-18T00:52:14.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f2864083740 1 -- 172.21.15.49:0/1993780714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03b60 msgr2=0x55b7b8a11970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f2864083740 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03b60 0x55b7b8a11970 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x55b7b8a17f10 tx=0x7f2850016f70 comp rx=0 tx=0).stop 2024-03-18T00:52:14.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f2864083740 1 -- 172.21.15.49:0/1993780714 shutdown_connections 2024-03-18T00:52:14.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f2864083740 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b7b8a7eeb0 0x55b7b8a812d0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f2864083740 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b7b8a03b60 0x55b7b8a11970 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f2864083740 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b7b8a08cd0 0x55b7b8a11eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f283eff5640 0 client.25673 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:14.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f2864083740 1 --2- 172.21.15.49:0/1993780714 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b7b8a03170 0x55b7b8a14e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.968+0000 7f2864083740 1 -- 172.21.15.49:0/1993780714 >> 172.21.15.49:0/1993780714 conn(0x55b7b89fe6e0 msgr2=0x55b7b8a05240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:14.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.969+0000 7f2864083740 1 -- 172.21.15.49:0/1993780714 shutdown_connections 2024-03-18T00:52:14.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.969+0000 7f2864083740 1 -- 172.21.15.49:0/1993780714 wait complete. 2024-03-18T00:52:14.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.969+0000 7f2864083740 20 client.25673 trim_cache size 0 max 16384 2024-03-18T00:52:14.970 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:52:14.970 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-18T00:52:14.970 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:52:14.970 INFO:teuthology.orchestra.run.smithi049.stdout: "2f6cef41-4cfb-4150-bbd7-005032be35ac": { 2024-03-18T00:52:14.970 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:52:14.970 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "2f6cef41-4cfb-4150-bbd7-005032be35ac", 2024-03-18T00:52:14.971 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:52:14.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:52:14.971 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:52:14.971 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:52:14.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 -- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f28440a2980 msgr2=0x7f28440a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f28440a2980 0x7f28440a4e40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f28540b9800 tx=0x7f2854004140 comp rx=0 tx=0).stop 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 -- 172.21.15.49:0/1704296098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f285c107e00 msgr2=0x7f285c0702b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f285c107e00 0x7f285c0702b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f285402c750 tx=0x7f2854035d80 comp rx=0 tx=0).stop 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 -- 172.21.15.49:0/1704296098 shutdown_connections 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f28440a2980 0x7f28440a4e40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f285c1099b0 0x7f285c1ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f285c108ff0 0x7f285c0707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 --2- 172.21.15.49:0/1704296098 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f285c107e00 0x7f285c0702b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 -- 172.21.15.49:0/1704296098 >> 172.21.15.49:0/1704296098 conn(0x7f285c076d90 msgr2=0x7f285c10c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 -- 172.21.15.49:0/1704296098 shutdown_connections 2024-03-18T00:52:14.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:14.970+0000 7f2861497640 1 -- 172.21.15.49:0/1704296098 wait complete. 2024-03-18T00:52:15.002 INFO:tasks.cephfs.filesystem:scrub status for tag:2f6cef41-4cfb-4150-bbd7-005032be35ac - {'path': '/', 'tag': '2f6cef41-4cfb-4150-bbd7-005032be35ac', 'options': 'recursive,force'} 2024-03-18T00:52:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:15 smithi049 ceph-mon[32130]: pgmap v414: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 1.98k op/s 2024-03-18T00:52:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:15 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:52:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:15 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:52:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:15 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:52:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:15 smithi107 ceph-mon[36270]: pgmap v414: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 1.98k op/s 2024-03-18T00:52:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:15 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:52:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:15 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:52:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:15 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:52:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:15 smithi106 ceph-mon[35899]: pgmap v414: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 1.98k op/s 2024-03-18T00:52:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:15 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:52:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:15 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:52:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:15 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:52:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:17 smithi107 ceph-mon[36270]: pgmap v415: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 798 MiB/s wr, 1.65k op/s 2024-03-18T00:52:17.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:17 smithi106 ceph-mon[35899]: pgmap v415: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 798 MiB/s wr, 1.65k op/s 2024-03-18T00:52:17.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:17 smithi049 ceph-mon[32130]: pgmap v415: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 798 MiB/s wr, 1.65k op/s 2024-03-18T00:52:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:19 smithi107 ceph-mon[36270]: pgmap v416: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 943 MiB/s wr, 1.89k op/s 2024-03-18T00:52:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:19 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:52:19.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:19 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:52:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:19 smithi106 ceph-mon[35899]: pgmap v416: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 943 MiB/s wr, 1.89k op/s 2024-03-18T00:52:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:19 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:52:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:19 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:52:19.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:19 smithi049 ceph-mon[32130]: pgmap v416: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 943 MiB/s wr, 1.89k op/s 2024-03-18T00:52:19.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:19 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:52:19.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:19 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:52:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.190+0000 7f090143f640 1 -- 172.21.15.49:0/3057494120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc072a90 msgr2=0x7f08fc072eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:21.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.190+0000 7f090143f640 1 --2- 172.21.15.49:0/3057494120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc072a90 0x7f08fc072eb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009900 tx=0x7f08ec02bc60 comp rx=0 tx=0).stop 2024-03-18T00:52:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.200+0000 7f090143f640 1 -- 172.21.15.49:0/3057494120 shutdown_connections 2024-03-18T00:52:21.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.200+0000 7f090143f640 1 --2- 172.21.15.49:0/3057494120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08fc1094d0 0x7f08fc10e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.200+0000 7f090143f640 1 --2- 172.21.15.49:0/3057494120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc0733f0 0x7f08fc108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.200+0000 7f090143f640 1 --2- 172.21.15.49:0/3057494120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc072a90 0x7f08fc072eb0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.202 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.200+0000 7f090143f640 1 -- 172.21.15.49:0/3057494120 >> 172.21.15.49:0/3057494120 conn(0x7f08fc06e650 msgr2=0x7f08fc070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:21.208 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.205+0000 7f090143f640 1 -- 172.21.15.49:0/3057494120 shutdown_connections 2024-03-18T00:52:21.209 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.208+0000 7f090143f640 1 -- 172.21.15.49:0/3057494120 wait complete. 2024-03-18T00:52:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.209+0000 7f090143f640 1 Processor -- start 2024-03-18T00:52:21.210 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.209+0000 7f090143f640 1 -- start start 2024-03-18T00:52:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.210+0000 7f090143f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08fc072a90 0x7f08fc114ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.210+0000 7f090143f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0733f0 0x7f08fc115410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.210+0000 7f090143f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1094d0 0x7f08fc1b6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.210+0000 7f090143f640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc11c110 con 0x7f08fc0733f0 2024-03-18T00:52:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.210+0000 7f090143f640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc11bf90 con 0x7f08fc072a90 2024-03-18T00:52:21.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.210+0000 7f090143f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc11c290 con 0x7f08fc1094d0 2024-03-18T00:52:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.212+0000 7f08fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0733f0 0x7f08fc115410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.212+0000 7f08fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0733f0 0x7f08fc115410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50026/0 (socket says 172.21.15.49:50026) 2024-03-18T00:52:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.212+0000 7f08fb7fe640 1 -- 172.21.15.49:0/206086198 learned_addr learned my addr 172.21.15.49:0/206086198 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.213+0000 7f08fb7fe640 1 -- 172.21.15.49:0/206086198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1094d0 msgr2=0x7f08fc1b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.213+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1094d0 0x7f08fc1b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.213+0000 7f08fb7fe640 1 -- 172.21.15.49:0/206086198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08fc072a90 msgr2=0x7f08fc114ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:52:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.213+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08fc072a90 0x7f08fc114ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.213+0000 7f08fb7fe640 1 -- 172.21.15.49:0/206086198 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08fc1b6d40 con 0x7f08fc0733f0 2024-03-18T00:52:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.215+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0733f0 0x7f08fc115410 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f08f000d770 tx=0x7f08f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.216+0000 7f08f97fa640 1 -- 172.21.15.49:0/206086198 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f0017070 con 0x7f08fc0733f0 2024-03-18T00:52:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.216+0000 7f08f97fa640 1 -- 172.21.15.49:0/206086198 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f0009c30 con 0x7f08fc0733f0 2024-03-18T00:52:21.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.216+0000 7f08f97fa640 1 -- 172.21.15.49:0/206086198 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f000f620 con 0x7f08fc0733f0 2024-03-18T00:52:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.219+0000 7f090143f640 1 -- 172.21.15.49:0/206086198 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08fc1b6f10 con 0x7f08fc0733f0 2024-03-18T00:52:21.221 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.219+0000 7f090143f640 1 -- 172.21.15.49:0/206086198 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08fc1b7320 con 0x7f08fc0733f0 2024-03-18T00:52:21.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.227+0000 7f090536f740 1 Processor -- start 2024-03-18T00:52:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.230+0000 7f090536f740 1 -- start start 2024-03-18T00:52:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.230+0000 7f090536f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.230+0000 7f090536f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167ebec0 0x562c167f09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.230+0000 7f090536f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167f1030 0x562c167f34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.230+0000 7f090536f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x562c16750a40 con 0x562c167ebec0 2024-03-18T00:52:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.230+0000 7f090536f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x562c166f9920 con 0x562c167eb4d0 2024-03-18T00:52:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.230+0000 7f090536f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562c1674cae0 con 0x562c167f1030 2024-03-18T00:52:21.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.231+0000 7f08f97fa640 1 -- 172.21.15.49:0/206086198 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f08f001a070 con 0x7f08fc0733f0 2024-03-18T00:52:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.232+0000 7f08fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167f1030 0x562c167f34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.232+0000 7f0900c3e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.232+0000 7f0900c3e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167eb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47236/0 (socket says 172.21.15.49:47236) 2024-03-18T00:52:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.232+0000 7f0900c3e640 1 -- 172.21.15.49:0/1762850550 learned_addr learned my addr 172.21.15.49:0/1762850550 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.233+0000 7f08f97fa640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e40a2a00 0x7f08e40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.234+0000 7f0900c3e640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e40a2a00 0x7f08e40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.235 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.234+0000 7f08f97fa640 1 -- 172.21.15.49:0/206086198 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f08f00ce5d0 con 0x7f08fc0733f0 2024-03-18T00:52:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.237+0000 7f0900c3e640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e40a2a00 0x7f08e40a4ec0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f08f4013690 tx=0x7f08f40077e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.238+0000 7f0900c3e640 1 -- 172.21.15.49:0/1762850550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167f1030 msgr2=0x562c167f34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.238+0000 7f0900c3e640 1 --2- 172.21.15.49:0/1762850550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167f1030 0x562c167f34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.238+0000 7f08fbfff640 1 --2- 172.21.15.49:0/1762850550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167ebec0 0x562c167f09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.238+0000 7f0900c3e640 1 -- 172.21.15.49:0/1762850550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167ebec0 msgr2=0x562c167f09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.238+0000 7f0900c3e640 1 --2- 172.21.15.49:0/1762850550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167ebec0 0x562c167f09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.238+0000 7f0900c3e640 1 -- 172.21.15.49:0/1762850550 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c16711460 con 0x562c167eb4d0 2024-03-18T00:52:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.238+0000 7f08fbfff640 1 --2- 172.21.15.49:0/1762850550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167ebec0 0x562c167f09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:52:21.243 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.241+0000 7f0900c3e640 1 --2- 172.21.15.49:0/1762850550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167eb8f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f08f40118b0 tx=0x7f08f400d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0619d83efd5dcdd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.248+0000 7f08e27fc640 1 -- 172.21.15.49:0/1762850550 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f400da40 con 0x562c167eb4d0 2024-03-18T00:52:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.248+0000 7f08e27fc640 1 -- 172.21.15.49:0/1762850550 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f400dbe0 con 0x562c167eb4d0 2024-03-18T00:52:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.248+0000 7f08e27fc640 1 -- 172.21.15.49:0/1762850550 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f401a3b0 con 0x562c167eb4d0 2024-03-18T00:52:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.249+0000 7f090536f740 1 -- 172.21.15.49:0/1762850550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 msgr2=0x562c167eb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:21.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.249+0000 7f090536f740 1 --2- 172.21.15.49:0/1762850550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167eb8f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f08f40118b0 tx=0x7f08f400d000 comp rx=0 tx=0).stop 2024-03-18T00:52:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.252+0000 7f090536f740 1 -- 172.21.15.49:0/1762850550 shutdown_connections 2024-03-18T00:52:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.252+0000 7f090536f740 1 --2- 172.21.15.49:0/1762850550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167f1030 0x562c167f34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.252+0000 7f090536f740 1 --2- 172.21.15.49:0/1762850550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167ebec0 0x562c167f09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.252+0000 7f090536f740 1 --2- 172.21.15.49:0/1762850550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167eb8f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.257 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.252+0000 7f090536f740 1 -- 172.21.15.49:0/1762850550 >> 172.21.15.49:0/1762850550 conn(0x562c167e6a40 msgr2=0x562c167e6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:21.261 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.261+0000 7f090536f740 1 -- 172.21.15.49:0/1762850550 shutdown_connections 2024-03-18T00:52:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.261+0000 7f090536f740 1 -- 172.21.15.49:0/1762850550 wait complete. 2024-03-18T00:52:21.262 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.261+0000 7f090536f740 1 -- start start 2024-03-18T00:52:21.263 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.262+0000 7f090536f740 1 Processor -- start 2024-03-18T00:52:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.263+0000 7f090536f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167f5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.263+0000 7f090536f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.263+0000 7f090536f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.263+0000 7f090536f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x562c167f3f80 con 0x562c167f1030 2024-03-18T00:52:21.264 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.263+0000 7f090536f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x562c1670cd70 con 0x562c167eb4d0 2024-03-18T00:52:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.263+0000 7f090536f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562c167f4100 con 0x562c167ebec0 2024-03-18T00:52:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.263+0000 7f090536f740 10 client.0 _ll_register_callbacks cb 0x562c16464a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.264+0000 7f08fbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.264+0000 7f08fb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.264+0000 7f08fbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167f5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47242/0 (socket says 172.21.15.49:47242) 2024-03-18T00:52:21.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.264+0000 7f08fbfff640 1 -- 172.21.15.49:0/78200456 learned_addr learned my addr 172.21.15.49:0/78200456 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.265+0000 7f0900c3e640 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.266 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.265+0000 7f08fbfff640 1 -- 172.21.15.49:0/78200456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 msgr2=0x562c167f5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.265+0000 7f08fbfff640 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167f5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.265+0000 7f08fbfff640 1 -- 172.21.15.49:0/78200456 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 msgr2=0x562c167fcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.265+0000 7f08fbfff640 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.265+0000 7f08fbfff640 1 -- 172.21.15.49:0/78200456 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c16465210 con 0x562c167eb4d0 2024-03-18T00:52:21.267 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.266+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:52:21.269 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.267+0000 7f08fbfff640 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f08ec02c1f0 tx=0x7f08ec009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0088409be0b52f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.267+0000 7f08db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:52:21.270 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.269+0000 7f08db7fe640 1 -- 172.21.15.49:0/78200456 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec043020 con 0x562c167eb4d0 2024-03-18T00:52:21.271 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.269+0000 7f090536f740 1 -- 172.21.15.49:0/78200456 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08f400dbe0 con 0x562c167eb4d0 2024-03-18T00:52:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.269+0000 7f08db7fe640 1 -- 172.21.15.49:0/78200456 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f08ec004350 con 0x562c167eb4d0 2024-03-18T00:52:21.276 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.275+0000 7f08db7fe640 1 -- 172.21.15.49:0/78200456 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f08ec04e050 con 0x562c167eb4d0 2024-03-18T00:52:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.277+0000 7f090536f740 10 client.25679 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.277+0000 7f090536f740 1 -- 172.21.15.49:0/78200456 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562c16531620 con 0x562c167eb4d0 2024-03-18T00:52:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.280+0000 7f08db7fe640 1 -- 172.21.15.49:0/78200456 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f400dbe0 con 0x562c167eb4d0 2024-03-18T00:52:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.280+0000 7f08db7fe640 1 -- 172.21.15.49:0/78200456 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f08ec004b90 con 0x562c167eb4d0 2024-03-18T00:52:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.280+0000 7f090536f740 10 client.25679 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.280+0000 7f090536f740 10 client.25679 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.280+0000 7f090536f740 20 client.25679 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.280+0000 7f090536f740 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562c16801a20 0x562c16803ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.280+0000 7f090536f740 4 client.25679 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:21.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.280+0000 7f090536f740 1 -- 172.21.15.49:0/78200456 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c1675df80 con 0x562c16801a20 2024-03-18T00:52:21.284 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.282+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562c16801a20 0x562c16803ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.307+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562c16801a20 0x562c16803ec0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f78bad0707ee6914 server_cookie=d64251f5df1fff60 in_seq=0 out_seq=0 2024-03-18T00:52:21.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.308+0000 7f08db7fe640 10 client.25679 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.314+0000 7f08db7fe640 1 -- 172.21.15.49:0/78200456 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562c1675df80 con 0x562c16801a20 2024-03-18T00:52:21.316 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.315+0000 7f08db7fe640 10 client.25679 handle_command_reply: tid=0 2024-03-18T00:52:21.319 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.319+0000 7f090536f740 1 client.25679 shutdown 2024-03-18T00:52:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.319+0000 7f090536f740 1 -- 172.21.15.49:0/78200456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 msgr2=0x562c167f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.319+0000 7f090536f740 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f08ec02c1f0 tx=0x7f08ec009630 comp rx=0 tx=0).stop 2024-03-18T00:52:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.319+0000 7f090536f740 1 -- 172.21.15.49:0/78200456 shutdown_connections 2024-03-18T00:52:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.319+0000 7f090536f740 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562c16801a20 0x562c16803ec0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.319+0000 7f090536f740 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.319+0000 7f090536f740 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.319+0000 7f090536f740 1 --2- 172.21.15.49:0/78200456 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167eb4d0 0x562c167f5830 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f08ec02c1f0 tx=0x7f08ec009630 comp rx=0 tx=0).stop 2024-03-18T00:52:21.323 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.319+0000 7f090536f740 1 -- 172.21.15.49:0/78200456 >> 172.21.15.49:0/78200456 conn(0x562c167e6a40 msgr2=0x562c167e5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.323+0000 7f08db7fe640 0 client.25679 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:21.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.323+0000 7f090536f740 1 -- 172.21.15.49:0/78200456 shutdown_connections 2024-03-18T00:52:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.327+0000 7f090536f740 1 -- 172.21.15.49:0/78200456 wait complete. 2024-03-18T00:52:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.328+0000 7f090536f740 20 client.25679 trim_cache size 0 max 16384 2024-03-18T00:52:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.366+0000 7f090536f740 1 Processor -- start 2024-03-18T00:52:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.366+0000 7f090536f740 1 -- start start 2024-03-18T00:52:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.366+0000 7f090536f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.366+0000 7f090536f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.367+0000 7f08fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.367+0000 7f08fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50054/0 (socket says 172.21.15.49:50054) 2024-03-18T00:52:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.367+0000 7f08fb7fe640 1 -- 172.21.15.49:0/241778225 learned_addr learned my addr 172.21.15.49:0/241778225 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.367+0000 7f090536f740 1 --2- 172.21.15.49:0/241778225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167e53d0 0x562c167e5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.367+0000 7f090536f740 1 -- 172.21.15.49:0/241778225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc1c03a0 con 0x562c167f1030 2024-03-18T00:52:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.367+0000 7f08fbfff640 1 --2- 172.21.15.49:0/241778225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.367+0000 7f090536f740 1 -- 172.21.15.49:0/241778225 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc1c1f10 con 0x562c167e53d0 2024-03-18T00:52:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.367+0000 7f090536f740 1 -- 172.21.15.49:0/241778225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc1c0a70 con 0x562c167ebec0 2024-03-18T00:52:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.368+0000 7f08fb7fe640 1 -- 172.21.15.49:0/241778225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 msgr2=0x562c167f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:21.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.368+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/241778225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167f5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.368+0000 7f08fb7fe640 1 -- 172.21.15.49:0/241778225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167e53d0 msgr2=0x562c167e5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.368+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/241778225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167e53d0 0x562c167e5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.368+0000 7f08fb7fe640 1 -- 172.21.15.49:0/241778225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c167f0d10 con 0x562c167f1030 2024-03-18T00:52:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.369+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/241778225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcd70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f08f0097940 tx=0x7f08f00ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38faa6463b8ae1a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:21.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.370+0000 7f08e1ffb640 1 -- 172.21.15.49:0/241778225 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f00b7030 con 0x562c167f1030 2024-03-18T00:52:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.370+0000 7f08e1ffb640 1 -- 172.21.15.49:0/241778225 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08f00a3710 con 0x562c167f1030 2024-03-18T00:52:21.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.370+0000 7f08e1ffb640 1 -- 172.21.15.49:0/241778225 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f00a3280 con 0x562c167f1030 2024-03-18T00:52:21.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.371+0000 7f090536f740 1 -- 172.21.15.49:0/241778225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 msgr2=0x562c167fcd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:21.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.371+0000 7f090536f740 1 --2- 172.21.15.49:0/241778225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcd70 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f08f0097940 tx=0x7f08f00ce040 comp rx=0 tx=0).stop 2024-03-18T00:52:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.372+0000 7f090536f740 1 -- 172.21.15.49:0/241778225 shutdown_connections 2024-03-18T00:52:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.372+0000 7f090536f740 1 --2- 172.21.15.49:0/241778225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167e53d0 0x562c167e5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.372+0000 7f090536f740 1 --2- 172.21.15.49:0/241778225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167f1030 0x562c167fcd70 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.372+0000 7f090536f740 1 --2- 172.21.15.49:0/241778225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167f5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.372+0000 7f090536f740 1 -- 172.21.15.49:0/241778225 >> 172.21.15.49:0/241778225 conn(0x562c167e6a40 msgr2=0x562c167f5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.372+0000 7f090536f740 1 -- 172.21.15.49:0/241778225 shutdown_connections 2024-03-18T00:52:21.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.372+0000 7f090536f740 1 -- 172.21.15.49:0/241778225 wait complete. 2024-03-18T00:52:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.373+0000 7f090536f740 1 -- start start 2024-03-18T00:52:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.373+0000 7f090536f740 1 Processor -- start 2024-03-18T00:52:21.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.374+0000 7f090536f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167e53d0 0x562c167fa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.374+0000 7f090536f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167fa900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.374+0000 7f08fbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167e53d0 0x562c167fa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.374+0000 7f090536f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167f1030 0x562c167fb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.375 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f08fbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167e53d0 0x562c167fa3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50062/0 (socket says 172.21.15.49:50062) 2024-03-18T00:52:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f090536f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc1c1810 con 0x562c167e53d0 2024-03-18T00:52:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f090536f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc1c1140 con 0x562c167f1030 2024-03-18T00:52:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f090536f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc1be380 con 0x562c167ebec0 2024-03-18T00:52:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f08fbfff640 1 -- 172.21.15.49:0/2816706097 learned_addr learned my addr 172.21.15.49:0/2816706097 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:21.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f0900c3e640 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167f1030 0x562c167fb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f090536f740 10 client.0 _ll_register_callbacks cb 0x7f08ec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167fa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f08fbfff640 1 -- 172.21.15.49:0/2816706097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 msgr2=0x562c167fa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f08fbfff640 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167fa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f08fbfff640 1 -- 172.21.15.49:0/2816706097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167f1030 msgr2=0x562c167fb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f08fbfff640 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167f1030 0x562c167fb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.375+0000 7f08fbfff640 1 -- 172.21.15.49:0/2816706097 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c167fbf00 con 0x562c167e53d0 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.376+0000 7f090536f740 1 -- 172.21.15.49:0/2816706097 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08f00a3710 con 0x562c167e53d0 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.376+0000 7f08fbfff640 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167e53d0 0x562c167fa3c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f08ec02c270 tx=0x7f08ec004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a5ad245180339c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.376+0000 7f08db7fe640 10 client.16029 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:21.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.376+0000 7f08db7fe640 1 -- 172.21.15.49:0/2816706097 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec043040 con 0x562c167e53d0 2024-03-18T00:52:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.377+0000 7f08db7fe640 1 -- 172.21.15.49:0/2816706097 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f08ec003a40 con 0x562c167e53d0 2024-03-18T00:52:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.377+0000 7f08db7fe640 1 -- 172.21.15.49:0/2816706097 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f08ec003cd0 con 0x562c167e53d0 2024-03-18T00:52:21.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.377+0000 7f090536f740 10 client.16029 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.377+0000 7f090536f740 1 -- 172.21.15.49:0/2816706097 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({fsmap=26}) -- 0x562c16867e00 con 0x562c167e53d0 2024-03-18T00:52:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.377+0000 7f08db7fe640 1 -- 172.21.15.49:0/2816706097 <== mon.0 v2:172.21.15.49:3300/0 4 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f08ec005020 con 0x562c167e53d0 2024-03-18T00:52:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.378+0000 7f090536f740 10 client.16029 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.378+0000 7f090536f740 10 client.16029 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.378+0000 7f090536f740 20 client.16029 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.378+0000 7f090536f740 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562c168684d0 0x562c1686a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.378+0000 7f090536f740 4 client.16029 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:52:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.378+0000 7f090536f740 1 -- 172.21.15.49:0/2816706097 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c167582d0 con 0x562c168684d0 2024-03-18T00:52:21.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.378+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562c168684d0 0x562c1686a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:21.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.380+0000 7f08db7fe640 10 client.16029 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:21.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:21.380+0000 7f08fb7fe640 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562c168684d0 0x562c1686a8f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc593c963bcfa775 server_cookie=499e5363cc71575f in_seq=0 out_seq=0 2024-03-18T00:52:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:21 smithi107 ceph-mon[36270]: pgmap v417: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 924 MiB/s wr, 1.69k op/s 2024-03-18T00:52:21.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:21 smithi106 ceph-mon[35899]: pgmap v417: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 924 MiB/s wr, 1.69k op/s 2024-03-18T00:52:21.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:21 smithi049 ceph-mon[32130]: pgmap v417: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 924 MiB/s wr, 1.69k op/s 2024-03-18T00:52:22.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:22 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:52:22.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:22 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:52:22.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:22 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:52:23.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:23 smithi049 ceph-mon[32130]: pgmap v418: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 824 MiB/s wr, 1.37k op/s 2024-03-18T00:52:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:23 smithi107 ceph-mon[36270]: pgmap v418: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 824 MiB/s wr, 1.37k op/s 2024-03-18T00:52:23.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:23 smithi106 ceph-mon[35899]: pgmap v418: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 824 MiB/s wr, 1.37k op/s 2024-03-18T00:52:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:25 smithi049 ceph-mon[32130]: pgmap v419: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 1.69k op/s 2024-03-18T00:52:25.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:25 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:52:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:25 smithi107 ceph-mon[36270]: pgmap v419: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 1.69k op/s 2024-03-18T00:52:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:25 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:52:25.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:25 smithi106 ceph-mon[35899]: pgmap v419: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.0 GiB/s wr, 1.69k op/s 2024-03-18T00:52:25.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:25 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:52:26.634 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:26 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:52:26.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:26 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:52:26.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:26 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:52:27.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:27 smithi049 ceph-mon[32130]: pgmap v420: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 797 MiB/s wr, 1.28k op/s 2024-03-18T00:52:27.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:27 smithi107 ceph-mon[36270]: pgmap v420: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 797 MiB/s wr, 1.28k op/s 2024-03-18T00:52:27.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:27 smithi106 ceph-mon[35899]: pgmap v420: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 797 MiB/s wr, 1.28k op/s 2024-03-18T00:52:29.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:29 smithi049 ceph-mon[32130]: pgmap v421: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 942 MiB/s wr, 1.84k op/s 2024-03-18T00:52:29.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:29 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:52:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:29 smithi107 ceph-mon[36270]: pgmap v421: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 942 MiB/s wr, 1.84k op/s 2024-03-18T00:52:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:29 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:52:29.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:29 smithi106 ceph-mon[35899]: pgmap v421: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 942 MiB/s wr, 1.84k op/s 2024-03-18T00:52:29.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:29 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:52:29.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.792+0000 7f08db7fe640 1 -- 172.21.15.49:0/2816706097 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c167582d0 con 0x562c168684d0 2024-03-18T00:52:29.794 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.792+0000 7f08db7fe640 10 client.16029 handle_command_reply: tid=0 2024-03-18T00:52:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.796+0000 7f090536f740 1 client.16029 shutdown 2024-03-18T00:52:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.797+0000 7f090536f740 1 -- 172.21.15.49:0/2816706097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167e53d0 msgr2=0x562c167fa3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.797+0000 7f090536f740 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167e53d0 0x562c167fa3c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f08ec02c270 tx=0x7f08ec004600 comp rx=0 tx=0).stop 2024-03-18T00:52:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.797+0000 7f090536f740 1 -- 172.21.15.49:0/2816706097 shutdown_connections 2024-03-18T00:52:29.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.797+0000 7f090536f740 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562c168684d0 0x562c1686a8f0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.798+0000 7f090536f740 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562c167f1030 0x562c167fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.798+0000 7f090536f740 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562c167ebec0 0x562c167fa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.798+0000 7f090536f740 1 --2- 172.21.15.49:0/2816706097 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562c167e53d0 0x562c167fa3c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.798+0000 7f090536f740 1 -- 172.21.15.49:0/2816706097 >> 172.21.15.49:0/2816706097 conn(0x562c167e6a40 msgr2=0x562c167f5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:29.798 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.798+0000 7f08db7fe640 0 client.16029 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.800+0000 7f090536f740 1 -- 172.21.15.49:0/2816706097 shutdown_connections 2024-03-18T00:52:29.799 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.800+0000 7f090536f740 1 -- 172.21.15.49:0/2816706097 wait complete. 2024-03-18T00:52:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.800+0000 7f090536f740 20 client.16029 trim_cache size 0 max 16384 2024-03-18T00:52:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:52:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:52:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:52:29.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:52:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.811+0000 7f090143f640 1 -- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e40a2a00 msgr2=0x7f08e40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.811+0000 7f090143f640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e40a2a00 0x7f08e40a4ec0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f08f4013690 tx=0x7f08f40077e0 comp rx=0 tx=0).stop 2024-03-18T00:52:29.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.811+0000 7f090143f640 1 -- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0733f0 msgr2=0x7f08fc115410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:29.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.811+0000 7f090143f640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0733f0 0x7f08fc115410 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f08f000d770 tx=0x7f08f000dc30 comp rx=0 tx=0).stop 2024-03-18T00:52:29.816 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.816+0000 7f090143f640 1 -- 172.21.15.49:0/206086198 shutdown_connections 2024-03-18T00:52:29.816 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.816+0000 7f090143f640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f08e40a2a00 0x7f08e40a4ec0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:29.816 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.816+0000 7f090143f640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08fc1094d0 0x7f08fc1b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:29.816 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.816+0000 7f090143f640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f08fc0733f0 0x7f08fc115410 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:29.816 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.816+0000 7f090143f640 1 --2- 172.21.15.49:0/206086198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f08fc072a90 0x7f08fc114ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:29.817 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.816+0000 7f090143f640 1 -- 172.21.15.49:0/206086198 >> 172.21.15.49:0/206086198 conn(0x7f08fc06e650 msgr2=0x7f08fc070320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:29.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.817+0000 7f090143f640 1 -- 172.21.15.49:0/206086198 shutdown_connections 2024-03-18T00:52:29.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:29.818+0000 7f090143f640 1 -- 172.21.15.49:0/206086198 wait complete. 2024-03-18T00:52:30.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:30 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:30 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:30 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:30 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:30 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:30 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:30 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:30 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:30 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:30 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:30 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:30.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:30 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:31 smithi107 ceph-mon[36270]: pgmap v422: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 928 MiB/s wr, 2.07k op/s 2024-03-18T00:52:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:52:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:52:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:31 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:31.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:31 smithi106 ceph-mon[35899]: pgmap v422: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 928 MiB/s wr, 2.07k op/s 2024-03-18T00:52:31.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:52:31.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:52:31.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:31 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:31.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:31 smithi049 ceph-mon[32130]: pgmap v422: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 928 MiB/s wr, 2.07k op/s 2024-03-18T00:52:31.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:52:31.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:52:31.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:31 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:52:33.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:33 smithi107 ceph-mon[36270]: pgmap v423: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 834 MiB/s wr, 1.96k op/s 2024-03-18T00:52:33.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:33 smithi106 ceph-mon[35899]: pgmap v423: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 834 MiB/s wr, 1.96k op/s 2024-03-18T00:52:33.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:33 smithi049 ceph-mon[32130]: pgmap v423: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 834 MiB/s wr, 1.96k op/s 2024-03-18T00:52:34.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:34 smithi049 ceph-mon[32130]: pgmap v424: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 GiB/s wr, 2.21k op/s 2024-03-18T00:52:34.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:34 smithi107 ceph-mon[36270]: pgmap v424: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 GiB/s wr, 2.21k op/s 2024-03-18T00:52:34.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:34 smithi106 ceph-mon[35899]: pgmap v424: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 GiB/s wr, 2.21k op/s 2024-03-18T00:52:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:36 smithi107 ceph-mon[36270]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 806 MiB/s wr, 1.94k op/s 2024-03-18T00:52:36.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:36 smithi106 ceph-mon[35899]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 806 MiB/s wr, 1.94k op/s 2024-03-18T00:52:37.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:36 smithi049 ceph-mon[32130]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 806 MiB/s wr, 1.94k op/s 2024-03-18T00:52:38.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:38 smithi107 ceph-mon[36270]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 919 MiB/s wr, 2.14k op/s 2024-03-18T00:52:38.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:38 smithi106 ceph-mon[35899]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 919 MiB/s wr, 2.14k op/s 2024-03-18T00:52:39.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:38 smithi049 ceph-mon[32130]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 919 MiB/s wr, 2.14k op/s 2024-03-18T00:52:40.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:40 smithi107 ceph-mon[36270]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 932 MiB/s wr, 1.90k op/s 2024-03-18T00:52:40.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:40 smithi106 ceph-mon[35899]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 932 MiB/s wr, 1.90k op/s 2024-03-18T00:52:41.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:40 smithi049 ceph-mon[32130]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 932 MiB/s wr, 1.90k op/s 2024-03-18T00:52:42.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:42 smithi107 ceph-mon[36270]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 831 MiB/s wr, 1.43k op/s 2024-03-18T00:52:42.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:42 smithi106 ceph-mon[35899]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 831 MiB/s wr, 1.43k op/s 2024-03-18T00:52:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.118+0000 7f4eacf37640 1 -- 172.21.15.49:0/2946394034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ea8072ab0 msgr2=0x7f4ea8072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:43.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.118+0000 7f4eacf37640 1 --2- 172.21.15.49:0/2946394034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ea8072ab0 0x7f4ea8072ed0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009900 tx=0x7f4e9802bc40 comp rx=0 tx=0).stop 2024-03-18T00:52:43.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:42 smithi049 ceph-mon[32130]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 831 MiB/s wr, 1.43k op/s 2024-03-18T00:52:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.127+0000 7f4eacf37640 1 -- 172.21.15.49:0/2946394034 shutdown_connections 2024-03-18T00:52:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.127+0000 7f4eacf37640 1 --2- 172.21.15.49:0/2946394034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea8109510 0x7f4ea810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.127+0000 7f4eacf37640 1 --2- 172.21.15.49:0/2946394034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea8073410 0x7f4ea8108e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.127+0000 7f4eacf37640 1 --2- 172.21.15.49:0/2946394034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ea8072ab0 0x7f4ea8072ed0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.127 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.127+0000 7f4eacf37640 1 -- 172.21.15.49:0/2946394034 >> 172.21.15.49:0/2946394034 conn(0x7f4ea806e650 msgr2=0x7f4ea8070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:43.130 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.130+0000 7f4eacf37640 1 -- 172.21.15.49:0/2946394034 shutdown_connections 2024-03-18T00:52:43.130 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.130+0000 7f4eacf37640 1 -- 172.21.15.49:0/2946394034 wait complete. 2024-03-18T00:52:43.132 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.132+0000 7f4eacf37640 1 Processor -- start 2024-03-18T00:52:43.134 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.134+0000 7f4eacf37640 1 -- start start 2024-03-18T00:52:43.138 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.135+0000 7f4eacf37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea8072ab0 0x7f4ea81a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.135+0000 7f4eacf37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea8073410 0x7f4ea81a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.135+0000 7f4eacf37640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ea8109510 0x7f4ea81ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.135+0000 7f4eacf37640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea811c230 con 0x7f4ea8072ab0 2024-03-18T00:52:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.135+0000 7f4eacf37640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea811c0b0 con 0x7f4ea8109510 2024-03-18T00:52:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.135+0000 7f4eacf37640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea811c3b0 con 0x7f4ea8073410 2024-03-18T00:52:43.139 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.138+0000 7f4ea659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea8072ab0 0x7f4ea81a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.138+0000 7f4ea659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea8072ab0 0x7f4ea81a6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44402/0 (socket says 172.21.15.49:44402) 2024-03-18T00:52:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.138+0000 7f4ea659c640 1 -- 172.21.15.49:0/182227797 learned_addr learned my addr 172.21.15.49:0/182227797 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:43.140 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.139+0000 7f4ea5d9b640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea8073410 0x7f4ea81a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.142 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.141+0000 7f4ea6d9d640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ea8109510 0x7f4ea81ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.143+0000 7f4ea5d9b640 1 -- 172.21.15.49:0/182227797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ea8109510 msgr2=0x7f4ea81ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.143+0000 7f4ea5d9b640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ea8109510 0x7f4ea81ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.143+0000 7f4ea5d9b640 1 -- 172.21.15.49:0/182227797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea8072ab0 msgr2=0x7f4ea81a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.143+0000 7f4ea5d9b640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea8072ab0 0x7f4ea81a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.144 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.143+0000 7f4ea5d9b640 1 -- 172.21.15.49:0/182227797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea81ae4a0 con 0x7f4ea8073410 2024-03-18T00:52:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.147+0000 7f4ea5d9b640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea8073410 0x7f4ea81a6dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00b3c0 tx=0x7f4e9c00b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.151+0000 7f4e977fe640 1 -- 172.21.15.49:0/182227797 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c011830 con 0x7f4ea8073410 2024-03-18T00:52:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.152+0000 7f4eacf37640 1 -- 172.21.15.49:0/182227797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea81ae790 con 0x7f4ea8073410 2024-03-18T00:52:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.152+0000 7f4eacf37640 1 -- 172.21.15.49:0/182227797 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea81aece0 con 0x7f4ea8073410 2024-03-18T00:52:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.152+0000 7f4e977fe640 1 -- 172.21.15.49:0/182227797 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e9c016070 con 0x7f4ea8073410 2024-03-18T00:52:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.153+0000 7f4e977fe640 1 -- 172.21.15.49:0/182227797 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00f2e0 con 0x7f4ea8073410 2024-03-18T00:52:43.180 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.179+0000 7f4e977fe640 1 -- 172.21.15.49:0/182227797 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4e9c00f480 con 0x7f4ea8073410 2024-03-18T00:52:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.181+0000 7f4eafbf6740 1 Processor -- start 2024-03-18T00:52:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.181+0000 7f4e977fe640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4e900a2980 0x7f4e900a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.182 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.181+0000 7f4e977fe640 1 -- 172.21.15.49:0/182227797 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4e9c0cef70 con 0x7f4ea8073410 2024-03-18T00:52:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.186+0000 7f4eafbf6740 1 -- start start 2024-03-18T00:52:43.188 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.187+0000 7f4ea6d9d640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4e900a2980 0x7f4e900a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.187+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 0x559ae9431920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.187+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431ef0 0x559ae94369f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.187+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9437060 0x559ae94394f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.187+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559ae9396a40 con 0x559ae9431ef0 2024-03-18T00:52:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.187+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559ae933f920 con 0x559ae9437060 2024-03-18T00:52:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.187+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ae9392ae0 con 0x559ae9431500 2024-03-18T00:52:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.188+0000 7f4ea659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 0x559ae9431920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.188+0000 7f4ea659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 0x559ae9431920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:40756/0 (socket says 172.21.15.49:40756) 2024-03-18T00:52:43.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.188+0000 7f4ea659c640 1 -- 172.21.15.49:0/2291005879 learned_addr learned my addr 172.21.15.49:0/2291005879 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.189+0000 7f4ea659c640 1 -- 172.21.15.49:0/2291005879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9437060 msgr2=0x559ae94394f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.189+0000 7f4ea659c640 1 --2- 172.21.15.49:0/2291005879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9437060 0x559ae94394f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.189+0000 7f4ea659c640 1 -- 172.21.15.49:0/2291005879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431ef0 msgr2=0x559ae94369f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:52:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.189+0000 7f4ea659c640 1 --2- 172.21.15.49:0/2291005879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431ef0 0x559ae94369f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.189+0000 7f4ea659c640 1 -- 172.21.15.49:0/2291005879 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ae9357460 con 0x559ae9431500 2024-03-18T00:52:43.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.190+0000 7f4ea659c640 1 --2- 172.21.15.49:0/2291005879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 0x559ae9431920 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4e9802c720 tx=0x7f4e98000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c10c54451cbc3cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:43.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.192+0000 7f4e94ff9640 1 -- 172.21.15.49:0/2291005879 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e98042070 con 0x559ae9431500 2024-03-18T00:52:43.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.192+0000 7f4e94ff9640 1 -- 172.21.15.49:0/2291005879 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e9802ccd0 con 0x559ae9431500 2024-03-18T00:52:43.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.192+0000 7f4e94ff9640 1 -- 172.21.15.49:0/2291005879 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e980053c0 con 0x559ae9431500 2024-03-18T00:52:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.196+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2291005879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 msgr2=0x559ae9431920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:43.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.196+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2291005879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 0x559ae9431920 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4e9802c720 tx=0x7f4e98000bc0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.198+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2291005879 shutdown_connections 2024-03-18T00:52:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.198+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2291005879 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9437060 0x559ae94394f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.198+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2291005879 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431ef0 0x559ae94369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.198+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2291005879 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 0x559ae9431920 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.198+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2291005879 >> 172.21.15.49:0/2291005879 conn(0x559ae942ca70 msgr2=0x559ae942ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.200+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2291005879 shutdown_connections 2024-03-18T00:52:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.200+0000 7f4ea6d9d640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4e900a2980 0x7f4e900a4e40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009f30 tx=0x7f4ea0009940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.200+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2291005879 wait complete. 2024-03-18T00:52:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.200+0000 7f4eafbf6740 1 -- start start 2024-03-18T00:52:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.201+0000 7f4eafbf6740 1 Processor -- start 2024-03-18T00:52:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.201+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431500 0x559ae943b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.201+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431ef0 0x559ae943bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.201+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 0x559ae9442d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.202+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559ae9439fb0 con 0x559ae9431500 2024-03-18T00:52:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.202+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559ae9352d70 con 0x559ae9431ef0 2024-03-18T00:52:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.202+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559ae943a130 con 0x559ae9437060 2024-03-18T00:52:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.202+0000 7f4eafbf6740 10 client.0 _ll_register_callbacks cb 0x559ae90aaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.204+0000 7f4ea659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431500 0x559ae943b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.204+0000 7f4ea659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431500 0x559ae943b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44412/0 (socket says 172.21.15.49:44412) 2024-03-18T00:52:43.204 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.204+0000 7f4ea659c640 1 -- 172.21.15.49:0/1806677285 learned_addr learned my addr 172.21.15.49:0/1806677285 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:43.206 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.206+0000 7f4ea6d9d640 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 0x559ae9442d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.211+0000 7f4ea659c640 1 -- 172.21.15.49:0/1806677285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 msgr2=0x559ae9442d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.211+0000 7f4ea659c640 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 0x559ae9442d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.211+0000 7f4ea659c640 1 -- 172.21.15.49:0/1806677285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431ef0 msgr2=0x559ae943bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.211+0000 7f4ea659c640 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431ef0 0x559ae943bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.211+0000 7f4ea659c640 1 -- 172.21.15.49:0/1806677285 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ae90ab210 con 0x559ae9431500 2024-03-18T00:52:43.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.214+0000 7f4ea659c640 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431500 0x559ae943b7c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009560 tx=0x7f4e98009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85e548690891347 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:43.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.214+0000 7f4e8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:43.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.216+0000 7f4e8dffb640 1 -- 172.21.15.49:0/1806677285 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9804d030 con 0x559ae9431500 2024-03-18T00:52:43.217 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.216+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1806677285 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e9802ccd0 con 0x559ae9431500 2024-03-18T00:52:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.216+0000 7f4e8dffb640 1 -- 172.21.15.49:0/1806677285 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4e9802cad0 con 0x559ae9431500 2024-03-18T00:52:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.217+0000 7f4e8dffb640 1 -- 172.21.15.49:0/1806677285 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4e98058070 con 0x559ae9431500 2024-03-18T00:52:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.217+0000 7f4eafbf6740 10 client.16035 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.217+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1806677285 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559ae9177620 con 0x559ae9431500 2024-03-18T00:52:43.221 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.220+0000 7f4e8dffb640 1 -- 172.21.15.49:0/1806677285 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e9802ccd0 con 0x559ae9431500 2024-03-18T00:52:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.222+0000 7f4e8dffb640 1 -- 172.21.15.49:0/1806677285 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f4e980544d0 con 0x559ae9431500 2024-03-18T00:52:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.222+0000 7f4eafbf6740 10 client.16035 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.222+0000 7f4eafbf6740 10 client.16035 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.222+0000 7f4eafbf6740 20 client.16035 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.222+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559ae9447790 0x559ae9449bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.222+0000 7f4eafbf6740 4 client.16035 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.222+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1806677285 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ae93a3f80 con 0x559ae9447790 2024-03-18T00:52:43.227 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.227+0000 7f4ea5d9b640 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559ae9447790 0x559ae9449bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.250 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.250+0000 7f4ea5d9b640 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559ae9447790 0x559ae9449bf0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8881f8b1f7d75105 server_cookie=2c6bc5597d24893c in_seq=0 out_seq=0 2024-03-18T00:52:43.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.250+0000 7f4e8dffb640 10 client.16035 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.265+0000 7f4e8dffb640 1 -- 172.21.15.49:0/1806677285 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559ae93a3f80 con 0x559ae9447790 2024-03-18T00:52:43.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.265+0000 7f4e8dffb640 10 client.16035 handle_command_reply: tid=0 2024-03-18T00:52:43.267 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.265+0000 7f4eafbf6740 1 client.16035 shutdown 2024-03-18T00:52:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.270+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1806677285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431500 msgr2=0x559ae943b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.270+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431500 0x559ae943b7c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009560 tx=0x7f4e98009260 comp rx=0 tx=0).stop 2024-03-18T00:52:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.273+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1806677285 shutdown_connections 2024-03-18T00:52:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.273+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559ae9447790 0x559ae9449bf0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.273+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431ef0 0x559ae943bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.273+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 0x559ae9442d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.273+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1806677285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431500 0x559ae943b7c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.273+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1806677285 >> 172.21.15.49:0/1806677285 conn(0x559ae942ca70 msgr2=0x559ae942bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.274+0000 7f4e8dffb640 0 client.16035 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.275+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1806677285 shutdown_connections 2024-03-18T00:52:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.275+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1806677285 wait complete. 2024-03-18T00:52:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.275+0000 7f4eafbf6740 20 client.16035 trim_cache size 0 max 16384 2024-03-18T00:52:43.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.342+0000 7f4eafbf6740 1 Processor -- start 2024-03-18T00:52:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.345+0000 7f4eafbf6740 1 -- start start 2024-03-18T00:52:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.346+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 0x559ae942ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.346+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431ef0 0x559ae9442c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.346+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9437060 0x559ae943ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.346+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8800e4c0 con 0x559ae9437060 2024-03-18T00:52:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.346+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4e88010030 con 0x559ae9431ef0 2024-03-18T00:52:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.346+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8800eb90 con 0x559ae9431500 2024-03-18T00:52:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.346+0000 7f4ea6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9437060 0x559ae943ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.346+0000 7f4ea6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9437060 0x559ae943ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44424/0 (socket says 172.21.15.49:44424) 2024-03-18T00:52:43.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.347+0000 7f4ea6d9d640 1 -- 172.21.15.49:0/1504941592 learned_addr learned my addr 172.21.15.49:0/1504941592 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.348+0000 7f4ea6d9d640 1 -- 172.21.15.49:0/1504941592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 msgr2=0x559ae942ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.348+0000 7f4ea6d9d640 1 --2- 172.21.15.49:0/1504941592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 0x559ae942ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.348+0000 7f4ea6d9d640 1 -- 172.21.15.49:0/1504941592 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431ef0 msgr2=0x559ae9442c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.348+0000 7f4ea6d9d640 1 --2- 172.21.15.49:0/1504941592 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431ef0 0x559ae9442c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.348+0000 7f4ea6d9d640 1 -- 172.21.15.49:0/1504941592 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ae942c3c0 con 0x559ae9437060 2024-03-18T00:52:43.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.352+0000 7f4ea6d9d640 1 --2- 172.21.15.49:0/1504941592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9437060 0x559ae943ade0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0017810 tx=0x7f4ea0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3649c24d6fc08de3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.356+0000 7f4e8ffff640 1 -- 172.21.15.49:0/1504941592 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0023410 con 0x559ae9437060 2024-03-18T00:52:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.356+0000 7f4e8ffff640 1 -- 172.21.15.49:0/1504941592 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ea0028070 con 0x559ae9437060 2024-03-18T00:52:43.356 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.356+0000 7f4e8ffff640 1 -- 172.21.15.49:0/1504941592 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ea00219a0 con 0x559ae9437060 2024-03-18T00:52:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.359+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1504941592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9437060 msgr2=0x559ae943ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.359+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1504941592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9437060 0x559ae943ade0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0017810 tx=0x7f4ea0015000 comp rx=0 tx=0).stop 2024-03-18T00:52:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.359+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1504941592 shutdown_connections 2024-03-18T00:52:43.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.359+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1504941592 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431ef0 0x559ae9442c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.359+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1504941592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9437060 0x559ae943ade0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.359+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/1504941592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9431500 0x559ae942ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.359+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1504941592 >> 172.21.15.49:0/1504941592 conn(0x559ae942ca70 msgr2=0x559ae942bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.361+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1504941592 shutdown_connections 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.362+0000 7f4eafbf6740 1 -- 172.21.15.49:0/1504941592 wait complete. 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.363+0000 7f4eafbf6740 1 -- start start 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.363+0000 7f4eafbf6740 1 Processor -- start 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.363+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431500 0x559ae94403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.363+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431ef0 0x559ae9440900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.363+0000 7f4eafbf6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 0x559ae94418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.363+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8800f930 con 0x559ae9431ef0 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.363+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8800f260 con 0x559ae9431500 2024-03-18T00:52:43.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.363+0000 7f4eafbf6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8800c550 con 0x559ae9437060 2024-03-18T00:52:43.365 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.363+0000 7f4eafbf6740 10 client.0 _ll_register_callbacks cb 0x7f4e98004b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.364+0000 7f4ea659c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431500 0x559ae94403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.364+0000 7f4ea659c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431500 0x559ae94403c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:58998/0 (socket says 172.21.15.49:58998) 2024-03-18T00:52:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.364+0000 7f4ea659c640 1 -- 172.21.15.49:0/2944151627 learned_addr learned my addr 172.21.15.49:0/2944151627 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.365+0000 7f4ea6d9d640 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 0x559ae94418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.366+0000 7f4ea5d9b640 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431ef0 0x559ae9440900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.368+0000 7f4ea659c640 1 -- 172.21.15.49:0/2944151627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 msgr2=0x559ae94418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.368+0000 7f4ea659c640 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 0x559ae94418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.368+0000 7f4ea659c640 1 -- 172.21.15.49:0/2944151627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431ef0 msgr2=0x559ae9440900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.368+0000 7f4ea659c640 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431ef0 0x559ae9440900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:43.375 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.368+0000 7f4ea659c640 1 -- 172.21.15.49:0/2944151627 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ae9441eb0 con 0x559ae9431500 2024-03-18T00:52:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.376+0000 7f4ea659c640 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431500 0x559ae94403c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e98004f50 tx=0x7f4e98054cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90689bd945ffb90b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:43.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.376+0000 7f4e8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:52:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.380+0000 7f4e8dffb640 1 -- 172.21.15.49:0/2944151627 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9804d070 con 0x559ae9431500 2024-03-18T00:52:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.380+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2944151627 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ea0028070 con 0x559ae9431500 2024-03-18T00:52:43.380 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.380+0000 7f4e8dffb640 1 -- 172.21.15.49:0/2944151627 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4e98005000 con 0x559ae9431500 2024-03-18T00:52:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.384+0000 7f4e8dffb640 1 -- 172.21.15.49:0/2944151627 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4e98042050 con 0x559ae9431500 2024-03-18T00:52:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.384+0000 7f4eafbf6740 10 client.25691 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:43.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.384+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2944151627 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559ae94adaf0 con 0x559ae9431500 2024-03-18T00:52:43.386 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.386+0000 7f4e8dffb640 1 -- 172.21.15.49:0/2944151627 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ea0028070 con 0x559ae9431500 2024-03-18T00:52:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.386+0000 7f4e8dffb640 1 -- 172.21.15.49:0/2944151627 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f4e980542d0 con 0x559ae9431500 2024-03-18T00:52:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.387+0000 7f4eafbf6740 10 client.25691 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.387+0000 7f4eafbf6740 10 client.25691 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.387+0000 7f4eafbf6740 20 client.25691 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.387+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559ae94ae310 0x559ae94b0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.387+0000 7f4eafbf6740 4 client.25691 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:52:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.387+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2944151627 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ae939e2d0 con 0x559ae94ae310 2024-03-18T00:52:43.388 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.388+0000 7f4ea5d9b640 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559ae94ae310 0x559ae94b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:43.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.394+0000 7f4ea5d9b640 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559ae94ae310 0x559ae94b0730 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f799a9b5345abfb server_cookie=b53529c595c0eaa7 in_seq=0 out_seq=0 2024-03-18T00:52:43.394 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:43.394+0000 7f4e8dffb640 10 client.25691 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:44.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:44 smithi106 ceph-mon[35899]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1015 MiB/s wr, 1.64k op/s 2024-03-18T00:52:45.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:52:45.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:52:45.055 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:44 smithi049 ceph-mon[32130]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1015 MiB/s wr, 1.64k op/s 2024-03-18T00:52:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:44 smithi107 ceph-mon[36270]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1015 MiB/s wr, 1.64k op/s 2024-03-18T00:52:45.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.336+0000 7fdf02c8a640 1 -- 172.21.15.49:0/2327983312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 msgr2=0x7fdefc10d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:45.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.336+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/2327983312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 0x7fdefc10d5c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009860 tx=0x7fdef002bb90 comp rx=0 tx=0).stop 2024-03-18T00:52:45.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.336+0000 7fdf02c8a640 1 -- 172.21.15.49:0/2327983312 shutdown_connections 2024-03-18T00:52:45.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.336+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/2327983312 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdefc10db00 0x7fdefc10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.336+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/2327983312 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 0x7fdefc10d5c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.336+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/2327983312 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdefc1087e0 0x7fdefc10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.336+0000 7fdf02c8a640 1 -- 172.21.15.49:0/2327983312 >> 172.21.15.49:0/2327983312 conn(0x7fdefc06e640 msgr2=0x7fdefc070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:45.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.347+0000 7fdf02c8a640 1 -- 172.21.15.49:0/2327983312 shutdown_connections 2024-03-18T00:52:45.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.347+0000 7fdf02c8a640 1 -- 172.21.15.49:0/2327983312 wait complete. 2024-03-18T00:52:45.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.348+0000 7fdf02c8a640 1 Processor -- start 2024-03-18T00:52:45.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.348+0000 7fdf02c8a640 1 -- start start 2024-03-18T00:52:45.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.348+0000 7fdf02c8a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdefc1087e0 0x7fdefc19e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.348+0000 7fdf02c8a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 0x7fdefc19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.349+0000 7fdf02c8a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdefc10db00 0x7fdefc1c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.349+0000 7fdf02c8a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc111b00 con 0x7fdefc1087e0 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.349+0000 7fdf02c8a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc111980 con 0x7fdefc10db00 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.349+0000 7fdf02c8a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdefc111c80 con 0x7fdefc10b150 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.349+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 0x7fdefc19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.349+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 0x7fdefc19e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:40802/0 (socket says 172.21.15.49:40802) 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.349+0000 7fdefbfff640 1 -- 172.21.15.49:0/3043264981 learned_addr learned my addr 172.21.15.49:0/3043264981 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.349+0000 7fdf01227640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdefc10db00 0x7fdefc1c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.350+0000 7fdefbfff640 1 -- 172.21.15.49:0/3043264981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdefc10db00 msgr2=0x7fdefc1c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.350+0000 7fdefbfff640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdefc10db00 0x7fdefc1c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.350+0000 7fdefbfff640 1 -- 172.21.15.49:0/3043264981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdefc1087e0 msgr2=0x7fdefc19e020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.350+0000 7fdefbfff640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdefc1087e0 0x7fdefc19e020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.350+0000 7fdefbfff640 1 -- 172.21.15.49:0/3043264981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdefc1c7c50 con 0x7fdefc10b150 2024-03-18T00:52:45.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.351+0000 7fdefbfff640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 0x7fdefc19e560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdef002c1a0 tx=0x7fdef0004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.364+0000 7fdef9ffb640 1 -- 172.21.15.49:0/3043264981 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0041020 con 0x7fdefc10b150 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.364+0000 7fdf02c8a640 1 -- 172.21.15.49:0/3043264981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdefc1c7ee0 con 0x7fdefc10b150 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.364+0000 7fdf02c8a640 1 -- 172.21.15.49:0/3043264981 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdefc1c83c0 con 0x7fdefc10b150 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.364+0000 7fdef9ffb640 1 -- 172.21.15.49:0/3043264981 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef002ca10 con 0x7fdefc10b150 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.364+0000 7fdef9ffb640 1 -- 172.21.15.49:0/3043264981 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0039040 con 0x7fdefc10b150 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.368+0000 7fdef9ffb640 1 -- 172.21.15.49:0/3043264981 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdef0044020 con 0x7fdefc10b150 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.370+0000 7fdef9ffb640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdee40a2980 0x7fdee40a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.370+0000 7fdef9ffb640 1 -- 172.21.15.49:0/3043264981 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fdef00f0490 con 0x7fdefc10b150 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.371+0000 7fdf05871740 1 Processor -- start 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.372+0000 7fdf05871740 1 -- start start 2024-03-18T00:52:45.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.372+0000 7fdf05871740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770af590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.372+0000 7fdf05871740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.372+0000 7fdf05871740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770b7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.372+0000 7fdf05871740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558c77014500 con 0x558c770afb60 2024-03-18T00:52:45.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.372+0000 7fdf05871740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558c76fbcbe0 con 0x558c770b4cd0 2024-03-18T00:52:45.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.372+0000 7fdf05871740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558c770105a0 con 0x558c770af170 2024-03-18T00:52:45.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.373+0000 7fdf01227640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:45.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.373+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:45.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.373+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44458/0 (socket says 172.21.15.49:44458) 2024-03-18T00:52:45.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.373+0000 7fdefbfff640 1 -- 172.21.15.49:0/736582499 learned_addr learned my addr 172.21.15.49:0/736582499 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:45.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.373+0000 7fdf00a26640 1 --2- 172.21.15.49:0/736582499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:45.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.373+0000 7fdf00a26640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdee40a2980 0x7fdee40a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:45.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.374+0000 7fdf00a26640 1 -- 172.21.15.49:0/736582499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 msgr2=0x558c770b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:45.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.374+0000 7fdf00a26640 1 --2- 172.21.15.49:0/736582499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.374+0000 7fdf00a26640 1 -- 172.21.15.49:0/736582499 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 msgr2=0x558c770b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:45.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.374+0000 7fdf00a26640 1 --2- 172.21.15.49:0/736582499 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.374+0000 7fdf00a26640 1 -- 172.21.15.49:0/736582499 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c76fd04b0 con 0x558c770af170 2024-03-18T00:52:45.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.375+0000 7fdf00a26640 1 --2- 172.21.15.49:0/736582499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770af590 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdef4012810 tx=0x7fdef4012cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27b944c249bdd21f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:45.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.376+0000 7fdee2ffd640 1 -- 172.21.15.49:0/736582499 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef4012e00 con 0x558c770af170 2024-03-18T00:52:45.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.376+0000 7fdee2ffd640 1 -- 172.21.15.49:0/736582499 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef4004d90 con 0x558c770af170 2024-03-18T00:52:45.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.377+0000 7fdf05871740 1 -- 172.21.15.49:0/736582499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 msgr2=0x558c770af590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:45.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.377+0000 7fdf05871740 1 --2- 172.21.15.49:0/736582499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770af590 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdef4012810 tx=0x7fdef4012cd0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.377+0000 7fdf05871740 1 -- 172.21.15.49:0/736582499 shutdown_connections 2024-03-18T00:52:45.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.377+0000 7fdf05871740 1 --2- 172.21.15.49:0/736582499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.377+0000 7fdf05871740 1 --2- 172.21.15.49:0/736582499 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.377+0000 7fdf05871740 1 --2- 172.21.15.49:0/736582499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770af590 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.377+0000 7fdf05871740 1 -- 172.21.15.49:0/736582499 >> 172.21.15.49:0/736582499 conn(0x558c770aa6e0 msgr2=0x558c770aaae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:45.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.380+0000 7fdf00a26640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdee40a2980 0x7fdee40a4e40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdef4004fd0 tx=0x7fdef401ad30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:45.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.384+0000 7fdf05871740 1 -- 172.21.15.49:0/736582499 shutdown_connections 2024-03-18T00:52:45.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.385+0000 7fdf05871740 1 -- 172.21.15.49:0/736582499 wait complete. 2024-03-18T00:52:45.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.385+0000 7fdf05871740 1 -- start start 2024-03-18T00:52:45.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.386+0000 7fdf05871740 1 Processor -- start 2024-03-18T00:52:45.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.386+0000 7fdf05871740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770af170 0x558c770b9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf05871740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770afb60 0x558c770b9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf05871740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770b4cd0 0x558c770c0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf05871740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558c770b7d20 con 0x558c770b4cd0 2024-03-18T00:52:45.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf05871740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558c76fd0160 con 0x558c770af170 2024-03-18T00:52:45.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf05871740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558c770b7ea0 con 0x558c770afb60 2024-03-18T00:52:45.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf05871740 10 client.0 _ll_register_callbacks cb 0x558c76fb4150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:45.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf00a26640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770af170 0x558c770b9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:45.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf01227640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770b4cd0 0x558c770c0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:45.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf01227640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770b4cd0 0x558c770c0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44466/0 (socket says 172.21.15.49:44466) 2024-03-18T00:52:45.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.387+0000 7fdf01227640 1 -- 172.21.15.49:0/2032562799 learned_addr learned my addr 172.21.15.49:0/2032562799 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:45.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.391+0000 7fdf00a26640 1 -- 172.21.15.49:0/2032562799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770afb60 msgr2=0x558c770b9ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:45.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.391+0000 7fdf00a26640 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770afb60 0x558c770b9ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.391+0000 7fdf00a26640 1 -- 172.21.15.49:0/2032562799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770b4cd0 msgr2=0x558c770c0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:45.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.391+0000 7fdf00a26640 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770b4cd0 0x558c770c0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:45.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.391+0000 7fdf00a26640 1 -- 172.21.15.49:0/2032562799 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c76df27d0 con 0x558c770af170 2024-03-18T00:52:45.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.392+0000 7fdf00a26640 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770af170 0x558c770b9590 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdef40127e0 tx=0x7fdef400b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6906df14e0576d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:45.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.394+0000 7fdedbfff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:52:45.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.394+0000 7fdedbfff640 1 -- 172.21.15.49:0/2032562799 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef400b940 con 0x558c770af170 2024-03-18T00:52:45.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.394+0000 7fdedbfff640 1 -- 172.21.15.49:0/2032562799 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fdef40060c0 con 0x558c770af170 2024-03-18T00:52:45.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.394+0000 7fdf05871740 1 -- 172.21.15.49:0/2032562799 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef4004d90 con 0x558c770af170 2024-03-18T00:52:45.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.394+0000 7fdedbfff640 1 -- 172.21.15.49:0/2032562799 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdef401bac0 con 0x558c770af170 2024-03-18T00:52:45.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.394+0000 7fdf05871740 10 client.25700 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:45.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.394+0000 7fdf05871740 1 -- 172.21.15.49:0/2032562799 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558c76d283f0 con 0x558c770af170 2024-03-18T00:52:45.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.395+0000 7fdedbfff640 1 -- 172.21.15.49:0/2032562799 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef4004d90 con 0x558c770af170 2024-03-18T00:52:45.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.395+0000 7fdedbfff640 1 -- 172.21.15.49:0/2032562799 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fdef4019070 con 0x558c770af170 2024-03-18T00:52:45.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.395+0000 7fdf05871740 10 client.25700 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:45.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.395+0000 7fdf05871740 10 client.25700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:45.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.395+0000 7fdf05871740 20 client.25700 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:45.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.396+0000 7fdf05871740 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558c770c53e0 0x558c770c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:45.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.396+0000 7fdf05871740 4 client.25700 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:45.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.396+0000 7fdf05871740 1 -- 172.21.15.49:0/2032562799 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c77021a40 con 0x558c770c53e0 2024-03-18T00:52:45.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.408+0000 7fdefbfff640 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558c770c53e0 0x558c770c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:45.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.414+0000 7fdefbfff640 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558c770c53e0 0x558c770c78a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b58d3824be8ff11d server_cookie=9de93c3bff40cc9 in_seq=0 out_seq=0 2024-03-18T00:52:45.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:45.414+0000 7fdedbfff640 10 client.25700 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:46.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:46 smithi106 ceph-mon[35899]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 824 MiB/s wr, 1.38k op/s 2024-03-18T00:52:47.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:46 smithi049 ceph-mon[32130]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 824 MiB/s wr, 1.38k op/s 2024-03-18T00:52:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:46 smithi107 ceph-mon[36270]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 824 MiB/s wr, 1.38k op/s 2024-03-18T00:52:48.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:48 smithi107 ceph-mon[36270]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 891 MiB/s wr, 1.43k op/s 2024-03-18T00:52:49.128 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:48 smithi049 ceph-mon[32130]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 891 MiB/s wr, 1.43k op/s 2024-03-18T00:52:49.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:48 smithi106 ceph-mon[35899]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 891 MiB/s wr, 1.43k op/s 2024-03-18T00:52:51.128 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:50 smithi049 ceph-mon[32130]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 940 MiB/s wr, 1.38k op/s 2024-03-18T00:52:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:50 smithi107 ceph-mon[36270]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 940 MiB/s wr, 1.38k op/s 2024-03-18T00:52:51.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:50 smithi106 ceph-mon[35899]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 940 MiB/s wr, 1.38k op/s 2024-03-18T00:52:52.996 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:52 smithi106 ceph-mon[35899]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 807 MiB/s wr, 1.08k op/s 2024-03-18T00:52:53.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.089+0000 7f4e8dffb640 1 -- 172.21.15.49:0/2944151627 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559ae939e2d0 con 0x559ae94ae310 2024-03-18T00:52:53.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.089+0000 7f4e8dffb640 10 client.25691 handle_command_reply: tid=0 2024-03-18T00:52:53.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.089+0000 7f4eafbf6740 1 client.25691 shutdown 2024-03-18T00:52:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdedbfff640 1 -- 172.21.15.49:0/2032562799 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558c77021a40 con 0x558c770c53e0 2024-03-18T00:52:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdedbfff640 10 client.25700 handle_command_reply: tid=0 2024-03-18T00:52:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdf05871740 1 client.25700 shutdown 2024-03-18T00:52:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdf05871740 1 -- 172.21.15.49:0/2032562799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770af170 msgr2=0x558c770b9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdf05871740 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770af170 0x558c770b9590 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdef40127e0 tx=0x7fdef400b390 comp rx=0 tx=0).stop 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdf05871740 1 -- 172.21.15.49:0/2032562799 shutdown_connections 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdf05871740 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558c770c53e0 0x558c770c78a0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdf05871740 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770b4cd0 0x558c770c0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdf05871740 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770afb60 0x558c770b9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdf05871740 1 --2- 172.21.15.49:0/2032562799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770af170 0x558c770b9590 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.089+0000 7fdf05871740 1 -- 172.21.15.49:0/2032562799 >> 172.21.15.49:0/2032562799 conn(0x558c770aa6e0 msgr2=0x558c770a9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.090+0000 7fdedbfff640 0 client.25700 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.090+0000 7fdf05871740 1 -- 172.21.15.49:0/2032562799 shutdown_connections 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.090+0000 7fdf05871740 1 -- 172.21.15.49:0/2032562799 wait complete. 2024-03-18T00:52:53.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.090+0000 7fdf05871740 20 client.25700 trim_cache size 0 max 16384 2024-03-18T00:52:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.092+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2944151627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431500 msgr2=0x559ae94403c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.093 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.092+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431500 0x559ae94403c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4e98004f50 tx=0x7f4e98054cf0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.094+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2944151627 shutdown_connections 2024-03-18T00:52:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.095+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559ae94ae310 0x559ae94b0730 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.095+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559ae9437060 0x559ae94418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.095+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559ae9431ef0 0x559ae9440900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.095+0000 7f4eafbf6740 1 --2- 172.21.15.49:0/2944151627 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559ae9431500 0x559ae94403c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.095 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.095+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2944151627 >> 172.21.15.49:0/2944151627 conn(0x559ae942ca70 msgr2=0x559ae942bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:53.096 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.095+0000 7f4e8dffb640 0 client.25691 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:53.096 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.095+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2944151627 shutdown_connections 2024-03-18T00:52:53.096 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.095+0000 7f4eafbf6740 1 -- 172.21.15.49:0/2944151627 wait complete. 2024-03-18T00:52:53.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:52:53.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:52:53.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:52:53.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:52:53.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.095+0000 7f4eafbf6740 20 client.25691 trim_cache size 0 max 16384 2024-03-18T00:52:53.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.101+0000 7f4eacf37640 1 -- 172.21.15.49:0/182227797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4e900a2980 msgr2=0x7f4e900a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.102+0000 7f4eacf37640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4e900a2980 0x7f4e900a4e40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009f30 tx=0x7f4ea0009940 comp rx=0 tx=0).stop 2024-03-18T00:52:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.102+0000 7f4eacf37640 1 -- 172.21.15.49:0/182227797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea8073410 msgr2=0x7f4ea81a6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.102+0000 7f4eacf37640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea8073410 0x7f4ea81a6dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00b3c0 tx=0x7f4e9c00b880 comp rx=0 tx=0).stop 2024-03-18T00:52:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.106+0000 7f4eacf37640 1 -- 172.21.15.49:0/182227797 shutdown_connections 2024-03-18T00:52:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.106+0000 7f4eacf37640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4e900a2980 0x7f4e900a4e40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.106+0000 7f4eacf37640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4ea8109510 0x7f4ea81ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.106+0000 7f4eacf37640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4ea8073410 0x7f4ea81a6dc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.106+0000 7f4eacf37640 1 --2- 172.21.15.49:0/182227797 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4ea8072ab0 0x7f4ea81a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.106+0000 7f4eacf37640 1 -- 172.21.15.49:0/182227797 >> 172.21.15.49:0/182227797 conn(0x7f4ea806e650 msgr2=0x7f4ea80702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:53.108 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.108+0000 7f4eacf37640 1 -- 172.21.15.49:0/182227797 shutdown_connections 2024-03-18T00:52:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:52:53.112+0000 7f4eacf37640 1 -- 172.21.15.49:0/182227797 wait complete. 2024-03-18T00:52:53.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:52 smithi049 ceph-mon[32130]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 807 MiB/s wr, 1.08k op/s 2024-03-18T00:52:53.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.162+0000 7fdf05871740 1 Processor -- start 2024-03-18T00:52:53.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.162+0000 7fdf05871740 1 -- start start 2024-03-18T00:52:53.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.162+0000 7fdf05871740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.163+0000 7fdf05871740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.163+0000 7fdf05871740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770b98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.163+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:53.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.163+0000 7fdefbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770c0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40370/0 (socket says 172.21.15.49:40370) 2024-03-18T00:52:53.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.163+0000 7fdefbfff640 1 -- 172.21.15.49:0/710379658 learned_addr learned my addr 172.21.15.49:0/710379658 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:53.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.163+0000 7fdf05871740 1 -- 172.21.15.49:0/710379658 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdee800e640 con 0x558c770afb60 2024-03-18T00:52:53.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.163+0000 7fdf05871740 1 -- 172.21.15.49:0/710379658 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdee80101b0 con 0x558c770b4cd0 2024-03-18T00:52:53.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.163+0000 7fdf05871740 1 -- 172.21.15.49:0/710379658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdee800ed10 con 0x558c770af170 2024-03-18T00:52:53.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.164+0000 7fdf01227640 1 --2- 172.21.15.49:0/710379658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770b98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:53.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.164+0000 7fdf00a26640 1 --2- 172.21.15.49:0/710379658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:53.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.164+0000 7fdefbfff640 1 -- 172.21.15.49:0/710379658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 msgr2=0x558c770b8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:53.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.164+0000 7fdefbfff640 1 --2- 172.21.15.49:0/710379658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770b8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.164+0000 7fdefbfff640 1 -- 172.21.15.49:0/710379658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 msgr2=0x558c770b98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:53.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.164+0000 7fdefbfff640 1 --2- 172.21.15.49:0/710379658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770b98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.164+0000 7fdefbfff640 1 -- 172.21.15.49:0/710379658 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c76ce80f0 con 0x558c770afb60 2024-03-18T00:52:53.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.165+0000 7fdefbfff640 1 --2- 172.21.15.49:0/710379658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770c0ad0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdef0005b60 tx=0x7fdef0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccdd673983d5a5c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:53.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.174+0000 7fdee27fc640 1 -- 172.21.15.49:0/710379658 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00c0f00 con 0x558c770afb60 2024-03-18T00:52:53.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.174+0000 7fdee27fc640 1 -- 172.21.15.49:0/710379658 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef00c5550 con 0x558c770afb60 2024-03-18T00:52:53.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.175+0000 7fdee27fc640 1 -- 172.21.15.49:0/710379658 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef004d300 con 0x558c770afb60 2024-03-18T00:52:53.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.176+0000 7fdf05871740 1 -- 172.21.15.49:0/710379658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 msgr2=0x558c770c0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.177+0000 7fdf05871740 1 --2- 172.21.15.49:0/710379658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770c0ad0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fdef0005b60 tx=0x7fdef0000bc0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.178+0000 7fdf05871740 1 -- 172.21.15.49:0/710379658 shutdown_connections 2024-03-18T00:52:53.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.178+0000 7fdf05871740 1 --2- 172.21.15.49:0/710379658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770b98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.178+0000 7fdf05871740 1 --2- 172.21.15.49:0/710379658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770c0ad0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.178+0000 7fdf05871740 1 --2- 172.21.15.49:0/710379658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770b8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.178+0000 7fdf05871740 1 -- 172.21.15.49:0/710379658 >> 172.21.15.49:0/710379658 conn(0x558c770aa6e0 msgr2=0x558c770b91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:53.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:52 smithi107 ceph-mon[36270]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 807 MiB/s wr, 1.08k op/s 2024-03-18T00:52:53.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.180+0000 7fdf05871740 1 -- 172.21.15.49:0/710379658 shutdown_connections 2024-03-18T00:52:53.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.181+0000 7fdf05871740 1 -- 172.21.15.49:0/710379658 wait complete. 2024-03-18T00:52:53.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.181+0000 7fdf05871740 1 -- start start 2024-03-18T00:52:53.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.182+0000 7fdf05871740 1 Processor -- start 2024-03-18T00:52:53.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.184+0000 7fdf05871740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770bde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.184+0000 7fdf05871740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770be340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.184+0000 7fdf05871740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770bf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.185+0000 7fdf05871740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdee800fab0 con 0x558c770afb60 2024-03-18T00:52:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.185+0000 7fdf05871740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdee800f3e0 con 0x558c770b4cd0 2024-03-18T00:52:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.185+0000 7fdf05871740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdee800c620 con 0x558c770af170 2024-03-18T00:52:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.185+0000 7fdf05871740 10 client.0 _ll_register_callbacks cb 0x7fdeec01a7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.185+0000 7fdf00a26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770bde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.185+0000 7fdf00a26640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770bde00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34552/0 (socket says 172.21.15.49:34552) 2024-03-18T00:52:53.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.185+0000 7fdf00a26640 1 -- 172.21.15.49:0/2355943209 learned_addr learned my addr 172.21.15.49:0/2355943209 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:53.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.185+0000 7fdf01227640 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770bf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:53.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.187+0000 7fdefbfff640 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770be340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.190+0000 7fdf00a26640 1 -- 172.21.15.49:0/2355943209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 msgr2=0x558c770bf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.190+0000 7fdf00a26640 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770bf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.190+0000 7fdf00a26640 1 -- 172.21.15.49:0/2355943209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 msgr2=0x558c770be340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.190+0000 7fdf00a26640 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770be340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.190+0000 7fdf00a26640 1 -- 172.21.15.49:0/2355943209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c76fdb9d0 con 0x558c770af170 2024-03-18T00:52:53.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.194+0000 7fdf00a26640 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770bde00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdef401a600 tx=0x7fdef4010cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b79e631d22c652 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:53.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.197+0000 7fdf01227640 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770bf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:52:53.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.202+0000 7fdedbfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:52:53.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.202+0000 7fdedbfff640 1 -- 172.21.15.49:0/2355943209 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef403e030 con 0x558c770af170 2024-03-18T00:52:53.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.202+0000 7fdedbfff640 1 -- 172.21.15.49:0/2355943209 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fdef4020070 con 0x558c770af170 2024-03-18T00:52:53.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.203+0000 7fdf05871740 1 -- 172.21.15.49:0/2355943209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdef00c5550 con 0x558c770af170 2024-03-18T00:52:53.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.206+0000 7fdedbfff640 1 -- 172.21.15.49:0/2355943209 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdef401bc60 con 0x558c770af170 2024-03-18T00:52:53.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.206+0000 7fdf05871740 10 client.25783 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:53.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.206+0000 7fdf05871740 1 -- 172.21.15.49:0/2355943209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558c770c0aa0 con 0x558c770af170 2024-03-18T00:52:53.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.208+0000 7fdedbfff640 1 -- 172.21.15.49:0/2355943209 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdef00c5550 con 0x558c770af170 2024-03-18T00:52:53.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.209+0000 7fdedbfff640 1 -- 172.21.15.49:0/2355943209 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fdef4019040 con 0x558c770af170 2024-03-18T00:52:53.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.209+0000 7fdf05871740 10 client.25783 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:53.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.209+0000 7fdf05871740 10 client.25783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:53.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.209+0000 7fdf05871740 20 client.25783 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:53.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.209+0000 7fdf05871740 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558c7712bc00 0x558c7712e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.209+0000 7fdf05871740 4 client.25783 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:52:53.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.209+0000 7fdf05871740 1 -- 172.21.15.49:0/2355943209 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c7701bd90 con 0x558c7712bc00 2024-03-18T00:52:53.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.211+0000 7fdefbfff640 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558c7712bc00 0x558c7712e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:53.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.221+0000 7fdefbfff640 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558c7712bc00 0x558c7712e020 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=453648f1c905f7e6 server_cookie=a1a33d9d41b9562 in_seq=0 out_seq=0 2024-03-18T00:52:53.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.221+0000 7fdedbfff640 10 client.25783 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:53.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.231+0000 7fdedbfff640 1 -- 172.21.15.49:0/2355943209 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558c7701bd90 con 0x558c7712bc00 2024-03-18T00:52:53.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.231+0000 7fdedbfff640 10 client.25783 handle_command_reply: tid=0 2024-03-18T00:52:53.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.232+0000 7fdf05871740 1 client.25783 shutdown 2024-03-18T00:52:53.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.238+0000 7fdf05871740 1 -- 172.21.15.49:0/2355943209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 msgr2=0x558c770bde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.238+0000 7fdf05871740 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770bde00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdef401a600 tx=0x7fdef4010cb0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.243+0000 7fdf05871740 1 -- 172.21.15.49:0/2355943209 shutdown_connections 2024-03-18T00:52:53.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.243+0000 7fdf05871740 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558c7712bc00 0x558c7712e020 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.243+0000 7fdf05871740 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c770b4cd0 0x558c770bf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.243+0000 7fdf05871740 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c770afb60 0x558c770be340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.243+0000 7fdf05871740 1 --2- 172.21.15.49:0/2355943209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c770af170 0x558c770bde00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.243+0000 7fdf05871740 1 -- 172.21.15.49:0/2355943209 >> 172.21.15.49:0/2355943209 conn(0x558c770aa6e0 msgr2=0x558c770ad940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:53.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.243+0000 7fdedbfff640 0 client.25783 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:53.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.243+0000 7fdf05871740 1 -- 172.21.15.49:0/2355943209 shutdown_connections 2024-03-18T00:52:53.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.247+0000 7fdf05871740 1 -- 172.21.15.49:0/2355943209 wait complete. 2024-03-18T00:52:53.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.253+0000 7fdf05871740 20 client.25783 trim_cache size 0 max 16384 2024-03-18T00:52:53.260 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:52:53.260 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:52:53.260 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:52:53.260 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:52:53.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.262+0000 7fdf02c8a640 1 -- 172.21.15.49:0/3043264981 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdee40a2980 msgr2=0x7fdee40a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.262+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdee40a2980 0x7fdee40a4e40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdef4004fd0 tx=0x7fdef401ad30 comp rx=0 tx=0).stop 2024-03-18T00:52:53.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.263+0000 7fdf02c8a640 1 -- 172.21.15.49:0/3043264981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 msgr2=0x7fdefc19e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.263+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 0x7fdefc19e560 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdef002c1a0 tx=0x7fdef0004270 comp rx=0 tx=0).stop 2024-03-18T00:52:53.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.263+0000 7fdf02c8a640 1 -- 172.21.15.49:0/3043264981 shutdown_connections 2024-03-18T00:52:53.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.263+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdee40a2980 0x7fdee40a4e40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.263+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdefc10db00 0x7fdefc1c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.263+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdefc10b150 0x7fdefc19e560 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.264+0000 7fdf02c8a640 1 --2- 172.21.15.49:0/3043264981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdefc1087e0 0x7fdefc19e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.264+0000 7fdf02c8a640 1 -- 172.21.15.49:0/3043264981 >> 172.21.15.49:0/3043264981 conn(0x7fdefc06e640 msgr2=0x7fdefc10b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:53.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.265+0000 7fdf02c8a640 1 -- 172.21.15.49:0/3043264981 shutdown_connections 2024-03-18T00:52:53.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.265+0000 7fdf02c8a640 1 -- 172.21.15.49:0/3043264981 wait complete. 2024-03-18T00:52:53.306 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:52:53.307 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:52:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.690+0000 7f4be604b640 1 -- 172.21.15.49:0/2037135078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 msgr2=0x7f4be0108fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.690+0000 7f4be604b640 1 --2- 172.21.15.49:0/2037135078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 0x7f4be0108fa0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009900 tx=0x7f4bd402bc30 comp rx=0 tx=0).stop 2024-03-18T00:52:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.691+0000 7f4be604b640 1 -- 172.21.15.49:0/2037135078 shutdown_connections 2024-03-18T00:52:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.691+0000 7f4be604b640 1 --2- 172.21.15.49:0/2037135078 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be010a230 0x7f4be0110ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.691+0000 7f4be604b640 1 --2- 172.21.15.49:0/2037135078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0109890 0x7f4be0109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.691+0000 7f4be604b640 1 --2- 172.21.15.49:0/2037135078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 0x7f4be0108fa0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.691+0000 7f4be604b640 1 -- 172.21.15.49:0/2037135078 >> 172.21.15.49:0/2037135078 conn(0x7f4be006e650 msgr2=0x7f4be0070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:53.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.691+0000 7f4be604b640 1 -- 172.21.15.49:0/2037135078 shutdown_connections 2024-03-18T00:52:53.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.692+0000 7f4be604b640 1 -- 172.21.15.49:0/2037135078 wait complete. 2024-03-18T00:52:53.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.693+0000 7f4be604b640 1 Processor -- start 2024-03-18T00:52:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.694+0000 7f4be604b640 1 -- start start 2024-03-18T00:52:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.695+0000 7f4be604b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 0x7f4be0118150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.695+0000 7f4be604b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0109890 0x7f4be0118690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.695+0000 7f4be604b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be010a230 0x7f4be0119540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.695+0000 7f4be604b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4be0112780 con 0x7f4be0108b80 2024-03-18T00:52:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.695+0000 7f4be604b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4be0112600 con 0x7f4be010a230 2024-03-18T00:52:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.695+0000 7f4be604b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4be0112900 con 0x7f4be0109890 2024-03-18T00:52:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.695+0000 7f4bdf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 0x7f4be0118150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.695+0000 7f4bdf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 0x7f4be0118150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40416/0 (socket says 172.21.15.49:40416) 2024-03-18T00:52:53.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.695+0000 7f4bdf7fe640 1 -- 172.21.15.49:0/1503621198 learned_addr learned my addr 172.21.15.49:0/1503621198 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.696+0000 7f4bdf7fe640 1 -- 172.21.15.49:0/1503621198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0109890 msgr2=0x7f4be0118690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.696+0000 7f4bdf7fe640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0109890 0x7f4be0118690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.696+0000 7f4bdf7fe640 1 -- 172.21.15.49:0/1503621198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be010a230 msgr2=0x7f4be0119540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.696+0000 7f4bdf7fe640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be010a230 0x7f4be0119540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.696+0000 7f4bdf7fe640 1 -- 172.21.15.49:0/1503621198 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be0119cc0 con 0x7f4be0108b80 2024-03-18T00:52:53.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.697+0000 7f4bdf7fe640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 0x7f4be0118150 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4bd402c240 tx=0x7f4bd402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:53.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.699+0000 7f4bdcff9640 1 -- 172.21.15.49:0/1503621198 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4043020 con 0x7f4be0108b80 2024-03-18T00:52:53.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.699+0000 7f4bdcff9640 1 -- 172.21.15.49:0/1503621198 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd40054e0 con 0x7f4be0108b80 2024-03-18T00:52:53.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.699+0000 7f4bdcff9640 1 -- 172.21.15.49:0/1503621198 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4003cd0 con 0x7f4be0108b80 2024-03-18T00:52:53.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.699+0000 7f4be604b640 1 -- 172.21.15.49:0/1503621198 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be0119f50 con 0x7f4be0108b80 2024-03-18T00:52:53.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.699+0000 7f4be604b640 1 -- 172.21.15.49:0/1503621198 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be01b7610 con 0x7f4be0108b80 2024-03-18T00:52:53.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.708+0000 7f4be604b640 1 -- 172.21.15.49:0/1503621198 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bcc0052c0 con 0x7f4be0108b80 2024-03-18T00:52:53.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.708+0000 7f4bdcff9640 1 -- 172.21.15.49:0/1503621198 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4bd4046020 con 0x7f4be0108b80 2024-03-18T00:52:53.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.709+0000 7f4bdcff9640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2a00 0x7f4bc80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:53.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.710+0000 7f4bdcff9640 1 -- 172.21.15.49:0/1503621198 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4bd40f0cd0 con 0x7f4be0108b80 2024-03-18T00:52:53.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.713+0000 7f4bdeffd640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2a00 0x7f4bc80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:53.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.714+0000 7f4bdeffd640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2a00 0x7f4bc80a4ec0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000b990 tx=0x7f4bd000b370 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:53.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:53.724+0000 7f4bdcff9640 1 -- 172.21.15.49:0/1503621198 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4bd404d020 con 0x7f4be0108b80 2024-03-18T00:52:54.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.369+0000 7f4be604b640 1 -- 172.21.15.49:0/1503621198 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bcc0055b0 con 0x7f4be0108b80 2024-03-18T00:52:54.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.373+0000 7f4bdcff9640 1 -- 172.21.15.49:0/1503621198 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11933 (secure 0 0 0) 0x7f4bd40bdcf0 con 0x7f4be0108b80 2024-03-18T00:52:54.388 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:52:54.388 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:51:14.980346+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24506,24499,24481,24463,24470]},"id":1}]} 2024-03-18T00:52:54.388 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 26 2024-03-18T00:52:54.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.398+0000 7f4bc67fc640 1 -- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2a00 msgr2=0x7f4bc80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:54.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.398+0000 7f4bc67fc640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2a00 0x7f4bc80a4ec0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000b990 tx=0x7f4bd000b370 comp rx=0 tx=0).stop 2024-03-18T00:52:54.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.398+0000 7f4bc67fc640 1 -- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 msgr2=0x7f4be0118150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:54.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.398+0000 7f4bc67fc640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 0x7f4be0118150 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4bd402c240 tx=0x7f4bd402c5d0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.399+0000 7f4bc67fc640 1 -- 172.21.15.49:0/1503621198 shutdown_connections 2024-03-18T00:52:54.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.399+0000 7f4bc67fc640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4bc80a2a00 0x7f4bc80a4ec0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.399+0000 7f4bc67fc640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4be010a230 0x7f4be0119540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.399+0000 7f4bc67fc640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4be0109890 0x7f4be0118690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.399+0000 7f4bc67fc640 1 --2- 172.21.15.49:0/1503621198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4be0108b80 0x7f4be0118150 secure :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4bd402c240 tx=0x7f4bd402c5d0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.399+0000 7f4bc67fc640 1 -- 172.21.15.49:0/1503621198 >> 172.21.15.49:0/1503621198 conn(0x7f4be006e650 msgr2=0x7f4be010cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:54.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.404+0000 7f4bc67fc640 1 -- 172.21.15.49:0/1503621198 shutdown_connections 2024-03-18T00:52:54.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.404+0000 7f4bc67fc640 1 -- 172.21.15.49:0/1503621198 wait complete. 2024-03-18T00:52:54.492 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:52:54.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:54 smithi049 ceph-mon[32130]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 996 MiB/s wr, 1.49k op/s 2024-03-18T00:52:54.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:54 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1503621198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:52:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.980+0000 7f4926aa7640 1 -- 172.21.15.49:0/2830041555 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4920109870 msgr2=0x7f492010e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:54.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.981+0000 7f4926aa7640 1 --2- 172.21.15.49:0/2830041555 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4920109870 0x7f492010e110 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f491c009900 tx=0x7f491c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:52:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.985+0000 7f4926aa7640 1 -- 172.21.15.49:0/2830041555 shutdown_connections 2024-03-18T00:52:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.985+0000 7f4926aa7640 1 --2- 172.21.15.49:0/2830041555 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4920109870 0x7f492010e110 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.985+0000 7f4926aa7640 1 --2- 172.21.15.49:0/2830041555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4920108eb0 0x7f4920109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.985+0000 7f4926aa7640 1 --2- 172.21.15.49:0/2830041555 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4920072bf0 0x7f4920073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.985+0000 7f4926aa7640 1 -- 172.21.15.49:0/2830041555 >> 172.21.15.49:0/2830041555 conn(0x7f492006e6c0 msgr2=0x7f4920070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:54.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.986+0000 7f4926aa7640 1 -- 172.21.15.49:0/2830041555 shutdown_connections 2024-03-18T00:52:54.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.987+0000 7f4926aa7640 1 -- 172.21.15.49:0/2830041555 wait complete. 2024-03-18T00:52:54.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.988+0000 7f4926aa7640 1 Processor -- start 2024-03-18T00:52:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.988+0000 7f4926aa7640 1 -- start start 2024-03-18T00:52:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.993+0000 7f4926aa7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4920072bf0 0x7f49201a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:54.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.993+0000 7f4926aa7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4920108eb0 0x7f49201a6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:54.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.993+0000 7f4926aa7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4920109870 0x7f49201add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:54.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.993+0000 7f4926aa7640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f492011c170 con 0x7f4920109870 2024-03-18T00:52:54.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.993+0000 7f4926aa7640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f492011bff0 con 0x7f4920072bf0 2024-03-18T00:52:54.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.993+0000 7f4926aa7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f492011c2f0 con 0x7f4920108eb0 2024-03-18T00:52:54.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.993+0000 7f4925aa5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4920072bf0 0x7f49201a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:54.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.993+0000 7f4925aa5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4920072bf0 0x7f49201a6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:50270/0 (socket says 172.21.15.49:50270) 2024-03-18T00:52:54.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.993+0000 7f4925aa5640 1 -- 172.21.15.49:0/3050903921 learned_addr learned my addr 172.21.15.49:0/3050903921 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:54.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.994+0000 7f4925aa5640 1 -- 172.21.15.49:0/3050903921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4920108eb0 msgr2=0x7f49201a6cb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:54.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.994+0000 7f4925aa5640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4920108eb0 0x7f49201a6cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.994+0000 7f4925aa5640 1 -- 172.21.15.49:0/3050903921 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4920109870 msgr2=0x7f49201add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:54.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.994+0000 7f4925aa5640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4920109870 0x7f49201add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:54.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.994+0000 7f4925aa5640 1 -- 172.21.15.49:0/3050903921 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49201ae390 con 0x7f4920072bf0 2024-03-18T00:52:54.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.996+0000 7f4925aa5640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4920072bf0 0x7f49201a6770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f491000d950 tx=0x7f491000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:54.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.998+0000 7f4916ffd640 1 -- 172.21.15.49:0/3050903921 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4910017020 con 0x7f4920072bf0 2024-03-18T00:52:54.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.998+0000 7f4916ffd640 1 -- 172.21.15.49:0/3050903921 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4910009bb0 con 0x7f4920072bf0 2024-03-18T00:52:54.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.998+0000 7f4926aa7640 1 -- 172.21.15.49:0/3050903921 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49201ae680 con 0x7f4920072bf0 2024-03-18T00:52:54.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.999+0000 7f4926aa7640 1 -- 172.21.15.49:0/3050903921 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49201aeb70 con 0x7f4920072bf0 2024-03-18T00:52:54.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:54.999+0000 7f4916ffd640 1 -- 172.21.15.49:0/3050903921 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491000f6b0 con 0x7f4920072bf0 2024-03-18T00:52:55.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.005+0000 7f492a9d1740 1 Processor -- start 2024-03-18T00:52:55.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.005+0000 7f4916ffd640 1 -- 172.21.15.49:0/3050903921 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f491001a020 con 0x7f4920072bf0 2024-03-18T00:52:55.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.006+0000 7f492a9d1740 1 -- start start 2024-03-18T00:52:55.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.007+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c262677a40 0x55c262677e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.007+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 0x55c26267cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.007+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c26267fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.007+0000 7f492a9d1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c2625dc490 con 0x55c262677a40 2024-03-18T00:52:55.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.007+0000 7f492a9d1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c262584c90 con 0x55c26267d5a0 2024-03-18T00:52:55.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.007+0000 7f492a9d1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c2625d8530 con 0x55c262678430 2024-03-18T00:52:55.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.008+0000 7f49252a4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c26267fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.008+0000 7f49252a4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c26267fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:50272/0 (socket says 172.21.15.49:50272) 2024-03-18T00:52:55.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.008+0000 7f49252a4640 1 -- 172.21.15.49:0/1194567183 learned_addr learned my addr 172.21.15.49:0/1194567183 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:55.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.008+0000 7f49262a6640 1 --2- 172.21.15.49:0/1194567183 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c262677a40 0x55c262677e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.009+0000 7f4925aa5640 1 --2- 172.21.15.49:0/1194567183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 0x55c26267cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.009+0000 7f49252a4640 1 -- 172.21.15.49:0/1194567183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 msgr2=0x55c26267cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.009+0000 7f49252a4640 1 --2- 172.21.15.49:0/1194567183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 0x55c26267cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.009+0000 7f49252a4640 1 -- 172.21.15.49:0/1194567183 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c262677a40 msgr2=0x55c262677e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.009+0000 7f49252a4640 1 --2- 172.21.15.49:0/1194567183 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c262677a40 0x55c262677e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.009+0000 7f49252a4640 1 -- 172.21.15.49:0/1194567183 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c26263f500 con 0x55c26267d5a0 2024-03-18T00:52:55.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.011+0000 7f49252a4640 1 --2- 172.21.15.49:0/1194567183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c26267fa30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4918009cd0 tx=0x7f491800e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af66c181c8bc36b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.012+0000 7f490bfff640 1 -- 172.21.15.49:0/1194567183 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4918016070 con 0x55c26267d5a0 2024-03-18T00:52:55.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.012+0000 7f490bfff640 1 -- 172.21.15.49:0/1194567183 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4918005020 con 0x55c26267d5a0 2024-03-18T00:52:55.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.012+0000 7f490bfff640 1 -- 172.21.15.49:0/1194567183 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49180116d0 con 0x55c26267d5a0 2024-03-18T00:52:55.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.012+0000 7f4916ffd640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f490c0a2a00 0x7f490c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.013+0000 7f4916ffd640 1 -- 172.21.15.49:0/3050903921 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f49100ce400 con 0x7f4920072bf0 2024-03-18T00:52:55.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.013+0000 7f492a9d1740 1 -- 172.21.15.49:0/1194567183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 msgr2=0x55c26267fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.013+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1194567183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c26267fa30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4918009cd0 tx=0x7f491800e6c0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.014+0000 7f49262a6640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f490c0a2a00 0x7f490c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.016+0000 7f49262a6640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f490c0a2a00 0x7f490c0a4ec0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f491c02c240 tx=0x7f491c02c880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.016+0000 7f492a9d1740 1 -- 172.21.15.49:0/1194567183 shutdown_connections 2024-03-18T00:52:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.016+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1194567183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c26267fa30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.016+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1194567183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 0x55c26267cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.016+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1194567183 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c262677a40 0x55c262677e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.016+0000 7f492a9d1740 1 -- 172.21.15.49:0/1194567183 >> 172.21.15.49:0/1194567183 conn(0x55c262672fb0 msgr2=0x55c2626733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.020+0000 7f492a9d1740 1 -- 172.21.15.49:0/1194567183 shutdown_connections 2024-03-18T00:52:55.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.022+0000 7f492a9d1740 1 -- 172.21.15.49:0/1194567183 wait complete. 2024-03-18T00:52:55.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.023+0000 7f492a9d1740 1 -- start start 2024-03-18T00:52:55.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.023+0000 7f492a9d1740 1 Processor -- start 2024-03-18T00:52:55.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.024+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262677a40 0x55c262681fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.024+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c262678430 0x55c262682520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.024+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c2626894f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.024+0000 7f492a9d1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c262680550 con 0x55c26267d5a0 2024-03-18T00:52:55.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.024+0000 7f492a9d1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c262598230 con 0x55c262678430 2024-03-18T00:52:55.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.024+0000 7f492a9d1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c2626806d0 con 0x55c262677a40 2024-03-18T00:52:55.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.024+0000 7f492a9d1740 10 client.0 _ll_register_callbacks cb 0x55c2625d4010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:55.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.025+0000 7f49262a6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c2626894f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.025+0000 7f49262a6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c2626894f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40464/0 (socket says 172.21.15.49:40464) 2024-03-18T00:52:55.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.025+0000 7f49262a6640 1 -- 172.21.15.49:0/1466983823 learned_addr learned my addr 172.21.15.49:0/1466983823 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:55.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.025+0000 7f4925aa5640 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262677a40 0x55c262681fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.026+0000 7f49262a6640 1 -- 172.21.15.49:0/1466983823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262677a40 msgr2=0x55c262681fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.026+0000 7f49262a6640 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262677a40 0x55c262681fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.026+0000 7f49262a6640 1 -- 172.21.15.49:0/1466983823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c262678430 msgr2=0x55c262682520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.026+0000 7f49262a6640 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c262678430 0x55c262682520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.026+0000 7f49262a6640 1 -- 172.21.15.49:0/1466983823 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2625a7100 con 0x55c26267d5a0 2024-03-18T00:52:55.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.026+0000 7f49262a6640 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c2626894f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f491c035000 tx=0x7f491c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6df41d85b67886 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.027+0000 7f49097fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:55.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.027+0000 7f49097fa640 1 -- 172.21.15.49:0/1466983823 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c004ad0 con 0x55c26267d5a0 2024-03-18T00:52:55.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.028+0000 7f492a9d1740 1 -- 172.21.15.49:0/1466983823 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4918005020 con 0x55c26267d5a0 2024-03-18T00:52:55.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.029+0000 7f49097fa640 1 -- 172.21.15.49:0/1466983823 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f491c005680 con 0x55c26267d5a0 2024-03-18T00:52:55.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.030+0000 7f49097fa640 1 -- 172.21.15.49:0/1466983823 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f491c004c70 con 0x55c26267d5a0 2024-03-18T00:52:55.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.030+0000 7f492a9d1740 10 client.16071 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:55.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.030+0000 7f492a9d1740 1 -- 172.21.15.49:0/1466983823 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c2622f03f0 con 0x55c26267d5a0 2024-03-18T00:52:55.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.032+0000 7f49097fa640 1 -- 172.21.15.49:0/1466983823 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4918005020 con 0x55c26267d5a0 2024-03-18T00:52:55.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.033+0000 7f49097fa640 1 -- 172.21.15.49:0/1466983823 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f491c04a600 con 0x55c26267d5a0 2024-03-18T00:52:55.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.034+0000 7f492a9d1740 10 client.16071 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:55.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.034+0000 7f492a9d1740 10 client.16071 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:52:55.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.034+0000 7f492a9d1740 20 client.16071 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:55.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.034+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c26268e2e0 0x55c2626907a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.034+0000 7f492a9d1740 4 client.16071 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:55.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.034+0000 7f492a9d1740 1 -- 172.21.15.49:0/1466983823 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2625e99d0 con 0x55c26268e2e0 2024-03-18T00:52:55.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.034+0000 7f4925aa5640 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c26268e2e0 0x55c2626907a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.038+0000 7f4925aa5640 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c26268e2e0 0x55c2626907a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=53ed0b8d8f701c16 server_cookie=5cc1e1ee06a97e8a in_seq=0 out_seq=0 2024-03-18T00:52:55.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.038+0000 7f49097fa640 10 client.16071 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:52:55.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.042+0000 7f49097fa640 1 -- 172.21.15.49:0/1466983823 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c2625e99d0 con 0x55c26268e2e0 2024-03-18T00:52:55.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.042+0000 7f49097fa640 10 client.16071 handle_command_reply: tid=0 2024-03-18T00:52:55.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.042+0000 7f492a9d1740 1 client.16071 shutdown 2024-03-18T00:52:55.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.042+0000 7f492a9d1740 1 -- 172.21.15.49:0/1466983823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 msgr2=0x55c2626894f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.042+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c2626894f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f491c035000 tx=0x7f491c009260 comp rx=0 tx=0).stop 2024-03-18T00:52:55.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.043+0000 7f492a9d1740 1 -- 172.21.15.49:0/1466983823 shutdown_connections 2024-03-18T00:52:55.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.043+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c26268e2e0 0x55c2626907a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.043+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c262678430 0x55c262682520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.043+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c2626894f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.043+0000 7f492a9d1740 1 --2- 172.21.15.49:0/1466983823 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262677a40 0x55c262681fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.043+0000 7f492a9d1740 1 -- 172.21.15.49:0/1466983823 >> 172.21.15.49:0/1466983823 conn(0x55c262672fb0 msgr2=0x55c262672140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.043+0000 7f49097fa640 0 client.16071 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:52:55.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.048+0000 7f492a9d1740 1 -- 172.21.15.49:0/1466983823 shutdown_connections 2024-03-18T00:52:55.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.048+0000 7f492a9d1740 1 -- 172.21.15.49:0/1466983823 wait complete. 2024-03-18T00:52:55.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.048+0000 7f492a9d1740 20 client.16071 trim_cache size 0 max 16384 2024-03-18T00:52:55.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.089+0000 7f492a9d1740 1 Processor -- start 2024-03-18T00:52:55.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.090+0000 7f492a9d1740 1 -- start start 2024-03-18T00:52:55.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.091+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c262677a40 0x55c2626718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.092+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 0x55c2626894b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.092+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c262671df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.092+0000 7f492a9d1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f490400e470 con 0x55c26267d5a0 2024-03-18T00:52:55.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.092+0000 7f49262a6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c262671df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.092+0000 7f49262a6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c262671df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40472/0 (socket says 172.21.15.49:40472) 2024-03-18T00:52:55.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.092+0000 7f49262a6640 1 -- 172.21.15.49:0/2383171963 learned_addr learned my addr 172.21.15.49:0/2383171963 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:55.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.092+0000 7f492a9d1740 1 -- 172.21.15.49:0/2383171963 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f490400ffe0 con 0x55c262677a40 2024-03-18T00:52:55.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.092+0000 7f492a9d1740 1 -- 172.21.15.49:0/2383171963 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f490400eb40 con 0x55c262678430 2024-03-18T00:52:55.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.093+0000 7f4925aa5640 1 --2- 172.21.15.49:0/2383171963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c262677a40 0x55c2626718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.093+0000 7f49252a4640 1 --2- 172.21.15.49:0/2383171963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 0x55c2626894b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.094+0000 7f49262a6640 1 -- 172.21.15.49:0/2383171963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 msgr2=0x55c2626894b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.094+0000 7f49262a6640 1 --2- 172.21.15.49:0/2383171963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 0x55c2626894b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.094+0000 7f49262a6640 1 -- 172.21.15.49:0/2383171963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c262677a40 msgr2=0x55c2626718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.094+0000 7f49262a6640 1 --2- 172.21.15.49:0/2383171963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c262677a40 0x55c2626718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.094+0000 7f49262a6640 1 -- 172.21.15.49:0/2383171963 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2623ba7d0 con 0x55c26267d5a0 2024-03-18T00:52:55.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.095+0000 7f49262a6640 1 --2- 172.21.15.49:0/2383171963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c262671df0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f491c04aac0 tx=0x7f491c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0e737f214916cb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.107+0000 7f490b7fe640 1 -- 172.21.15.49:0/2383171963 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c005680 con 0x55c26267d5a0 2024-03-18T00:52:55.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.107+0000 7f490b7fe640 1 -- 172.21.15.49:0/2383171963 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f491c062070 con 0x55c26267d5a0 2024-03-18T00:52:55.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.108+0000 7f492a9d1740 1 -- 172.21.15.49:0/2383171963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 msgr2=0x55c262671df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.108+0000 7f492a9d1740 1 --2- 172.21.15.49:0/2383171963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c262671df0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f491c04aac0 tx=0x7f491c005fd0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.116+0000 7f492a9d1740 1 -- 172.21.15.49:0/2383171963 shutdown_connections 2024-03-18T00:52:55.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.116+0000 7f492a9d1740 1 --2- 172.21.15.49:0/2383171963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c26267d5a0 0x55c262671df0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.116+0000 7f492a9d1740 1 --2- 172.21.15.49:0/2383171963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262678430 0x55c2626894b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.116+0000 7f492a9d1740 1 --2- 172.21.15.49:0/2383171963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c262677a40 0x55c2626718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.116+0000 7f492a9d1740 1 -- 172.21.15.49:0/2383171963 >> 172.21.15.49:0/2383171963 conn(0x55c262672fb0 msgr2=0x55c262679930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.124+0000 7f492a9d1740 1 -- 172.21.15.49:0/2383171963 shutdown_connections 2024-03-18T00:52:55.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.125+0000 7f492a9d1740 1 -- 172.21.15.49:0/2383171963 wait complete. 2024-03-18T00:52:55.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.125+0000 7f492a9d1740 1 -- start start 2024-03-18T00:52:55.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.125+0000 7f492a9d1740 1 Processor -- start 2024-03-18T00:52:55.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.126+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262677a40 0x55c2626894d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.126+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c262678430 0x55c262685fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.126+0000 7f492a9d1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c262686540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.126+0000 7f492a9d1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f490400f8e0 con 0x55c262678430 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.126+0000 7f492a9d1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f490400f210 con 0x55c26267d5a0 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.126+0000 7f492a9d1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f490400c500 con 0x55c262677a40 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.126+0000 7f492a9d1740 10 client.0 _ll_register_callbacks cb 0x7f491c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.128+0000 7f49262a6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c262686540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.128+0000 7f49262a6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c262686540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:50286/0 (socket says 172.21.15.49:50286) 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.128+0000 7f49262a6640 1 -- 172.21.15.49:0/3467552526 learned_addr learned my addr 172.21.15.49:0/3467552526 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.129+0000 7f49262a6640 1 -- 172.21.15.49:0/3467552526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262677a40 msgr2=0x55c2626894d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.129+0000 7f49262a6640 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262677a40 0x55c2626894d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.129+0000 7f49262a6640 1 -- 172.21.15.49:0/3467552526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c262678430 msgr2=0x55c262685fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.129+0000 7f49262a6640 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c262678430 0x55c262685fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.129+0000 7f49262a6640 1 -- 172.21.15.49:0/3467552526 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2626f26d0 con 0x55c26267d5a0 2024-03-18T00:52:55.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.130+0000 7f49262a6640 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c262686540 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f491c04aa40 tx=0x7f491c003fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=909ca977780382ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.130+0000 7f49097fa640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:52:55.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.136+0000 7f49097fa640 1 -- 172.21.15.49:0/3467552526 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c04a660 con 0x55c26267d5a0 2024-03-18T00:52:55.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.136+0000 7f49097fa640 1 -- 172.21.15.49:0/3467552526 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f491c03e430 con 0x55c26267d5a0 2024-03-18T00:52:55.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.136+0000 7f492a9d1740 1 -- 172.21.15.49:0/3467552526 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f491c062070 con 0x55c26267d5a0 2024-03-18T00:52:55.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.139+0000 7f49097fa640 1 -- 172.21.15.49:0/3467552526 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f491c04a800 con 0x55c26267d5a0 2024-03-18T00:52:55.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.140+0000 7f492a9d1740 10 client.25724 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:55.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.140+0000 7f492a9d1740 1 -- 172.21.15.49:0/3467552526 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c2626f39d0 con 0x55c26267d5a0 2024-03-18T00:52:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.146+0000 7f49097fa640 1 -- 172.21.15.49:0/3467552526 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f491c062070 con 0x55c26267d5a0 2024-03-18T00:52:55.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.146+0000 7f49097fa640 1 -- 172.21.15.49:0/3467552526 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f491c0778e0 con 0x55c26267d5a0 2024-03-18T00:52:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.146+0000 7f492a9d1740 10 client.25724 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.146+0000 7f492a9d1740 10 client.25724 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:52:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.146+0000 7f492a9d1740 20 client.25724 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.146+0000 7f492a9d1740 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c2626f4300 0x55c2626f6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.146+0000 7f492a9d1740 4 client.25724 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:55.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.146+0000 7f492a9d1740 1 -- 172.21.15.49:0/3467552526 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c2625e3d20 con 0x55c2626f4300 2024-03-18T00:52:55.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.148+0000 7f4925aa5640 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c2626f4300 0x55c2626f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.159+0000 7f4925aa5640 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c2626f4300 0x55c2626f6720 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bab1ed8583fcc5c8 server_cookie=eccc484ffc119c28 in_seq=0 out_seq=0 2024-03-18T00:52:55.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.159+0000 7f49097fa640 10 client.25724 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:52:55.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.160+0000 7f49097fa640 1 -- 172.21.15.49:0/3467552526 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c2625e3d20 con 0x55c2626f4300 2024-03-18T00:52:55.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.160+0000 7f49097fa640 10 client.25724 handle_command_reply: tid=0 2024-03-18T00:52:55.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.160+0000 7f492a9d1740 1 client.25724 shutdown 2024-03-18T00:52:55.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.161+0000 7f492a9d1740 1 -- 172.21.15.49:0/3467552526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 msgr2=0x55c262686540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.161+0000 7f492a9d1740 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c262686540 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f491c04aa40 tx=0x7f491c003fd0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.163+0000 7f492a9d1740 1 -- 172.21.15.49:0/3467552526 shutdown_connections 2024-03-18T00:52:55.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.163+0000 7f492a9d1740 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c2626f4300 0x55c2626f6720 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.163+0000 7f492a9d1740 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c26267d5a0 0x55c262686540 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.163+0000 7f492a9d1740 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c262678430 0x55c262685fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.163+0000 7f492a9d1740 1 --2- 172.21.15.49:0/3467552526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c262677a40 0x55c2626894d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.163+0000 7f492a9d1740 1 -- 172.21.15.49:0/3467552526 >> 172.21.15.49:0/3467552526 conn(0x55c262672fb0 msgr2=0x55c2626718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.164+0000 7f49097fa640 0 client.25724 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:52:55.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.169+0000 7f492a9d1740 1 -- 172.21.15.49:0/3467552526 shutdown_connections 2024-03-18T00:52:55.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.169+0000 7f492a9d1740 1 -- 172.21.15.49:0/3467552526 wait complete. 2024-03-18T00:52:55.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.177+0000 7f492a9d1740 20 client.25724 trim_cache size 0 max 16384 2024-03-18T00:52:55.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:54 smithi107 ceph-mon[36270]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 996 MiB/s wr, 1.49k op/s 2024-03-18T00:52:55.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:54 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1503621198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:52:55.183 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:55.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.184+0000 7f4926aa7640 1 -- 172.21.15.49:0/3050903921 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f490c0a2a00 msgr2=0x7f490c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.184+0000 7f4926aa7640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f490c0a2a00 0x7f490c0a4ec0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f491c02c240 tx=0x7f491c02c880 comp rx=0 tx=0).stop 2024-03-18T00:52:55.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.184+0000 7f4926aa7640 1 -- 172.21.15.49:0/3050903921 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4920072bf0 msgr2=0x7f49201a6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.184+0000 7f4926aa7640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4920072bf0 0x7f49201a6770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f491000d950 tx=0x7f491000de10 comp rx=0 tx=0).stop 2024-03-18T00:52:55.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.185+0000 7f4926aa7640 1 -- 172.21.15.49:0/3050903921 shutdown_connections 2024-03-18T00:52:55.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.185+0000 7f4926aa7640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f490c0a2a00 0x7f490c0a4ec0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.185+0000 7f4926aa7640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4920109870 0x7f49201add20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.185+0000 7f4926aa7640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4920108eb0 0x7f49201a6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.185+0000 7f4926aa7640 1 --2- 172.21.15.49:0/3050903921 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4920072bf0 0x7f49201a6770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.185+0000 7f4926aa7640 1 -- 172.21.15.49:0/3050903921 >> 172.21.15.49:0/3050903921 conn(0x7f492006e6c0 msgr2=0x7f49200702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.190+0000 7f4926aa7640 1 -- 172.21.15.49:0/3050903921 shutdown_connections 2024-03-18T00:52:55.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.190+0000 7f4926aa7640 1 -- 172.21.15.49:0/3050903921 wait complete. 2024-03-18T00:52:55.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:54 smithi106 ceph-mon[35899]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 996 MiB/s wr, 1.49k op/s 2024-03-18T00:52:55.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:54 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1503621198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:52:55.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:52:55.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.503+0000 7fcbe233b640 1 -- 172.21.15.49:0/381849500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc101970 msgr2=0x7fcbdc108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.503+0000 7fcbe233b640 1 --2- 172.21.15.49:0/381849500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc101970 0x7fcbdc108290 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009ca0 tx=0x7fcbd402c140 comp rx=0 tx=0).stop 2024-03-18T00:52:55.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.503+0000 7fcbe233b640 1 -- 172.21.15.49:0/381849500 shutdown_connections 2024-03-18T00:52:55.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.503+0000 7fcbe233b640 1 --2- 172.21.15.49:0/381849500 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc101970 0x7fcbdc108290 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.503+0000 7fcbe233b640 1 --2- 172.21.15.49:0/381849500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbdc100fb0 0x7fcbdc101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.503+0000 7fcbe233b640 1 --2- 172.21.15.49:0/381849500 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbdc0ffe60 0x7fcbdc100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.503+0000 7fcbe233b640 1 -- 172.21.15.49:0/381849500 >> 172.21.15.49:0/381849500 conn(0x7fcbdc0fb3d0 msgr2=0x7fcbdc0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.504+0000 7fcbe233b640 1 -- 172.21.15.49:0/381849500 shutdown_connections 2024-03-18T00:52:55.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.504+0000 7fcbe233b640 1 -- 172.21.15.49:0/381849500 wait complete. 2024-03-18T00:52:55.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.505+0000 7fcbe233b640 1 Processor -- start 2024-03-18T00:52:55.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.506+0000 7fcbe233b640 1 -- start start 2024-03-18T00:52:55.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.506+0000 7fcbe233b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc0ffe60 0x7fcbdc19e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.506+0000 7fcbe233b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbdc100fb0 0x7fcbdc19e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.506+0000 7fcbe233b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbdc101970 0x7fcbdc1a57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.506+0000 7fcbe233b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc10d5f0 con 0x7fcbdc0ffe60 2024-03-18T00:52:55.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.506+0000 7fcbe233b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc10d470 con 0x7fcbdc101970 2024-03-18T00:52:55.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.506+0000 7fcbe233b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc10d770 con 0x7fcbdc100fb0 2024-03-18T00:52:55.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.507+0000 7fcbdbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc0ffe60 0x7fcbdc19e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.507+0000 7fcbdbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc0ffe60 0x7fcbdc19e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40488/0 (socket says 172.21.15.49:40488) 2024-03-18T00:52:55.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.507+0000 7fcbdbfff640 1 -- 172.21.15.49:0/1978818395 learned_addr learned my addr 172.21.15.49:0/1978818395 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:55.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.507+0000 7fcbdbfff640 1 -- 172.21.15.49:0/1978818395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbdc100fb0 msgr2=0x7fcbdc19e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.516 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.507+0000 7fcbdb7fe640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbdc100fb0 0x7fcbdc19e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.508+0000 7fcbe08d8640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbdc101970 0x7fcbdc1a57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.508+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbdc100fb0 0x7fcbdc19e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.508+0000 7fcbdbfff640 1 -- 172.21.15.49:0/1978818395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbdc101970 msgr2=0x7fcbdc1a57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.508+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbdc101970 0x7fcbdc1a57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.508+0000 7fcbdbfff640 1 -- 172.21.15.49:0/1978818395 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc06b330 con 0x7fcbdc0ffe60 2024-03-18T00:52:55.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.508+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc0ffe60 0x7fcbdc19e220 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00b450 tx=0x7fcbcc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.510+0000 7fcbd97fa640 1 -- 172.21.15.49:0/1978818395 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc015020 con 0x7fcbdc0ffe60 2024-03-18T00:52:55.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.510+0000 7fcbe233b640 1 -- 172.21.15.49:0/1978818395 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbdc06b5c0 con 0x7fcbdc0ffe60 2024-03-18T00:52:55.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.510+0000 7fcbe233b640 1 -- 172.21.15.49:0/1978818395 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbdc06bbe0 con 0x7fcbdc0ffe60 2024-03-18T00:52:55.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.510+0000 7fcbd97fa640 1 -- 172.21.15.49:0/1978818395 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbcc011d70 con 0x7fcbdc0ffe60 2024-03-18T00:52:55.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.510+0000 7fcbd97fa640 1 -- 172.21.15.49:0/1978818395 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc00f340 con 0x7fcbdc0ffe60 2024-03-18T00:52:55.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.515+0000 7fcbd97fa640 1 -- 172.21.15.49:0/1978818395 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcbcc018020 con 0x7fcbdc0ffe60 2024-03-18T00:52:55.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.516+0000 7fcbe5008740 1 Processor -- start 2024-03-18T00:52:55.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.516+0000 7fcbe5008740 1 -- start start 2024-03-18T00:52:55.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.516+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd69d850 0x555abd69dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.516+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69e240 0x555abd6a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.516+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 0x555abd6a5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.516+0000 7fcbe5008740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555abd602490 con 0x555abd69e240 2024-03-18T00:52:55.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.516+0000 7fcbe5008740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555abd5aac90 con 0x555abd6a33b0 2024-03-18T00:52:55.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.517+0000 7fcbe5008740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555abd5fe530 con 0x555abd69d850 2024-03-18T00:52:55.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.517+0000 7fcbd97fa640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbc80c4950 0x7fcbc80c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.517+0000 7fcbdbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69e240 0x555abd6a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.517+0000 7fcbdbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69e240 0x555abd6a2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40500/0 (socket says 172.21.15.49:40500) 2024-03-18T00:52:55.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.517+0000 7fcbdbfff640 1 -- 172.21.15.49:0/3127480754 learned_addr learned my addr 172.21.15.49:0/3127480754 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:55.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.517+0000 7fcbe08d8640 1 --2- 172.21.15.49:0/3127480754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd69d850 0x555abd69dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.517+0000 7fcbd97fa640 1 -- 172.21.15.49:0/1978818395 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcbcc0ce4e0 con 0x7fcbdc0ffe60 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.517+0000 7fcbe08d8640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbc80c4950 0x7fcbc80c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.518+0000 7fcbdbfff640 1 -- 172.21.15.49:0/3127480754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd69d850 msgr2=0x555abd69dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.518+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/3127480754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd69d850 0x555abd69dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.518+0000 7fcbdbfff640 1 -- 172.21.15.49:0/3127480754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 msgr2=0x555abd6a5840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.518+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/3127480754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 0x555abd6a5840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.518+0000 7fcbdbfff640 1 -- 172.21.15.49:0/3127480754 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555abd665500 con 0x555abd69e240 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.518+0000 7fcbe08d8640 1 --2- 172.21.15.49:0/3127480754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd69d850 0x555abd69dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.518+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/3127480754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69e240 0x555abd6a2d40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc09ab00 tx=0x7fcbcc09aa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bda1429055affef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.518+0000 7fcbe08d8640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbc80c4950 0x7fcbc80c6e10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x555abd69eaa0 tx=0x7fcbd40095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.519+0000 7fcbd27fc640 1 -- 172.21.15.49:0/3127480754 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc09ae00 con 0x555abd69e240 2024-03-18T00:52:55.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.519+0000 7fcbd27fc640 1 -- 172.21.15.49:0/3127480754 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbcc09e070 con 0x555abd69e240 2024-03-18T00:52:55.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.520+0000 7fcbe5008740 1 -- 172.21.15.49:0/3127480754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69e240 msgr2=0x555abd6a2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.520+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3127480754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69e240 0x555abd6a2d40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc09ab00 tx=0x7fcbcc09aa00 comp rx=0 tx=0).stop 2024-03-18T00:52:55.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.521+0000 7fcbe5008740 1 -- 172.21.15.49:0/3127480754 shutdown_connections 2024-03-18T00:52:55.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.521+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3127480754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 0x555abd6a5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.521+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3127480754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69e240 0x555abd6a2d40 secure :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc09ab00 tx=0x7fcbcc09aa00 comp rx=0 tx=0).stop 2024-03-18T00:52:55.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.521+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3127480754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd69d850 0x555abd69dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.521+0000 7fcbe5008740 1 -- 172.21.15.49:0/3127480754 >> 172.21.15.49:0/3127480754 conn(0x555abd698dc0 msgr2=0x555abd6991c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.521+0000 7fcbe5008740 1 -- 172.21.15.49:0/3127480754 shutdown_connections 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.521+0000 7fcbe5008740 1 -- 172.21.15.49:0/3127480754 wait complete. 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.522+0000 7fcbe5008740 1 -- start start 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.522+0000 7fcbe5008740 1 Processor -- start 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.523+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd6a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.523+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 0x555abd6a8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.523+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 0x555abd6af890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.523+0000 7fcbe5008740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555abd6a63b0 con 0x555abd69d850 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.523+0000 7fcbe5008740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555abd5be230 con 0x555abd6a33b0 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.523+0000 7fcbe5008740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555abd6a6530 con 0x555abd6ad4a0 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.523+0000 7fcbe5008740 10 client.0 _ll_register_callbacks cb 0x555abd5fa010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.524+0000 7fcbe08d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 0x555abd6af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.524+0000 7fcbe08d8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 0x555abd6af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34646/0 (socket says 172.21.15.49:34646) 2024-03-18T00:52:55.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.524+0000 7fcbe08d8640 1 -- 172.21.15.49:0/3640522305 learned_addr learned my addr 172.21.15.49:0/3640522305 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:55.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.524+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd6a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.525+0000 7fcbe08d8640 1 -- 172.21.15.49:0/3640522305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 msgr2=0x555abd6a8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.525+0000 7fcbe08d8640 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 0x555abd6a8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.525+0000 7fcbe08d8640 1 -- 172.21.15.49:0/3640522305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 msgr2=0x555abd6a7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.525+0000 7fcbe08d8640 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd6a7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.525+0000 7fcbe08d8640 1 -- 172.21.15.49:0/3640522305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555abd5cd100 con 0x555abd6ad4a0 2024-03-18T00:52:55.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.525+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd6a7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.525+0000 7fcbe08d8640 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 0x555abd6af890 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009ca0 tx=0x7fcbd402c140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d632dddb723ee027 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.525+0000 7fcbbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.525+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/3640522305 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4039070 con 0x555abd6ad4a0 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.526+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/3640522305 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcbd4004450 con 0x555abd6ad4a0 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.527+0000 7fcbe5008740 1 -- 172.21.15.49:0/3640522305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbcc09e070 con 0x555abd6ad4a0 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.527+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/3640522305 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcbd4003e40 con 0x555abd6ad4a0 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.527+0000 7fcbe5008740 10 client.25804 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.527+0000 7fcbe5008740 1 -- 172.21.15.49:0/3640522305 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555abd3163f0 con 0x555abd6ad4a0 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.528+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/3640522305 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbcc09e070 con 0x555abd6ad4a0 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.528+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/3640522305 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fcbd4042420 con 0x555abd6ad4a0 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.528+0000 7fcbe5008740 10 client.25804 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:55.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.528+0000 7fcbe5008740 10 client.25804 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:52:55.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.528+0000 7fcbe5008740 20 client.25804 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:55.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.528+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x555abd6b4700 0x555abd6b6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.528+0000 7fcbe5008740 4 client.25804 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:55.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.528+0000 7fcbe5008740 1 -- 172.21.15.49:0/3640522305 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555abd60f9d0 con 0x555abd6b4700 2024-03-18T00:52:55.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.530+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x555abd6b4700 0x555abd6b6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.532+0000 7fcbbb7fe640 10 client.25804 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:52:55.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.532+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x555abd6b4700 0x555abd6b6ba0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6dddc46d3da04772 server_cookie=ff9c491c5c9a9011 in_seq=0 out_seq=0 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.541+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/3640522305 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555abd60f9d0 con 0x555abd6b4700 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.541+0000 7fcbbb7fe640 10 client.25804 handle_command_reply: tid=0 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.541+0000 7fcbe5008740 1 client.25804 shutdown 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.542+0000 7fcbe5008740 1 -- 172.21.15.49:0/3640522305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 msgr2=0x555abd6af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.542+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 0x555abd6af890 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4009ca0 tx=0x7fcbd402c140 comp rx=0 tx=0).stop 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.543+0000 7fcbe5008740 1 -- 172.21.15.49:0/3640522305 shutdown_connections 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.543+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x555abd6b4700 0x555abd6b6ba0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.543+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 0x555abd6a8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.543+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 0x555abd6af890 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.543+0000 7fcbe5008740 1 --2- 172.21.15.49:0/3640522305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd6a7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.543+0000 7fcbe5008740 1 -- 172.21.15.49:0/3640522305 >> 172.21.15.49:0/3640522305 conn(0x555abd698dc0 msgr2=0x555abd697fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.543+0000 7fcbbb7fe640 0 client.25804 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:52:55.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.559+0000 7fcbe5008740 1 -- 172.21.15.49:0/3640522305 shutdown_connections 2024-03-18T00:52:55.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.559+0000 7fcbe5008740 1 -- 172.21.15.49:0/3640522305 wait complete. 2024-03-18T00:52:55.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.560+0000 7fcbe5008740 20 client.25804 trim_cache size 0 max 16384 2024-03-18T00:52:55.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.637+0000 7fcbe5008740 1 Processor -- start 2024-03-18T00:52:55.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.637+0000 7fcbe5008740 1 -- start start 2024-03-18T00:52:55.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.637+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd694130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.638+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 0x555abd6a69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.638+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 0x555abd6991a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.638+0000 7fcbe5008740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc00e550 con 0x555abd69d850 2024-03-18T00:52:55.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.638+0000 7fcbe5008740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc0100c0 con 0x555abd6a33b0 2024-03-18T00:52:55.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.638+0000 7fcbe5008740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc00ec20 con 0x555abd6ad4a0 2024-03-18T00:52:55.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.639+0000 7fcbdbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd694130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.640+0000 7fcbdbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd694130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40508/0 (socket says 172.21.15.49:40508) 2024-03-18T00:52:55.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.640+0000 7fcbdbfff640 1 -- 172.21.15.49:0/1532210690 learned_addr learned my addr 172.21.15.49:0/1532210690 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:55.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.641+0000 7fcbdbfff640 1 -- 172.21.15.49:0/1532210690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 msgr2=0x555abd6991a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:55.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.641+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/1532210690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 0x555abd6991a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.641+0000 7fcbdbfff640 1 -- 172.21.15.49:0/1532210690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 msgr2=0x555abd6a69b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:55.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.641+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/1532210690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 0x555abd6a69b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.641+0000 7fcbdbfff640 1 -- 172.21.15.49:0/1532210690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555abd3e07d0 con 0x555abd69d850 2024-03-18T00:52:55.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.642+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/1532210690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd694130 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc09ad90 tx=0x7fcbcc00b3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1333cf4290094612 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.643+0000 7fcbd1ffb640 1 -- 172.21.15.49:0/1532210690 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc09ae00 con 0x555abd69d850 2024-03-18T00:52:55.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.643+0000 7fcbd1ffb640 1 -- 172.21.15.49:0/1532210690 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbcc09e070 con 0x555abd69d850 2024-03-18T00:52:55.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.645+0000 7fcbe5008740 1 -- 172.21.15.49:0/1532210690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 msgr2=0x555abd694130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.645+0000 7fcbe5008740 1 --2- 172.21.15.49:0/1532210690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd694130 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc09ad90 tx=0x7fcbcc00b3d0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.650+0000 7fcbe5008740 1 -- 172.21.15.49:0/1532210690 shutdown_connections 2024-03-18T00:52:55.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.650+0000 7fcbe5008740 1 --2- 172.21.15.49:0/1532210690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd6a33b0 0x555abd6a69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.650+0000 7fcbe5008740 1 --2- 172.21.15.49:0/1532210690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6ad4a0 0x555abd6991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.650+0000 7fcbe5008740 1 --2- 172.21.15.49:0/1532210690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd69d850 0x555abd694130 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.650+0000 7fcbe5008740 1 -- 172.21.15.49:0/1532210690 >> 172.21.15.49:0/1532210690 conn(0x555abd698dc0 msgr2=0x555abd69ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.651+0000 7fcbe5008740 1 -- 172.21.15.49:0/1532210690 shutdown_connections 2024-03-18T00:52:55.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.652+0000 7fcbe5008740 1 -- 172.21.15.49:0/1532210690 wait complete. 2024-03-18T00:52:55.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.653+0000 7fcbe5008740 1 -- start start 2024-03-18T00:52:55.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.654+0000 7fcbe5008740 1 Processor -- start 2024-03-18T00:52:55.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.654+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd69d850 0x555abd6c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.655+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6a33b0 0x555abd6c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.656+0000 7fcbe5008740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd6ad4a0 0x555abd6c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.657+0000 7fcbe5008740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc00f9c0 con 0x555abd6ad4a0 2024-03-18T00:52:55.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.657+0000 7fcbe5008740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc00f2f0 con 0x555abd69d850 2024-03-18T00:52:55.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.657+0000 7fcbe5008740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcbbc00c530 con 0x555abd6a33b0 2024-03-18T00:52:55.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.657+0000 7fcbe5008740 10 client.0 _ll_register_callbacks cb 0x7fcbd4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:55.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.658+0000 7fcbdbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd69d850 0x555abd6c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.658+0000 7fcbdbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd69d850 0x555abd6c7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:50338/0 (socket says 172.21.15.49:50338) 2024-03-18T00:52:55.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.658+0000 7fcbdbfff640 1 -- 172.21.15.49:0/714078809 learned_addr learned my addr 172.21.15.49:0/714078809 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:55.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.659+0000 7fcbdb7fe640 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6a33b0 0x555abd6c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.659+0000 7fcbdbfff640 1 -- 172.21.15.49:0/714078809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6a33b0 msgr2=0x555abd6c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.659+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6a33b0 0x555abd6c46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.659+0000 7fcbdbfff640 1 -- 172.21.15.49:0/714078809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd6ad4a0 msgr2=0x555abd6c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:55.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.659+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd6ad4a0 0x555abd6c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.659+0000 7fcbdbfff640 1 -- 172.21.15.49:0/714078809 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555abd718490 con 0x555abd69d850 2024-03-18T00:52:55.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.660+0000 7fcbdbfff640 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd69d850 0x555abd6c7bc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0977d0 tx=0x7fcbcc0108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12bf56edd4cb3fae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:55.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.661+0000 7fcbbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:52:55.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.661+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/714078809 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0a3310 con 0x555abd69d850 2024-03-18T00:52:55.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.662+0000 7fcbe5008740 1 -- 172.21.15.49:0/714078809 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbcc09e070 con 0x555abd69d850 2024-03-18T00:52:55.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.662+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/714078809 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcbcc09a250 con 0x555abd69d850 2024-03-18T00:52:55.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.664+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/714078809 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcbcc098ea0 con 0x555abd69d850 2024-03-18T00:52:55.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.664+0000 7fcbe5008740 10 client.25727 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:55.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.664+0000 7fcbe5008740 1 -- 172.21.15.49:0/714078809 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555abd719730 con 0x555abd69d850 2024-03-18T00:52:55.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.670+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/714078809 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbcc09e070 con 0x555abd69d850 2024-03-18T00:52:55.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.670+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/714078809 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fcbcc099cc0 con 0x555abd69d850 2024-03-18T00:52:55.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.670+0000 7fcbe5008740 10 client.25727 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:55.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.670+0000 7fcbe5008740 10 client.25727 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:52:55.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.670+0000 7fcbe5008740 20 client.25727 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:55.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.670+0000 7fcbe5008740 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x555abd719e70 0x555abd71c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:55.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.670+0000 7fcbe5008740 4 client.25727 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:55.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.670+0000 7fcbe5008740 1 -- 172.21.15.49:0/714078809 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555abd609d20 con 0x555abd719e70 2024-03-18T00:52:55.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.676+0000 7fcbdb7fe640 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x555abd719e70 0x555abd71c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:55.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.685+0000 7fcbdb7fe640 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x555abd719e70 0x555abd71c290 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3a56a4d4b36563b6 server_cookie=f50ed7416e748af0 in_seq=0 out_seq=0 2024-03-18T00:52:55.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.686+0000 7fcbbb7fe640 10 client.25727 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:52:55.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.693+0000 7fcbbb7fe640 1 -- 172.21.15.49:0/714078809 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555abd609d20 con 0x555abd719e70 2024-03-18T00:52:55.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.693+0000 7fcbbb7fe640 10 client.25727 handle_command_reply: tid=0 2024-03-18T00:52:55.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.693+0000 7fcbe5008740 1 client.25727 shutdown 2024-03-18T00:52:55.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.698+0000 7fcbe5008740 1 -- 172.21.15.49:0/714078809 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd69d850 msgr2=0x555abd6c7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.698+0000 7fcbe5008740 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd69d850 0x555abd6c7bc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0977d0 tx=0x7fcbcc0108e0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.702+0000 7fcbe5008740 1 -- 172.21.15.49:0/714078809 shutdown_connections 2024-03-18T00:52:55.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.702+0000 7fcbe5008740 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x555abd719e70 0x555abd71c290 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.702+0000 7fcbe5008740 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555abd6ad4a0 0x555abd6c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.702+0000 7fcbe5008740 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555abd6a33b0 0x555abd6c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.702+0000 7fcbe5008740 1 --2- 172.21.15.49:0/714078809 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555abd69d850 0x555abd6c7bc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.702+0000 7fcbe5008740 1 -- 172.21.15.49:0/714078809 >> 172.21.15.49:0/714078809 conn(0x555abd698dc0 msgr2=0x555abd6940a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.702+0000 7fcbbb7fe640 0 client.25727 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:52:55.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.706+0000 7fcbe5008740 1 -- 172.21.15.49:0/714078809 shutdown_connections 2024-03-18T00:52:55.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.706+0000 7fcbe5008740 1 -- 172.21.15.49:0/714078809 wait complete. 2024-03-18T00:52:55.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.708+0000 7fcbe5008740 20 client.25727 trim_cache size 0 max 16384 2024-03-18T00:52:55.718 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:55.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.724+0000 7fcbe233b640 1 -- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbc80c4950 msgr2=0x7fcbc80c6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.724+0000 7fcbe233b640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbc80c4950 0x7fcbc80c6e10 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x555abd69eaa0 tx=0x7fcbd40095c0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.724+0000 7fcbe233b640 1 -- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc0ffe60 msgr2=0x7fcbdc19e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:55.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.724+0000 7fcbe233b640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc0ffe60 0x7fcbdc19e220 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00b450 tx=0x7fcbcc00b910 comp rx=0 tx=0).stop 2024-03-18T00:52:55.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.729+0000 7fcbe233b640 1 -- 172.21.15.49:0/1978818395 shutdown_connections 2024-03-18T00:52:55.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.729+0000 7fcbe233b640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcbc80c4950 0x7fcbc80c6e10 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.729+0000 7fcbe233b640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcbdc101970 0x7fcbdc1a57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.729+0000 7fcbe233b640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcbdc100fb0 0x7fcbdc19e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.729+0000 7fcbe233b640 1 --2- 172.21.15.49:0/1978818395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcbdc0ffe60 0x7fcbdc19e220 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:55.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.729+0000 7fcbe233b640 1 -- 172.21.15.49:0/1978818395 >> 172.21.15.49:0/1978818395 conn(0x7fcbdc0fb3d0 msgr2=0x7fcbdc102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:55.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.730+0000 7fcbe233b640 1 -- 172.21.15.49:0/1978818395 shutdown_connections 2024-03-18T00:52:55.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:55.730+0000 7fcbe233b640 1 -- 172.21.15.49:0/1978818395 wait complete. 2024-03-18T00:52:55.806 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:52:56.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.123+0000 7fc157fff640 1 -- 172.21.15.49:0/3907878560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc158108eb0 msgr2=0x7fc158109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.123+0000 7fc157fff640 1 --2- 172.21.15.49:0/3907878560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc158108eb0 0x7fc158109330 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009900 tx=0x7fc14c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:52:56.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.123+0000 7fc157fff640 1 -- 172.21.15.49:0/3907878560 shutdown_connections 2024-03-18T00:52:56.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.123+0000 7fc157fff640 1 --2- 172.21.15.49:0/3907878560 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc158109870 0x7fc15810e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.123+0000 7fc157fff640 1 --2- 172.21.15.49:0/3907878560 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc158108eb0 0x7fc158109330 secure :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009900 tx=0x7fc14c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:52:56.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.124+0000 7fc157fff640 1 --2- 172.21.15.49:0/3907878560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc158072bf0 0x7fc158073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.124+0000 7fc157fff640 1 -- 172.21.15.49:0/3907878560 >> 172.21.15.49:0/3907878560 conn(0x7fc15806e6c0 msgr2=0x7fc158070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.131+0000 7fc157fff640 1 -- 172.21.15.49:0/3907878560 shutdown_connections 2024-03-18T00:52:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.136+0000 7fc157fff640 1 -- 172.21.15.49:0/3907878560 wait complete. 2024-03-18T00:52:56.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.137+0000 7fc157fff640 1 Processor -- start 2024-03-18T00:52:56.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc157fff640 1 -- start start 2024-03-18T00:52:56.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc157fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc158072bf0 0x7fc158117100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc157fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc158109870 0x7fc158117640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc157fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1581b4900 0x7fc1581b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc157fff640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc15811c200 con 0x7fc1581b4900 2024-03-18T00:52:56.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc157fff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc15811c080 con 0x7fc158072bf0 2024-03-18T00:52:56.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc157fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc15811c380 con 0x7fc158109870 2024-03-18T00:52:56.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1581b4900 0x7fc1581b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1581b4900 0x7fc1581b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40540/0 (socket says 172.21.15.49:40540) 2024-03-18T00:52:56.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.140+0000 7fc1577fe640 1 -- 172.21.15.49:0/3971530444 learned_addr learned my addr 172.21.15.49:0/3971530444 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.142+0000 7fc156ffd640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc158072bf0 0x7fc158117100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.146+0000 7fc156ffd640 1 -- 172.21.15.49:0/3971530444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc158109870 msgr2=0x7fc158117640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.146+0000 7fc156ffd640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc158109870 0x7fc158117640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.146+0000 7fc156ffd640 1 -- 172.21.15.49:0/3971530444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1581b4900 msgr2=0x7fc1581b6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.146+0000 7fc156ffd640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1581b4900 0x7fc1581b6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.146+0000 7fc156ffd640 1 -- 172.21.15.49:0/3971530444 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1581b7280 con 0x7fc158072bf0 2024-03-18T00:52:56.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.148+0000 7fc156ffd640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc158072bf0 0x7fc158117100 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc148009fd0 tx=0x7fc14800b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.151+0000 7fc13ffff640 1 -- 172.21.15.49:0/3971530444 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc148015020 con 0x7fc158072bf0 2024-03-18T00:52:56.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.151+0000 7fc157fff640 1 -- 172.21.15.49:0/3971530444 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1581b7570 con 0x7fc158072bf0 2024-03-18T00:52:56.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.151+0000 7fc157fff640 1 -- 172.21.15.49:0/3971530444 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1581b7ac0 con 0x7fc158072bf0 2024-03-18T00:52:56.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.151+0000 7fc13ffff640 1 -- 172.21.15.49:0/3971530444 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc148011d10 con 0x7fc158072bf0 2024-03-18T00:52:56.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.151+0000 7fc13ffff640 1 -- 172.21.15.49:0/3971530444 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14800f2e0 con 0x7fc158072bf0 2024-03-18T00:52:56.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.159+0000 7fc160184740 1 Processor -- start 2024-03-18T00:52:56.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.163+0000 7fc160184740 1 -- start start 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.163+0000 7fc160184740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d57a40 0x55a180d57e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.163+0000 7fc160184740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d58430 0x55a180d5cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.163+0000 7fc160184740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d5d5a0 0x55a180d5fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.163+0000 7fc160184740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a180cbc490 con 0x55a180d5d5a0 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.163+0000 7fc160184740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a180c64c90 con 0x55a180d58430 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.163+0000 7fc160184740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a180cb8530 con 0x55a180d57a40 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.164+0000 7fc13ffff640 1 -- 172.21.15.49:0/3971530444 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc148018020 con 0x7fc158072bf0 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.165+0000 7fc13ffff640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1400a29d0 0x7fc1400a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.165+0000 7fc13ffff640 1 -- 172.21.15.49:0/3971530444 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc1480ce160 con 0x7fc158072bf0 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.167+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d57a40 0x55a180d57e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.167+0000 7fc156ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d58430 0x55a180d5cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.167+0000 7fc1567fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d5d5a0 0x55a180d5fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.167+0000 7fc1567fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d5d5a0 0x55a180d5fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40550/0 (socket says 172.21.15.49:40550) 2024-03-18T00:52:56.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.167+0000 7fc1567fc640 1 -- 172.21.15.49:0/1848363787 learned_addr learned my addr 172.21.15.49:0/1848363787 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.168+0000 7fc1567fc640 1 -- 172.21.15.49:0/1848363787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d57a40 msgr2=0x55a180d57e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:56.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.168+0000 7fc1567fc640 1 --2- 172.21.15.49:0/1848363787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d57a40 0x55a180d57e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.168+0000 7fc1567fc640 1 -- 172.21.15.49:0/1848363787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d58430 msgr2=0x55a180d5cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:56.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.168+0000 7fc1567fc640 1 --2- 172.21.15.49:0/1848363787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d58430 0x55a180d5cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.168+0000 7fc1567fc640 1 -- 172.21.15.49:0/1848363787 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a180d1f500 con 0x55a180d5d5a0 2024-03-18T00:52:56.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.169+0000 7fc1567fc640 1 --2- 172.21.15.49:0/1848363787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d5d5a0 0x55a180d5fa30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc14c02c7c0 tx=0x7fc14c009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cafcee321b25278 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.170+0000 7fc13d7fa640 1 -- 172.21.15.49:0/1848363787 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c043020 con 0x55a180d5d5a0 2024-03-18T00:52:56.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.171+0000 7fc1577fe640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1400a29d0 0x7fc1400a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.171+0000 7fc13d7fa640 1 -- 172.21.15.49:0/1848363787 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc14c02ca10 con 0x55a180d5d5a0 2024-03-18T00:52:56.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.171+0000 7fc13d7fa640 1 -- 172.21.15.49:0/1848363787 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c0046c0 con 0x55a180d5d5a0 2024-03-18T00:52:56.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.175+0000 7fc160184740 1 -- 172.21.15.49:0/1848363787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d5d5a0 msgr2=0x55a180d5fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.175+0000 7fc160184740 1 --2- 172.21.15.49:0/1848363787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d5d5a0 0x55a180d5fa30 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc14c02c7c0 tx=0x7fc14c009540 comp rx=0 tx=0).stop 2024-03-18T00:52:56.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.175+0000 7fc160184740 1 -- 172.21.15.49:0/1848363787 shutdown_connections 2024-03-18T00:52:56.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.175+0000 7fc160184740 1 --2- 172.21.15.49:0/1848363787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d58430 0x55a180d5cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.175+0000 7fc160184740 1 --2- 172.21.15.49:0/1848363787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d5d5a0 0x55a180d5fa30 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.175+0000 7fc160184740 1 --2- 172.21.15.49:0/1848363787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d57a40 0x55a180d57e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.175+0000 7fc160184740 1 -- 172.21.15.49:0/1848363787 >> 172.21.15.49:0/1848363787 conn(0x55a180d52fb0 msgr2=0x55a180d533b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.176+0000 7fc160184740 1 -- 172.21.15.49:0/1848363787 shutdown_connections 2024-03-18T00:52:56.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.177+0000 7fc160184740 1 -- 172.21.15.49:0/1848363787 wait complete. 2024-03-18T00:52:56.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.178+0000 7fc160184740 1 -- start start 2024-03-18T00:52:56.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.178+0000 7fc160184740 1 Processor -- start 2024-03-18T00:52:56.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.179+0000 7fc1577fe640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1400a29d0 0x7fc1400a4e90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55a180d58c90 tx=0x7fc150007560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.179+0000 7fc160184740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d57a40 0x55a180d61f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.179+0000 7fc160184740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d58430 0x55a180d624d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.179+0000 7fc160184740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d69510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.179+0000 7fc160184740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a180d604f0 con 0x55a180d57a40 2024-03-18T00:52:56.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.179+0000 7fc160184740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a180c78230 con 0x55a180d58430 2024-03-18T00:52:56.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.179+0000 7fc160184740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a180d60670 con 0x55a180d5d5a0 2024-03-18T00:52:56.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.179+0000 7fc160184740 10 client.0 _ll_register_callbacks cb 0x55a180cb4010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:56.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.180+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d69510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.181+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d69510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34696/0 (socket says 172.21.15.49:34696) 2024-03-18T00:52:56.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.181+0000 7fc1577fe640 1 -- 172.21.15.49:0/641144395 learned_addr learned my addr 172.21.15.49:0/641144395 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.183+0000 7fc1577fe640 1 -- 172.21.15.49:0/641144395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d58430 msgr2=0x55a180d624d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:56.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.184+0000 7fc1577fe640 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d58430 0x55a180d624d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.184+0000 7fc1577fe640 1 -- 172.21.15.49:0/641144395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d57a40 msgr2=0x55a180d61f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:56.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.184+0000 7fc1577fe640 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d57a40 0x55a180d61f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.184+0000 7fc1577fe640 1 -- 172.21.15.49:0/641144395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a180c87100 con 0x55a180d5d5a0 2024-03-18T00:52:56.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.186+0000 7fc1577fe640 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d69510 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc150012e20 tx=0x7fc150004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b55dc2a967b9c8b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.196+0000 7fc13aff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:52:56.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.196+0000 7fc13aff5640 1 -- 172.21.15.49:0/641144395 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150023070 con 0x55a180d5d5a0 2024-03-18T00:52:56.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.196+0000 7fc13aff5640 1 -- 172.21.15.49:0/641144395 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc15001f650 con 0x55a180d5d5a0 2024-03-18T00:52:56.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.196+0000 7fc160184740 1 -- 172.21.15.49:0/641144395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc14c02ca10 con 0x55a180d5d5a0 2024-03-18T00:52:56.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.199+0000 7fc13aff5640 1 -- 172.21.15.49:0/641144395 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc15002d050 con 0x55a180d5d5a0 2024-03-18T00:52:56.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.205+0000 7fc160184740 10 client.25807 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:56.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.205+0000 7fc160184740 1 -- 172.21.15.49:0/641144395 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a1809d03f0 con 0x55a180d5d5a0 2024-03-18T00:52:56.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.208+0000 7fc13aff5640 1 -- 172.21.15.49:0/641144395 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc14c02ca10 con 0x55a180d5d5a0 2024-03-18T00:52:56.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.208+0000 7fc13aff5640 1 -- 172.21.15.49:0/641144395 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fc150028a50 con 0x55a180d5d5a0 2024-03-18T00:52:56.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.208+0000 7fc160184740 10 client.25807 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:56.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.208+0000 7fc160184740 10 client.25807 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:52:56.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.208+0000 7fc160184740 20 client.25807 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:56.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.208+0000 7fc160184740 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a180d6df20 0x55a180d703e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.208+0000 7fc160184740 4 client.25807 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:56.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.208+0000 7fc160184740 1 -- 172.21.15.49:0/641144395 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a180cc99d0 con 0x55a180d6df20 2024-03-18T00:52:56.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.209+0000 7fc156ffd640 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a180d6df20 0x55a180d703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.216+0000 7fc13aff5640 10 client.25807 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:52:56.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.216+0000 7fc156ffd640 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a180d6df20 0x55a180d703e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f0f48eeb4bd79bdf server_cookie=43cab5015339216c in_seq=0 out_seq=0 2024-03-18T00:52:56.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.221+0000 7fc13aff5640 1 -- 172.21.15.49:0/641144395 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a180cc99d0 con 0x55a180d6df20 2024-03-18T00:52:56.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.221+0000 7fc13aff5640 10 client.25807 handle_command_reply: tid=0 2024-03-18T00:52:56.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.221+0000 7fc160184740 1 client.25807 shutdown 2024-03-18T00:52:56.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.225+0000 7fc160184740 1 -- 172.21.15.49:0/641144395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 msgr2=0x55a180d69510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.225+0000 7fc160184740 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d69510 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc150012e20 tx=0x7fc150004ab0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.225+0000 7fc160184740 1 -- 172.21.15.49:0/641144395 shutdown_connections 2024-03-18T00:52:56.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.225+0000 7fc160184740 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a180d6df20 0x55a180d703e0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.225+0000 7fc160184740 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d58430 0x55a180d624d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.225+0000 7fc160184740 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d69510 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.225+0000 7fc160184740 1 --2- 172.21.15.49:0/641144395 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d57a40 0x55a180d61f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.225+0000 7fc160184740 1 -- 172.21.15.49:0/641144395 >> 172.21.15.49:0/641144395 conn(0x55a180d52fb0 msgr2=0x55a180d52140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.225+0000 7fc13aff5640 0 client.25807 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:52:56.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.228+0000 7fc160184740 1 -- 172.21.15.49:0/641144395 shutdown_connections 2024-03-18T00:52:56.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.228+0000 7fc160184740 1 -- 172.21.15.49:0/641144395 wait complete. 2024-03-18T00:52:56.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.228+0000 7fc160184740 20 client.25807 trim_cache size 0 max 16384 2024-03-18T00:52:56.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.298+0000 7fc160184740 1 Processor -- start 2024-03-18T00:52:56.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.298+0000 7fc160184740 1 -- start start 2024-03-18T00:52:56.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.299+0000 7fc160184740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d57a40 0x55a180d4dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.299+0000 7fc160184740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d58430 0x55a180d694d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.299+0000 7fc160184740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d518b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.299+0000 7fc160184740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc14400e5f0 con 0x55a180d58430 2024-03-18T00:52:56.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.299+0000 7fc160184740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc144010160 con 0x55a180d57a40 2024-03-18T00:52:56.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.299+0000 7fc160184740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc14400ecc0 con 0x55a180d5d5a0 2024-03-18T00:52:56.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.300+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.300+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d518b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34698/0 (socket says 172.21.15.49:34698) 2024-03-18T00:52:56.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.300+0000 7fc1577fe640 1 -- 172.21.15.49:0/3010577373 learned_addr learned my addr 172.21.15.49:0/3010577373 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.300+0000 7fc156ffd640 1 --2- 172.21.15.49:0/3010577373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d57a40 0x55a180d4dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.303+0000 7fc1577fe640 1 -- 172.21.15.49:0/3010577373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d57a40 msgr2=0x55a180d4dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:56.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.303+0000 7fc1577fe640 1 --2- 172.21.15.49:0/3010577373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d57a40 0x55a180d4dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.303+0000 7fc1577fe640 1 -- 172.21.15.49:0/3010577373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d58430 msgr2=0x55a180d694d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:56.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.303+0000 7fc1577fe640 1 --2- 172.21.15.49:0/3010577373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d58430 0x55a180d694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.303+0000 7fc1577fe640 1 -- 172.21.15.49:0/3010577373 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a180a9a7d0 con 0x55a180d5d5a0 2024-03-18T00:52:56.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.308+0000 7fc1577fe640 1 --2- 172.21.15.49:0/3010577373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d518b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc150000f80 tx=0x7fc15000ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bacf0f80f4cc4b19 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.310+0000 7fc13cff9640 1 -- 172.21.15.49:0/3010577373 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150023070 con 0x55a180d5d5a0 2024-03-18T00:52:56.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.310+0000 7fc13cff9640 1 -- 172.21.15.49:0/3010577373 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc150027600 con 0x55a180d5d5a0 2024-03-18T00:52:56.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.314+0000 7fc160184740 1 -- 172.21.15.49:0/3010577373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 msgr2=0x55a180d518b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.314+0000 7fc160184740 1 --2- 172.21.15.49:0/3010577373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d518b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc150000f80 tx=0x7fc15000ed50 comp rx=0 tx=0).stop 2024-03-18T00:52:56.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.319+0000 7fc160184740 1 -- 172.21.15.49:0/3010577373 shutdown_connections 2024-03-18T00:52:56.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.319+0000 7fc160184740 1 --2- 172.21.15.49:0/3010577373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d5d5a0 0x55a180d518b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.319+0000 7fc160184740 1 --2- 172.21.15.49:0/3010577373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d58430 0x55a180d694d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.319+0000 7fc160184740 1 --2- 172.21.15.49:0/3010577373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d57a40 0x55a180d4dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.319+0000 7fc160184740 1 -- 172.21.15.49:0/3010577373 >> 172.21.15.49:0/3010577373 conn(0x55a180d52fb0 msgr2=0x55a180d59970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.329+0000 7fc160184740 1 -- 172.21.15.49:0/3010577373 shutdown_connections 2024-03-18T00:52:56.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.331+0000 7fc160184740 1 -- 172.21.15.49:0/3010577373 wait complete. 2024-03-18T00:52:56.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.331+0000 7fc160184740 1 -- start start 2024-03-18T00:52:56.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.332+0000 7fc160184740 1 Processor -- start 2024-03-18T00:52:56.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.332+0000 7fc160184740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d57a40 0x55a180d67070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.332+0000 7fc160184740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d58430 0x55a180d675b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.332+0000 7fc160184740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d5d5a0 0x55a180d684e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.332+0000 7fc160184740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc14400fa60 con 0x55a180d58430 2024-03-18T00:52:56.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.332+0000 7fc160184740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc14400f390 con 0x55a180d5d5a0 2024-03-18T00:52:56.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.332+0000 7fc160184740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc14400c5d0 con 0x55a180d57a40 2024-03-18T00:52:56.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.333+0000 7fc160184740 10 client.0 _ll_register_callbacks cb 0x7fc150016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:56.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.333+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d5d5a0 0x55a180d684e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.334+0000 7fc1577fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d5d5a0 0x55a180d684e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:50372/0 (socket says 172.21.15.49:50372) 2024-03-18T00:52:56.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.334+0000 7fc1577fe640 1 -- 172.21.15.49:0/1780385436 learned_addr learned my addr 172.21.15.49:0/1780385436 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.335+0000 7fc1577fe640 1 -- 172.21.15.49:0/1780385436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d57a40 msgr2=0x55a180d67070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:56.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.335+0000 7fc1577fe640 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d57a40 0x55a180d67070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.335+0000 7fc1577fe640 1 -- 172.21.15.49:0/1780385436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d58430 msgr2=0x55a180d675b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:52:56.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.335+0000 7fc1577fe640 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d58430 0x55a180d675b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.335+0000 7fc1577fe640 1 -- 172.21.15.49:0/1780385436 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a180d68ad0 con 0x55a180d5d5a0 2024-03-18T00:52:56.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.337+0000 7fc1577fe640 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d5d5a0 0x55a180d684e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc150027fd0 tx=0x7fc150028ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5700ae4bff64a5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.337+0000 7fc13aff5640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:52:56.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.339+0000 7fc13aff5640 1 -- 172.21.15.49:0/1780385436 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc150023040 con 0x55a180d5d5a0 2024-03-18T00:52:56.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.339+0000 7fc160184740 1 -- 172.21.15.49:0/1780385436 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc150027600 con 0x55a180d5d5a0 2024-03-18T00:52:56.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.339+0000 7fc13aff5640 1 -- 172.21.15.49:0/1780385436 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc150004c10 con 0x55a180d5d5a0 2024-03-18T00:52:56.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.342+0000 7fc13aff5640 1 -- 172.21.15.49:0/1780385436 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc150004e10 con 0x55a180d5d5a0 2024-03-18T00:52:56.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.345+0000 7fc160184740 10 client.25736 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:56.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.345+0000 7fc160184740 1 -- 172.21.15.49:0/1780385436 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a180dd4660 con 0x55a180d5d5a0 2024-03-18T00:52:56.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.346+0000 7fc13aff5640 1 -- 172.21.15.49:0/1780385436 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc150027600 con 0x55a180d5d5a0 2024-03-18T00:52:56.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.346+0000 7fc13aff5640 1 -- 172.21.15.49:0/1780385436 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fc150039030 con 0x55a180d5d5a0 2024-03-18T00:52:56.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.346+0000 7fc160184740 10 client.25736 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:56.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.346+0000 7fc160184740 10 client.25736 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:52:56.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.346+0000 7fc160184740 20 client.25736 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:56.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.346+0000 7fc160184740 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a180dd4ef0 0x55a180dd7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.347+0000 7fc160184740 4 client.25736 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:56.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.347+0000 7fc160184740 1 -- 172.21.15.49:0/1780385436 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a180cc3d20 con 0x55a180dd4ef0 2024-03-18T00:52:56.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.348+0000 7fc156ffd640 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a180dd4ef0 0x55a180dd7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.353+0000 7fc156ffd640 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a180dd4ef0 0x55a180dd7310 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b314cd9b6953991 server_cookie=20cfc2a6ccaa1c95 in_seq=0 out_seq=0 2024-03-18T00:52:56.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.353+0000 7fc13aff5640 10 client.25736 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:52:56.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.353+0000 7fc13aff5640 1 -- 172.21.15.49:0/1780385436 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a180cc3d20 con 0x55a180dd4ef0 2024-03-18T00:52:56.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.353+0000 7fc13aff5640 10 client.25736 handle_command_reply: tid=0 2024-03-18T00:52:56.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.355+0000 7fc160184740 1 client.25736 shutdown 2024-03-18T00:52:56.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.358+0000 7fc160184740 1 -- 172.21.15.49:0/1780385436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d5d5a0 msgr2=0x55a180d684e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.358+0000 7fc160184740 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d5d5a0 0x55a180d684e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc150027fd0 tx=0x7fc150028ab0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.360+0000 7fc160184740 1 -- 172.21.15.49:0/1780385436 shutdown_connections 2024-03-18T00:52:56.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.360+0000 7fc160184740 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55a180dd4ef0 0x55a180dd7310 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.360+0000 7fc160184740 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a180d5d5a0 0x55a180d684e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.360+0000 7fc160184740 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a180d58430 0x55a180d675b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.360+0000 7fc160184740 1 --2- 172.21.15.49:0/1780385436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a180d57a40 0x55a180d67070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.360+0000 7fc160184740 1 -- 172.21.15.49:0/1780385436 >> 172.21.15.49:0/1780385436 conn(0x55a180d52fb0 msgr2=0x55a180d518b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.360+0000 7fc13aff5640 0 client.25736 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:52:56.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.362+0000 7fc160184740 1 -- 172.21.15.49:0/1780385436 shutdown_connections 2024-03-18T00:52:56.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.362+0000 7fc160184740 1 -- 172.21.15.49:0/1780385436 wait complete. 2024-03-18T00:52:56.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.362+0000 7fc160184740 20 client.25736 trim_cache size 0 max 16384 2024-03-18T00:52:56.368 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:56.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.368+0000 7fc157fff640 1 -- 172.21.15.49:0/3971530444 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1400a29d0 msgr2=0x7fc1400a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.368+0000 7fc157fff640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1400a29d0 0x7fc1400a4e90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55a180d58c90 tx=0x7fc150007560 comp rx=0 tx=0).stop 2024-03-18T00:52:56.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.368+0000 7fc157fff640 1 -- 172.21.15.49:0/3971530444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc158072bf0 msgr2=0x7fc158117100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.368+0000 7fc157fff640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc158072bf0 0x7fc158117100 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc148009fd0 tx=0x7fc14800b870 comp rx=0 tx=0).stop 2024-03-18T00:52:56.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.369+0000 7fc157fff640 1 -- 172.21.15.49:0/3971530444 shutdown_connections 2024-03-18T00:52:56.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.369+0000 7fc157fff640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1400a29d0 0x7fc1400a4e90 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.369+0000 7fc157fff640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1581b4900 0x7fc1581b6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.369+0000 7fc157fff640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc158109870 0x7fc158117640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.369+0000 7fc157fff640 1 --2- 172.21.15.49:0/3971530444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc158072bf0 0x7fc158117100 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.369+0000 7fc157fff640 1 -- 172.21.15.49:0/3971530444 >> 172.21.15.49:0/3971530444 conn(0x7fc15806e6c0 msgr2=0x7fc15810c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.369+0000 7fc157fff640 1 -- 172.21.15.49:0/3971530444 shutdown_connections 2024-03-18T00:52:56.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.369+0000 7fc157fff640 1 -- 172.21.15.49:0/3971530444 wait complete. 2024-03-18T00:52:56.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:52:56.751 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:56 smithi049 ceph-mon[32130]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 818 MiB/s wr, 1.35k op/s 2024-03-18T00:52:56.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.770+0000 7fe307cbf640 1 -- 172.21.15.49:0/829581846 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe30010b150 msgr2=0x7fe30010d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.770+0000 7fe307cbf640 1 --2- 172.21.15.49:0/829581846 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe30010b150 0x7fe30010d5c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009900 tx=0x7fe2fc02bc50 comp rx=0 tx=0).stop 2024-03-18T00:52:56.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.771+0000 7fe307cbf640 1 -- 172.21.15.49:0/829581846 shutdown_connections 2024-03-18T00:52:56.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.771+0000 7fe307cbf640 1 --2- 172.21.15.49:0/829581846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe30010db00 0x7fe30010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.771+0000 7fe307cbf640 1 --2- 172.21.15.49:0/829581846 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe30010b150 0x7fe30010d5c0 secure :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009900 tx=0x7fe2fc02bc50 comp rx=0 tx=0).stop 2024-03-18T00:52:56.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.771+0000 7fe307cbf640 1 --2- 172.21.15.49:0/829581846 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe3001087e0 0x7fe30010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.771+0000 7fe307cbf640 1 -- 172.21.15.49:0/829581846 >> 172.21.15.49:0/829581846 conn(0x7fe30006e640 msgr2=0x7fe300070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.779+0000 7fe307cbf640 1 -- 172.21.15.49:0/829581846 shutdown_connections 2024-03-18T00:52:56.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.781+0000 7fe307cbf640 1 -- 172.21.15.49:0/829581846 wait complete. 2024-03-18T00:52:56.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.782+0000 7fe307cbf640 1 Processor -- start 2024-03-18T00:52:56.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.782+0000 7fe307cbf640 1 -- start start 2024-03-18T00:52:56.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.782+0000 7fe307cbf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3001087e0 0x7fe300115090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.782+0000 7fe307cbf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe30010db00 0x7fe3001155d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.782+0000 7fe307cbf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3001b4a20 0x7fe3001b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.782+0000 7fe307cbf640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe30011c320 con 0x7fe3001087e0 2024-03-18T00:52:56.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.782+0000 7fe307cbf640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe30011c1a0 con 0x7fe30010db00 2024-03-18T00:52:56.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.782+0000 7fe307cbf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe30011c4a0 con 0x7fe3001b4a20 2024-03-18T00:52:56.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.784+0000 7fe30625c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3001b4a20 0x7fe3001b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.786+0000 7fe305a5b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3001087e0 0x7fe300115090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.786+0000 7fe305a5b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3001087e0 0x7fe300115090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40576/0 (socket says 172.21.15.49:40576) 2024-03-18T00:52:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.786+0000 7fe305a5b640 1 -- 172.21.15.49:0/1649139392 learned_addr learned my addr 172.21.15.49:0/1649139392 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.787+0000 7fe305a5b640 1 -- 172.21.15.49:0/1649139392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3001b4a20 msgr2=0x7fe3001b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.787+0000 7fe305a5b640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3001b4a20 0x7fe3001b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.787+0000 7fe305a5b640 1 -- 172.21.15.49:0/1649139392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe30010db00 msgr2=0x7fe3001155d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.787+0000 7fe305a5b640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe30010db00 0x7fe3001155d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.787+0000 7fe305a5b640 1 -- 172.21.15.49:0/1649139392 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3001b7380 con 0x7fe3001087e0 2024-03-18T00:52:56.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.788+0000 7fe305a5b640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3001087e0 0x7fe300115090 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800ba50 tx=0x7fe2f800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.791+0000 7fe2f6ffd640 1 -- 172.21.15.49:0/1649139392 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8019070 con 0x7fe3001087e0 2024-03-18T00:52:56.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.791+0000 7fe2f6ffd640 1 -- 172.21.15.49:0/1649139392 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2f800cbe0 con 0x7fe3001087e0 2024-03-18T00:52:56.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.791+0000 7fe307cbf640 1 -- 172.21.15.49:0/1649139392 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3001b7670 con 0x7fe3001087e0 2024-03-18T00:52:56.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.792+0000 7fe307cbf640 1 -- 172.21.15.49:0/1649139392 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3001b7bc0 con 0x7fe3001087e0 2024-03-18T00:52:56.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.792+0000 7fe2f6ffd640 1 -- 172.21.15.49:0/1649139392 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80076a0 con 0x7fe3001087e0 2024-03-18T00:52:56.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.796+0000 7fe30a982740 1 Processor -- start 2024-03-18T00:52:56.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.797+0000 7fe30a982740 1 -- start start 2024-03-18T00:52:56.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.798+0000 7fe30a982740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385928850 0x55e385928c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.799+0000 7fe30625c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385928850 0x55e385928c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.799+0000 7fe30a982740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385929240 0x55e38592dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.799+0000 7fe30a982740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e385930840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.799+0000 7fe30a982740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e38588d490 con 0x55e385928850 2024-03-18T00:52:56.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.799+0000 7fe30a982740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e385835c90 con 0x55e38592e3b0 2024-03-18T00:52:56.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.799+0000 7fe30a982740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e385889530 con 0x55e385929240 2024-03-18T00:52:56.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.799+0000 7fe30625c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385928850 0x55e385928c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40590/0 (socket says 172.21.15.49:40590) 2024-03-18T00:52:56.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.799+0000 7fe30625c640 1 -- 172.21.15.49:0/3228330174 learned_addr learned my addr 172.21.15.49:0/3228330174 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.801+0000 7fe2f6ffd640 1 -- 172.21.15.49:0/1649139392 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe2f801c070 con 0x7fe3001087e0 2024-03-18T00:52:56.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.803+0000 7fe2f6ffd640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2ec0a28c0 0x7fe2ec0a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.803+0000 7fe2f6ffd640 1 -- 172.21.15.49:0/1649139392 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe2f80ce000 con 0x7fe3001087e0 2024-03-18T00:52:56.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.805+0000 7fe30625c640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2ec0a28c0 0x7fe2ec0a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.805+0000 7fe30625c640 1 -- 172.21.15.49:0/3228330174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385929240 msgr2=0x55e38592dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:56.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.805+0000 7fe30625c640 1 --2- 172.21.15.49:0/3228330174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385929240 0x55e38592dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.805+0000 7fe30625c640 1 -- 172.21.15.49:0/3228330174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 msgr2=0x55e385930840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:56.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.805+0000 7fe30625c640 1 --2- 172.21.15.49:0/3228330174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e385930840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.805+0000 7fe30625c640 1 -- 172.21.15.49:0/3228330174 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3858f0500 con 0x55e385928850 2024-03-18T00:52:56.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.808+0000 7fe30625c640 1 --2- 172.21.15.49:0/3228330174 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385928850 0x55e385928c70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0005fd0 tx=0x7fe2f0015cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f1b98d27d4cfb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.808+0000 7fe30625c640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2ec0a28c0 0x7fe2ec0a4d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000c000 tx=0x7fe2f001b000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.808+0000 7fe2ebfff640 1 -- 172.21.15.49:0/3228330174 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f0022070 con 0x55e385928850 2024-03-18T00:52:56.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.813+0000 7fe2ebfff640 1 -- 172.21.15.49:0/3228330174 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2f0004cc0 con 0x55e385928850 2024-03-18T00:52:56.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.815+0000 7fe30a982740 1 -- 172.21.15.49:0/3228330174 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385928850 msgr2=0x55e385928c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.815+0000 7fe30a982740 1 --2- 172.21.15.49:0/3228330174 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385928850 0x55e385928c70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0005fd0 tx=0x7fe2f0015cd0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.815+0000 7fe30a982740 1 -- 172.21.15.49:0/3228330174 shutdown_connections 2024-03-18T00:52:56.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.815+0000 7fe30a982740 1 --2- 172.21.15.49:0/3228330174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e385930840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.815+0000 7fe30a982740 1 --2- 172.21.15.49:0/3228330174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385929240 0x55e38592dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.815+0000 7fe30a982740 1 --2- 172.21.15.49:0/3228330174 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385928850 0x55e385928c70 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.815+0000 7fe30a982740 1 -- 172.21.15.49:0/3228330174 >> 172.21.15.49:0/3228330174 conn(0x55e385923dc0 msgr2=0x55e3859241c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.818+0000 7fe30a982740 1 -- 172.21.15.49:0/3228330174 shutdown_connections 2024-03-18T00:52:56.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.819+0000 7fe30a982740 1 -- 172.21.15.49:0/3228330174 wait complete. 2024-03-18T00:52:56.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.819+0000 7fe30a982740 1 -- start start 2024-03-18T00:52:56.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.819+0000 7fe30a982740 1 Processor -- start 2024-03-18T00:52:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.820+0000 7fe30a982740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385932ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.820+0000 7fe30a982740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e3859331e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.820+0000 7fe30a982740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e38593a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.820+0000 7fe30a982740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e3859313a0 con 0x55e385929240 2024-03-18T00:52:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.820+0000 7fe30a982740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e385849230 con 0x55e38592e3b0 2024-03-18T00:52:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.820+0000 7fe30a982740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e385931520 con 0x55e385938200 2024-03-18T00:52:56.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.820+0000 7fe30a982740 10 client.0 _ll_register_callbacks cb 0x55e385885010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:56.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.821+0000 7fe305a5b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385932ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.821+0000 7fe305a5b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385932ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40594/0 (socket says 172.21.15.49:40594) 2024-03-18T00:52:56.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.821+0000 7fe305a5b640 1 -- 172.21.15.49:0/2550206693 learned_addr learned my addr 172.21.15.49:0/2550206693 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.821+0000 7fe305a5b640 1 -- 172.21.15.49:0/2550206693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 msgr2=0x55e38593a5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:56.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.821+0000 7fe305a5b640 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e38593a5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.821+0000 7fe305a5b640 1 -- 172.21.15.49:0/2550206693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 msgr2=0x55e3859331e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:56.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.821+0000 7fe305a5b640 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e3859331e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.821+0000 7fe305a5b640 1 -- 172.21.15.49:0/2550206693 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e385858100 con 0x55e385929240 2024-03-18T00:52:56.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.822+0000 7fe305a5b640 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385932ca0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8097350 tx=0x7fe2f809b140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7586181897194160 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.822+0000 7fe2e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:56.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.833+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2550206693 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2f80a3120 con 0x55e385929240 2024-03-18T00:52:56.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.834+0000 7fe30a982740 1 -- 172.21.15.49:0/2550206693 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e38589a9d0 con 0x55e385929240 2024-03-18T00:52:56.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.834+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2550206693 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe2f8098c80 con 0x55e385929240 2024-03-18T00:52:56.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.835+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2550206693 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe2f80a2c90 con 0x55e385929240 2024-03-18T00:52:56.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.835+0000 7fe30a982740 10 client.16107 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:56.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.835+0000 7fe30a982740 1 -- 172.21.15.49:0/2550206693 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e3855a13f0 con 0x55e385929240 2024-03-18T00:52:56.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.836+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2550206693 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e38589a9d0 con 0x55e385929240 2024-03-18T00:52:56.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.837+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2550206693 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fe2f8099dd0 con 0x55e385929240 2024-03-18T00:52:56.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.837+0000 7fe30a982740 10 client.16107 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:56.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.837+0000 7fe30a982740 10 client.16107 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:52:56.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.837+0000 7fe30a982740 20 client.16107 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:56.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.837+0000 7fe30a982740 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55e38593f420 0x55e3859418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.837+0000 7fe30a982740 4 client.16107 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:56.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.837+0000 7fe30a982740 1 -- 172.21.15.49:0/2550206693 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e385894d20 con 0x55e38593f420 2024-03-18T00:52:56.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.838+0000 7fe30525a640 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55e38593f420 0x55e3859418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.840+0000 7fe2e97fa640 10 client.16107 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:52:56.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.840+0000 7fe30525a640 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55e38593f420 0x55e3859418c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=deec6507e80bc177 server_cookie=a63afa243b0f721b in_seq=0 out_seq=0 2024-03-18T00:52:56.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.842+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2550206693 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e385894d20 con 0x55e38593f420 2024-03-18T00:52:56.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.842+0000 7fe2e97fa640 10 client.16107 handle_command_reply: tid=0 2024-03-18T00:52:56.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.842+0000 7fe30a982740 1 client.16107 shutdown 2024-03-18T00:52:56.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.842+0000 7fe30a982740 1 -- 172.21.15.49:0/2550206693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 msgr2=0x55e385932ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.842+0000 7fe30a982740 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385932ca0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe2f8097350 tx=0x7fe2f809b140 comp rx=0 tx=0).stop 2024-03-18T00:52:56.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe30a982740 1 -- 172.21.15.49:0/2550206693 shutdown_connections 2024-03-18T00:52:56.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe30a982740 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55e38593f420 0x55e3859418c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe30a982740 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e3859331e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe30a982740 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e38593a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe30a982740 1 --2- 172.21.15.49:0/2550206693 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385932ca0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe30a982740 1 -- 172.21.15.49:0/2550206693 >> 172.21.15.49:0/2550206693 conn(0x55e385923dc0 msgr2=0x55e385928a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe2e97fa640 0 client.16107 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:52:56.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe30a982740 1 -- 172.21.15.49:0/2550206693 shutdown_connections 2024-03-18T00:52:56.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe30a982740 1 -- 172.21.15.49:0/2550206693 wait complete. 2024-03-18T00:52:56.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.843+0000 7fe30a982740 20 client.16107 trim_cache size 0 max 16384 2024-03-18T00:52:56.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.870+0000 7fe30a982740 1 Processor -- start 2024-03-18T00:52:56.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.870+0000 7fe30a982740 1 -- start start 2024-03-18T00:52:56.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.871+0000 7fe30a982740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e385929240 0x55e38592a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.871+0000 7fe30a982740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e38592e3b0 0x55e38592aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.871+0000 7fe30a982740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e385932c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.871+0000 7fe30a982740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e400e690 con 0x55e38592e3b0 2024-03-18T00:52:56.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.871+0000 7fe30a982740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e4010200 con 0x55e385929240 2024-03-18T00:52:56.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.871+0000 7fe30a982740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e400ed60 con 0x55e385938200 2024-03-18T00:52:56.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.871+0000 7fe30525a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e38592e3b0 0x55e38592aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.871+0000 7fe30525a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e38592e3b0 0x55e38592aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40598/0 (socket says 172.21.15.49:40598) 2024-03-18T00:52:56.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.871+0000 7fe30525a640 1 -- 172.21.15.49:0/3423336670 learned_addr learned my addr 172.21.15.49:0/3423336670 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.872+0000 7fe30525a640 1 -- 172.21.15.49:0/3423336670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 msgr2=0x55e385932c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:56.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.872+0000 7fe30525a640 1 --2- 172.21.15.49:0/3423336670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e385932c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.872+0000 7fe30525a640 1 -- 172.21.15.49:0/3423336670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e385929240 msgr2=0x55e38592a560 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:56.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.872+0000 7fe30525a640 1 --2- 172.21.15.49:0/3423336670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e385929240 0x55e38592a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.872+0000 7fe30525a640 1 -- 172.21.15.49:0/3423336670 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e38566b7d0 con 0x55e38592e3b0 2024-03-18T00:52:56.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.872+0000 7fe30525a640 1 --2- 172.21.15.49:0/3423336670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e38592e3b0 0x55e38592aaa0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009230 tx=0x7fe2fc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d9aa574b19ce02 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.872+0000 7fe2eb7fe640 1 -- 172.21.15.49:0/3423336670 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc02cba0 con 0x55e38592e3b0 2024-03-18T00:52:56.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.872+0000 7fe2eb7fe640 1 -- 172.21.15.49:0/3423336670 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2fc005d10 con 0x55e38592e3b0 2024-03-18T00:52:56.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.872+0000 7fe2eb7fe640 1 -- 172.21.15.49:0/3423336670 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc0035f0 con 0x55e38592e3b0 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.873+0000 7fe30a982740 1 -- 172.21.15.49:0/3423336670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e38592e3b0 msgr2=0x55e38592aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.873+0000 7fe30a982740 1 --2- 172.21.15.49:0/3423336670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e38592e3b0 0x55e38592aaa0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009230 tx=0x7fe2fc005fd0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.874+0000 7fe30a982740 1 -- 172.21.15.49:0/3423336670 shutdown_connections 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.874+0000 7fe30a982740 1 --2- 172.21.15.49:0/3423336670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e385932c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.874+0000 7fe30a982740 1 --2- 172.21.15.49:0/3423336670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e38592e3b0 0x55e38592aaa0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.874+0000 7fe30a982740 1 --2- 172.21.15.49:0/3423336670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e385929240 0x55e38592a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.874+0000 7fe30a982740 1 -- 172.21.15.49:0/3423336670 >> 172.21.15.49:0/3423336670 conn(0x55e385923dc0 msgr2=0x55e3859241a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.874+0000 7fe30a982740 1 -- 172.21.15.49:0/3423336670 shutdown_connections 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.874+0000 7fe30a982740 1 -- 172.21.15.49:0/3423336670 wait complete. 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.874+0000 7fe30a982740 1 -- start start 2024-03-18T00:52:56.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.875+0000 7fe30a982740 1 Processor -- start 2024-03-18T00:52:56.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.875+0000 7fe30a982740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385950110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.875+0000 7fe30a982740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e385950650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.875+0000 7fe30a982740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e385951660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.875+0000 7fe30a982740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e400fb00 con 0x55e385929240 2024-03-18T00:52:56.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.875+0000 7fe30a982740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e400f430 con 0x55e38592e3b0 2024-03-18T00:52:56.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.875+0000 7fe30a982740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2e400c670 con 0x55e385938200 2024-03-18T00:52:56.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.875+0000 7fe30a982740 10 client.0 _ll_register_callbacks cb 0x7fe2f80099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:56.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.876+0000 7fe30625c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e385951660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.876+0000 7fe30625c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e385951660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34750/0 (socket says 172.21.15.49:34750) 2024-03-18T00:52:56.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.876+0000 7fe30625c640 1 -- 172.21.15.49:0/2541208261 learned_addr learned my addr 172.21.15.49:0/2541208261 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:56.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.877+0000 7fe305a5b640 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385950110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.877+0000 7fe30525a640 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e385950650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.878+0000 7fe30525a640 1 -- 172.21.15.49:0/2541208261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 msgr2=0x55e385951660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:56.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.878+0000 7fe30525a640 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e385951660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.878+0000 7fe30525a640 1 -- 172.21.15.49:0/2541208261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 msgr2=0x55e385950110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:56.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.878+0000 7fe30525a640 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385950110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.878+0000 7fe30525a640 1 -- 172.21.15.49:0/2541208261 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e385951c50 con 0x55e38592e3b0 2024-03-18T00:52:56.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.878+0000 7fe305a5b640 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385950110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:52:56.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.879+0000 7fe30525a640 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e385950650 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc005ec0 tx=0x7fe2fc0436c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64544e06a44d9089 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:56.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.879+0000 7fe2e97fa640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:52:56.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.880+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2541208261 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc053070 con 0x55e38592e3b0 2024-03-18T00:52:56.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.881+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2541208261 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe2fc02cba0 con 0x55e38592e3b0 2024-03-18T00:52:56.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.881+0000 7fe30a982740 1 -- 172.21.15.49:0/2541208261 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2fc005d10 con 0x55e38592e3b0 2024-03-18T00:52:56.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.882+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2541208261 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe2fc05d050 con 0x55e38592e3b0 2024-03-18T00:52:56.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.882+0000 7fe30a982740 10 client.25739 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:56.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.883+0000 7fe30a982740 1 -- 172.21.15.49:0/2541208261 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e3859a5220 con 0x55e38592e3b0 2024-03-18T00:52:56.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.884+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2541208261 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2fc005d10 con 0x55e38592e3b0 2024-03-18T00:52:56.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.884+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2541208261 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fe2fc004730 con 0x55e38592e3b0 2024-03-18T00:52:56.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.884+0000 7fe30a982740 10 client.25739 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:56.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.884+0000 7fe30a982740 10 client.25739 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:52:56.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.884+0000 7fe30a982740 20 client.25739 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:56.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.884+0000 7fe30a982740 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55e3859a5c40 0x55e3859a8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:56.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.884+0000 7fe30a982740 4 client.25739 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:56.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.884+0000 7fe30a982740 1 -- 172.21.15.49:0/2541208261 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e3858126a0 con 0x55e3859a5c40 2024-03-18T00:52:56.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.885+0000 7fe305a5b640 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55e3859a5c40 0x55e3859a8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:56.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.887+0000 7fe305a5b640 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55e3859a5c40 0x55e3859a8060 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eaf6c95ab46346cf server_cookie=5843b04ea6cac4f9 in_seq=0 out_seq=0 2024-03-18T00:52:56.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.887+0000 7fe2e97fa640 10 client.25739 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:52:56.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.887+0000 7fe2e97fa640 1 -- 172.21.15.49:0/2541208261 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e3858126a0 con 0x55e3859a5c40 2024-03-18T00:52:56.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.887+0000 7fe2e97fa640 10 client.25739 handle_command_reply: tid=0 2024-03-18T00:52:56.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.887+0000 7fe30a982740 1 client.25739 shutdown 2024-03-18T00:52:56.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe30a982740 1 -- 172.21.15.49:0/2541208261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 msgr2=0x55e385950650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe30a982740 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e385950650 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc005ec0 tx=0x7fe2fc0436c0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe30a982740 1 -- 172.21.15.49:0/2541208261 shutdown_connections 2024-03-18T00:52:56.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe30a982740 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55e3859a5c40 0x55e3859a8060 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe30a982740 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e38592e3b0 0x55e385950650 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe30a982740 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e385938200 0x55e385951660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe2e97fa640 0 client.25739 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:52:56.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe30a982740 1 --2- 172.21.15.49:0/2541208261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e385929240 0x55e385950110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe30a982740 1 -- 172.21.15.49:0/2541208261 >> 172.21.15.49:0/2541208261 conn(0x55e385923dc0 msgr2=0x55e38592a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.888+0000 7fe30a982740 1 -- 172.21.15.49:0/2541208261 shutdown_connections 2024-03-18T00:52:56.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.889+0000 7fe30a982740 1 -- 172.21.15.49:0/2541208261 wait complete. 2024-03-18T00:52:56.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.889+0000 7fe30a982740 20 client.25739 trim_cache size 0 max 16384 2024-03-18T00:52:56.891 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:56.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 -- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2ec0a28c0 msgr2=0x7fe2ec0a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2ec0a28c0 0x7fe2ec0a4d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe2f000c000 tx=0x7fe2f001b000 comp rx=0 tx=0).stop 2024-03-18T00:52:56.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 -- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3001087e0 msgr2=0x7fe300115090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:56.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3001087e0 0x7fe300115090 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fe2f800ba50 tx=0x7fe2f800bf10 comp rx=0 tx=0).stop 2024-03-18T00:52:56.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 -- 172.21.15.49:0/1649139392 shutdown_connections 2024-03-18T00:52:56.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2ec0a28c0 0x7fe2ec0a4d80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe30010db00 0x7fe3001155d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3001b4a20 0x7fe3001b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 --2- 172.21.15.49:0/1649139392 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe3001087e0 0x7fe300115090 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:56.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 -- 172.21.15.49:0/1649139392 >> 172.21.15.49:0/1649139392 conn(0x7fe30006e640 msgr2=0x7fe300108c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:56.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 -- 172.21.15.49:0/1649139392 shutdown_connections 2024-03-18T00:52:56.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:56.890+0000 7fe307cbf640 1 -- 172.21.15.49:0/1649139392 wait complete. 2024-03-18T00:52:56.913 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:52:57.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:56 smithi107 ceph-mon[36270]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 818 MiB/s wr, 1.35k op/s 2024-03-18T00:52:57.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:56 smithi106 ceph-mon[35899]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 818 MiB/s wr, 1.35k op/s 2024-03-18T00:52:57.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.261+0000 7f7d83fff640 1 -- 172.21.15.49:0/1124131037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d84109890 msgr2=0x7f7d84110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:57.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.261+0000 7f7d83fff640 1 --2- 172.21.15.49:0/1124131037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d84109890 0x7f7d84110120 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009ca0 tx=0x7f7d7c02c120 comp rx=0 tx=0).stop 2024-03-18T00:52:57.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.262+0000 7f7d83fff640 1 -- 172.21.15.49:0/1124131037 shutdown_connections 2024-03-18T00:52:57.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.262+0000 7f7d83fff640 1 --2- 172.21.15.49:0/1124131037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d84109890 0x7f7d84110120 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.262+0000 7f7d83fff640 1 --2- 172.21.15.49:0/1124131037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d84108ed0 0x7f7d84109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.262+0000 7f7d83fff640 1 --2- 172.21.15.49:0/1124131037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d84107ce0 0x7f7d84108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.262+0000 7f7d83fff640 1 -- 172.21.15.49:0/1124131037 >> 172.21.15.49:0/1124131037 conn(0x7f7d84076c80 msgr2=0x7f7d840790c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:57.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.276+0000 7f7d83fff640 1 -- 172.21.15.49:0/1124131037 shutdown_connections 2024-03-18T00:52:57.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.279+0000 7f7d83fff640 1 -- 172.21.15.49:0/1124131037 wait complete. 2024-03-18T00:52:57.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.280+0000 7f7d83fff640 1 Processor -- start 2024-03-18T00:52:57.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.284+0000 7f7d83fff640 1 -- start start 2024-03-18T00:52:57.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.284+0000 7f7d83fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d84107ce0 0x7f7d8407fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.284+0000 7f7d83fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d84108ed0 0x7f7d84080350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.284+0000 7f7d83fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d84109890 0x7f7d84081390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.284+0000 7f7d83fff640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8407a640 con 0x7f7d84108ed0 2024-03-18T00:52:57.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.284+0000 7f7d83fff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8407a4c0 con 0x7f7d84107ce0 2024-03-18T00:52:57.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.284+0000 7f7d83fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8407a7c0 con 0x7f7d84109890 2024-03-18T00:52:57.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.288+0000 7f7d827fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d84108ed0 0x7f7d84080350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.288+0000 7f7d827fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d84108ed0 0x7f7d84080350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40616/0 (socket says 172.21.15.49:40616) 2024-03-18T00:52:57.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.288+0000 7f7d827fc640 1 -- 172.21.15.49:0/241773508 learned_addr learned my addr 172.21.15.49:0/241773508 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:57.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.289+0000 7f7d82ffd640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d84107ce0 0x7f7d8407fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.290+0000 7f7d827fc640 1 -- 172.21.15.49:0/241773508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d84109890 msgr2=0x7f7d84081390 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:57.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.290+0000 7f7d827fc640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d84109890 0x7f7d84081390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.290+0000 7f7d827fc640 1 -- 172.21.15.49:0/241773508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d84107ce0 msgr2=0x7f7d8407fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:57.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.290+0000 7f7d827fc640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d84107ce0 0x7f7d8407fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.290+0000 7f7d827fc640 1 -- 172.21.15.49:0/241773508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d84081a00 con 0x7f7d84108ed0 2024-03-18T00:52:57.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.290+0000 7f7d827fc640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d84108ed0 0x7f7d84080350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400d950 tx=0x7f7d7400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:57.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.292+0000 7f7d6bfff640 1 -- 172.21.15.49:0/241773508 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74017070 con 0x7f7d84108ed0 2024-03-18T00:52:57.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.292+0000 7f7d6bfff640 1 -- 172.21.15.49:0/241773508 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d74009c30 con 0x7f7d84108ed0 2024-03-18T00:52:57.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.292+0000 7f7d6bfff640 1 -- 172.21.15.49:0/241773508 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7400f6b0 con 0x7f7d84108ed0 2024-03-18T00:52:57.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.294+0000 7f7d83fff640 1 -- 172.21.15.49:0/241773508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d84081cf0 con 0x7f7d84108ed0 2024-03-18T00:52:57.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.294+0000 7f7d83fff640 1 -- 172.21.15.49:0/241773508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d840821f0 con 0x7f7d84108ed0 2024-03-18T00:52:57.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.299+0000 7f7d8bbfd740 1 Processor -- start 2024-03-18T00:52:57.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.299+0000 7f7d8bbfd740 1 -- start start 2024-03-18T00:52:57.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.299+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070ef8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.299+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070efdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.300+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f00a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.300+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a070e5d490 con 0x55a070efe5a0 2024-03-18T00:52:57.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.300+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a070e05c90 con 0x55a070ef8a40 2024-03-18T00:52:57.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.300+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a070e59530 con 0x55a070ef9430 2024-03-18T00:52:57.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.303+0000 7f7d82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070efdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.303+0000 7f7d82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070efdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:34780/0 (socket says 172.21.15.49:34780) 2024-03-18T00:52:57.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.303+0000 7f7d82ffd640 1 -- 172.21.15.49:0/865603459 learned_addr learned my addr 172.21.15.49:0/865603459 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:57.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.304+0000 7f7d6bfff640 1 -- 172.21.15.49:0/241773508 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7d740109d0 con 0x7f7d84108ed0 2024-03-18T00:52:57.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.306+0000 7f7d6bfff640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d6c0a2990 0x7f7d6c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.306+0000 7f7d6bfff640 1 -- 172.21.15.49:0/241773508 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7d740ce610 con 0x7f7d84108ed0 2024-03-18T00:52:57.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.307+0000 7f7d827fc640 1 --2- 172.21.15.49:0/865603459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f00a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.308+0000 7f7d837fe640 1 --2- 172.21.15.49:0/865603459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070ef8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.309+0000 7f7d82ffd640 1 -- 172.21.15.49:0/865603459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 msgr2=0x55a070ef8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:57.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.309+0000 7f7d82ffd640 1 --2- 172.21.15.49:0/865603459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070ef8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.309+0000 7f7d82ffd640 1 -- 172.21.15.49:0/865603459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 msgr2=0x55a070f00a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:57.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.309+0000 7f7d82ffd640 1 --2- 172.21.15.49:0/865603459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f00a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.309+0000 7f7d82ffd640 1 -- 172.21.15.49:0/865603459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a070ec0500 con 0x55a070ef9430 2024-03-18T00:52:57.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.314+0000 7f7d837fe640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d6c0a2990 0x7f7d6c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.315+0000 7f7d82ffd640 1 --2- 172.21.15.49:0/865603459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070efdf30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009b70 tx=0x7f7d78009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fff546687ff8da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:57.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.324+0000 7f7d697fa640 1 -- 172.21.15.49:0/865603459 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78016070 con 0x55a070ef9430 2024-03-18T00:52:57.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.324+0000 7f7d697fa640 1 -- 172.21.15.49:0/865603459 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d78005500 con 0x55a070ef9430 2024-03-18T00:52:57.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.324+0000 7f7d697fa640 1 -- 172.21.15.49:0/865603459 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d78011470 con 0x55a070ef9430 2024-03-18T00:52:57.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.329+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/865603459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 msgr2=0x55a070efdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:57.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.329+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/865603459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070efdf30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009b70 tx=0x7f7d78009f70 comp rx=0 tx=0).stop 2024-03-18T00:52:57.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.331+0000 7f7d837fe640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d6c0a2990 0x7f7d6c0a4e50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55a070ef9c90 tx=0x7f7d7c02c120 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:57.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.334+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/865603459 shutdown_connections 2024-03-18T00:52:57.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.334+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/865603459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f00a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.334+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/865603459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070efdf30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.334+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/865603459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070ef8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.334+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/865603459 >> 172.21.15.49:0/865603459 conn(0x55a070ef3fb0 msgr2=0x55a070ef43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:57.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.342+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/865603459 shutdown_connections 2024-03-18T00:52:57.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.346+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/865603459 wait complete. 2024-03-18T00:52:57.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.347+0000 7f7d8bbfd740 1 -- start start 2024-03-18T00:52:57.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.347+0000 7f7d8bbfd740 1 Processor -- start 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.351+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef8a40 0x55a070f03050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.351+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef9430 0x55a070f03590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.351+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f0a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.351+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a070f01550 con 0x55a070efe5a0 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.351+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a070e19230 con 0x55a070ef9430 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.351+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a070f016d0 con 0x55a070ef8a40 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.351+0000 7f7d8bbfd740 10 client.0 _ll_register_callbacks cb 0x55a070e55010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.353+0000 7f7d837fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f0a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.353+0000 7f7d837fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f0a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40632/0 (socket says 172.21.15.49:40632) 2024-03-18T00:52:57.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.353+0000 7f7d837fe640 1 -- 172.21.15.49:0/3178948859 learned_addr learned my addr 172.21.15.49:0/3178948859 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:57.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.354+0000 7f7d827fc640 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef9430 0x55a070f03590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.355+0000 7f7d837fe640 1 -- 172.21.15.49:0/3178948859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef8a40 msgr2=0x55a070f03050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:57.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.355+0000 7f7d837fe640 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef8a40 0x55a070f03050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.355+0000 7f7d837fe640 1 -- 172.21.15.49:0/3178948859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef9430 msgr2=0x55a070f03590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:57.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.355+0000 7f7d837fe640 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef9430 0x55a070f03590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.355+0000 7f7d837fe640 1 -- 172.21.15.49:0/3178948859 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a070e28100 con 0x55a070efe5a0 2024-03-18T00:52:57.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.356+0000 7f7d837fe640 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f0a560 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c036000 tx=0x7f7d7c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632c5743db7a71fb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:57.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.357+0000 7f7d66ff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:57.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.363+0000 7f7d66ff5640 1 -- 172.21.15.49:0/3178948859 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c04f060 con 0x55a070efe5a0 2024-03-18T00:52:57.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.363+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3178948859 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d78005500 con 0x55a070efe5a0 2024-03-18T00:52:57.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.363+0000 7f7d66ff5640 1 -- 172.21.15.49:0/3178948859 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7d7c004240 con 0x55a070efe5a0 2024-03-18T00:52:57.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.368+0000 7f7d66ff5640 1 -- 172.21.15.49:0/3178948859 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7d7c039070 con 0x55a070efe5a0 2024-03-18T00:52:57.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.368+0000 7f7d8bbfd740 10 client.16125 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:57.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.368+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3178948859 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a070b713f0 con 0x55a070efe5a0 2024-03-18T00:52:57.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.369+0000 7f7d66ff5640 1 -- 172.21.15.49:0/3178948859 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d78005500 con 0x55a070efe5a0 2024-03-18T00:52:57.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.370+0000 7f7d66ff5640 1 -- 172.21.15.49:0/3178948859 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f7d7c04a410 con 0x55a070efe5a0 2024-03-18T00:52:57.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.370+0000 7f7d8bbfd740 10 client.16125 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:57.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.370+0000 7f7d8bbfd740 10 client.16125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:57.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.370+0000 7f7d8bbfd740 20 client.16125 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:57.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.370+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a070f0eee0 0x55a070f11350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.370+0000 7f7d8bbfd740 4 client.16125 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:57.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.370+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3178948859 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a070e6a9d0 con 0x55a070f0eee0 2024-03-18T00:52:57.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.372+0000 7f7d82ffd640 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a070f0eee0 0x55a070f11350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.385+0000 7f7d82ffd640 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a070f0eee0 0x55a070f11350 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4664f4e2027fbe22 server_cookie=c38491c1daeb7f79 in_seq=0 out_seq=0 2024-03-18T00:52:57.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.385+0000 7f7d66ff5640 10 client.16125 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.397+0000 7f7d66ff5640 1 -- 172.21.15.49:0/3178948859 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a070e6a9d0 con 0x55a070f0eee0 2024-03-18T00:52:57.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.397+0000 7f7d66ff5640 10 client.16125 handle_command_reply: tid=0 2024-03-18T00:52:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.397+0000 7f7d8bbfd740 1 client.16125 shutdown 2024-03-18T00:52:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.397+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3178948859 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 msgr2=0x55a070f0a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.397+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f0a560 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c036000 tx=0x7f7d7c0095c0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.398+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3178948859 shutdown_connections 2024-03-18T00:52:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.398+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a070f0eee0 0x55a070f11350 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.398+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef9430 0x55a070f03590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.398+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f0a560 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.398+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3178948859 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef8a40 0x55a070f03050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.398+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3178948859 >> 172.21.15.49:0/3178948859 conn(0x55a070ef3fb0 msgr2=0x55a070ef3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.398+0000 7f7d66ff5640 0 client.16125 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.398+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3178948859 shutdown_connections 2024-03-18T00:52:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.398+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3178948859 wait complete. 2024-03-18T00:52:57.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.399+0000 7f7d8bbfd740 20 client.16125 trim_cache size 0 max 16384 2024-03-18T00:52:57.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.427+0000 7f7d8bbfd740 1 Processor -- start 2024-03-18T00:52:57.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.427+0000 7f7d8bbfd740 1 -- start start 2024-03-18T00:52:57.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.428+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070f10ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.428+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070f0a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.428+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070ef28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.428+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841c01d0 con 0x55a070efe5a0 2024-03-18T00:52:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.428+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841c1d10 con 0x55a070ef8a40 2024-03-18T00:52:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.428+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841c0870 con 0x55a070ef9430 2024-03-18T00:52:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.428+0000 7f7d837fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070ef28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.428+0000 7f7d837fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070ef28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40636/0 (socket says 172.21.15.49:40636) 2024-03-18T00:52:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.428+0000 7f7d837fe640 1 -- 172.21.15.49:0/3772147589 learned_addr learned my addr 172.21.15.49:0/3772147589 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.429+0000 7f7d837fe640 1 -- 172.21.15.49:0/3772147589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 msgr2=0x55a070f0a520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.429+0000 7f7d837fe640 1 --2- 172.21.15.49:0/3772147589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070f0a520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.429+0000 7f7d837fe640 1 -- 172.21.15.49:0/3772147589 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 msgr2=0x55a070f10ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.429+0000 7f7d837fe640 1 --2- 172.21.15.49:0/3772147589 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070f10ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.429+0000 7f7d837fe640 1 -- 172.21.15.49:0/3772147589 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a070f116c0 con 0x55a070efe5a0 2024-03-18T00:52:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.429+0000 7f7d837fe640 1 --2- 172.21.15.49:0/3772147589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070ef28b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009ca0 tx=0x7f7d7c04aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4650cd5bac727dac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.430+0000 7f7d68ff9640 1 -- 172.21.15.49:0/3772147589 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c04f070 con 0x55a070efe5a0 2024-03-18T00:52:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.430+0000 7f7d68ff9640 1 -- 172.21.15.49:0/3772147589 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d7c045050 con 0x55a070efe5a0 2024-03-18T00:52:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.430+0000 7f7d68ff9640 1 -- 172.21.15.49:0/3772147589 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c034920 con 0x55a070efe5a0 2024-03-18T00:52:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.431+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3772147589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 msgr2=0x55a070ef28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.431+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3772147589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070ef28b0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c009ca0 tx=0x7f7d7c04aaa0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.431+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3772147589 shutdown_connections 2024-03-18T00:52:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.432+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3772147589 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070ef28b0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.432+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3772147589 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070f0a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.432+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/3772147589 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070f10ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.432+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3772147589 >> 172.21.15.49:0/3772147589 conn(0x55a070ef3fb0 msgr2=0x55a070efaa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.432+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3772147589 shutdown_connections 2024-03-18T00:52:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.432+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/3772147589 wait complete. 2024-03-18T00:52:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.432+0000 7f7d8bbfd740 1 -- start start 2024-03-18T00:52:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.433+0000 7f7d8bbfd740 1 Processor -- start 2024-03-18T00:52:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.433+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070f07810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.433+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070f07d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.433+0000 7f7d8bbfd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f08d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.434+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841c1610 con 0x55a070efe5a0 2024-03-18T00:52:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.434+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841c0f40 con 0x55a070ef8a40 2024-03-18T00:52:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.434+0000 7f7d8bbfd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84071a10 con 0x55a070ef9430 2024-03-18T00:52:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.434+0000 7f7d8bbfd740 10 client.0 _ll_register_callbacks cb 0x7f7d7c041040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.434+0000 7f7d837fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f08d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.434+0000 7f7d837fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f08d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40640/0 (socket says 172.21.15.49:40640) 2024-03-18T00:52:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.434+0000 7f7d837fe640 1 -- 172.21.15.49:0/2237472309 learned_addr learned my addr 172.21.15.49:0/2237472309 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:57.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.435+0000 7f7d837fe640 1 -- 172.21.15.49:0/2237472309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 msgr2=0x55a070f07d50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.435+0000 7f7d837fe640 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070f07d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.435+0000 7f7d837fe640 1 -- 172.21.15.49:0/2237472309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 msgr2=0x55a070f07810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.435+0000 7f7d837fe640 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070f07810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.435+0000 7f7d837fe640 1 -- 172.21.15.49:0/2237472309 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a070c3b7d0 con 0x55a070efe5a0 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.435+0000 7f7d837fe640 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f08d60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c003ec0 tx=0x7f7d7c04bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179880f0dc10f285 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.435+0000 7f7d66ff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.436+0000 7f7d66ff5640 1 -- 172.21.15.49:0/2237472309 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c04f070 con 0x55a070efe5a0 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.436+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/2237472309 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d7c045050 con 0x55a070efe5a0 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.436+0000 7f7d66ff5640 1 -- 172.21.15.49:0/2237472309 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7d7c034850 con 0x55a070efe5a0 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.436+0000 7f7d66ff5640 1 -- 172.21.15.49:0/2237472309 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7d7c06c050 con 0x55a070efe5a0 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.436+0000 7f7d8bbfd740 10 client.16131 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.436+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/2237472309 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a070f0a480 con 0x55a070efe5a0 2024-03-18T00:52:57.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.437+0000 7f7d66ff5640 1 -- 172.21.15.49:0/2237472309 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d7c045050 con 0x55a070efe5a0 2024-03-18T00:52:57.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.437+0000 7f7d66ff5640 1 -- 172.21.15.49:0/2237472309 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f7d7c056ac0 con 0x55a070efe5a0 2024-03-18T00:52:57.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.437+0000 7f7d8bbfd740 10 client.16131 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:57.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.437+0000 7f7d8bbfd740 10 client.16131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:57.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.437+0000 7f7d8bbfd740 20 client.16131 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:57.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.438+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a070f75690 0x55a070f77ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:57.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.438+0000 7f7d8bbfd740 4 client.16131 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:52:57.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.438+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/2237472309 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a070e64d20 con 0x55a070f75690 2024-03-18T00:52:57.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.438+0000 7f7d82ffd640 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a070f75690 0x55a070f77ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:57.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.439+0000 7f7d82ffd640 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a070f75690 0x55a070f77ab0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8449805059ea5a5c server_cookie=d6afcac683979d95 in_seq=0 out_seq=0 2024-03-18T00:52:57.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.439+0000 7f7d66ff5640 10 client.16131 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.444+0000 7f7d66ff5640 1 -- 172.21.15.49:0/2237472309 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a070e64d20 con 0x55a070f75690 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.444+0000 7f7d66ff5640 10 client.16131 handle_command_reply: tid=0 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.444+0000 7f7d8bbfd740 1 client.16131 shutdown 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.444+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/2237472309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 msgr2=0x55a070f08d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.444+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f08d60 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c003ec0 tx=0x7f7d7c04bed0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/2237472309 shutdown_connections 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a070f75690 0x55a070f77ab0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a070efe5a0 0x55a070f08d60 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a070ef9430 0x55a070f07d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d8bbfd740 1 --2- 172.21.15.49:0/2237472309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a070ef8a40 0x55a070f07810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/2237472309 >> 172.21.15.49:0/2237472309 conn(0x55a070ef3fb0 msgr2=0x55a070ef28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:57.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d66ff5640 0 client.16131 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:57.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/2237472309 shutdown_connections 2024-03-18T00:52:57.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d8bbfd740 1 -- 172.21.15.49:0/2237472309 wait complete. 2024-03-18T00:52:57.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.445+0000 7f7d8bbfd740 20 client.16131 trim_cache size 0 max 16384 2024-03-18T00:52:57.449 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:52:57.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.446+0000 7f7d83fff640 1 -- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d6c0a2990 msgr2=0x7f7d6c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:57.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.447+0000 7f7d83fff640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d6c0a2990 0x7f7d6c0a4e50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55a070ef9c90 tx=0x7f7d7c02c120 comp rx=0 tx=0).stop 2024-03-18T00:52:57.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.447+0000 7f7d83fff640 1 -- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d84108ed0 msgr2=0x7f7d84080350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:57.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.447+0000 7f7d83fff640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d84108ed0 0x7f7d84080350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400d950 tx=0x7f7d7400de10 comp rx=0 tx=0).stop 2024-03-18T00:52:57.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.447+0000 7f7d83fff640 1 -- 172.21.15.49:0/241773508 shutdown_connections 2024-03-18T00:52:57.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.447+0000 7f7d83fff640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7d6c0a2990 0x7f7d6c0a4e50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.447+0000 7f7d83fff640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7d84109890 0x7f7d84081390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.447+0000 7f7d83fff640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7d84108ed0 0x7f7d84080350 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.447+0000 7f7d83fff640 1 --2- 172.21.15.49:0/241773508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7d84107ce0 0x7f7d8407fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:57.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.447+0000 7f7d83fff640 1 -- 172.21.15.49:0/241773508 >> 172.21.15.49:0/241773508 conn(0x7f7d84076c80 msgr2=0x7f7d8410c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:57.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.448+0000 7f7d83fff640 1 -- 172.21.15.49:0/241773508 shutdown_connections 2024-03-18T00:52:57.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:57.448+0000 7f7d83fff640 1 -- 172.21.15.49:0/241773508 wait complete. 2024-03-18T00:52:58.482 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:52:58.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:52:58.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:52:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.825+0000 7fa452e62640 1 -- 172.21.15.49:0/1844054794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa44c1087e0 msgr2=0x7fa44c10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.825+0000 7fa452e62640 1 --2- 172.21.15.49:0/1844054794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa44c1087e0 0x7fa44c10ac10 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009900 tx=0x7fa43c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:52:58.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.832+0000 7fa452e62640 1 -- 172.21.15.49:0/1844054794 shutdown_connections 2024-03-18T00:52:58.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.832+0000 7fa452e62640 1 --2- 172.21.15.49:0/1844054794 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa44c10db00 0x7fa44c10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.832+0000 7fa452e62640 1 --2- 172.21.15.49:0/1844054794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa44c10b150 0x7fa44c10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.832+0000 7fa452e62640 1 --2- 172.21.15.49:0/1844054794 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa44c1087e0 0x7fa44c10ac10 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.832+0000 7fa452e62640 1 -- 172.21.15.49:0/1844054794 >> 172.21.15.49:0/1844054794 conn(0x7fa44c06e640 msgr2=0x7fa44c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:58.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.835+0000 7fa452e62640 1 -- 172.21.15.49:0/1844054794 shutdown_connections 2024-03-18T00:52:58.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.838+0000 7fa452e62640 1 -- 172.21.15.49:0/1844054794 wait complete. 2024-03-18T00:52:58.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.839+0000 7fa452e62640 1 Processor -- start 2024-03-18T00:52:58.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.839+0000 7fa452e62640 1 -- start start 2024-03-18T00:52:58.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.840+0000 7fa452e62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa44c1087e0 0x7fa44c1a6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.840+0000 7fa452e62640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa44c10b150 0x7fa44c1a6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.840+0000 7fa452e62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa44c10db00 0x7fa44c1adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.840+0000 7fa452e62640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa44c11c320 con 0x7fa44c10db00 2024-03-18T00:52:58.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.840+0000 7fa452e62640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa44c11c1a0 con 0x7fa44c10b150 2024-03-18T00:52:58.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.840+0000 7fa452e62640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa44c11c4a0 con 0x7fa44c1087e0 2024-03-18T00:52:58.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.840+0000 7fa450bfe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa44c1087e0 0x7fa44c1a6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.840+0000 7fa450bfe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa44c1087e0 0x7fa44c1a6a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46190/0 (socket says 172.21.15.49:46190) 2024-03-18T00:52:58.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.840+0000 7fa450bfe640 1 -- 172.21.15.49:0/3218755138 learned_addr learned my addr 172.21.15.49:0/3218755138 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:58.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.841+0000 7fa4513ff640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa44c10db00 0x7fa44c1adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.841+0000 7fa450bfe640 1 -- 172.21.15.49:0/3218755138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa44c10b150 msgr2=0x7fa44c1a6f50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:58.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.841+0000 7fa450bfe640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa44c10b150 0x7fa44c1a6f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.841+0000 7fa450bfe640 1 -- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa44c10db00 msgr2=0x7fa44c1adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:58.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.841+0000 7fa450bfe640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa44c10db00 0x7fa44c1adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.841+0000 7fa450bfe640 1 -- 172.21.15.49:0/3218755138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44c1ae5e0 con 0x7fa44c1087e0 2024-03-18T00:52:58.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.842+0000 7fa450bfe640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa44c1087e0 0x7fa44c1a6a10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa43c000c00 tx=0x7fa43c02ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:58.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.843+0000 7fa441ffb640 1 -- 172.21.15.49:0/3218755138 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c041020 con 0x7fa44c1087e0 2024-03-18T00:52:58.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.843+0000 7fa452e62640 1 -- 172.21.15.49:0/3218755138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa44c1ae870 con 0x7fa44c1087e0 2024-03-18T00:52:58.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.844+0000 7fa452e62640 1 -- 172.21.15.49:0/3218755138 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa44c1aed60 con 0x7fa44c1087e0 2024-03-18T00:52:58.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.845+0000 7fa441ffb640 1 -- 172.21.15.49:0/3218755138 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa43c02e2e0 con 0x7fa44c1087e0 2024-03-18T00:52:58.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.845+0000 7fa441ffb640 1 -- 172.21.15.49:0/3218755138 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c005190 con 0x7fa44c1087e0 2024-03-18T00:52:58.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.850+0000 7fa441ffb640 1 -- 172.21.15.49:0/3218755138 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa43c044020 con 0x7fa44c1087e0 2024-03-18T00:52:58.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.856+0000 7fa441ffb640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4340a29d0 0x7fa4340a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.857+0000 7fa441ffb640 1 -- 172.21.15.49:0/3218755138 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa43c0f05a0 con 0x7fa44c1087e0 2024-03-18T00:52:58.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.857+0000 7fa455a45740 1 Processor -- start 2024-03-18T00:52:58.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.857+0000 7fa455a45740 1 -- start start 2024-03-18T00:52:58.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.858+0000 7fa455a45740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592ce990 0x55a6592cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.859+0000 7fa455a45740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592cf380 0x55a6592d3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.859+0000 7fa455a45740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592d44f0 0x55a6592d6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.859+0000 7fa455a45740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a6592330d0 con 0x55a6592cf380 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.859+0000 7fa455a45740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a6591dbc70 con 0x55a6592ce990 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.859+0000 7fa455a45740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a65922f170 con 0x55a6592d44f0 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.859+0000 7fa450bfe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592ce990 0x55a6592cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.859+0000 7fa450bfe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592ce990 0x55a6592cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52330/0 (socket says 172.21.15.49:52330) 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.859+0000 7fa450bfe640 1 -- 172.21.15.49:0/2249674667 learned_addr learned my addr 172.21.15.49:0/2249674667 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.860+0000 7fa450bfe640 1 -- 172.21.15.49:0/2249674667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592d44f0 msgr2=0x55a6592d6980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.860+0000 7fa450bfe640 1 --2- 172.21.15.49:0/2249674667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592d44f0 0x55a6592d6980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.860+0000 7fa450bfe640 1 -- 172.21.15.49:0/2249674667 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592cf380 msgr2=0x55a6592d3e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.860+0000 7fa450bfe640 1 --2- 172.21.15.49:0/2249674667 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592cf380 0x55a6592d3e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.860+0000 7fa450bfe640 1 -- 172.21.15.49:0/2249674667 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a65900f7e0 con 0x55a6592ce990 2024-03-18T00:52:58.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.861+0000 7fa450bfe640 1 --2- 172.21.15.49:0/2249674667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592ce990 0x55a6592cedb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa43c02e040 tx=0x7fa43c02bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afa385b5f2b389f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:58.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.863+0000 7fa443fff640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4340a29d0 0x7fa4340a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.865+0000 7fa443fff640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4340a29d0 0x7fa4340a4e90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa4480058c0 tx=0x7fa448005b20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:58.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.866+0000 7fa432ffd640 1 -- 172.21.15.49:0/2249674667 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c04d300 con 0x55a6592ce990 2024-03-18T00:52:58.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.866+0000 7fa432ffd640 1 -- 172.21.15.49:0/2249674667 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa43c0bd5c0 con 0x55a6592ce990 2024-03-18T00:52:58.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.868+0000 7fa455a45740 1 -- 172.21.15.49:0/2249674667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592ce990 msgr2=0x55a6592cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:58.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.868+0000 7fa455a45740 1 --2- 172.21.15.49:0/2249674667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592ce990 0x55a6592cedb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa43c02e040 tx=0x7fa43c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:52:58.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.868+0000 7fa455a45740 1 -- 172.21.15.49:0/2249674667 shutdown_connections 2024-03-18T00:52:58.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.868+0000 7fa455a45740 1 --2- 172.21.15.49:0/2249674667 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592d44f0 0x55a6592d6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.868+0000 7fa455a45740 1 --2- 172.21.15.49:0/2249674667 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592cf380 0x55a6592d3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.868+0000 7fa455a45740 1 --2- 172.21.15.49:0/2249674667 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592ce990 0x55a6592cedb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.868+0000 7fa455a45740 1 -- 172.21.15.49:0/2249674667 >> 172.21.15.49:0/2249674667 conn(0x55a659296c50 msgr2=0x55a6592ca300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:58.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.869+0000 7fa455a45740 1 -- 172.21.15.49:0/2249674667 shutdown_connections 2024-03-18T00:52:58.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.871+0000 7fa455a45740 1 -- 172.21.15.49:0/2249674667 wait complete. 2024-03-18T00:52:58.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.872+0000 7fa455a45740 1 -- start start 2024-03-18T00:52:58.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.872+0000 7fa455a45740 1 Processor -- start 2024-03-18T00:52:58.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:58 smithi049 ceph-mon[32130]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 893 MiB/s wr, 1.62k op/s 2024-03-18T00:52:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.878+0000 7fa455a45740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592cf380 0x55a6592d8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.878+0000 7fa455a45740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592d44f0 0x55a6592d9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.878+0000 7fa455a45740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592de090 0x55a6592e0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.878+0000 7fa455a45740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a6592d7260 con 0x55a6592d44f0 2024-03-18T00:52:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.878+0000 7fa455a45740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a6591ef0d0 con 0x55a6592cf380 2024-03-18T00:52:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.879+0000 7fa455a45740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a6592d73e0 con 0x55a6592de090 2024-03-18T00:52:58.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.879+0000 7fa455a45740 10 client.0 _ll_register_callbacks cb 0x55a6591b8760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:58.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.880+0000 7fa443fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592d44f0 0x55a6592d9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.880+0000 7fa443fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592d44f0 0x55a6592d9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33848/0 (socket says 172.21.15.49:33848) 2024-03-18T00:52:58.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.880+0000 7fa443fff640 1 -- 172.21.15.49:0/1224318658 learned_addr learned my addr 172.21.15.49:0/1224318658 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:58.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.882+0000 7fa450bfe640 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592cf380 0x55a6592d8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.882+0000 7fa4513ff640 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592de090 0x55a6592e0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.883+0000 7fa4513ff640 1 -- 172.21.15.49:0/1224318658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592cf380 msgr2=0x55a6592d8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:58.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.883+0000 7fa4513ff640 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592cf380 0x55a6592d8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.883+0000 7fa4513ff640 1 -- 172.21.15.49:0/1224318658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592d44f0 msgr2=0x55a6592d9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:58.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.883+0000 7fa4513ff640 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592d44f0 0x55a6592d9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.883+0000 7fa4513ff640 1 -- 172.21.15.49:0/1224318658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a65906ff70 con 0x55a6592de090 2024-03-18T00:52:58.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.894+0000 7fa4513ff640 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592de090 0x55a6592e0480 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa44c06b4d0 tx=0x7fa44400cea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ddc06f4392a870d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:58.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.894+0000 7fa42bfff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:52:58.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.895+0000 7fa42bfff640 1 -- 172.21.15.49:0/1224318658 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44401a060 con 0x55a6592de090 2024-03-18T00:52:58.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.895+0000 7fa42bfff640 1 -- 172.21.15.49:0/1224318658 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa4440078c0 con 0x55a6592de090 2024-03-18T00:52:58.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.896+0000 7fa455a45740 1 -- 172.21.15.49:0/1224318658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a659240610 con 0x55a6592de090 2024-03-18T00:52:58.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.899+0000 7fa42bfff640 1 -- 172.21.15.49:0/1224318658 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa444007b00 con 0x55a6592de090 2024-03-18T00:52:58.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.899+0000 7fa455a45740 10 client.25828 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:58.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.899+0000 7fa455a45740 1 -- 172.21.15.49:0/1224318658 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a658f07170 con 0x55a6592de090 2024-03-18T00:52:58.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.901+0000 7fa42bfff640 1 -- 172.21.15.49:0/1224318658 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a659240610 con 0x55a6592de090 2024-03-18T00:52:58.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.901+0000 7fa42bfff640 1 -- 172.21.15.49:0/1224318658 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fa444020ce0 con 0x55a6592de090 2024-03-18T00:52:58.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.901+0000 7fa455a45740 10 client.25828 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:58.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.901+0000 7fa455a45740 10 client.25828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:58.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.901+0000 7fa455a45740 20 client.25828 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:58.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.901+0000 7fa455a45740 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a6592e4af0 0x55a6592e6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.902+0000 7fa455a45740 4 client.25828 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:58.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.902+0000 7fa455a45740 1 -- 172.21.15.49:0/1224318658 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a65923a960 con 0x55a6592e4af0 2024-03-18T00:52:58.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.911+0000 7fa450bfe640 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a6592e4af0 0x55a6592e6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.926+0000 7fa450bfe640 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a6592e4af0 0x55a6592e6f80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c21ffe887ca5e67 server_cookie=f409d527931933 in_seq=0 out_seq=0 2024-03-18T00:52:58.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.927+0000 7fa42bfff640 10 client.25828 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:58.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.935+0000 7fa42bfff640 1 -- 172.21.15.49:0/1224318658 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a65923a960 con 0x55a6592e4af0 2024-03-18T00:52:58.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.935+0000 7fa42bfff640 10 client.25828 handle_command_reply: tid=0 2024-03-18T00:52:58.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.935+0000 7fa455a45740 1 client.25828 shutdown 2024-03-18T00:52:58.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.938+0000 7fa455a45740 1 -- 172.21.15.49:0/1224318658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592de090 msgr2=0x55a6592e0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:58.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.938+0000 7fa455a45740 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592de090 0x55a6592e0480 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa44c06b4d0 tx=0x7fa44400cea0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa455a45740 1 -- 172.21.15.49:0/1224318658 shutdown_connections 2024-03-18T00:52:58.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa455a45740 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a6592e4af0 0x55a6592e6f80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa455a45740 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592de090 0x55a6592e0480 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa455a45740 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a6592d44f0 0x55a6592d9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa455a45740 1 --2- 172.21.15.49:0/1224318658 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592cf380 0x55a6592d8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa455a45740 1 -- 172.21.15.49:0/1224318658 >> 172.21.15.49:0/1224318658 conn(0x55a659296c50 msgr2=0x55a6592c8c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:58.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa42bfff640 0 client.25828 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:58.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa455a45740 1 -- 172.21.15.49:0/1224318658 shutdown_connections 2024-03-18T00:52:58.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa455a45740 1 -- 172.21.15.49:0/1224318658 wait complete. 2024-03-18T00:52:58.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.939+0000 7fa455a45740 20 client.25828 trim_cache size 0 max 16384 2024-03-18T00:52:58.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.977+0000 7fa455a45740 1 Processor -- start 2024-03-18T00:52:58.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.977+0000 7fa455a45740 1 -- start start 2024-03-18T00:52:58.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa455a45740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592ec890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa455a45740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592cf380 0x55a6592d8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa455a45740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592d44f0 0x55a6592d9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa455a45740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa43800e640 con 0x55a659296c50 2024-03-18T00:52:58.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa455a45740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa4380101b0 con 0x55a6592d44f0 2024-03-18T00:52:58.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa455a45740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa43800ed10 con 0x55a6592cf380 2024-03-18T00:52:58.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa450bfe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592ec890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa450bfe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592ec890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33862/0 (socket says 172.21.15.49:33862) 2024-03-18T00:52:58.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa450bfe640 1 -- 172.21.15.49:0/1532016692 learned_addr learned my addr 172.21.15.49:0/1532016692 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:58.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.978+0000 7fa443fff640 1 --2- 172.21.15.49:0/1532016692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592cf380 0x55a6592d8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.979+0000 7fa4513ff640 1 --2- 172.21.15.49:0/1532016692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592d44f0 0x55a6592d9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.979+0000 7fa450bfe640 1 -- 172.21.15.49:0/1532016692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592cf380 msgr2=0x55a6592d8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:58.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.979+0000 7fa450bfe640 1 --2- 172.21.15.49:0/1532016692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592cf380 0x55a6592d8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.979+0000 7fa450bfe640 1 -- 172.21.15.49:0/1532016692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592d44f0 msgr2=0x55a6592d9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:58.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.979+0000 7fa450bfe640 1 --2- 172.21.15.49:0/1532016692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592d44f0 0x55a6592d9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.979+0000 7fa450bfe640 1 -- 172.21.15.49:0/1532016692 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6592e7210 con 0x55a659296c50 2024-03-18T00:52:58.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.980+0000 7fa450bfe640 1 --2- 172.21.15.49:0/1532016692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592ec890 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa43c0bcbc0 tx=0x7fa43c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd571180495cd64 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:58.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.980+0000 7fa4327fc640 1 -- 172.21.15.49:0/1532016692 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c04d300 con 0x55a659296c50 2024-03-18T00:52:58.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.981+0000 7fa4327fc640 1 -- 172.21.15.49:0/1532016692 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa43c0bcec0 con 0x55a659296c50 2024-03-18T00:52:58.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.981+0000 7fa4327fc640 1 -- 172.21.15.49:0/1532016692 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c0f03a0 con 0x55a659296c50 2024-03-18T00:52:58.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.982+0000 7fa455a45740 1 -- 172.21.15.49:0/1532016692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 msgr2=0x55a6592ec890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:58.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.982+0000 7fa455a45740 1 --2- 172.21.15.49:0/1532016692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592ec890 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa43c0bcbc0 tx=0x7fa43c009260 comp rx=0 tx=0).stop 2024-03-18T00:52:58.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.982+0000 7fa455a45740 1 -- 172.21.15.49:0/1532016692 shutdown_connections 2024-03-18T00:52:58.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.982+0000 7fa455a45740 1 --2- 172.21.15.49:0/1532016692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592d44f0 0x55a6592d9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.982+0000 7fa455a45740 1 --2- 172.21.15.49:0/1532016692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592cf380 0x55a6592d8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.982+0000 7fa455a45740 1 --2- 172.21.15.49:0/1532016692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592ec890 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.982+0000 7fa455a45740 1 -- 172.21.15.49:0/1532016692 >> 172.21.15.49:0/1532016692 conn(0x55a6592cef10 msgr2=0x55a6592e6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:58.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.982+0000 7fa455a45740 1 -- 172.21.15.49:0/1532016692 shutdown_connections 2024-03-18T00:52:58.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.982+0000 7fa455a45740 1 -- 172.21.15.49:0/1532016692 wait complete. 2024-03-18T00:52:58.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.983+0000 7fa455a45740 1 -- start start 2024-03-18T00:52:58.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.983+0000 7fa455a45740 1 Processor -- start 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa455a45740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa455a45740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592cf380 0x55a6592f6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa455a45740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592d44f0 0x55a6592f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa455a45740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa43800fab0 con 0x55a659296c50 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa455a45740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa43800f3e0 con 0x55a6592d44f0 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa455a45740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa43800c620 con 0x55a6592cf380 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa450bfe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa455a45740 10 client.0 _ll_register_callbacks cb 0x7fa44400c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa450bfe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592f60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33868/0 (socket says 172.21.15.49:33868) 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa450bfe640 1 -- 172.21.15.49:0/2011761539 learned_addr learned my addr 172.21.15.49:0/2011761539 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:58.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa450bfe640 1 -- 172.21.15.49:0/2011761539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592cf380 msgr2=0x55a6592f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.984+0000 7fa450bfe640 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592cf380 0x55a6592f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa450bfe640 1 -- 172.21.15.49:0/2011761539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592d44f0 msgr2=0x55a6592f75d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa450bfe640 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592d44f0 0x55a6592f75d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa450bfe640 1 -- 172.21.15.49:0/2011761539 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6592f7bc0 con 0x55a659296c50 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa450bfe640 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592f60f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa43c02e080 tx=0x7fa43c0bc920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6219cc480eff91af server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa42bfff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa42bfff640 1 -- 172.21.15.49:0/2011761539 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c0c5660 con 0x55a659296c50 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa455a45740 1 -- 172.21.15.49:0/2011761539 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa43c0bcec0 con 0x55a659296c50 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa42bfff640 1 -- 172.21.15.49:0/2011761539 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa43c0f0330 con 0x55a659296c50 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa42bfff640 1 -- 172.21.15.49:0/2011761539 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa43c0d1020 con 0x55a659296c50 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa455a45740 10 client.16146 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.985+0000 7fa455a45740 1 -- 172.21.15.49:0/2011761539 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55a65935c1b0 con 0x55a659296c50 2024-03-18T00:52:58.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.986+0000 7fa42bfff640 1 -- 172.21.15.49:0/2011761539 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa43c0bcec0 con 0x55a659296c50 2024-03-18T00:52:58.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.986+0000 7fa42bfff640 1 -- 172.21.15.49:0/2011761539 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7fa43c0bbd80 con 0x55a659296c50 2024-03-18T00:52:58.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.986+0000 7fa455a45740 10 client.16146 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:58.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.986+0000 7fa455a45740 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:58.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.986+0000 7fa455a45740 20 client.16146 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:58.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.986+0000 7fa455a45740 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a6592de090 0x55a65935e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:58.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.986+0000 7fa455a45740 4 client.16146 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:52:58.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.986+0000 7fa455a45740 1 -- 172.21.15.49:0/2011761539 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a6591b8be0 con 0x55a6592de090 2024-03-18T00:52:58.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.988+0000 7fa443fff640 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a6592de090 0x55a65935e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:58.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.992+0000 7fa443fff640 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a6592de090 0x55a65935e7d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c48a445e10f73743 server_cookie=1dcead88608b632f in_seq=0 out_seq=0 2024-03-18T00:52:58.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:58.992+0000 7fa42bfff640 10 client.16146 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa42bfff640 1 -- 172.21.15.49:0/2011761539 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a6591b8be0 con 0x55a6592de090 2024-03-18T00:52:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa42bfff640 10 client.16146 handle_command_reply: tid=0 2024-03-18T00:52:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa455a45740 1 client.16146 shutdown 2024-03-18T00:52:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa455a45740 1 -- 172.21.15.49:0/2011761539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 msgr2=0x55a6592f60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa455a45740 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592f60f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa43c02e080 tx=0x7fa43c0bc920 comp rx=0 tx=0).stop 2024-03-18T00:52:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa455a45740 1 -- 172.21.15.49:0/2011761539 shutdown_connections 2024-03-18T00:52:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa455a45740 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55a6592de090 0x55a65935e7d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa455a45740 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a6592d44f0 0x55a6592f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa455a45740 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a6592cf380 0x55a6592f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa455a45740 1 --2- 172.21.15.49:0/2011761539 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a659296c50 0x55a6592f60f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa455a45740 1 -- 172.21.15.49:0/2011761539 >> 172.21.15.49:0/2011761539 conn(0x55a6592cef10 msgr2=0x55a6592d8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:59.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.008+0000 7fa42bfff640 0 client.16146 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:59.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.009+0000 7fa455a45740 1 -- 172.21.15.49:0/2011761539 shutdown_connections 2024-03-18T00:52:59.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.009+0000 7fa455a45740 1 -- 172.21.15.49:0/2011761539 wait complete. 2024-03-18T00:52:59.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.009+0000 7fa455a45740 20 client.16146 trim_cache size 0 max 16384 2024-03-18T00:52:59.012 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:52:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:52:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "fbc54382-ae06-4f06-b963-12c19359aa4d", 2024-03-18T00:52:59.012 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:52:59.012 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:52:59.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 -- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4340a29d0 msgr2=0x7fa4340a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4340a29d0 0x7fa4340a4e90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa4480058c0 tx=0x7fa448005b20 comp rx=0 tx=0).stop 2024-03-18T00:52:59.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 -- 172.21.15.49:0/3218755138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa44c1087e0 msgr2=0x7fa44c1a6a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa44c1087e0 0x7fa44c1a6a10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa43c000c00 tx=0x7fa43c02ecd0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 -- 172.21.15.49:0/3218755138 shutdown_connections 2024-03-18T00:52:59.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa4340a29d0 0x7fa4340a4e90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa44c10b150 0x7fa44c1a6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa44c10db00 0x7fa44c1adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 --2- 172.21.15.49:0/3218755138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa44c1087e0 0x7fa44c1a6a10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.010+0000 7fa452e62640 1 -- 172.21.15.49:0/3218755138 >> 172.21.15.49:0/3218755138 conn(0x7fa44c06e640 msgr2=0x7fa44c10c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:59.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.012+0000 7fa452e62640 1 -- 172.21.15.49:0/3218755138 shutdown_connections 2024-03-18T00:52:59.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.012+0000 7fa452e62640 1 -- 172.21.15.49:0/3218755138 wait complete. 2024-03-18T00:52:59.042 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:52:59.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:52:59.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:58 smithi107 ceph-mon[36270]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 893 MiB/s wr, 1.62k op/s 2024-03-18T00:52:59.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:58 smithi106 ceph-mon[35899]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 893 MiB/s wr, 1.62k op/s 2024-03-18T00:52:59.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.409+0000 7f63b83d3640 1 -- 172.21.15.49:0/3663891327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b01087e0 msgr2=0x7f63b010ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.409+0000 7f63b83d3640 1 --2- 172.21.15.49:0/3663891327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b01087e0 0x7f63b010ac10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009ca0 tx=0x7f63a802c140 comp rx=0 tx=0).stop 2024-03-18T00:52:59.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.412+0000 7f63b83d3640 1 -- 172.21.15.49:0/3663891327 shutdown_connections 2024-03-18T00:52:59.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.413+0000 7f63b83d3640 1 --2- 172.21.15.49:0/3663891327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63b010db00 0x7f63b010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.413+0000 7f63b83d3640 1 --2- 172.21.15.49:0/3663891327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b010b150 0x7f63b010d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.413+0000 7f63b83d3640 1 --2- 172.21.15.49:0/3663891327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b01087e0 0x7f63b010ac10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.413+0000 7f63b83d3640 1 -- 172.21.15.49:0/3663891327 >> 172.21.15.49:0/3663891327 conn(0x7f63b006e640 msgr2=0x7f63b0070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:59.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.420+0000 7f63b83d3640 1 -- 172.21.15.49:0/3663891327 shutdown_connections 2024-03-18T00:52:59.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.421+0000 7f63b83d3640 1 -- 172.21.15.49:0/3663891327 wait complete. 2024-03-18T00:52:59.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.422+0000 7f63b83d3640 1 Processor -- start 2024-03-18T00:52:59.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.422+0000 7f63b83d3640 1 -- start start 2024-03-18T00:52:59.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.423+0000 7f63b83d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b01087e0 0x7f63b01a6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.423+0000 7f63b83d3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b010b150 0x7f63b01a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.423+0000 7f63b83d3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63b010db00 0x7f63b01adf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.423+0000 7f63b83d3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f63b011c320 con 0x7f63b010b150 2024-03-18T00:52:59.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.423+0000 7f63b83d3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f63b011c1a0 con 0x7f63b010db00 2024-03-18T00:52:59.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.423+0000 7f63b83d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f63b011c4a0 con 0x7f63b01087e0 2024-03-18T00:52:59.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.424+0000 7f63b596e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b010b150 0x7f63b01a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.424+0000 7f63b596e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b010b150 0x7f63b01a6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33876/0 (socket says 172.21.15.49:33876) 2024-03-18T00:52:59.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.424+0000 7f63b596e640 1 -- 172.21.15.49:0/11715408 learned_addr learned my addr 172.21.15.49:0/11715408 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.426+0000 7f63b596e640 1 -- 172.21.15.49:0/11715408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b01087e0 msgr2=0x7f63b01a6a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.426+0000 7f63b596e640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b01087e0 0x7f63b01a6a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.426+0000 7f63b596e640 1 -- 172.21.15.49:0/11715408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63b010db00 msgr2=0x7f63b01adf40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:52:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.426+0000 7f63b596e640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63b010db00 0x7f63b01adf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.426+0000 7f63b596e640 1 -- 172.21.15.49:0/11715408 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b01ae5b0 con 0x7f63b010b150 2024-03-18T00:52:59.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.428+0000 7f63b596e640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b010b150 0x7f63b01a6f70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b450 tx=0x7f63a000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:59.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.431+0000 7f63a77fe640 1 -- 172.21.15.49:0/11715408 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0015020 con 0x7f63b010b150 2024-03-18T00:52:59.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.432+0000 7f63a77fe640 1 -- 172.21.15.49:0/11715408 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a0011d70 con 0x7f63b010b150 2024-03-18T00:52:59.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.432+0000 7f63a77fe640 1 -- 172.21.15.49:0/11715408 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a000f340 con 0x7f63b010b150 2024-03-18T00:52:59.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.433+0000 7f63b83d3640 1 -- 172.21.15.49:0/11715408 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63b01ae8a0 con 0x7f63b010b150 2024-03-18T00:52:59.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.433+0000 7f63b83d3640 1 -- 172.21.15.49:0/11715408 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63b01aeda0 con 0x7f63b010b150 2024-03-18T00:52:59.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.448+0000 7f63a77fe640 1 -- 172.21.15.49:0/11715408 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f63a0018020 con 0x7f63b010b150 2024-03-18T00:52:59.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.450+0000 7f63a77fe640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f639c0a2a10 0x7f639c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.451+0000 7f63a77fe640 1 -- 172.21.15.49:0/11715408 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f63a00ce5a0 con 0x7f63b010b150 2024-03-18T00:52:59.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.454+0000 7f63b616f640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f639c0a2a10 0x7f639c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.456+0000 7f63bafb7740 1 Processor -- start 2024-03-18T00:52:59.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.456+0000 7f63bafb7740 1 -- start start 2024-03-18T00:52:59.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.457+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a2a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.457+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a2f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.457+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a32160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.457+0000 7f63bafb7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ddb598f500 con 0x55ddb5a2ab60 2024-03-18T00:52:59.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.457+0000 7f63bafb7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ddb5937be0 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.457+0000 7f63bafb7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ddb598b5a0 con 0x55ddb5a2a170 2024-03-18T00:52:59.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.458+0000 7f63b596e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a2f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.458+0000 7f63b596e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a2f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33882/0 (socket says 172.21.15.49:33882) 2024-03-18T00:52:59.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.458+0000 7f63b596e640 1 -- 172.21.15.49:0/1212719246 learned_addr learned my addr 172.21.15.49:0/1212719246 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:59.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.458+0000 7f63b616f640 1 --2- 172.21.15.49:0/1212719246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a2a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.460+0000 7f63b596e640 1 -- 172.21.15.49:0/1212719246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 msgr2=0x55ddb5a2a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:59.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.460+0000 7f63b596e640 1 --2- 172.21.15.49:0/1212719246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a2a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.460+0000 7f63b596e640 1 -- 172.21.15.49:0/1212719246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2fcd0 msgr2=0x55ddb5a32160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:59.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.460+0000 7f63b596e640 1 --2- 172.21.15.49:0/1212719246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a32160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.460+0000 7f63b596e640 1 -- 172.21.15.49:0/1212719246 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddb594b4b0 con 0x55ddb5a2ab60 2024-03-18T00:52:59.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.460+0000 7f63b616f640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f639c0a2a10 0x7f639c0a4ed0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63a8004af0 tx=0x7f63a8005eb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:59.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.460+0000 7f63b596e640 1 --2- 172.21.15.49:0/1212719246 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a2f660 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f63a009ae50 tx=0x7f63a009b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff9d103f46b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:59.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.462+0000 7f63a4bfe640 1 -- 172.21.15.49:0/1212719246 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a00a3660 con 0x55ddb5a2ab60 2024-03-18T00:52:59.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.462+0000 7f63a4bfe640 1 -- 172.21.15.49:0/1212719246 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a00991c0 con 0x55ddb5a2ab60 2024-03-18T00:52:59.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.462+0000 7f63a4bfe640 1 -- 172.21.15.49:0/1212719246 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0017b70 con 0x55ddb5a2ab60 2024-03-18T00:52:59.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.463+0000 7f63bafb7740 1 -- 172.21.15.49:0/1212719246 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2ab60 msgr2=0x55ddb5a2f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.463+0000 7f63bafb7740 1 --2- 172.21.15.49:0/1212719246 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a2f660 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f63a009ae50 tx=0x7f63a009b700 comp rx=0 tx=0).stop 2024-03-18T00:52:59.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.463+0000 7f63bafb7740 1 -- 172.21.15.49:0/1212719246 shutdown_connections 2024-03-18T00:52:59.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.463+0000 7f63bafb7740 1 --2- 172.21.15.49:0/1212719246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a32160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.464+0000 7f63bafb7740 1 --2- 172.21.15.49:0/1212719246 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a2f660 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.464+0000 7f63bafb7740 1 --2- 172.21.15.49:0/1212719246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a2a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.464+0000 7f63bafb7740 1 -- 172.21.15.49:0/1212719246 >> 172.21.15.49:0/1212719246 conn(0x55ddb5a256e0 msgr2=0x55ddb5a25ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:59.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.464+0000 7f63bafb7740 1 -- 172.21.15.49:0/1212719246 shutdown_connections 2024-03-18T00:52:59.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.464+0000 7f63bafb7740 1 -- 172.21.15.49:0/1212719246 wait complete. 2024-03-18T00:52:59.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.465+0000 7f63bafb7740 1 -- start start 2024-03-18T00:52:59.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.465+0000 7f63bafb7740 1 Processor -- start 2024-03-18T00:52:59.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.466+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2a170 0x55ddb5a34560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.468+0000 7f63b616f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2a170 0x55ddb5a34560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.468+0000 7f63b616f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2a170 0x55ddb5a34560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33884/0 (socket says 172.21.15.49:33884) 2024-03-18T00:52:59.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.468+0000 7f63b616f640 1 -- 172.21.15.49:0/680130637 learned_addr learned my addr 172.21.15.49:0/680130637 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:59.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.468+0000 7f63bafb7740 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a34aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.468+0000 7f63bafb7740 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a3bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.469+0000 7f63b596e640 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a34aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.469+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ddb5a32c80 con 0x55ddb5a2a170 2024-03-18T00:52:59.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.470+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ddb594b160 con 0x55ddb5a2ab60 2024-03-18T00:52:59.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.470+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ddb5a32e00 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.470+0000 7f63b6970640 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a3bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.471+0000 7f63bafb7740 10 client.0 _ll_register_callbacks cb 0x55ddb592f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:59.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.471+0000 7f63b6970640 1 -- 172.21.15.49:0/680130637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 msgr2=0x55ddb5a34aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:59.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.471+0000 7f63b6970640 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a34aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.471+0000 7f63b6970640 1 -- 172.21.15.49:0/680130637 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2a170 msgr2=0x55ddb5a34560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:59.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.471+0000 7f63b6970640 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2a170 0x55ddb5a34560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.471+0000 7f63b6970640 1 -- 172.21.15.49:0/680130637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddb576d7d0 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.471+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a00991c0 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.472+0000 7f63b6970640 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a3bae0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f63b006b4d0 tx=0x7f63ac00a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57962a9c2937c72e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:59.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.473+0000 7f6399ffb640 10 client.25837 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:52:59.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.475+0000 7f6399ffb640 1 -- 172.21.15.49:0/680130637 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ac005220 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.475+0000 7f6399ffb640 1 -- 172.21.15.49:0/680130637 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f63ac0053c0 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.475+0000 7f6399ffb640 1 -- 172.21.15.49:0/680130637 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f63ac005600 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.477+0000 7f63bafb7740 10 client.25837 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:59.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.477+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55ddb56a33f0 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.483+0000 7f6399ffb640 1 -- 172.21.15.49:0/680130637 <== mon.1 v2:172.21.15.107:3300/0 4 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f63ac004730 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.486+0000 7f63bafb7740 10 client.25837 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:59.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.486+0000 7f63bafb7740 10 client.25837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:59.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.486+0000 7f63bafb7740 20 client.25837 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:59.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.486+0000 7f63bafb7740 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddb5a3fed0 0x55ddb5a42340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.486+0000 7f63bafb7740 4 client.25837 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:52:59.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.487+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddb599ca40 con 0x55ddb5a3fed0 2024-03-18T00:52:59.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.488+0000 7f63b616f640 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddb5a3fed0 0x55ddb5a42340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.492+0000 7f63b616f640 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddb5a3fed0 0x55ddb5a42340 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58efa92158d46a02 server_cookie=243ded032a2fb457 in_seq=0 out_seq=0 2024-03-18T00:52:59.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.492+0000 7f6399ffb640 10 client.25837 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:59.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.498+0000 7f6399ffb640 1 -- 172.21.15.49:0/680130637 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ddb599ca40 con 0x55ddb5a3fed0 2024-03-18T00:52:59.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.498+0000 7f6399ffb640 10 client.25837 handle_command_reply: tid=0 2024-03-18T00:52:59.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.505+0000 7f63bafb7740 1 client.25837 shutdown 2024-03-18T00:52:59.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.506+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2fcd0 msgr2=0x55ddb5a3bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.506+0000 7f63bafb7740 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a3bae0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f63b006b4d0 tx=0x7f63ac00a620 comp rx=0 tx=0).stop 2024-03-18T00:52:59.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.510+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 shutdown_connections 2024-03-18T00:52:59.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.510+0000 7f63bafb7740 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddb5a3fed0 0x55ddb5a42340 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.510+0000 7f63bafb7740 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a34aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.510+0000 7f63bafb7740 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a3bae0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.510+0000 7f63bafb7740 1 --2- 172.21.15.49:0/680130637 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2a170 0x55ddb5a34560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.510+0000 7f6399ffb640 0 client.25837 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:59.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.510+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 >> 172.21.15.49:0/680130637 conn(0x55ddb5a256e0 msgr2=0x55ddb5a24900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:59.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.516+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 shutdown_connections 2024-03-18T00:52:59.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.516+0000 7f63bafb7740 1 -- 172.21.15.49:0/680130637 wait complete. 2024-03-18T00:52:59.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.518+0000 7f63bafb7740 20 client.25837 trim_cache size 0 max 16384 2024-03-18T00:52:59.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.562+0000 7f63bafb7740 1 Processor -- start 2024-03-18T00:52:59.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.562+0000 7f63bafb7740 1 -- start start 2024-03-18T00:52:59.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.563+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a24070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.563+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a3baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.563+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a245b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.563+0000 7f63b616f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a24070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.563+0000 7f63b616f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a24070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46250/0 (socket says 172.21.15.49:46250) 2024-03-18T00:52:59.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.564+0000 7f63b616f640 1 -- 172.21.15.49:0/3899531663 learned_addr learned my addr 172.21.15.49:0/3899531663 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:59.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.564+0000 7f63b596e640 1 --2- 172.21.15.49:0/3899531663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a3baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.564+0000 7f63bafb7740 1 -- 172.21.15.49:0/3899531663 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01b7c10 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.564+0000 7f63bafb7740 1 -- 172.21.15.49:0/3899531663 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01b9780 con 0x55ddb5a2ab60 2024-03-18T00:52:59.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.564+0000 7f63bafb7740 1 -- 172.21.15.49:0/3899531663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01b82e0 con 0x55ddb5a2a170 2024-03-18T00:52:59.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.564+0000 7f63b6970640 1 --2- 172.21.15.49:0/3899531663 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.565+0000 7f63b616f640 1 -- 172.21.15.49:0/3899531663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 msgr2=0x55ddb5a3baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:59.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.565+0000 7f63b616f640 1 --2- 172.21.15.49:0/3899531663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a3baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.565+0000 7f63b616f640 1 -- 172.21.15.49:0/3899531663 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 msgr2=0x55ddb5a245b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:59.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.565+0000 7f63b616f640 1 --2- 172.21.15.49:0/3899531663 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a245b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.565+0000 7f63b616f640 1 -- 172.21.15.49:0/3899531663 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddb56630f0 con 0x55ddb5a2a170 2024-03-18T00:52:59.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.566+0000 7f63b616f640 1 --2- 172.21.15.49:0/3899531663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a24070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f63a8037040 tx=0x7f63a8043000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0ddba0e764fa43e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:59.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.568+0000 7f639bfff640 1 -- 172.21.15.49:0/3899531663 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8054e00 con 0x55ddb5a2a170 2024-03-18T00:52:59.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.569+0000 7f639bfff640 1 -- 172.21.15.49:0/3899531663 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a805a070 con 0x55ddb5a2a170 2024-03-18T00:52:59.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.570+0000 7f63bafb7740 1 -- 172.21.15.49:0/3899531663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 msgr2=0x55ddb5a24070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.570+0000 7f63bafb7740 1 --2- 172.21.15.49:0/3899531663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a24070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f63a8037040 tx=0x7f63a8043000 comp rx=0 tx=0).stop 2024-03-18T00:52:59.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.570+0000 7f63bafb7740 1 -- 172.21.15.49:0/3899531663 shutdown_connections 2024-03-18T00:52:59.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.570+0000 7f63bafb7740 1 --2- 172.21.15.49:0/3899531663 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a3baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.570+0000 7f63bafb7740 1 --2- 172.21.15.49:0/3899531663 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a245b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.570+0000 7f63bafb7740 1 --2- 172.21.15.49:0/3899531663 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2a170 0x55ddb5a24070 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f63a8037040 tx=0x7f63a8043000 comp rx=0 tx=0).stop 2024-03-18T00:52:59.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.570+0000 7f63bafb7740 1 -- 172.21.15.49:0/3899531663 >> 172.21.15.49:0/3899531663 conn(0x55ddb5a256e0 msgr2=0x55ddb5a34560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:59.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.573+0000 7f63bafb7740 1 -- 172.21.15.49:0/3899531663 shutdown_connections 2024-03-18T00:52:59.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.573+0000 7f63bafb7740 1 -- 172.21.15.49:0/3899531663 wait complete. 2024-03-18T00:52:59.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.574+0000 7f63bafb7740 1 -- start start 2024-03-18T00:52:59.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.574+0000 7f63bafb7740 1 Processor -- start 2024-03-18T00:52:59.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.574+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a395a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.574+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a39ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.574+0000 7f63bafb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a3aa30 0x55ddb5a3aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.574+0000 7f63bafb7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01b9080 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.574+0000 7f63bafb7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01b89b0 con 0x55ddb5a3aa30 2024-03-18T00:52:59.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.574+0000 7f63bafb7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f63b01b5bf0 con 0x55ddb5a2ab60 2024-03-18T00:52:59.576 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.574+0000 7f63bafb7740 10 client.0 _ll_register_callbacks cb 0x7f63ac005d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:52:59.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.576+0000 7f63b596e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a39ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.576+0000 7f63b596e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a39ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33906/0 (socket says 172.21.15.49:33906) 2024-03-18T00:52:59.577 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.576+0000 7f63b596e640 1 -- 172.21.15.49:0/2881088718 learned_addr learned my addr 172.21.15.49:0/2881088718 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:52:59.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.576+0000 7f63b6970640 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a3aa30 0x55ddb5a3aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.576+0000 7f63b616f640 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a395a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.578+0000 7f63b596e640 1 -- 172.21.15.49:0/2881088718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2ab60 msgr2=0x55ddb5a395a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:59.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.578+0000 7f63b596e640 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a395a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.578+0000 7f63b596e640 1 -- 172.21.15.49:0/2881088718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a3aa30 msgr2=0x55ddb5a3aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:52:59.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.578+0000 7f63b596e640 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a3aa30 0x55ddb5a3aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.578+0000 7f63b596e640 1 -- 172.21.15.49:0/2881088718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddb59569d0 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.579+0000 7f63b596e640 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a39ae0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f63a00a3130 tx=0x7f63a0009340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb58744e6b1c705 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:52:59.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.580+0000 7f6399ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:52:59.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.580+0000 7f6399ffb640 1 -- 172.21.15.49:0/2881088718 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0017b70 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.580+0000 7f63bafb7740 1 -- 172.21.15.49:0/2881088718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a805a070 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.581+0000 7f6399ffb640 1 -- 172.21.15.49:0/2881088718 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f63a0017d10 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.581+0000 7f6399ffb640 1 -- 172.21.15.49:0/2881088718 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f63a009a310 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.581+0000 7f63bafb7740 10 client.16164 fetch_fsmap learned FSMap version 26 2024-03-18T00:52:59.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.582+0000 7f63bafb7740 1 -- 172.21.15.49:0/2881088718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ddb5aa67e0 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.583+0000 7f6399ffb640 1 -- 172.21.15.49:0/2881088718 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a805a070 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.584+0000 7f6399ffb640 1 -- 172.21.15.49:0/2881088718 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f63a0099d80 con 0x55ddb5a2fcd0 2024-03-18T00:52:59.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.584+0000 7f63bafb7740 10 client.16164 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:52:59.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.584+0000 7f63bafb7740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:52:59.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.584+0000 7f63bafb7740 20 client.16164 populate_metadata read hostname 'smithi049' 2024-03-18T00:52:59.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.584+0000 7f63bafb7740 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddb5aa7150 0x55ddb5aa9570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:52:59.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.584+0000 7f63bafb7740 4 client.16164 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:52:59.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.584+0000 7f63bafb7740 1 -- 172.21.15.49:0/2881088718 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ddb5996d90 con 0x55ddb5aa7150 2024-03-18T00:52:59.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.590+0000 7f63b616f640 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddb5aa7150 0x55ddb5aa9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:52:59.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.604+0000 7f63b616f640 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddb5aa7150 0x55ddb5aa9570 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d161f304a5aef410 server_cookie=1cfb0b9fd7849ba1 in_seq=0 out_seq=0 2024-03-18T00:52:59.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.604+0000 7f6399ffb640 10 client.16164 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:59.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.611+0000 7f6399ffb640 1 -- 172.21.15.49:0/2881088718 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55ddb5996d90 con 0x55ddb5aa7150 2024-03-18T00:52:59.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.611+0000 7f6399ffb640 10 client.16164 handle_command_reply: tid=0 2024-03-18T00:52:59.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.611+0000 7f63bafb7740 1 client.16164 shutdown 2024-03-18T00:52:59.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 -- 172.21.15.49:0/2881088718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 msgr2=0x55ddb5a39ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a39ae0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f63a00a3130 tx=0x7f63a0009340 comp rx=0 tx=0).stop 2024-03-18T00:52:59.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 -- 172.21.15.49:0/2881088718 shutdown_connections 2024-03-18T00:52:59.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddb5aa7150 0x55ddb5aa9570 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddb5a3aa30 0x55ddb5a3aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddb5a2fcd0 0x55ddb5a39ae0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 --2- 172.21.15.49:0/2881088718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddb5a2ab60 0x55ddb5a395a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 -- 172.21.15.49:0/2881088718 >> 172.21.15.49:0/2881088718 conn(0x55ddb5a256e0 msgr2=0x55ddb5a28940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:59.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f6399ffb640 0 client.16164 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:52:59.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 -- 172.21.15.49:0/2881088718 shutdown_connections 2024-03-18T00:52:59.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.613+0000 7f63bafb7740 1 -- 172.21.15.49:0/2881088718 wait complete. 2024-03-18T00:52:59.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.614+0000 7f63bafb7740 20 client.16164 trim_cache size 0 max 16384 2024-03-18T00:52:59.617 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:52:59.617 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-18T00:52:59.618 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:52:59.618 INFO:teuthology.orchestra.run.smithi049.stdout: "fbc54382-ae06-4f06-b963-12c19359aa4d": { 2024-03-18T00:52:59.618 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:52:59.618 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "fbc54382-ae06-4f06-b963-12c19359aa4d", 2024-03-18T00:52:59.618 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:52:59.618 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:52:59.618 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:52:59.618 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:52:59.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.617+0000 7f63b83d3640 1 -- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f639c0a2a10 msgr2=0x7f639c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.617+0000 7f63b83d3640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f639c0a2a10 0x7f639c0a4ed0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63a8004af0 tx=0x7f63a8005eb0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.617+0000 7f63b83d3640 1 -- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b010b150 msgr2=0x7f63b01a6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:52:59.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.618+0000 7f63b83d3640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b010b150 0x7f63b01a6f70 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f63a000b450 tx=0x7f63a000b910 comp rx=0 tx=0).stop 2024-03-18T00:52:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.620+0000 7f63b83d3640 1 -- 172.21.15.49:0/11715408 shutdown_connections 2024-03-18T00:52:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.620+0000 7f63b83d3640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f639c0a2a10 0x7f639c0a4ed0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.620+0000 7f63b83d3640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f63b010db00 0x7f63b01adf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.620+0000 7f63b83d3640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f63b010b150 0x7f63b01a6f70 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.620+0000 7f63b83d3640 1 --2- 172.21.15.49:0/11715408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f63b01087e0 0x7f63b01a6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:52:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.620+0000 7f63b83d3640 1 -- 172.21.15.49:0/11715408 >> 172.21.15.49:0/11715408 conn(0x7f63b006e640 msgr2=0x7f63b010c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:52:59.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.624+0000 7f63b83d3640 1 -- 172.21.15.49:0/11715408 shutdown_connections 2024-03-18T00:52:59.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:52:59.625+0000 7f63b83d3640 1 -- 172.21.15.49:0/11715408 wait complete. 2024-03-18T00:52:59.685 INFO:tasks.cephfs.filesystem:scrub status for tag:fbc54382-ae06-4f06-b963-12c19359aa4d - {'path': '/', 'tag': 'fbc54382-ae06-4f06-b963-12c19359aa4d', 'options': 'recursive,force'} 2024-03-18T00:53:00.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:59 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:53:00.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:59 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:00.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:59 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:53:00.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:59 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:53:00.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:52:59 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:00.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:59 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:53:00.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:59 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:00.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:59 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:53:00.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:59 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:53:00.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:52:59 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:00.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:59 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:53:00.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:59 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:00.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:59 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:53:00.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:59 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:53:00.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:52:59 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:01.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:00 smithi049 ceph-mon[32130]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 949 MiB/s wr, 1.74k op/s 2024-03-18T00:53:01.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:00 smithi107 ceph-mon[36270]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 949 MiB/s wr, 1.74k op/s 2024-03-18T00:53:01.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:00 smithi106 ceph-mon[35899]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 949 MiB/s wr, 1.74k op/s 2024-03-18T00:53:03.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:02 smithi049 ceph-mon[32130]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 819 MiB/s wr, 1.66k op/s 2024-03-18T00:53:03.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:02 smithi107 ceph-mon[36270]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 819 MiB/s wr, 1.66k op/s 2024-03-18T00:53:03.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:02 smithi106 ceph-mon[35899]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 819 MiB/s wr, 1.66k op/s 2024-03-18T00:53:04.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:03 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:04.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:03 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:04.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:03 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:05.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:04 smithi049 ceph-mon[32130]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1015 MiB/s wr, 2.48k op/s 2024-03-18T00:53:05.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:04 smithi107 ceph-mon[36270]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1015 MiB/s wr, 2.48k op/s 2024-03-18T00:53:05.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:04 smithi106 ceph-mon[35899]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1015 MiB/s wr, 2.48k op/s 2024-03-18T00:53:06.128 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:05 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:53:06.128 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:05 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:53:06.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:05 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:53:06.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:05 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:53:06.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:05 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:53:06.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:05 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:53:06.407 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.407+0000 7f82ff0ac640 1 -- 172.21.15.49:0/2959377648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82f810b150 msgr2=0x7f82f810d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:06.407 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.407+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/2959377648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82f810b150 0x7f82f810d5c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009ca0 tx=0x7f82f002c120 comp rx=0 tx=0).stop 2024-03-18T00:53:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.412+0000 7f82ff0ac640 1 -- 172.21.15.49:0/2959377648 shutdown_connections 2024-03-18T00:53:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.412+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/2959377648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82f810db00 0x7f82f810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.411 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.412+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/2959377648 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82f810b150 0x7f82f810d5c0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.412 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.412+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/2959377648 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82f81087e0 0x7f82f810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.412 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.412+0000 7f82ff0ac640 1 -- 172.21.15.49:0/2959377648 >> 172.21.15.49:0/2959377648 conn(0x7f82f806e820 msgr2=0x7f82f8070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:06.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.417+0000 7f82ff0ac640 1 -- 172.21.15.49:0/2959377648 shutdown_connections 2024-03-18T00:53:06.420 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.419+0000 7f82ff0ac640 1 -- 172.21.15.49:0/2959377648 wait complete. 2024-03-18T00:53:06.423 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.420+0000 7f82ff0ac640 1 Processor -- start 2024-03-18T00:53:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.424+0000 7f82ff0ac640 1 -- start start 2024-03-18T00:53:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.425+0000 7f82ff0ac640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82f81087e0 0x7f82f81a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.425+0000 7f82ff0ac640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82f810b150 0x7f82f81a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.425+0000 7f82ff0ac640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82f810db00 0x7f82f81adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.425+0000 7f82ff0ac640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f82f811c3b0 con 0x7f82f810b150 2024-03-18T00:53:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.425+0000 7f82ff0ac640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f82f811c230 con 0x7f82f810db00 2024-03-18T00:53:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.425+0000 7f82ff0ac640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f82f811c530 con 0x7f82f81087e0 2024-03-18T00:53:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.427+0000 7f82fce48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82f81087e0 0x7f82f81a6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.431 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.427+0000 7f82fce48640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82f81087e0 0x7f82f81a6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46266/0 (socket says 172.21.15.49:46266) 2024-03-18T00:53:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.427+0000 7f82fce48640 1 -- 172.21.15.49:0/3608175604 learned_addr learned my addr 172.21.15.49:0/3608175604 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.427+0000 7f82fd649640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82f810db00 0x7f82f81adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.429+0000 7f82fce48640 1 -- 172.21.15.49:0/3608175604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82f810db00 msgr2=0x7f82f81adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.429+0000 7f82fce48640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82f810db00 0x7f82f81adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.429+0000 7f82fce48640 1 -- 172.21.15.49:0/3608175604 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82f810b150 msgr2=0x7f82f81a6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.429+0000 7f82fce48640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82f810b150 0x7f82f81a6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.429+0000 7f82fce48640 1 -- 172.21.15.49:0/3608175604 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f81ae620 con 0x7f82f81087e0 2024-03-18T00:53:06.432 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.430+0000 7f82fce48640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82f81087e0 0x7f82f81a6a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f82e800b450 tx=0x7f82e800b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:06.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.436+0000 7f82f5ffb640 1 -- 172.21.15.49:0/3608175604 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8015020 con 0x7f82f81087e0 2024-03-18T00:53:06.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.436+0000 7f82f5ffb640 1 -- 172.21.15.49:0/3608175604 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e8011d70 con 0x7f82f81087e0 2024-03-18T00:53:06.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.436+0000 7f82f5ffb640 1 -- 172.21.15.49:0/3608175604 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e800f340 con 0x7f82f81087e0 2024-03-18T00:53:06.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.437+0000 7f82ff0ac640 1 -- 172.21.15.49:0/3608175604 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f81ae910 con 0x7f82f81087e0 2024-03-18T00:53:06.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.438+0000 7f82ff0ac640 1 -- 172.21.15.49:0/3608175604 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f81aee90 con 0x7f82f81087e0 2024-03-18T00:53:06.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.448+0000 7f82f5ffb640 1 -- 172.21.15.49:0/3608175604 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f82e8018020 con 0x7f82f81087e0 2024-03-18T00:53:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.449+0000 7f82f5ffb640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82e00a2ae0 0x7f82e00a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.450+0000 7f82f5ffb640 1 -- 172.21.15.49:0/3608175604 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f82e80ce9f0 con 0x7f82f81087e0 2024-03-18T00:53:06.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.451+0000 7f82f7fff640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82e00a2ae0 0x7f82e00a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.455 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.454+0000 7f82f7fff640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82e00a2ae0 0x7f82e00a4fa0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f82f0004af0 tx=0x7f82f0032340 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.458+0000 7f8301d73740 1 Processor -- start 2024-03-18T00:53:06.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.458+0000 7f8301d73740 1 -- start start 2024-03-18T00:53:06.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.459+0000 7f8301d73740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816dee7340 0x55816dee7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.459+0000 7f8301d73740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816dee7d30 0x55816deec830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.459+0000 7f8301d73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816deecea0 0x55816deef330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.460+0000 7f8301d73740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55816de4ca40 con 0x55816dee7340 2024-03-18T00:53:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.460+0000 7f8301d73740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55816ddf5920 con 0x55816dee7d30 2024-03-18T00:53:06.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.460+0000 7f8301d73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55816de48ae0 con 0x55816deecea0 2024-03-18T00:53:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.461+0000 7f82fce48640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816dee7340 0x55816dee7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.462 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.461+0000 7f82fce48640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816dee7340 0x55816dee7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33936/0 (socket says 172.21.15.49:33936) 2024-03-18T00:53:06.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.462+0000 7f82fce48640 1 -- 172.21.15.49:0/490358527 learned_addr learned my addr 172.21.15.49:0/490358527 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:06.467 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.466+0000 7f82fd649640 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816deecea0 0x55816deef330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.467 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.466+0000 7f82f7fff640 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816dee7d30 0x55816deec830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.468+0000 7f82fce48640 1 -- 172.21.15.49:0/490358527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816deecea0 msgr2=0x55816deef330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.468+0000 7f82fce48640 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816deecea0 0x55816deef330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.468+0000 7f82fce48640 1 -- 172.21.15.49:0/490358527 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816dee7d30 msgr2=0x55816deec830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.468+0000 7f82fce48640 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816dee7d30 0x55816deec830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.469+0000 7f82fce48640 1 -- 172.21.15.49:0/490358527 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55816de0d460 con 0x55816dee7340 2024-03-18T00:53:06.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.469+0000 7f82f7fff640 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816dee7d30 0x55816deec830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:53:06.486 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.485+0000 7f82fce48640 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816dee7340 0x55816dee7760 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097ce0 tx=0x7f82e809bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1948da1416a7a15e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.501+0000 7f82deffd640 1 -- 172.21.15.49:0/490358527 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e80a3ab0 con 0x55816dee7340 2024-03-18T00:53:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.501+0000 7f82deffd640 1 -- 172.21.15.49:0/490358527 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e8099610 con 0x55816dee7340 2024-03-18T00:53:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.501+0000 7f82deffd640 1 -- 172.21.15.49:0/490358527 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8097b10 con 0x55816dee7340 2024-03-18T00:53:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.502+0000 7f8301d73740 1 -- 172.21.15.49:0/490358527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816dee7340 msgr2=0x55816dee7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.503+0000 7f8301d73740 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816dee7340 0x55816dee7760 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097ce0 tx=0x7f82e809bad0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.503+0000 7f8301d73740 1 -- 172.21.15.49:0/490358527 shutdown_connections 2024-03-18T00:53:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.503+0000 7f8301d73740 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816dee7d30 0x55816deec830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.503+0000 7f8301d73740 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816deecea0 0x55816deef330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.503+0000 7f8301d73740 1 --2- 172.21.15.49:0/490358527 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816dee7340 0x55816dee7760 secure :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097ce0 tx=0x7f82e809bad0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.503+0000 7f8301d73740 1 -- 172.21.15.49:0/490358527 >> 172.21.15.49:0/490358527 conn(0x55816dee28b0 msgr2=0x55816dee2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:06.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.507+0000 7f8301d73740 1 -- 172.21.15.49:0/490358527 shutdown_connections 2024-03-18T00:53:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.508+0000 7f8301d73740 1 -- 172.21.15.49:0/490358527 wait complete. 2024-03-18T00:53:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.509+0000 7f8301d73740 1 -- start start 2024-03-18T00:53:06.509 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.509+0000 7f8301d73740 1 Processor -- start 2024-03-18T00:53:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.510+0000 7f8301d73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816dee7d30 0x55816def18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.510+0000 7f8301d73740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816def1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.510 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.510+0000 7f8301d73740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 0x55816def9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.510+0000 7f8301d73740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55816deefe50 con 0x55816deecea0 2024-03-18T00:53:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.510+0000 7f8301d73740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55816de08d70 con 0x55816def6e30 2024-03-18T00:53:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.510+0000 7f8301d73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55816deeffd0 con 0x55816dee7d30 2024-03-18T00:53:06.511 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.510+0000 7f8301d73740 10 client.0 _ll_register_callbacks cb 0x55816db60a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.514+0000 7f82fd649640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 0x55816def9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.514+0000 7f82fd649640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 0x55816def9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52456/0 (socket says 172.21.15.49:52456) 2024-03-18T00:53:06.514 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.514+0000 7f82fd649640 1 -- 172.21.15.49:0/2687114012 learned_addr learned my addr 172.21.15.49:0/2687114012 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:06.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.514+0000 7f82fce48640 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816dee7d30 0x55816def18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.516+0000 7f82f7fff640 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816def1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.517+0000 7f82fce48640 1 -- 172.21.15.49:0/2687114012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 msgr2=0x55816def9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.517+0000 7f82fce48640 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 0x55816def9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.517+0000 7f82fce48640 1 -- 172.21.15.49:0/2687114012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 msgr2=0x55816def1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.517+0000 7f82fce48640 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816def1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.517+0000 7f82fce48640 1 -- 172.21.15.49:0/2687114012 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55816db61210 con 0x55816dee7d30 2024-03-18T00:53:06.525 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.525+0000 7f82fce48640 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816dee7d30 0x55816def18c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097ce0 tx=0x7f82e8097af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaa41f534f7c49cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:06.525 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.525+0000 7f82d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:53:06.527 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.528+0000 7f82d7fff640 1 -- 172.21.15.49:0/2687114012 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e80a3ab0 con 0x55816dee7d30 2024-03-18T00:53:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.528+0000 7f8301d73740 1 -- 172.21.15.49:0/2687114012 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82e8099610 con 0x55816dee7d30 2024-03-18T00:53:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.528+0000 7f82d7fff640 1 -- 172.21.15.49:0/2687114012 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f82e80a3830 con 0x55816dee7d30 2024-03-18T00:53:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.530+0000 7f82d7fff640 1 -- 172.21.15.49:0/2687114012 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f82e809a760 con 0x55816dee7d30 2024-03-18T00:53:06.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.535+0000 7f8301d73740 10 client.25849 fetch_fsmap learned FSMap version 26 2024-03-18T00:53:06.535 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.535+0000 7f8301d73740 1 -- 172.21.15.49:0/2687114012 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55816dc2d620 con 0x55816dee7d30 2024-03-18T00:53:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.539+0000 7f82d7fff640 1 -- 172.21.15.49:0/2687114012 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82e8099610 con 0x55816dee7d30 2024-03-18T00:53:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.539+0000 7f82d7fff640 1 -- 172.21.15.49:0/2687114012 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f82e809a1d0 con 0x55816dee7d30 2024-03-18T00:53:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.540+0000 7f8301d73740 10 client.25849 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:53:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.540+0000 7f8301d73740 10 client.25849 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.540+0000 7f8301d73740 20 client.25849 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.540+0000 7f8301d73740 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55816defd710 0x55816deffb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.540+0000 7f8301d73740 4 client.25849 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.540+0000 7f8301d73740 1 -- 172.21.15.49:0/2687114012 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55816de59f80 con 0x55816defd710 2024-03-18T00:53:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.541+0000 7f82f7fff640 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55816defd710 0x55816deffb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.543+0000 7f82d7fff640 10 client.25849 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.543+0000 7f82f7fff640 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55816defd710 0x55816deffb00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41249d0610f0bf27 server_cookie=d23e0fcb88091b78 in_seq=0 out_seq=0 2024-03-18T00:53:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.557+0000 7f82d7fff640 1 -- 172.21.15.49:0/2687114012 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55816de59f80 con 0x55816defd710 2024-03-18T00:53:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.557+0000 7f82d7fff640 10 client.25849 handle_command_reply: tid=0 2024-03-18T00:53:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.558+0000 7f8301d73740 1 client.25849 shutdown 2024-03-18T00:53:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.563+0000 7f8301d73740 1 -- 172.21.15.49:0/2687114012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816dee7d30 msgr2=0x55816def18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.563+0000 7f8301d73740 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816dee7d30 0x55816def18c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097ce0 tx=0x7f82e8097af0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.563+0000 7f8301d73740 1 -- 172.21.15.49:0/2687114012 shutdown_connections 2024-03-18T00:53:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.563+0000 7f8301d73740 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55816defd710 0x55816deffb00 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.566 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.563+0000 7f8301d73740 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 0x55816def9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.563+0000 7f8301d73740 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816def1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.563+0000 7f8301d73740 1 --2- 172.21.15.49:0/2687114012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816dee7d30 0x55816def18c0 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097ce0 tx=0x7f82e8097af0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.563+0000 7f8301d73740 1 -- 172.21.15.49:0/2687114012 >> 172.21.15.49:0/2687114012 conn(0x55816dee28b0 msgr2=0x55816dee1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.563+0000 7f82d7fff640 0 client.25849 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.564+0000 7f8301d73740 1 -- 172.21.15.49:0/2687114012 shutdown_connections 2024-03-18T00:53:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.565+0000 7f8301d73740 1 -- 172.21.15.49:0/2687114012 wait complete. 2024-03-18T00:53:06.567 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.565+0000 7f8301d73740 20 client.25849 trim_cache size 0 max 16384 2024-03-18T00:53:06.617 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.616+0000 7f8301d73740 1 Processor -- start 2024-03-18T00:53:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.616+0000 7f8301d73740 1 -- start start 2024-03-18T00:53:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.617+0000 7f8301d73740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816deecea0 0x55816dedda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.617+0000 7f8301d73740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816def6e30 0x55816def14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.617+0000 7f8301d73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 0x55816dee1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.617+0000 7f8301d73740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f82e400e560 con 0x55816def6e30 2024-03-18T00:53:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.617+0000 7f8301d73740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f82e40100d0 con 0x55816deecea0 2024-03-18T00:53:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.617+0000 7f8301d73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f82e400ec30 con 0x55816def1a10 2024-03-18T00:53:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.619+0000 7f82fce48640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816deecea0 0x55816dedda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.619+0000 7f82fce48640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816deecea0 0x55816dedda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52470/0 (socket says 172.21.15.49:52470) 2024-03-18T00:53:06.619 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.619+0000 7f82fce48640 1 -- 172.21.15.49:0/1647561933 learned_addr learned my addr 172.21.15.49:0/1647561933 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:06.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.620+0000 7f82f7fff640 1 --2- 172.21.15.49:0/1647561933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816def6e30 0x55816def14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.621+0000 7f82fd649640 1 --2- 172.21.15.49:0/1647561933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 0x55816dee1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.621+0000 7f82f7fff640 1 -- 172.21.15.49:0/1647561933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 msgr2=0x55816dee1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.621+0000 7f82f7fff640 1 --2- 172.21.15.49:0/1647561933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 0x55816dee1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.621+0000 7f82f7fff640 1 -- 172.21.15.49:0/1647561933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816deecea0 msgr2=0x55816dedda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.621+0000 7f82f7fff640 1 --2- 172.21.15.49:0/1647561933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816deecea0 0x55816dedda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.624 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.621+0000 7f82f7fff640 1 -- 172.21.15.49:0/1647561933 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55816dee3620 con 0x55816def6e30 2024-03-18T00:53:06.627 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.627+0000 7f82f7fff640 1 --2- 172.21.15.49:0/1647561933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816def6e30 0x55816def14d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f82f0032530 tx=0x7f82f003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec6fda5d952fddc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:06.630 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.630+0000 7f82de7fc640 1 -- 172.21.15.49:0/1647561933 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f0039040 con 0x55816def6e30 2024-03-18T00:53:06.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.631+0000 7f82de7fc640 1 -- 172.21.15.49:0/1647561933 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f0003e50 con 0x55816def6e30 2024-03-18T00:53:06.635 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.632+0000 7f82de7fc640 1 -- 172.21.15.49:0/1647561933 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82f004f7f0 con 0x55816def6e30 2024-03-18T00:53:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.636+0000 7f8301d73740 1 -- 172.21.15.49:0/1647561933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816def6e30 msgr2=0x55816def14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.636+0000 7f8301d73740 1 --2- 172.21.15.49:0/1647561933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816def6e30 0x55816def14d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f82f0032530 tx=0x7f82f003e000 comp rx=0 tx=0).stop 2024-03-18T00:53:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.637+0000 7f8301d73740 1 -- 172.21.15.49:0/1647561933 shutdown_connections 2024-03-18T00:53:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.637+0000 7f8301d73740 1 --2- 172.21.15.49:0/1647561933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 0x55816dee1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.637+0000 7f8301d73740 1 --2- 172.21.15.49:0/1647561933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816def6e30 0x55816def14d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.637+0000 7f8301d73740 1 --2- 172.21.15.49:0/1647561933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816deecea0 0x55816dedda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.637+0000 7f8301d73740 1 -- 172.21.15.49:0/1647561933 >> 172.21.15.49:0/1647561933 conn(0x55816dee28b0 msgr2=0x55816dee2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:06.640 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.641+0000 7f8301d73740 1 -- 172.21.15.49:0/1647561933 shutdown_connections 2024-03-18T00:53:06.641 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.641+0000 7f8301d73740 1 -- 172.21.15.49:0/1647561933 wait complete. 2024-03-18T00:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.641+0000 7f8301d73740 1 -- start start 2024-03-18T00:53:06.642 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.642+0000 7f8301d73740 1 Processor -- start 2024-03-18T00:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.643+0000 7f8301d73740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816df0e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.643+0000 7f8301d73740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 0x55816df0eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.643+0000 7f8301d73740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 0x55816df0fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.643+0000 7f8301d73740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f82e400f9d0 con 0x55816deecea0 2024-03-18T00:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.643+0000 7f8301d73740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f82e400f300 con 0x55816def6e30 2024-03-18T00:53:06.643 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.643+0000 7f8301d73740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f82e400c500 con 0x55816def1a10 2024-03-18T00:53:06.644 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.643+0000 7f8301d73740 10 client.0 _ll_register_callbacks cb 0x7f82e809ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.645+0000 7f82fce48640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816df0e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.645+0000 7f82fce48640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816df0e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33960/0 (socket says 172.21.15.49:33960) 2024-03-18T00:53:06.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.645+0000 7f82fce48640 1 -- 172.21.15.49:0/4051564874 learned_addr learned my addr 172.21.15.49:0/4051564874 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.645+0000 7f82fd649640 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 0x55816df0fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.646+0000 7f82f7fff640 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 0x55816df0eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.646+0000 7f82fce48640 1 -- 172.21.15.49:0/4051564874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 msgr2=0x55816df0eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:06.648 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.646+0000 7f82fce48640 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 0x55816df0eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.646+0000 7f82fce48640 1 -- 172.21.15.49:0/4051564874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 msgr2=0x55816df0fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.646+0000 7f82fce48640 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 0x55816df0fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.646+0000 7f82fce48640 1 -- 172.21.15.49:0/4051564874 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55816df10140 con 0x55816deecea0 2024-03-18T00:53:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.646+0000 7f82fce48640 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816df0e650 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097ce0 tx=0x7f82e809b3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6716649841f48dec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.646+0000 7f82d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:06.649 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.647+0000 7f82d7fff640 1 -- 172.21.15.49:0/4051564874 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e809a760 con 0x55816deecea0 2024-03-18T00:53:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.647+0000 7f8301d73740 1 -- 172.21.15.49:0/4051564874 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82f0003e50 con 0x55816deecea0 2024-03-18T00:53:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.647+0000 7f82d7fff640 1 -- 172.21.15.49:0/4051564874 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f82e8099610 con 0x55816deecea0 2024-03-18T00:53:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.647+0000 7f82d7fff640 1 -- 172.21.15.49:0/4051564874 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f82e809f230 con 0x55816deecea0 2024-03-18T00:53:06.650 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.647+0000 7f8301d73740 10 client.16179 fetch_fsmap learned FSMap version 26 2024-03-18T00:53:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.648+0000 7f8301d73740 1 -- 172.21.15.49:0/4051564874 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55816df746f0 con 0x55816deecea0 2024-03-18T00:53:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.648+0000 7f82d7fff640 1 -- 172.21.15.49:0/4051564874 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82f0003e50 con 0x55816deecea0 2024-03-18T00:53:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.648+0000 7f82d7fff640 1 -- 172.21.15.49:0/4051564874 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7f82e807d070 con 0x55816deecea0 2024-03-18T00:53:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.648+0000 7f8301d73740 10 client.16179 fetch_fsmap finished waiting for FSMap version 26 2024-03-18T00:53:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.648+0000 7f8301d73740 10 client.16179 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:06.651 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.649+0000 7f8301d73740 20 client.16179 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:06.652 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.649+0000 7f8301d73740 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55816df74e50 0x55816df77270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:06.652 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.649+0000 7f8301d73740 4 client.16179 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:53:06.652 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.649+0000 7f8301d73740 1 -- 172.21.15.49:0/4051564874 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55816de542d0 con 0x55816df74e50 2024-03-18T00:53:06.652 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.650+0000 7f82f7fff640 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55816df74e50 0x55816df77270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.655+0000 7f82f7fff640 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55816df74e50 0x55816df77270 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3041ee282fee045c server_cookie=46233a3a4cfbd14e in_seq=0 out_seq=0 2024-03-18T00:53:06.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:06.655+0000 7f82d7fff640 10 client.16179 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:07.128 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:06 smithi049 ceph-mon[32130]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 832 MiB/s wr, 2.26k op/s 2024-03-18T00:53:07.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:06 smithi107 ceph-mon[36270]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 832 MiB/s wr, 2.26k op/s 2024-03-18T00:53:07.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:06 smithi106 ceph-mon[35899]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 832 MiB/s wr, 2.26k op/s 2024-03-18T00:53:09.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:08 smithi107 ceph-mon[36270]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 912 MiB/s wr, 2.59k op/s 2024-03-18T00:53:09.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:08 smithi106 ceph-mon[35899]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 912 MiB/s wr, 2.59k op/s 2024-03-18T00:53:09.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:08 smithi049 ceph-mon[32130]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 912 MiB/s wr, 2.59k op/s 2024-03-18T00:53:11.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:10 smithi107 ceph-mon[36270]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 970 MiB/s wr, 2.66k op/s 2024-03-18T00:53:11.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:10 smithi106 ceph-mon[35899]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 970 MiB/s wr, 2.66k op/s 2024-03-18T00:53:11.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:10 smithi049 ceph-mon[32130]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 970 MiB/s wr, 2.66k op/s 2024-03-18T00:53:13.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:12 smithi107 ceph-mon[36270]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 834 MiB/s wr, 2.53k op/s 2024-03-18T00:53:13.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:12 smithi106 ceph-mon[35899]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 834 MiB/s wr, 2.53k op/s 2024-03-18T00:53:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.335+0000 7f82d7fff640 1 -- 172.21.15.49:0/4051564874 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55816de542d0 con 0x55816df74e50 2024-03-18T00:53:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.336+0000 7f82d7fff640 10 client.16179 handle_command_reply: tid=0 2024-03-18T00:53:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.336+0000 7f8301d73740 1 client.16179 shutdown 2024-03-18T00:53:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.336+0000 7f8301d73740 1 -- 172.21.15.49:0/4051564874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 msgr2=0x55816df0e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:13.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.336+0000 7f8301d73740 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816df0e650 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f82e8097ce0 tx=0x7f82e809b3d0 comp rx=0 tx=0).stop 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.337+0000 7f8301d73740 1 -- 172.21.15.49:0/4051564874 shutdown_connections 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.337+0000 7f8301d73740 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55816df74e50 0x55816df77270 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.337+0000 7f8301d73740 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55816def6e30 0x55816df0fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.337+0000 7f8301d73740 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55816def1a10 0x55816df0eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.337+0000 7f8301d73740 1 --2- 172.21.15.49:0/4051564874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55816deecea0 0x55816df0e650 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.337+0000 7f8301d73740 1 -- 172.21.15.49:0/4051564874 >> 172.21.15.49:0/4051564874 conn(0x55816dee28b0 msgr2=0x55816dedda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.337+0000 7f82d7fff640 0 client.16179 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.339+0000 7f8301d73740 1 -- 172.21.15.49:0/4051564874 shutdown_connections 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.339+0000 7f8301d73740 1 -- 172.21.15.49:0/4051564874 wait complete. 2024-03-18T00:53:13.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.339+0000 7f8301d73740 20 client.16179 trim_cache size 0 max 16384 2024-03-18T00:53:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:53:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:53:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:53:13.340 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:53:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.346+0000 7f82ff0ac640 1 -- 172.21.15.49:0/3608175604 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82e00a2ae0 msgr2=0x7f82e00a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.346+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82e00a2ae0 0x7f82e00a4fa0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f82f0004af0 tx=0x7f82f0032340 comp rx=0 tx=0).stop 2024-03-18T00:53:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.346+0000 7f82ff0ac640 1 -- 172.21.15.49:0/3608175604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82f81087e0 msgr2=0x7f82f81a6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.346+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82f81087e0 0x7f82f81a6a00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f82e800b450 tx=0x7f82e800b910 comp rx=0 tx=0).stop 2024-03-18T00:53:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.354+0000 7f82ff0ac640 1 -- 172.21.15.49:0/3608175604 shutdown_connections 2024-03-18T00:53:13.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.354+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f82e00a2ae0 0x7f82e00a4fa0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.354+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f82f810db00 0x7f82f81adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.354+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f82f810b150 0x7f82f81a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.354+0000 7f82ff0ac640 1 --2- 172.21.15.49:0/3608175604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82f81087e0 0x7f82f81a6a00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:13.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.354+0000 7f82ff0ac640 1 -- 172.21.15.49:0/3608175604 >> 172.21.15.49:0/3608175604 conn(0x7f82f806e820 msgr2=0x7f82f810a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:13.357 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.358+0000 7f82ff0ac640 1 -- 172.21.15.49:0/3608175604 shutdown_connections 2024-03-18T00:53:13.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:13.359+0000 7f82ff0ac640 1 -- 172.21.15.49:0/3608175604 wait complete. 2024-03-18T00:53:13.381 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:12 smithi049 ceph-mon[32130]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 834 MiB/s wr, 2.53k op/s 2024-03-18T00:53:15.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:14 smithi107 ceph-mon[36270]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1024 MiB/s wr, 3.05k op/s 2024-03-18T00:53:15.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:14 smithi106 ceph-mon[35899]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1024 MiB/s wr, 3.05k op/s 2024-03-18T00:53:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:14 smithi049 ceph-mon[32130]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1024 MiB/s wr, 3.05k op/s 2024-03-18T00:53:17.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:16 smithi107 ceph-mon[36270]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 839 MiB/s wr, 2.36k op/s 2024-03-18T00:53:17.198 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:16 smithi106 ceph-mon[35899]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 839 MiB/s wr, 2.36k op/s 2024-03-18T00:53:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:16 smithi049 ceph-mon[32130]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 839 MiB/s wr, 2.36k op/s 2024-03-18T00:53:19.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:18 smithi106 ceph-mon[35899]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 924 MiB/s wr, 2.50k op/s 2024-03-18T00:53:19.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:18 smithi049 ceph-mon[32130]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 924 MiB/s wr, 2.50k op/s 2024-03-18T00:53:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:18 smithi107 ceph-mon[36270]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 924 MiB/s wr, 2.50k op/s 2024-03-18T00:53:21.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:20 smithi049 ceph-mon[32130]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 970 MiB/s wr, 2.47k op/s 2024-03-18T00:53:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:20 smithi107 ceph-mon[36270]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 970 MiB/s wr, 2.47k op/s 2024-03-18T00:53:21.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:20 smithi106 ceph-mon[35899]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 970 MiB/s wr, 2.47k op/s 2024-03-18T00:53:23.246 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:22 smithi106 ceph-mon[35899]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 832 MiB/s wr, 2.11k op/s 2024-03-18T00:53:23.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:22 smithi049 ceph-mon[32130]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 832 MiB/s wr, 2.11k op/s 2024-03-18T00:53:23.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:22 smithi107 ceph-mon[36270]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 832 MiB/s wr, 2.11k op/s 2024-03-18T00:53:25.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:24 smithi049 ceph-mon[32130]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 GiB/s wr, 2.35k op/s 2024-03-18T00:53:25.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:24 smithi107 ceph-mon[36270]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 GiB/s wr, 2.35k op/s 2024-03-18T00:53:25.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:24 smithi106 ceph-mon[35899]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 GiB/s wr, 2.35k op/s 2024-03-18T00:53:26.683 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.682+0000 7fca255da640 1 -- 172.21.15.49:0/3152968022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca2010b230 msgr2=0x7fca2010d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:26.683 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.682+0000 7fca255da640 1 --2- 172.21.15.49:0/3152968022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca2010b230 0x7fca2010d680 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fca18005fd0 tx=0x7fca1802cc70 comp rx=0 tx=0).stop 2024-03-18T00:53:26.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.686+0000 7fca255da640 1 -- 172.21.15.49:0/3152968022 shutdown_connections 2024-03-18T00:53:26.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.686+0000 7fca255da640 1 --2- 172.21.15.49:0/3152968022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca2010dd10 0x7fca201101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.686+0000 7fca255da640 1 --2- 172.21.15.49:0/3152968022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca2010b230 0x7fca2010d680 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.686+0000 7fca255da640 1 --2- 172.21.15.49:0/3152968022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca20108870 0x7fca2010ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.689 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.686+0000 7fca255da640 1 -- 172.21.15.49:0/3152968022 >> 172.21.15.49:0/3152968022 conn(0x7fca2006e860 msgr2=0x7fca20070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:26.700 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.699+0000 7fca255da640 1 -- 172.21.15.49:0/3152968022 shutdown_connections 2024-03-18T00:53:26.701 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.701+0000 7fca255da640 1 -- 172.21.15.49:0/3152968022 wait complete. 2024-03-18T00:53:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.702+0000 7fca255da640 1 Processor -- start 2024-03-18T00:53:26.703 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.702+0000 7fca255da640 1 -- start start 2024-03-18T00:53:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.711+0000 7fca255da640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca20108870 0x7fca201a66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.711+0000 7fca255da640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2010b230 0x7fca201a6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.711+0000 7fca255da640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca2010dd10 0x7fca201adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.711+0000 7fca255da640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fca201138f0 con 0x7fca2010dd10 2024-03-18T00:53:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.711+0000 7fca255da640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fca20113770 con 0x7fca20108870 2024-03-18T00:53:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.711+0000 7fca255da640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fca20113a70 con 0x7fca2010b230 2024-03-18T00:53:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.712+0000 7fca1effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca20108870 0x7fca201a66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.712+0000 7fca1effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca20108870 0x7fca201a66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47066/0 (socket says 172.21.15.49:47066) 2024-03-18T00:53:26.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.712+0000 7fca1effd640 1 -- 172.21.15.49:0/1765850190 learned_addr learned my addr 172.21.15.49:0/1765850190 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:26.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.714+0000 7fca1f7fe640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca2010dd10 0x7fca201adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.714+0000 7fca1effd640 1 -- 172.21.15.49:0/1765850190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2010b230 msgr2=0x7fca201a6bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:53:26.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.714+0000 7fca1effd640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2010b230 0x7fca201a6bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.714+0000 7fca1effd640 1 -- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca2010dd10 msgr2=0x7fca201adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.714+0000 7fca1effd640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca2010dd10 0x7fca201adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.714+0000 7fca1effd640 1 -- 172.21.15.49:0/1765850190 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca201ae2d0 con 0x7fca20108870 2024-03-18T00:53:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.715+0000 7fca1effd640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca20108870 0x7fca201a66b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c890 tx=0x7fca1000cd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:26.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.715+0000 7fca1f7fe640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca2010dd10 0x7fca201adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:53:26.726 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.725+0000 7fca07fff640 1 -- 172.21.15.49:0/1765850190 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10004220 con 0x7fca20108870 2024-03-18T00:53:26.726 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.725+0000 7fca07fff640 1 -- 172.21.15.49:0/1765850190 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca100043c0 con 0x7fca20108870 2024-03-18T00:53:26.726 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.725+0000 7fca07fff640 1 -- 172.21.15.49:0/1765850190 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100049e0 con 0x7fca20108870 2024-03-18T00:53:26.730 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.727+0000 7fca255da640 1 -- 172.21.15.49:0/1765850190 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca201ae5c0 con 0x7fca20108870 2024-03-18T00:53:26.730 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.727+0000 7fca255da640 1 -- 172.21.15.49:0/1765850190 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca201aeac0 con 0x7fca20108870 2024-03-18T00:53:26.738 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.738+0000 7fca07fff640 1 -- 172.21.15.49:0/1765850190 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fca10004c40 con 0x7fca20108870 2024-03-18T00:53:26.740 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.740+0000 7fca07fff640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca080a2a00 0x7fca080a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.741+0000 7fca07fff640 1 -- 172.21.15.49:0/1765850190 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fca100ceac0 con 0x7fca20108870 2024-03-18T00:53:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.742+0000 7fca281bd740 1 Processor -- start 2024-03-18T00:53:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.743+0000 7fca281bd740 1 -- start start 2024-03-18T00:53:26.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.744+0000 7fca281bd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39e9fe930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.744+0000 7fca281bd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fef00 0x55f39ea03a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.744+0000 7fca281bd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39ea04070 0x55f39ea06500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.744+0000 7fca281bd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f39e952a40 con 0x55f39ea04070 2024-03-18T00:53:26.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.744+0000 7fca281bd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f39e8fb920 con 0x55f39e9fe510 2024-03-18T00:53:26.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.744+0000 7fca281bd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f39e94eae0 con 0x55f39e9fef00 2024-03-18T00:53:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.746+0000 7fca1e7fc640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca080a2a00 0x7fca080a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.747+0000 7fca1effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39e9fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.747+0000 7fca1effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39e9fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47070/0 (socket says 172.21.15.49:47070) 2024-03-18T00:53:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.747+0000 7fca1effd640 1 -- 172.21.15.49:0/157364074 learned_addr learned my addr 172.21.15.49:0/157364074 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.748+0000 7fca1effd640 1 -- 172.21.15.49:0/157364074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fef00 msgr2=0x55f39ea03a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.748+0000 7fca1effd640 1 --2- 172.21.15.49:0/157364074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fef00 0x55f39ea03a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.748+0000 7fca1effd640 1 -- 172.21.15.49:0/157364074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39ea04070 msgr2=0x55f39ea06500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.748+0000 7fca1effd640 1 --2- 172.21.15.49:0/157364074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39ea04070 0x55f39ea06500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.748+0000 7fca1effd640 1 -- 172.21.15.49:0/157364074 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f39e913460 con 0x55f39e9fe510 2024-03-18T00:53:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.750+0000 7fca1effd640 1 --2- 172.21.15.49:0/157364074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39e9fe930 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fca100a3650 tx=0x7fca1009bba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=daf3e51498bd6827 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:26.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.750+0000 7fca1e7fc640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca080a2a00 0x7fca080a4ec0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fca18037040 tx=0x7fca18005ed0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.751+0000 7fca057fa640 1 -- 172.21.15.49:0/157364074 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100b4070 con 0x55f39e9fe510 2024-03-18T00:53:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.751+0000 7fca057fa640 1 -- 172.21.15.49:0/157364074 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca100a3b80 con 0x55f39e9fe510 2024-03-18T00:53:26.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.751+0000 7fca057fa640 1 -- 172.21.15.49:0/157364074 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1001d040 con 0x55f39e9fe510 2024-03-18T00:53:26.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.759+0000 7fca281bd740 1 -- 172.21.15.49:0/157364074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 msgr2=0x55f39e9fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:26.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.759+0000 7fca281bd740 1 --2- 172.21.15.49:0/157364074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39e9fe930 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fca100a3650 tx=0x7fca1009bba0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.763+0000 7fca281bd740 1 -- 172.21.15.49:0/157364074 shutdown_connections 2024-03-18T00:53:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.763+0000 7fca281bd740 1 --2- 172.21.15.49:0/157364074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39ea04070 0x55f39ea06500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.763+0000 7fca281bd740 1 --2- 172.21.15.49:0/157364074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fef00 0x55f39ea03a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.763+0000 7fca281bd740 1 --2- 172.21.15.49:0/157364074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39e9fe930 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.764+0000 7fca281bd740 1 -- 172.21.15.49:0/157364074 >> 172.21.15.49:0/157364074 conn(0x55f39e9e8a70 msgr2=0x55f39e9e8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:26.770 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.769+0000 7fca281bd740 1 -- 172.21.15.49:0/157364074 shutdown_connections 2024-03-18T00:53:26.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.769+0000 7fca281bd740 1 -- 172.21.15.49:0/157364074 wait complete. 2024-03-18T00:53:26.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.769+0000 7fca281bd740 1 -- start start 2024-03-18T00:53:26.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.770+0000 7fca281bd740 1 Processor -- start 2024-03-18T00:53:26.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.771+0000 7fca281bd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39ea08870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.771+0000 7fca281bd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea08db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.771+0000 7fca281bd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 0x55f39ea0fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.771+0000 7fca281bd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f39ea07020 con 0x55f39e9fef00 2024-03-18T00:53:26.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.771+0000 7fca281bd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f39e90ed70 con 0x55f39e9fe510 2024-03-18T00:53:26.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.771+0000 7fca281bd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f39ea071a0 con 0x55f39ea04070 2024-03-18T00:53:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.771+0000 7fca281bd740 10 client.0 _ll_register_callbacks cb 0x55f39e666a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.772+0000 7fca1e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea08db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.775 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.772+0000 7fca1e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea08db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55738/0 (socket says 172.21.15.49:55738) 2024-03-18T00:53:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.772+0000 7fca1e7fc640 1 -- 172.21.15.49:0/602669445 learned_addr learned my addr 172.21.15.49:0/602669445 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.775+0000 7fca1effd640 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39ea08870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.776 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.775+0000 7fca1f7fe640 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 0x55f39ea0fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.778+0000 7fca1f7fe640 1 -- 172.21.15.49:0/602669445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 msgr2=0x55f39ea08870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.778+0000 7fca1f7fe640 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39ea08870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.778+0000 7fca1f7fe640 1 -- 172.21.15.49:0/602669445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 msgr2=0x55f39ea08db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.779+0000 7fca1f7fe640 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea08db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.779+0000 7fca1f7fe640 1 -- 172.21.15.49:0/602669445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f39e667210 con 0x55f39ea04070 2024-03-18T00:53:26.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.779+0000 7fca1e7fc640 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea08db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:53:26.789 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.790+0000 7fca1f7fe640 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 0x55f39ea0fdf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca1400d3c0 tx=0x7fca1400d880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9d70c13435b4a8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:26.792 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.792+0000 7fca02ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:53:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.792+0000 7fca02ff5640 1 -- 172.21.15.49:0/602669445 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1400f8f0 con 0x55f39ea04070 2024-03-18T00:53:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.792+0000 7fca281bd740 1 -- 172.21.15.49:0/602669445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca100a3b80 con 0x55f39ea04070 2024-03-18T00:53:26.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.792+0000 7fca02ff5640 1 -- 172.21.15.49:0/602669445 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fca1400fa90 con 0x55f39ea04070 2024-03-18T00:53:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.797+0000 7fca02ff5640 1 -- 172.21.15.49:0/602669445 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fca1400fc90 con 0x55f39ea04070 2024-03-18T00:53:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.798+0000 7fca281bd740 10 client.25858 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.798+0000 7fca281bd740 1 -- 172.21.15.49:0/602669445 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f39e733620 con 0x55f39ea04070 2024-03-18T00:53:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.799+0000 7fca02ff5640 1 -- 172.21.15.49:0/602669445 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca100a3b80 con 0x55f39ea04070 2024-03-18T00:53:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.800+0000 7fca02ff5640 1 -- 172.21.15.49:0/602669445 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fca140213f0 con 0x55f39ea04070 2024-03-18T00:53:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.800+0000 7fca281bd740 10 client.25858 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.800+0000 7fca281bd740 10 client.25858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.800+0000 7fca281bd740 20 client.25858 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.800+0000 7fca281bd740 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f39ea147d0 0x55f39ea16c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.800+0000 7fca281bd740 4 client.25858 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:26.800 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.800+0000 7fca281bd740 1 -- 172.21.15.49:0/602669445 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f39e95ff80 con 0x55f39ea147d0 2024-03-18T00:53:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.804+0000 7fca1effd640 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f39ea147d0 0x55f39ea16c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.816+0000 7fca1effd640 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f39ea147d0 0x55f39ea16c30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1f0e12372d9a699 server_cookie=475e6b6f72466c89 in_seq=0 out_seq=0 2024-03-18T00:53:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.816+0000 7fca02ff5640 10 client.25858 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.824+0000 7fca02ff5640 1 -- 172.21.15.49:0/602669445 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f39e95ff80 con 0x55f39ea147d0 2024-03-18T00:53:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.824+0000 7fca02ff5640 10 client.25858 handle_command_reply: tid=0 2024-03-18T00:53:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.824+0000 7fca281bd740 1 client.25858 shutdown 2024-03-18T00:53:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.825+0000 7fca281bd740 1 -- 172.21.15.49:0/602669445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 msgr2=0x55f39ea0fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.825+0000 7fca281bd740 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 0x55f39ea0fdf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca1400d3c0 tx=0x7fca1400d880 comp rx=0 tx=0).stop 2024-03-18T00:53:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.825+0000 7fca281bd740 1 -- 172.21.15.49:0/602669445 shutdown_connections 2024-03-18T00:53:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.825+0000 7fca281bd740 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f39ea147d0 0x55f39ea16c30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.825+0000 7fca281bd740 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 0x55f39ea0fdf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.825+0000 7fca281bd740 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea08db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.825+0000 7fca281bd740 1 --2- 172.21.15.49:0/602669445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39ea08870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.825+0000 7fca281bd740 1 -- 172.21.15.49:0/602669445 >> 172.21.15.49:0/602669445 conn(0x55f39e9e8a70 msgr2=0x55f39e9e7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.825+0000 7fca02ff5640 0 client.25858 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.827+0000 7fca281bd740 1 -- 172.21.15.49:0/602669445 shutdown_connections 2024-03-18T00:53:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.827+0000 7fca281bd740 1 -- 172.21.15.49:0/602669445 wait complete. 2024-03-18T00:53:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.829+0000 7fca281bd740 20 client.25858 trim_cache size 0 max 16384 2024-03-18T00:53:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.884+0000 7fca281bd740 1 Processor -- start 2024-03-18T00:53:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.888+0000 7fca281bd740 1 -- start start 2024-03-18T00:53:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.888+0000 7fca281bd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39ea00280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.888+0000 7fca281bd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea007c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.888+0000 7fca281bd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 0x55f39ea0fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.888+0000 7fca281bd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c00e650 con 0x55f39e9fef00 2024-03-18T00:53:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.888+0000 7fca281bd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c010190 con 0x55f39e9fe510 2024-03-18T00:53:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.888+0000 7fca281bd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c00ed20 con 0x55f39ea04070 2024-03-18T00:53:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.890+0000 7fca1effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39ea00280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.890+0000 7fca1f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 0x55f39ea0fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.890+0000 7fca1e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea007c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.890+0000 7fca1e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea007c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55744/0 (socket says 172.21.15.49:55744) 2024-03-18T00:53:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.891+0000 7fca1e7fc640 1 -- 172.21.15.49:0/882017785 learned_addr learned my addr 172.21.15.49:0/882017785 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.891+0000 7fca1e7fc640 1 -- 172.21.15.49:0/882017785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 msgr2=0x55f39ea0fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.891+0000 7fca1e7fc640 1 --2- 172.21.15.49:0/882017785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 0x55f39ea0fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.891+0000 7fca1e7fc640 1 -- 172.21.15.49:0/882017785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 msgr2=0x55f39ea00280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:26.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.891+0000 7fca1e7fc640 1 --2- 172.21.15.49:0/882017785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39ea00280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.891+0000 7fca1e7fc640 1 -- 172.21.15.49:0/882017785 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f39e9e97e0 con 0x55f39e9fef00 2024-03-18T00:53:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.892+0000 7fca1e7fc640 1 --2- 172.21.15.49:0/882017785 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea007c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fca18005fa0 tx=0x7fca1803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7733f159898652e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.893+0000 7fca04ff9640 1 -- 172.21.15.49:0/882017785 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1800a3d0 con 0x55f39e9fef00 2024-03-18T00:53:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.893+0000 7fca04ff9640 1 -- 172.21.15.49:0/882017785 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca18050070 con 0x55f39e9fef00 2024-03-18T00:53:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.894+0000 7fca04ff9640 1 -- 172.21.15.49:0/882017785 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1802f890 con 0x55f39e9fef00 2024-03-18T00:53:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.894+0000 7fca281bd740 1 -- 172.21.15.49:0/882017785 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 msgr2=0x55f39ea007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.895+0000 7fca281bd740 1 --2- 172.21.15.49:0/882017785 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea007c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fca18005fa0 tx=0x7fca1803c000 comp rx=0 tx=0).stop 2024-03-18T00:53:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.895+0000 7fca281bd740 1 -- 172.21.15.49:0/882017785 shutdown_connections 2024-03-18T00:53:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.895+0000 7fca281bd740 1 --2- 172.21.15.49:0/882017785 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39ea04070 0x55f39ea0fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.895+0000 7fca281bd740 1 --2- 172.21.15.49:0/882017785 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea007c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.895+0000 7fca281bd740 1 --2- 172.21.15.49:0/882017785 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39e9fe510 0x55f39ea00280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.895+0000 7fca281bd740 1 -- 172.21.15.49:0/882017785 >> 172.21.15.49:0/882017785 conn(0x55f39e9e8a70 msgr2=0x55f39ea167e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.895+0000 7fca281bd740 1 -- 172.21.15.49:0/882017785 shutdown_connections 2024-03-18T00:53:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.895+0000 7fca281bd740 1 -- 172.21.15.49:0/882017785 wait complete. 2024-03-18T00:53:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.896+0000 7fca281bd740 1 -- start start 2024-03-18T00:53:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.897+0000 7fca281bd740 1 Processor -- start 2024-03-18T00:53:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.898+0000 7fca281bd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fe510 0x55f39ea0cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.899+0000 7fca281bd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea0d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.899+0000 7fca281bd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39ea04070 0x55f39ea78b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.899+0000 7fca1e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea0d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.899+0000 7fca1e7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea0d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:55752/0 (socket says 172.21.15.49:55752) 2024-03-18T00:53:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.899+0000 7fca1e7fc640 1 -- 172.21.15.49:0/178332652 learned_addr learned my addr 172.21.15.49:0/178332652 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c00fac0 con 0x55f39e9fef00 2024-03-18T00:53:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c00f3f0 con 0x55f39ea04070 2024-03-18T00:53:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c00c630 con 0x55f39e9fe510 2024-03-18T00:53:26.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca1effd640 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fe510 0x55f39ea0cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca1e7fc640 1 -- 172.21.15.49:0/178332652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fe510 msgr2=0x55f39ea0cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca1e7fc640 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fe510 0x55f39ea0cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca1e7fc640 1 -- 172.21.15.49:0/178332652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39ea04070 msgr2=0x55f39ea78b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca1e7fc640 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39ea04070 0x55f39ea78b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca1e7fc640 1 -- 172.21.15.49:0/178332652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f39ea790a0 con 0x55f39e9fef00 2024-03-18T00:53:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.900+0000 7fca1effd640 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fe510 0x55f39ea0cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:53:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.901+0000 7fca281bd740 10 client.0 _ll_register_callbacks cb 0x7fca14005e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.902+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca18050070 con 0x55f39e9fef00 2024-03-18T00:53:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.902+0000 7fca1e7fc640 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea0d130 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fca1800a740 tx=0x7fca18035cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5ea6d8d6bdf897 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.902+0000 7fca02ff5640 10 client.16191 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.902+0000 7fca02ff5640 1 -- 172.21.15.49:0/178332652 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca1805a020 con 0x55f39e9fef00 2024-03-18T00:53:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.903+0000 7fca02ff5640 1 -- 172.21.15.49:0/178332652 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fca1802f830 con 0x55f39e9fef00 2024-03-18T00:53:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.903+0000 7fca02ff5640 1 -- 172.21.15.49:0/178332652 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fca1800a3d0 con 0x55f39e9fef00 2024-03-18T00:53:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.903+0000 7fca281bd740 10 client.16191 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.903+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55f39ea7a140 con 0x55f39e9fef00 2024-03-18T00:53:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.904+0000 7fca02ff5640 1 -- 172.21.15.49:0/178332652 <== mon.0 v2:172.21.15.49:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fca1802fa70 con 0x55f39e9fef00 2024-03-18T00:53:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.904+0000 7fca281bd740 10 client.16191 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.904+0000 7fca281bd740 10 client.16191 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.904+0000 7fca281bd740 20 client.16191 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.904+0000 7fca281bd740 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f39ea7a940 0x55f39ea7cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.904+0000 7fca281bd740 4 client.16191 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:53:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.904+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f39e95a2d0 con 0x55f39ea7a940 2024-03-18T00:53:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.905+0000 7fca1effd640 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f39ea7a940 0x55f39ea7cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:26.922 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.921+0000 7fca1effd640 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f39ea7a940 0x55f39ea7cd60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f0b7c1339d0ee2b server_cookie=3490f2855e01e95 in_seq=0 out_seq=0 2024-03-18T00:53:26.922 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:26.922+0000 7fca02ff5640 10 client.16191 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:27.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:26 smithi049 ceph-mon[32130]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 847 MiB/s wr, 1.80k op/s 2024-03-18T00:53:27.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:26 smithi049 ceph-mon[32130]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:53:27.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:26 smithi107 ceph-mon[36270]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 847 MiB/s wr, 1.80k op/s 2024-03-18T00:53:27.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:26 smithi107 ceph-mon[36270]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:53:27.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:26 smithi106 ceph-mon[35899]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 847 MiB/s wr, 1.80k op/s 2024-03-18T00:53:27.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:26 smithi106 ceph-mon[35899]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:53:29.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:28 smithi049 ceph-mon[32130]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 923 MiB/s wr, 1.83k op/s 2024-03-18T00:53:29.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:28 smithi107 ceph-mon[36270]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 923 MiB/s wr, 1.83k op/s 2024-03-18T00:53:29.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:28 smithi106 ceph-mon[35899]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 923 MiB/s wr, 1.83k op/s 2024-03-18T00:53:29.686 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:53:29.686 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:53:30.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.051+0000 7fd132747640 1 -- 172.21.15.49:0/3026238619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c1087e0 msgr2=0x7fd12c108c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:30.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.051+0000 7fd132747640 1 --2- 172.21.15.49:0/3026238619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c1087e0 0x7fd12c108c60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd12800aa40 tx=0x7fd12802cf90 comp rx=0 tx=0).stop 2024-03-18T00:53:30.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.051+0000 7fd132747640 1 -- 172.21.15.49:0/3026238619 shutdown_connections 2024-03-18T00:53:30.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.051+0000 7fd132747640 1 --2- 172.21.15.49:0/3026238619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd12c1091a0 0x7fd12c110a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.051+0000 7fd132747640 1 --2- 172.21.15.49:0/3026238619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c1087e0 0x7fd12c108c60 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.051+0000 7fd132747640 1 --2- 172.21.15.49:0/3026238619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd12c109b40 0x7fd12c109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.051+0000 7fd132747640 1 -- 172.21.15.49:0/3026238619 >> 172.21.15.49:0/3026238619 conn(0x7fd12c06e8b0 msgr2=0x7fd12c070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:30.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.051+0000 7fd132747640 1 -- 172.21.15.49:0/3026238619 shutdown_connections 2024-03-18T00:53:30.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.051+0000 7fd132747640 1 -- 172.21.15.49:0/3026238619 wait complete. 2024-03-18T00:53:30.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.052+0000 7fd132747640 1 Processor -- start 2024-03-18T00:53:30.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.052+0000 7fd132747640 1 -- start start 2024-03-18T00:53:30.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd132747640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd12c1087e0 0x7fd12c1a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd132747640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd12c1091a0 0x7fd12c1a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd132747640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c109b40 0x7fd12c1a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd132747640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1139c0 con 0x7fd12c109b40 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd132747640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c113840 con 0x7fd12c1087e0 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd132747640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c113b40 con 0x7fd12c1091a0 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd131f46640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c109b40 0x7fd12c1a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd131f46640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c109b40 0x7fd12c1a99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58046/0 (socket says 172.21.15.49:58046) 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd131f46640 1 -- 172.21.15.49:0/1559343965 learned_addr learned my addr 172.21.15.49:0/1559343965 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.053+0000 7fd130f44640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd12c1091a0 0x7fd12c1a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.054+0000 7fd131f46640 1 -- 172.21.15.49:0/1559343965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd12c1091a0 msgr2=0x7fd12c1a2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.054+0000 7fd131f46640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd12c1091a0 0x7fd12c1a2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.054+0000 7fd131f46640 1 -- 172.21.15.49:0/1559343965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd12c1087e0 msgr2=0x7fd12c1a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.054+0000 7fd131f46640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd12c1087e0 0x7fd12c1a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.054+0000 7fd131f46640 1 -- 172.21.15.49:0/1559343965 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12c1aa030 con 0x7fd12c109b40 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.054+0000 7fd131f46640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c109b40 0x7fd12c1a99c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd12400b990 tx=0x7fd12400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:30.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.055+0000 7fd1227fc640 1 -- 172.21.15.49:0/1559343965 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd124019070 con 0x7fd12c109b40 2024-03-18T00:53:30.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.055+0000 7fd1227fc640 1 -- 172.21.15.49:0/1559343965 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd12400cba0 con 0x7fd12c109b40 2024-03-18T00:53:30.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.055+0000 7fd1227fc640 1 -- 172.21.15.49:0/1559343965 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd124007650 con 0x7fd12c109b40 2024-03-18T00:53:30.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.055+0000 7fd132747640 1 -- 172.21.15.49:0/1559343965 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd12c1aa320 con 0x7fd12c109b40 2024-03-18T00:53:30.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.055+0000 7fd132747640 1 -- 172.21.15.49:0/1559343965 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd12c1aa860 con 0x7fd12c109b40 2024-03-18T00:53:30.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.060+0000 7fd1227fc640 1 -- 172.21.15.49:0/1559343965 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd12401c070 con 0x7fd12c109b40 2024-03-18T00:53:30.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.060+0000 7fd136594740 1 Processor -- start 2024-03-18T00:53:30.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.060+0000 7fd136594740 1 -- start start 2024-03-18T00:53:30.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd136594740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c3a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd136594740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 0x55f536c3f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd136594740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 0x55f536c42330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd1227fc640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1180a2930 0x7fd1180a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd130f44640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c3a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd130f44640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c3a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58048/0 (socket says 172.21.15.49:58048) 2024-03-18T00:53:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd130f44640 1 -- 172.21.15.49:0/1571157688 learned_addr learned my addr 172.21.15.49:0/1571157688 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd1227fc640 1 -- 172.21.15.49:0/1559343965 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd1240cdfe0 con 0x7fd12c109b40 2024-03-18T00:53:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd136594740 1 -- 172.21.15.49:0/1571157688 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f536b9f500 con 0x55f536c3a340 2024-03-18T00:53:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd136594740 1 -- 172.21.15.49:0/1571157688 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f536b47be0 con 0x55f536c3fea0 2024-03-18T00:53:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd136594740 1 -- 172.21.15.49:0/1571157688 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f536b9b5a0 con 0x55f536c3ad30 2024-03-18T00:53:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.061+0000 7fd131745640 1 --2- 172.21.15.49:0/1571157688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 0x55f536c3f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:30.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.062+0000 7fd130f44640 1 --2- 172.21.15.49:0/1571157688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 0x55f536c42330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:30.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.062+0000 7fd131f46640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1180a2930 0x7fd1180a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:30.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.062+0000 7fd130f44640 1 -- 172.21.15.49:0/1571157688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 msgr2=0x55f536c3f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:30.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.062+0000 7fd130f44640 1 --2- 172.21.15.49:0/1571157688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 0x55f536c3f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.062+0000 7fd130f44640 1 -- 172.21.15.49:0/1571157688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 msgr2=0x55f536c42330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:30.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.062+0000 7fd130f44640 1 --2- 172.21.15.49:0/1571157688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 0x55f536c42330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.062+0000 7fd130f44640 1 -- 172.21.15.49:0/1571157688 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f536b5b4b0 con 0x55f536c3a340 2024-03-18T00:53:30.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.063+0000 7fd131f46640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1180a2930 0x7fd1180a4df0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd1240a2bd0 tx=0x7fd12400e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:30.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.063+0000 7fd130f44640 1 --2- 172.21.15.49:0/1571157688 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c3a760 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd12800aa40 tx=0x7fd128004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4731cc530ff4d89e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:30.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.063+0000 7fd1177fe640 1 -- 172.21.15.49:0/1571157688 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12802dd40 con 0x55f536c3a340 2024-03-18T00:53:30.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.064+0000 7fd1177fe640 1 -- 172.21.15.49:0/1571157688 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1280356c0 con 0x55f536c3a340 2024-03-18T00:53:30.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.064+0000 7fd1177fe640 1 -- 172.21.15.49:0/1571157688 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd128035bf0 con 0x55f536c3a340 2024-03-18T00:53:30.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.065+0000 7fd136594740 1 -- 172.21.15.49:0/1571157688 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 msgr2=0x55f536c3a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:30.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.065+0000 7fd136594740 1 --2- 172.21.15.49:0/1571157688 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c3a760 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fd12800aa40 tx=0x7fd128004e30 comp rx=0 tx=0).stop 2024-03-18T00:53:30.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.065+0000 7fd136594740 1 -- 172.21.15.49:0/1571157688 shutdown_connections 2024-03-18T00:53:30.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.065+0000 7fd136594740 1 --2- 172.21.15.49:0/1571157688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 0x55f536c42330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.065+0000 7fd136594740 1 --2- 172.21.15.49:0/1571157688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 0x55f536c3f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.065+0000 7fd136594740 1 --2- 172.21.15.49:0/1571157688 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c3a760 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.065+0000 7fd136594740 1 -- 172.21.15.49:0/1571157688 >> 172.21.15.49:0/1571157688 conn(0x55f536c358b0 msgr2=0x55f536c35cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:30.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.066+0000 7fd136594740 1 -- 172.21.15.49:0/1571157688 shutdown_connections 2024-03-18T00:53:30.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.066+0000 7fd136594740 1 -- 172.21.15.49:0/1571157688 wait complete. 2024-03-18T00:53:30.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.066+0000 7fd136594740 1 -- start start 2024-03-18T00:53:30.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.066+0000 7fd136594740 1 Processor -- start 2024-03-18T00:53:30.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd136594740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd136594740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3ad30 0x55f536c44bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd136594740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 0x55f536c4bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd136594740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f536c42df0 con 0x55f536c3a340 2024-03-18T00:53:30.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd136594740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f536b5b160 con 0x55f536c3ad30 2024-03-18T00:53:30.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd136594740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f536c42f70 con 0x55f536c3fea0 2024-03-18T00:53:30.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd136594740 10 client.0 _ll_register_callbacks cb 0x55f536b3f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:30.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd130f44640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:30.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd130f44640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58052/0 (socket says 172.21.15.49:58052) 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.067+0000 7fd130f44640 1 -- 172.21.15.49:0/1179468556 learned_addr learned my addr 172.21.15.49:0/1179468556 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.068+0000 7fd131745640 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3ad30 0x55f536c44bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.068+0000 7fd130f44640 1 -- 172.21.15.49:0/1179468556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 msgr2=0x55f536c4bbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.068+0000 7fd130f44640 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 0x55f536c4bbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.068+0000 7fd130f44640 1 -- 172.21.15.49:0/1179468556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3ad30 msgr2=0x55f536c44bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.068+0000 7fd130f44640 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3ad30 0x55f536c44bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.068+0000 7fd130f44640 1 -- 172.21.15.49:0/1179468556 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f53697d7d0 con 0x55f536c3a340 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.068+0000 7fd131745640 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3ad30 0x55f536c44bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.069+0000 7fd130f44640 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44670 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd12802d4d0 tx=0x7fd12802dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f5933ecb594daf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:30.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.069+0000 7fd114ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:30.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.069+0000 7fd114ff9640 1 -- 172.21.15.49:0/1179468556 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd128004ad0 con 0x55f536c3a340 2024-03-18T00:53:30.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.069+0000 7fd114ff9640 1 -- 172.21.15.49:0/1179468556 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd1280359b0 con 0x55f536c3a340 2024-03-18T00:53:30.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.069+0000 7fd136594740 1 -- 172.21.15.49:0/1179468556 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f536baca40 con 0x55f536c3a340 2024-03-18T00:53:30.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.070+0000 7fd114ff9640 1 -- 172.21.15.49:0/1179468556 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd128035bb0 con 0x55f536c3a340 2024-03-18T00:53:30.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.070+0000 7fd136594740 10 client.16203 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:30.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.070+0000 7fd136594740 1 -- 172.21.15.49:0/1179468556 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f5368b33f0 con 0x55f536c3a340 2024-03-18T00:53:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.070+0000 7fd114ff9640 1 -- 172.21.15.49:0/1179468556 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f536baca40 con 0x55f536c3a340 2024-03-18T00:53:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.070+0000 7fd114ff9640 1 -- 172.21.15.49:0/1179468556 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd12800b040 con 0x55f536c3a340 2024-03-18T00:53:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.071+0000 7fd136594740 10 client.16203 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.071+0000 7fd136594740 10 client.16203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.071+0000 7fd136594740 20 client.16203 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.071+0000 7fd136594740 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f536c50180 0x55f536c525a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.071+0000 7fd136594740 4 client.16203 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:30.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.071+0000 7fd136594740 1 -- 172.21.15.49:0/1179468556 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f536ba6d90 con 0x55f536c50180 2024-03-18T00:53:30.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.073+0000 7fd131745640 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f536c50180 0x55f536c525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:30.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.075+0000 7fd131745640 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f536c50180 0x55f536c525a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c1124472286daad server_cookie=c50e41fc41e66176 in_seq=0 out_seq=0 2024-03-18T00:53:30.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:30.075+0000 7fd114ff9640 10 client.16203 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:31.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:31 smithi049 ceph-mon[32130]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 964 MiB/s wr, 1.69k op/s 2024-03-18T00:53:31.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:31 smithi107 ceph-mon[36270]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 964 MiB/s wr, 1.69k op/s 2024-03-18T00:53:31.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:31 smithi106 ceph-mon[35899]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 964 MiB/s wr, 1.69k op/s 2024-03-18T00:53:32.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:32 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:53:32.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:32 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:53:32.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:32 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:53:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.829+0000 7fca02ff5640 1 -- 172.21.15.49:0/178332652 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f39e95a2d0 con 0x55f39ea7a940 2024-03-18T00:53:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.829+0000 7fca02ff5640 10 client.16191 handle_command_reply: tid=0 2024-03-18T00:53:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.829+0000 7fca281bd740 1 client.16191 shutdown 2024-03-18T00:53:32.830 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.829+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 msgr2=0x55f39ea0d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:32.831 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.829+0000 7fca281bd740 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea0d130 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fca1800a740 tx=0x7fca18035cc0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.830+0000 7fd114ff9640 1 -- 172.21.15.49:0/1179468556 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f536ba6d90 con 0x55f536c50180 2024-03-18T00:53:32.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.830+0000 7fd114ff9640 10 client.16203 handle_command_reply: tid=0 2024-03-18T00:53:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 shutdown_connections 2024-03-18T00:53:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca281bd740 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f39ea7a940 0x55f39ea7cd60 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca281bd740 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f39ea04070 0x55f39ea78b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca281bd740 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f39e9fef00 0x55f39ea0d130 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca281bd740 1 --2- 172.21.15.49:0/178332652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f39e9fe510 0x55f39ea0cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca02ff5640 0 client.16191 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 >> 172.21.15.49:0/178332652 conn(0x55f39e9e8a70 msgr2=0x55f39ea00280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:32.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.830+0000 7fd136594740 1 client.16203 shutdown 2024-03-18T00:53:32.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.831+0000 7fd136594740 1 -- 172.21.15.49:0/1179468556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 msgr2=0x55f536c44670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:32.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.831+0000 7fd136594740 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44670 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd12802d4d0 tx=0x7fd12802dd70 comp rx=0 tx=0).stop 2024-03-18T00:53:32.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.831+0000 7fd136594740 1 -- 172.21.15.49:0/1179468556 shutdown_connections 2024-03-18T00:53:32.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.831+0000 7fd136594740 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f536c50180 0x55f536c525a0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.831+0000 7fd136594740 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3ad30 0x55f536c44bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.831+0000 7fd136594740 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 0x55f536c4bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.831+0000 7fd136594740 1 --2- 172.21.15.49:0/1179468556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44670 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.831+0000 7fd136594740 1 -- 172.21.15.49:0/1179468556 >> 172.21.15.49:0/1179468556 conn(0x55f536c358b0 msgr2=0x55f536c34a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:32.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.831+0000 7fd114ff9640 0 client.16203 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 shutdown_connections 2024-03-18T00:53:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca281bd740 1 -- 172.21.15.49:0/178332652 wait complete. 2024-03-18T00:53:32.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.832+0000 7fca281bd740 20 client.16191 trim_cache size 0 max 16384 2024-03-18T00:53:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:53:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:53:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:53:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:53:32.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.833+0000 7fd136594740 1 -- 172.21.15.49:0/1179468556 shutdown_connections 2024-03-18T00:53:32.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.833+0000 7fd136594740 1 -- 172.21.15.49:0/1179468556 wait complete. 2024-03-18T00:53:32.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.833+0000 7fd136594740 20 client.16203 trim_cache size 0 max 16384 2024-03-18T00:53:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.834+0000 7fca255da640 1 -- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca080a2a00 msgr2=0x7fca080a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.834+0000 7fca255da640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca080a2a00 0x7fca080a4ec0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fca18037040 tx=0x7fca18005ed0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.834+0000 7fca255da640 1 -- 172.21.15.49:0/1765850190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca20108870 msgr2=0x7fca201a66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:32.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.834+0000 7fca255da640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca20108870 0x7fca201a66b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fca1000c890 tx=0x7fca1000cd50 comp rx=0 tx=0).stop 2024-03-18T00:53:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.840+0000 7fca255da640 1 -- 172.21.15.49:0/1765850190 shutdown_connections 2024-03-18T00:53:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.840+0000 7fca255da640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fca080a2a00 0x7fca080a4ec0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.840+0000 7fca255da640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fca2010dd10 0x7fca201adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.840+0000 7fca255da640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca2010b230 0x7fca201a6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.840+0000 7fca255da640 1 --2- 172.21.15.49:0/1765850190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fca20108870 0x7fca201a66b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.840+0000 7fca255da640 1 -- 172.21.15.49:0/1765850190 >> 172.21.15.49:0/1765850190 conn(0x7fca2006e860 msgr2=0x7fca20109110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.842+0000 7fca255da640 1 -- 172.21.15.49:0/1765850190 shutdown_connections 2024-03-18T00:53:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:32.842+0000 7fca255da640 1 -- 172.21.15.49:0/1765850190 wait complete. 2024-03-18T00:53:32.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.955+0000 7fd136594740 1 Processor -- start 2024-03-18T00:53:32.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.955+0000 7fd136594740 1 -- start start 2024-03-18T00:53:32.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.961+0000 7fd136594740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:32.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.961+0000 7fd136594740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 0x55f536c4bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:32.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.961+0000 7fd136594740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 0x55f536c308f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:32.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.961+0000 7fd136594740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1b3540 con 0x55f536c3a340 2024-03-18T00:53:32.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.961+0000 7fd136594740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1b5080 con 0x55f536c3fea0 2024-03-18T00:53:32.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.961+0000 7fd136594740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1b3c10 con 0x55f536c3ad30 2024-03-18T00:53:32.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.967+0000 7fd131745640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 0x55f536c4bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:32.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.967+0000 7fd130f44640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:32.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.967+0000 7fd130f44640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58054/0 (socket says 172.21.15.49:58054) 2024-03-18T00:53:32.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.967+0000 7fd130f44640 1 -- 172.21.15.49:0/1116015622 learned_addr learned my addr 172.21.15.49:0/1116015622 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:32.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.967+0000 7fd130f44640 1 --2- 172.21.15.49:0/1116015622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 0x55f536c308f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:32.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.968+0000 7fd130f44640 1 -- 172.21.15.49:0/1116015622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 msgr2=0x55f536c4bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:32.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.968+0000 7fd130f44640 1 --2- 172.21.15.49:0/1116015622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 0x55f536c4bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.968+0000 7fd130f44640 1 -- 172.21.15.49:0/1116015622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 msgr2=0x55f536c308f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:32.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.968+0000 7fd130f44640 1 --2- 172.21.15.49:0/1116015622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 0x55f536c308f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.968+0000 7fd130f44640 1 -- 172.21.15.49:0/1116015622 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5368730f0 con 0x55f536c3a340 2024-03-18T00:53:32.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.969+0000 7fd130f44640 1 --2- 172.21.15.49:0/1116015622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44b10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd128004e90 tx=0x7fd128001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1534ba4c1e76fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:32.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.970+0000 7fd116ffd640 1 -- 172.21.15.49:0/1116015622 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12800a630 con 0x55f536c3a340 2024-03-18T00:53:32.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.970+0000 7fd116ffd640 1 -- 172.21.15.49:0/1116015622 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd128004490 con 0x55f536c3a340 2024-03-18T00:53:32.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.970+0000 7fd116ffd640 1 -- 172.21.15.49:0/1116015622 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd128004b30 con 0x55f536c3a340 2024-03-18T00:53:32.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.972+0000 7fd136594740 1 -- 172.21.15.49:0/1116015622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 msgr2=0x55f536c44b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:32.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.972+0000 7fd136594740 1 --2- 172.21.15.49:0/1116015622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44b10 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fd128004e90 tx=0x7fd128001e70 comp rx=0 tx=0).stop 2024-03-18T00:53:32.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.975+0000 7fd136594740 1 -- 172.21.15.49:0/1116015622 shutdown_connections 2024-03-18T00:53:32.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.975+0000 7fd136594740 1 --2- 172.21.15.49:0/1116015622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3fea0 0x55f536c308f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.975+0000 7fd136594740 1 --2- 172.21.15.49:0/1116015622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3ad30 0x55f536c4bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.975+0000 7fd136594740 1 --2- 172.21.15.49:0/1116015622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3a340 0x55f536c44b10 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.975+0000 7fd136594740 1 -- 172.21.15.49:0/1116015622 >> 172.21.15.49:0/1116015622 conn(0x55f536c358b0 msgr2=0x55f536c341d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:32.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.976+0000 7fd136594740 1 -- 172.21.15.49:0/1116015622 shutdown_connections 2024-03-18T00:53:32.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.976+0000 7fd136594740 1 -- 172.21.15.49:0/1116015622 wait complete. 2024-03-18T00:53:32.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.976+0000 7fd136594740 1 -- start start 2024-03-18T00:53:32.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.976+0000 7fd136594740 1 Processor -- start 2024-03-18T00:53:32.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.977+0000 7fd136594740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3a340 0x55f536c49af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:32.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.977+0000 7fd136594740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3ad30 0x55f536c4a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.977+0000 7fd136594740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 0x55f536c4afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.977+0000 7fd136594740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1b49b0 con 0x55f536c3ad30 2024-03-18T00:53:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.977+0000 7fd136594740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1b42e0 con 0x55f536c3a340 2024-03-18T00:53:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.977+0000 7fd136594740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd12c1b1520 con 0x55f536c3fea0 2024-03-18T00:53:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.978+0000 7fd136594740 10 client.0 _ll_register_callbacks cb 0x7fd11c005f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.978+0000 7fd130f44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 0x55f536c4afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:32.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.978+0000 7fd130f44640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 0x55f536c4afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:48316/0 (socket says 172.21.15.49:48316) 2024-03-18T00:53:32.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.978+0000 7fd130f44640 1 -- 172.21.15.49:0/2367318285 learned_addr learned my addr 172.21.15.49:0/2367318285 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:32.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.978+0000 7fd130f44640 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3a340 0x55f536c49af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:32.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.979+0000 7fd131745640 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3ad30 0x55f536c4a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:32.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.980+0000 7fd130f44640 1 -- 172.21.15.49:0/2367318285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3a340 msgr2=0x55f536c49af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:32.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.980+0000 7fd130f44640 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3a340 0x55f536c49af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.980+0000 7fd130f44640 1 -- 172.21.15.49:0/2367318285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3ad30 msgr2=0x55f536c4a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:32.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.980+0000 7fd130f44640 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3ad30 0x55f536c4a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:32.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.980+0000 7fd130f44640 1 -- 172.21.15.49:0/2367318285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f536b669d0 con 0x55f536c3fea0 2024-03-18T00:53:32.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.981+0000 7fd130f44640 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 0x55f536c4afd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd128003ae0 tx=0x7fd128005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=997fe857d2e09a6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:32.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.981+0000 7fd114ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:53:32.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.981+0000 7fd114ff9640 1 -- 172.21.15.49:0/2367318285 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd128003b60 con 0x55f536c3fea0 2024-03-18T00:53:32.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.982+0000 7fd136594740 1 -- 172.21.15.49:0/2367318285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd128004490 con 0x55f536c3fea0 2024-03-18T00:53:32.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.982+0000 7fd114ff9640 1 -- 172.21.15.49:0/2367318285 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd128003d00 con 0x55f536c3fea0 2024-03-18T00:53:32.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.982+0000 7fd114ff9640 1 -- 172.21.15.49:0/2367318285 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd128003f00 con 0x55f536c3fea0 2024-03-18T00:53:32.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.983+0000 7fd136594740 10 client.25873 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:32.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.983+0000 7fd136594740 1 -- 172.21.15.49:0/2367318285 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f536cb09b0 con 0x55f536c3fea0 2024-03-18T00:53:32.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.984+0000 7fd114ff9640 1 -- 172.21.15.49:0/2367318285 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd128004490 con 0x55f536c3fea0 2024-03-18T00:53:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.989+0000 7fd114ff9640 1 -- 172.21.15.49:0/2367318285 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd12805d700 con 0x55f536c3fea0 2024-03-18T00:53:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.989+0000 7fd136594740 10 client.25873 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.989+0000 7fd136594740 10 client.25873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.989+0000 7fd136594740 20 client.25873 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.989+0000 7fd136594740 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f536cb12c0 0x55f536cb36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.989+0000 7fd136594740 4 client.25873 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:53:32.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.989+0000 7fd136594740 1 -- 172.21.15.49:0/2367318285 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f536b24540 con 0x55f536cb12c0 2024-03-18T00:53:32.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:32.993+0000 7fd131745640 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f536cb12c0 0x55f536cb36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:33.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.002+0000 7fd131745640 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f536cb12c0 0x55f536cb36e0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5de89f472ec10818 server_cookie=678c11931b7cfff3 in_seq=0 out_seq=0 2024-03-18T00:53:33.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.006+0000 7fd114ff9640 10 client.25873 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:33.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.010+0000 7fd114ff9640 1 -- 172.21.15.49:0/2367318285 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f536b24540 con 0x55f536cb12c0 2024-03-18T00:53:33.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.011+0000 7fd114ff9640 10 client.25873 handle_command_reply: tid=0 2024-03-18T00:53:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.011+0000 7fd136594740 1 client.25873 shutdown 2024-03-18T00:53:33.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.011+0000 7fd136594740 1 -- 172.21.15.49:0/2367318285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 msgr2=0x55f536c4afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.011+0000 7fd136594740 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 0x55f536c4afd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd128003ae0 tx=0x7fd128005d50 comp rx=0 tx=0).stop 2024-03-18T00:53:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.012+0000 7fd136594740 1 -- 172.21.15.49:0/2367318285 shutdown_connections 2024-03-18T00:53:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.012+0000 7fd136594740 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f536cb12c0 0x55f536cb36e0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.012+0000 7fd136594740 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f536c3fea0 0x55f536c4afd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.012+0000 7fd136594740 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f536c3ad30 0x55f536c4a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.012+0000 7fd136594740 1 --2- 172.21.15.49:0/2367318285 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f536c3a340 0x55f536c49af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.012+0000 7fd136594740 1 -- 172.21.15.49:0/2367318285 >> 172.21.15.49:0/2367318285 conn(0x55f536c358b0 msgr2=0x55f536c34a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:33.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.012+0000 7fd114ff9640 0 client.25873 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.015+0000 7fd136594740 1 -- 172.21.15.49:0/2367318285 shutdown_connections 2024-03-18T00:53:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.015+0000 7fd136594740 1 -- 172.21.15.49:0/2367318285 wait complete. 2024-03-18T00:53:33.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.015+0000 7fd136594740 20 client.25873 trim_cache size 0 max 16384 2024-03-18T00:53:33.016 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:53:33.016 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:53:33.017 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:53:33.017 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:53:33.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.024+0000 7fd132747640 1 -- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1180a2930 msgr2=0x7fd1180a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:33.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.024+0000 7fd132747640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1180a2930 0x7fd1180a4df0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd1240a2bd0 tx=0x7fd12400e040 comp rx=0 tx=0).stop 2024-03-18T00:53:33.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.024+0000 7fd132747640 1 -- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c109b40 msgr2=0x7fd12c1a99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:33.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.024+0000 7fd132747640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c109b40 0x7fd12c1a99c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd12400b990 tx=0x7fd12400be50 comp rx=0 tx=0).stop 2024-03-18T00:53:33.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.025+0000 7fd132747640 1 -- 172.21.15.49:0/1559343965 shutdown_connections 2024-03-18T00:53:33.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.025+0000 7fd132747640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd1180a2930 0x7fd1180a4df0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.025+0000 7fd132747640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd12c109b40 0x7fd12c1a99c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.025+0000 7fd132747640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd12c1091a0 0x7fd12c1a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.025+0000 7fd132747640 1 --2- 172.21.15.49:0/1559343965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd12c1087e0 0x7fd12c1a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.025+0000 7fd132747640 1 -- 172.21.15.49:0/1559343965 >> 172.21.15.49:0/1559343965 conn(0x7fd12c06e8b0 msgr2=0x7fd12c10caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:33.032 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.032+0000 7fd132747640 1 -- 172.21.15.49:0/1559343965 shutdown_connections 2024-03-18T00:53:33.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.032+0000 7fd132747640 1 -- 172.21.15.49:0/1559343965 wait complete. 2024-03-18T00:53:33.093 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:53:33.094 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:53:33.378 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:33 smithi049 ceph-mon[32130]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 827 MiB/s wr, 1.27k op/s 2024-03-18T00:53:33.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:33 smithi107 ceph-mon[36270]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 827 MiB/s wr, 1.27k op/s 2024-03-18T00:53:33.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:33 smithi106 ceph-mon[35899]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 827 MiB/s wr, 1.27k op/s 2024-03-18T00:53:33.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.531+0000 7f6eb1227640 1 -- 172.21.15.49:0/639528851 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10b150 msgr2=0x7f6eac10d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:33.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.531+0000 7f6eb1227640 1 --2- 172.21.15.49:0/639528851 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10b150 0x7f6eac10d5e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c009900 tx=0x7f6e9c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:53:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.537+0000 7f6eb1227640 1 -- 172.21.15.49:0/639528851 shutdown_connections 2024-03-18T00:53:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.537+0000 7f6eb1227640 1 --2- 172.21.15.49:0/639528851 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6eac10db20 0x7f6eac10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.537+0000 7f6eb1227640 1 --2- 172.21.15.49:0/639528851 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10b150 0x7f6eac10d5e0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.537+0000 7f6eb1227640 1 --2- 172.21.15.49:0/639528851 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eac1087e0 0x7f6eac10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.537+0000 7f6eb1227640 1 -- 172.21.15.49:0/639528851 >> 172.21.15.49:0/639528851 conn(0x7f6eac06e650 msgr2=0x7f6eac070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.542+0000 7f6eb1227640 1 -- 172.21.15.49:0/639528851 shutdown_connections 2024-03-18T00:53:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.543+0000 7f6eb1227640 1 -- 172.21.15.49:0/639528851 wait complete. 2024-03-18T00:53:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.544+0000 7f6eb1227640 1 Processor -- start 2024-03-18T00:53:33.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.551+0000 7f6eb1227640 1 -- start start 2024-03-18T00:53:33.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.552+0000 7f6eb1227640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6eac1087e0 0x7f6eac119ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:33.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.552+0000 7f6eb1227640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eac10b150 0x7f6eac119ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:33.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.552+0000 7f6eb1227640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10db20 0x7f6eac1180f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:33.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.552+0000 7f6eb1227640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac110960 con 0x7f6eac10db20 2024-03-18T00:53:33.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.552+0000 7f6eb1227640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac1107e0 con 0x7f6eac1087e0 2024-03-18T00:53:33.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.552+0000 7f6eb1227640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac110ae0 con 0x7f6eac10b150 2024-03-18T00:53:33.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.554+0000 7f6eab59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10db20 0x7f6eac1180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:33.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.554+0000 7f6eab59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10db20 0x7f6eac1180f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58072/0 (socket says 172.21.15.49:58072) 2024-03-18T00:53:33.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.554+0000 7f6eab59e640 1 -- 172.21.15.49:0/3572897 learned_addr learned my addr 172.21.15.49:0/3572897 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:33.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.558+0000 7f6eaad9d640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6eac1087e0 0x7f6eac119ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:33.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.559+0000 7f6eaa59c640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eac10b150 0x7f6eac119ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:33.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.561+0000 7f6eab59e640 1 -- 172.21.15.49:0/3572897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eac10b150 msgr2=0x7f6eac119ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:33.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.561+0000 7f6eab59e640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eac10b150 0x7f6eac119ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.561+0000 7f6eab59e640 1 -- 172.21.15.49:0/3572897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6eac1087e0 msgr2=0x7f6eac119ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:33.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.561+0000 7f6eab59e640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6eac1087e0 0x7f6eac119ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:33.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.561+0000 7f6eab59e640 1 -- 172.21.15.49:0/3572897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac1188c0 con 0x7f6eac10db20 2024-03-18T00:53:33.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.565+0000 7f6eab59e640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10db20 0x7f6eac1180f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000b3c0 tx=0x7f6ea000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:33.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.569+0000 7f6e93fff640 1 -- 172.21.15.49:0/3572897 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0011830 con 0x7f6eac10db20 2024-03-18T00:53:33.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.569+0000 7f6e93fff640 1 -- 172.21.15.49:0/3572897 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ea0016070 con 0x7f6eac10db20 2024-03-18T00:53:33.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.569+0000 7f6e93fff640 1 -- 172.21.15.49:0/3572897 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000f2e0 con 0x7f6eac10db20 2024-03-18T00:53:33.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.571+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eac118bb0 con 0x7f6eac10db20 2024-03-18T00:53:33.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.571+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eac1b7700 con 0x7f6eac10db20 2024-03-18T00:53:33.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.573+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eac1122a0 con 0x7f6eac10db20 2024-03-18T00:53:33.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.578+0000 7f6e93fff640 1 -- 172.21.15.49:0/3572897 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6ea001a410 con 0x7f6eac10db20 2024-03-18T00:53:33.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.579+0000 7f6e93fff640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6e940a2a00 0x7f6e940a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:33.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.579+0000 7f6e93fff640 1 -- 172.21.15.49:0/3572897 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6ea00ce730 con 0x7f6eac10db20 2024-03-18T00:53:33.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.580+0000 7f6eaad9d640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6e940a2a00 0x7f6e940a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:33.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.581+0000 7f6eaad9d640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6e940a2a00 0x7f6e940a4ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0728d0 tx=0x7f6ea400d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:33.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:33.590+0000 7f6e93fff640 1 -- 172.21.15.49:0/3572897 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6ea001f020 con 0x7f6eac10db20 2024-03-18T00:53:34.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.329+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6eac1b7990 con 0x7f6eac10db20 2024-03-18T00:53:34.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.330+0000 7f6e93fff640 1 -- 172.21.15.49:0/3572897 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f6ea009b860 con 0x7f6eac10db20 2024-03-18T00:53:34.331 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:53:34.331 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:53:25.016081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24470,24463,24481,24506,24499]},"id":1}]} 2024-03-18T00:53:34.332 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 27 2024-03-18T00:53:34.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6e940a2a00 msgr2=0x7f6e940a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6e940a2a00 0x7f6e940a4ec0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0728d0 tx=0x7f6ea400d040 comp rx=0 tx=0).stop 2024-03-18T00:53:34.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10db20 msgr2=0x7f6eac1180f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10db20 0x7f6eac1180f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6ea000b3c0 tx=0x7f6ea000b880 comp rx=0 tx=0).stop 2024-03-18T00:53:34.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 shutdown_connections 2024-03-18T00:53:34.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6e940a2a00 0x7f6e940a4ec0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6eac10db20 0x7f6eac1180f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6eac10b150 0x7f6eac119ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 --2- 172.21.15.49:0/3572897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6eac1087e0 0x7f6eac119ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.336+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 >> 172.21.15.49:0/3572897 conn(0x7f6eac06e650 msgr2=0x7f6eac10b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:34.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.337+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 shutdown_connections 2024-03-18T00:53:34.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.337+0000 7f6eb1227640 1 -- 172.21.15.49:0/3572897 wait complete. 2024-03-18T00:53:34.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:53:34.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:53:34.366 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:34 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:53:34.370 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:53:34.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:53:34.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:53:34.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:34 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:53:34.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:53:34.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:53:34.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:34 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:53:34.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.795+0000 7f17eed91640 1 -- 172.21.15.49:0/4210764342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 msgr2=0x7f17e8108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.795+0000 7f17eed91640 1 --2- 172.21.15.49:0/4210764342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 0x7f17e8108a00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009ca0 tx=0x7f17e002c100 comp rx=0 tx=0).stop 2024-03-18T00:53:34.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.795+0000 7f17eed91640 1 -- 172.21.15.49:0/4210764342 shutdown_connections 2024-03-18T00:53:34.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.795+0000 7f17eed91640 1 --2- 172.21.15.49:0/4210764342 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17e81017b0 0x7f17e8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.795+0000 7f17eed91640 1 --2- 172.21.15.49:0/4210764342 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 0x7f17e8108a00 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.795+0000 7f17eed91640 1 --2- 172.21.15.49:0/4210764342 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e81005c0 0x7f17e81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.795+0000 7f17eed91640 1 -- 172.21.15.49:0/4210764342 >> 172.21.15.49:0/4210764342 conn(0x7f17e80fbb50 msgr2=0x7f17e80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:34.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.795+0000 7f17eed91640 1 -- 172.21.15.49:0/4210764342 shutdown_connections 2024-03-18T00:53:34.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.796+0000 7f17eed91640 1 -- 172.21.15.49:0/4210764342 wait complete. 2024-03-18T00:53:34.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.797+0000 7f17eed91640 1 Processor -- start 2024-03-18T00:53:34.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.797+0000 7f17eed91640 1 -- start start 2024-03-18T00:53:34.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.797+0000 7f17eed91640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e81005c0 0x7f17e81956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.797+0000 7f17eed91640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17e81017b0 0x7f17e8195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.797+0000 7f17eed91640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 0x7f17e819cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.797+0000 7f17eed91640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f17e810b220 con 0x7f17e8102170 2024-03-18T00:53:34.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17ee590640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 0x7f17e819cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17ee590640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 0x7f17e819cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58104/0 (socket says 172.21.15.49:58104) 2024-03-18T00:53:34.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17ee590640 1 -- 172.21.15.49:0/27076640 learned_addr learned my addr 172.21.15.49:0/27076640 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:34.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f17e810b0a0 con 0x7f17e81017b0 2024-03-18T00:53:34.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17e810b3a0 con 0x7f17e81005c0 2024-03-18T00:53:34.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17ee590640 1 -- 172.21.15.49:0/27076640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e81005c0 msgr2=0x7f17e81956b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17ee590640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e81005c0 0x7f17e81956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17ee590640 1 -- 172.21.15.49:0/27076640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17e81017b0 msgr2=0x7f17e8195bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17ee590640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17e81017b0 0x7f17e8195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.798+0000 7f17ee590640 1 -- 172.21.15.49:0/27076640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e819d2d0 con 0x7f17e8102170 2024-03-18T00:53:34.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.799+0000 7f17ee590640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 0x7f17e819cc60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f17e00098d0 tx=0x7f17e0032830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:34.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.799+0000 7f17deffd640 1 -- 172.21.15.49:0/27076640 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e0039040 con 0x7f17e8102170 2024-03-18T00:53:34.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.799+0000 7f17deffd640 1 -- 172.21.15.49:0/27076640 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e0044070 con 0x7f17e8102170 2024-03-18T00:53:34.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.799+0000 7f17deffd640 1 -- 172.21.15.49:0/27076640 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e003f440 con 0x7f17e8102170 2024-03-18T00:53:34.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.799+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e819d560 con 0x7f17e8102170 2024-03-18T00:53:34.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.799+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e819dad0 con 0x7f17e8102170 2024-03-18T00:53:34.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.804+0000 7f17f2bdb740 1 Processor -- start 2024-03-18T00:53:34.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.805+0000 7f17deffd640 1 -- 172.21.15.49:0/27076640 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f17e003f5e0 con 0x7f17e8102170 2024-03-18T00:53:34.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.805+0000 7f17f2bdb740 1 -- start start 2024-03-18T00:53:34.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.805+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7250 0x5580e10c7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.806+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7c40 0x5580e10cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.806+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.806+0000 7f17deffd640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17d40a29d0 0x7f17d40a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.806+0000 7f17ed58e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.806+0000 7f17ed58e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10cf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58112/0 (socket says 172.21.15.49:58112) 2024-03-18T00:53:34.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.807+0000 7f17ee590640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17d40a29d0 0x7f17d40a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.807+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5580e1023490 con 0x5580e10ccdb0 2024-03-18T00:53:34.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.807+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5580e0fcbc90 con 0x5580e10c7c40 2024-03-18T00:53:34.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.807+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580e101f530 con 0x5580e10c7250 2024-03-18T00:53:34.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.807+0000 7f17ed58e640 1 -- 172.21.15.49:0/571408734 learned_addr learned my addr 172.21.15.49:0/571408734 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:34.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.807+0000 7f17deffd640 1 -- 172.21.15.49:0/27076640 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f17e00f1640 con 0x7f17e8102170 2024-03-18T00:53:34.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.808+0000 7f17edd8f640 1 --2- 172.21.15.49:0/571408734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7c40 0x5580e10cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.808+0000 7f17ee590640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17d40a29d0 0x7f17d40a4e90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005ff0 tx=0x7f17e0035f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:34.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.808+0000 7f17ed58e640 1 --2- 172.21.15.49:0/571408734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7250 0x5580e10c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.809+0000 7f17ed58e640 1 -- 172.21.15.49:0/571408734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7250 msgr2=0x5580e10c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:34.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.809+0000 7f17ed58e640 1 --2- 172.21.15.49:0/571408734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7250 0x5580e10c7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.809+0000 7f17ed58e640 1 -- 172.21.15.49:0/571408734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7c40 msgr2=0x5580e10cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:34.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.809+0000 7f17ed58e640 1 --2- 172.21.15.49:0/571408734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7c40 0x5580e10cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.809+0000 7f17ed58e640 1 -- 172.21.15.49:0/571408734 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580e1086500 con 0x5580e10ccdb0 2024-03-18T00:53:34.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.810+0000 7f17ed58e640 1 --2- 172.21.15.49:0/571408734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10cf240 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f17d8001910 tx=0x7f17d8001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d2fdd19ae25cce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:34.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.810+0000 7f17d3fff640 1 -- 172.21.15.49:0/571408734 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8006b60 con 0x5580e10ccdb0 2024-03-18T00:53:34.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.810+0000 7f17d3fff640 1 -- 172.21.15.49:0/571408734 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17d8011040 con 0x5580e10ccdb0 2024-03-18T00:53:34.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.811+0000 7f17d3fff640 1 -- 172.21.15.49:0/571408734 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8007770 con 0x5580e10ccdb0 2024-03-18T00:53:34.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.812+0000 7f17f2bdb740 1 -- 172.21.15.49:0/571408734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 msgr2=0x5580e10cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.812+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/571408734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10cf240 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f17d8001910 tx=0x7f17d8001360 comp rx=0 tx=0).stop 2024-03-18T00:53:34.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.814+0000 7f17f2bdb740 1 -- 172.21.15.49:0/571408734 shutdown_connections 2024-03-18T00:53:34.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.814+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/571408734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7c40 0x5580e10cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.814+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/571408734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10cf240 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.814+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/571408734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7250 0x5580e10c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.814+0000 7f17f2bdb740 1 -- 172.21.15.49:0/571408734 >> 172.21.15.49:0/571408734 conn(0x5580e10c27c0 msgr2=0x5580e10c2bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:34.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.815+0000 7f17f2bdb740 1 -- 172.21.15.49:0/571408734 shutdown_connections 2024-03-18T00:53:34.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.815+0000 7f17f2bdb740 1 -- 172.21.15.49:0/571408734 wait complete. 2024-03-18T00:53:34.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.816+0000 7f17f2bdb740 1 -- start start 2024-03-18T00:53:34.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.816+0000 7f17f2bdb740 1 Processor -- start 2024-03-18T00:53:34.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7250 0x5580e10d1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7c40 0x5580e10d1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10ccdb0 0x5580e10d8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5580e10cfd60 con 0x5580e10c7c40 2024-03-18T00:53:34.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5580e0fdf230 con 0x5580e10ccdb0 2024-03-18T00:53:34.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5580e10cfee0 con 0x5580e10c7250 2024-03-18T00:53:34.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17f2bdb740 10 client.0 _ll_register_callbacks cb 0x5580e101b010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:34.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17edd8f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7c40 0x5580e10d1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17edd8f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7c40 0x5580e10d1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58116/0 (socket says 172.21.15.49:58116) 2024-03-18T00:53:34.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.819+0000 7f17edd8f640 1 -- 172.21.15.49:0/3694511465 learned_addr learned my addr 172.21.15.49:0/3694511465 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:34.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.820+0000 7f17edd8f640 1 -- 172.21.15.49:0/3694511465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7250 msgr2=0x5580e10d1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:34.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.820+0000 7f17edd8f640 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7250 0x5580e10d1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.820+0000 7f17edd8f640 1 -- 172.21.15.49:0/3694511465 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10ccdb0 msgr2=0x5580e10d8db0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:34.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.820+0000 7f17edd8f640 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10ccdb0 0x5580e10d8db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.820+0000 7f17edd8f640 1 -- 172.21.15.49:0/3694511465 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580e0fee100 con 0x5580e10c7c40 2024-03-18T00:53:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.821+0000 7f17edd8f640 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7c40 0x5580e10d1d70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f17e400bfb0 tx=0x7f17e400d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d75105deb383ba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.821+0000 7f17d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.821+0000 7f17d17fa640 1 -- 172.21.15.49:0/3694511465 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4014430 con 0x5580e10c7c40 2024-03-18T00:53:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.821+0000 7f17d17fa640 1 -- 172.21.15.49:0/3694511465 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f17e40145d0 con 0x5580e10c7c40 2024-03-18T00:53:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.821+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3694511465 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d8011040 con 0x5580e10c7c40 2024-03-18T00:53:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.822+0000 7f17d17fa640 1 -- 172.21.15.49:0/3694511465 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f17e40147f0 con 0x5580e10c7c40 2024-03-18T00:53:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.822+0000 7f17f2bdb740 10 client.16227 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:34.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.822+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3694511465 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5580e0d373f0 con 0x5580e10c7c40 2024-03-18T00:53:34.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.822+0000 7f17d17fa640 1 -- 172.21.15.49:0/3694511465 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17d8011040 con 0x5580e10c7c40 2024-03-18T00:53:34.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.823+0000 7f17d17fa640 1 -- 172.21.15.49:0/3694511465 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f17e4013070 con 0x5580e10c7c40 2024-03-18T00:53:34.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.823+0000 7f17f2bdb740 10 client.16227 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:34.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.823+0000 7f17f2bdb740 10 client.16227 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:53:34.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.823+0000 7f17f2bdb740 20 client.16227 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:34.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.823+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5580e10dd800 0x5580e10dfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.823+0000 7f17f2bdb740 4 client.16227 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:34.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.823+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3694511465 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580e10309d0 con 0x5580e10dd800 2024-03-18T00:53:34.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.825+0000 7f17ed58e640 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5580e10dd800 0x5580e10dfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.826+0000 7f17ed58e640 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5580e10dd800 0x5580e10dfca0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=405a225839a23e0b server_cookie=b1ce4488f2b4ae2d in_seq=0 out_seq=0 2024-03-18T00:53:34.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.826+0000 7f17d17fa640 10 client.16227 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:53:34.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.828+0000 7f17d17fa640 1 -- 172.21.15.49:0/3694511465 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5580e10309d0 con 0x5580e10dd800 2024-03-18T00:53:34.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.828+0000 7f17d17fa640 10 client.16227 handle_command_reply: tid=0 2024-03-18T00:53:34.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.828+0000 7f17f2bdb740 1 client.16227 shutdown 2024-03-18T00:53:34.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3694511465 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7c40 msgr2=0x5580e10d1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7c40 0x5580e10d1d70 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f17e400bfb0 tx=0x7f17e400d5f0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3694511465 shutdown_connections 2024-03-18T00:53:34.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5580e10dd800 0x5580e10dfca0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10ccdb0 0x5580e10d8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7c40 0x5580e10d1d70 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3694511465 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7250 0x5580e10d1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3694511465 >> 172.21.15.49:0/3694511465 conn(0x5580e10c27c0 msgr2=0x5580e10b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:34.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17d17fa640 0 client.16227 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:53:34.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3694511465 shutdown_connections 2024-03-18T00:53:34.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3694511465 wait complete. 2024-03-18T00:53:34.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.829+0000 7f17f2bdb740 20 client.16227 trim_cache size 0 max 16384 2024-03-18T00:53:34.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.860+0000 7f17f2bdb740 1 Processor -- start 2024-03-18T00:53:34.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.861+0000 7f17f2bdb740 1 -- start start 2024-03-18T00:53:34.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.861+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7250 0x5580e10c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.862+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10c9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.862+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10ccdb0 0x5580e10d8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.862+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81a67f0 con 0x5580e10c7250 2024-03-18T00:53:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.862+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81a8330 con 0x5580e10ccdb0 2024-03-18T00:53:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.862+0000 7f17edd8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.862+0000 7f17edd8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10c9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:48388/0 (socket says 172.21.15.49:48388) 2024-03-18T00:53:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.862+0000 7f17edd8f640 1 -- 172.21.15.49:0/1283665354 learned_addr learned my addr 172.21.15.49:0/1283665354 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.862+0000 7f17f2bdb740 1 -- 172.21.15.49:0/1283665354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81a6ec0 con 0x5580e10c7c40 2024-03-18T00:53:34.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.863+0000 7f17ed58e640 1 --2- 172.21.15.49:0/1283665354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10ccdb0 0x5580e10d8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.864+0000 7f17edd8f640 1 -- 172.21.15.49:0/1283665354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10ccdb0 msgr2=0x5580e10d8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:34.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.864+0000 7f17edd8f640 1 --2- 172.21.15.49:0/1283665354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10ccdb0 0x5580e10d8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.864+0000 7f17edd8f640 1 -- 172.21.15.49:0/1283665354 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7250 msgr2=0x5580e10c9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:34.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.864+0000 7f17edd8f640 1 --2- 172.21.15.49:0/1283665354 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7250 0x5580e10c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.864+0000 7f17edd8f640 1 -- 172.21.15.49:0/1283665354 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580e0e017d0 con 0x5580e10c7c40 2024-03-18T00:53:34.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.872+0000 7f17edd8f640 1 --2- 172.21.15.49:0/1283665354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10c9750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f17e4001eb0 tx=0x7f17e4006e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd0b80f6885439db server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:34.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.874+0000 7f17d37fe640 1 -- 172.21.15.49:0/1283665354 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4014430 con 0x5580e10c7c40 2024-03-18T00:53:34.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.874+0000 7f17d37fe640 1 -- 172.21.15.49:0/1283665354 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e4019070 con 0x5580e10c7c40 2024-03-18T00:53:34.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.876+0000 7f17f2bdb740 1 -- 172.21.15.49:0/1283665354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 msgr2=0x5580e10c9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.876+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/1283665354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10c9750 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f17e4001eb0 tx=0x7f17e4006e70 comp rx=0 tx=0).stop 2024-03-18T00:53:34.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.877+0000 7f17f2bdb740 1 -- 172.21.15.49:0/1283665354 shutdown_connections 2024-03-18T00:53:34.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.877+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/1283665354 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10ccdb0 0x5580e10d8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.877+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/1283665354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10c9750 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.877+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/1283665354 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10c7250 0x5580e10c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.877+0000 7f17f2bdb740 1 -- 172.21.15.49:0/1283665354 >> 172.21.15.49:0/1283665354 conn(0x5580e10c27c0 msgr2=0x5580e10d1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:34.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.878+0000 7f17f2bdb740 1 -- 172.21.15.49:0/1283665354 shutdown_connections 2024-03-18T00:53:34.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.878+0000 7f17f2bdb740 1 -- 172.21.15.49:0/1283665354 wait complete. 2024-03-18T00:53:34.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.879+0000 7f17f2bdb740 1 -- start start 2024-03-18T00:53:34.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.879+0000 7f17f2bdb740 1 Processor -- start 2024-03-18T00:53:34.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.880+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7250 0x5580e10d8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.880+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10d5550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.880+0000 7f17f2bdb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10d5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.880+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81a7c60 con 0x5580e10ccdb0 2024-03-18T00:53:34.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.880+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81a7590 con 0x5580e10c7250 2024-03-18T00:53:34.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.880+0000 7f17f2bdb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17e81a47d0 con 0x5580e10c7c40 2024-03-18T00:53:34.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.880+0000 7f17f2bdb740 10 client.0 _ll_register_callbacks cb 0x7f17e400b0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:34.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.882+0000 7f17edd8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10d5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.882+0000 7f17edd8f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10d5550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:48394/0 (socket says 172.21.15.49:48394) 2024-03-18T00:53:34.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.882+0000 7f17edd8f640 1 -- 172.21.15.49:0/3473316676 learned_addr learned my addr 172.21.15.49:0/3473316676 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:34.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.883+0000 7f17ed58e640 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7250 0x5580e10d8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.883+0000 7f17ed58e640 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10d5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.884+0000 7f17edd8f640 1 -- 172.21.15.49:0/3473316676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7250 msgr2=0x5580e10d8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:34.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.884+0000 7f17edd8f640 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7250 0x5580e10d8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.884+0000 7f17edd8f640 1 -- 172.21.15.49:0/3473316676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 msgr2=0x5580e10d5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:34.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.884+0000 7f17edd8f640 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10d5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.884+0000 7f17edd8f640 1 -- 172.21.15.49:0/3473316676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580e10d6100 con 0x5580e10c7c40 2024-03-18T00:53:34.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.887+0000 7f17edd8f640 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10d5550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f17e4012c50 tx=0x7f17e400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5215b89157bb295b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:34.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.888+0000 7f17d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:53:34.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.901+0000 7f17d17fa640 1 -- 172.21.15.49:0/3473316676 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e402d050 con 0x5580e10c7c40 2024-03-18T00:53:34.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.901+0000 7f17d17fa640 1 -- 172.21.15.49:0/3473316676 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f17e4014ba0 con 0x5580e10c7c40 2024-03-18T00:53:34.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.902+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3473316676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e4019070 con 0x5580e10c7c40 2024-03-18T00:53:34.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.905+0000 7f17d17fa640 1 -- 172.21.15.49:0/3473316676 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f17e4014de0 con 0x5580e10c7c40 2024-03-18T00:53:34.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.905+0000 7f17f2bdb740 10 client.25882 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:34.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.905+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3473316676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5580e1153ca0 con 0x5580e10c7c40 2024-03-18T00:53:34.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.912+0000 7f17d17fa640 1 -- 172.21.15.49:0/3473316676 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e4019070 con 0x5580e10c7c40 2024-03-18T00:53:34.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.912+0000 7f17d17fa640 1 -- 172.21.15.49:0/3473316676 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f17e4030070 con 0x5580e10c7c40 2024-03-18T00:53:34.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.912+0000 7f17f2bdb740 10 client.25882 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:34.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.912+0000 7f17f2bdb740 10 client.25882 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:53:34.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.912+0000 7f17f2bdb740 20 client.25882 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:34.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.912+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5580e11544f0 0x5580e1156910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:34.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.912+0000 7f17f2bdb740 4 client.25882 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:53:34.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.912+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3473316676 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5580e102ad20 con 0x5580e11544f0 2024-03-18T00:53:34.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.922+0000 7f17ed58e640 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5580e11544f0 0x5580e1156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:34.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.939+0000 7f17ed58e640 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5580e11544f0 0x5580e1156910 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=66d4f4a532c0d86b server_cookie=750bafecd068e1d0 in_seq=0 out_seq=0 2024-03-18T00:53:34.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.939+0000 7f17d17fa640 10 client.25882 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:53:34.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.945+0000 7f17d17fa640 1 -- 172.21.15.49:0/3473316676 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5580e102ad20 con 0x5580e11544f0 2024-03-18T00:53:34.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.946+0000 7f17d17fa640 10 client.25882 handle_command_reply: tid=0 2024-03-18T00:53:34.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.948+0000 7f17f2bdb740 1 client.25882 shutdown 2024-03-18T00:53:34.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.952+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3473316676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 msgr2=0x5580e10d5550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.952+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10d5550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f17e4012c50 tx=0x7f17e400b8a0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.952+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3473316676 shutdown_connections 2024-03-18T00:53:34.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.952+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5580e11544f0 0x5580e1156910 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.952+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5580e10ccdb0 0x5580e10d5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.952+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5580e10c7c40 0x5580e10d5550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.952+0000 7f17f2bdb740 1 --2- 172.21.15.49:0/3473316676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5580e10c7250 0x5580e10d8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.952+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3473316676 >> 172.21.15.49:0/3473316676 conn(0x5580e10c27c0 msgr2=0x5580e10d1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:34.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.953+0000 7f17d17fa640 0 client.25882 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:53:34.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.953+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3473316676 shutdown_connections 2024-03-18T00:53:34.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.956+0000 7f17f2bdb740 1 -- 172.21.15.49:0/3473316676 wait complete. 2024-03-18T00:53:34.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.963+0000 7f17f2bdb740 20 client.25882 trim_cache size 0 max 16384 2024-03-18T00:53:34.966 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:53:34.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.972+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17d40a29d0 msgr2=0x7f17d40a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.972+0000 7f17eed91640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17d40a29d0 0x7f17d40a4e90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005ff0 tx=0x7f17e0035f10 comp rx=0 tx=0).stop 2024-03-18T00:53:34.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.972+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 msgr2=0x7f17e819cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:34.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.972+0000 7f17eed91640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 0x7f17e819cc60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f17e00098d0 tx=0x7f17e0032830 comp rx=0 tx=0).stop 2024-03-18T00:53:34.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.976+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 shutdown_connections 2024-03-18T00:53:34.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.976+0000 7f17eed91640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17d40a29d0 0x7f17d40a4e90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.976+0000 7f17eed91640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17e81017b0 0x7f17e8195bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.976+0000 7f17eed91640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17e8102170 0x7f17e819cc60 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.976+0000 7f17eed91640 1 --2- 172.21.15.49:0/27076640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17e81005c0 0x7f17e81956b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:34.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.976+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 >> 172.21.15.49:0/27076640 conn(0x7f17e80fbb50 msgr2=0x7f17e81049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:34.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.983+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 shutdown_connections 2024-03-18T00:53:34.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:34.985+0000 7f17eed91640 1 -- 172.21.15.49:0/27076640 wait complete. 2024-03-18T00:53:35.031 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:53:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.361+0000 7f8f899f3640 1 -- 172.21.15.49:0/867168608 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 msgr2=0x7f8f84101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.361+0000 7f8f899f3640 1 --2- 172.21.15.49:0/867168608 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 0x7f8f84101d90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009900 tx=0x7f8f7802bc30 comp rx=0 tx=0).stop 2024-03-18T00:53:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.361+0000 7f8f899f3640 1 -- 172.21.15.49:0/867168608 shutdown_connections 2024-03-18T00:53:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.361+0000 7f8f899f3640 1 --2- 172.21.15.49:0/867168608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f841022d0 0x7f8f84108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.361+0000 7f8f899f3640 1 --2- 172.21.15.49:0/867168608 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 0x7f8f84101d90 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.361+0000 7f8f899f3640 1 --2- 172.21.15.49:0/867168608 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f84100c00 0x7f8f84101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.361+0000 7f8f899f3640 1 -- 172.21.15.49:0/867168608 >> 172.21.15.49:0/867168608 conn(0x7f8f840fb3d0 msgr2=0x7f8f840fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.361+0000 7f8f899f3640 1 -- 172.21.15.49:0/867168608 shutdown_connections 2024-03-18T00:53:35.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.362+0000 7f8f899f3640 1 -- 172.21.15.49:0/867168608 wait complete. 2024-03-18T00:53:35.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.362+0000 7f8f899f3640 1 Processor -- start 2024-03-18T00:53:35.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.362+0000 7f8f899f3640 1 -- start start 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f899f3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f84100c00 0x7f8f8410af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f899f3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 0x7f8f84109560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f899f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f841022d0 0x7f8f84109aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f899f3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8410d560 con 0x7f8f84101910 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f899f3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8410d3e0 con 0x7f8f84100c00 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f899f3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8410d6e0 con 0x7f8f841022d0 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f827fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 0x7f8f84109560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f827fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 0x7f8f84109560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58152/0 (socket says 172.21.15.49:58152) 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f827fc640 1 -- 172.21.15.49:0/1957703079 learned_addr learned my addr 172.21.15.49:0/1957703079 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f82ffd640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f84100c00 0x7f8f8410af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f827fc640 1 -- 172.21.15.49:0/1957703079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f841022d0 msgr2=0x7f8f84109aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f837fe640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f841022d0 0x7f8f84109aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f827fc640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f841022d0 0x7f8f84109aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f827fc640 1 -- 172.21.15.49:0/1957703079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f84100c00 msgr2=0x7f8f8410af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f827fc640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f84100c00 0x7f8f8410af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f827fc640 1 -- 172.21.15.49:0/1957703079 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8410a330 con 0x7f8f84101910 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.363+0000 7f8f837fe640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f841022d0 0x7f8f84109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.364+0000 7f8f827fc640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 0x7f8f84109560 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009230 tx=0x7f8f78004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.364+0000 7f8f889f1640 1 -- 172.21.15.49:0/1957703079 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78041070 con 0x7f8f84101910 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.364+0000 7f8f899f3640 1 -- 172.21.15.49:0/1957703079 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f841a1cc0 con 0x7f8f84101910 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.364+0000 7f8f899f3640 1 -- 172.21.15.49:0/1957703079 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f841a21b0 con 0x7f8f84101910 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.364+0000 7f8f889f1640 1 -- 172.21.15.49:0/1957703079 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f780054e0 con 0x7f8f84101910 2024-03-18T00:53:35.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.364+0000 7f8f889f1640 1 -- 172.21.15.49:0/1957703079 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78034440 con 0x7f8f84101910 2024-03-18T00:53:35.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.367+0000 7f8f8c5d6740 1 Processor -- start 2024-03-18T00:53:35.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.369+0000 7f8f8c5d6740 1 -- start start 2024-03-18T00:53:35.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.369+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9d060 0x55dc1aa9d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.369+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 0x55dc1aaa2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.369+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aaa5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.369+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55dc1a9f9490 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.369+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55dc1a9a1c90 con 0x55dc1aa9d060 2024-03-18T00:53:35.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.369+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dc1a9f5530 con 0x55dc1aa9da50 2024-03-18T00:53:35.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.369+0000 7f8f889f1640 1 -- 172.21.15.49:0/1957703079 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8f78044070 con 0x7f8f84101910 2024-03-18T00:53:35.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.370+0000 7f8f837fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9d060 0x55dc1aa9d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.370+0000 7f8f827fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aaa5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.370+0000 7f8f827fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aaa5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58166/0 (socket says 172.21.15.49:58166) 2024-03-18T00:53:35.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.370+0000 7f8f827fc640 1 -- 172.21.15.49:0/920831628 learned_addr learned my addr 172.21.15.49:0/920831628 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:35.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.371+0000 7f8f889f1640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f6c0a2900 0x7f8f6c0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.371+0000 7f8f827fc640 1 -- 172.21.15.49:0/920831628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 msgr2=0x55dc1aaa2550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:53:35.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.371+0000 7f8f827fc640 1 --2- 172.21.15.49:0/920831628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 0x55dc1aaa2550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.371+0000 7f8f827fc640 1 -- 172.21.15.49:0/920831628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9d060 msgr2=0x55dc1aa9d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:35.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.371+0000 7f8f827fc640 1 --2- 172.21.15.49:0/920831628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9d060 0x55dc1aa9d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.371+0000 7f8f827fc640 1 -- 172.21.15.49:0/920831628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc1aa5c500 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.371+0000 7f8f837fe640 1 --2- 172.21.15.49:0/920831628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9d060 0x55dc1aa9d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:53:35.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.372+0000 7f8f889f1640 1 -- 172.21.15.49:0/1957703079 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8f780f07a0 con 0x7f8f84101910 2024-03-18T00:53:35.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.372+0000 7f8f827fc640 1 --2- 172.21.15.49:0/920831628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aaa5050 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8f780c2ad0 tx=0x7f8f780b8e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4731a3defb70c64 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:35.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.373+0000 7f8f837fe640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f6c0a2900 0x7f8f6c0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.374+0000 7f8f837fe640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f6c0a2900 0x7f8f6c0a4dc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55dc1aa9e2b0 tx=0x7f8f7c004b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:35.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.375+0000 7f8f69ffb640 1 -- 172.21.15.49:0/920831628 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7804b070 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.375+0000 7f8f69ffb640 1 -- 172.21.15.49:0/920831628 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f780c24b0 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.375+0000 7f8f69ffb640 1 -- 172.21.15.49:0/920831628 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f780bce90 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:35 smithi049 ceph-mon[32130]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1012 MiB/s wr, 1.59k op/s 2024-03-18T00:53:35.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:35 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3572897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:53:35.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.380+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/920831628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 msgr2=0x55dc1aaa5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.380+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/920831628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aaa5050 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8f780c2ad0 tx=0x7f8f780b8e10 comp rx=0 tx=0).stop 2024-03-18T00:53:35.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.385+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/920831628 shutdown_connections 2024-03-18T00:53:35.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.385+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/920831628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aaa5050 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.385+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/920831628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 0x55dc1aaa2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.385+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/920831628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9d060 0x55dc1aa9d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.385+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/920831628 >> 172.21.15.49:0/920831628 conn(0x55dc1aa985d0 msgr2=0x55dc1aa989d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:35.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.395+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/920831628 shutdown_connections 2024-03-18T00:53:35.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.398+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/920831628 wait complete. 2024-03-18T00:53:35.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.399+0000 7f8f8c5d6740 1 -- start start 2024-03-18T00:53:35.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.399+0000 7f8f8c5d6740 1 Processor -- start 2024-03-18T00:53:35.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.400+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aab82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.400+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9da50 0x55dc1aab8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.400+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.400+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55dc1aab6940 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.400+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55dc1a9b5230 con 0x55dc1aa9da50 2024-03-18T00:53:35.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.400+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dc1aab6ac0 con 0x55dc1aa9d060 2024-03-18T00:53:35.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.400+0000 7f8f8c5d6740 10 client.0 _ll_register_callbacks cb 0x55dc1a9f1010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:35.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.401+0000 7f8f82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aab82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.401+0000 7f8f82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aab82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:48420/0 (socket says 172.21.15.49:48420) 2024-03-18T00:53:35.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.401+0000 7f8f82ffd640 1 -- 172.21.15.49:0/3496612651 learned_addr learned my addr 172.21.15.49:0/3496612651 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:35.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.401+0000 7f8f837fe640 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.401+0000 7f8f82ffd640 1 -- 172.21.15.49:0/3496612651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9da50 msgr2=0x55dc1aab8820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:53:35.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.403+0000 7f8f82ffd640 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9da50 0x55dc1aab8820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.403+0000 7f8f82ffd640 1 -- 172.21.15.49:0/3496612651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 msgr2=0x55dc1aabf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:35.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.403+0000 7f8f82ffd640 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.403+0000 7f8f82ffd640 1 -- 172.21.15.49:0/3496612651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc1a9c4100 con 0x55dc1aa9d060 2024-03-18T00:53:35.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.403+0000 7f8f837fe640 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:53:35.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.404+0000 7f8f82ffd640 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aab82e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400e1d0 tx=0x7f8f7400e690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa648f6b56794a8d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:35.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.405+0000 7f8f62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:53:35.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.405+0000 7f8f62ffd640 1 -- 172.21.15.49:0/3496612651 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f74017070 con 0x55dc1aa9d060 2024-03-18T00:53:35.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.405+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3496612651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dc1aa069d0 con 0x55dc1aa9d060 2024-03-18T00:53:35.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.405+0000 7f8f62ffd640 1 -- 172.21.15.49:0/3496612651 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8f74013470 con 0x55dc1aa9d060 2024-03-18T00:53:35.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.413+0000 7f8f62ffd640 1 -- 172.21.15.49:0/3496612651 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8f74023050 con 0x55dc1aa9d060 2024-03-18T00:53:35.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.415+0000 7f8f8c5d6740 10 client.25885 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:35.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.415+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3496612651 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dc1a70d3f0 con 0x55dc1aa9d060 2024-03-18T00:53:35.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.419+0000 7f8f62ffd640 1 -- 172.21.15.49:0/3496612651 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55dc1aa069d0 con 0x55dc1aa9d060 2024-03-18T00:53:35.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.419+0000 7f8f62ffd640 1 -- 172.21.15.49:0/3496612651 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8f74012910 con 0x55dc1aa9d060 2024-03-18T00:53:35.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.419+0000 7f8f8c5d6740 10 client.25885 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:35.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.419+0000 7f8f8c5d6740 10 client.25885 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:53:35.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.419+0000 7f8f8c5d6740 20 client.25885 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:35.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.419+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55dc1aaa8a40 0x55dc1aaaaee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.419+0000 7f8f8c5d6740 4 client.25885 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:35.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.419+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3496612651 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc1aa00d20 con 0x55dc1aaa8a40 2024-03-18T00:53:35.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.422+0000 7f8f827fc640 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55dc1aaa8a40 0x55dc1aaaaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:35 smithi107 ceph-mon[36270]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1012 MiB/s wr, 1.59k op/s 2024-03-18T00:53:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:35 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3572897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:53:35.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.436+0000 7f8f827fc640 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55dc1aaa8a40 0x55dc1aaaaee0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fe1541f152e77b12 server_cookie=6f23f6c587ae1fd6 in_seq=0 out_seq=0 2024-03-18T00:53:35.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.436+0000 7f8f62ffd640 10 client.25885 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:53:35.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.438+0000 7f8f62ffd640 1 -- 172.21.15.49:0/3496612651 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dc1aa00d20 con 0x55dc1aaa8a40 2024-03-18T00:53:35.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.438+0000 7f8f62ffd640 10 client.25885 handle_command_reply: tid=0 2024-03-18T00:53:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:35 smithi106 ceph-mon[35899]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1012 MiB/s wr, 1.59k op/s 2024-03-18T00:53:35.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:35 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3572897' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:53:35.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.442+0000 7f8f8c5d6740 1 client.25885 shutdown 2024-03-18T00:53:35.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.450+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3496612651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 msgr2=0x55dc1aab82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.450+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aab82e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400e1d0 tx=0x7f8f7400e690 comp rx=0 tx=0).stop 2024-03-18T00:53:35.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.453+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3496612651 shutdown_connections 2024-03-18T00:53:35.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.453+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55dc1aaa8a40 0x55dc1aaaaee0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.453+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aa9da50 0x55dc1aab8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.453+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.453+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3496612651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aab82e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.453+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3496612651 >> 172.21.15.49:0/3496612651 conn(0x55dc1aa985d0 msgr2=0x55dc1aa8efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:35.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.453+0000 7f8f62ffd640 0 client.25885 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:53:35.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.453+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3496612651 shutdown_connections 2024-03-18T00:53:35.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.454+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3496612651 wait complete. 2024-03-18T00:53:35.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.454+0000 7f8f8c5d6740 20 client.25885 trim_cache size 0 max 16384 2024-03-18T00:53:35.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.510+0000 7f8f8c5d6740 1 Processor -- start 2024-03-18T00:53:35.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.510+0000 7f8f8c5d6740 1 -- start start 2024-03-18T00:53:35.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.511+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9d060 0x55dc1aa8e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.511+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 0x55dc1aabf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.511+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aa8ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.511+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000e650 con 0x55dc1aa9d060 2024-03-18T00:53:35.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.511+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8f70010190 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.511+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000ed20 con 0x55dc1aa9da50 2024-03-18T00:53:35.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.512+0000 7f8f827fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 0x55dc1aabf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.512+0000 7f8f827fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 0x55dc1aabf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:48426/0 (socket says 172.21.15.49:48426) 2024-03-18T00:53:35.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.512+0000 7f8f827fc640 1 -- 172.21.15.49:0/3710128007 learned_addr learned my addr 172.21.15.49:0/3710128007 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:35.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.513+0000 7f8f827fc640 1 -- 172.21.15.49:0/3710128007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 msgr2=0x55dc1aa8ec90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:35.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.513+0000 7f8f827fc640 1 --2- 172.21.15.49:0/3710128007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aa8ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.513+0000 7f8f827fc640 1 -- 172.21.15.49:0/3710128007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9d060 msgr2=0x55dc1aa8e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:35.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.513+0000 7f8f827fc640 1 --2- 172.21.15.49:0/3710128007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9d060 0x55dc1aa8e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.513+0000 7f8f827fc640 1 -- 172.21.15.49:0/3710128007 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc1a7d77d0 con 0x55dc1aa9da50 2024-03-18T00:53:35.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.514+0000 7f8f827fc640 1 --2- 172.21.15.49:0/3710128007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 0x55dc1aabf710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8f780bd050 tx=0x7f8f78005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2260a6016517763 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:35.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.515+0000 7f8f697fa640 1 -- 172.21.15.49:0/3710128007 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7804b040 con 0x55dc1aa9da50 2024-03-18T00:53:35.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.515+0000 7f8f697fa640 1 -- 172.21.15.49:0/3710128007 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f780dd070 con 0x55dc1aa9da50 2024-03-18T00:53:35.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.515+0000 7f8f697fa640 1 -- 172.21.15.49:0/3710128007 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f780c1ad0 con 0x55dc1aa9da50 2024-03-18T00:53:35.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.516+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3710128007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 msgr2=0x55dc1aabf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.516+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3710128007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 0x55dc1aabf710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8f780bd050 tx=0x7f8f78005c90 comp rx=0 tx=0).stop 2024-03-18T00:53:35.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.523+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3710128007 shutdown_connections 2024-03-18T00:53:35.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.523+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3710128007 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aa8ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.523+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3710128007 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9da50 0x55dc1aabf710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.523+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/3710128007 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9d060 0x55dc1aa8e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.523+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3710128007 >> 172.21.15.49:0/3710128007 conn(0x55dc1aa985d0 msgr2=0x55dc1aa9c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:35.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.523+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3710128007 shutdown_connections 2024-03-18T00:53:35.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.524+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/3710128007 wait complete. 2024-03-18T00:53:35.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.525+0000 7f8f8c5d6740 1 -- start start 2024-03-18T00:53:35.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.526+0000 7f8f8c5d6740 1 Processor -- start 2024-03-18T00:53:35.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.527+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aabcec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.527+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9da50 0x55dc1aabd400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.527+0000 7f8f8c5d6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabe3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.527+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000fac0 con 0x55dc1aa9da50 2024-03-18T00:53:35.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.527+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000f3f0 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.527+0000 7f8f8c5d6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8f7000c630 con 0x55dc1aa9d060 2024-03-18T00:53:35.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.527+0000 7f8f8c5d6740 10 client.0 _ll_register_callbacks cb 0x7f8f74009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:35.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.530+0000 7f8f82ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aabcec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.530+0000 7f8f827fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9da50 0x55dc1aabd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.530+0000 7f8f827fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9da50 0x55dc1aabd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58196/0 (socket says 172.21.15.49:58196) 2024-03-18T00:53:35.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.530+0000 7f8f827fc640 1 -- 172.21.15.49:0/1296732834 learned_addr learned my addr 172.21.15.49:0/1296732834 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:35.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.534+0000 7f8f837fe640 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabe3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.538+0000 7f8f837fe640 1 -- 172.21.15.49:0/1296732834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 msgr2=0x55dc1aabcec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:35.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.538+0000 7f8f837fe640 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aabcec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.538+0000 7f8f837fe640 1 -- 172.21.15.49:0/1296732834 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9da50 msgr2=0x55dc1aabd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:35.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.538+0000 7f8f837fe640 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9da50 0x55dc1aabd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.538+0000 7f8f837fe640 1 -- 172.21.15.49:0/1296732834 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc1aabe9b0 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.540+0000 7f8f837fe640 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabe3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00dad0 tx=0x7f8f7c0073b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e85b9faa0d5d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:35.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.541+0000 7f8f62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:53:35.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.542+0000 7f8f62ffd640 1 -- 172.21.15.49:0/1296732834 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c0198b0 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.542+0000 7f8f62ffd640 1 -- 172.21.15.49:0/1296732834 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8f7c019a50 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.543+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/1296732834 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f780dd070 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.552+0000 7f8f62ffd640 1 -- 172.21.15.49:0/1296732834 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8f7c019c50 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.553+0000 7f8f8c5d6740 10 client.25811 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:35.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.553+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/1296732834 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55dc1ab3aee0 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.554+0000 7f8f62ffd640 1 -- 172.21.15.49:0/1296732834 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f780dd070 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.555+0000 7f8f62ffd640 1 -- 172.21.15.49:0/1296732834 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8f7c023020 con 0x55dc1aaa2bc0 2024-03-18T00:53:35.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.555+0000 7f8f8c5d6740 10 client.25811 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:35.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.555+0000 7f8f8c5d6740 10 client.25811 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:53:35.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.555+0000 7f8f8c5d6740 20 client.25811 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:35.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.555+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55dc1ab3b830 0x55dc1ab3dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.555+0000 7f8f8c5d6740 4 client.25811 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:53:35.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.555+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/1296732834 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc1a97e6a0 con 0x55dc1ab3b830 2024-03-18T00:53:35.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.563+0000 7f8f82ffd640 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55dc1ab3b830 0x55dc1ab3dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.578+0000 7f8f62ffd640 10 client.25811 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:53:35.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.578+0000 7f8f82ffd640 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55dc1ab3b830 0x55dc1ab3dc50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=302fb258d60dcd74 server_cookie=9c3d041962bea7ad in_seq=0 out_seq=0 2024-03-18T00:53:35.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.582+0000 7f8f62ffd640 1 -- 172.21.15.49:0/1296732834 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc1a97e6a0 con 0x55dc1ab3b830 2024-03-18T00:53:35.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.582+0000 7f8f62ffd640 10 client.25811 handle_command_reply: tid=0 2024-03-18T00:53:35.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f8c5d6740 1 client.25811 shutdown 2024-03-18T00:53:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/1296732834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 msgr2=0x55dc1aabe3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabe3c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00dad0 tx=0x7f8f7c0073b0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/1296732834 shutdown_connections 2024-03-18T00:53:35.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55dc1ab3b830 0x55dc1ab3dc50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dc1aaa2bc0 0x55dc1aabe3c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dc1aa9da50 0x55dc1aabd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f8c5d6740 1 --2- 172.21.15.49:0/1296732834 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dc1aa9d060 0x55dc1aabcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f62ffd640 0 client.25811 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:53:35.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.587+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/1296732834 >> 172.21.15.49:0/1296732834 conn(0x55dc1aa985d0 msgr2=0x55dc1aab57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:35.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.589+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/1296732834 shutdown_connections 2024-03-18T00:53:35.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.589+0000 7f8f8c5d6740 1 -- 172.21.15.49:0/1296732834 wait complete. 2024-03-18T00:53:35.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.590+0000 7f8f8c5d6740 20 client.25811 trim_cache size 0 max 16384 2024-03-18T00:53:35.593 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:53:35.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.592+0000 7f8f899f3640 1 -- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f6c0a2900 msgr2=0x7f8f6c0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.592+0000 7f8f899f3640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f6c0a2900 0x7f8f6c0a4dc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55dc1aa9e2b0 tx=0x7f8f7c004b50 comp rx=0 tx=0).stop 2024-03-18T00:53:35.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.592+0000 7f8f899f3640 1 -- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 msgr2=0x7f8f84109560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.592+0000 7f8f899f3640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 0x7f8f84109560 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009230 tx=0x7f8f78004040 comp rx=0 tx=0).stop 2024-03-18T00:53:35.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.593+0000 7f8f899f3640 1 -- 172.21.15.49:0/1957703079 shutdown_connections 2024-03-18T00:53:35.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.593+0000 7f8f899f3640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f6c0a2900 0x7f8f6c0a4dc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.593+0000 7f8f899f3640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f841022d0 0x7f8f84109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.593+0000 7f8f899f3640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f84101910 0x7f8f84109560 secure :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009230 tx=0x7f8f78004040 comp rx=0 tx=0).stop 2024-03-18T00:53:35.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.593+0000 7f8f899f3640 1 --2- 172.21.15.49:0/1957703079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f84100c00 0x7f8f8410af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.593+0000 7f8f899f3640 1 -- 172.21.15.49:0/1957703079 >> 172.21.15.49:0/1957703079 conn(0x7f8f840fb3d0 msgr2=0x7f8f84104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:35.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.593+0000 7f8f899f3640 1 -- 172.21.15.49:0/1957703079 shutdown_connections 2024-03-18T00:53:35.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.594+0000 7f8f899f3640 1 -- 172.21.15.49:0/1957703079 wait complete. 2024-03-18T00:53:35.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:53:35.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.948+0000 7f367b2a1640 1 -- 172.21.15.49:0/279677883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36741087e0 msgr2=0x7f367410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.948+0000 7f367b2a1640 1 --2- 172.21.15.49:0/279677883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36741087e0 0x7f367410ac10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3670009900 tx=0x7f367002bc30 comp rx=0 tx=0).stop 2024-03-18T00:53:35.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.949+0000 7f367b2a1640 1 -- 172.21.15.49:0/279677883 shutdown_connections 2024-03-18T00:53:35.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.949+0000 7f367b2a1640 1 --2- 172.21.15.49:0/279677883 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f367410db00 0x7f367410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.949+0000 7f367b2a1640 1 --2- 172.21.15.49:0/279677883 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f367410b150 0x7f367410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.949+0000 7f367b2a1640 1 --2- 172.21.15.49:0/279677883 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f36741087e0 0x7f367410ac10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.949+0000 7f367b2a1640 1 -- 172.21.15.49:0/279677883 >> 172.21.15.49:0/279677883 conn(0x7f367406e640 msgr2=0x7f3674070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:35.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.952+0000 7f367b2a1640 1 -- 172.21.15.49:0/279677883 shutdown_connections 2024-03-18T00:53:35.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.955+0000 7f367b2a1640 1 -- 172.21.15.49:0/279677883 wait complete. 2024-03-18T00:53:35.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.956+0000 7f367b2a1640 1 Processor -- start 2024-03-18T00:53:35.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.957+0000 7f367b2a1640 1 -- start start 2024-03-18T00:53:35.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.957+0000 7f367b2a1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36741087e0 0x7f3674114f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.957+0000 7f367b2a1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f367410b150 0x7f3674115470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.957+0000 7f367b2a1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f367410db00 0x7f36741b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.957+0000 7f367b2a1640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f367411c300 con 0x7f36741087e0 2024-03-18T00:53:35.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.957+0000 7f367b2a1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f367411c180 con 0x7f367410db00 2024-03-18T00:53:35.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.957+0000 7f367b2a1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f367411c480 con 0x7f367410b150 2024-03-18T00:53:35.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.958+0000 7f367903d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36741087e0 0x7f3674114f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.958+0000 7f367903d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36741087e0 0x7f3674114f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58224/0 (socket says 172.21.15.49:58224) 2024-03-18T00:53:35.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.959+0000 7f367903d640 1 -- 172.21.15.49:0/1058477898 learned_addr learned my addr 172.21.15.49:0/1058477898 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:35.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.960+0000 7f367983e640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f367410db00 0x7f36741b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.962+0000 7f367903d640 1 -- 172.21.15.49:0/1058477898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f367410b150 msgr2=0x7f3674115470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:53:35.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.962+0000 7f367903d640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f367410b150 0x7f3674115470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.962+0000 7f367903d640 1 -- 172.21.15.49:0/1058477898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f367410db00 msgr2=0x7f36741b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:35.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.962+0000 7f367903d640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f367410db00 0x7f36741b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.962+0000 7f367903d640 1 -- 172.21.15.49:0/1058477898 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36741b6f50 con 0x7f36741087e0 2024-03-18T00:53:35.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.963+0000 7f367903d640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36741087e0 0x7f3674114f30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f367002c240 tx=0x7f3670004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:35.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.964+0000 7f36667fc640 1 -- 172.21.15.49:0/1058477898 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670043020 con 0x7f36741087e0 2024-03-18T00:53:35.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.964+0000 7f36667fc640 1 -- 172.21.15.49:0/1058477898 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36700054e0 con 0x7f36741087e0 2024-03-18T00:53:35.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.964+0000 7f36667fc640 1 -- 172.21.15.49:0/1058477898 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3670003ce0 con 0x7f36741087e0 2024-03-18T00:53:35.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.966+0000 7f367b2a1640 1 -- 172.21.15.49:0/1058477898 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36741b7120 con 0x7f36741087e0 2024-03-18T00:53:35.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.966+0000 7f367b2a1640 1 -- 172.21.15.49:0/1058477898 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36741b75c0 con 0x7f36741087e0 2024-03-18T00:53:35.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.979+0000 7f367df6a740 1 Processor -- start 2024-03-18T00:53:35.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.983+0000 7f367df6a740 1 -- start start 2024-03-18T00:53:35.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.983+0000 7f367df6a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfa850 0x55f8bfbfac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.983+0000 7f367df6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfbfb240 0x55f8bfbffd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.983+0000 7f367df6a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc02840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.983+0000 7f367df6a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f8bfb5f490 con 0x55f8bfbfa850 2024-03-18T00:53:35.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.983+0000 7f367df6a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f8bfb07c90 con 0x55f8bfc003b0 2024-03-18T00:53:35.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.983+0000 7f367df6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f8bfb5b530 con 0x55f8bfbfb240 2024-03-18T00:53:35.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.984+0000 7f367983e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfa850 0x55f8bfbfac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.984+0000 7f367983e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfa850 0x55f8bfbfac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58230/0 (socket says 172.21.15.49:58230) 2024-03-18T00:53:35.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.985+0000 7f367983e640 1 -- 172.21.15.49:0/213413742 learned_addr learned my addr 172.21.15.49:0/213413742 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:35.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.986+0000 7f367883c640 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc02840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.986+0000 7f367903d640 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfbfb240 0x55f8bfbffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:35.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.987+0000 7f367983e640 1 -- 172.21.15.49:0/213413742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfbfb240 msgr2=0x55f8bfbffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:35.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.987+0000 7f367983e640 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfbfb240 0x55f8bfbffd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.987+0000 7f367983e640 1 -- 172.21.15.49:0/213413742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 msgr2=0x55f8bfc02840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:35.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.987+0000 7f367983e640 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc02840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:35.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.987+0000 7f367983e640 1 -- 172.21.15.49:0/213413742 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8bfbc2500 con 0x55f8bfbfa850 2024-03-18T00:53:35.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.987+0000 7f367883c640 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc02840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:53:35.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.991+0000 7f36667fc640 1 -- 172.21.15.49:0/1058477898 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3670046020 con 0x7f36741087e0 2024-03-18T00:53:35.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.992+0000 7f36667fc640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f36600a2a00 0x7f36600a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:35.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.992+0000 7f367983e640 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfa850 0x55f8bfbfac70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f366c009f70 tx=0x7f366c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1774b941028a2980 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:35.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.994+0000 7f36667fc640 1 -- 172.21.15.49:0/1058477898 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f36700f0bb0 con 0x7f36741087e0 2024-03-18T00:53:35.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.995+0000 7f367903d640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f36600a2a00 0x7f36600a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:35.999+0000 7f367903d640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f36600a2a00 0x7f36600a4ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f36700dbfd0 tx=0x7f36700dd040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.000+0000 7f365f7fe640 1 -- 172.21.15.49:0/213413742 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c004b20 con 0x55f8bfbfa850 2024-03-18T00:53:36.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.002+0000 7f365f7fe640 1 -- 172.21.15.49:0/213413742 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f366c004cc0 con 0x55f8bfbfa850 2024-03-18T00:53:36.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.003+0000 7f367df6a740 1 -- 172.21.15.49:0/213413742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfa850 msgr2=0x55f8bfbfac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.003+0000 7f367df6a740 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfa850 0x55f8bfbfac70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f366c009f70 tx=0x7f366c0118d0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.003+0000 7f367df6a740 1 -- 172.21.15.49:0/213413742 shutdown_connections 2024-03-18T00:53:36.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.003+0000 7f367df6a740 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc02840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.003+0000 7f367df6a740 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfbfb240 0x55f8bfbffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.003+0000 7f367df6a740 1 --2- 172.21.15.49:0/213413742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfa850 0x55f8bfbfac70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.003+0000 7f367df6a740 1 -- 172.21.15.49:0/213413742 >> 172.21.15.49:0/213413742 conn(0x55f8bfbf5dc0 msgr2=0x55f8bfbf61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.011+0000 7f367df6a740 1 -- 172.21.15.49:0/213413742 shutdown_connections 2024-03-18T00:53:36.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.011+0000 7f367df6a740 1 -- 172.21.15.49:0/213413742 wait complete. 2024-03-18T00:53:36.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.012+0000 7f367df6a740 1 -- start start 2024-03-18T00:53:36.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.012+0000 7f367df6a740 1 Processor -- start 2024-03-18T00:53:36.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.013+0000 7f367df6a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc04cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.013+0000 7f367df6a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.013+0000 7f367df6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfc0a210 0x55f8bfc0c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.013+0000 7f367df6a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f8bfc033b0 con 0x55f8bfbfb240 2024-03-18T00:53:36.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.013+0000 7f367df6a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f8bfb1b230 con 0x55f8bfc003b0 2024-03-18T00:53:36.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.013+0000 7f367df6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f8bfc03530 con 0x55f8bfc0a210 2024-03-18T00:53:36.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.013+0000 7f367df6a740 10 client.0 _ll_register_callbacks cb 0x55f8bfb57010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:36.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.015+0000 7f367983e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc04cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.015+0000 7f367983e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc04cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58240/0 (socket says 172.21.15.49:58240) 2024-03-18T00:53:36.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.015+0000 7f367983e640 1 -- 172.21.15.49:0/1613030301 learned_addr learned my addr 172.21.15.49:0/1613030301 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:36.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.017+0000 7f367983e640 1 -- 172.21.15.49:0/1613030301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfc0a210 msgr2=0x55f8bfc0c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.017+0000 7f367983e640 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfc0a210 0x55f8bfc0c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.017+0000 7f367983e640 1 -- 172.21.15.49:0/1613030301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 msgr2=0x55f8bfc051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.017+0000 7f367983e640 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.017+0000 7f367983e640 1 -- 172.21.15.49:0/1613030301 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8bfb2a100 con 0x55f8bfbfb240 2024-03-18T00:53:36.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.019+0000 7f367983e640 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc04cb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f366c01b040 tx=0x7f366c004f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d7019653f9f854 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.019+0000 7f365cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:36.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.020+0000 7f365cff9640 1 -- 172.21.15.49:0/1613030301 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c028070 con 0x55f8bfbfb240 2024-03-18T00:53:36.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.020+0000 7f367df6a740 1 -- 172.21.15.49:0/1613030301 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f8bfb6c9d0 con 0x55f8bfbfb240 2024-03-18T00:53:36.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.020+0000 7f365cff9640 1 -- 172.21.15.49:0/1613030301 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f366c005740 con 0x55f8bfbfb240 2024-03-18T00:53:36.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.022+0000 7f365cff9640 1 -- 172.21.15.49:0/1613030301 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f366c005980 con 0x55f8bfbfb240 2024-03-18T00:53:36.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.024+0000 7f367df6a740 10 client.16257 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:36.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.024+0000 7f367df6a740 1 -- 172.21.15.49:0/1613030301 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8bf8733f0 con 0x55f8bfbfb240 2024-03-18T00:53:36.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.034+0000 7f365cff9640 1 -- 172.21.15.49:0/1613030301 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f8bfb6c9d0 con 0x55f8bfbfb240 2024-03-18T00:53:36.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.035+0000 7f365cff9640 1 -- 172.21.15.49:0/1613030301 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f366c019690 con 0x55f8bfbfb240 2024-03-18T00:53:36.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.035+0000 7f367df6a740 10 client.16257 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:36.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.035+0000 7f367df6a740 10 client.16257 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:53:36.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.035+0000 7f367df6a740 20 client.16257 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:36.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.035+0000 7f367df6a740 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55f8bfc33200 0x55f8bfc35660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.035+0000 7f367df6a740 4 client.16257 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:36.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.035+0000 7f367df6a740 1 -- 172.21.15.49:0/1613030301 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8bfb66d20 con 0x55f8bfc33200 2024-03-18T00:53:36.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.036+0000 7f367883c640 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55f8bfc33200 0x55f8bfc35660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.037+0000 7f365cff9640 10 client.16257 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:53:36.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.038+0000 7f367883c640 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55f8bfc33200 0x55f8bfc35660 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8b6893f8cff549df server_cookie=f56bf148adafe7fb in_seq=0 out_seq=0 2024-03-18T00:53:36.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.041+0000 7f365cff9640 1 -- 172.21.15.49:0/1613030301 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f8bfb66d20 con 0x55f8bfc33200 2024-03-18T00:53:36.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.041+0000 7f365cff9640 10 client.16257 handle_command_reply: tid=0 2024-03-18T00:53:36.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.041+0000 7f367df6a740 1 client.16257 shutdown 2024-03-18T00:53:36.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.045+0000 7f367df6a740 1 -- 172.21.15.49:0/1613030301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 msgr2=0x55f8bfc04cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.045+0000 7f367df6a740 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc04cb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f366c01b040 tx=0x7f366c004f00 comp rx=0 tx=0).stop 2024-03-18T00:53:36.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.049+0000 7f367df6a740 1 -- 172.21.15.49:0/1613030301 shutdown_connections 2024-03-18T00:53:36.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.049+0000 7f367df6a740 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55f8bfc33200 0x55f8bfc35660 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.049+0000 7f367df6a740 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.049+0000 7f367df6a740 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfc0a210 0x55f8bfc0c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.049+0000 7f367df6a740 1 --2- 172.21.15.49:0/1613030301 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc04cb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.049+0000 7f367df6a740 1 -- 172.21.15.49:0/1613030301 >> 172.21.15.49:0/1613030301 conn(0x55f8bfbf5dc0 msgr2=0x55f8bfbfaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.049+0000 7f365cff9640 0 client.16257 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:53:36.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.050+0000 7f367df6a740 1 -- 172.21.15.49:0/1613030301 shutdown_connections 2024-03-18T00:53:36.051 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.050+0000 7f367df6a740 1 -- 172.21.15.49:0/1613030301 wait complete. 2024-03-18T00:53:36.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.051+0000 7f367df6a740 20 client.16257 trim_cache size 0 max 16384 2024-03-18T00:53:36.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.130+0000 7f367df6a740 1 Processor -- start 2024-03-18T00:53:36.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.130+0000 7f367df6a740 1 -- start start 2024-03-18T00:53:36.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.130+0000 7f367df6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfbfb240 0x55f8bfc03240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.130+0000 7f367df6a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfbf61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.130+0000 7f367df6a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfc0a210 0x55f8bfbf9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.131+0000 7f367df6a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f36741aa5f0 con 0x55f8bfc0a210 2024-03-18T00:53:36.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.131+0000 7f367df6a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f36741ac130 con 0x55f8bfc003b0 2024-03-18T00:53:36.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.131+0000 7f367df6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f36741aacc0 con 0x55f8bfbfb240 2024-03-18T00:53:36.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.132+0000 7f367983e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfc0a210 0x55f8bfbf9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.132+0000 7f367983e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfc0a210 0x55f8bfbf9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58246/0 (socket says 172.21.15.49:58246) 2024-03-18T00:53:36.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.132+0000 7f367983e640 1 -- 172.21.15.49:0/4264468487 learned_addr learned my addr 172.21.15.49:0/4264468487 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:36.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.135+0000 7f367883c640 1 --2- 172.21.15.49:0/4264468487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfbf61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.135+0000 7f367983e640 1 -- 172.21.15.49:0/4264468487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfbfb240 msgr2=0x55f8bfc03240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.135+0000 7f367983e640 1 --2- 172.21.15.49:0/4264468487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfbfb240 0x55f8bfc03240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.135+0000 7f367983e640 1 -- 172.21.15.49:0/4264468487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 msgr2=0x55f8bfbf61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.135+0000 7f367983e640 1 --2- 172.21.15.49:0/4264468487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfbf61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.135+0000 7f367983e640 1 -- 172.21.15.49:0/4264468487 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8bf93d7d0 con 0x55f8bfc0a210 2024-03-18T00:53:36.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.136+0000 7f367883c640 1 --2- 172.21.15.49:0/4264468487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfbf61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:53:36.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.136+0000 7f367983e640 1 --2- 172.21.15.49:0/4264468487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfc0a210 0x55f8bfbf9020 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f366c005b40 tx=0x7f366c0051c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df82ce2e8b5e57b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.142+0000 7f365effd640 1 -- 172.21.15.49:0/4264468487 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c015030 con 0x55f8bfc0a210 2024-03-18T00:53:36.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.142+0000 7f365effd640 1 -- 172.21.15.49:0/4264468487 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f366c005740 con 0x55f8bfc0a210 2024-03-18T00:53:36.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.144+0000 7f367df6a740 1 -- 172.21.15.49:0/4264468487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfc0a210 msgr2=0x55f8bfbf9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.144+0000 7f367df6a740 1 --2- 172.21.15.49:0/4264468487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfc0a210 0x55f8bfbf9020 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f366c005b40 tx=0x7f366c0051c0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.150+0000 7f367df6a740 1 -- 172.21.15.49:0/4264468487 shutdown_connections 2024-03-18T00:53:36.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.150+0000 7f367df6a740 1 --2- 172.21.15.49:0/4264468487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfbf61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.150+0000 7f367df6a740 1 --2- 172.21.15.49:0/4264468487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfc0a210 0x55f8bfbf9020 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.151+0000 7f367df6a740 1 --2- 172.21.15.49:0/4264468487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfbfb240 0x55f8bfc03240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.151+0000 7f367df6a740 1 -- 172.21.15.49:0/4264468487 >> 172.21.15.49:0/4264468487 conn(0x55f8bfbf5dc0 msgr2=0x55f8bfbfaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.154+0000 7f367df6a740 1 -- 172.21.15.49:0/4264468487 shutdown_connections 2024-03-18T00:53:36.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.155+0000 7f367df6a740 1 -- 172.21.15.49:0/4264468487 wait complete. 2024-03-18T00:53:36.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.156+0000 7f367df6a740 1 -- start start 2024-03-18T00:53:36.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.156+0000 7f367df6a740 1 Processor -- start 2024-03-18T00:53:36.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.157+0000 7f367df6a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc43540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.157+0000 7f367df6a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc43a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.157+0000 7f367df6a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfc0a210 0x55f8bfc96f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.158+0000 7f367df6a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f36741aba60 con 0x55f8bfbfb240 2024-03-18T00:53:36.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.158+0000 7f367df6a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f36741ab390 con 0x55f8bfc003b0 2024-03-18T00:53:36.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.158+0000 7f367df6a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f36741a85d0 con 0x55f8bfc0a210 2024-03-18T00:53:36.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.158+0000 7f367983e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc43540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.158+0000 7f367df6a740 10 client.0 _ll_register_callbacks cb 0x7f366c009e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:36.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.158+0000 7f367983e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc43540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58248/0 (socket says 172.21.15.49:58248) 2024-03-18T00:53:36.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.158+0000 7f367983e640 1 -- 172.21.15.49:0/3910489590 learned_addr learned my addr 172.21.15.49:0/3910489590 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:36.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.161+0000 7f367983e640 1 -- 172.21.15.49:0/3910489590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfc0a210 msgr2=0x55f8bfc96f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.161+0000 7f367983e640 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfc0a210 0x55f8bfc96f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.161+0000 7f367983e640 1 -- 172.21.15.49:0/3910489590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 msgr2=0x55f8bfc43a80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:36.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.161+0000 7f367983e640 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc43a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.161+0000 7f367983e640 1 -- 172.21.15.49:0/3910489590 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8bfc469c0 con 0x55f8bfbfb240 2024-03-18T00:53:36.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.165+0000 7f367983e640 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc43540 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f366c005910 tx=0x7f366c00ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb06d5792f3a00b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.169+0000 7f365cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:36.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.169+0000 7f365cff9640 1 -- 172.21.15.49:0/3910489590 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c03e020 con 0x55f8bfbfb240 2024-03-18T00:53:36.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.169+0000 7f365cff9640 1 -- 172.21.15.49:0/3910489590 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f366c024920 con 0x55f8bfbfb240 2024-03-18T00:53:36.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.169+0000 7f367df6a740 1 -- 172.21.15.49:0/3910489590 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f366c005740 con 0x55f8bfbfb240 2024-03-18T00:53:36.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.186+0000 7f365cff9640 1 -- 172.21.15.49:0/3910489590 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f366c019e00 con 0x55f8bfbfb240 2024-03-18T00:53:36.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.186+0000 7f367df6a740 10 client.16263 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:36.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.186+0000 7f367df6a740 1 -- 172.21.15.49:0/3910489590 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8bfc983b0 con 0x55f8bfbfb240 2024-03-18T00:53:36.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.192+0000 7f365cff9640 1 -- 172.21.15.49:0/3910489590 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f366c005740 con 0x55f8bfbfb240 2024-03-18T00:53:36.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.193+0000 7f365cff9640 1 -- 172.21.15.49:0/3910489590 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f366c037030 con 0x55f8bfbfb240 2024-03-18T00:53:36.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.193+0000 7f367df6a740 10 client.16263 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:36.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.193+0000 7f367df6a740 10 client.16263 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:53:36.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.193+0000 7f367df6a740 20 client.16263 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:36.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.193+0000 7f367df6a740 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55f8bfc98af0 0x55f8bfc9af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.193+0000 7f367df6a740 4 client.16263 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:53:36.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.193+0000 7f367df6a740 1 -- 172.21.15.49:0/3910489590 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f8bfae46a0 con 0x55f8bfc98af0 2024-03-18T00:53:36.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.196+0000 7f367883c640 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55f8bfc98af0 0x55f8bfc9af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.207+0000 7f367883c640 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55f8bfc98af0 0x55f8bfc9af10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b4cd993378d66a09 server_cookie=d716d2b4da6ef470 in_seq=0 out_seq=0 2024-03-18T00:53:36.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.207+0000 7f365cff9640 10 client.16263 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:53:36.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.209+0000 7f365cff9640 1 -- 172.21.15.49:0/3910489590 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f8bfae46a0 con 0x55f8bfc98af0 2024-03-18T00:53:36.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.210+0000 7f365cff9640 10 client.16263 handle_command_reply: tid=0 2024-03-18T00:53:36.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.210+0000 7f367df6a740 1 client.16263 shutdown 2024-03-18T00:53:36.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.210+0000 7f367df6a740 1 -- 172.21.15.49:0/3910489590 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 msgr2=0x55f8bfc43540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.210+0000 7f367df6a740 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc43540 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f366c005910 tx=0x7f366c00ffd0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.210+0000 7f367df6a740 1 -- 172.21.15.49:0/3910489590 shutdown_connections 2024-03-18T00:53:36.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.211+0000 7f367df6a740 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55f8bfc98af0 0x55f8bfc9af10 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.211+0000 7f367df6a740 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8bfc003b0 0x55f8bfc43a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.211+0000 7f367df6a740 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8bfc0a210 0x55f8bfc96f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.211+0000 7f367df6a740 1 --2- 172.21.15.49:0/3910489590 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8bfbfb240 0x55f8bfc43540 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.211+0000 7f367df6a740 1 -- 172.21.15.49:0/3910489590 >> 172.21.15.49:0/3910489590 conn(0x55f8bfbf5dc0 msgr2=0x55f8bfbf61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.211+0000 7f365cff9640 0 client.16263 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:53:36.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.213+0000 7f367df6a740 1 -- 172.21.15.49:0/3910489590 shutdown_connections 2024-03-18T00:53:36.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.213+0000 7f367df6a740 1 -- 172.21.15.49:0/3910489590 wait complete. 2024-03-18T00:53:36.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.213+0000 7f367df6a740 20 client.16263 trim_cache size 0 max 16384 2024-03-18T00:53:36.217 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.217+0000 7f367b2a1640 1 -- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f36600a2a00 msgr2=0x7f36600a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f36600a2a00 0x7f36600a4ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f36700dbfd0 tx=0x7f36700dd040 comp rx=0 tx=0).stop 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 -- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36741087e0 msgr2=0x7f3674114f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36741087e0 0x7f3674114f30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f367002c240 tx=0x7f3670004e30 comp rx=0 tx=0).stop 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 -- 172.21.15.49:0/1058477898 shutdown_connections 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f36600a2a00 0x7f36600a4ec0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f367410db00 0x7f36741b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f367410b150 0x7f3674115470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 --2- 172.21.15.49:0/1058477898 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f36741087e0 0x7f3674114f30 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 -- 172.21.15.49:0/1058477898 >> 172.21.15.49:0/1058477898 conn(0x7f367406e640 msgr2=0x7f367410c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 -- 172.21.15.49:0/1058477898 shutdown_connections 2024-03-18T00:53:36.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.218+0000 7f367b2a1640 1 -- 172.21.15.49:0/1058477898 wait complete. 2024-03-18T00:53:36.248 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:53:36.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.545+0000 7f79dab97640 1 -- 172.21.15.49:0/2451054094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 msgr2=0x7f79d410df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.545+0000 7f79dab97640 1 --2- 172.21.15.49:0/2451054094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 0x7f79d410df80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f79cc01acb0 tx=0x7f79cc03d130 comp rx=0 tx=0).stop 2024-03-18T00:53:36.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.546+0000 7f79dab97640 1 -- 172.21.15.49:0/2451054094 shutdown_connections 2024-03-18T00:53:36.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.546+0000 7f79dab97640 1 --2- 172.21.15.49:0/2451054094 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 0x7f79d410df80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.546+0000 7f79dab97640 1 --2- 172.21.15.49:0/2451054094 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d410b150 0x7f79d410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.546+0000 7f79dab97640 1 --2- 172.21.15.49:0/2451054094 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d41087e0 0x7f79d410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.546+0000 7f79dab97640 1 -- 172.21.15.49:0/2451054094 >> 172.21.15.49:0/2451054094 conn(0x7f79d406e640 msgr2=0x7f79d4070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.548+0000 7f79dab97640 1 -- 172.21.15.49:0/2451054094 shutdown_connections 2024-03-18T00:53:36.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.551+0000 7f79dab97640 1 -- 172.21.15.49:0/2451054094 wait complete. 2024-03-18T00:53:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.552+0000 7f79dab97640 1 Processor -- start 2024-03-18T00:53:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.552+0000 7f79dab97640 1 -- start start 2024-03-18T00:53:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.552+0000 7f79dab97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d41087e0 0x7f79d419e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.552+0000 7f79dab97640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d410b150 0x7f79d419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.552+0000 7f79dab97640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 0x7f79d4079a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.552+0000 7f79dab97640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4111ae0 con 0x7f79d410db00 2024-03-18T00:53:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.552+0000 7f79dab97640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4111960 con 0x7f79d410b150 2024-03-18T00:53:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.552+0000 7f79dab97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f79d4111c60 con 0x7f79d41087e0 2024-03-18T00:53:36.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.554+0000 7f79d9134640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 0x7f79d4079a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.554+0000 7f79d9134640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 0x7f79d4079a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58260/0 (socket says 172.21.15.49:58260) 2024-03-18T00:53:36.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.554+0000 7f79d9134640 1 -- 172.21.15.49:0/3990608812 learned_addr learned my addr 172.21.15.49:0/3990608812 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:36.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.554+0000 7f79d3fff640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d410b150 0x7f79d419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.555+0000 7f79d8933640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d41087e0 0x7f79d419e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.556+0000 7f79d9134640 1 -- 172.21.15.49:0/3990608812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d41087e0 msgr2=0x7f79d419e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.556+0000 7f79d9134640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d41087e0 0x7f79d419e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.556+0000 7f79d9134640 1 -- 172.21.15.49:0/3990608812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d410b150 msgr2=0x7f79d419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.556+0000 7f79d9134640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d410b150 0x7f79d419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.556+0000 7f79d9134640 1 -- 172.21.15.49:0/3990608812 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d407a0b0 con 0x7f79d410db00 2024-03-18T00:53:36.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.559+0000 7f79d9134640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 0x7f79d4079a40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f79cc01ac00 tx=0x7f79cc003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.563+0000 7f79d1ffb640 1 -- 172.21.15.49:0/3990608812 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc04a040 con 0x7f79d410db00 2024-03-18T00:53:36.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.563+0000 7f79dab97640 1 -- 172.21.15.49:0/3990608812 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79d407a340 con 0x7f79d410db00 2024-03-18T00:53:36.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.564+0000 7f79dab97640 1 -- 172.21.15.49:0/3990608812 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79d407a830 con 0x7f79d410db00 2024-03-18T00:53:36.572 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.570+0000 7f79dd786740 1 Processor -- start 2024-03-18T00:53:36.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.570+0000 7f79dd786740 1 -- start start 2024-03-18T00:53:36.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.571+0000 7f79dd786740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 0x556d52cf5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.571+0000 7f79dd786740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52cfad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.571+0000 7f79d1ffb640 1 -- 172.21.15.49:0/3990608812 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79cc004520 con 0x7f79d410db00 2024-03-18T00:53:36.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.571+0000 7f79d1ffb640 1 -- 172.21.15.49:0/3990608812 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc045bd0 con 0x7f79d410db00 2024-03-18T00:53:36.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.571+0000 7f79dd786740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52cfd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.574 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.571+0000 7f79dd786740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x556d52c5a490 con 0x556d52cf6240 2024-03-18T00:53:36.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.571+0000 7f79dd786740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x556d52c02c90 con 0x556d52cf5850 2024-03-18T00:53:36.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.571+0000 7f79dd786740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556d52c56530 con 0x556d52cfb3b0 2024-03-18T00:53:36.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.573+0000 7f79d3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 0x556d52cf5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.573+0000 7f79d3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 0x556d52cf5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:36816/0 (socket says 172.21.15.49:36816) 2024-03-18T00:53:36.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.573+0000 7f79d3fff640 1 -- 172.21.15.49:0/484295974 learned_addr learned my addr 172.21.15.49:0/484295974 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:36.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.573+0000 7f79d3fff640 1 --2- 172.21.15.49:0/484295974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52cfd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.573+0000 7f79d8933640 1 --2- 172.21.15.49:0/484295974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52cfad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.576+0000 7f79d1ffb640 1 -- 172.21.15.49:0/3990608812 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f79cc057030 con 0x7f79d410db00 2024-03-18T00:53:36.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.577+0000 7f79d8933640 1 -- 172.21.15.49:0/484295974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 msgr2=0x556d52cfd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.577+0000 7f79d8933640 1 --2- 172.21.15.49:0/484295974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52cfd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.577+0000 7f79d8933640 1 -- 172.21.15.49:0/484295974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 msgr2=0x556d52cf5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.577+0000 7f79d8933640 1 --2- 172.21.15.49:0/484295974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 0x556d52cf5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.577+0000 7f79d8933640 1 -- 172.21.15.49:0/484295974 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d52cbd500 con 0x556d52cf6240 2024-03-18T00:53:36.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.578+0000 7f79d8933640 1 --2- 172.21.15.49:0/484295974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52cfad40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f79c400a830 tx=0x7f79c400acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08b67745e3f4775 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.578+0000 7f79d1ffb640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f79bc0a2a10 0x7f79bc0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.579+0000 7f79d1ffb640 1 -- 172.21.15.49:0/3990608812 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f79cc101840 con 0x7f79d410db00 2024-03-18T00:53:36.580 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.579+0000 7f79d3fff640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f79bc0a2a10 0x7f79bc0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.579+0000 7f79baffd640 1 -- 172.21.15.49:0/484295974 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c4015070 con 0x556d52cf6240 2024-03-18T00:53:36.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.579+0000 7f79baffd640 1 -- 172.21.15.49:0/484295974 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c400cb40 con 0x556d52cf6240 2024-03-18T00:53:36.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.581+0000 7f79dd786740 1 -- 172.21.15.49:0/484295974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 msgr2=0x556d52cfad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.581+0000 7f79dd786740 1 --2- 172.21.15.49:0/484295974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52cfad40 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f79c400a830 tx=0x7f79c400acf0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.581+0000 7f79dd786740 1 -- 172.21.15.49:0/484295974 shutdown_connections 2024-03-18T00:53:36.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.581+0000 7f79dd786740 1 --2- 172.21.15.49:0/484295974 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52cfd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.581+0000 7f79dd786740 1 --2- 172.21.15.49:0/484295974 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52cfad40 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.581+0000 7f79dd786740 1 --2- 172.21.15.49:0/484295974 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 0x556d52cf5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.581+0000 7f79dd786740 1 -- 172.21.15.49:0/484295974 >> 172.21.15.49:0/484295974 conn(0x556d52cf0dc0 msgr2=0x556d52cf11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.588+0000 7f79d3fff640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f79bc0a2a10 0x7f79bc0a4ed0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f79c80058a0 tx=0x7f79c800b710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.593+0000 7f79dd786740 1 -- 172.21.15.49:0/484295974 shutdown_connections 2024-03-18T00:53:36.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.593+0000 7f79dd786740 1 -- 172.21.15.49:0/484295974 wait complete. 2024-03-18T00:53:36.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.594+0000 7f79dd786740 1 -- start start 2024-03-18T00:53:36.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.594+0000 7f79dd786740 1 Processor -- start 2024-03-18T00:53:36.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.595+0000 7f79dd786740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf5850 0x556d52cffd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.595+0000 7f79dd786740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf6240 0x556d52d002b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.595+0000 7f79dd786740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52d072b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.595+0000 7f79dd786740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x556d52cfe3b0 con 0x556d52cf5850 2024-03-18T00:53:36.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.595+0000 7f79dd786740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x556d52c16230 con 0x556d52cf6240 2024-03-18T00:53:36.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.595+0000 7f79dd786740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556d52cfe530 con 0x556d52cfb3b0 2024-03-18T00:53:36.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.595+0000 7f79dd786740 10 client.0 _ll_register_callbacks cb 0x556d52c52010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:36.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.595+0000 7f79d8933640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf5850 0x556d52cffd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.596+0000 7f79d8933640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf5850 0x556d52cffd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58282/0 (socket says 172.21.15.49:58282) 2024-03-18T00:53:36.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.596+0000 7f79d8933640 1 -- 172.21.15.49:0/2037496100 learned_addr learned my addr 172.21.15.49:0/2037496100 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:36.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.596+0000 7f79d9134640 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52d072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.597+0000 7f79d9134640 1 -- 172.21.15.49:0/2037496100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf6240 msgr2=0x556d52d002b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:36.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.597+0000 7f79d9134640 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf6240 0x556d52d002b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.597+0000 7f79d9134640 1 -- 172.21.15.49:0/2037496100 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf5850 msgr2=0x556d52cffd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.597+0000 7f79d9134640 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf5850 0x556d52cffd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.597+0000 7f79d9134640 1 -- 172.21.15.49:0/2037496100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d52c25100 con 0x556d52cfb3b0 2024-03-18T00:53:36.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.597+0000 7f79d9134640 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52d072b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0049d0 tx=0x7f79cc045eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3195c7d07a8c29ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.600+0000 7f79b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:53:36.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.600+0000 7f79b3fff640 1 -- 172.21.15.49:0/2037496100 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc0ce160 con 0x556d52cfb3b0 2024-03-18T00:53:36.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.600+0000 7f79dd786740 1 -- 172.21.15.49:0/2037496100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79c400cb40 con 0x556d52cfb3b0 2024-03-18T00:53:36.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.600+0000 7f79b3fff640 1 -- 172.21.15.49:0/2037496100 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f79cc0ff3d0 con 0x556d52cfb3b0 2024-03-18T00:53:36.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.601+0000 7f79b3fff640 1 -- 172.21.15.49:0/2037496100 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f79cc0d22b0 con 0x556d52cfb3b0 2024-03-18T00:53:36.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.601+0000 7f79dd786740 10 client.25897 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:36.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.601+0000 7f79dd786740 1 -- 172.21.15.49:0/2037496100 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556d5296e3f0 con 0x556d52cfb3b0 2024-03-18T00:53:36.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.603+0000 7f79b3fff640 1 -- 172.21.15.49:0/2037496100 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c400cb40 con 0x556d52cfb3b0 2024-03-18T00:53:36.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.603+0000 7f79b3fff640 1 -- 172.21.15.49:0/2037496100 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f79cc03dce0 con 0x556d52cfb3b0 2024-03-18T00:53:36.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.603+0000 7f79dd786740 10 client.25897 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:36.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.603+0000 7f79dd786740 10 client.25897 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:53:36.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.603+0000 7f79dd786740 20 client.25897 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:36.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.603+0000 7f79dd786740 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x556d52d0bed0 0x556d52d0e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.603+0000 7f79dd786740 4 client.25897 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:36.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.603+0000 7f79dd786740 1 -- 172.21.15.49:0/2037496100 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d52c679d0 con 0x556d52d0bed0 2024-03-18T00:53:36.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.606+0000 7f79d8933640 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x556d52d0bed0 0x556d52d0e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.612+0000 7f79d8933640 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x556d52d0bed0 0x556d52d0e370 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fc390e6c9ae3c75d server_cookie=f86f4cd54481472f in_seq=0 out_seq=0 2024-03-18T00:53:36.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.612+0000 7f79b3fff640 10 client.25897 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:53:36.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.616+0000 7f79b3fff640 1 -- 172.21.15.49:0/2037496100 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556d52c679d0 con 0x556d52d0bed0 2024-03-18T00:53:36.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.616+0000 7f79b3fff640 10 client.25897 handle_command_reply: tid=0 2024-03-18T00:53:36.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.616+0000 7f79dd786740 1 client.25897 shutdown 2024-03-18T00:53:36.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.622+0000 7f79dd786740 1 -- 172.21.15.49:0/2037496100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 msgr2=0x556d52d072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.622+0000 7f79dd786740 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52d072b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0049d0 tx=0x7f79cc045eb0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.622+0000 7f79dd786740 1 -- 172.21.15.49:0/2037496100 shutdown_connections 2024-03-18T00:53:36.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.622+0000 7f79dd786740 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x556d52d0bed0 0x556d52d0e370 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.622+0000 7f79dd786740 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf6240 0x556d52d002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.622+0000 7f79dd786740 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52d072b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.622+0000 7f79dd786740 1 --2- 172.21.15.49:0/2037496100 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf5850 0x556d52cffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.622+0000 7f79dd786740 1 -- 172.21.15.49:0/2037496100 >> 172.21.15.49:0/2037496100 conn(0x556d52cf0dc0 msgr2=0x556d52ceffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.622+0000 7f79b3fff640 0 client.25897 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:53:36.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.624+0000 7f79dd786740 1 -- 172.21.15.49:0/2037496100 shutdown_connections 2024-03-18T00:53:36.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.624+0000 7f79dd786740 1 -- 172.21.15.49:0/2037496100 wait complete. 2024-03-18T00:53:36.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.628+0000 7f79dd786740 20 client.25897 trim_cache size 0 max 16384 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.690+0000 7f79dd786740 1 Processor -- start 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.690+0000 7f79dd786740 1 -- start start 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.691+0000 7f79dd786740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cf5850 0x556d52cef750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.691+0000 7f79dd786740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf6240 0x556d52d07270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.691+0000 7f79dd786740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cfb3b0 0x556d52cefc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.691+0000 7f79dd786740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f79c000e640 con 0x556d52cfb3b0 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.691+0000 7f79dd786740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f79c0010180 con 0x556d52cf6240 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.691+0000 7f79dd786740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f79c000ed10 con 0x556d52cf5850 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.691+0000 7f79d9134640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cfb3b0 0x556d52cefc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.691+0000 7f79d9134640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cfb3b0 0x556d52cefc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58284/0 (socket says 172.21.15.49:58284) 2024-03-18T00:53:36.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.691+0000 7f79d9134640 1 -- 172.21.15.49:0/1766061327 learned_addr learned my addr 172.21.15.49:0/1766061327 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:36.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.692+0000 7f79d3fff640 1 --2- 172.21.15.49:0/1766061327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf6240 0x556d52d07270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.693+0000 7f79d8933640 1 --2- 172.21.15.49:0/1766061327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cf5850 0x556d52cef750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.696+0000 7f79d9134640 1 -- 172.21.15.49:0/1766061327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cf5850 msgr2=0x556d52cef750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.696+0000 7f79d9134640 1 --2- 172.21.15.49:0/1766061327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cf5850 0x556d52cef750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.696+0000 7f79d9134640 1 -- 172.21.15.49:0/1766061327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf6240 msgr2=0x556d52d07270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.696+0000 7f79d9134640 1 --2- 172.21.15.49:0/1766061327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf6240 0x556d52d07270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.696+0000 7f79d9134640 1 -- 172.21.15.49:0/1766061327 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d52a387d0 con 0x556d52cfb3b0 2024-03-18T00:53:36.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.697+0000 7f79d9134640 1 --2- 172.21.15.49:0/1766061327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cfb3b0 0x556d52cefc90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0cde60 tx=0x7f79cc0d2190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43b29549f439a1c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.700+0000 7f79ba7fc640 1 -- 172.21.15.49:0/1766061327 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc0ce160 con 0x556d52cfb3b0 2024-03-18T00:53:36.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.700+0000 7f79ba7fc640 1 -- 172.21.15.49:0/1766061327 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79cc0d6900 con 0x556d52cfb3b0 2024-03-18T00:53:36.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.700+0000 7f79ba7fc640 1 -- 172.21.15.49:0/1766061327 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc0046d0 con 0x556d52cfb3b0 2024-03-18T00:53:36.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.702+0000 7f79dd786740 1 -- 172.21.15.49:0/1766061327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cfb3b0 msgr2=0x556d52cefc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.702+0000 7f79dd786740 1 --2- 172.21.15.49:0/1766061327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cfb3b0 0x556d52cefc90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0cde60 tx=0x7f79cc0d2190 comp rx=0 tx=0).stop 2024-03-18T00:53:36.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.703+0000 7f79dd786740 1 -- 172.21.15.49:0/1766061327 shutdown_connections 2024-03-18T00:53:36.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.703+0000 7f79dd786740 1 --2- 172.21.15.49:0/1766061327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf6240 0x556d52d07270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.703+0000 7f79dd786740 1 --2- 172.21.15.49:0/1766061327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cfb3b0 0x556d52cefc90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.703+0000 7f79dd786740 1 --2- 172.21.15.49:0/1766061327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cf5850 0x556d52cef750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.703+0000 7f79dd786740 1 -- 172.21.15.49:0/1766061327 >> 172.21.15.49:0/1766061327 conn(0x556d52cf0dc0 msgr2=0x556d52cf7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.704+0000 7f79dd786740 1 -- 172.21.15.49:0/1766061327 shutdown_connections 2024-03-18T00:53:36.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.705+0000 7f79dd786740 1 -- 172.21.15.49:0/1766061327 wait complete. 2024-03-18T00:53:36.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.706+0000 7f79dd786740 1 -- start start 2024-03-18T00:53:36.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.706+0000 7f79dd786740 1 Processor -- start 2024-03-18T00:53:36.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.707+0000 7f79dd786740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 0x556d52d04450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.707+0000 7f79dd786740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52d04990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.707+0000 7f79dd786740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52d059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.707+0000 7f79dd786740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f79c000fab0 con 0x556d52cf6240 2024-03-18T00:53:36.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.707+0000 7f79dd786740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f79c000f3e0 con 0x556d52cf5850 2024-03-18T00:53:36.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.707+0000 7f79dd786740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f79c000c620 con 0x556d52cfb3b0 2024-03-18T00:53:36.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.708+0000 7f79dd786740 10 client.0 _ll_register_callbacks cb 0x7f79cc0c9ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:36.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.708+0000 7f79d3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52d04990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.708+0000 7f79d3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52d04990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58292/0 (socket says 172.21.15.49:58292) 2024-03-18T00:53:36.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.708+0000 7f79d3fff640 1 -- 172.21.15.49:0/3586175508 learned_addr learned my addr 172.21.15.49:0/3586175508 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:36.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.708+0000 7f79d8933640 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 0x556d52d04450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.709+0000 7f79d3fff640 1 -- 172.21.15.49:0/3586175508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 msgr2=0x556d52d059a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.709+0000 7f79d3fff640 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52d059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.709+0000 7f79d3fff640 1 -- 172.21.15.49:0/3586175508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 msgr2=0x556d52d04450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:36.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.709+0000 7f79d3fff640 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 0x556d52d04450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.709+0000 7f79d3fff640 1 -- 172.21.15.49:0/3586175508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d52d05f90 con 0x556d52cf6240 2024-03-18T00:53:36.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.710+0000 7f79d3fff640 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52d04990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f79c801da00 tx=0x7f79c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f037c97000091a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:36.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.710+0000 7f79b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:36.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.711+0000 7f79b3fff640 1 -- 172.21.15.49:0/3586175508 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c801caa0 con 0x556d52cf6240 2024-03-18T00:53:36.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.711+0000 7f79dd786740 1 -- 172.21.15.49:0/3586175508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79cc0d6900 con 0x556d52cf6240 2024-03-18T00:53:36.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.711+0000 7f79b3fff640 1 -- 172.21.15.49:0/3586175508 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f79c801cc40 con 0x556d52cf6240 2024-03-18T00:53:36.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.712+0000 7f79b3fff640 1 -- 172.21.15.49:0/3586175508 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f79c801ce40 con 0x556d52cf6240 2024-03-18T00:53:36.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.712+0000 7f79dd786740 10 client.16281 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:36.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.712+0000 7f79dd786740 1 -- 172.21.15.49:0/3586175508 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556d52d07330 con 0x556d52cf6240 2024-03-18T00:53:36.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.715+0000 7f79b3fff640 1 -- 172.21.15.49:0/3586175508 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79c8005500 con 0x556d52cf6240 2024-03-18T00:53:36.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.716+0000 7f79b3fff640 1 -- 172.21.15.49:0/3586175508 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f79c8027530 con 0x556d52cf6240 2024-03-18T00:53:36.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.716+0000 7f79dd786740 10 client.16281 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:36.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.716+0000 7f79dd786740 10 client.16281 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:53:36.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.716+0000 7f79dd786740 20 client.16281 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:36.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.716+0000 7f79dd786740 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x556d52d72870 0x556d52d74c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:36.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.717+0000 7f79dd786740 4 client.16281 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:53:36.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.717+0000 7f79dd786740 1 -- 172.21.15.49:0/3586175508 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f79cc0d6900 con 0x556d52d72870 2024-03-18T00:53:36.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.718+0000 7f79d8933640 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x556d52d72870 0x556d52d74c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:36.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.722+0000 7f79d8933640 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x556d52d72870 0x556d52d74c90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac67199197e59697 server_cookie=ea4041dc184a6003 in_seq=0 out_seq=0 2024-03-18T00:53:36.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.723+0000 7f79b3fff640 10 client.16281 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:53:36.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.723+0000 7f79b3fff640 1 -- 172.21.15.49:0/3586175508 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f79cc0d6900 con 0x556d52d72870 2024-03-18T00:53:36.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.723+0000 7f79b3fff640 10 client.16281 handle_command_reply: tid=0 2024-03-18T00:53:36.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.723+0000 7f79dd786740 1 client.16281 shutdown 2024-03-18T00:53:36.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.725+0000 7f79dd786740 1 -- 172.21.15.49:0/3586175508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 msgr2=0x556d52d04990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.725+0000 7f79dd786740 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52d04990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f79c801da00 tx=0x7f79c8004730 comp rx=0 tx=0).stop 2024-03-18T00:53:36.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.730+0000 7f79dd786740 1 -- 172.21.15.49:0/3586175508 shutdown_connections 2024-03-18T00:53:36.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.730+0000 7f79dd786740 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x556d52d72870 0x556d52d74c90 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.730+0000 7f79dd786740 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556d52cfb3b0 0x556d52d059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.730+0000 7f79dd786740 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556d52cf6240 0x556d52d04990 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.730+0000 7f79dd786740 1 --2- 172.21.15.49:0/3586175508 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556d52cf5850 0x556d52d04450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.730+0000 7f79dd786740 1 -- 172.21.15.49:0/3586175508 >> 172.21.15.49:0/3586175508 conn(0x556d52cf0dc0 msgr2=0x556d52cef750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.730+0000 7f79b3fff640 0 client.16281 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:53:36.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.733+0000 7f79dd786740 1 -- 172.21.15.49:0/3586175508 shutdown_connections 2024-03-18T00:53:36.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.737+0000 7f79dd786740 1 -- 172.21.15.49:0/3586175508 wait complete. 2024-03-18T00:53:36.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.738+0000 7f79dd786740 20 client.16281 trim_cache size 0 max 16384 2024-03-18T00:53:36.741 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:53:36.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.739+0000 7f79dab97640 1 -- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f79bc0a2a10 msgr2=0x7f79bc0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.739+0000 7f79dab97640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f79bc0a2a10 0x7f79bc0a4ed0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f79c80058a0 tx=0x7f79c800b710 comp rx=0 tx=0).stop 2024-03-18T00:53:36.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.739+0000 7f79dab97640 1 -- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 msgr2=0x7f79d4079a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:36.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.739+0000 7f79dab97640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 0x7f79d4079a40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f79cc01ac00 tx=0x7f79cc003f10 comp rx=0 tx=0).stop 2024-03-18T00:53:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.740+0000 7f79dab97640 1 -- 172.21.15.49:0/3990608812 shutdown_connections 2024-03-18T00:53:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.740+0000 7f79dab97640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f79bc0a2a10 0x7f79bc0a4ed0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.740+0000 7f79dab97640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79d410b150 0x7f79d419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.740+0000 7f79dab97640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79d410db00 0x7f79d4079a40 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.740+0000 7f79dab97640 1 --2- 172.21.15.49:0/3990608812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79d41087e0 0x7f79d419e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:36.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.740+0000 7f79dab97640 1 -- 172.21.15.49:0/3990608812 >> 172.21.15.49:0/3990608812 conn(0x7f79d406e640 msgr2=0x7f79d4108f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:36.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.741+0000 7f79dab97640 1 -- 172.21.15.49:0/3990608812 shutdown_connections 2024-03-18T00:53:36.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:36.741+0000 7f79dab97640 1 -- 172.21.15.49:0/3990608812 wait complete. 2024-03-18T00:53:36.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:53:37.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.053+0000 7fc1cf59e640 1 -- 172.21.15.49:0/2437120562 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 msgr2=0x7fc1d010e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:37.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.053+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/2437120562 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 0x7fc1d010e110 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc1c401a910 tx=0x7fc1c403cca0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.053+0000 7fc1cf59e640 1 -- 172.21.15.49:0/2437120562 shutdown_connections 2024-03-18T00:53:37.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.053+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/2437120562 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 0x7fc1d010e110 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.053+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/2437120562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d0108eb0 0x7fc1d0109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.053+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/2437120562 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1d0072bf0 0x7fc1d0073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.053+0000 7fc1cf59e640 1 -- 172.21.15.49:0/2437120562 >> 172.21.15.49:0/2437120562 conn(0x7fc1d006e6c0 msgr2=0x7fc1d0070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:37.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.064+0000 7fc1cf59e640 1 -- 172.21.15.49:0/2437120562 shutdown_connections 2024-03-18T00:53:37.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.068+0000 7fc1cf59e640 1 -- 172.21.15.49:0/2437120562 wait complete. 2024-03-18T00:53:37.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.069+0000 7fc1cf59e640 1 Processor -- start 2024-03-18T00:53:37.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.069+0000 7fc1cf59e640 1 -- start start 2024-03-18T00:53:37.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.069+0000 7fc1cf59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d0072bf0 0x7fc1d019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.069+0000 7fc1cf59e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1d0108eb0 0x7fc1d019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.069+0000 7fc1cf59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 0x7fc1d01a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.069+0000 7fc1cf59e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0111ac0 con 0x7fc1d0109870 2024-03-18T00:53:37.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.069+0000 7fc1cf59e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0111940 con 0x7fc1d0108eb0 2024-03-18T00:53:37.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.069+0000 7fc1cf59e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d0111c40 con 0x7fc1d0072bf0 2024-03-18T00:53:37.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.070+0000 7fc1ced9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 0x7fc1d01a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.070+0000 7fc1ced9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 0x7fc1d01a54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58312/0 (socket says 172.21.15.49:58312) 2024-03-18T00:53:37.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.070+0000 7fc1ced9d640 1 -- 172.21.15.49:0/1823424327 learned_addr learned my addr 172.21.15.49:0/1823424327 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:37.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.071+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d0072bf0 0x7fc1d019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.072+0000 7fc1ced9d640 1 -- 172.21.15.49:0/1823424327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d0072bf0 msgr2=0x7fc1d019df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:37.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.072+0000 7fc1ced9d640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d0072bf0 0x7fc1d019df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.072+0000 7fc1ced9d640 1 -- 172.21.15.49:0/1823424327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1d0108eb0 msgr2=0x7fc1d019e440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:53:37.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.072+0000 7fc1ced9d640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1d0108eb0 0x7fc1d019e440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.072+0000 7fc1ced9d640 1 -- 172.21.15.49:0/1823424327 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d01a5b20 con 0x7fc1d0109870 2024-03-18T00:53:37.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.078+0000 7fc1ced9d640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 0x7fc1d01a54b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4004a30 tx=0x7fc1c4004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:37.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.080+0000 7fc1bf7fe640 1 -- 172.21.15.49:0/1823424327 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4051020 con 0x7fc1d0109870 2024-03-18T00:53:37.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.080+0000 7fc1cf59e640 1 -- 172.21.15.49:0/1823424327 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1d01a5db0 con 0x7fc1d0109870 2024-03-18T00:53:37.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.080+0000 7fc1cf59e640 1 -- 172.21.15.49:0/1823424327 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1d01a6240 con 0x7fc1d0109870 2024-03-18T00:53:37.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.087+0000 7fc1bf7fe640 1 -- 172.21.15.49:0/1823424327 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c4004e50 con 0x7fc1d0109870 2024-03-18T00:53:37.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.087+0000 7fc1bf7fe640 1 -- 172.21.15.49:0/1823424327 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c404d5e0 con 0x7fc1d0109870 2024-03-18T00:53:37.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.091+0000 7fc1bf7fe640 1 -- 172.21.15.49:0/1823424327 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc1c4054020 con 0x7fc1d0109870 2024-03-18T00:53:37.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.092+0000 7fc1d75ab740 1 Processor -- start 2024-03-18T00:53:37.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.092+0000 7fc1d75ab740 1 -- start start 2024-03-18T00:53:37.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.093+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216ca40 0x55d97216ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.093+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972171f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.093+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9721725a0 0x55d972174a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.093+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d9720d1490 con 0x55d97216d430 2024-03-18T00:53:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.093+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d972079c90 con 0x55d9721725a0 2024-03-18T00:53:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.093+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d9720cd530 con 0x55d97216ca40 2024-03-18T00:53:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.094+0000 7fc1bf7fe640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1b80a2a00 0x7fc1b80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.094+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972171f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.094+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972171f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58316/0 (socket says 172.21.15.49:58316) 2024-03-18T00:53:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.094+0000 7fc1ce59c640 1 -- 172.21.15.49:0/1724797916 learned_addr learned my addr 172.21.15.49:0/1724797916 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:37.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.095+0000 7fc1bf7fe640 1 -- 172.21.15.49:0/1823424327 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc1c41024d0 con 0x7fc1d0109870 2024-03-18T00:53:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.095+0000 7fc1ced9d640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1b80a2a00 0x7fc1b80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.095+0000 7fc1ce59c640 1 -- 172.21.15.49:0/1724797916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216ca40 msgr2=0x55d97216ce60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.095+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/1724797916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216ca40 0x55d97216ce60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.095+0000 7fc1ce59c640 1 -- 172.21.15.49:0/1724797916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9721725a0 msgr2=0x55d972174a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.095+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/1724797916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9721725a0 0x55d972174a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.095+0000 7fc1ce59c640 1 -- 172.21.15.49:0/1724797916 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d972134500 con 0x55d97216d430 2024-03-18T00:53:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.096+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/1724797916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972171f30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800af10 tx=0x7fc1c800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec6988405754b9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:37.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.096+0000 7fc1ced9d640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1b80a2a00 0x7fc1b80a4ec0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4004ad0 tx=0x7fc1c40057f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:37.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.097+0000 7fc1bcff9640 1 -- 172.21.15.49:0/1724797916 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8010840 con 0x55d97216d430 2024-03-18T00:53:37.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.097+0000 7fc1bcff9640 1 -- 172.21.15.49:0/1724797916 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c8018430 con 0x55d97216d430 2024-03-18T00:53:37.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.097+0000 7fc1bcff9640 1 -- 172.21.15.49:0/1724797916 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8017640 con 0x55d97216d430 2024-03-18T00:53:37.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.099+0000 7fc1d75ab740 1 -- 172.21.15.49:0/1724797916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 msgr2=0x55d972171f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:37.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.099+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/1724797916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972171f30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800af10 tx=0x7fc1c800f790 comp rx=0 tx=0).stop 2024-03-18T00:53:37.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.099+0000 7fc1d75ab740 1 -- 172.21.15.49:0/1724797916 shutdown_connections 2024-03-18T00:53:37.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.099+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/1724797916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9721725a0 0x55d972174a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.099+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/1724797916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972171f30 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.099+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/1724797916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216ca40 0x55d97216ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.099+0000 7fc1d75ab740 1 -- 172.21.15.49:0/1724797916 >> 172.21.15.49:0/1724797916 conn(0x55d972167fb0 msgr2=0x55d9721683b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:37.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.099+0000 7fc1d75ab740 1 -- 172.21.15.49:0/1724797916 shutdown_connections 2024-03-18T00:53:37.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.100+0000 7fc1d75ab740 1 -- 172.21.15.49:0/1724797916 wait complete. 2024-03-18T00:53:37.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.100+0000 7fc1d75ab740 1 -- start start 2024-03-18T00:53:37.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.101+0000 7fc1d75ab740 1 Processor -- start 2024-03-18T00:53:37.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.101+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 0x55d972176fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.101+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972177500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.101+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97217e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.101+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d972175550 con 0x55d97216d430 2024-03-18T00:53:37.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.101+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d97208d230 con 0x55d97216ca40 2024-03-18T00:53:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.101+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d9721756d0 con 0x55d9721725a0 2024-03-18T00:53:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.101+0000 7fc1d75ab740 10 client.0 _ll_register_callbacks cb 0x55d9720c9010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.102+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972177500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.102+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972177500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58326/0 (socket says 172.21.15.49:58326) 2024-03-18T00:53:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.102+0000 7fc1ce59c640 1 -- 172.21.15.49:0/3139679825 learned_addr learned my addr 172.21.15.49:0/3139679825 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:37.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.103+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 0x55d972176fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.103+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97217e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.104+0000 7fc1ce59c640 1 -- 172.21.15.49:0/3139679825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 msgr2=0x55d97217e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:37.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.104+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97217e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.104+0000 7fc1ce59c640 1 -- 172.21.15.49:0/3139679825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 msgr2=0x55d972176fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:37.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.104+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 0x55d972176fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.104+0000 7fc1ce59c640 1 -- 172.21.15.49:0/3139679825 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d97209c100 con 0x55d97216d430 2024-03-18T00:53:37.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.105+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972177500 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8018d40 tx=0x7fc1c8003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=240f951fe5d0e1e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:37.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.108+0000 7fc1b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:37.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.108+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3139679825 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8026020 con 0x55d97216d430 2024-03-18T00:53:37.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.108+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3139679825 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1c8018430 con 0x55d97216d430 2024-03-18T00:53:37.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.111+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3139679825 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc1c8004c40 con 0x55d97216d430 2024-03-18T00:53:37.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.112+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3139679825 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc1c8031050 con 0x55d97216d430 2024-03-18T00:53:37.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.117+0000 7fc1d75ab740 10 client.16293 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:37.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.117+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3139679825 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d971de53f0 con 0x55d97216d430 2024-03-18T00:53:37.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.118+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3139679825 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c8018430 con 0x55d97216d430 2024-03-18T00:53:37.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.118+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3139679825 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc1c8010920 con 0x55d97216d430 2024-03-18T00:53:37.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.119+0000 7fc1d75ab740 10 client.16293 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:37.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.119+0000 7fc1d75ab740 10 client.16293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:37.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.119+0000 7fc1d75ab740 20 client.16293 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:37.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.119+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d972182b70 0x55d972184f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.119+0000 7fc1d75ab740 4 client.16293 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:37.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.119+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3139679825 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9720de9d0 con 0x55d972182b70 2024-03-18T00:53:37.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.120+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d972182b70 0x55d972184f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.125+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d972182b70 0x55d972184f60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=217f03dbadb1159b server_cookie=51a77cfc4f60b0d8 in_seq=0 out_seq=0 2024-03-18T00:53:37.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.128+0000 7fc1b5ffb640 10 client.16293 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:37.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.130+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3139679825 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d9720de9d0 con 0x55d972182b70 2024-03-18T00:53:37.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.130+0000 7fc1b5ffb640 10 client.16293 handle_command_reply: tid=0 2024-03-18T00:53:37.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.131+0000 7fc1d75ab740 1 client.16293 shutdown 2024-03-18T00:53:37.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.133+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3139679825 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 msgr2=0x55d972177500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:37.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.133+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972177500 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8018d40 tx=0x7fc1c8003ba0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.135+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3139679825 shutdown_connections 2024-03-18T00:53:37.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.135+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d972182b70 0x55d972184f60 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.135+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97217e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.135+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216d430 0x55d972177500 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.135+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3139679825 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 0x55d972176fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.135+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3139679825 >> 172.21.15.49:0/3139679825 conn(0x55d972167fb0 msgr2=0x55d972167140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:37.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.135+0000 7fc1b5ffb640 0 client.16293 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:37.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.135+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3139679825 shutdown_connections 2024-03-18T00:53:37.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.136+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3139679825 wait complete. 2024-03-18T00:53:37.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.136+0000 7fc1d75ab740 20 client.16293 trim_cache size 0 max 16384 2024-03-18T00:53:37.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.172+0000 7fc1d75ab740 1 Processor -- start 2024-03-18T00:53:37.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.173+0000 7fc1d75ab740 1 -- start start 2024-03-18T00:53:37.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.174+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216ca40 0x55d97216e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.175+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216d430 0x55d97217e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.175+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97216ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.175+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b000e6d0 con 0x55d97216ca40 2024-03-18T00:53:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.175+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b0010210 con 0x55d97216d430 2024-03-18T00:53:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.175+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b000eda0 con 0x55d9721725a0 2024-03-18T00:53:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.175+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97216ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.175+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97216ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:48590/0 (socket says 172.21.15.49:48590) 2024-03-18T00:53:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.175+0000 7fc1cdd9b640 1 -- 172.21.15.49:0/2581956783 learned_addr learned my addr 172.21.15.49:0/2581956783 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:37.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.176+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/2581956783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216ca40 0x55d97216e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.177+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/2581956783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216d430 0x55d97217e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.177+0000 7fc1cdd9b640 1 -- 172.21.15.49:0/2581956783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216d430 msgr2=0x55d97217e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.177+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/2581956783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216d430 0x55d97217e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.177+0000 7fc1cdd9b640 1 -- 172.21.15.49:0/2581956783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216ca40 msgr2=0x55d97216e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.177+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/2581956783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216ca40 0x55d97216e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.177+0000 7fc1cdd9b640 1 -- 172.21.15.49:0/2581956783 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d971eaf7d0 con 0x55d9721725a0 2024-03-18T00:53:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.178+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/2581956783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97216ee30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0000c00 tx=0x7fc1c0001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1055d4aaa4d7ded server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:37.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.182+0000 7fc1b7fff640 1 -- 172.21.15.49:0/2581956783 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c001bc70 con 0x55d9721725a0 2024-03-18T00:53:37.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.182+0000 7fc1b7fff640 1 -- 172.21.15.49:0/2581956783 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c001be10 con 0x55d9721725a0 2024-03-18T00:53:37.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.182+0000 7fc1b7fff640 1 -- 172.21.15.49:0/2581956783 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c0004a00 con 0x55d9721725a0 2024-03-18T00:53:37.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.184+0000 7fc1d75ab740 1 -- 172.21.15.49:0/2581956783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 msgr2=0x55d97216ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:37.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.184+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/2581956783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97216ee30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0000c00 tx=0x7fc1c0001e70 comp rx=0 tx=0).stop 2024-03-18T00:53:37.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.187+0000 7fc1d75ab740 1 -- 172.21.15.49:0/2581956783 shutdown_connections 2024-03-18T00:53:37.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.187+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/2581956783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216d430 0x55d97217e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.187+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/2581956783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9721725a0 0x55d97216ee30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.187+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/2581956783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d97216ca40 0x55d97216e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.187+0000 7fc1d75ab740 1 -- 172.21.15.49:0/2581956783 >> 172.21.15.49:0/2581956783 conn(0x55d972167fb0 msgr2=0x55d972176fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:37.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.187+0000 7fc1d75ab740 1 -- 172.21.15.49:0/2581956783 shutdown_connections 2024-03-18T00:53:37.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.187+0000 7fc1d75ab740 1 -- 172.21.15.49:0/2581956783 wait complete. 2024-03-18T00:53:37.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.188+0000 7fc1d75ab740 1 -- start start 2024-03-18T00:53:37.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.188+0000 7fc1d75ab740 1 Processor -- start 2024-03-18T00:53:37.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.188+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 0x55d97217b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.189+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216d430 0x55d97217b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.189+0000 7fc1d75ab740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9721725a0 0x55d9721e6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.189+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b000fb40 con 0x55d9721725a0 2024-03-18T00:53:37.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.189+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b000f470 con 0x55d97216ca40 2024-03-18T00:53:37.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.189+0000 7fc1d75ab740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc1b000c6b0 con 0x55d97216d430 2024-03-18T00:53:37.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.189+0000 7fc1d75ab740 10 client.0 _ll_register_callbacks cb 0x7fc1c800ad60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:37.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.190+0000 7fc1cdd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 0x55d97217b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.190+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216d430 0x55d97217b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.190+0000 7fc1ce59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216d430 0x55d97217b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:48594/0 (socket says 172.21.15.49:48594) 2024-03-18T00:53:37.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.190+0000 7fc1ce59c640 1 -- 172.21.15.49:0/3485643491 learned_addr learned my addr 172.21.15.49:0/3485643491 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:37.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.191+0000 7fc1ce59c640 1 -- 172.21.15.49:0/3485643491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 msgr2=0x55d97217b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:37.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.191+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 0x55d97217b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.191+0000 7fc1ce59c640 1 -- 172.21.15.49:0/3485643491 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9721725a0 msgr2=0x55d9721e6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:37.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.191+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9721725a0 0x55d9721e6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.191+0000 7fc1ce59c640 1 -- 172.21.15.49:0/3485643491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d97217e700 con 0x55d97216d430 2024-03-18T00:53:37.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.192+0000 7fc1ce59c640 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216d430 0x55d97217b630 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8000f80 tx=0x7fc1c8003c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d4199a0c6948dac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:37.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.193+0000 7fc1b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:53:37.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.196+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3485643491 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8026030 con 0x55d97216d430 2024-03-18T00:53:37.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.196+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3485643491 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc1c8004a00 con 0x55d97216d430 2024-03-18T00:53:37.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.196+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3485643491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1c001be10 con 0x55d97216d430 2024-03-18T00:53:37.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.209+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3485643491 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc1c8018430 con 0x55d97216d430 2024-03-18T00:53:37.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.209+0000 7fc1d75ab740 10 client.25912 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:37.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.209+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3485643491 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d9721e8280 con 0x55d97216d430 2024-03-18T00:53:37.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.210+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3485643491 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1c001be10 con 0x55d97216d430 2024-03-18T00:53:37.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.210+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3485643491 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc1c8022850 con 0x55d97216d430 2024-03-18T00:53:37.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.211+0000 7fc1d75ab740 10 client.25912 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:37.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.211+0000 7fc1d75ab740 10 client.25912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:37.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.211+0000 7fc1d75ab740 20 client.25912 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:37.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.211+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9721e8c50 0x55d9721eb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:37.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.211+0000 7fc1d75ab740 4 client.25912 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:53:37.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.211+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3485643491 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9720d8d20 con 0x55d9721e8c50 2024-03-18T00:53:37.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.215+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9721e8c50 0x55d9721eb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:37.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.231+0000 7fc1cdd9b640 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9721e8c50 0x55d9721eb070 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad47f7046a3773bc server_cookie=6943b31c123b935 in_seq=0 out_seq=0 2024-03-18T00:53:37.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.233+0000 7fc1b5ffb640 10 client.25912 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:37.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1b5ffb640 1 -- 172.21.15.49:0/3485643491 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9720d8d20 con 0x55d9721e8c50 2024-03-18T00:53:37.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1b5ffb640 10 client.25912 handle_command_reply: tid=0 2024-03-18T00:53:37.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1d75ab740 1 client.25912 shutdown 2024-03-18T00:53:37.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3485643491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216d430 msgr2=0x55d97217b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:37.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216d430 0x55d97217b630 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc1c8000f80 tx=0x7fc1c8003c90 comp rx=0 tx=0).stop 2024-03-18T00:53:37.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3485643491 shutdown_connections 2024-03-18T00:53:37.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9721e8c50 0x55d9721eb070 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9721725a0 0x55d9721e6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d97216d430 0x55d97217b630 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1d75ab740 1 --2- 172.21.15.49:0/3485643491 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d97216ca40 0x55d97217b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.235+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3485643491 >> 172.21.15.49:0/3485643491 conn(0x55d972167fb0 msgr2=0x55d97216e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:37.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.236+0000 7fc1b5ffb640 0 client.25912 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:37.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.236+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3485643491 shutdown_connections 2024-03-18T00:53:37.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.236+0000 7fc1d75ab740 1 -- 172.21.15.49:0/3485643491 wait complete. 2024-03-18T00:53:37.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.236+0000 7fc1d75ab740 20 client.25912 trim_cache size 0 max 16384 2024-03-18T00:53:37.239 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:53:37.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.238+0000 7fc1cf59e640 1 -- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1b80a2a00 msgr2=0x7fc1b80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:37.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.239+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1b80a2a00 0x7fc1b80a4ec0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4004ad0 tx=0x7fc1c40057f0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.239+0000 7fc1cf59e640 1 -- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 msgr2=0x7fc1d01a54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:37.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.239+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 0x7fc1d01a54b0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4004a30 tx=0x7fc1c4004a60 comp rx=0 tx=0).stop 2024-03-18T00:53:37.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.239+0000 7fc1cf59e640 1 -- 172.21.15.49:0/1823424327 shutdown_connections 2024-03-18T00:53:37.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.239+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc1b80a2a00 0x7fc1b80a4ec0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.239+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc1d0108eb0 0x7fc1d019e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.239+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc1d0109870 0x7fc1d01a54b0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.239+0000 7fc1cf59e640 1 --2- 172.21.15.49:0/1823424327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc1d0072bf0 0x7fc1d019df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:37.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.239+0000 7fc1cf59e640 1 -- 172.21.15.49:0/1823424327 >> 172.21.15.49:0/1823424327 conn(0x7fc1d006e6c0 msgr2=0x7fc1d010c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:37.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.241+0000 7fc1cf59e640 1 -- 172.21.15.49:0/1823424327 shutdown_connections 2024-03-18T00:53:37.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:37.243+0000 7fc1cf59e640 1 -- 172.21.15.49:0/1823424327 wait complete. 2024-03-18T00:53:37.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:37 smithi049 ceph-mon[32130]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 834 MiB/s wr, 1.31k op/s 2024-03-18T00:53:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:37 smithi107 ceph-mon[36270]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 834 MiB/s wr, 1.31k op/s 2024-03-18T00:53:37.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:37 smithi106 ceph-mon[35899]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 834 MiB/s wr, 1.31k op/s 2024-03-18T00:53:38.279 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:53:38.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:53:38.279 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:53:38.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.594+0000 7f79a97b7640 1 -- 172.21.15.49:0/438952622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79a4078de0 msgr2=0x7f79a407b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:38.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.594+0000 7f79a97b7640 1 --2- 172.21.15.49:0/438952622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79a4078de0 0x7f79a407b270 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f79940097d0 tx=0x7f799402bc60 comp rx=0 tx=0).stop 2024-03-18T00:53:38.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.594+0000 7f79a97b7640 1 -- 172.21.15.49:0/438952622 shutdown_connections 2024-03-18T00:53:38.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.594+0000 7f79a97b7640 1 --2- 172.21.15.49:0/438952622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79a4078de0 0x7f79a407b270 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.594+0000 7f79a97b7640 1 --2- 172.21.15.49:0/438952622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79a4076430 0x7f79a40788a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.594+0000 7f79a97b7640 1 --2- 172.21.15.49:0/438952622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79a4073550 0x7f79a4073970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.594+0000 7f79a97b7640 1 -- 172.21.15.49:0/438952622 >> 172.21.15.49:0/438952622 conn(0x7f79a406e6c0 msgr2=0x7f79a4070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:38.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.595+0000 7f79a97b7640 1 -- 172.21.15.49:0/438952622 shutdown_connections 2024-03-18T00:53:38.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.595+0000 7f79a97b7640 1 -- 172.21.15.49:0/438952622 wait complete. 2024-03-18T00:53:38.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.596+0000 7f79a97b7640 1 Processor -- start 2024-03-18T00:53:38.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.596+0000 7f79a97b7640 1 -- start start 2024-03-18T00:53:38.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.596+0000 7f79a97b7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79a4073550 0x7f79a41b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.596+0000 7f79a97b7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79a4076430 0x7f79a41b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.596+0000 7f79a97b7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79a4078de0 0x7f79a41b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.596+0000 7f79a97b7640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f79a407ec20 con 0x7f79a4073550 2024-03-18T00:53:38.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.596+0000 7f79a97b7640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f79a407eaa0 con 0x7f79a4076430 2024-03-18T00:53:38.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.596+0000 7f79a97b7640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f79a407eda0 con 0x7f79a4078de0 2024-03-18T00:53:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.599+0000 7f79a3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79a4073550 0x7f79a41b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.599+0000 7f79a3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79a4073550 0x7f79a41b1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36610/0 (socket says 172.21.15.49:36610) 2024-03-18T00:53:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.599+0000 7f79a3fff640 1 -- 172.21.15.49:0/2282788202 learned_addr learned my addr 172.21.15.49:0/2282788202 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.599+0000 7f79a8fb6640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79a4078de0 0x7f79a41b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.599+0000 7f79a37fe640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79a4076430 0x7f79a41b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.600+0000 7f79a37fe640 1 -- 172.21.15.49:0/2282788202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79a4078de0 msgr2=0x7f79a41b62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.601+0000 7f79a37fe640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79a4078de0 0x7f79a41b62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.601+0000 7f79a37fe640 1 -- 172.21.15.49:0/2282788202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79a4073550 msgr2=0x7f79a41b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:38.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.601+0000 7f79a37fe640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79a4073550 0x7f79a41b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.601+0000 7f79a37fe640 1 -- 172.21.15.49:0/2282788202 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a41b6b90 con 0x7f79a4076430 2024-03-18T00:53:38.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.601+0000 7f79a8fb6640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79a4078de0 0x7f79a41b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:53:38.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.601+0000 7f79a37fe640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79a4076430 0x7f79a41b3c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f799800ea90 tx=0x7f799800ef50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:38.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.605+0000 7f79a17fa640 1 -- 172.21.15.49:0/2282788202 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799801b070 con 0x7f79a4076430 2024-03-18T00:53:38.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.605+0000 7f79a97b7640 1 -- 172.21.15.49:0/2282788202 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a41b6e80 con 0x7f79a4076430 2024-03-18T00:53:38.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.605+0000 7f79a97b7640 1 -- 172.21.15.49:0/2282788202 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a41b73d0 con 0x7f79a4076430 2024-03-18T00:53:38.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.609+0000 7f79ad608740 1 Processor -- start 2024-03-18T00:53:38.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.609+0000 7f79ad608740 1 -- start start 2024-03-18T00:53:38.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.610+0000 7f79a17fa640 1 -- 172.21.15.49:0/2282788202 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f799800cc80 con 0x7f79a4076430 2024-03-18T00:53:38.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.610+0000 7f79ad608740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fe8b80 0x55b5e5fe8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.610+0000 7f79ad608740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5fee070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.610+0000 7f79ad608740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.610+0000 7f79ad608740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b5e5f4d0d0 con 0x55b5e5fe8b80 2024-03-18T00:53:38.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.610+0000 7f79ad608740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b5e5ef5c70 con 0x55b5e5fe9570 2024-03-18T00:53:38.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.610+0000 7f79ad608740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b5e5f49170 con 0x55b5e5fee6e0 2024-03-18T00:53:38.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.611+0000 7f79a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.611+0000 7f79a37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:52406/0 (socket says 172.21.15.49:52406) 2024-03-18T00:53:38.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.611+0000 7f79a37fe640 1 -- 172.21.15.49:0/675809308 learned_addr learned my addr 172.21.15.49:0/675809308 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:38.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.613+0000 7f79a3fff640 1 --2- 172.21.15.49:0/675809308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5fee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.613+0000 7f79a8fb6640 1 --2- 172.21.15.49:0/675809308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fe8b80 0x55b5e5fe8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.614+0000 7f79a37fe640 1 -- 172.21.15.49:0/675809308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 msgr2=0x55b5e5fee070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:38.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.614+0000 7f79a37fe640 1 --2- 172.21.15.49:0/675809308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5fee070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.614+0000 7f79a37fe640 1 -- 172.21.15.49:0/675809308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fe8b80 msgr2=0x55b5e5fe8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:38.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.614+0000 7f79a37fe640 1 --2- 172.21.15.49:0/675809308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fe8b80 0x55b5e5fe8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.614+0000 7f79a37fe640 1 -- 172.21.15.49:0/675809308 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e5d297e0 con 0x55b5e5fee6e0 2024-03-18T00:53:38.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.615+0000 7f79a17fa640 1 -- 172.21.15.49:0/2282788202 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799800b760 con 0x7f79a4076430 2024-03-18T00:53:38.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.615+0000 7f79a37fe640 1 --2- 172.21.15.49:0/675809308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff0b70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f79980d3c50 tx=0x7f79980d6d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=167a60569eac7536 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:38.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.615+0000 7f79a17fa640 1 -- 172.21.15.49:0/2282788202 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f799801e070 con 0x7f79a4076430 2024-03-18T00:53:38.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.616+0000 7f798a7fc640 1 -- 172.21.15.49:0/675809308 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980dd070 con 0x55b5e5fee6e0 2024-03-18T00:53:38.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.618+0000 7f798a7fc640 1 -- 172.21.15.49:0/675809308 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79980db070 con 0x55b5e5fee6e0 2024-03-18T00:53:38.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.619+0000 7f79ad608740 1 -- 172.21.15.49:0/675809308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 msgr2=0x55b5e5ff0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:38.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.619+0000 7f79ad608740 1 --2- 172.21.15.49:0/675809308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff0b70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f79980d3c50 tx=0x7f79980d6d40 comp rx=0 tx=0).stop 2024-03-18T00:53:38.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.620+0000 7f79a17fa640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f798c0a28f0 0x7f798c0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.620+0000 7f79a17fa640 1 -- 172.21.15.49:0/2282788202 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f79980d61b0 con 0x7f79a4076430 2024-03-18T00:53:38.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.621+0000 7f79a8fb6640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f798c0a28f0 0x7f798c0a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.622+0000 7f79ad608740 1 -- 172.21.15.49:0/675809308 shutdown_connections 2024-03-18T00:53:38.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.622+0000 7f79ad608740 1 --2- 172.21.15.49:0/675809308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5fee070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.622+0000 7f79ad608740 1 --2- 172.21.15.49:0/675809308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff0b70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.622+0000 7f79ad608740 1 --2- 172.21.15.49:0/675809308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fe8b80 0x55b5e5fe8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.622+0000 7f79ad608740 1 -- 172.21.15.49:0/675809308 >> 172.21.15.49:0/675809308 conn(0x55b5e5fb0c50 msgr2=0x55b5e5fe44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:38.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.625+0000 7f79ad608740 1 -- 172.21.15.49:0/675809308 shutdown_connections 2024-03-18T00:53:38.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.626+0000 7f79ad608740 1 -- 172.21.15.49:0/675809308 wait complete. 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.627+0000 7f79ad608740 1 -- start start 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.627+0000 7f79ad608740 1 Processor -- start 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.628+0000 7f79a8fb6640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f798c0a28f0 0x7f798c0a4db0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7994009590 tx=0x7f7994005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.629+0000 7f79ad608740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fe8b80 0x55b5e5ff3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.629+0000 7f79ad608740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.629+0000 7f79ad608740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ffa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.629+0000 7f79ad608740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b5e5ff16e0 con 0x55b5e5fe8b80 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.629+0000 7f79ad608740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b5e5f090d0 con 0x55b5e5fe9570 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.629+0000 7f79ad608740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b5e5ff1860 con 0x55b5e5fee6e0 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.629+0000 7f79ad608740 10 client.0 _ll_register_callbacks cb 0x55b5e5ed2760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.630+0000 7f79a37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.630+0000 7f79a37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47998/0 (socket says 172.21.15.49:47998) 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.630+0000 7f79a37fe640 1 -- 172.21.15.49:0/2963764772 learned_addr learned my addr 172.21.15.49:0/2963764772 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.630+0000 7f79a37fe640 1 -- 172.21.15.49:0/2963764772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 msgr2=0x55b5e5ffa6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.630+0000 7f79a37fe640 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ffa6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.630+0000 7f79a37fe640 1 -- 172.21.15.49:0/2963764772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fe8b80 msgr2=0x55b5e5ff3160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.630+0000 7f79a37fe640 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fe8b80 0x55b5e5ff3160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.630+0000 7f79a37fe640 1 -- 172.21.15.49:0/2963764772 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e5d89f70 con 0x55b5e5fe9570 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.631+0000 7f79a37fe640 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff36a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7998000c00 tx=0x7f799800cf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11e416bd6391df48 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.631+0000 7f7987ff7640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.631+0000 7f7987ff7640 1 -- 172.21.15.49:0/2963764772 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980dd070 con 0x55b5e5fe9570 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.631+0000 7f79ad608740 1 -- 172.21.15.49:0/2963764772 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79980db070 con 0x55b5e5fe9570 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.632+0000 7f7987ff7640 1 -- 172.21.15.49:0/2963764772 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f79980d63b0 con 0x55b5e5fe9570 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.633+0000 7f7987ff7640 1 -- 172.21.15.49:0/2963764772 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f799809ed60 con 0x55b5e5fe9570 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.633+0000 7f79ad608740 10 client.25847 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:38.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.633+0000 7f79ad608740 1 -- 172.21.15.49:0/2963764772 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5e5c21170 con 0x55b5e5fe9570 2024-03-18T00:53:38.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.634+0000 7f7987ff7640 1 -- 172.21.15.49:0/2963764772 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79980db070 con 0x55b5e5fe9570 2024-03-18T00:53:38.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.634+0000 7f7987ff7640 1 -- 172.21.15.49:0/2963764772 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f799800b040 con 0x55b5e5fe9570 2024-03-18T00:53:38.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.634+0000 7f79ad608740 10 client.25847 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:38.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.634+0000 7f79ad608740 10 client.25847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:38.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.634+0000 7f79ad608740 20 client.25847 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:38.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.634+0000 7f79ad608740 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b5e5ffebd0 0x55b5e6001090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.634+0000 7f79ad608740 4 client.25847 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:38.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.634+0000 7f79ad608740 1 -- 172.21.15.49:0/2963764772 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5e5f5a610 con 0x55b5e5ffebd0 2024-03-18T00:53:38.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.640+0000 7f79a3fff640 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b5e5ffebd0 0x55b5e6001090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.659+0000 7f7987ff7640 10 client.25847 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:38.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.659+0000 7f79a3fff640 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b5e5ffebd0 0x55b5e6001090 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bcc7e012d999c4a server_cookie=812262926c638434 in_seq=0 out_seq=0 2024-03-18T00:53:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.665+0000 7f7987ff7640 1 -- 172.21.15.49:0/2963764772 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b5e5f5a610 con 0x55b5e5ffebd0 2024-03-18T00:53:38.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.666+0000 7f7987ff7640 10 client.25847 handle_command_reply: tid=0 2024-03-18T00:53:38.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.666+0000 7f79ad608740 1 client.25847 shutdown 2024-03-18T00:53:38.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.668+0000 7f79ad608740 1 -- 172.21.15.49:0/2963764772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 msgr2=0x55b5e5ff36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:38.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.668+0000 7f79ad608740 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff36a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7998000c00 tx=0x7f799800cf00 comp rx=0 tx=0).stop 2024-03-18T00:53:38.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.671+0000 7f79ad608740 1 -- 172.21.15.49:0/2963764772 shutdown_connections 2024-03-18T00:53:38.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.671+0000 7f79ad608740 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b5e5ffebd0 0x55b5e6001090 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.671+0000 7f79ad608740 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff36a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.671+0000 7f79ad608740 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ffa6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.671+0000 7f79ad608740 1 --2- 172.21.15.49:0/2963764772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fe8b80 0x55b5e5ff3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.671+0000 7f7987ff7640 0 client.25847 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:38.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.671+0000 7f79ad608740 1 -- 172.21.15.49:0/2963764772 >> 172.21.15.49:0/2963764772 conn(0x55b5e5fb0c50 msgr2=0x55b5e5fe36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:38.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.671+0000 7f79ad608740 1 -- 172.21.15.49:0/2963764772 shutdown_connections 2024-03-18T00:53:38.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.672+0000 7f79ad608740 1 -- 172.21.15.49:0/2963764772 wait complete. 2024-03-18T00:53:38.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.672+0000 7f79ad608740 20 client.25847 trim_cache size 0 max 16384 2024-03-18T00:53:38.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.751+0000 7f79ad608740 1 Processor -- start 2024-03-18T00:53:38.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.751+0000 7f79ad608740 1 -- start start 2024-03-18T00:53:38.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.752+0000 7f79ad608740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe8b80 0x55b5e6006960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.752+0000 7f79ad608740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ffa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.753+0000 7f79ad608740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.753+0000 7f79ad608740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f79900304c0 con 0x55b5e5fee6e0 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.753+0000 7f79ad608740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7990032000 con 0x55b5e5fe9570 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.753+0000 7f79ad608740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7990030b90 con 0x55b5e5fe8b80 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.753+0000 7f79a37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ffa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.753+0000 7f79a37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ffa6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48002/0 (socket says 172.21.15.49:48002) 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.753+0000 7f79a37fe640 1 -- 172.21.15.49:0/986137936 learned_addr learned my addr 172.21.15.49:0/986137936 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.754+0000 7f79a8fb6640 1 --2- 172.21.15.49:0/986137936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.754+0000 7f79a37fe640 1 -- 172.21.15.49:0/986137936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe8b80 msgr2=0x55b5e6006960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.754+0000 7f79a37fe640 1 --2- 172.21.15.49:0/986137936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe8b80 0x55b5e6006960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.754+0000 7f79a37fe640 1 -- 172.21.15.49:0/986137936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 msgr2=0x55b5e5ff2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:38.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.754+0000 7f79a37fe640 1 --2- 172.21.15.49:0/986137936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.754+0000 7f79a37fe640 1 -- 172.21.15.49:0/986137936 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5e5fe5240 con 0x55b5e5fe9570 2024-03-18T00:53:38.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.755+0000 7f79a37fe640 1 --2- 172.21.15.49:0/986137936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ffa6a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7998016e30 tx=0x7f79980d6d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d36814015e6203c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:38.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.757+0000 7f7989ffb640 1 -- 172.21.15.49:0/986137936 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980dd050 con 0x55b5e5fe9570 2024-03-18T00:53:38.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.757+0000 7f7989ffb640 1 -- 172.21.15.49:0/986137936 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79980db070 con 0x55b5e5fe9570 2024-03-18T00:53:38.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.757+0000 7f7989ffb640 1 -- 172.21.15.49:0/986137936 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980a0c90 con 0x55b5e5fe9570 2024-03-18T00:53:38.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.759+0000 7f79ad608740 1 -- 172.21.15.49:0/986137936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 msgr2=0x55b5e5ffa6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:38.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.759+0000 7f79ad608740 1 --2- 172.21.15.49:0/986137936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ffa6a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7998016e30 tx=0x7f79980d6d40 comp rx=0 tx=0).stop 2024-03-18T00:53:38.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.760+0000 7f79ad608740 1 -- 172.21.15.49:0/986137936 shutdown_connections 2024-03-18T00:53:38.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.760+0000 7f79ad608740 1 --2- 172.21.15.49:0/986137936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe9570 0x55b5e5ffa6a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.760+0000 7f79ad608740 1 --2- 172.21.15.49:0/986137936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.760+0000 7f79ad608740 1 --2- 172.21.15.49:0/986137936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe8b80 0x55b5e6006960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.760+0000 7f79ad608740 1 -- 172.21.15.49:0/986137936 >> 172.21.15.49:0/986137936 conn(0x55b5e5fb0c50 msgr2=0x55b5e5ff2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:38.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.760+0000 7f79ad608740 1 -- 172.21.15.49:0/986137936 shutdown_connections 2024-03-18T00:53:38.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.760+0000 7f79ad608740 1 -- 172.21.15.49:0/986137936 wait complete. 2024-03-18T00:53:38.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.761+0000 7f79ad608740 1 -- start start 2024-03-18T00:53:38.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.761+0000 7f79ad608740 1 Processor -- start 2024-03-18T00:53:38.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79ad608740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe8b80 0x55b5e5ff7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79ad608740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79ad608740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79ad608740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7990031930 con 0x55b5e5fee6e0 2024-03-18T00:53:38.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79ad608740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7990031260 con 0x55b5e5fe8b80 2024-03-18T00:53:38.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79ad608740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f799002e4a0 con 0x55b5e5fe9570 2024-03-18T00:53:38.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79ad608740 10 client.0 _ll_register_callbacks cb 0x7f79980d66c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:38.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79a8fb6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79a8fb6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36630/0 (socket says 172.21.15.49:36630) 2024-03-18T00:53:38.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.762+0000 7f79a8fb6640 1 -- 172.21.15.49:0/652774546 learned_addr learned my addr 172.21.15.49:0/652774546 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:38.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.763+0000 7f79a37fe640 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.764+0000 7f79a37fe640 1 -- 172.21.15.49:0/652774546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe8b80 msgr2=0x55b5e5ff7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:38.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.764+0000 7f79a37fe640 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe8b80 0x55b5e5ff7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.764+0000 7f79a37fe640 1 -- 172.21.15.49:0/652774546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 msgr2=0x55b5e5ff9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:38.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.764+0000 7f79a37fe640 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:38.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.764+0000 7f79a37fe640 1 -- 172.21.15.49:0/652774546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5e5ff98f0 con 0x55b5e5fe9570 2024-03-18T00:53:38.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.764+0000 7f79a37fe640 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff8340 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f79980a2850 tx=0x7f799800a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b96f59d79545f0b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:38.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.764+0000 7f7987ff7640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:53:38.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.765+0000 7f7987ff7640 1 -- 172.21.15.49:0/652774546 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79980dd040 con 0x55b5e5fe9570 2024-03-18T00:53:38.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.765+0000 7f79ad608740 1 -- 172.21.15.49:0/652774546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79980db070 con 0x55b5e5fe9570 2024-03-18T00:53:38.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.765+0000 7f7987ff7640 1 -- 172.21.15.49:0/652774546 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f79980c1270 con 0x55b5e5fe9570 2024-03-18T00:53:38.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.767+0000 7f7987ff7640 1 -- 172.21.15.49:0/652774546 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f79980a24b0 con 0x55b5e5fe9570 2024-03-18T00:53:38.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.767+0000 7f79ad608740 10 client.25921 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:38.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.767+0000 7f79ad608740 1 -- 172.21.15.49:0/652774546 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5e5ff98f0 con 0x55b5e5fe9570 2024-03-18T00:53:38.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.769+0000 7f7987ff7640 1 -- 172.21.15.49:0/652774546 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79980db070 con 0x55b5e5fe9570 2024-03-18T00:53:38.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.770+0000 7f7987ff7640 1 -- 172.21.15.49:0/652774546 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f79980aae60 con 0x55b5e5fe9570 2024-03-18T00:53:38.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.771+0000 7f79ad608740 10 client.25921 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:38.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.771+0000 7f79ad608740 10 client.25921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:38.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.771+0000 7f79ad608740 20 client.25921 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:38.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.771+0000 7f79ad608740 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b5e6065530 0x55b5e6067950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:38.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.771+0000 7f79ad608740 4 client.25921 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:53:38.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.771+0000 7f79ad608740 1 -- 172.21.15.49:0/652774546 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b5e5f54960 con 0x55b5e6065530 2024-03-18T00:53:38.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.772+0000 7f79a3fff640 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b5e6065530 0x55b5e6067950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:38.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.778+0000 7f7987ff7640 10 client.25921 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:38.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:38.778+0000 7f79a3fff640 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b5e6065530 0x55b5e6067950 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1e3b81bb30cb23b server_cookie=fad39a2cc9136b6e in_seq=0 out_seq=0 2024-03-18T00:53:39.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.441+0000 7f7987ff7640 1 -- 172.21.15.49:0/652774546 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b5e5f54960 con 0x55b5e6065530 2024-03-18T00:53:39.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.441+0000 7f7987ff7640 10 client.25921 handle_command_reply: tid=0 2024-03-18T00:53:39.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.441+0000 7f79ad608740 1 client.25921 shutdown 2024-03-18T00:53:39.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 -- 172.21.15.49:0/652774546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe9570 msgr2=0x55b5e5ff8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:39.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff8340 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f79980a2850 tx=0x7f799800a430 comp rx=0 tx=0).stop 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 -- 172.21.15.49:0/652774546 shutdown_connections 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b5e6065530 0x55b5e6067950 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5e5fee6e0 0x55b5e5ff9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5e5fe9570 0x55b5e5ff8340 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 --2- 172.21.15.49:0/652774546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5e5fe8b80 0x55b5e5ff7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 -- 172.21.15.49:0/652774546 >> 172.21.15.49:0/652774546 conn(0x55b5e5fb0c50 msgr2=0x55b5e5ff3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f7987ff7640 0 client.25921 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 -- 172.21.15.49:0/652774546 shutdown_connections 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 1 -- 172.21.15.49:0/652774546 wait complete. 2024-03-18T00:53:39.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.442+0000 7f79ad608740 20 client.25921 trim_cache size 0 max 16384 2024-03-18T00:53:39.445 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:53:39.445 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:53:39.445 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "a5257771-97b0-429d-ae49-72f0997ff82e", 2024-03-18T00:53:39.445 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:53:39.445 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:53:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.443+0000 7f79a97b7640 1 -- 172.21.15.49:0/2282788202 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f798c0a28f0 msgr2=0x7f798c0a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.443+0000 7f79a97b7640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f798c0a28f0 0x7f798c0a4db0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7994009590 tx=0x7f7994005b50 comp rx=0 tx=0).stop 2024-03-18T00:53:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 -- 172.21.15.49:0/2282788202 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79a4076430 msgr2=0x7f79a41b3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79a4076430 0x7f79a41b3c80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f799800ea90 tx=0x7f799800ef50 comp rx=0 tx=0).stop 2024-03-18T00:53:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 -- 172.21.15.49:0/2282788202 shutdown_connections 2024-03-18T00:53:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f798c0a28f0 0x7f798c0a4db0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f79a4076430 0x7f79a41b3c80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79a4078de0 0x7f79a41b62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 --2- 172.21.15.49:0/2282788202 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f79a4073550 0x7f79a41b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 -- 172.21.15.49:0/2282788202 >> 172.21.15.49:0/2282788202 conn(0x7f79a406e6c0 msgr2=0x7f79a4083d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:39.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 -- 172.21.15.49:0/2282788202 shutdown_connections 2024-03-18T00:53:39.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.444+0000 7f79a97b7640 1 -- 172.21.15.49:0/2282788202 wait complete. 2024-03-18T00:53:39.475 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:39 smithi049 ceph-mon[32130]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 862 MiB/s wr, 1.34k op/s 2024-03-18T00:53:39.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:53:39.477 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:53:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:39 smithi107 ceph-mon[36270]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 862 MiB/s wr, 1.34k op/s 2024-03-18T00:53:39.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:39 smithi106 ceph-mon[35899]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 862 MiB/s wr, 1.34k op/s 2024-03-18T00:53:39.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.890+0000 7f6274b10640 1 -- 172.21.15.49:0/1013035666 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62701018d0 msgr2=0x7f6270101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.890+0000 7f6274b10640 1 --2- 172.21.15.49:0/1013035666 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62701018d0 0x7f6270101d50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f6264009900 tx=0x7f626402bc30 comp rx=0 tx=0).stop 2024-03-18T00:53:39.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.892+0000 7f6274b10640 1 -- 172.21.15.49:0/1013035666 shutdown_connections 2024-03-18T00:53:39.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.892+0000 7f6274b10640 1 --2- 172.21.15.49:0/1013035666 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6270102290 0x7f6270108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.892+0000 7f6274b10640 1 --2- 172.21.15.49:0/1013035666 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62701018d0 0x7f6270101d50 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.892+0000 7f6274b10640 1 --2- 172.21.15.49:0/1013035666 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62701006e0 0x7f6270100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.892+0000 7f6274b10640 1 -- 172.21.15.49:0/1013035666 >> 172.21.15.49:0/1013035666 conn(0x7f62700fbc70 msgr2=0x7f62700fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:39.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.892+0000 7f6274b10640 1 -- 172.21.15.49:0/1013035666 shutdown_connections 2024-03-18T00:53:39.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.892+0000 7f6274b10640 1 -- 172.21.15.49:0/1013035666 wait complete. 2024-03-18T00:53:39.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.894+0000 7f6274b10640 1 Processor -- start 2024-03-18T00:53:39.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.894+0000 7f6274b10640 1 -- start start 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.894+0000 7f6274b10640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62701006e0 0x7f62701112e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.894+0000 7f6274b10640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62701018d0 0x7f6270111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.894+0000 7f6274b10640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6270102290 0x7f627010f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.894+0000 7f6274b10640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f627010a1c0 con 0x7f6270102290 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.894+0000 7f6274b10640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f627010a040 con 0x7f62701018d0 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.894+0000 7f6274b10640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f627010a340 con 0x7f62701006e0 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.895+0000 7f626ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6270102290 0x7f627010f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.895+0000 7f626ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6270102290 0x7f627010f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36660/0 (socket says 172.21.15.49:36660) 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.895+0000 7f626ed9d640 1 -- 172.21.15.49:0/530292847 learned_addr learned my addr 172.21.15.49:0/530292847 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.895+0000 7f626ed9d640 1 -- 172.21.15.49:0/530292847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62701006e0 msgr2=0x7f62701112e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.895+0000 7f626ed9d640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62701006e0 0x7f62701112e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.895+0000 7f626ed9d640 1 -- 172.21.15.49:0/530292847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62701018d0 msgr2=0x7f6270111820 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:53:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.895+0000 7f626ed9d640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62701018d0 0x7f6270111820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.895+0000 7f626ed9d640 1 -- 172.21.15.49:0/530292847 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62701100f0 con 0x7f6270102290 2024-03-18T00:53:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.895+0000 7f626ed9d640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6270102290 0x7f627010f920 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f626800eaa0 tx=0x7f626800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.896+0000 7f625f7fe640 1 -- 172.21.15.49:0/530292847 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626800cdb0 con 0x7f6270102290 2024-03-18T00:53:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.896+0000 7f625f7fe640 1 -- 172.21.15.49:0/530292847 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6268005020 con 0x7f6270102290 2024-03-18T00:53:39.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.896+0000 7f625f7fe640 1 -- 172.21.15.49:0/530292847 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62680116b0 con 0x7f6270102290 2024-03-18T00:53:39.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.896+0000 7f6274b10640 1 -- 172.21.15.49:0/530292847 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62701103e0 con 0x7f6270102290 2024-03-18T00:53:39.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.896+0000 7f6274b10640 1 -- 172.21.15.49:0/530292847 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62701aee20 con 0x7f6270102290 2024-03-18T00:53:39.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.899+0000 7f625f7fe640 1 -- 172.21.15.49:0/530292847 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6268005560 con 0x7f6270102290 2024-03-18T00:53:39.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.900+0000 7f62777dc740 1 Processor -- start 2024-03-18T00:53:39.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.900+0000 7f62777dc740 1 -- start start 2024-03-18T00:53:39.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.901+0000 7f62777dc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473170 0x560eea473590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.901+0000 7f62777dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473b60 0x560eea478660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.901+0000 7f62777dc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea47b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.901+0000 7f625f7fe640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62580a2880 0x7f62580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.901+0000 7f62777dc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x560eea3d8500 con 0x560eea478cd0 2024-03-18T00:53:39.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.901+0000 7f62777dc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560eea380be0 con 0x560eea473170 2024-03-18T00:53:39.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.901+0000 7f626ed9d640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62580a2880 0x7f62580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.901+0000 7f62777dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560eea3d45a0 con 0x560eea473b60 2024-03-18T00:53:39.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.901+0000 7f625f7fe640 1 -- 172.21.15.49:0/530292847 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f62680cecd0 con 0x7f6270102290 2024-03-18T00:53:39.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.902+0000 7f626dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea47b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.902+0000 7f626dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea47b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36668/0 (socket says 172.21.15.49:36668) 2024-03-18T00:53:39.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.902+0000 7f626dd9b640 1 -- 172.21.15.49:0/3563670617 learned_addr learned my addr 172.21.15.49:0/3563670617 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:39.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.902+0000 7f626ed9d640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62580a2880 0x7f62580a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6268000f80 tx=0x7f6268090b30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:39.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.902+0000 7f626dd9b640 1 --2- 172.21.15.49:0/3563670617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473170 0x560eea473590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.902+0000 7f626e59c640 1 --2- 172.21.15.49:0/3563670617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473b60 0x560eea478660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.903+0000 7f626dd9b640 1 -- 172.21.15.49:0/3563670617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473b60 msgr2=0x560eea478660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.903+0000 7f626dd9b640 1 --2- 172.21.15.49:0/3563670617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473b60 0x560eea478660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.903+0000 7f626dd9b640 1 -- 172.21.15.49:0/3563670617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473170 msgr2=0x560eea473590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.903+0000 7f626dd9b640 1 --2- 172.21.15.49:0/3563670617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473170 0x560eea473590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.903+0000 7f626dd9b640 1 -- 172.21.15.49:0/3563670617 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eea3944b0 con 0x560eea478cd0 2024-03-18T00:53:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.903+0000 7f626dd9b640 1 --2- 172.21.15.49:0/3563670617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea47b160 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6264009230 tx=0x7f626402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb01c673d7503cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.903+0000 7f625cff9640 1 -- 172.21.15.49:0/3563670617 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626402ca20 con 0x560eea478cd0 2024-03-18T00:53:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.904+0000 7f625cff9640 1 -- 172.21.15.49:0/3563670617 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f626402cbc0 con 0x560eea478cd0 2024-03-18T00:53:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.904+0000 7f625cff9640 1 -- 172.21.15.49:0/3563670617 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6264004920 con 0x560eea478cd0 2024-03-18T00:53:39.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.904+0000 7f62777dc740 1 -- 172.21.15.49:0/3563670617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 msgr2=0x560eea47b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:39.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.904+0000 7f62777dc740 1 --2- 172.21.15.49:0/3563670617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea47b160 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6264009230 tx=0x7f626402e040 comp rx=0 tx=0).stop 2024-03-18T00:53:39.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.905+0000 7f62777dc740 1 -- 172.21.15.49:0/3563670617 shutdown_connections 2024-03-18T00:53:39.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.905+0000 7f62777dc740 1 --2- 172.21.15.49:0/3563670617 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea47b160 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.905+0000 7f62777dc740 1 --2- 172.21.15.49:0/3563670617 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473b60 0x560eea478660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.905+0000 7f62777dc740 1 --2- 172.21.15.49:0/3563670617 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473170 0x560eea473590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.905+0000 7f62777dc740 1 -- 172.21.15.49:0/3563670617 >> 172.21.15.49:0/3563670617 conn(0x560eea46e6e0 msgr2=0x560eea46eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:39.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.906+0000 7f62777dc740 1 -- 172.21.15.49:0/3563670617 shutdown_connections 2024-03-18T00:53:39.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.906+0000 7f62777dc740 1 -- 172.21.15.49:0/3563670617 wait complete. 2024-03-18T00:53:39.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.906+0000 7f62777dc740 1 -- start start 2024-03-18T00:53:39.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.906+0000 7f62777dc740 1 Processor -- start 2024-03-18T00:53:39.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.907+0000 7f62777dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea47d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.907+0000 7f62777dc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea47dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.907+0000 7f62777dc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea484c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f62777dc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x560eea47bc20 con 0x560eea478cd0 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f62777dc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560eea394160 con 0x560eea473b60 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f62777dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560eea47bda0 con 0x560eea473170 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea484c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea484c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36674/0 (socket says 172.21.15.49:36674) 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626dd9b640 1 -- 172.21.15.49:0/3576790139 learned_addr learned my addr 172.21.15.49:0/3576790139 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f62777dc740 10 client.0 _ll_register_callbacks cb 0x560eea378150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626e59c640 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea47dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626dd9b640 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea47d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626dd9b640 1 -- 172.21.15.49:0/3576790139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 msgr2=0x560eea47d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626dd9b640 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea47d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626dd9b640 1 -- 172.21.15.49:0/3576790139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 msgr2=0x560eea47dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626dd9b640 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea47dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.908+0000 7f626dd9b640 1 -- 172.21.15.49:0/3576790139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eea1b67d0 con 0x560eea478cd0 2024-03-18T00:53:39.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.909+0000 7f626e59c640 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea47dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:53:39.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.909+0000 7f626dd9b640 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea484c00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6264005fd0 tx=0x7f626402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801d30ef787cc10a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:39.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.909+0000 7f6255ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:39.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.909+0000 7f6255ffb640 1 -- 172.21.15.49:0/3576790139 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6264005290 con 0x560eea478cd0 2024-03-18T00:53:39.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.910+0000 7f62777dc740 1 -- 172.21.15.49:0/3576790139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f626402cbc0 con 0x560eea478cd0 2024-03-18T00:53:39.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.910+0000 7f6255ffb640 1 -- 172.21.15.49:0/3576790139 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f62640046c0 con 0x560eea478cd0 2024-03-18T00:53:39.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.910+0000 7f6255ffb640 1 -- 172.21.15.49:0/3576790139 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6264005430 con 0x560eea478cd0 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.910+0000 7f62777dc740 10 client.16323 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.910+0000 7f62777dc740 1 -- 172.21.15.49:0/3576790139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560eea0ec3f0 con 0x560eea478cd0 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.911+0000 7f6255ffb640 1 -- 172.21.15.49:0/3576790139 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f626402cbc0 con 0x560eea478cd0 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.911+0000 7f6255ffb640 1 -- 172.21.15.49:0/3576790139 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f626404d040 con 0x560eea478cd0 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.911+0000 7f62777dc740 10 client.16323 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.911+0000 7f62777dc740 10 client.16323 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.911+0000 7f62777dc740 20 client.16323 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.911+0000 7f62777dc740 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560eea488d20 0x560eea48b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.911+0000 7f62777dc740 4 client.16323 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.911+0000 7f62777dc740 1 -- 172.21.15.49:0/3576790139 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560eea3e5a40 con 0x560eea488d20 2024-03-18T00:53:39.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.912+0000 7f626e59c640 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560eea488d20 0x560eea48b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.913+0000 7f626e59c640 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560eea488d20 0x560eea48b110 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b96b80409bb7212d server_cookie=3097c3b0b50297a9 in_seq=0 out_seq=0 2024-03-18T00:53:39.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.913+0000 7f6255ffb640 10 client.16323 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:39.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f6255ffb640 1 -- 172.21.15.49:0/3576790139 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560eea3e5a40 con 0x560eea488d20 2024-03-18T00:53:39.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f6255ffb640 10 client.16323 handle_command_reply: tid=0 2024-03-18T00:53:39.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 client.16323 shutdown 2024-03-18T00:53:39.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 -- 172.21.15.49:0/3576790139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 msgr2=0x560eea484c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:39.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea484c00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6264005fd0 tx=0x7f626402bc30 comp rx=0 tx=0).stop 2024-03-18T00:53:39.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 -- 172.21.15.49:0/3576790139 shutdown_connections 2024-03-18T00:53:39.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560eea488d20 0x560eea48b110 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea47dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea484c00 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 --2- 172.21.15.49:0/3576790139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea47d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f6255ffb640 0 client.16323 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:39.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 -- 172.21.15.49:0/3576790139 >> 172.21.15.49:0/3576790139 conn(0x560eea46e6e0 msgr2=0x560eea46d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:39.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.923+0000 7f62777dc740 1 -- 172.21.15.49:0/3576790139 shutdown_connections 2024-03-18T00:53:39.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.924+0000 7f62777dc740 1 -- 172.21.15.49:0/3576790139 wait complete. 2024-03-18T00:53:39.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.924+0000 7f62777dc740 20 client.16323 trim_cache size 0 max 16384 2024-03-18T00:53:39.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.964+0000 7f62777dc740 1 Processor -- start 2024-03-18T00:53:39.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.964+0000 7f62777dc740 1 -- start start 2024-03-18T00:53:39.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.964+0000 7f62777dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea46eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.964+0000 7f62777dc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea484bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.964+0000 7f62777dc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea46d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.964+0000 7f626dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea46d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea46d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36684/0 (socket says 172.21.15.49:36684) 2024-03-18T00:53:39.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626dd9b640 1 -- 172.21.15.49:0/1202197933 learned_addr learned my addr 172.21.15.49:0/1202197933 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:39.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f62777dc740 1 -- 172.21.15.49:0/1202197933 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f62701b7b50 con 0x560eea478cd0 2024-03-18T00:53:39.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f62777dc740 1 -- 172.21.15.49:0/1202197933 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f62701b9690 con 0x560eea473b60 2024-03-18T00:53:39.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f62777dc740 1 -- 172.21.15.49:0/1202197933 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62701b8220 con 0x560eea473170 2024-03-18T00:53:39.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626dd9b640 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea46eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626e59c640 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea484bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626dd9b640 1 -- 172.21.15.49:0/1202197933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 msgr2=0x560eea46eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:39.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626dd9b640 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea46eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626dd9b640 1 -- 172.21.15.49:0/1202197933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 msgr2=0x560eea484bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:39.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626dd9b640 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea484bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626dd9b640 1 -- 172.21.15.49:0/1202197933 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560eea475850 con 0x560eea478cd0 2024-03-18T00:53:39.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626e59c640 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea484bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:53:39.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.965+0000 7f626dd9b640 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea46d070 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f626402c7b0 tx=0x7f6264003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca24d02145286c60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:39.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.966+0000 7f6257fff640 1 -- 172.21.15.49:0/1202197933 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6264005430 con 0x560eea478cd0 2024-03-18T00:53:39.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.966+0000 7f6257fff640 1 -- 172.21.15.49:0/1202197933 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6264055070 con 0x560eea478cd0 2024-03-18T00:53:39.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.966+0000 7f6257fff640 1 -- 172.21.15.49:0/1202197933 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62640508a0 con 0x560eea478cd0 2024-03-18T00:53:39.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.969+0000 7f62777dc740 1 -- 172.21.15.49:0/1202197933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 msgr2=0x560eea46d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:39.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.969+0000 7f62777dc740 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea46d070 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f626402c7b0 tx=0x7f6264003fb0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.979+0000 7f62777dc740 1 -- 172.21.15.49:0/1202197933 shutdown_connections 2024-03-18T00:53:39.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.979+0000 7f62777dc740 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea473b60 0x560eea484bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.979+0000 7f62777dc740 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea478cd0 0x560eea46d070 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.979+0000 7f62777dc740 1 --2- 172.21.15.49:0/1202197933 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea46eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.979+0000 7f62777dc740 1 -- 172.21.15.49:0/1202197933 >> 172.21.15.49:0/1202197933 conn(0x560eea46e6e0 msgr2=0x560eea47c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:39.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.984+0000 7f62777dc740 1 -- 172.21.15.49:0/1202197933 shutdown_connections 2024-03-18T00:53:39.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.984+0000 7f62777dc740 1 -- 172.21.15.49:0/1202197933 wait complete. 2024-03-18T00:53:39.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.985+0000 7f62777dc740 1 -- start start 2024-03-18T00:53:39.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.985+0000 7f62777dc740 1 Processor -- start 2024-03-18T00:53:39.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.986+0000 7f62777dc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea481cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.986+0000 7f62777dc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea473b60 0x560eea4821f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.986+0000 7f62777dc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea478cd0 0x560eea483190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:39.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.986+0000 7f62777dc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f62701b8fc0 con 0x560eea473b60 2024-03-18T00:53:39.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.986+0000 7f62777dc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f62701b88f0 con 0x560eea478cd0 2024-03-18T00:53:39.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.986+0000 7f62777dc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62701b5b30 con 0x560eea473170 2024-03-18T00:53:39.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.987+0000 7f62777dc740 10 client.0 _ll_register_callbacks cb 0x7f6264009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:39.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.988+0000 7f626e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea473b60 0x560eea4821f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.988+0000 7f626e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea473b60 0x560eea4821f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36692/0 (socket says 172.21.15.49:36692) 2024-03-18T00:53:39.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.988+0000 7f626e59c640 1 -- 172.21.15.49:0/2985918563 learned_addr learned my addr 172.21.15.49:0/2985918563 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:39.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.988+0000 7f626dd9b640 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea481cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.993+0000 7f626dd9b640 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea478cd0 0x560eea483190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:39.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.993+0000 7f626e59c640 1 -- 172.21.15.49:0/2985918563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 msgr2=0x560eea481cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:39.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.993+0000 7f626e59c640 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea481cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.993+0000 7f626e59c640 1 -- 172.21.15.49:0/2985918563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea478cd0 msgr2=0x560eea483190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:39.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.993+0000 7f626e59c640 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea478cd0 0x560eea483190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:39.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.994+0000 7f626e59c640 1 -- 172.21.15.49:0/2985918563 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560eea0ac0f0 con 0x560eea473b60 2024-03-18T00:53:39.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.995+0000 7f626e59c640 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea473b60 0x560eea4821f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x560eea473f60 tx=0x7f626000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6e69ed821f86fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:39.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.995+0000 7f6255ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:39.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.997+0000 7f6255ffb640 1 -- 172.21.15.49:0/2985918563 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260017b20 con 0x560eea473b60 2024-03-18T00:53:39.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.997+0000 7f62777dc740 1 -- 172.21.15.49:0/2985918563 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6264055070 con 0x560eea473b60 2024-03-18T00:53:39.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.998+0000 7f6255ffb640 1 -- 172.21.15.49:0/2985918563 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6260017cc0 con 0x560eea473b60 2024-03-18T00:53:39.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.998+0000 7f6255ffb640 1 -- 172.21.15.49:0/2985918563 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f62600190b0 con 0x560eea473b60 2024-03-18T00:53:39.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.998+0000 7f62777dc740 10 client.16329 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:39.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:39.998+0000 7f62777dc740 1 -- 172.21.15.49:0/2985918563 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560eea39f9d0 con 0x560eea473b60 2024-03-18T00:53:40.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.010+0000 7f6255ffb640 1 -- 172.21.15.49:0/2985918563 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6264055070 con 0x560eea473b60 2024-03-18T00:53:40.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.010+0000 7f6255ffb640 1 -- 172.21.15.49:0/2985918563 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f62600229c0 con 0x560eea473b60 2024-03-18T00:53:40.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.010+0000 7f62777dc740 10 client.16329 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:40.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.010+0000 7f62777dc740 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:40.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.010+0000 7f62777dc740 20 client.16329 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:40.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.010+0000 7f62777dc740 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560eea4ef010 0x560eea4f1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:40.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.010+0000 7f62777dc740 4 client.16329 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:53:40.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.010+0000 7f62777dc740 1 -- 172.21.15.49:0/2985918563 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560eea3dfd90 con 0x560eea4ef010 2024-03-18T00:53:40.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.014+0000 7f626dd9b640 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560eea4ef010 0x560eea4f1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:40.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.025+0000 7f626dd9b640 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560eea4ef010 0x560eea4f1430 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b29da7184e2ffe3 server_cookie=40ba30d0fcce16c2 in_seq=0 out_seq=0 2024-03-18T00:53:40.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.026+0000 7f6255ffb640 10 client.16329 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:40.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.035+0000 7f6255ffb640 1 -- 172.21.15.49:0/2985918563 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x560eea3dfd90 con 0x560eea4ef010 2024-03-18T00:53:40.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.035+0000 7f6255ffb640 10 client.16329 handle_command_reply: tid=0 2024-03-18T00:53:40.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.036+0000 7f62777dc740 1 client.16329 shutdown 2024-03-18T00:53:40.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.042+0000 7f62777dc740 1 -- 172.21.15.49:0/2985918563 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea473b60 msgr2=0x560eea4821f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:40.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.042+0000 7f62777dc740 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea473b60 0x560eea4821f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x560eea473f60 tx=0x7f626000bcb0 comp rx=0 tx=0).stop 2024-03-18T00:53:40.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.043+0000 7f62777dc740 1 -- 172.21.15.49:0/2985918563 shutdown_connections 2024-03-18T00:53:40.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.043+0000 7f62777dc740 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560eea4ef010 0x560eea4f1430 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:40.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.043+0000 7f62777dc740 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560eea478cd0 0x560eea483190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:40.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.043+0000 7f62777dc740 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560eea473b60 0x560eea4821f0 secure :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0x560eea473f60 tx=0x7f626000bcb0 comp rx=0 tx=0).stop 2024-03-18T00:53:40.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.043+0000 7f62777dc740 1 --2- 172.21.15.49:0/2985918563 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560eea473170 0x560eea481cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:40.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.043+0000 7f62777dc740 1 -- 172.21.15.49:0/2985918563 >> 172.21.15.49:0/2985918563 conn(0x560eea46e6e0 msgr2=0x560eea471940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:40.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.044+0000 7f6255ffb640 0 client.16329 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:40.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.050+0000 7f62777dc740 1 -- 172.21.15.49:0/2985918563 shutdown_connections 2024-03-18T00:53:40.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.055+0000 7f62777dc740 1 -- 172.21.15.49:0/2985918563 wait complete. 2024-03-18T00:53:40.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.056+0000 7f62777dc740 20 client.16329 trim_cache size 0 max 16384 2024-03-18T00:53:40.057 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:53:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-18T00:53:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:53:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "a5257771-97b0-429d-ae49-72f0997ff82e": { 2024-03-18T00:53:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:53:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "a5257771-97b0-429d-ae49-72f0997ff82e", 2024-03-18T00:53:40.057 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:53:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:53:40.058 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:53:40.058 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:53:40.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.063+0000 7f6274b10640 1 -- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62580a2880 msgr2=0x7f62580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:40.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.063+0000 7f6274b10640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62580a2880 0x7f62580a4d40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6268000f80 tx=0x7f6268090b30 comp rx=0 tx=0).stop 2024-03-18T00:53:40.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.063+0000 7f6274b10640 1 -- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6270102290 msgr2=0x7f627010f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:40.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.063+0000 7f6274b10640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6270102290 0x7f627010f920 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f626800eaa0 tx=0x7f626800ef60 comp rx=0 tx=0).stop 2024-03-18T00:53:40.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.068+0000 7f6274b10640 1 -- 172.21.15.49:0/530292847 shutdown_connections 2024-03-18T00:53:40.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.068+0000 7f6274b10640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62580a2880 0x7f62580a4d40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:40.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.068+0000 7f6274b10640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62701018d0 0x7f6270111820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:40.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.068+0000 7f6274b10640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6270102290 0x7f627010f920 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:40.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.068+0000 7f6274b10640 1 --2- 172.21.15.49:0/530292847 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62701006e0 0x7f62701112e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:40.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.068+0000 7f6274b10640 1 -- 172.21.15.49:0/530292847 >> 172.21.15.49:0/530292847 conn(0x7f62700fbc70 msgr2=0x7f6270104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:40.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.075+0000 7f6274b10640 1 -- 172.21.15.49:0/530292847 shutdown_connections 2024-03-18T00:53:40.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:53:40.080+0000 7f6274b10640 1 -- 172.21.15.49:0/530292847 wait complete. 2024-03-18T00:53:40.145 INFO:tasks.cephfs.filesystem:scrub status for tag:a5257771-97b0-429d-ae49-72f0997ff82e - {'path': '/', 'tag': 'a5257771-97b0-429d-ae49-72f0997ff82e', 'options': 'recursive,force'} 2024-03-18T00:53:40.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:40 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:53:40.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:40 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:40.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:40 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:53:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:40 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:53:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:40 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:40 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:53:40.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:40 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:53:40.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:40 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:40.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:40 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:53:41.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:41 smithi049 ceph-mon[32130]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 959 MiB/s wr, 1.54k op/s 2024-03-18T00:53:41.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:41 smithi107 ceph-mon[36270]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 959 MiB/s wr, 1.54k op/s 2024-03-18T00:53:41.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:41 smithi106 ceph-mon[35899]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 959 MiB/s wr, 1.54k op/s 2024-03-18T00:53:42.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:42 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:53:42.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:42 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:42.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:42 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:53:42.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:42 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:42.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:42 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:53:42.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:42 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:43.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:43 smithi049 ceph-mon[32130]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 823 MiB/s wr, 1.37k op/s 2024-03-18T00:53:43.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:43 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:43 smithi107 ceph-mon[36270]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 823 MiB/s wr, 1.37k op/s 2024-03-18T00:53:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:43 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:43.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:43 smithi106 ceph-mon[35899]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 823 MiB/s wr, 1.37k op/s 2024-03-18T00:53:43.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:43 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:53:45.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:45 smithi049 ceph-mon[32130]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1009 MiB/s wr, 1.70k op/s 2024-03-18T00:53:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:45 smithi107 ceph-mon[36270]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1009 MiB/s wr, 1.70k op/s 2024-03-18T00:53:45.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:45 smithi106 ceph-mon[35899]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1009 MiB/s wr, 1.70k op/s 2024-03-18T00:53:46.145 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.146+0000 7f34feffd640 1 -- 172.21.15.49:0/1595071028 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f002cc80 con 0x7f35001087e0 2024-03-18T00:53:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.146+0000 7f350673b640 1 -- 172.21.15.49:0/1595071028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35001087e0 msgr2=0x7f350010ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.147+0000 7f350673b640 1 --2- 172.21.15.49:0/1595071028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35001087e0 0x7f350010ac10 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f34f0009900 tx=0x7f34f002bc30 comp rx=0 tx=0).stop 2024-03-18T00:53:46.146 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.147+0000 7f350673b640 1 -- 172.21.15.49:0/1595071028 shutdown_connections 2024-03-18T00:53:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.147+0000 7f350673b640 1 --2- 172.21.15.49:0/1595071028 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f350010db00 0x7f350010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.147+0000 7f350673b640 1 --2- 172.21.15.49:0/1595071028 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f350010b150 0x7f350010d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.147+0000 7f350673b640 1 --2- 172.21.15.49:0/1595071028 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35001087e0 0x7f350010ac10 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.147+0000 7f350673b640 1 -- 172.21.15.49:0/1595071028 >> 172.21.15.49:0/1595071028 conn(0x7f350006e820 msgr2=0x7f3500070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.147+0000 7f350673b640 1 -- 172.21.15.49:0/1595071028 shutdown_connections 2024-03-18T00:53:46.147 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.147+0000 7f350673b640 1 -- 172.21.15.49:0/1595071028 wait complete. 2024-03-18T00:53:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.148+0000 7f350673b640 1 Processor -- start 2024-03-18T00:53:46.148 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.149+0000 7f350673b640 1 -- start start 2024-03-18T00:53:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.150+0000 7f350673b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35001087e0 0x7f35001a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.150+0000 7f350673b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f350010b150 0x7f35001a6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.150+0000 7f350673b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f350010db00 0x7f35001ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.150+0000 7f350673b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f350011c330 con 0x7f350010db00 2024-03-18T00:53:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.150+0000 7f350673b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f350011c1b0 con 0x7f350010b150 2024-03-18T00:53:46.150 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.150+0000 7f350673b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f350011c4b0 con 0x7f35001087e0 2024-03-18T00:53:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.151+0000 7f3504cd8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f350010db00 0x7f35001ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.151+0000 7f3504cd8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f350010db00 0x7f35001ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36718/0 (socket says 172.21.15.49:36718) 2024-03-18T00:53:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.151+0000 7f3504cd8640 1 -- 172.21.15.49:0/982319927 learned_addr learned my addr 172.21.15.49:0/982319927 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.151+0000 7f34fffff640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35001087e0 0x7f35001a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.151+0000 7f3504cd8640 1 -- 172.21.15.49:0/982319927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35001087e0 msgr2=0x7f35001a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.151+0000 7f3504cd8640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35001087e0 0x7f35001a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.151+0000 7f3504cd8640 1 -- 172.21.15.49:0/982319927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f350010b150 msgr2=0x7f35001a6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.152+0000 7f3504cd8640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f350010b150 0x7f35001a6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.152 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.152+0000 7f3504cd8640 1 -- 172.21.15.49:0/982319927 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35001ae4c0 con 0x7f350010db00 2024-03-18T00:53:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.152+0000 7f3504cd8640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f350010db00 0x7f35001ade50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f34f800b970 tx=0x7f34f800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.153+0000 7f34fd7fa640 1 -- 172.21.15.49:0/982319927 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8019070 con 0x7f350010db00 2024-03-18T00:53:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.153+0000 7f34fd7fa640 1 -- 172.21.15.49:0/982319927 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f800cb40 con 0x7f350010db00 2024-03-18T00:53:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.153+0000 7f350673b640 1 -- 172.21.15.49:0/982319927 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35001ae7b0 con 0x7f350010db00 2024-03-18T00:53:46.153 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.153+0000 7f34fd7fa640 1 -- 172.21.15.49:0/982319927 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f8007650 con 0x7f350010db00 2024-03-18T00:53:46.154 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.154+0000 7f350673b640 1 -- 172.21.15.49:0/982319927 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35001aed00 con 0x7f350010db00 2024-03-18T00:53:46.158 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.159+0000 7f3509401740 1 Processor -- start 2024-03-18T00:53:46.159 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.160+0000 7f34fd7fa640 1 -- 172.21.15.49:0/982319927 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f34f801c070 con 0x7f350010db00 2024-03-18T00:53:46.160 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.160+0000 7f3509401740 1 -- start start 2024-03-18T00:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.160+0000 7f3509401740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedbd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.160+0000 7f3509401740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 0x55ddcedc2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.160+0000 7f3509401740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedc2ea0 0x55ddcedc5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.161+0000 7f3509401740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ddced22a40 con 0x55ddcedc2ea0 2024-03-18T00:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.161+0000 7f3509401740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ddceccb920 con 0x55ddcedbd340 2024-03-18T00:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.161+0000 7f3509401740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ddced1eae0 con 0x55ddcedbdd30 2024-03-18T00:53:46.161 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.161+0000 7f34ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedc2ea0 0x55ddcedc5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.162+0000 7f34ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedc2ea0 0x55ddcedc5330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36732/0 (socket says 172.21.15.49:36732) 2024-03-18T00:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.162+0000 7f34ff7fe640 1 -- 172.21.15.49:0/1426320433 learned_addr learned my addr 172.21.15.49:0/1426320433 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.162+0000 7f34fd7fa640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34e80a2920 0x7f34e80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.162+0000 7f34fffff640 1 --2- 172.21.15.49:0/1426320433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 0x55ddcedc2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.162 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.162+0000 7f3504cd8640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34e80a2920 0x7f34e80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.162+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/1426320433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedbd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.162+0000 7f34ff7fe640 1 -- 172.21.15.49:0/1426320433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 msgr2=0x55ddcedc2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.162+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/1426320433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 0x55ddcedc2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.163+0000 7f34ff7fe640 1 -- 172.21.15.49:0/1426320433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 msgr2=0x55ddcedbd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.163+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/1426320433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedbd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.163+0000 7f34ff7fe640 1 -- 172.21.15.49:0/1426320433 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddcece3460 con 0x55ddcedc2ea0 2024-03-18T00:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.163+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/1426320433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedc2ea0 0x55ddcedc5330 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f34f4005d00 tx=0x7f34f40094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89e4d41bf4edbe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:46.163 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.163+0000 7f34fd7fa640 1 -- 172.21.15.49:0/982319927 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f34f80cdfb0 con 0x7f350010db00 2024-03-18T00:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.164+0000 7f3504cd8640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34e80a2920 0x7f34e80a4de0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f34f80c6700 tx=0x7f34f80c6670 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.164+0000 7f34e67fc640 1 -- 172.21.15.49:0/1426320433 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4016070 con 0x55ddcedc2ea0 2024-03-18T00:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.164+0000 7f34e67fc640 1 -- 172.21.15.49:0/1426320433 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f4005500 con 0x55ddcedc2ea0 2024-03-18T00:53:46.164 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.164+0000 7f34e67fc640 1 -- 172.21.15.49:0/1426320433 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4011410 con 0x55ddcedc2ea0 2024-03-18T00:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.166+0000 7f3509401740 1 -- 172.21.15.49:0/1426320433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedc2ea0 msgr2=0x55ddcedc5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.166+0000 7f3509401740 1 --2- 172.21.15.49:0/1426320433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedc2ea0 0x55ddcedc5330 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f34f4005d00 tx=0x7f34f40094e0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.166+0000 7f3509401740 1 -- 172.21.15.49:0/1426320433 shutdown_connections 2024-03-18T00:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.166+0000 7f3509401740 1 --2- 172.21.15.49:0/1426320433 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedc2ea0 0x55ddcedc5330 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.166 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.166+0000 7f3509401740 1 --2- 172.21.15.49:0/1426320433 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 0x55ddcedc2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.166+0000 7f3509401740 1 --2- 172.21.15.49:0/1426320433 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedbd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.166+0000 7f3509401740 1 -- 172.21.15.49:0/1426320433 >> 172.21.15.49:0/1426320433 conn(0x55ddcedb88b0 msgr2=0x55ddcedb8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.167+0000 7f3509401740 1 -- 172.21.15.49:0/1426320433 shutdown_connections 2024-03-18T00:53:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.167+0000 7f3509401740 1 -- 172.21.15.49:0/1426320433 wait complete. 2024-03-18T00:53:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.167+0000 7f3509401740 1 -- start start 2024-03-18T00:53:46.167 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.168+0000 7f3509401740 1 Processor -- start 2024-03-18T00:53:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.168+0000 7f3509401740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedc7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.168+0000 7f3509401740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedbdd30 0x55ddcedc7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.168+0000 7f3509401740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedccd70 0x55ddcedcf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.168+0000 7f3509401740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ddcedc5e50 con 0x55ddcedbdd30 2024-03-18T00:53:46.168 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.168+0000 7f3509401740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ddcecded70 con 0x55ddcedbd340 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.168+0000 7f3509401740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ddcedc5fd0 con 0x55ddcedccd70 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.168+0000 7f34fffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedbdd30 0x55ddcedc7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.168+0000 7f3509401740 10 client.0 _ll_register_callbacks cb 0x55ddcea36a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.169+0000 7f34fffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedbdd30 0x55ddcedc7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36734/0 (socket says 172.21.15.49:36734) 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.169+0000 7f34fffff640 1 -- 172.21.15.49:0/3906516846 learned_addr learned my addr 172.21.15.49:0/3906516846 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.169+0000 7f34fffff640 1 -- 172.21.15.49:0/3906516846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedccd70 msgr2=0x55ddcedcf160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.169+0000 7f34fffff640 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedccd70 0x55ddcedcf160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.169+0000 7f34fffff640 1 -- 172.21.15.49:0/3906516846 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 msgr2=0x55ddcedc7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.169+0000 7f34fffff640 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedc7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.169+0000 7f34fffff640 1 -- 172.21.15.49:0/3906516846 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddcea37210 con 0x55ddcedbdd30 2024-03-18T00:53:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.170+0000 7f34fffff640 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedbdd30 0x55ddcedc7d40 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f34f0006010 tx=0x7f34f0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c74f825f4cb7b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.170+0000 7f34df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.171+0000 7f34df7fe640 1 -- 172.21.15.49:0/3906516846 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0005000 con 0x55ddcedbdd30 2024-03-18T00:53:46.170 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.171+0000 7f3509401740 1 -- 172.21.15.49:0/3906516846 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34f4005500 con 0x55ddcedbdd30 2024-03-18T00:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.171+0000 7f34df7fe640 1 -- 172.21.15.49:0/3906516846 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f34f00051a0 con 0x55ddcedbdd30 2024-03-18T00:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.171+0000 7f34df7fe640 1 -- 172.21.15.49:0/3906516846 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f34f00053a0 con 0x55ddcedbdd30 2024-03-18T00:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.172+0000 7f3509401740 10 client.16341 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:46.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.172+0000 7f3509401740 1 -- 172.21.15.49:0/3906516846 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ddceb03620 con 0x55ddcedbdd30 2024-03-18T00:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.172+0000 7f34df7fe640 1 -- 172.21.15.49:0/3906516846 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f4005500 con 0x55ddcedbdd30 2024-03-18T00:53:46.172 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.173+0000 7f34df7fe640 1 -- 172.21.15.49:0/3906516846 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f34f0003940 con 0x55ddcedbdd30 2024-03-18T00:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.174+0000 7f3509401740 10 client.16341 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.174+0000 7f3509401740 10 client.16341 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.174+0000 7f3509401740 20 client.16341 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.174+0000 7f3509401740 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddcedd3ab0 0x55ddcedd5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.175+0000 7f3509401740 4 client.16341 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:53:46.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.175+0000 7f3509401740 1 -- 172.21.15.49:0/3906516846 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddced2ff80 con 0x55ddcedd3ab0 2024-03-18T00:53:46.175 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.175+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddcedd3ab0 0x55ddcedd5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.177+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddcedd3ab0 0x55ddcedd5f50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0e3f4d9c83317a7 server_cookie=be02d6a0e0a93da2 in_seq=0 out_seq=0 2024-03-18T00:53:46.177 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.177+0000 7f34df7fe640 10 client.16341 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.184+0000 7f34df7fe640 1 -- 172.21.15.49:0/3906516846 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ddced2ff80 con 0x55ddcedd3ab0 2024-03-18T00:53:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.184+0000 7f34df7fe640 10 client.16341 handle_command_reply: tid=0 2024-03-18T00:53:46.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.184+0000 7f3509401740 1 client.16341 shutdown 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 -- 172.21.15.49:0/3906516846 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedbdd30 msgr2=0x55ddcedc7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedbdd30 0x55ddcedc7d40 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f34f0006010 tx=0x7f34f0004270 comp rx=0 tx=0).stop 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 -- 172.21.15.49:0/3906516846 shutdown_connections 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddcedd3ab0 0x55ddcedd5f50 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedccd70 0x55ddcedcf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedbdd30 0x55ddcedc7d40 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f34df7fe640 0 client.16341 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 --2- 172.21.15.49:0/3906516846 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedc7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 -- 172.21.15.49:0/3906516846 >> 172.21.15.49:0/3906516846 conn(0x55ddcedb88b0 msgr2=0x55ddcedb7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:46.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 -- 172.21.15.49:0/3906516846 shutdown_connections 2024-03-18T00:53:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 1 -- 172.21.15.49:0/3906516846 wait complete. 2024-03-18T00:53:46.186 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.185+0000 7f3509401740 20 client.16341 trim_cache size 0 max 16384 2024-03-18T00:53:46.223 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.220+0000 7f3509401740 1 Processor -- start 2024-03-18T00:53:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.229+0000 7f3509401740 1 -- start start 2024-03-18T00:53:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.229+0000 7f3509401740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedb3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.230+0000 7f3509401740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 0x55ddcedbbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.230+0000 7f3509401740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcedbc050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.230+0000 7f3509401740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f35001b7cc0 con 0x55ddcedccd70 2024-03-18T00:53:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.230+0000 7f3509401740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f35001b9800 con 0x55ddcedbd340 2024-03-18T00:53:46.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.230+0000 7f3509401740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f35001b8390 con 0x55ddcedbdd30 2024-03-18T00:53:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.238+0000 7f34ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcedbc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.238+0000 7f34ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcedbc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36748/0 (socket says 172.21.15.49:36748) 2024-03-18T00:53:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.238+0000 7f34ff7fe640 1 -- 172.21.15.49:0/3337863167 learned_addr learned my addr 172.21.15.49:0/3337863167 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:46.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.239+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/3337863167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedb3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.248+0000 7f34fffff640 1 --2- 172.21.15.49:0/3337863167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 0x55ddcedbbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.250+0000 7f34fffff640 1 -- 172.21.15.49:0/3337863167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 msgr2=0x55ddcedb3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.250+0000 7f34fffff640 1 --2- 172.21.15.49:0/3337863167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedb3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.250+0000 7f34fffff640 1 -- 172.21.15.49:0/3337863167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 msgr2=0x55ddcedbc050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.250+0000 7f34fffff640 1 --2- 172.21.15.49:0/3337863167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcedbc050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.250+0000 7f34fffff640 1 -- 172.21.15.49:0/3337863167 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddcedbc950 con 0x55ddcedbdd30 2024-03-18T00:53:46.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.251+0000 7f34fffff640 1 --2- 172.21.15.49:0/3337863167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 0x55ddcedbbb10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f34f0003d40 tx=0x7f34f0005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e053e81bc80ae2b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.257+0000 7f34e5ffb640 1 -- 172.21.15.49:0/3337863167 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0034440 con 0x55ddcedbdd30 2024-03-18T00:53:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.257+0000 7f34e5ffb640 1 -- 172.21.15.49:0/3337863167 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f0056070 con 0x55ddcedbdd30 2024-03-18T00:53:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.257+0000 7f34e5ffb640 1 -- 172.21.15.49:0/3337863167 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f00046c0 con 0x55ddcedbdd30 2024-03-18T00:53:46.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.260+0000 7f3509401740 1 -- 172.21.15.49:0/3337863167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 msgr2=0x55ddcedbbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:46.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.260+0000 7f3509401740 1 --2- 172.21.15.49:0/3337863167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 0x55ddcedbbb10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f34f0003d40 tx=0x7f34f0005a20 comp rx=0 tx=0).stop 2024-03-18T00:53:46.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.260+0000 7f3509401740 1 -- 172.21.15.49:0/3337863167 shutdown_connections 2024-03-18T00:53:46.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.260+0000 7f3509401740 1 --2- 172.21.15.49:0/3337863167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcedbc050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.260+0000 7f3509401740 1 --2- 172.21.15.49:0/3337863167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbdd30 0x55ddcedbbb10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.260+0000 7f3509401740 1 --2- 172.21.15.49:0/3337863167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbd340 0x55ddcedb3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.260+0000 7f3509401740 1 -- 172.21.15.49:0/3337863167 >> 172.21.15.49:0/3337863167 conn(0x55ddcedb88b0 msgr2=0x55ddcedb8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.266+0000 7f3509401740 1 -- 172.21.15.49:0/3337863167 shutdown_connections 2024-03-18T00:53:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.266+0000 7f3509401740 1 -- 172.21.15.49:0/3337863167 wait complete. 2024-03-18T00:53:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.267+0000 7f3509401740 1 -- start start 2024-03-18T00:53:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.277+0000 7f3509401740 1 Processor -- start 2024-03-18T00:53:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.278+0000 7f3509401740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbd340 0x55ddcede3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.277 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.278+0000 7f3509401740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbdd30 0x55ddcede6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.278+0000 7f3509401740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcede6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.278+0000 7f3509401740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f35001b9130 con 0x55ddcedccd70 2024-03-18T00:53:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.278+0000 7f3509401740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f35001b8a60 con 0x55ddcedbdd30 2024-03-18T00:53:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.278+0000 7f3509401740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f35001b5ca0 con 0x55ddcedbd340 2024-03-18T00:53:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.278+0000 7f3509401740 10 client.0 _ll_register_callbacks cb 0x7f34f0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:53:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.282+0000 7f34ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcede6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.282+0000 7f34ff7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcede6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36750/0 (socket says 172.21.15.49:36750) 2024-03-18T00:53:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.282+0000 7f34ff7fe640 1 -- 172.21.15.49:0/976528991 learned_addr learned my addr 172.21.15.49:0/976528991 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:53:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.282+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbd340 0x55ddcede3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.283+0000 7f34fffff640 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbdd30 0x55ddcede6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.288+0000 7f34ff7fe640 1 -- 172.21.15.49:0/976528991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbd340 msgr2=0x55ddcede3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.288+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbd340 0x55ddcede3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.288+0000 7f34ff7fe640 1 -- 172.21.15.49:0/976528991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbdd30 msgr2=0x55ddcede6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:53:46.288 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.288+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbdd30 0x55ddcede6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:46.289 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.288+0000 7f34ff7fe640 1 -- 172.21.15.49:0/976528991 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddcede6ea0 con 0x55ddcedccd70 2024-03-18T00:53:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.294+0000 7f34ff7fe640 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcede6670 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f34f4001eb0 tx=0x7f34f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b784eb6d10e79df1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:53:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.294+0000 7f34df7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:53:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.300+0000 7f34df7fe640 1 -- 172.21.15.49:0/976528991 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4016040 con 0x55ddcedccd70 2024-03-18T00:53:46.300 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.300+0000 7f34df7fe640 1 -- 172.21.15.49:0/976528991 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f34f4026540 con 0x55ddcedccd70 2024-03-18T00:53:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.300+0000 7f3509401740 1 -- 172.21.15.49:0/976528991 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34f0056070 con 0x55ddcedccd70 2024-03-18T00:53:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.305+0000 7f34df7fe640 1 -- 172.21.15.49:0/976528991 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f34f4026740 con 0x55ddcedccd70 2024-03-18T00:53:46.308 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.307+0000 7f3509401740 10 client.16347 fetch_fsmap learned FSMap version 27 2024-03-18T00:53:46.308 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.307+0000 7f3509401740 1 -- 172.21.15.49:0/976528991 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ddcee4b3d0 con 0x55ddcedccd70 2024-03-18T00:53:46.311 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.309+0000 7f34df7fe640 1 -- 172.21.15.49:0/976528991 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34f0056070 con 0x55ddcedccd70 2024-03-18T00:53:46.311 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.310+0000 7f34df7fe640 1 -- 172.21.15.49:0/976528991 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f34f40124e0 con 0x55ddcedccd70 2024-03-18T00:53:46.311 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.311+0000 7f3509401740 10 client.16347 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:53:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.311+0000 7f3509401740 10 client.16347 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:53:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.311+0000 7f3509401740 20 client.16347 populate_metadata read hostname 'smithi049' 2024-03-18T00:53:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.311+0000 7f3509401740 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddcee4b9a0 0x55ddcee4dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:53:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.311+0000 7f3509401740 4 client.16347 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:53:46.312 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.311+0000 7f3509401740 1 -- 172.21.15.49:0/976528991 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ddced2a2d0 con 0x55ddcee4b9a0 2024-03-18T00:53:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.315+0000 7f34fffff640 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddcee4b9a0 0x55ddcee4dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:53:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.325+0000 7f34fffff640 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddcee4b9a0 0x55ddcee4dd90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c02dc4eff904d158 server_cookie=98249779b4199edf in_seq=0 out_seq=0 2024-03-18T00:53:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:46.326+0000 7f34df7fe640 10 client.16347 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:47.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:47 smithi049 ceph-mon[32130]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 842 MiB/s wr, 1.44k op/s 2024-03-18T00:53:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:47 smithi107 ceph-mon[36270]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 842 MiB/s wr, 1.44k op/s 2024-03-18T00:53:47.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:47 smithi106 ceph-mon[35899]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 842 MiB/s wr, 1.44k op/s 2024-03-18T00:53:49.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:49 smithi049 ceph-mon[32130]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 869 MiB/s wr, 1.58k op/s 2024-03-18T00:53:49.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:49 smithi107 ceph-mon[36270]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 869 MiB/s wr, 1.58k op/s 2024-03-18T00:53:49.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:49 smithi106 ceph-mon[35899]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 869 MiB/s wr, 1.58k op/s 2024-03-18T00:53:50.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:50 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:53:50.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:50 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:53:50.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:50 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:53:51.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:51 smithi049 ceph-mon[32130]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1005 MiB/s wr, 2.08k op/s 2024-03-18T00:53:51.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:51 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:53:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:51 smithi107 ceph-mon[36270]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1005 MiB/s wr, 2.08k op/s 2024-03-18T00:53:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:51 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:53:51.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:51 smithi106 ceph-mon[35899]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1005 MiB/s wr, 2.08k op/s 2024-03-18T00:53:51.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:51 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:53:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.497+0000 7f34df7fe640 1 -- 172.21.15.49:0/976528991 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ddced2a2d0 con 0x55ddcee4b9a0 2024-03-18T00:53:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.497+0000 7f34df7fe640 10 client.16347 handle_command_reply: tid=0 2024-03-18T00:53:52.499 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.497+0000 7f3509401740 1 client.16347 shutdown 2024-03-18T00:53:52.500 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.499+0000 7f3509401740 1 -- 172.21.15.49:0/976528991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 msgr2=0x55ddcede6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:52.500 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.499+0000 7f3509401740 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcede6670 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f34f4001eb0 tx=0x7f34f40046c0 comp rx=0 tx=0).stop 2024-03-18T00:53:52.501 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.500+0000 7f3509401740 1 -- 172.21.15.49:0/976528991 shutdown_connections 2024-03-18T00:53:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.501+0000 7f3509401740 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ddcee4b9a0 0x55ddcee4dd90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.501+0000 7f3509401740 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ddcedbdd30 0x55ddcede6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.501+0000 7f3509401740 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ddcedccd70 0x55ddcede6670 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.501+0000 7f3509401740 1 --2- 172.21.15.49:0/976528991 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ddcedbd340 0x55ddcede3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.501+0000 7f3509401740 1 -- 172.21.15.49:0/976528991 >> 172.21.15.49:0/976528991 conn(0x55ddcedb88b0 msgr2=0x55ddcedb3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.501+0000 7f34df7fe640 0 client.16347 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:53:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.501+0000 7f3509401740 1 -- 172.21.15.49:0/976528991 shutdown_connections 2024-03-18T00:53:52.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.502+0000 7f3509401740 1 -- 172.21.15.49:0/976528991 wait complete. 2024-03-18T00:53:52.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.502+0000 7f3509401740 20 client.16347 trim_cache size 0 max 16384 2024-03-18T00:53:52.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:53:52.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:53:52.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:53:52.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.504+0000 7f350673b640 1 -- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34e80a2920 msgr2=0x7f34e80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.504+0000 7f350673b640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34e80a2920 0x7f34e80a4de0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f34f80c6700 tx=0x7f34f80c6670 comp rx=0 tx=0).stop 2024-03-18T00:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.504+0000 7f350673b640 1 -- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f350010db00 msgr2=0x7f35001ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:53:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.504+0000 7f350673b640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f350010db00 0x7f35001ade50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f34f800b970 tx=0x7f34f800be30 comp rx=0 tx=0).stop 2024-03-18T00:53:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.507+0000 7f350673b640 1 -- 172.21.15.49:0/982319927 shutdown_connections 2024-03-18T00:53:52.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.507+0000 7f350673b640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34e80a2920 0x7f34e80a4de0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.507+0000 7f350673b640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f350010b150 0x7f35001a6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.507+0000 7f350673b640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f350010db00 0x7f35001ade50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.507+0000 7f350673b640 1 --2- 172.21.15.49:0/982319927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f35001087e0 0x7f35001a68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:53:52.508 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.507+0000 7f350673b640 1 -- 172.21.15.49:0/982319927 >> 172.21.15.49:0/982319927 conn(0x7f350006e820 msgr2=0x7f350010d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:53:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.511+0000 7f350673b640 1 -- 172.21.15.49:0/982319927 shutdown_connections 2024-03-18T00:53:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:53:52.511+0000 7f350673b640 1 -- 172.21.15.49:0/982319927 wait complete. 2024-03-18T00:53:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:53 smithi049 ceph-mon[32130]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 823 MiB/s wr, 1.78k op/s 2024-03-18T00:53:53.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:53 smithi107 ceph-mon[36270]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 823 MiB/s wr, 1.78k op/s 2024-03-18T00:53:53.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:53 smithi106 ceph-mon[35899]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 823 MiB/s wr, 1.78k op/s 2024-03-18T00:53:55.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:55 smithi049 ceph-mon[32130]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1017 MiB/s wr, 2.28k op/s 2024-03-18T00:53:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:55 smithi107 ceph-mon[36270]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1017 MiB/s wr, 2.28k op/s 2024-03-18T00:53:55.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:55 smithi106 ceph-mon[35899]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1017 MiB/s wr, 2.28k op/s 2024-03-18T00:53:57.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:57 smithi049 ceph-mon[32130]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 848 MiB/s wr, 2.04k op/s 2024-03-18T00:53:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:57 smithi107 ceph-mon[36270]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 848 MiB/s wr, 2.04k op/s 2024-03-18T00:53:57.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:57 smithi106 ceph-mon[35899]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 848 MiB/s wr, 2.04k op/s 2024-03-18T00:53:59.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:53:59 smithi049 ceph-mon[32130]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 870 MiB/s wr, 2.10k op/s 2024-03-18T00:53:59.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:53:59 smithi107 ceph-mon[36270]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 870 MiB/s wr, 2.10k op/s 2024-03-18T00:53:59.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:53:59 smithi106 ceph-mon[35899]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 870 MiB/s wr, 2.10k op/s 2024-03-18T00:54:01.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:01 smithi049 ceph-mon[32130]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1000 MiB/s wr, 2.24k op/s 2024-03-18T00:54:01.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:01 smithi107 ceph-mon[36270]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1000 MiB/s wr, 2.24k op/s 2024-03-18T00:54:01.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:01 smithi106 ceph-mon[35899]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1000 MiB/s wr, 2.24k op/s 2024-03-18T00:54:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:03 smithi107 ceph-mon[36270]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 816 MiB/s wr, 1.73k op/s 2024-03-18T00:54:03.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:03 smithi106 ceph-mon[35899]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 816 MiB/s wr, 1.73k op/s 2024-03-18T00:54:03.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:03 smithi049 ceph-mon[32130]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 816 MiB/s wr, 1.73k op/s 2024-03-18T00:54:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:05 smithi107 ceph-mon[36270]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1010 MiB/s wr, 2.25k op/s 2024-03-18T00:54:05.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:05 smithi106 ceph-mon[35899]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1010 MiB/s wr, 2.25k op/s 2024-03-18T00:54:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.852+0000 7fa22b27a640 1 -- 172.21.15.49:0/3212124677 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa224072a90 msgr2=0x7fa224072eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.852+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3212124677 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa224072a90 0x7fa224072eb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa21c01acb0 tx=0x7fa21c03d170 comp rx=0 tx=0).stop 2024-03-18T00:54:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.854+0000 7fa22b27a640 1 -- 172.21.15.49:0/3212124677 shutdown_connections 2024-03-18T00:54:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.854+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3212124677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2240733f0 0x7fa224108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.854+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3212124677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa2241094d0 0x7fa22410e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.854+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3212124677 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa224072a90 0x7fa224072eb0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.854+0000 7fa22b27a640 1 -- 172.21.15.49:0/3212124677 >> 172.21.15.49:0/3212124677 conn(0x7fa22406e650 msgr2=0x7fa224070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.854+0000 7fa22b27a640 1 -- 172.21.15.49:0/3212124677 shutdown_connections 2024-03-18T00:54:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.855+0000 7fa22b27a640 1 -- 172.21.15.49:0/3212124677 wait complete. 2024-03-18T00:54:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.856+0000 7fa22b27a640 1 Processor -- start 2024-03-18T00:54:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.856+0000 7fa22b27a640 1 -- start start 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.856+0000 7fa22b27a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa224072a90 0x7fa224073df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.856+0000 7fa22b27a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2240733f0 0x7fa224074330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.856+0000 7fa22b27a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2241094d0 0x7fa22407b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.856+0000 7fa22b27a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241119f0 con 0x7fa2240733f0 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.856+0000 7fa22b27a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa224111870 con 0x7fa2241094d0 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.856+0000 7fa22b27a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa224111b70 con 0x7fa224072a90 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.857+0000 7fa229a77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2240733f0 0x7fa224074330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.857+0000 7fa229a77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2240733f0 0x7fa224074330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43726/0 (socket says 172.21.15.49:43726) 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.857+0000 7fa229a77640 1 -- 172.21.15.49:0/3114623226 learned_addr learned my addr 172.21.15.49:0/3114623226 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.857+0000 7fa22a278640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa224072a90 0x7fa224073df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.858+0000 7fa22aa79640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2241094d0 0x7fa22407b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.858+0000 7fa229a77640 1 -- 172.21.15.49:0/3114623226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa224072a90 msgr2=0x7fa224073df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:05.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.858+0000 7fa229a77640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa224072a90 0x7fa224073df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.858+0000 7fa229a77640 1 -- 172.21.15.49:0/3114623226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2241094d0 msgr2=0x7fa22407b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.858+0000 7fa229a77640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2241094d0 0x7fa22407b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.858+0000 7fa229a77640 1 -- 172.21.15.49:0/3114623226 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22407ba10 con 0x7fa2240733f0 2024-03-18T00:54:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.859+0000 7fa229a77640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2240733f0 0x7fa224074330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa22000b470 tx=0x7fa22000b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.859+0000 7fa21b7fe640 1 -- 172.21.15.49:0/3114623226 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220015020 con 0x7fa2240733f0 2024-03-18T00:54:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.859+0000 7fa22b27a640 1 -- 172.21.15.49:0/3114623226 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa22407bd00 con 0x7fa2240733f0 2024-03-18T00:54:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.859+0000 7fa21b7fe640 1 -- 172.21.15.49:0/3114623226 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa220011d70 con 0x7fa2240733f0 2024-03-18T00:54:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.859+0000 7fa21b7fe640 1 -- 172.21.15.49:0/3114623226 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22000f340 con 0x7fa2240733f0 2024-03-18T00:54:05.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.860+0000 7fa22b27a640 1 -- 172.21.15.49:0/3114623226 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa22407c2d0 con 0x7fa2240733f0 2024-03-18T00:54:05.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.865+0000 7fa22f1a0740 1 Processor -- start 2024-03-18T00:54:05.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.865+0000 7fa22f1a0740 1 -- start start 2024-03-18T00:54:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.866+0000 7fa21b7fe640 1 -- 172.21.15.49:0/3114623226 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa220018020 con 0x7fa2240733f0 2024-03-18T00:54:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.866+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f4a500 0x556523f4a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.866+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f4aef0 0x556523f4f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.866+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f50060 0x556523f524f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.866+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x556523eafa40 con 0x556523f4aef0 2024-03-18T00:54:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.866+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x556523e58920 con 0x556523f4a500 2024-03-18T00:54:05.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.866+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556523eabae0 con 0x556523f50060 2024-03-18T00:54:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.866+0000 7fa22a278640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f4aef0 0x556523f4f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.867+0000 7fa22a278640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f4aef0 0x556523f4f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43730/0 (socket says 172.21.15.49:43730) 2024-03-18T00:54:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.867+0000 7fa22a278640 1 -- 172.21.15.49:0/181419793 learned_addr learned my addr 172.21.15.49:0/181419793 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.867+0000 7fa22a278640 1 -- 172.21.15.49:0/181419793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f50060 msgr2=0x556523f524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:05.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.867+0000 7fa22a278640 1 --2- 172.21.15.49:0/181419793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f50060 0x556523f524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.867+0000 7fa22a278640 1 -- 172.21.15.49:0/181419793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f4a500 msgr2=0x556523f4a920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.867+0000 7fa22a278640 1 --2- 172.21.15.49:0/181419793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f4a500 0x556523f4a920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.867+0000 7fa22a278640 1 -- 172.21.15.49:0/181419793 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556523e70460 con 0x556523f4aef0 2024-03-18T00:54:05.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.868+0000 7fa21b7fe640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa2100a28b0 0x7fa2100a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.868+0000 7fa22a278640 1 --2- 172.21.15.49:0/181419793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f4aef0 0x556523f4f9f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa21c000f80 tx=0x7fa21c01a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642a5d9b22270ecb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.868+0000 7fa21b7fe640 1 -- 172.21.15.49:0/3114623226 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa2200ce5c0 con 0x7fa2240733f0 2024-03-18T00:54:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.869+0000 7fa218bfe640 1 -- 172.21.15.49:0/181419793 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c04a050 con 0x556523f4aef0 2024-03-18T00:54:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.869+0000 7fa218bfe640 1 -- 172.21.15.49:0/181419793 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa21c056070 con 0x556523f4aef0 2024-03-18T00:54:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.869+0000 7fa218bfe640 1 -- 172.21.15.49:0/181419793 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c004040 con 0x556523f4aef0 2024-03-18T00:54:05.869 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.869+0000 7fa22aa79640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa2100a28b0 0x7fa2100a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.870+0000 7fa22f1a0740 1 -- 172.21.15.49:0/181419793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f4aef0 msgr2=0x556523f4f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.871+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/181419793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f4aef0 0x556523f4f9f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa21c000f80 tx=0x7fa21c01a880 comp rx=0 tx=0).stop 2024-03-18T00:54:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.871+0000 7fa22aa79640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa2100a28b0 0x7fa2100a4d70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x556523f4b750 tx=0x7fa214008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.871+0000 7fa22f1a0740 1 -- 172.21.15.49:0/181419793 shutdown_connections 2024-03-18T00:54:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.871+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/181419793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f50060 0x556523f524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.871+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/181419793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f4aef0 0x556523f4f9f0 secure :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa21c000f80 tx=0x7fa21c01a880 comp rx=0 tx=0).stop 2024-03-18T00:54:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.871+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/181419793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f4a500 0x556523f4a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.871+0000 7fa22f1a0740 1 -- 172.21.15.49:0/181419793 >> 172.21.15.49:0/181419793 conn(0x556523f45a70 msgr2=0x556523f45e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.871+0000 7fa22f1a0740 1 -- 172.21.15.49:0/181419793 shutdown_connections 2024-03-18T00:54:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.871+0000 7fa22f1a0740 1 -- 172.21.15.49:0/181419793 wait complete. 2024-03-18T00:54:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.872+0000 7fa22f1a0740 1 -- start start 2024-03-18T00:54:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.872+0000 7fa22f1a0740 1 Processor -- start 2024-03-18T00:54:05.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:05 smithi049 ceph-mon[32130]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1010 MiB/s wr, 2.25k op/s 2024-03-18T00:54:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.875+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f4a500 0x556523f54a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.875+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f54f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.875+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f59f80 0x556523f5c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.875+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x556523f52fb0 con 0x556523f50060 2024-03-18T00:54:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.875+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x556523e6bd70 con 0x556523f59f80 2024-03-18T00:54:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.875+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556523f53130 con 0x556523f4a500 2024-03-18T00:54:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.875+0000 7fa22f1a0740 10 client.0 _ll_register_callbacks cb 0x556523bc3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.876+0000 7fa229a77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f54f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.876+0000 7fa229a77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f54f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43746/0 (socket says 172.21.15.49:43746) 2024-03-18T00:54:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.876+0000 7fa229a77640 1 -- 172.21.15.49:0/1210851630 learned_addr learned my addr 172.21.15.49:0/1210851630 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.877+0000 7fa22aa79640 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f59f80 0x556523f5c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.877+0000 7fa229a77640 1 -- 172.21.15.49:0/1210851630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f4a500 msgr2=0x556523f54a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.877+0000 7fa229a77640 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f4a500 0x556523f54a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.877+0000 7fa229a77640 1 -- 172.21.15.49:0/1210851630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f59f80 msgr2=0x556523f5c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.877+0000 7fa229a77640 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f59f80 0x556523f5c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.877+0000 7fa229a77640 1 -- 172.21.15.49:0/1210851630 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556523bc4210 con 0x556523f50060 2024-03-18T00:54:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.878+0000 7fa229a77640 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f54f50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa220097340 tx=0x7fa22009a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a91def233409472 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.879+0000 7fa20dffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.879+0000 7fa20dffb640 1 -- 172.21.15.49:0/1210851630 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22009f030 con 0x556523f50060 2024-03-18T00:54:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.879+0000 7fa22f1a0740 1 -- 172.21.15.49:0/1210851630 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa21c056070 con 0x556523f50060 2024-03-18T00:54:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.879+0000 7fa20dffb640 1 -- 172.21.15.49:0/1210851630 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa22009a330 con 0x556523f50060 2024-03-18T00:54:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.880+0000 7fa20dffb640 1 -- 172.21.15.49:0/1210851630 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa2200a3680 con 0x556523f50060 2024-03-18T00:54:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.880+0000 7fa22f1a0740 10 client.16359 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.880+0000 7fa22f1a0740 1 -- 172.21.15.49:0/1210851630 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556523c90620 con 0x556523f50060 2024-03-18T00:54:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.881+0000 7fa20dffb640 1 -- 172.21.15.49:0/1210851630 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa21c056070 con 0x556523f50060 2024-03-18T00:54:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.881+0000 7fa20dffb640 1 -- 172.21.15.49:0/1210851630 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa220096c30 con 0x556523f50060 2024-03-18T00:54:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.881+0000 7fa22f1a0740 10 client.16359 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.881+0000 7fa22f1a0740 10 client.16359 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.881+0000 7fa22f1a0740 20 client.16359 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.881+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556523f61080 0x556523f63500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.881+0000 7fa22f1a0740 4 client.16359 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.881+0000 7fa22f1a0740 1 -- 172.21.15.49:0/1210851630 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556523ebcf80 con 0x556523f61080 2024-03-18T00:54:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.882+0000 7fa22a278640 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556523f61080 0x556523f63500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.886+0000 7fa22a278640 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556523f61080 0x556523f63500 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26152e8bd8f1812f server_cookie=5a78fec191580bd5 in_seq=0 out_seq=0 2024-03-18T00:54:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.886+0000 7fa20dffb640 10 client.16359 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.889+0000 7fa20dffb640 1 -- 172.21.15.49:0/1210851630 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556523ebcf80 con 0x556523f61080 2024-03-18T00:54:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.889+0000 7fa20dffb640 10 client.16359 handle_command_reply: tid=0 2024-03-18T00:54:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.891+0000 7fa22f1a0740 1 client.16359 shutdown 2024-03-18T00:54:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.891+0000 7fa22f1a0740 1 -- 172.21.15.49:0/1210851630 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 msgr2=0x556523f54f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.891+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f54f50 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa220097340 tx=0x7fa22009a840 comp rx=0 tx=0).stop 2024-03-18T00:54:05.895 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.896+0000 7fa22f1a0740 1 -- 172.21.15.49:0/1210851630 shutdown_connections 2024-03-18T00:54:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.896+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556523f61080 0x556523f63500 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.896+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f59f80 0x556523f5c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.896+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f54f50 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.896+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/1210851630 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f4a500 0x556523f54a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.896+0000 7fa22f1a0740 1 -- 172.21.15.49:0/1210851630 >> 172.21.15.49:0/1210851630 conn(0x556523f45a70 msgr2=0x556523f44c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.896+0000 7fa20dffb640 0 client.16359 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.897+0000 7fa22f1a0740 1 -- 172.21.15.49:0/1210851630 shutdown_connections 2024-03-18T00:54:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.897+0000 7fa22f1a0740 1 -- 172.21.15.49:0/1210851630 wait complete. 2024-03-18T00:54:05.897 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.897+0000 7fa22f1a0740 20 client.16359 trim_cache size 0 max 16384 2024-03-18T00:54:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.929+0000 7fa22f1a0740 1 Processor -- start 2024-03-18T00:54:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.929+0000 7fa22f1a0740 1 -- start start 2024-03-18T00:54:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f4a500 0x556523f524d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f4c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f59f80 0x556523f4c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.930 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241c0380 con 0x556523f50060 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241c1ec0 con 0x556523f4a500 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241c0a50 con 0x556523f59f80 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa229a77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f4c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa229a77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f4c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43752/0 (socket says 172.21.15.49:43752) 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa229a77640 1 -- 172.21.15.49:0/4091274179 learned_addr learned my addr 172.21.15.49:0/4091274179 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa229a77640 1 -- 172.21.15.49:0/4091274179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f59f80 msgr2=0x556523f4c970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa229a77640 1 --2- 172.21.15.49:0/4091274179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f59f80 0x556523f4c970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa229a77640 1 -- 172.21.15.49:0/4091274179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f4a500 msgr2=0x556523f524d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa229a77640 1 --2- 172.21.15.49:0/4091274179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f4a500 0x556523f524d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.930+0000 7fa229a77640 1 -- 172.21.15.49:0/4091274179 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556523f4a140 con 0x556523f50060 2024-03-18T00:54:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.931+0000 7fa229a77640 1 --2- 172.21.15.49:0/4091274179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f4c430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa22009abe0 tx=0x7fa22009a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd0281eceb4369d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.931+0000 7fa20ffff640 1 -- 172.21.15.49:0/4091274179 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa220099f70 con 0x556523f50060 2024-03-18T00:54:05.933 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.931+0000 7fa20ffff640 1 -- 172.21.15.49:0/4091274179 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa220096c30 con 0x556523f50060 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.931+0000 7fa20ffff640 1 -- 172.21.15.49:0/4091274179 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22009a330 con 0x556523f50060 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.933+0000 7fa22f1a0740 1 -- 172.21.15.49:0/4091274179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 msgr2=0x556523f4c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.933+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/4091274179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f4c430 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa22009abe0 tx=0x7fa22009a970 comp rx=0 tx=0).stop 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.933+0000 7fa22f1a0740 1 -- 172.21.15.49:0/4091274179 shutdown_connections 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.933+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/4091274179 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f59f80 0x556523f4c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.933+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/4091274179 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f50060 0x556523f4c430 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.933+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/4091274179 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f4a500 0x556523f524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.933+0000 7fa22f1a0740 1 -- 172.21.15.49:0/4091274179 >> 172.21.15.49:0/4091274179 conn(0x556523f45a70 msgr2=0x556523f45e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.933+0000 7fa22f1a0740 1 -- 172.21.15.49:0/4091274179 shutdown_connections 2024-03-18T00:54:05.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.933+0000 7fa22f1a0740 1 -- 172.21.15.49:0/4091274179 wait complete. 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.934+0000 7fa22f1a0740 1 -- start start 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.934+0000 7fa22f1a0740 1 Processor -- start 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.935+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f4a500 0x556523f71ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.935+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f50060 0x556523f72410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.935+0000 7fa22f1a0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f59f80 0x556523f73420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.936+0000 7fa22f1a0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241c17f0 con 0x556523f59f80 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.936+0000 7fa22aa79640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f59f80 0x556523f73420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.936+0000 7fa22aa79640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f59f80 0x556523f73420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43754/0 (socket says 172.21.15.49:43754) 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.936+0000 7fa22aa79640 1 -- 172.21.15.49:0/309777056 learned_addr learned my addr 172.21.15.49:0/309777056 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:05.936 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.936+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa2241c1120 con 0x556523f50060 2024-03-18T00:54:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.936+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa224083140 con 0x556523f4a500 2024-03-18T00:54:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.937+0000 7fa22aa79640 1 -- 172.21.15.49:0/309777056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f4a500 msgr2=0x556523f71ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.937+0000 7fa22aa79640 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f4a500 0x556523f71ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.937+0000 7fa22aa79640 1 -- 172.21.15.49:0/309777056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f50060 msgr2=0x556523f72410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.937+0000 7fa22aa79640 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f50060 0x556523f72410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:05.937 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.937+0000 7fa22aa79640 1 -- 172.21.15.49:0/309777056 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556523f4ab10 con 0x556523f59f80 2024-03-18T00:54:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.937+0000 7fa22f1a0740 10 client.0 _ll_register_callbacks cb 0x7fa220010f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.937+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa220096c30 con 0x556523f59f80 2024-03-18T00:54:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.938+0000 7fa22aa79640 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f59f80 0x556523f73420 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa21400dd90 tx=0x7fa214004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974239381c6e98e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:05.938 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.938+0000 7fa20dffb640 10 client.16365 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.939+0000 7fa20dffb640 1 -- 172.21.15.49:0/309777056 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21400ddc0 con 0x556523f59f80 2024-03-18T00:54:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.939+0000 7fa20dffb640 1 -- 172.21.15.49:0/309777056 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa21400f040 con 0x556523f59f80 2024-03-18T00:54:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.939+0000 7fa20dffb640 1 -- 172.21.15.49:0/309777056 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa214005990 con 0x556523f59f80 2024-03-18T00:54:05.939 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.940+0000 7fa22f1a0740 10 client.16365 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.940+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({fsmap=27}) -- 0x556523fc6e00 con 0x556523f59f80 2024-03-18T00:54:05.940 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.940+0000 7fa20dffb640 1 -- 172.21.15.49:0/309777056 <== mon.0 v2:172.21.15.49:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa214013070 con 0x556523f59f80 2024-03-18T00:54:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.941+0000 7fa22f1a0740 10 client.16365 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.941+0000 7fa22f1a0740 10 client.16365 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.941+0000 7fa22f1a0740 20 client.16365 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.941+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556523fc7690 0x556523fc9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.941+0000 7fa22f1a0740 4 client.16365 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:54:05.941 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.941+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556523eb72d0 con 0x556523fc7690 2024-03-18T00:54:05.942 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.943+0000 7fa22a278640 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556523fc7690 0x556523fc9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:05.948 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.948+0000 7fa20dffb640 10 client.16365 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:05.948 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:05.948+0000 7fa22a278640 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556523fc7690 0x556523fc9b50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9670e3b2c128d09a server_cookie=6dd85fef79a502a6 in_seq=0 out_seq=0 2024-03-18T00:54:07.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:07 smithi107 ceph-mon[36270]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 831 MiB/s wr, 1.74k op/s 2024-03-18T00:54:07.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:07 smithi106 ceph-mon[35899]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 831 MiB/s wr, 1.74k op/s 2024-03-18T00:54:07.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:07 smithi049 ceph-mon[32130]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 831 MiB/s wr, 1.74k op/s 2024-03-18T00:54:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:09 smithi107 ceph-mon[36270]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 842 MiB/s wr, 1.68k op/s 2024-03-18T00:54:09.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:09 smithi106 ceph-mon[35899]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 842 MiB/s wr, 1.68k op/s 2024-03-18T00:54:09.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:09 smithi049 ceph-mon[32130]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 842 MiB/s wr, 1.68k op/s 2024-03-18T00:54:10.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:54:10.146 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:54:10.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.523+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2871373442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e98109140 msgr2=0x7f0e9810fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:10.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.523+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2871373442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e98109140 0x7f0e9810fb20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c009900 tx=0x7f0e8c02bc50 comp rx=0 tx=0).stop 2024-03-18T00:54:10.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.523+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2871373442 shutdown_connections 2024-03-18T00:54:10.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.523+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2871373442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e98110060 0x7f0e98112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.523+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2871373442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e98109140 0x7f0e9810fb20 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.523+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2871373442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e981087e0 0x7f0e98108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.523+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2871373442 >> 172.21.15.49:0/2871373442 conn(0x7f0e9806e640 msgr2=0x7f0e98070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:10.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.523+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2871373442 shutdown_connections 2024-03-18T00:54:10.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.523+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2871373442 wait complete. 2024-03-18T00:54:10.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.524+0000 7f0e9dc36640 1 Processor -- start 2024-03-18T00:54:10.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.525+0000 7f0e9dc36640 1 -- start start 2024-03-18T00:54:10.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.525+0000 7f0e9dc36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e981087e0 0x7f0e9810b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.525+0000 7f0e9dc36640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e98110060 0x7f0e98109ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.525+0000 7f0e9dc36640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e981a4150 0x7f0e98109ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.526+0000 7f0e9dc36640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98115c90 con 0x7f0e981a4150 2024-03-18T00:54:10.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.526+0000 7f0e9dc36640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98115b10 con 0x7f0e98110060 2024-03-18T00:54:10.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.526+0000 7f0e97fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e981a4150 0x7f0e98109ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:10.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.526+0000 7f0e97fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e981a4150 0x7f0e98109ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42212/0 (socket says 172.21.15.49:42212) 2024-03-18T00:54:10.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.526+0000 7f0e97fff640 1 -- 172.21.15.49:0/2590288426 learned_addr learned my addr 172.21.15.49:0/2590288426 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:10.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.526+0000 7f0e977fe640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e981087e0 0x7f0e9810b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:10.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.535+0000 7f0e96ffd640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e98110060 0x7f0e98109ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:10.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.535+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2590288426 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0e98115e10 con 0x7f0e981087e0 2024-03-18T00:54:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.551+0000 7f0e97fff640 1 -- 172.21.15.49:0/2590288426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e981087e0 msgr2=0x7f0e9810b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.551+0000 7f0e97fff640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e981087e0 0x7f0e9810b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.551+0000 7f0e97fff640 1 -- 172.21.15.49:0/2590288426 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e98110060 msgr2=0x7f0e98109ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.551+0000 7f0e97fff640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e98110060 0x7f0e98109ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.551+0000 7f0e97fff640 1 -- 172.21.15.49:0/2590288426 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9810a880 con 0x7f0e981a4150 2024-03-18T00:54:10.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.555+0000 7f0e97fff640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e981a4150 0x7f0e98109ff0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000cce0 tx=0x7f0e9000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:10.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.556+0000 7f0e94ff9640 1 -- 172.21.15.49:0/2590288426 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e90003970 con 0x7f0e981a4150 2024-03-18T00:54:10.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.556+0000 7f0e94ff9640 1 -- 172.21.15.49:0/2590288426 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e90004030 con 0x7f0e981a4150 2024-03-18T00:54:10.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.556+0000 7f0e94ff9640 1 -- 172.21.15.49:0/2590288426 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9000f7a0 con 0x7f0e981a4150 2024-03-18T00:54:10.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.556+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2590288426 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e981aee20 con 0x7f0e981a4150 2024-03-18T00:54:10.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.556+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2590288426 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e981af1d0 con 0x7f0e981a4150 2024-03-18T00:54:10.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.566+0000 7f0e94ff9640 1 -- 172.21.15.49:0/2590288426 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0e90003b10 con 0x7f0e981a4150 2024-03-18T00:54:10.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.567+0000 7f0e94ff9640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e800a29d0 0x7f0e800a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.567+0000 7f0e94ff9640 1 -- 172.21.15.49:0/2590288426 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0e900cea00 con 0x7f0e981a4150 2024-03-18T00:54:10.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.569+0000 7f0e977fe640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e800a29d0 0x7f0e800a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:10.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.573+0000 7f0ea0827740 1 Processor -- start 2024-03-18T00:54:10.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.579+0000 7f0ea0827740 1 -- start start 2024-03-18T00:54:10.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.583+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264a170 0x55770264a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.583+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264ab60 0x55770264f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.583+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264fcd0 0x557702652160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.583+0000 7f0ea0827740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5577025af500 con 0x55770264a170 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.583+0000 7f0ea0827740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557702557be0 con 0x55770264fcd0 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.583+0000 7f0ea0827740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5577025ab5a0 con 0x55770264ab60 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.585+0000 7f0e977fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264a170 0x55770264a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.585+0000 7f0e977fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264a170 0x55770264a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42228/0 (socket says 172.21.15.49:42228) 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.585+0000 7f0e977fe640 1 -- 172.21.15.49:0/3082164505 learned_addr learned my addr 172.21.15.49:0/3082164505 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.588+0000 7f0e977fe640 1 -- 172.21.15.49:0/3082164505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264ab60 msgr2=0x55770264f660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.588+0000 7f0e977fe640 1 --2- 172.21.15.49:0/3082164505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264ab60 0x55770264f660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.588+0000 7f0e977fe640 1 -- 172.21.15.49:0/3082164505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264fcd0 msgr2=0x557702652160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.588+0000 7f0e977fe640 1 --2- 172.21.15.49:0/3082164505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264fcd0 0x557702652160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.588+0000 7f0e977fe640 1 -- 172.21.15.49:0/3082164505 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55770256b4b0 con 0x55770264a170 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.589+0000 7f0e977fe640 1 --2- 172.21.15.49:0/3082164505 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264a170 0x55770264a590 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0e88012a60 tx=0x7f0e88012f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3625674e4aadc308 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:10.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.593+0000 7f0e7dffb640 1 -- 172.21.15.49:0/3082164505 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88023070 con 0x55770264a170 2024-03-18T00:54:10.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.593+0000 7f0e7dffb640 1 -- 172.21.15.49:0/3082164505 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e8800f8a0 con 0x55770264a170 2024-03-18T00:54:10.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.595+0000 7f0ea0827740 1 -- 172.21.15.49:0/3082164505 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264a170 msgr2=0x55770264a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:10.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.595+0000 7f0ea0827740 1 --2- 172.21.15.49:0/3082164505 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264a170 0x55770264a590 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f0e88012a60 tx=0x7f0e88012f20 comp rx=0 tx=0).stop 2024-03-18T00:54:10.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.598+0000 7f0ea0827740 1 -- 172.21.15.49:0/3082164505 shutdown_connections 2024-03-18T00:54:10.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.598+0000 7f0ea0827740 1 --2- 172.21.15.49:0/3082164505 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264fcd0 0x557702652160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.598+0000 7f0ea0827740 1 --2- 172.21.15.49:0/3082164505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264ab60 0x55770264f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.598+0000 7f0ea0827740 1 --2- 172.21.15.49:0/3082164505 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264a170 0x55770264a590 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.598+0000 7f0ea0827740 1 -- 172.21.15.49:0/3082164505 >> 172.21.15.49:0/3082164505 conn(0x5577026456e0 msgr2=0x557702645ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:10.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.602+0000 7f0e977fe640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e800a29d0 0x7f0e800a4e90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0e88020000 tx=0x7f0e88015000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:10.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.602+0000 7f0ea0827740 1 -- 172.21.15.49:0/3082164505 shutdown_connections 2024-03-18T00:54:10.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.605+0000 7f0ea0827740 1 -- 172.21.15.49:0/3082164505 wait complete. 2024-03-18T00:54:10.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.605+0000 7f0ea0827740 1 -- start start 2024-03-18T00:54:10.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.606+0000 7f0ea0827740 1 Processor -- start 2024-03-18T00:54:10.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026546b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264ab60 0x557702654bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264fcd0 0x55770265bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0ea0827740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557702652cd0 con 0x55770264ab60 2024-03-18T00:54:10.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0ea0827740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55770256b160 con 0x55770264a170 2024-03-18T00:54:10.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0ea0827740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557702652e50 con 0x55770264fcd0 2024-03-18T00:54:10.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0ea0827740 10 client.0 _ll_register_callbacks cb 0x55770254f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:10.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0e96ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264ab60 0x557702654bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:10.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0e96ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264ab60 0x557702654bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42242/0 (socket says 172.21.15.49:42242) 2024-03-18T00:54:10.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0e96ffd640 1 -- 172.21.15.49:0/1466948680 learned_addr learned my addr 172.21.15.49:0/1466948680 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:10.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.607+0000 7f0e977fe640 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:10.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.608+0000 7f0e96ffd640 1 -- 172.21.15.49:0/1466948680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264fcd0 msgr2=0x55770265bc30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:10.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.608+0000 7f0e96ffd640 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264fcd0 0x55770265bc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.608+0000 7f0e96ffd640 1 -- 172.21.15.49:0/1466948680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 msgr2=0x5577026546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:10.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.608+0000 7f0e96ffd640 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026546b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:10.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.608+0000 7f0e96ffd640 1 -- 172.21.15.49:0/1466948680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55770238d7d0 con 0x55770264ab60 2024-03-18T00:54:10.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.612+0000 7f0e96ffd640 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264ab60 0x557702654bf0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c02c800 tx=0x7f0e8c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea35c063350007fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:10.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.613+0000 7f0e76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:10.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.618+0000 7f0e76ffd640 1 -- 172.21.15.49:0/1466948680 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e8c005d50 con 0x55770264ab60 2024-03-18T00:54:10.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.619+0000 7f0e76ffd640 1 -- 172.21.15.49:0/1466948680 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0e8c005240 con 0x55770264ab60 2024-03-18T00:54:10.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.623+0000 7f0ea0827740 1 -- 172.21.15.49:0/1466948680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e8800f8a0 con 0x55770264ab60 2024-03-18T00:54:10.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.625+0000 7f0e76ffd640 1 -- 172.21.15.49:0/1466948680 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e8c005460 con 0x55770264ab60 2024-03-18T00:54:10.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.626+0000 7f0ea0827740 10 client.16377 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:10.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.626+0000 7f0ea0827740 1 -- 172.21.15.49:0/1466948680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5577022c33f0 con 0x55770264ab60 2024-03-18T00:54:10.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.629+0000 7f0e76ffd640 1 -- 172.21.15.49:0/1466948680 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e8800f8a0 con 0x55770264ab60 2024-03-18T00:54:10.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.630+0000 7f0e76ffd640 1 -- 172.21.15.49:0/1466948680 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0e8c0038f0 con 0x55770264ab60 2024-03-18T00:54:10.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.630+0000 7f0ea0827740 10 client.16377 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:10.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.630+0000 7f0ea0827740 10 client.16377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:10.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.630+0000 7f0ea0827740 20 client.16377 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:10.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.630+0000 7f0ea0827740 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5577026601d0 0x557702662650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:10.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.630+0000 7f0ea0827740 4 client.16377 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:10.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.630+0000 7f0ea0827740 1 -- 172.21.15.49:0/1466948680 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5577025bca40 con 0x5577026601d0 2024-03-18T00:54:10.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.631+0000 7f0e977fe640 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5577026601d0 0x557702662650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:10.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.633+0000 7f0e977fe640 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5577026601d0 0x557702662650 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d9dd0ed318901b8 server_cookie=410b14d478ddca98 in_seq=0 out_seq=0 2024-03-18T00:54:10.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:10.633+0000 7f0e76ffd640 10 client.16377 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:11 smithi107 ceph-mon[36270]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 988 MiB/s wr, 1.90k op/s 2024-03-18T00:54:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:11 smithi106 ceph-mon[35899]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 988 MiB/s wr, 1.90k op/s 2024-03-18T00:54:11.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:11 smithi049 ceph-mon[32130]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 988 MiB/s wr, 1.90k op/s 2024-03-18T00:54:13.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:13 smithi107 ceph-mon[36270]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 814 MiB/s wr, 1.61k op/s 2024-03-18T00:54:13.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:13 smithi106 ceph-mon[35899]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 814 MiB/s wr, 1.61k op/s 2024-03-18T00:54:13.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:13 smithi049 ceph-mon[32130]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 814 MiB/s wr, 1.61k op/s 2024-03-18T00:54:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.054+0000 7fa20dffb640 1 -- 172.21.15.49:0/309777056 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556523eb72d0 con 0x556523fc7690 2024-03-18T00:54:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.054+0000 7fa20dffb640 10 client.16365 handle_command_reply: tid=0 2024-03-18T00:54:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.054+0000 7fa22f1a0740 1 client.16365 shutdown 2024-03-18T00:54:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f59f80 msgr2=0x556523f73420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.060 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f59f80 0x556523f73420 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa21400dd90 tx=0x7fa214004700 comp rx=0 tx=0).stop 2024-03-18T00:54:14.061 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 shutdown_connections 2024-03-18T00:54:14.062 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556523fc7690 0x556523fc9b50 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.062 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556523f50060 0x556523f72410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.062 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556523f59f80 0x556523f73420 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 --2- 172.21.15.49:0/309777056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556523f4a500 0x556523f71ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 >> 172.21.15.49:0/309777056 conn(0x556523f45a70 msgr2=0x556523f4c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa20dffb640 0 client.16365 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 shutdown_connections 2024-03-18T00:54:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 1 -- 172.21.15.49:0/309777056 wait complete. 2024-03-18T00:54:14.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.055+0000 7fa22f1a0740 20 client.16365 trim_cache size 0 max 16384 2024-03-18T00:54:14.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.056+0000 7f0e76ffd640 1 -- 172.21.15.49:0/1466948680 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5577025bca40 con 0x5577026601d0 2024-03-18T00:54:14.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.056+0000 7f0e76ffd640 10 client.16377 handle_command_reply: tid=0 2024-03-18T00:54:14.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.056+0000 7f0ea0827740 1 client.16377 shutdown 2024-03-18T00:54:14.065 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:54:14.065 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:54:14.065 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:54:14.065 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:54:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.059+0000 7fa22b27a640 1 -- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa2100a28b0 msgr2=0x7fa2100a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.059+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa2100a28b0 0x7fa2100a4d70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x556523f4b750 tx=0x7fa214008040 comp rx=0 tx=0).stop 2024-03-18T00:54:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.059+0000 7fa22b27a640 1 -- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2240733f0 msgr2=0x7fa224074330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.060+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2240733f0 0x7fa224074330 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa22000b470 tx=0x7fa22000b930 comp rx=0 tx=0).stop 2024-03-18T00:54:14.067 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.060+0000 7fa22b27a640 1 -- 172.21.15.49:0/3114623226 shutdown_connections 2024-03-18T00:54:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.060+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa2100a28b0 0x7fa2100a4d70 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.060+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa2241094d0 0x7fa22407b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.060+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa2240733f0 0x7fa224074330 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.060+0000 7fa22b27a640 1 --2- 172.21.15.49:0/3114623226 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa224072a90 0x7fa224073df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.060+0000 7fa22b27a640 1 -- 172.21.15.49:0/3114623226 >> 172.21.15.49:0/3114623226 conn(0x7fa22406e650 msgr2=0x7fa22410c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:14.068 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.061+0000 7fa22b27a640 1 -- 172.21.15.49:0/3114623226 shutdown_connections 2024-03-18T00:54:14.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.061+0000 7f0ea0827740 1 -- 172.21.15.49:0/1466948680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264ab60 msgr2=0x557702654bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.061+0000 7f0ea0827740 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264ab60 0x557702654bf0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c02c800 tx=0x7f0e8c004270 comp rx=0 tx=0).stop 2024-03-18T00:54:14.069 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:14.062+0000 7fa22b27a640 1 -- 172.21.15.49:0/3114623226 wait complete. 2024-03-18T00:54:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.061+0000 7f0ea0827740 1 -- 172.21.15.49:0/1466948680 shutdown_connections 2024-03-18T00:54:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.061+0000 7f0ea0827740 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5577026601d0 0x557702662650 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.061+0000 7f0ea0827740 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264fcd0 0x55770265bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.061+0000 7f0ea0827740 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264ab60 0x557702654bf0 secure :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0e8c02c800 tx=0x7f0e8c004270 comp rx=0 tx=0).stop 2024-03-18T00:54:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.061+0000 7f0ea0827740 1 --2- 172.21.15.49:0/1466948680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.061+0000 7f0ea0827740 1 -- 172.21.15.49:0/1466948680 >> 172.21.15.49:0/1466948680 conn(0x5577026456e0 msgr2=0x557702644900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.061+0000 7f0e76ffd640 0 client.16377 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.062+0000 7f0ea0827740 1 -- 172.21.15.49:0/1466948680 shutdown_connections 2024-03-18T00:54:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.062+0000 7f0ea0827740 1 -- 172.21.15.49:0/1466948680 wait complete. 2024-03-18T00:54:14.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.062+0000 7f0ea0827740 20 client.16377 trim_cache size 0 max 16384 2024-03-18T00:54:14.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.115+0000 7f0ea0827740 1 Processor -- start 2024-03-18T00:54:14.117 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.116+0000 7f0ea0827740 1 -- start start 2024-03-18T00:54:14.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.116+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026540d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.117+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.117+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557702644070 0x5577026444b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.117+0000 7f0ea0827740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8400e640 con 0x55770264fcd0 2024-03-18T00:54:14.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.117+0000 7f0e977fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026540d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.117+0000 7f0e977fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026540d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59520/0 (socket says 172.21.15.49:59520) 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.117+0000 7f0e977fe640 1 -- 172.21.15.49:0/2299367754 learned_addr learned my addr 172.21.15.49:0/2299367754 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.117+0000 7f0ea0827740 1 -- 172.21.15.49:0/2299367754 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0e84010180 con 0x55770264a170 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.117+0000 7f0ea0827740 1 -- 172.21.15.49:0/2299367754 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8400ed10 con 0x557702644070 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.118+0000 7f0e96ffd640 1 --2- 172.21.15.49:0/2299367754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.118+0000 7f0e97fff640 1 --2- 172.21.15.49:0/2299367754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557702644070 0x5577026444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.118+0000 7f0e977fe640 1 -- 172.21.15.49:0/2299367754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557702644070 msgr2=0x5577026444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.118+0000 7f0e977fe640 1 --2- 172.21.15.49:0/2299367754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557702644070 0x5577026444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.118+0000 7f0e977fe640 1 -- 172.21.15.49:0/2299367754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 msgr2=0x55770265bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.118+0000 7f0e977fe640 1 --2- 172.21.15.49:0/2299367754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.119+0000 7f0e977fe640 1 -- 172.21.15.49:0/2299367754 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577022830f0 con 0x55770264a170 2024-03-18T00:54:14.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.119+0000 7f0e977fe640 1 --2- 172.21.15.49:0/2299367754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026540d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0e880058c0 tx=0x7f0e8801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a63c0ec8c91235c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:14.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.121+0000 7f0e7d7fa640 1 -- 172.21.15.49:0/2299367754 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88023040 con 0x55770264a170 2024-03-18T00:54:14.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.124+0000 7f0e7d7fa640 1 -- 172.21.15.49:0/2299367754 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e88036030 con 0x55770264a170 2024-03-18T00:54:14.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.126+0000 7f0ea0827740 1 -- 172.21.15.49:0/2299367754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 msgr2=0x5577026540d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.126+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2299367754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026540d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0e880058c0 tx=0x7f0e8801a000 comp rx=0 tx=0).stop 2024-03-18T00:54:14.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.130+0000 7f0ea0827740 1 -- 172.21.15.49:0/2299367754 shutdown_connections 2024-03-18T00:54:14.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.130+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2299367754 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557702644070 0x5577026444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.131+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2299367754 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.131+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2299367754 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55770264a170 0x5577026540d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.131+0000 7f0ea0827740 1 -- 172.21.15.49:0/2299367754 >> 172.21.15.49:0/2299367754 conn(0x5577026456e0 msgr2=0x5577026539c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:14.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.131+0000 7f0ea0827740 1 -- 172.21.15.49:0/2299367754 shutdown_connections 2024-03-18T00:54:14.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.131+0000 7f0ea0827740 1 -- 172.21.15.49:0/2299367754 wait complete. 2024-03-18T00:54:14.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.131+0000 7f0ea0827740 1 -- start start 2024-03-18T00:54:14.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.132+0000 7f0ea0827740 1 Processor -- start 2024-03-18T00:54:14.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.132+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557702644070 0x5577026597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.132+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264a170 0x557702659d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.141 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.132+0000 7f0ea0827740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.132+0000 7f0ea0827740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8400fab0 con 0x55770264fcd0 2024-03-18T00:54:14.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.132+0000 7f0ea0827740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8400f3e0 con 0x557702644070 2024-03-18T00:54:14.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.132+0000 7f0ea0827740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0e8400c620 con 0x55770264a170 2024-03-18T00:54:14.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.132+0000 7f0ea0827740 10 client.0 _ll_register_callbacks cb 0x7f0e8c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:14.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.133+0000 7f0e97fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.133+0000 7f0e97fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42256/0 (socket says 172.21.15.49:42256) 2024-03-18T00:54:14.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.133+0000 7f0e97fff640 1 -- 172.21.15.49:0/2731513453 learned_addr learned my addr 172.21.15.49:0/2731513453 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:14.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.133+0000 7f0e96ffd640 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264a170 0x557702659d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.134+0000 7f0e97fff640 1 -- 172.21.15.49:0/2731513453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264a170 msgr2=0x557702659d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:14.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.134+0000 7f0e97fff640 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264a170 0x557702659d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.134+0000 7f0e97fff640 1 -- 172.21.15.49:0/2731513453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557702644070 msgr2=0x5577026597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:14.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.134+0000 7f0e97fff640 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557702644070 0x5577026597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.134+0000 7f0e97fff640 1 -- 172.21.15.49:0/2731513453 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577025769d0 con 0x55770264fcd0 2024-03-18T00:54:14.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.134+0000 7f0e97fff640 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265ad30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0e90097cf0 tx=0x7f0e9009ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727f89c35c5e9ab4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:14.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.135+0000 7f0e76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.135+0000 7f0e76ffd640 1 -- 172.21.15.49:0/2731513453 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e9001d070 con 0x55770264fcd0 2024-03-18T00:54:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.135+0000 7f0ea0827740 1 -- 172.21.15.49:0/2731513453 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e88036030 con 0x55770264fcd0 2024-03-18T00:54:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.135+0000 7f0e76ffd640 1 -- 172.21.15.49:0/2731513453 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0e900cc590 con 0x55770264fcd0 2024-03-18T00:54:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.136+0000 7f0e76ffd640 1 -- 172.21.15.49:0/2731513453 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0e9009f470 con 0x55770264fcd0 2024-03-18T00:54:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.136+0000 7f0ea0827740 10 client.16383 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.136+0000 7f0ea0827740 1 -- 172.21.15.49:0/2731513453 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5577026c6900 con 0x55770264fcd0 2024-03-18T00:54:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.136+0000 7f0e76ffd640 1 -- 172.21.15.49:0/2731513453 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e88036030 con 0x55770264fcd0 2024-03-18T00:54:14.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.138+0000 7f0e76ffd640 1 -- 172.21.15.49:0/2731513453 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0e900a36c0 con 0x55770264fcd0 2024-03-18T00:54:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.138+0000 7f0ea0827740 10 client.16383 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.138+0000 7f0ea0827740 10 client.16383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.138+0000 7f0ea0827740 20 client.16383 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.138+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5577026c7280 0x5577026c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.138+0000 7f0ea0827740 4 client.16383 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:54:14.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.138+0000 7f0ea0827740 1 -- 172.21.15.49:0/2731513453 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5577025b6d90 con 0x5577026c7280 2024-03-18T00:54:14.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.143+0000 7f0e977fe640 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5577026c7280 0x5577026c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.155+0000 7f0e977fe640 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5577026c7280 0x5577026c96a0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd3e48f0b3578a09 server_cookie=65ddd3c2f57db01a in_seq=0 out_seq=0 2024-03-18T00:54:14.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.155+0000 7f0e76ffd640 10 client.16383 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:14.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.158+0000 7f0e76ffd640 1 -- 172.21.15.49:0/2731513453 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5577025b6d90 con 0x5577026c7280 2024-03-18T00:54:14.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.158+0000 7f0e76ffd640 10 client.16383 handle_command_reply: tid=0 2024-03-18T00:54:14.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.158+0000 7f0ea0827740 1 client.16383 shutdown 2024-03-18T00:54:14.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0ea0827740 1 -- 172.21.15.49:0/2731513453 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 msgr2=0x55770265ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265ad30 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0e90097cf0 tx=0x7f0e9009ba20 comp rx=0 tx=0).stop 2024-03-18T00:54:14.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0ea0827740 1 -- 172.21.15.49:0/2731513453 shutdown_connections 2024-03-18T00:54:14.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5577026c7280 0x5577026c96a0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55770264fcd0 0x55770265ad30 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55770264a170 0x557702659d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0ea0827740 1 --2- 172.21.15.49:0/2731513453 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557702644070 0x5577026597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0ea0827740 1 -- 172.21.15.49:0/2731513453 >> 172.21.15.49:0/2731513453 conn(0x5577026456e0 msgr2=0x5577026491d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:14.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0e76ffd640 0 client.16383 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:14.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.159+0000 7f0ea0827740 1 -- 172.21.15.49:0/2731513453 shutdown_connections 2024-03-18T00:54:14.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.163+0000 7f0ea0827740 1 -- 172.21.15.49:0/2731513453 wait complete. 2024-03-18T00:54:14.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.163+0000 7f0ea0827740 20 client.16383 trim_cache size 0 max 16384 2024-03-18T00:54:14.167 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:54:14.167 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:54:14.167 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:54:14.168 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:54:14.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.167+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e800a29d0 msgr2=0x7f0e800a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.167+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e800a29d0 0x7f0e800a4e90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0e88020000 tx=0x7f0e88015000 comp rx=0 tx=0).stop 2024-03-18T00:54:14.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.167+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e981a4150 msgr2=0x7f0e98109ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.167+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e981a4150 0x7f0e98109ff0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0e9000cce0 tx=0x7f0e9000a620 comp rx=0 tx=0).stop 2024-03-18T00:54:14.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.168+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2590288426 shutdown_connections 2024-03-18T00:54:14.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.168+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0e800a29d0 0x7f0e800a4e90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.168+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0e98110060 0x7f0e98109ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.168+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0e981a4150 0x7f0e98109ff0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.168+0000 7f0e9dc36640 1 --2- 172.21.15.49:0/2590288426 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0e981087e0 0x7f0e9810b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.168+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2590288426 >> 172.21.15.49:0/2590288426 conn(0x7f0e9806e640 msgr2=0x7f0e98070390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:14.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.168+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2590288426 shutdown_connections 2024-03-18T00:54:14.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.168+0000 7f0e9dc36640 1 -- 172.21.15.49:0/2590288426 wait complete. 2024-03-18T00:54:14.198 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:54:14.198 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:54:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.567+0000 7f462091d640 1 -- 172.21.15.49:0/1886612920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c109930 msgr2=0x7f461c110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.567+0000 7f461959a640 1 -- 172.21.15.49:0/1886612920 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461402ca90 con 0x7f461c109930 2024-03-18T00:54:14.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.567+0000 7f462091d640 1 --2- 172.21.15.49:0/1886612920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c109930 0x7f461c110250 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4614009be0 tx=0x7f461402bef0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.570+0000 7f462091d640 1 -- 172.21.15.49:0/1886612920 shutdown_connections 2024-03-18T00:54:14.571 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.570+0000 7f462091d640 1 --2- 172.21.15.49:0/1886612920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c109930 0x7f461c110250 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.570+0000 7f462091d640 1 --2- 172.21.15.49:0/1886612920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f461c108f70 0x7f461c1093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.570+0000 7f462091d640 1 --2- 172.21.15.49:0/1886612920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f461c107e20 0x7f461c108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.570+0000 7f462091d640 1 -- 172.21.15.49:0/1886612920 >> 172.21.15.49:0/1886612920 conn(0x7f461c076db0 msgr2=0x7f461c0791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:14.578 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.575+0000 7f462091d640 1 -- 172.21.15.49:0/1886612920 shutdown_connections 2024-03-18T00:54:14.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.578+0000 7f462091d640 1 -- 172.21.15.49:0/1886612920 wait complete. 2024-03-18T00:54:14.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.579+0000 7f462091d640 1 Processor -- start 2024-03-18T00:54:14.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.582+0000 7f462091d640 1 -- start start 2024-03-18T00:54:14.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.583+0000 7f462091d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f461c107e20 0x7f461c071150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.583+0000 7f462091d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f461c108f70 0x7f461c071690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.583 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.583+0000 7f462091d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c109930 0x7f461c072680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.583+0000 7f462091d640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f461c06b810 con 0x7f461c107e20 2024-03-18T00:54:14.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.583+0000 7f462091d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f461c06b690 con 0x7f461c108f70 2024-03-18T00:54:14.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.583+0000 7f462091d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f461c06b990 con 0x7f461c109930 2024-03-18T00:54:14.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.584+0000 7f461a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f461c107e20 0x7f461c071150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.584+0000 7f461a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f461c107e20 0x7f461c071150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42276/0 (socket says 172.21.15.49:42276) 2024-03-18T00:54:14.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.584+0000 7f461a59c640 1 -- 172.21.15.49:0/2701324832 learned_addr learned my addr 172.21.15.49:0/2701324832 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:14.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.585+0000 7f461ad9d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c109930 0x7f461c072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.587+0000 7f4619d9b640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f461c108f70 0x7f461c071690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.591+0000 7f461ad9d640 1 -- 172.21.15.49:0/2701324832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f461c108f70 msgr2=0x7f461c071690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.591+0000 7f461ad9d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f461c108f70 0x7f461c071690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.591+0000 7f461ad9d640 1 -- 172.21.15.49:0/2701324832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f461c107e20 msgr2=0x7f461c071150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:14.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.591+0000 7f461ad9d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f461c107e20 0x7f461c071150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:14.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.591+0000 7f461ad9d640 1 -- 172.21.15.49:0/2701324832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c072cf0 con 0x7f461c109930 2024-03-18T00:54:14.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.600+0000 7f461ad9d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c109930 0x7f461c072680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f46140334c0 tx=0x7f4614005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:14.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.603+0000 7f460b7fe640 1 -- 172.21.15.49:0/2701324832 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614043020 con 0x7f461c109930 2024-03-18T00:54:14.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.603+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c072f80 con 0x7f461c109930 2024-03-18T00:54:14.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.604+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c073520 con 0x7f461c109930 2024-03-18T00:54:14.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.604+0000 7f460b7fe640 1 -- 172.21.15.49:0/2701324832 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4614033cf0 con 0x7f461c109930 2024-03-18T00:54:14.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.604+0000 7f460b7fe640 1 -- 172.21.15.49:0/2701324832 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614043020 con 0x7f461c109930 2024-03-18T00:54:14.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.615+0000 7f460b7fe640 1 -- 172.21.15.49:0/2701324832 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4614046020 con 0x7f461c109930 2024-03-18T00:54:14.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.615+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45fc0052c0 con 0x7f461c109930 2024-03-18T00:54:14.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.617+0000 7f460b7fe640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46040a2cc0 0x7f46040a5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:14.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.617+0000 7f460b7fe640 1 -- 172.21.15.49:0/2701324832 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f461403a040 con 0x7f461c109930 2024-03-18T00:54:14.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.625+0000 7f461a59c640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46040a2cc0 0x7f46040a5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:14.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.626+0000 7f461a59c640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46040a2cc0 0x7f46040a5180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4610009c90 tx=0x7f4610005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:14.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:14.631+0000 7f460b7fe640 1 -- 172.21.15.49:0/2701324832 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f461404d070 con 0x7f461c109930 2024-03-18T00:54:15.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.447+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45fc0055b0 con 0x7f461c109930 2024-03-18T00:54:15.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.454+0000 7f460b7fe640 1 -- 172.21.15.49:0/2701324832 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f46140bd3e0 con 0x7f461c109930 2024-03-18T00:54:15.459 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:54:15.459 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:53:25.016081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24470,24463,24481,24506,24499]},"id":1}]} 2024-03-18T00:54:15.459 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 27 2024-03-18T00:54:15.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.469+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46040a2cc0 msgr2=0x7f46040a5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:15.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.469+0000 7f462091d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46040a2cc0 0x7f46040a5180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4610009c90 tx=0x7f4610005f90 comp rx=0 tx=0).stop 2024-03-18T00:54:15.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.469+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c109930 msgr2=0x7f461c072680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:15.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.470+0000 7f462091d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c109930 0x7f461c072680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f46140334c0 tx=0x7f4614005bd0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.470+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 shutdown_connections 2024-03-18T00:54:15.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.470+0000 7f462091d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46040a2cc0 0x7f46040a5180 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4610009c90 tx=0x7f4610005f90 comp rx=0 tx=0).stop 2024-03-18T00:54:15.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.470+0000 7f462091d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f461c108f70 0x7f461c071690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.470+0000 7f462091d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f461c109930 0x7f461c072680 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.470+0000 7f462091d640 1 --2- 172.21.15.49:0/2701324832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f461c107e20 0x7f461c071150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.470+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 >> 172.21.15.49:0/2701324832 conn(0x7f461c076db0 msgr2=0x7f461c10c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:15.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.477+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 shutdown_connections 2024-03-18T00:54:15.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.481+0000 7f462091d640 1 -- 172.21.15.49:0/2701324832 wait complete. 2024-03-18T00:54:15.534 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:54:15.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:15 smithi106 ceph-mon[35899]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1014 MiB/s wr, 1.84k op/s 2024-03-18T00:54:15.880 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:15 smithi049 ceph-mon[32130]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1014 MiB/s wr, 1.84k op/s 2024-03-18T00:54:15.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:15 smithi107 ceph-mon[36270]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1014 MiB/s wr, 1.84k op/s 2024-03-18T00:54:15.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.929+0000 7fc3fb84b640 1 -- 172.21.15.49:0/1428084297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f410db00 msgr2=0x7fc3f410df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:15.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.929+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/1428084297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f410db00 0x7fc3f410df80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009900 tx=0x7fc3f002bc50 comp rx=0 tx=0).stop 2024-03-18T00:54:15.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.930+0000 7fc3fb84b640 1 -- 172.21.15.49:0/1428084297 shutdown_connections 2024-03-18T00:54:15.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.930+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/1428084297 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f410db00 0x7fc3f410df80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.930+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/1428084297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f410b150 0x7fc3f410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.930+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/1428084297 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3f41087e0 0x7fc3f410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.930+0000 7fc3fb84b640 1 -- 172.21.15.49:0/1428084297 >> 172.21.15.49:0/1428084297 conn(0x7fc3f406e640 msgr2=0x7fc3f4070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:15.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.930+0000 7fc3fb84b640 1 -- 172.21.15.49:0/1428084297 shutdown_connections 2024-03-18T00:54:15.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.930+0000 7fc3fb84b640 1 -- 172.21.15.49:0/1428084297 wait complete. 2024-03-18T00:54:15.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.932+0000 7fc3fb84b640 1 Processor -- start 2024-03-18T00:54:15.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.932+0000 7fc3fb84b640 1 -- start start 2024-03-18T00:54:15.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.933+0000 7fc3fb84b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f41087e0 0x7fc3f41a6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:15.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.933+0000 7fc3fb84b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3f410b150 0x7fc3f41a6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:15.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.933+0000 7fc3fb84b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f410db00 0x7fc3f4079a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:15.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.933+0000 7fc3fb84b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f411c2e0 con 0x7fc3f410b150 2024-03-18T00:54:15.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.933+0000 7fc3fb84b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f411c160 con 0x7fc3f41087e0 2024-03-18T00:54:15.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.933+0000 7fc3fb84b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f411c460 con 0x7fc3f410db00 2024-03-18T00:54:15.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.934+0000 7fc3f95e7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f41087e0 0x7fc3f41a6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:15.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.935+0000 7fc3f95e7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f41087e0 0x7fc3f41a6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59556/0 (socket says 172.21.15.49:59556) 2024-03-18T00:54:15.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.935+0000 7fc3f95e7640 1 -- 172.21.15.49:0/2093837615 learned_addr learned my addr 172.21.15.49:0/2093837615 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:15.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.937+0000 7fc3f9de8640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f410db00 0x7fc3f4079a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:15.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.937+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3f410b150 0x7fc3f41a6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:15.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.938+0000 7fc3f95e7640 1 -- 172.21.15.49:0/2093837615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f410db00 msgr2=0x7fc3f4079a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:15.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.939+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f410db00 0x7fc3f4079a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.939+0000 7fc3f95e7640 1 -- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3f410b150 msgr2=0x7fc3f41a6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:15.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.939+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3f410b150 0x7fc3f41a6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.939+0000 7fc3f95e7640 1 -- 172.21.15.49:0/2093837615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f407a0e0 con 0x7fc3f41087e0 2024-03-18T00:54:15.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.939+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3f410b150 0x7fc3f41a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:54:15.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.942+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f41087e0 0x7fc3f41a6850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400e9f0 tx=0x7fc3e400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:15.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.944+0000 7fc3ea7fc640 1 -- 172.21.15.49:0/2093837615 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e400cd80 con 0x7fc3f41087e0 2024-03-18T00:54:15.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.944+0000 7fc3ea7fc640 1 -- 172.21.15.49:0/2093837615 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3e4010910 con 0x7fc3f41087e0 2024-03-18T00:54:15.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.944+0000 7fc3ea7fc640 1 -- 172.21.15.49:0/2093837615 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4018990 con 0x7fc3f41087e0 2024-03-18T00:54:15.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.944+0000 7fc3fb84b640 1 -- 172.21.15.49:0/2093837615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f407a3d0 con 0x7fc3f41087e0 2024-03-18T00:54:15.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.944+0000 7fc3fb84b640 1 -- 172.21.15.49:0/2093837615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f407a8d0 con 0x7fc3f41087e0 2024-03-18T00:54:15.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.949+0000 7fc3fe432740 1 Processor -- start 2024-03-18T00:54:15.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.951+0000 7fc3fe432740 1 -- start start 2024-03-18T00:54:15.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.951+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3b850 0x557c0ce3bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:15.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.951+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3c240 0x557c0ce40d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:15.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.951+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce43840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:15.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.951+0000 7fc3fe432740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557c0cda0490 con 0x557c0ce3b850 2024-03-18T00:54:15.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.951+0000 7fc3fe432740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557c0cd48c90 con 0x557c0ce413b0 2024-03-18T00:54:15.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.951+0000 7fc3fe432740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557c0cd9c530 con 0x557c0ce3c240 2024-03-18T00:54:15.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.953+0000 7fc3f9de8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3b850 0x557c0ce3bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:15.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.953+0000 7fc3f9de8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3b850 0x557c0ce3bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42294/0 (socket says 172.21.15.49:42294) 2024-03-18T00:54:15.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.953+0000 7fc3f9de8640 1 -- 172.21.15.49:0/1323613257 learned_addr learned my addr 172.21.15.49:0/1323613257 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:15.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.955+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/1323613257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3c240 0x557c0ce40d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:15.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.956+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/1323613257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce43840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:15.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.957+0000 7fc3f8de6640 1 -- 172.21.15.49:0/1323613257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3c240 msgr2=0x557c0ce40d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:15.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.957+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/1323613257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3c240 0x557c0ce40d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.957+0000 7fc3f8de6640 1 -- 172.21.15.49:0/1323613257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3b850 msgr2=0x557c0ce3bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:15.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.957+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/1323613257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3b850 0x557c0ce3bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.957+0000 7fc3f8de6640 1 -- 172.21.15.49:0/1323613257 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c0ce03500 con 0x557c0ce413b0 2024-03-18T00:54:15.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.962+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/1323613257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce43840 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc3f41a7920 tx=0x7fc3ec011c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbb66a7a1b3a44a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:15.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.963+0000 7fc3df7fe640 1 -- 172.21.15.49:0/1323613257 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec004d20 con 0x557c0ce413b0 2024-03-18T00:54:15.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.964+0000 7fc3df7fe640 1 -- 172.21.15.49:0/1323613257 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3ec00f970 con 0x557c0ce413b0 2024-03-18T00:54:15.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.964+0000 7fc3df7fe640 1 -- 172.21.15.49:0/1323613257 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec0044c0 con 0x557c0ce413b0 2024-03-18T00:54:15.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.967+0000 7fc3fe432740 1 -- 172.21.15.49:0/1323613257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 msgr2=0x557c0ce43840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:15.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.967+0000 7fc3fe432740 1 --2- 172.21.15.49:0/1323613257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce43840 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc3f41a7920 tx=0x7fc3ec011c70 comp rx=0 tx=0).stop 2024-03-18T00:54:15.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.972+0000 7fc3fe432740 1 -- 172.21.15.49:0/1323613257 shutdown_connections 2024-03-18T00:54:15.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.972+0000 7fc3fe432740 1 --2- 172.21.15.49:0/1323613257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce43840 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.972+0000 7fc3fe432740 1 --2- 172.21.15.49:0/1323613257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3c240 0x557c0ce40d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.972+0000 7fc3fe432740 1 --2- 172.21.15.49:0/1323613257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3b850 0x557c0ce3bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.972+0000 7fc3fe432740 1 -- 172.21.15.49:0/1323613257 >> 172.21.15.49:0/1323613257 conn(0x557c0ce36dc0 msgr2=0x557c0ce371c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:15.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.978+0000 7fc3fe432740 1 -- 172.21.15.49:0/1323613257 shutdown_connections 2024-03-18T00:54:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.980+0000 7fc3fe432740 1 -- 172.21.15.49:0/1323613257 wait complete. 2024-03-18T00:54:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.981+0000 7fc3fe432740 1 -- start start 2024-03-18T00:54:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.982+0000 7fc3fe432740 1 Processor -- start 2024-03-18T00:54:15.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.983+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce3b850 0x557c0ce45be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:15.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.983+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3c240 0x557c0ce46120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:15.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.983+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce413b0 0x557c0ce4d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:15.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.983+0000 7fc3fe432740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557c0ce44360 con 0x557c0ce413b0 2024-03-18T00:54:15.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.983+0000 7fc3fe432740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557c0cd5c230 con 0x557c0ce3b850 2024-03-18T00:54:15.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.983+0000 7fc3fe432740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557c0ce444e0 con 0x557c0ce3c240 2024-03-18T00:54:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.983+0000 7fc3fe432740 10 client.0 _ll_register_callbacks cb 0x557c0cd98010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.985+0000 7fc3f8de6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce413b0 0x557c0ce4d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.985+0000 7fc3f8de6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce413b0 0x557c0ce4d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42304/0 (socket says 172.21.15.49:42304) 2024-03-18T00:54:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.985+0000 7fc3f8de6640 1 -- 172.21.15.49:0/537485862 learned_addr learned my addr 172.21.15.49:0/537485862 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.989+0000 7fc3f8de6640 1 -- 172.21.15.49:0/537485862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3c240 msgr2=0x557c0ce46120 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:15.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.989+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3c240 0x557c0ce46120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.989+0000 7fc3f8de6640 1 -- 172.21.15.49:0/537485862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce3b850 msgr2=0x557c0ce45be0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:15.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.989+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce3b850 0x557c0ce45be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:15.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:15.989+0000 7fc3f8de6640 1 -- 172.21.15.49:0/537485862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c0cd6b100 con 0x557c0ce413b0 2024-03-18T00:54:16.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.000+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce413b0 0x557c0ce4d160 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec003c90 tx=0x7fc3ec003cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b43e0815d82f09e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.002+0000 7fc3dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:16.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.005+0000 7fc3dcff9640 1 -- 172.21.15.49:0/537485862 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec028020 con 0x557c0ce413b0 2024-03-18T00:54:16.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.005+0000 7fc3dcff9640 1 -- 172.21.15.49:0/537485862 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc3ec0041c0 con 0x557c0ce413b0 2024-03-18T00:54:16.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.005+0000 7fc3fe432740 1 -- 172.21.15.49:0/537485862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3ec00f970 con 0x557c0ce413b0 2024-03-18T00:54:16.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.008+0000 7fc3dcff9640 1 -- 172.21.15.49:0/537485862 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc3ec033050 con 0x557c0ce413b0 2024-03-18T00:54:16.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.014+0000 7fc3fe432740 10 client.16401 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:16.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.014+0000 7fc3fe432740 1 -- 172.21.15.49:0/537485862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557c0cab43f0 con 0x557c0ce413b0 2024-03-18T00:54:16.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.025+0000 7fc3dcff9640 1 -- 172.21.15.49:0/537485862 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3ec00f970 con 0x557c0ce413b0 2024-03-18T00:54:16.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.025+0000 7fc3dcff9640 1 -- 172.21.15.49:0/537485862 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc3ec018070 con 0x557c0ce413b0 2024-03-18T00:54:16.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.025+0000 7fc3fe432740 10 client.16401 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:16.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.025+0000 7fc3fe432740 10 client.16401 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:54:16.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.025+0000 7fc3fe432740 20 client.16401 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:16.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.025+0000 7fc3fe432740 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x557c0ce62850 0x557c0ce64cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.025+0000 7fc3fe432740 4 client.16401 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:16.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.025+0000 7fc3fe432740 1 -- 172.21.15.49:0/537485862 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c0cdad9d0 con 0x557c0ce62850 2024-03-18T00:54:16.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.033+0000 7fc3f9de8640 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x557c0ce62850 0x557c0ce64cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.050+0000 7fc3f9de8640 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x557c0ce62850 0x557c0ce64cc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb768db3facb7339 server_cookie=fdc244b3c353ce76 in_seq=0 out_seq=0 2024-03-18T00:54:16.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.051+0000 7fc3dcff9640 10 client.16401 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:54:16.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.055+0000 7fc3dcff9640 1 -- 172.21.15.49:0/537485862 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c0cdad9d0 con 0x557c0ce62850 2024-03-18T00:54:16.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.055+0000 7fc3dcff9640 10 client.16401 handle_command_reply: tid=0 2024-03-18T00:54:16.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.056+0000 7fc3fe432740 1 client.16401 shutdown 2024-03-18T00:54:16.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.056+0000 7fc3fe432740 1 -- 172.21.15.49:0/537485862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce413b0 msgr2=0x557c0ce4d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.056+0000 7fc3fe432740 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce413b0 0x557c0ce4d160 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec003c90 tx=0x7fc3ec003cc0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.060+0000 7fc3fe432740 1 -- 172.21.15.49:0/537485862 shutdown_connections 2024-03-18T00:54:16.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.060+0000 7fc3fe432740 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x557c0ce62850 0x557c0ce64cc0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.060+0000 7fc3fe432740 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce413b0 0x557c0ce4d160 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.060+0000 7fc3fe432740 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3c240 0x557c0ce46120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.060+0000 7fc3fe432740 1 --2- 172.21.15.49:0/537485862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce3b850 0x557c0ce45be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.060+0000 7fc3fe432740 1 -- 172.21.15.49:0/537485862 >> 172.21.15.49:0/537485862 conn(0x557c0ce36dc0 msgr2=0x557c0ce35fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.061+0000 7fc3dcff9640 0 client.16401 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:54:16.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.061+0000 7fc3fe432740 1 -- 172.21.15.49:0/537485862 shutdown_connections 2024-03-18T00:54:16.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.064+0000 7fc3fe432740 1 -- 172.21.15.49:0/537485862 wait complete. 2024-03-18T00:54:16.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.066+0000 7fc3fe432740 20 client.16401 trim_cache size 0 max 16384 2024-03-18T00:54:16.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.125+0000 7fc3fe432740 1 Processor -- start 2024-03-18T00:54:16.128 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.125+0000 7fc3fe432740 1 -- start start 2024-03-18T00:54:16.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.129+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 0x557c0ce3d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.130+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce4d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.130+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce3dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.131+0000 7fc3fe432740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f407b2c0 con 0x557c0ce3c240 2024-03-18T00:54:16.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.131+0000 7fc3fe432740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f407ce00 con 0x557c0ce413b0 2024-03-18T00:54:16.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.131+0000 7fc3fe432740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f407b990 con 0x557c0ce3b850 2024-03-18T00:54:16.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.132+0000 7fc3f8de6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce3dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.132+0000 7fc3f8de6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce3dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59570/0 (socket says 172.21.15.49:59570) 2024-03-18T00:54:16.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.132+0000 7fc3f8de6640 1 -- 172.21.15.49:0/2381378942 learned_addr learned my addr 172.21.15.49:0/2381378942 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:16.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.132+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2381378942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce4d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.133+0000 7fc3f9de8640 1 --2- 172.21.15.49:0/2381378942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 0x557c0ce3d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.133+0000 7fc3f95e7640 1 -- 172.21.15.49:0/2381378942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 msgr2=0x557c0ce3d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.133+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2381378942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 0x557c0ce3d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.133+0000 7fc3f95e7640 1 -- 172.21.15.49:0/2381378942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 msgr2=0x557c0ce3dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.133+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2381378942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce3dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.133+0000 7fc3f95e7640 1 -- 172.21.15.49:0/2381378942 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c0cb7e7d0 con 0x557c0ce3c240 2024-03-18T00:54:16.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.134+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2381378942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce4d120 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc3f407aaa0 tx=0x7fc3e4018f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca22afcc2e25058 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.145+0000 7fc3deffd640 1 -- 172.21.15.49:0/2381378942 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4031070 con 0x557c0ce3c240 2024-03-18T00:54:16.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.145+0000 7fc3deffd640 1 -- 172.21.15.49:0/2381378942 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3e4005510 con 0x557c0ce3c240 2024-03-18T00:54:16.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.145+0000 7fc3deffd640 1 -- 172.21.15.49:0/2381378942 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4026060 con 0x557c0ce3c240 2024-03-18T00:54:16.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.147+0000 7fc3fe432740 1 -- 172.21.15.49:0/2381378942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 msgr2=0x557c0ce4d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.147+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2381378942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce4d120 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc3f407aaa0 tx=0x7fc3e4018f80 comp rx=0 tx=0).stop 2024-03-18T00:54:16.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.155+0000 7fc3fe432740 1 -- 172.21.15.49:0/2381378942 shutdown_connections 2024-03-18T00:54:16.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.155+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2381378942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce3dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.155+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2381378942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce4d120 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.155+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2381378942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 0x557c0ce3d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.155+0000 7fc3fe432740 1 -- 172.21.15.49:0/2381378942 >> 172.21.15.49:0/2381378942 conn(0x557c0ce36dc0 msgr2=0x557c0ce31e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.156+0000 7fc3fe432740 1 -- 172.21.15.49:0/2381378942 shutdown_connections 2024-03-18T00:54:16.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.156+0000 7fc3fe432740 1 -- 172.21.15.49:0/2381378942 wait complete. 2024-03-18T00:54:16.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.156+0000 7fc3fe432740 1 -- start start 2024-03-18T00:54:16.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.157+0000 7fc3fe432740 1 Processor -- start 2024-03-18T00:54:16.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.158+0000 7fc3ea7fc640 1 -- 172.21.15.49:0/2093837615 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc3e4034070 con 0x7fc3f41087e0 2024-03-18T00:54:16.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.160+0000 7fc3ea7fc640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3e00a28d0 0x7fc3e00a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.162+0000 7fc3f9de8640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3e00a28d0 0x7fc3e00a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.164+0000 7fc3f9de8640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3e00a28d0 0x7fc3e00a4d90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x557c0ce65530 tx=0x7fc3f0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.165+0000 7fc3ea7fc640 1 -- 172.21.15.49:0/2093837615 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc3e40d2380 con 0x7fc3f41087e0 2024-03-18T00:54:16.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.170+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 0x557c0ce4d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.170+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce49bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.170+0000 7fc3fe432740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce4a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.171+0000 7fc3fe432740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f407c730 con 0x557c0ce3c240 2024-03-18T00:54:16.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.171+0000 7fc3fe432740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f407c060 con 0x557c0ce413b0 2024-03-18T00:54:16.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.171+0000 7fc3fe432740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f4116d40 con 0x557c0ce3b850 2024-03-18T00:54:16.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.171+0000 7fc3f8de6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce49bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.171+0000 7fc3f8de6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce49bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42318/0 (socket says 172.21.15.49:42318) 2024-03-18T00:54:16.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.171+0000 7fc3f8de6640 1 -- 172.21.15.49:0/2490872374 learned_addr learned my addr 172.21.15.49:0/2490872374 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:16.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.171+0000 7fc3fe432740 10 client.0 _ll_register_callbacks cb 0x7fc3e4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:16.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.173+0000 7fc3f8de6640 1 -- 172.21.15.49:0/2490872374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 msgr2=0x557c0ce4d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.173+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 0x557c0ce4d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.173+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 0x557c0ce4d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.173+0000 7fc3f8de6640 1 -- 172.21.15.49:0/2490872374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 msgr2=0x557c0ce4a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.173+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce4a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.173+0000 7fc3f8de6640 1 -- 172.21.15.49:0/2490872374 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c0cec6bb0 con 0x557c0ce3c240 2024-03-18T00:54:16.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.176+0000 7fc3f8de6640 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce49bc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0057c0 tx=0x7fc3ec024650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c203c9be85582411 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.176+0000 7fc3dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:16.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.180+0000 7fc3dcff9640 1 -- 172.21.15.49:0/2490872374 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec03c060 con 0x557c0ce3c240 2024-03-18T00:54:16.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.180+0000 7fc3dcff9640 1 -- 172.21.15.49:0/2490872374 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc3ec024ba0 con 0x557c0ce3c240 2024-03-18T00:54:16.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.182+0000 7fc3fe432740 1 -- 172.21.15.49:0/2490872374 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3e4005510 con 0x557c0ce3c240 2024-03-18T00:54:16.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.183+0000 7fc3dcff9640 1 -- 172.21.15.49:0/2490872374 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc3ec028070 con 0x557c0ce3c240 2024-03-18T00:54:16.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.183+0000 7fc3fe432740 10 client.16407 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:16.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.183+0000 7fc3fe432740 1 -- 172.21.15.49:0/2490872374 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557c0cec7e40 con 0x557c0ce3c240 2024-03-18T00:54:16.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.186+0000 7fc3dcff9640 1 -- 172.21.15.49:0/2490872374 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3e4005510 con 0x557c0ce3c240 2024-03-18T00:54:16.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.188+0000 7fc3dcff9640 1 -- 172.21.15.49:0/2490872374 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc3ec018040 con 0x557c0ce3c240 2024-03-18T00:54:16.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.188+0000 7fc3fe432740 10 client.16407 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:16.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.188+0000 7fc3fe432740 10 client.16407 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:54:16.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.188+0000 7fc3fe432740 20 client.16407 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:16.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.188+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x557c0cec8710 0x557c0cecab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.188+0000 7fc3fe432740 4 client.16407 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:54:16.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.188+0000 7fc3fe432740 1 -- 172.21.15.49:0/2490872374 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c0cda7d20 con 0x557c0cec8710 2024-03-18T00:54:16.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.190+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x557c0cec8710 0x557c0cecab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.194+0000 7fc3dcff9640 10 client.16407 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:54:16.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.194+0000 7fc3f95e7640 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x557c0cec8710 0x557c0cecab30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1e10ad09b95e8362 server_cookie=50313930de7ff405 in_seq=0 out_seq=0 2024-03-18T00:54:16.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.195+0000 7fc3dcff9640 1 -- 172.21.15.49:0/2490872374 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c0cda7d20 con 0x557c0cec8710 2024-03-18T00:54:16.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.195+0000 7fc3dcff9640 10 client.16407 handle_command_reply: tid=0 2024-03-18T00:54:16.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.195+0000 7fc3fe432740 1 client.16407 shutdown 2024-03-18T00:54:16.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.197+0000 7fc3fe432740 1 -- 172.21.15.49:0/2490872374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 msgr2=0x557c0ce49bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.197+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce49bc0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec0057c0 tx=0x7fc3ec024650 comp rx=0 tx=0).stop 2024-03-18T00:54:16.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.203+0000 7fc3fe432740 1 -- 172.21.15.49:0/2490872374 shutdown_connections 2024-03-18T00:54:16.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.203+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x557c0cec8710 0x557c0cecab30 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.203+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557c0ce413b0 0x557c0ce4a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.203+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557c0ce3c240 0x557c0ce49bc0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.203+0000 7fc3fe432740 1 --2- 172.21.15.49:0/2490872374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557c0ce3b850 0x557c0ce4d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.203+0000 7fc3dcff9640 0 client.16407 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:54:16.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.203+0000 7fc3fe432740 1 -- 172.21.15.49:0/2490872374 >> 172.21.15.49:0/2490872374 conn(0x557c0ce36dc0 msgr2=0x557c0ce3d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.205+0000 7fc3fe432740 1 -- 172.21.15.49:0/2490872374 shutdown_connections 2024-03-18T00:54:16.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.208+0000 7fc3fe432740 1 -- 172.21.15.49:0/2490872374 wait complete. 2024-03-18T00:54:16.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.209+0000 7fc3fe432740 20 client.16407 trim_cache size 0 max 16384 2024-03-18T00:54:16.216 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:54:16.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.217+0000 7fc3fb84b640 1 -- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3e00a28d0 msgr2=0x7fc3e00a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.217+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3e00a28d0 0x7fc3e00a4d90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x557c0ce65530 tx=0x7fc3f0009260 comp rx=0 tx=0).stop 2024-03-18T00:54:16.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.217+0000 7fc3fb84b640 1 -- 172.21.15.49:0/2093837615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f41087e0 msgr2=0x7fc3f41a6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.217+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f41087e0 0x7fc3f41a6850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400e9f0 tx=0x7fc3e400eeb0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.217+0000 7fc3fb84b640 1 -- 172.21.15.49:0/2093837615 shutdown_connections 2024-03-18T00:54:16.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.217+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3e00a28d0 0x7fc3e00a4d90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.217+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3f410db00 0x7fc3f4079a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.217+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3f410b150 0x7fc3f41a6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.217+0000 7fc3fb84b640 1 --2- 172.21.15.49:0/2093837615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3f41087e0 0x7fc3f41a6850 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.218+0000 7fc3fb84b640 1 -- 172.21.15.49:0/2093837615 >> 172.21.15.49:0/2093837615 conn(0x7fc3f406e640 msgr2=0x7fc3f410a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.218+0000 7fc3fb84b640 1 -- 172.21.15.49:0/2093837615 shutdown_connections 2024-03-18T00:54:16.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.218+0000 7fc3fb84b640 1 -- 172.21.15.49:0/2093837615 wait complete. 2024-03-18T00:54:16.253 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:54:16.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.583+0000 7f8b03316640 1 -- 172.21.15.49:0/767790064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc10db00 msgr2=0x7f8afc10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.583+0000 7f8b03316640 1 --2- 172.21.15.49:0/767790064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc10db00 0x7f8afc10df80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8aec009900 tx=0x7f8aec02bc80 comp rx=0 tx=0).stop 2024-03-18T00:54:16.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.583+0000 7f8b03316640 1 -- 172.21.15.49:0/767790064 shutdown_connections 2024-03-18T00:54:16.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.583+0000 7f8b03316640 1 --2- 172.21.15.49:0/767790064 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8afc10b150 0x7f8afc10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.583+0000 7f8b03316640 1 --2- 172.21.15.49:0/767790064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc10db00 0x7f8afc10df80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.583+0000 7f8b03316640 1 --2- 172.21.15.49:0/767790064 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8afc1087e0 0x7f8afc10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.583+0000 7f8b03316640 1 -- 172.21.15.49:0/767790064 >> 172.21.15.49:0/767790064 conn(0x7f8afc06e640 msgr2=0x7f8afc070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.593+0000 7f8b03316640 1 -- 172.21.15.49:0/767790064 shutdown_connections 2024-03-18T00:54:16.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.598+0000 7f8b03316640 1 -- 172.21.15.49:0/767790064 wait complete. 2024-03-18T00:54:16.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.599+0000 7f8b03316640 1 Processor -- start 2024-03-18T00:54:16.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.602+0000 7f8b03316640 1 -- start start 2024-03-18T00:54:16.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.602+0000 7f8b03316640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc1087e0 0x7f8afc1a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.602+0000 7f8b03316640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8afc10b150 0x7f8afc1a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.602+0000 7f8b03316640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8afc10db00 0x7f8afc1adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.602+0000 7f8b03316640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc11c2d0 con 0x7f8afc10db00 2024-03-18T00:54:16.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.602+0000 7f8b03316640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc11c150 con 0x7f8afc10b150 2024-03-18T00:54:16.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.602+0000 7f8b03316640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc11c450 con 0x7f8afc1087e0 2024-03-18T00:54:16.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.603+0000 7f8b018b3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8afc10db00 0x7f8afc1adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.603+0000 7f8b018b3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8afc10db00 0x7f8afc1adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42326/0 (socket says 172.21.15.49:42326) 2024-03-18T00:54:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.603+0000 7f8b018b3640 1 -- 172.21.15.49:0/2380250852 learned_addr learned my addr 172.21.15.49:0/2380250852 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.606+0000 7f8b018b3640 1 -- 172.21.15.49:0/2380250852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc1087e0 msgr2=0x7f8afc1a6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.606+0000 7f8b010b2640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc1087e0 0x7f8afc1a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.606+0000 7f8b018b3640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc1087e0 0x7f8afc1a6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.606+0000 7f8b018b3640 1 -- 172.21.15.49:0/2380250852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8afc10b150 msgr2=0x7f8afc1a6e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:54:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.606+0000 7f8b018b3640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8afc10b150 0x7f8afc1a6e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.606+0000 7f8b018b3640 1 -- 172.21.15.49:0/2380250852 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc1ae560 con 0x7f8afc10db00 2024-03-18T00:54:16.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.609+0000 7f8b018b3640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8afc10db00 0x7f8afc1adef0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8aec036040 tx=0x7f8aec02c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.609+0000 7f8af27fc640 1 -- 172.21.15.49:0/2380250852 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec041070 con 0x7f8afc10db00 2024-03-18T00:54:16.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.610+0000 7f8af27fc640 1 -- 172.21.15.49:0/2380250852 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8aec0054e0 con 0x7f8afc10db00 2024-03-18T00:54:16.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.610+0000 7f8af27fc640 1 -- 172.21.15.49:0/2380250852 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec034440 con 0x7f8afc10db00 2024-03-18T00:54:16.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.613+0000 7f8b03316640 1 -- 172.21.15.49:0/2380250852 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8afc1ae7f0 con 0x7f8afc10db00 2024-03-18T00:54:16.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.613+0000 7f8b03316640 1 -- 172.21.15.49:0/2380250852 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8afc1aec90 con 0x7f8afc10db00 2024-03-18T00:54:16.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:16 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2701324832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:54:16.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.624+0000 7f8af27fc640 1 -- 172.21.15.49:0/2380250852 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8aec044070 con 0x7f8afc10db00 2024-03-18T00:54:16.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.625+0000 7f8af27fc640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ae80a2980 0x7f8ae80a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.626+0000 7f8af27fc640 1 -- 172.21.15.49:0/2380250852 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8aec0f09c0 con 0x7f8afc10db00 2024-03-18T00:54:16.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.627+0000 7f8b010b2640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ae80a2980 0x7f8ae80a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.630 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.629+0000 7f8b010b2640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ae80a2980 0x7f8ae80a4e40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8af4007bf0 tx=0x7f8af4007240 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.637+0000 7f8b05fd6740 1 Processor -- start 2024-03-18T00:54:16.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.637+0000 7f8b05fd6740 1 -- start start 2024-03-18T00:54:16.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.638+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffbbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.639+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbc240 0x55adeffc0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.639+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 0x55adeffc3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.639+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55adeff20490 con 0x55adeffbb850 2024-03-18T00:54:16.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.639+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55adefec8c90 con 0x55adeffc13b0 2024-03-18T00:54:16.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.639+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55adeff1c530 con 0x55adeffbc240 2024-03-18T00:54:16.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.642+0000 7f8b010b2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffbbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.642+0000 7f8b010b2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffbbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42334/0 (socket says 172.21.15.49:42334) 2024-03-18T00:54:16.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.642+0000 7f8b010b2640 1 -- 172.21.15.49:0/1091665225 learned_addr learned my addr 172.21.15.49:0/1091665225 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:16.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.642+0000 7f8b018b3640 1 --2- 172.21.15.49:0/1091665225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 0x55adeffc3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.646+0000 7f8b010b2640 1 -- 172.21.15.49:0/1091665225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbc240 msgr2=0x55adeffc0d40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:16.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.646+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1091665225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbc240 0x55adeffc0d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.646+0000 7f8b010b2640 1 -- 172.21.15.49:0/1091665225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 msgr2=0x55adeffc3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.646+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1091665225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 0x55adeffc3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.648 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.646+0000 7f8b010b2640 1 -- 172.21.15.49:0/1091665225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adeff83500 con 0x55adeffbb850 2024-03-18T00:54:16.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.651+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1091665225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffbbc70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8af400cdb0 tx=0x7f8af400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77bf62ed8d2b485 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.658+0000 7f8ae77fe640 1 -- 172.21.15.49:0/1091665225 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4023070 con 0x55adeffbb850 2024-03-18T00:54:16.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.659+0000 7f8ae77fe640 1 -- 172.21.15.49:0/1091665225 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8af4013bc0 con 0x55adeffbb850 2024-03-18T00:54:16.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.661+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1091665225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 msgr2=0x55adeffbbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.661+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1091665225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffbbc70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8af400cdb0 tx=0x7f8af400f000 comp rx=0 tx=0).stop 2024-03-18T00:54:16.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.668+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1091665225 shutdown_connections 2024-03-18T00:54:16.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.668+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1091665225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 0x55adeffc3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.668+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1091665225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbc240 0x55adeffc0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.668+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1091665225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffbbc70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.668+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1091665225 >> 172.21.15.49:0/1091665225 conn(0x55adeffb6dc0 msgr2=0x55adeffb71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.671+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1091665225 shutdown_connections 2024-03-18T00:54:16.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.674+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1091665225 wait complete. 2024-03-18T00:54:16.674 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.674+0000 7f8b05fd6740 1 -- start start 2024-03-18T00:54:16.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.675+0000 7f8b05fd6740 1 Processor -- start 2024-03-18T00:54:16.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.677+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffc5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.677+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 0x55adeffc61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.677+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffc13b0 0x55adeffcd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.677+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55adeffc43b0 con 0x55adeffbb850 2024-03-18T00:54:16.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.678+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55adefedc230 con 0x55adeffbc240 2024-03-18T00:54:16.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.678+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55adeffc4530 con 0x55adeffc13b0 2024-03-18T00:54:16.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.678+0000 7f8b05fd6740 10 client.0 _ll_register_callbacks cb 0x55adeff18010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.679+0000 7f8b010b2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffc5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.679+0000 7f8b010b2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffc5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42342/0 (socket says 172.21.15.49:42342) 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.679+0000 7f8b010b2640 1 -- 172.21.15.49:0/1141616291 learned_addr learned my addr 172.21.15.49:0/1141616291 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.681+0000 7f8b008b1640 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 0x55adeffc61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.681+0000 7f8b018b3640 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffc13b0 0x55adeffcd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.681+0000 7f8b010b2640 1 -- 172.21.15.49:0/1141616291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffc13b0 msgr2=0x55adeffcd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.681+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffc13b0 0x55adeffcd190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.681+0000 7f8b010b2640 1 -- 172.21.15.49:0/1141616291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 msgr2=0x55adeffc61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.681+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 0x55adeffc61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.681+0000 7f8b010b2640 1 -- 172.21.15.49:0/1141616291 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adefeeb100 con 0x55adeffbb850 2024-03-18T00:54:16.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.682+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffc5c80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8af400cc00 tx=0x7f8af4007400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=506823f188805162 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.683+0000 7f8ae4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:16.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.685+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/1141616291 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af402c020 con 0x55adeffbb850 2024-03-18T00:54:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.685+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1141616291 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8af4013bc0 con 0x55adeffbb850 2024-03-18T00:54:16.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.685+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/1141616291 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8af401e600 con 0x55adeffbb850 2024-03-18T00:54:16.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.687+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/1141616291 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8af4037050 con 0x55adeffbb850 2024-03-18T00:54:16.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.688+0000 7f8b05fd6740 10 client.16416 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:16.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.688+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1141616291 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55adefc343f0 con 0x55adeffbb850 2024-03-18T00:54:16.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:16 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2701324832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:54:16.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.694+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/1141616291 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8af4013bc0 con 0x55adeffbb850 2024-03-18T00:54:16.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.694+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/1141616291 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8af4010030 con 0x55adeffbb850 2024-03-18T00:54:16.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.696+0000 7f8b05fd6740 10 client.16416 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:16.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.696+0000 7f8b05fd6740 10 client.16416 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:54:16.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.696+0000 7f8b05fd6740 20 client.16416 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:16.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.696+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55adeffd1780 0x55adeffd3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.696+0000 7f8b05fd6740 4 client.16416 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:16.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.696+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1141616291 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adeff2d9d0 con 0x55adeffd1780 2024-03-18T00:54:16.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.701+0000 7f8b008b1640 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55adeffd1780 0x55adeffd3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.721+0000 7f8ae4ff9640 10 client.16416 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:54:16.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.721+0000 7f8b008b1640 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55adeffd1780 0x55adeffd3bb0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fda345b2dde0950c server_cookie=5b6eb43414391e1 in_seq=0 out_seq=0 2024-03-18T00:54:16.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.724+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/1141616291 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55adeff2d9d0 con 0x55adeffd1780 2024-03-18T00:54:16.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.724+0000 7f8ae4ff9640 10 client.16416 handle_command_reply: tid=0 2024-03-18T00:54:16.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.724+0000 7f8b05fd6740 1 client.16416 shutdown 2024-03-18T00:54:16.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.727+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1141616291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 msgr2=0x55adeffc5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.727+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffc5c80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8af400cc00 tx=0x7f8af4007400 comp rx=0 tx=0).stop 2024-03-18T00:54:16.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.728+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1141616291 shutdown_connections 2024-03-18T00:54:16.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.728+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55adeffd1780 0x55adeffd3bb0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.728+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 0x55adeffc61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.728+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffc13b0 0x55adeffcd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.728+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1141616291 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbb850 0x55adeffc5c80 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.728+0000 7f8ae4ff9640 0 client.16416 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:54:16.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.728+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1141616291 >> 172.21.15.49:0/1141616291 conn(0x55adeffb6dc0 msgr2=0x55adeffb5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.731+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1141616291 shutdown_connections 2024-03-18T00:54:16.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.737+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1141616291 wait complete. 2024-03-18T00:54:16.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.740+0000 7f8b05fd6740 20 client.16416 trim_cache size 0 max 16384 2024-03-18T00:54:16.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.778+0000 7f8b05fd6740 1 Processor -- start 2024-03-18T00:54:16.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.779+0000 7f8b05fd6740 1 -- start start 2024-03-18T00:54:16.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.779+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbb850 0x55adeffc5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.779+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 0x55adeffcd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.779+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffc13b0 0x55adeffb5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.779+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1b7d00 con 0x55adeffc13b0 2024-03-18T00:54:16.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.779+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1b9840 con 0x55adeffbc240 2024-03-18T00:54:16.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.779+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1b83d0 con 0x55adeffbb850 2024-03-18T00:54:16.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.780+0000 7f8b008b1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 0x55adeffcd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.780+0000 7f8b008b1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 0x55adeffcd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59632/0 (socket says 172.21.15.49:59632) 2024-03-18T00:54:16.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.780+0000 7f8b008b1640 1 -- 172.21.15.49:0/1742524939 learned_addr learned my addr 172.21.15.49:0/1742524939 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:16.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.781+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1742524939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbb850 0x55adeffc5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.783+0000 7f8b018b3640 1 --2- 172.21.15.49:0/1742524939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffc13b0 0x55adeffb5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.785+0000 7f8b010b2640 1 -- 172.21.15.49:0/1742524939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 msgr2=0x55adeffcd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.785+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1742524939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 0x55adeffcd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.785+0000 7f8b010b2640 1 -- 172.21.15.49:0/1742524939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffc13b0 msgr2=0x55adeffb5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.785+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1742524939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffc13b0 0x55adeffb5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.785+0000 7f8b010b2640 1 -- 172.21.15.49:0/1742524939 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adeffb2640 con 0x55adeffbb850 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.789+0000 7f8b010b2640 1 --2- 172.21.15.49:0/1742524939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbb850 0x55adeffc5fa0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8af40044d0 tx=0x7f8af4007650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b498769638c6e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.789+0000 7f8ae6ffd640 1 -- 172.21.15.49:0/1742524939 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4040070 con 0x55adeffbb850 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.789+0000 7f8ae6ffd640 1 -- 172.21.15.49:0/1742524939 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8af401e8e0 con 0x55adeffbb850 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.791+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1742524939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbb850 msgr2=0x55adeffc5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.791+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1742524939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbb850 0x55adeffc5fa0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8af40044d0 tx=0x7f8af4007650 comp rx=0 tx=0).stop 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.791+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1742524939 shutdown_connections 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.791+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1742524939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffbc240 0x55adeffcd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.791+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1742524939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffc13b0 0x55adeffb5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.791+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/1742524939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffbb850 0x55adeffc5fa0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.791+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1742524939 >> 172.21.15.49:0/1742524939 conn(0x55adeffb6dc0 msgr2=0x55adeffc5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.791+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1742524939 shutdown_connections 2024-03-18T00:54:16.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.792+0000 7f8b05fd6740 1 -- 172.21.15.49:0/1742524939 wait complete. 2024-03-18T00:54:16.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.792+0000 7f8b05fd6740 1 -- start start 2024-03-18T00:54:16.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.792+0000 7f8b05fd6740 1 Processor -- start 2024-03-18T00:54:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.795+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbc240 0x55adeffca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.795+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 0x55adeffca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.795+0000 7f8b05fd6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffcb720 0x55adeffcbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.795+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1b9170 con 0x55adeffbc240 2024-03-18T00:54:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.795+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1b8aa0 con 0x55adeffc13b0 2024-03-18T00:54:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.795+0000 7f8b05fd6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc1b5ce0 con 0x55adeffcb720 2024-03-18T00:54:16.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.795+0000 7f8b05fd6740 10 client.0 _ll_register_callbacks cb 0x7f8af4021070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:16.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.796+0000 7f8b018b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffcb720 0x55adeffcbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.796+0000 7f8b008b1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 0x55adeffca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.796+0000 7f8b018b3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffcb720 0x55adeffcbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:55078/0 (socket says 172.21.15.49:55078) 2024-03-18T00:54:16.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.796+0000 7f8b018b3640 1 -- 172.21.15.49:0/3027785296 learned_addr learned my addr 172.21.15.49:0/3027785296 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:16.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.798+0000 7f8b018b3640 1 -- 172.21.15.49:0/3027785296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 msgr2=0x55adeffca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.798+0000 7f8b018b3640 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 0x55adeffca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.798+0000 7f8b018b3640 1 -- 172.21.15.49:0/3027785296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbc240 msgr2=0x55adeffca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:16.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.798+0000 7f8b018b3640 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbc240 0x55adeffca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.798+0000 7f8b018b3640 1 -- 172.21.15.49:0/3027785296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adefcfe7d0 con 0x55adeffcb720 2024-03-18T00:54:16.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.799+0000 7f8b018b3640 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffcb720 0x55adeffcbba0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0ee4f0 tx=0x7f8aec0035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8647c6e10b86f552 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:16.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.800+0000 7f8ae4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:54:16.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.800+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/3027785296 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec04b040 con 0x55adeffcb720 2024-03-18T00:54:16.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.801+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/3027785296 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8aec003680 con 0x55adeffcb720 2024-03-18T00:54:16.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.801+0000 7f8b05fd6740 1 -- 172.21.15.49:0/3027785296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55adeff27d20 con 0x55adeffcb720 2024-03-18T00:54:16.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.801+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/3027785296 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8aec0d6200 con 0x55adeffcb720 2024-03-18T00:54:16.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.801+0000 7f8b05fd6740 10 client.25975 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.802+0000 7f8b05fd6740 1 -- 172.21.15.49:0/3027785296 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55adeffcd310 con 0x55adeffcb720 2024-03-18T00:54:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.802+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/3027785296 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55adeff27d20 con 0x55adeffcb720 2024-03-18T00:54:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.803+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/3027785296 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8aec0bd5a0 con 0x55adeffcb720 2024-03-18T00:54:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.803+0000 7f8b05fd6740 10 client.25975 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.803+0000 7f8b05fd6740 10 client.25975 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:54:16.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.803+0000 7f8b05fd6740 20 client.25975 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:16.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.803+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55adf0038120 0x55adf003a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:16.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.803+0000 7f8b05fd6740 4 client.25975 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:54:16.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.803+0000 7f8b05fd6740 1 -- 172.21.15.49:0/3027785296 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55adefea56a0 con 0x55adf0038120 2024-03-18T00:54:16.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.807+0000 7f8b010b2640 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55adf0038120 0x55adf003a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:16.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.811+0000 7f8b010b2640 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55adf0038120 0x55adf003a540 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8d9c58c2709f8c86 server_cookie=e2079df33c0abb3a in_seq=0 out_seq=0 2024-03-18T00:54:16.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.811+0000 7f8ae4ff9640 10 client.25975 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:54:16.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.812+0000 7f8ae4ff9640 1 -- 172.21.15.49:0/3027785296 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55adefea56a0 con 0x55adf0038120 2024-03-18T00:54:16.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.812+0000 7f8ae4ff9640 10 client.25975 handle_command_reply: tid=0 2024-03-18T00:54:16.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.812+0000 7f8b05fd6740 1 client.25975 shutdown 2024-03-18T00:54:16.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.815+0000 7f8b05fd6740 1 -- 172.21.15.49:0/3027785296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffcb720 msgr2=0x55adeffcbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.815+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffcb720 0x55adeffcbba0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8aec0ee4f0 tx=0x7f8aec0035f0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.816+0000 7f8b05fd6740 1 -- 172.21.15.49:0/3027785296 shutdown_connections 2024-03-18T00:54:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.816+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55adf0038120 0x55adf003a540 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.816+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adeffc13b0 0x55adeffca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.816+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adeffcb720 0x55adeffcbba0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.816+0000 7f8b05fd6740 1 --2- 172.21.15.49:0/3027785296 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adeffbc240 0x55adeffca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.816+0000 7f8b05fd6740 1 -- 172.21.15.49:0/3027785296 >> 172.21.15.49:0/3027785296 conn(0x55adeffb6dc0 msgr2=0x55adeffba020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.816+0000 7f8ae4ff9640 0 client.25975 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:54:16.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.817+0000 7f8b05fd6740 1 -- 172.21.15.49:0/3027785296 shutdown_connections 2024-03-18T00:54:16.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.818+0000 7f8b05fd6740 1 -- 172.21.15.49:0/3027785296 wait complete. 2024-03-18T00:54:16.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.819+0000 7f8b05fd6740 20 client.25975 trim_cache size 0 max 16384 2024-03-18T00:54:16.819 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:54:16.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.820+0000 7f8b03316640 1 -- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ae80a2980 msgr2=0x7f8ae80a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.820+0000 7f8b03316640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ae80a2980 0x7f8ae80a4e40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8af4007bf0 tx=0x7f8af4007240 comp rx=0 tx=0).stop 2024-03-18T00:54:16.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.820+0000 7f8b03316640 1 -- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8afc10db00 msgr2=0x7f8afc1adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:16.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.820+0000 7f8b03316640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8afc10db00 0x7f8afc1adef0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8aec036040 tx=0x7f8aec02c6b0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.821+0000 7f8b03316640 1 -- 172.21.15.49:0/2380250852 shutdown_connections 2024-03-18T00:54:16.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.821+0000 7f8b03316640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8ae80a2980 0x7f8ae80a4e40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.821+0000 7f8b03316640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8afc10b150 0x7f8afc1a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.821+0000 7f8b03316640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8afc10db00 0x7f8afc1adef0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.821+0000 7f8b03316640 1 --2- 172.21.15.49:0/2380250852 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8afc1087e0 0x7f8afc1a6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:16.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.821+0000 7f8b03316640 1 -- 172.21.15.49:0/2380250852 >> 172.21.15.49:0/2380250852 conn(0x7f8afc06e640 msgr2=0x7f8afc108c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:16.823 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.823+0000 7f8b03316640 1 -- 172.21.15.49:0/2380250852 shutdown_connections 2024-03-18T00:54:16.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:16.824+0000 7f8b03316640 1 -- 172.21.15.49:0/2380250852 wait complete. 2024-03-18T00:54:16.870 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:54:16.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:16 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2701324832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:54:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.345+0000 7f3a2a050640 1 -- 172.21.15.49:0/3984278828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a24072bf0 msgr2=0x7f3a24073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:17.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.345+0000 7f3a2a050640 1 --2- 172.21.15.49:0/3984278828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a24072bf0 0x7f3a24073010 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c009be0 tx=0x7f3a1c02bf30 comp rx=0 tx=0).stop 2024-03-18T00:54:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.347+0000 7f3a2a050640 1 -- 172.21.15.49:0/3984278828 shutdown_connections 2024-03-18T00:54:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.347+0000 7f3a2a050640 1 --2- 172.21.15.49:0/3984278828 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a24109870 0x7f3a2410e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.347+0000 7f3a2a050640 1 --2- 172.21.15.49:0/3984278828 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a24108eb0 0x7f3a24109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.347+0000 7f3a2a050640 1 --2- 172.21.15.49:0/3984278828 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a24072bf0 0x7f3a24073010 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.347+0000 7f3a2a050640 1 -- 172.21.15.49:0/3984278828 >> 172.21.15.49:0/3984278828 conn(0x7f3a2406e6c0 msgr2=0x7f3a24070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.348+0000 7f3a2a050640 1 -- 172.21.15.49:0/3984278828 shutdown_connections 2024-03-18T00:54:17.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.348+0000 7f3a2a050640 1 -- 172.21.15.49:0/3984278828 wait complete. 2024-03-18T00:54:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.349+0000 7f3a2a050640 1 Processor -- start 2024-03-18T00:54:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.349+0000 7f3a2a050640 1 -- start start 2024-03-18T00:54:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.349+0000 7f3a2a050640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a24072bf0 0x7f3a241a67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.349+0000 7f3a2a050640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a24108eb0 0x7f3a241a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.349+0000 7f3a2a050640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a24109870 0x7f3a241add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.349+0000 7f3a2a050640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2411c130 con 0x7f3a24109870 2024-03-18T00:54:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.349+0000 7f3a2a050640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2411bfb0 con 0x7f3a24108eb0 2024-03-18T00:54:17.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.349+0000 7f3a2a050640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2411c2b0 con 0x7f3a24072bf0 2024-03-18T00:54:17.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.350+0000 7f3a2984f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a24109870 0x7f3a241add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.350+0000 7f3a2984f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a24109870 0x7f3a241add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42382/0 (socket says 172.21.15.49:42382) 2024-03-18T00:54:17.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.350+0000 7f3a2984f640 1 -- 172.21.15.49:0/979476353 learned_addr learned my addr 172.21.15.49:0/979476353 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:17.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.350+0000 7f3a2904e640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a24072bf0 0x7f3a241a67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.350+0000 7f3a2884d640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a24108eb0 0x7f3a241a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.351+0000 7f3a2984f640 1 -- 172.21.15.49:0/979476353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a24072bf0 msgr2=0x7f3a241a67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:17.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.351+0000 7f3a2984f640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a24072bf0 0x7f3a241a67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.351+0000 7f3a2984f640 1 -- 172.21.15.49:0/979476353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a24108eb0 msgr2=0x7f3a241a6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:17.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.351+0000 7f3a2984f640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a24108eb0 0x7f3a241a6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.351+0000 7f3a2984f640 1 -- 172.21.15.49:0/979476353 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a241ae3d0 con 0x7f3a24109870 2024-03-18T00:54:17.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.351+0000 7f3a2984f640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a24109870 0x7f3a241add60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000d750 tx=0x7f3a2000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.351+0000 7f3a1a7fc640 1 -- 172.21.15.49:0/979476353 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20017070 con 0x7f3a24109870 2024-03-18T00:54:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.352+0000 7f3a2a050640 1 -- 172.21.15.49:0/979476353 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a241ae6c0 con 0x7f3a24109870 2024-03-18T00:54:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.352+0000 7f3a2a050640 1 -- 172.21.15.49:0/979476353 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a241aec10 con 0x7f3a24109870 2024-03-18T00:54:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.352+0000 7f3a1a7fc640 1 -- 172.21.15.49:0/979476353 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a20005500 con 0x7f3a24109870 2024-03-18T00:54:17.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.352+0000 7f3a1a7fc640 1 -- 172.21.15.49:0/979476353 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a20020060 con 0x7f3a24109870 2024-03-18T00:54:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.357+0000 7f3a2df80740 1 Processor -- start 2024-03-18T00:54:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.359+0000 7f3a1a7fc640 1 -- 172.21.15.49:0/979476353 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3a20009bd0 con 0x7f3a24109870 2024-03-18T00:54:17.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.359+0000 7f3a2df80740 1 -- start start 2024-03-18T00:54:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.359+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.359+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b21f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.359+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b24a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.359+0000 7f3a2df80740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7a81490 con 0x55e3a7b1ca40 2024-03-18T00:54:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.359+0000 7f3a2df80740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7a29c90 con 0x55e3a7b1d430 2024-03-18T00:54:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.359+0000 7f3a2df80740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7a7d530 con 0x55e3a7b225a0 2024-03-18T00:54:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.360+0000 7f3a2884d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.360+0000 7f3a2884d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42390/0 (socket says 172.21.15.49:42390) 2024-03-18T00:54:17.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.360+0000 7f3a2884d640 1 -- 172.21.15.49:0/555833238 learned_addr learned my addr 172.21.15.49:0/555833238 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:17.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.361+0000 7f3a2884d640 1 -- 172.21.15.49:0/555833238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 msgr2=0x55e3a7b24a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:17.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.361+0000 7f3a2884d640 1 --2- 172.21.15.49:0/555833238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b24a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.361+0000 7f3a2884d640 1 -- 172.21.15.49:0/555833238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 msgr2=0x55e3a7b21f30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:17.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.361+0000 7f3a2884d640 1 --2- 172.21.15.49:0/555833238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b21f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.361+0000 7f3a2884d640 1 -- 172.21.15.49:0/555833238 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3a7ae4500 con 0x55e3a7b1ca40 2024-03-18T00:54:17.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.362+0000 7f3a2884d640 1 --2- 172.21.15.49:0/555833238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1ce60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400dd60 tx=0x7f3a1400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520488f3076549f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:17.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.363+0000 7f3a0f7fe640 1 -- 172.21.15.49:0/555833238 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14011640 con 0x55e3a7b1ca40 2024-03-18T00:54:17.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.364+0000 7f3a0f7fe640 1 -- 172.21.15.49:0/555833238 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a1400fe40 con 0x55e3a7b1ca40 2024-03-18T00:54:17.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.365+0000 7f3a1a7fc640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3a100a2800 0x7f3a100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.365+0000 7f3a2df80740 1 -- 172.21.15.49:0/555833238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 msgr2=0x55e3a7b1ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:17.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.365+0000 7f3a2df80740 1 --2- 172.21.15.49:0/555833238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1ce60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400dd60 tx=0x7f3a1400f730 comp rx=0 tx=0).stop 2024-03-18T00:54:17.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.366+0000 7f3a1a7fc640 1 -- 172.21.15.49:0/979476353 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3a200ce520 con 0x7f3a24109870 2024-03-18T00:54:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.366+0000 7f3a2df80740 1 -- 172.21.15.49:0/555833238 shutdown_connections 2024-03-18T00:54:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.366+0000 7f3a2df80740 1 --2- 172.21.15.49:0/555833238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b21f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.366+0000 7f3a2df80740 1 --2- 172.21.15.49:0/555833238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b24a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.366+0000 7f3a2df80740 1 --2- 172.21.15.49:0/555833238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1ce60 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.366+0000 7f3a2df80740 1 -- 172.21.15.49:0/555833238 >> 172.21.15.49:0/555833238 conn(0x55e3a7b17fb0 msgr2=0x55e3a7b183b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.366+0000 7f3a2df80740 1 -- 172.21.15.49:0/555833238 shutdown_connections 2024-03-18T00:54:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.367+0000 7f3a2df80740 1 -- 172.21.15.49:0/555833238 wait complete. 2024-03-18T00:54:17.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.368+0000 7f3a2904e640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3a100a2800 0x7f3a100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.369+0000 7f3a2904e640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3a100a2800 0x7f3a100a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c02c470 tx=0x7f3a1c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:17.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.369+0000 7f3a2df80740 1 -- start start 2024-03-18T00:54:17.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.369+0000 7f3a2df80740 1 Processor -- start 2024-03-18T00:54:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.370+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b27010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.370+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b27550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.371+0000 7f3a2904e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b27010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.371+0000 7f3a2904e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b27010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42406/0 (socket says 172.21.15.49:42406) 2024-03-18T00:54:17.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.371+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b2e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.371+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7b255f0 con 0x55e3a7b1ca40 2024-03-18T00:54:17.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.371+0000 7f3a2904e640 1 -- 172.21.15.49:0/3428374612 learned_addr learned my addr 172.21.15.49:0/3428374612 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:17.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.371+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7a3d230 con 0x55e3a7b1d430 2024-03-18T00:54:17.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.371+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e3a7b25770 con 0x55e3a7b225a0 2024-03-18T00:54:17.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.371+0000 7f3a2df80740 10 client.0 _ll_register_callbacks cb 0x55e3a7a79010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.372+0000 7f3a2904e640 1 -- 172.21.15.49:0/3428374612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 msgr2=0x55e3a7b2e590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.372+0000 7f3a2904e640 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b2e590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.372+0000 7f3a2904e640 1 -- 172.21.15.49:0/3428374612 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 msgr2=0x55e3a7b27550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.372+0000 7f3a2904e640 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b27550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.372+0000 7f3a2904e640 1 -- 172.21.15.49:0/3428374612 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3a7a4c100 con 0x55e3a7b1ca40 2024-03-18T00:54:17.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.373+0000 7f3a2904e640 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b27010 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3a24073940 tx=0x7f3a1c0333e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15e343899f758ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:17.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.373+0000 7f3a0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:17.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.379+0000 7f3a0cff9640 1 -- 172.21.15.49:0/3428374612 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c050060 con 0x55e3a7b1ca40 2024-03-18T00:54:17.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.379+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e3a7a8e9d0 con 0x55e3a7b1ca40 2024-03-18T00:54:17.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.380+0000 7f3a0cff9640 1 -- 172.21.15.49:0/3428374612 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3a1c034430 con 0x55e3a7b1ca40 2024-03-18T00:54:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.381+0000 7f3a0cff9640 1 -- 172.21.15.49:0/3428374612 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3a1c034630 con 0x55e3a7b1ca40 2024-03-18T00:54:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.381+0000 7f3a2df80740 10 client.16431 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:17.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.381+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e3a77953f0 con 0x55e3a7b1ca40 2024-03-18T00:54:17.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.384+0000 7f3a0cff9640 1 -- 172.21.15.49:0/3428374612 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a1c034cb0 con 0x55e3a7b1ca40 2024-03-18T00:54:17.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.384+0000 7f3a0cff9640 1 -- 172.21.15.49:0/3428374612 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3a1c03e420 con 0x55e3a7b1ca40 2024-03-18T00:54:17.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.387+0000 7f3a2df80740 10 client.16431 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:17.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.387+0000 7f3a2df80740 10 client.16431 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:54:17.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.387+0000 7f3a2df80740 20 client.16431 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:17.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.387+0000 7f3a2df80740 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e3a7b33380 0x55e3a7b35840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.387+0000 7f3a2df80740 4 client.16431 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:17.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.387+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e3a7a8e9d0 con 0x55e3a7b33380 2024-03-18T00:54:17.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.390+0000 7f3a2884d640 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e3a7b33380 0x55e3a7b35840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.411+0000 7f3a2884d640 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e3a7b33380 0x55e3a7b35840 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=56f8cb9f741eda6a server_cookie=2d695aca09ea4e45 in_seq=0 out_seq=0 2024-03-18T00:54:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.412+0000 7f3a0cff9640 10 client.16431 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:54:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.412+0000 7f3a0cff9640 1 -- 172.21.15.49:0/3428374612 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e3a7a8e9d0 con 0x55e3a7b33380 2024-03-18T00:54:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.412+0000 7f3a0cff9640 10 client.16431 handle_command_reply: tid=0 2024-03-18T00:54:17.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.412+0000 7f3a2df80740 1 client.16431 shutdown 2024-03-18T00:54:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.413+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 msgr2=0x55e3a7b27010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.413+0000 7f3a2df80740 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b27010 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f3a24073940 tx=0x7f3a1c0333e0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.413+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 shutdown_connections 2024-03-18T00:54:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.413+0000 7f3a2df80740 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e3a7b33380 0x55e3a7b35840 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.413+0000 7f3a2df80740 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b27550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.413+0000 7f3a2df80740 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b2e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.413+0000 7f3a2df80740 1 --2- 172.21.15.49:0/3428374612 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b27010 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.414+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 >> 172.21.15.49:0/3428374612 conn(0x55e3a7b17fb0 msgr2=0x55e3a7b17140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.414+0000 7f3a0cff9640 0 client.16431 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:54:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.414+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 shutdown_connections 2024-03-18T00:54:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.414+0000 7f3a2df80740 1 -- 172.21.15.49:0/3428374612 wait complete. 2024-03-18T00:54:17.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.415+0000 7f3a2df80740 20 client.16431 trim_cache size 0 max 16384 2024-03-18T00:54:17.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.468+0000 7f3a2df80740 1 Processor -- start 2024-03-18T00:54:17.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.469+0000 7f3a2df80740 1 -- start start 2024-03-18T00:54:17.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.470+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.470+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.470+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b1ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.470+0000 7f3a2df80740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0800e690 con 0x55e3a7b1d430 2024-03-18T00:54:17.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.470+0000 7f3a2df80740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3a080101d0 con 0x55e3a7b1ca40 2024-03-18T00:54:17.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.470+0000 7f3a2df80740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0800ed60 con 0x55e3a7b225a0 2024-03-18T00:54:17.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.470+0000 7f3a2884d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.470+0000 7f3a2884d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42410/0 (socket says 172.21.15.49:42410) 2024-03-18T00:54:17.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.470+0000 7f3a2884d640 1 -- 172.21.15.49:0/2134327062 learned_addr learned my addr 172.21.15.49:0/2134327062 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:17.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.472+0000 7f3a2904e640 1 --2- 172.21.15.49:0/2134327062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.472+0000 7f3a2984f640 1 --2- 172.21.15.49:0/2134327062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b1ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.475+0000 7f3a2884d640 1 -- 172.21.15.49:0/2134327062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 msgr2=0x55e3a7b1ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.475+0000 7f3a2884d640 1 --2- 172.21.15.49:0/2134327062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b1ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.475+0000 7f3a2884d640 1 -- 172.21.15.49:0/2134327062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1ca40 msgr2=0x55e3a7b1e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.475+0000 7f3a2884d640 1 --2- 172.21.15.49:0/2134327062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.475+0000 7f3a2884d640 1 -- 172.21.15.49:0/2134327062 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e3a785f7d0 con 0x55e3a7b1d430 2024-03-18T00:54:17.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.477+0000 7f3a2884d640 1 --2- 172.21.15.49:0/2134327062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e550 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400dd60 tx=0x7f3a1400d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529ab20609d1e310 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.483+0000 7f3a0effd640 1 -- 172.21.15.49:0/2134327062 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1402c070 con 0x55e3a7b1d430 2024-03-18T00:54:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.483+0000 7f3a0effd640 1 -- 172.21.15.49:0/2134327062 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a1401b6f0 con 0x55e3a7b1d430 2024-03-18T00:54:17.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.483+0000 7f3a0effd640 1 -- 172.21.15.49:0/2134327062 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14026330 con 0x55e3a7b1d430 2024-03-18T00:54:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.484+0000 7f3a2df80740 1 -- 172.21.15.49:0/2134327062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1d430 msgr2=0x55e3a7b2e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:17.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.485+0000 7f3a2df80740 1 --2- 172.21.15.49:0/2134327062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e550 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f3a1400dd60 tx=0x7f3a1400d9c0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.489+0000 7f3a2df80740 1 -- 172.21.15.49:0/2134327062 shutdown_connections 2024-03-18T00:54:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.489+0000 7f3a2df80740 1 --2- 172.21.15.49:0/2134327062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b225a0 0x55e3a7b1ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.489+0000 7f3a2df80740 1 --2- 172.21.15.49:0/2134327062 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e550 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.489+0000 7f3a2df80740 1 --2- 172.21.15.49:0/2134327062 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b1e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.489+0000 7f3a2df80740 1 -- 172.21.15.49:0/2134327062 >> 172.21.15.49:0/2134327062 conn(0x55e3a7b17fb0 msgr2=0x55e3a7b18390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:17.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.498+0000 7f3a2df80740 1 -- 172.21.15.49:0/2134327062 shutdown_connections 2024-03-18T00:54:17.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.502+0000 7f3a2df80740 1 -- 172.21.15.49:0/2134327062 wait complete. 2024-03-18T00:54:17.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.503+0000 7f3a2df80740 1 -- start start 2024-03-18T00:54:17.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.503+0000 7f3a2df80740 1 Processor -- start 2024-03-18T00:54:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.505+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b2df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.506+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.506+0000 7f3a2df80740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b225a0 0x55e3a7b2ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.506+0000 7f3a2df80740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0800fb00 con 0x55e3a7b225a0 2024-03-18T00:54:17.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.506+0000 7f3a2df80740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0800f430 con 0x55e3a7b1d430 2024-03-18T00:54:17.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.506+0000 7f3a2df80740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0800c670 con 0x55e3a7b1ca40 2024-03-18T00:54:17.507 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.507+0000 7f3a2df80740 10 client.0 _ll_register_callbacks cb 0x7f3a1c009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:17.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.508+0000 7f3a2984f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b225a0 0x55e3a7b2ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.508+0000 7f3a2884d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.508+0000 7f3a2984f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b225a0 0x55e3a7b2ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:42414/0 (socket says 172.21.15.49:42414) 2024-03-18T00:54:17.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.508+0000 7f3a2984f640 1 -- 172.21.15.49:0/134972362 learned_addr learned my addr 172.21.15.49:0/134972362 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:17.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.508+0000 7f3a2904e640 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b2df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.509+0000 7f3a2984f640 1 -- 172.21.15.49:0/134972362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b1ca40 msgr2=0x55e3a7b2df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.509+0000 7f3a2984f640 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b2df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.509+0000 7f3a2984f640 1 -- 172.21.15.49:0/134972362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 msgr2=0x55e3a7b2e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.509+0000 7f3a2984f640 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.509+0000 7f3a2984f640 1 -- 172.21.15.49:0/134972362 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3a7b2b2b0 con 0x55e3a7b225a0 2024-03-18T00:54:17.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.510+0000 7f3a2984f640 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b225a0 0x55e3a7b2ab40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f3a20097810 tx=0x7f3a2009b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c13891bdcf7769 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:17.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.511+0000 7f3a0cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:17.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.511+0000 7f3a0cff9640 1 -- 172.21.15.49:0/134972362 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a2009d030 con 0x55e3a7b225a0 2024-03-18T00:54:17.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.511+0000 7f3a0cff9640 1 -- 172.21.15.49:0/134972362 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3a200974b0 con 0x55e3a7b225a0 2024-03-18T00:54:17.511 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.511+0000 7f3a2df80740 1 -- 172.21.15.49:0/134972362 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a1401b6f0 con 0x55e3a7b225a0 2024-03-18T00:54:17.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.512+0000 7f3a0cff9640 1 -- 172.21.15.49:0/134972362 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3a2009ef90 con 0x55e3a7b225a0 2024-03-18T00:54:17.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.513+0000 7f3a2df80740 10 client.16437 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:17.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.513+0000 7f3a2df80740 1 -- 172.21.15.49:0/134972362 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e3a7b98470 con 0x55e3a7b225a0 2024-03-18T00:54:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.514+0000 7f3a0cff9640 1 -- 172.21.15.49:0/134972362 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a200a35e0 con 0x55e3a7b225a0 2024-03-18T00:54:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.514+0000 7f3a0cff9640 1 -- 172.21.15.49:0/134972362 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3a20099d00 con 0x55e3a7b225a0 2024-03-18T00:54:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.515+0000 7f3a2df80740 10 client.16437 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.515+0000 7f3a2df80740 10 client.16437 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:54:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.515+0000 7f3a2df80740 20 client.16437 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.515+0000 7f3a2df80740 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e3a7b98e50 0x55e3a7b9b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.515+0000 7f3a2df80740 4 client.16437 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:54:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.515+0000 7f3a2df80740 1 -- 172.21.15.49:0/134972362 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f3a1401b6f0 con 0x55e3a7b98e50 2024-03-18T00:54:17.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.519+0000 7f3a2904e640 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e3a7b98e50 0x55e3a7b9b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:17.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.534+0000 7f3a2904e640 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e3a7b98e50 0x55e3a7b9b270 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d296cb3a899916e9 server_cookie=c473d3bb0ed73f0e in_seq=0 out_seq=0 2024-03-18T00:54:17.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.534+0000 7f3a0cff9640 10 client.16437 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:54:17.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.542+0000 7f3a0cff9640 1 -- 172.21.15.49:0/134972362 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f3a1401b6f0 con 0x55e3a7b98e50 2024-03-18T00:54:17.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.542+0000 7f3a0cff9640 10 client.16437 handle_command_reply: tid=0 2024-03-18T00:54:17.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.542+0000 7f3a2df80740 1 client.16437 shutdown 2024-03-18T00:54:17.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.543+0000 7f3a2df80740 1 -- 172.21.15.49:0/134972362 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b225a0 msgr2=0x55e3a7b2ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:17.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.543+0000 7f3a2df80740 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b225a0 0x55e3a7b2ab40 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f3a20097810 tx=0x7f3a2009b540 comp rx=0 tx=0).stop 2024-03-18T00:54:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.543+0000 7f3a2df80740 1 -- 172.21.15.49:0/134972362 shutdown_connections 2024-03-18T00:54:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.543+0000 7f3a2df80740 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e3a7b98e50 0x55e3a7b9b270 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.543+0000 7f3a2df80740 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e3a7b1d430 0x55e3a7b2e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.544+0000 7f3a2df80740 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e3a7b225a0 0x55e3a7b2ab40 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.544+0000 7f3a2df80740 1 --2- 172.21.15.49:0/134972362 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e3a7b1ca40 0x55e3a7b2df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.544+0000 7f3a2df80740 1 -- 172.21.15.49:0/134972362 >> 172.21.15.49:0/134972362 conn(0x55e3a7b17fb0 msgr2=0x55e3a7b1e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:17.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.546+0000 7f3a0cff9640 0 client.16437 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:54:17.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.555+0000 7f3a2df80740 1 -- 172.21.15.49:0/134972362 shutdown_connections 2024-03-18T00:54:17.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.559+0000 7f3a2df80740 1 -- 172.21.15.49:0/134972362 wait complete. 2024-03-18T00:54:17.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.563+0000 7f3a2df80740 20 client.16437 trim_cache size 0 max 16384 2024-03-18T00:54:17.566 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:54:17.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.568+0000 7f3a2a050640 1 -- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3a100a2800 msgr2=0x7f3a100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:17.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.568+0000 7f3a2a050640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3a100a2800 0x7f3a100a4cc0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c02c470 tx=0x7f3a1c005fd0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.568+0000 7f3a2a050640 1 -- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a24109870 msgr2=0x7f3a241add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:17.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.568+0000 7f3a2a050640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a24109870 0x7f3a241add60 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000d750 tx=0x7f3a2000dc10 comp rx=0 tx=0).stop 2024-03-18T00:54:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.569+0000 7f3a2a050640 1 -- 172.21.15.49:0/979476353 shutdown_connections 2024-03-18T00:54:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.569+0000 7f3a2a050640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3a100a2800 0x7f3a100a4cc0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.569+0000 7f3a2a050640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3a24108eb0 0x7f3a241a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.569+0000 7f3a2a050640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3a24109870 0x7f3a241add60 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.569+0000 7f3a2a050640 1 --2- 172.21.15.49:0/979476353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a24072bf0 0x7f3a241a67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:17.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.569+0000 7f3a2a050640 1 -- 172.21.15.49:0/979476353 >> 172.21.15.49:0/979476353 conn(0x7f3a2406e6c0 msgr2=0x7f3a24070330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:17.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.570+0000 7f3a2a050640 1 -- 172.21.15.49:0/979476353 shutdown_connections 2024-03-18T00:54:17.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:17.570+0000 7f3a2a050640 1 -- 172.21.15.49:0/979476353 wait complete. 2024-03-18T00:54:17.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:54:17.774 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:17 smithi049 ceph-mon[32130]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 837 MiB/s wr, 1.27k op/s 2024-03-18T00:54:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:17 smithi107 ceph-mon[36270]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 837 MiB/s wr, 1.27k op/s 2024-03-18T00:54:17.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:17 smithi106 ceph-mon[35899]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 837 MiB/s wr, 1.27k op/s 2024-03-18T00:54:18.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.023+0000 7f98382c6640 1 -- 172.21.15.49:0/4025904127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9830073690 msgr2=0x7f9830073ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.023+0000 7f98382c6640 1 --2- 172.21.15.49:0/4025904127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9830073690 0x7f9830073ab0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f982c009900 tx=0x7f982c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:54:18.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.032+0000 7f98382c6640 1 -- 172.21.15.49:0/4025904127 shutdown_connections 2024-03-18T00:54:18.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.032+0000 7f98382c6640 1 --2- 172.21.15.49:0/4025904127 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9830078f20 0x7f983007b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.032+0000 7f98382c6640 1 --2- 172.21.15.49:0/4025904127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9830076570 0x7f98300789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.032+0000 7f98382c6640 1 --2- 172.21.15.49:0/4025904127 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9830073690 0x7f9830073ab0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.032+0000 7f98382c6640 1 -- 172.21.15.49:0/4025904127 >> 172.21.15.49:0/4025904127 conn(0x7f983006e640 msgr2=0x7f9830070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.038+0000 7f98382c6640 1 -- 172.21.15.49:0/4025904127 shutdown_connections 2024-03-18T00:54:18.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.042+0000 7f98382c6640 1 -- 172.21.15.49:0/4025904127 wait complete. 2024-03-18T00:54:18.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.043+0000 7f98382c6640 1 Processor -- start 2024-03-18T00:54:18.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.043+0000 7f98382c6640 1 -- start start 2024-03-18T00:54:18.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.043+0000 7f98382c6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9830073690 0x7f98301af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.043+0000 7f98382c6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9830076570 0x7f98301afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.043+0000 7f98382c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9830078f20 0x7f98301baa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.043+0000 7f98382c6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f983007edb0 con 0x7f9830076570 2024-03-18T00:54:18.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.043+0000 7f98382c6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f983007ec30 con 0x7f9830073690 2024-03-18T00:54:18.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.043+0000 7f98382c6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f983007ef30 con 0x7f9830078f20 2024-03-18T00:54:18.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.044+0000 7f9836863640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9830078f20 0x7f98301baa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.044+0000 7f9836863640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9830078f20 0x7f98301baa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60280/0 (socket says 172.21.15.49:60280) 2024-03-18T00:54:18.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.044+0000 7f9836863640 1 -- 172.21.15.49:0/1972386033 learned_addr learned my addr 172.21.15.49:0/1972386033 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.045+0000 7f9836863640 1 -- 172.21.15.49:0/1972386033 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9830073690 msgr2=0x7f98301af720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:54:18.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.045+0000 7f9836863640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9830073690 0x7f98301af720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.045+0000 7f9836863640 1 -- 172.21.15.49:0/1972386033 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9830076570 msgr2=0x7f98301afc60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:54:18.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.045+0000 7f9836863640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9830076570 0x7f98301afc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.045+0000 7f9836863640 1 -- 172.21.15.49:0/1972386033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98301b46e0 con 0x7f9830078f20 2024-03-18T00:54:18.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.046+0000 7f9836863640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9830078f20 0x7f98301baa80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f982000d720 tx=0x7f982000dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.054+0000 7f98277fe640 1 -- 172.21.15.49:0/1972386033 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820017070 con 0x7f9830078f20 2024-03-18T00:54:18.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.054+0000 7f98277fe640 1 -- 172.21.15.49:0/1972386033 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f982000de30 con 0x7f9830078f20 2024-03-18T00:54:18.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.054+0000 7f98382c6640 1 -- 172.21.15.49:0/1972386033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98301b49d0 con 0x7f9830078f20 2024-03-18T00:54:18.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.054+0000 7f98382c6640 1 -- 172.21.15.49:0/1972386033 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98301b4f20 con 0x7f9830078f20 2024-03-18T00:54:18.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.055+0000 7f98277fe640 1 -- 172.21.15.49:0/1972386033 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820004d90 con 0x7f9830078f20 2024-03-18T00:54:18.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.059+0000 7f98277fe640 1 -- 172.21.15.49:0/1972386033 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9820010460 con 0x7f9830078f20 2024-03-18T00:54:18.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.062+0000 7f983af90740 1 Processor -- start 2024-03-18T00:54:18.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.063+0000 7f98277fe640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f981c0a2a00 0x7f981c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.064+0000 7f98277fe640 1 -- 172.21.15.49:0/1972386033 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f98200cf470 con 0x7f9830078f20 2024-03-18T00:54:18.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.065+0000 7f983af90740 1 -- start start 2024-03-18T00:54:18.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.065+0000 7f983af90740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e51850 0x55fba2e51c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.065+0000 7f983af90740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 0x55fba2e56d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.065+0000 7f983af90740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e573b0 0x55fba2e59840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.065+0000 7f983af90740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55fba2db6490 con 0x55fba2e573b0 2024-03-18T00:54:18.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.065+0000 7f983af90740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55fba2d5ec90 con 0x55fba2e51850 2024-03-18T00:54:18.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.065+0000 7f983af90740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fba2db2530 con 0x55fba2e52240 2024-03-18T00:54:18.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.066+0000 7f9835861640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f981c0a2a00 0x7f981c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.067+0000 7f9836863640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e573b0 0x55fba2e59840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.067+0000 7f9836863640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e573b0 0x55fba2e59840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47436/0 (socket says 172.21.15.49:47436) 2024-03-18T00:54:18.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.067+0000 7f9836863640 1 -- 172.21.15.49:0/345762236 learned_addr learned my addr 172.21.15.49:0/345762236 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.067+0000 7f9835861640 1 --2- 172.21.15.49:0/345762236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 0x55fba2e56d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.070+0000 7f9835861640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f981c0a2a00 0x7f981c0a4ec0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9828005fd0 tx=0x7f98280077b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.070+0000 7f9835861640 1 -- 172.21.15.49:0/345762236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e51850 msgr2=0x55fba2e51c70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:18.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.070+0000 7f9835861640 1 --2- 172.21.15.49:0/345762236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e51850 0x55fba2e51c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.070+0000 7f9835861640 1 -- 172.21.15.49:0/345762236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e573b0 msgr2=0x55fba2e59840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.070+0000 7f9835861640 1 --2- 172.21.15.49:0/345762236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e573b0 0x55fba2e59840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.070+0000 7f9835861640 1 -- 172.21.15.49:0/345762236 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba2e19500 con 0x55fba2e52240 2024-03-18T00:54:18.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.072+0000 7f9835861640 1 --2- 172.21.15.49:0/345762236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 0x55fba2e56d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9828005c40 tx=0x7f982801a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b29074b6097da2f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.074+0000 7f9824bfe640 1 -- 172.21.15.49:0/345762236 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982801d860 con 0x55fba2e52240 2024-03-18T00:54:18.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.074+0000 7f9824bfe640 1 -- 172.21.15.49:0/345762236 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9828022070 con 0x55fba2e52240 2024-03-18T00:54:18.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.076+0000 7f983af90740 1 -- 172.21.15.49:0/345762236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 msgr2=0x55fba2e56d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.076+0000 7f983af90740 1 --2- 172.21.15.49:0/345762236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 0x55fba2e56d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9828005c40 tx=0x7f982801a000 comp rx=0 tx=0).stop 2024-03-18T00:54:18.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.076+0000 7f983af90740 1 -- 172.21.15.49:0/345762236 shutdown_connections 2024-03-18T00:54:18.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.076+0000 7f983af90740 1 --2- 172.21.15.49:0/345762236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e573b0 0x55fba2e59840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.076+0000 7f983af90740 1 --2- 172.21.15.49:0/345762236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 0x55fba2e56d40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.076+0000 7f983af90740 1 --2- 172.21.15.49:0/345762236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e51850 0x55fba2e51c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.076+0000 7f983af90740 1 -- 172.21.15.49:0/345762236 >> 172.21.15.49:0/345762236 conn(0x55fba2e4cdc0 msgr2=0x55fba2e4d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.076+0000 7f983af90740 1 -- 172.21.15.49:0/345762236 shutdown_connections 2024-03-18T00:54:18.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.077+0000 7f983af90740 1 -- 172.21.15.49:0/345762236 wait complete. 2024-03-18T00:54:18.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.077+0000 7f983af90740 1 -- start start 2024-03-18T00:54:18.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.077+0000 7f983af90740 1 Processor -- start 2024-03-18T00:54:18.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.078+0000 7f983af90740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e51850 0x55fba2e5bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.078+0000 7f983af90740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 0x55fba2e5c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.078+0000 7f983af90740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 0x55fba2e632f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.078+0000 7f983af90740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55fba2e5a350 con 0x55fba2e52240 2024-03-18T00:54:18.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.078+0000 7f983af90740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55fba2d72230 con 0x55fba2e573b0 2024-03-18T00:54:18.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.078+0000 7f983af90740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fba2e5a4d0 con 0x55fba2e51850 2024-03-18T00:54:18.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.078+0000 7f983af90740 10 client.0 _ll_register_callbacks cb 0x55fba2dae010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:18.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.078+0000 7f9836062640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e51850 0x55fba2e5bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.079+0000 7f9836062640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e51850 0x55fba2e5bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60296/0 (socket says 172.21.15.49:60296) 2024-03-18T00:54:18.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.079+0000 7f9836062640 1 -- 172.21.15.49:0/2485026430 learned_addr learned my addr 172.21.15.49:0/2485026430 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.079+0000 7f9835861640 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 0x55fba2e5c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.079+0000 7f9836863640 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 0x55fba2e632f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.079+0000 7f9836062640 1 -- 172.21.15.49:0/2485026430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 msgr2=0x55fba2e632f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.079+0000 7f9836062640 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 0x55fba2e632f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.079+0000 7f9836062640 1 -- 172.21.15.49:0/2485026430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 msgr2=0x55fba2e5c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.079+0000 7f9836062640 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 0x55fba2e5c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.079+0000 7f9836062640 1 -- 172.21.15.49:0/2485026430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba2d81100 con 0x55fba2e51850 2024-03-18T00:54:18.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.080+0000 7f9836062640 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e51850 0x55fba2e5bd70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f982c009230 tx=0x7f982c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=385c7d881ef9e012 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.080+0000 7f9819ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:54:18.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.080+0000 7f9819ffb640 1 -- 172.21.15.49:0/2485026430 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c039040 con 0x55fba2e51850 2024-03-18T00:54:18.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.080+0000 7f9819ffb640 1 -- 172.21.15.49:0/2485026430 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f982c0043b0 con 0x55fba2e51850 2024-03-18T00:54:18.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.081+0000 7f983af90740 1 -- 172.21.15.49:0/2485026430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9828022070 con 0x55fba2e51850 2024-03-18T00:54:18.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.082+0000 7f9819ffb640 1 -- 172.21.15.49:0/2485026430 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f982c02c8e0 con 0x55fba2e51850 2024-03-18T00:54:18.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.083+0000 7f983af90740 10 client.25993 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:18.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.083+0000 7f983af90740 1 -- 172.21.15.49:0/2485026430 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fba2aca3f0 con 0x55fba2e51850 2024-03-18T00:54:18.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.086+0000 7f9819ffb640 1 -- 172.21.15.49:0/2485026430 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9828022070 con 0x55fba2e51850 2024-03-18T00:54:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.087+0000 7f9819ffb640 1 -- 172.21.15.49:0/2485026430 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f982c03c070 con 0x55fba2e51850 2024-03-18T00:54:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.087+0000 7f983af90740 10 client.25993 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.087+0000 7f983af90740 10 client.25993 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:54:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.087+0000 7f983af90740 20 client.25993 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.087+0000 7f983af90740 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55fba2e78c80 0x55fba2e7b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.087+0000 7f983af90740 4 client.25993 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:18.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.087+0000 7f983af90740 1 -- 172.21.15.49:0/2485026430 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fba2dc39d0 con 0x55fba2e78c80 2024-03-18T00:54:18.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.088+0000 7f9835861640 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55fba2e78c80 0x55fba2e7b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.093+0000 7f9835861640 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55fba2e78c80 0x55fba2e7b100 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a2c27cd9121dd5de server_cookie=95ee2fbcb949e8f8 in_seq=0 out_seq=0 2024-03-18T00:54:18.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.093+0000 7f9819ffb640 10 client.25993 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:54:18.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.096+0000 7f9819ffb640 1 -- 172.21.15.49:0/2485026430 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fba2dc39d0 con 0x55fba2e78c80 2024-03-18T00:54:18.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.096+0000 7f9819ffb640 10 client.25993 handle_command_reply: tid=0 2024-03-18T00:54:18.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.098+0000 7f983af90740 1 client.25993 shutdown 2024-03-18T00:54:18.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.108+0000 7f983af90740 1 -- 172.21.15.49:0/2485026430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e51850 msgr2=0x55fba2e5bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.108+0000 7f983af90740 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e51850 0x55fba2e5bd70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f982c009230 tx=0x7f982c004040 comp rx=0 tx=0).stop 2024-03-18T00:54:18.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.110+0000 7f983af90740 1 -- 172.21.15.49:0/2485026430 shutdown_connections 2024-03-18T00:54:18.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.110+0000 7f983af90740 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55fba2e78c80 0x55fba2e7b100 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.110+0000 7f983af90740 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 0x55fba2e632f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.110+0000 7f983af90740 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 0x55fba2e5c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.110+0000 7f983af90740 1 --2- 172.21.15.49:0/2485026430 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e51850 0x55fba2e5bd70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.110+0000 7f983af90740 1 -- 172.21.15.49:0/2485026430 >> 172.21.15.49:0/2485026430 conn(0x55fba2e4cdc0 msgr2=0x55fba2e4bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.110+0000 7f9819ffb640 0 client.25993 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:54:18.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.111+0000 7f983af90740 1 -- 172.21.15.49:0/2485026430 shutdown_connections 2024-03-18T00:54:18.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.111+0000 7f983af90740 1 -- 172.21.15.49:0/2485026430 wait complete. 2024-03-18T00:54:18.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.115+0000 7f983af90740 20 client.25993 trim_cache size 0 max 16384 2024-03-18T00:54:18.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.165+0000 7f983af90740 1 Processor -- start 2024-03-18T00:54:18.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.165+0000 7f983af90740 1 -- start start 2024-03-18T00:54:18.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.165+0000 7f983af90740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e51850 0x55fba2e50730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.166+0000 7f983af90740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 0x55fba2e5b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.166+0000 7f983af90740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 0x55fba2e5b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.166+0000 7f983af90740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f981400e3e0 con 0x55fba2e51850 2024-03-18T00:54:18.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.166+0000 7f983af90740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f981400ff20 con 0x55fba2e573b0 2024-03-18T00:54:18.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.166+0000 7f983af90740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f981400eab0 con 0x55fba2e52240 2024-03-18T00:54:18.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.170+0000 7f9836062640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e51850 0x55fba2e50730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.170+0000 7f9836062640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e51850 0x55fba2e50730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47444/0 (socket says 172.21.15.49:47444) 2024-03-18T00:54:18.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.170+0000 7f9836062640 1 -- 172.21.15.49:0/1262968079 learned_addr learned my addr 172.21.15.49:0/1262968079 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.171+0000 7f9836863640 1 --2- 172.21.15.49:0/1262968079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 0x55fba2e5b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.171+0000 7f9835861640 1 --2- 172.21.15.49:0/1262968079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 0x55fba2e5b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.174+0000 7f9836062640 1 -- 172.21.15.49:0/1262968079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 msgr2=0x55fba2e5b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.174+0000 7f9836062640 1 --2- 172.21.15.49:0/1262968079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 0x55fba2e5b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.174+0000 7f9836062640 1 -- 172.21.15.49:0/1262968079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 msgr2=0x55fba2e5b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.174+0000 7f9836062640 1 --2- 172.21.15.49:0/1262968079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 0x55fba2e5b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.174+0000 7f9836062640 1 -- 172.21.15.49:0/1262968079 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba2b947d0 con 0x55fba2e51850 2024-03-18T00:54:18.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.181+0000 7f9836062640 1 --2- 172.21.15.49:0/1262968079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e51850 0x55fba2e50730 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f982c003dd0 tx=0x7f982c02c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b96a86cb1f48aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.189+0000 7f981bfff640 1 -- 172.21.15.49:0/1262968079 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c004fb0 con 0x55fba2e51850 2024-03-18T00:54:18.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.189+0000 7f981bfff640 1 -- 172.21.15.49:0/1262968079 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f982c042070 con 0x55fba2e51850 2024-03-18T00:54:18.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.189+0000 7f981bfff640 1 -- 172.21.15.49:0/1262968079 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c0035f0 con 0x55fba2e51850 2024-03-18T00:54:18.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.193+0000 7f983af90740 1 -- 172.21.15.49:0/1262968079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e51850 msgr2=0x55fba2e50730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.193+0000 7f983af90740 1 --2- 172.21.15.49:0/1262968079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e51850 0x55fba2e50730 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f982c003dd0 tx=0x7f982c02c1a0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.199+0000 7f983af90740 1 -- 172.21.15.49:0/1262968079 shutdown_connections 2024-03-18T00:54:18.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.199+0000 7f983af90740 1 --2- 172.21.15.49:0/1262968079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e573b0 0x55fba2e5b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.199+0000 7f983af90740 1 --2- 172.21.15.49:0/1262968079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e52240 0x55fba2e5b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.199+0000 7f983af90740 1 --2- 172.21.15.49:0/1262968079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e51850 0x55fba2e50730 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.199+0000 7f983af90740 1 -- 172.21.15.49:0/1262968079 >> 172.21.15.49:0/1262968079 conn(0x55fba2e4cdc0 msgr2=0x55fba2e50020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.203+0000 7f983af90740 1 -- 172.21.15.49:0/1262968079 shutdown_connections 2024-03-18T00:54:18.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.203+0000 7f983af90740 1 -- 172.21.15.49:0/1262968079 wait complete. 2024-03-18T00:54:18.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.204+0000 7f983af90740 1 -- start start 2024-03-18T00:54:18.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.204+0000 7f983af90740 1 Processor -- start 2024-03-18T00:54:18.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.205+0000 7f983af90740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e51850 0x55fba2e603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.205+0000 7f983af90740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 0x55fba2e60910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.205+0000 7f983af90740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e573b0 0x55fba2e61920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.206+0000 7f983af90740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f981400f850 con 0x55fba2e52240 2024-03-18T00:54:18.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.206+0000 7f983af90740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f981400f180 con 0x55fba2e51850 2024-03-18T00:54:18.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.206+0000 7f983af90740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f981400c470 con 0x55fba2e573b0 2024-03-18T00:54:18.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.206+0000 7f983af90740 10 client.0 _ll_register_callbacks cb 0x7f982c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:18.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.206+0000 7f9836863640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e573b0 0x55fba2e61920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.206+0000 7f9836863640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e573b0 0x55fba2e61920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60318/0 (socket says 172.21.15.49:60318) 2024-03-18T00:54:18.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.206+0000 7f9836863640 1 -- 172.21.15.49:0/3171426550 learned_addr learned my addr 172.21.15.49:0/3171426550 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.207+0000 7f9836062640 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e51850 0x55fba2e603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.208+0000 7f9835861640 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 0x55fba2e60910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.208+0000 7f9836863640 1 -- 172.21.15.49:0/3171426550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e51850 msgr2=0x55fba2e603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.208+0000 7f9836863640 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e51850 0x55fba2e603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.208+0000 7f9836863640 1 -- 172.21.15.49:0/3171426550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 msgr2=0x55fba2e60910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.208+0000 7f9836863640 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 0x55fba2e60910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.208+0000 7f9836863640 1 -- 172.21.15.49:0/3171426550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba2e61f10 con 0x55fba2e573b0 2024-03-18T00:54:18.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.210+0000 7f9836863640 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e573b0 0x55fba2e61920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f982009ba90 tx=0x7f982009ffa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e83add1fc02ec9f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.212+0000 7f9819ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:54:18.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.212+0000 7f9819ffb640 1 -- 172.21.15.49:0/3171426550 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98200b8020 con 0x55fba2e573b0 2024-03-18T00:54:18.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.212+0000 7f9819ffb640 1 -- 172.21.15.49:0/3171426550 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9820098400 con 0x55fba2e573b0 2024-03-18T00:54:18.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.212+0000 7f983af90740 1 -- 172.21.15.49:0/3171426550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f982c042070 con 0x55fba2e573b0 2024-03-18T00:54:18.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.222+0000 7f9819ffb640 1 -- 172.21.15.49:0/3171426550 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f98200b1040 con 0x55fba2e573b0 2024-03-18T00:54:18.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.222+0000 7f983af90740 10 client.25999 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:18.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.222+0000 7f983af90740 1 -- 172.21.15.49:0/3171426550 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fba2e632c0 con 0x55fba2e573b0 2024-03-18T00:54:18.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.223+0000 7f9819ffb640 1 -- 172.21.15.49:0/3171426550 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f982c042070 con 0x55fba2e573b0 2024-03-18T00:54:18.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.223+0000 7f9819ffb640 1 -- 172.21.15.49:0/3171426550 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f982009b300 con 0x55fba2e573b0 2024-03-18T00:54:18.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.223+0000 7f983af90740 10 client.25999 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:18.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.223+0000 7f983af90740 10 client.25999 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:54:18.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.223+0000 7f983af90740 20 client.25999 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:18.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.223+0000 7f983af90740 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55fba2edf840 0x55fba2ee1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.223+0000 7f983af90740 4 client.25999 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:54:18.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.223+0000 7f983af90740 1 -- 172.21.15.49:0/3171426550 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fba2dbdd20 con 0x55fba2edf840 2024-03-18T00:54:18.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.226+0000 7f9836062640 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55fba2edf840 0x55fba2ee1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.228+0000 7f9836062640 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55fba2edf840 0x55fba2ee1c60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c0ed23771dcf20a server_cookie=a2e118f744da50a9 in_seq=0 out_seq=0 2024-03-18T00:54:18.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.228+0000 7f9819ffb640 10 client.25999 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:54:18.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.232+0000 7f9819ffb640 1 -- 172.21.15.49:0/3171426550 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fba2dbdd20 con 0x55fba2edf840 2024-03-18T00:54:18.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.232+0000 7f9819ffb640 10 client.25999 handle_command_reply: tid=0 2024-03-18T00:54:18.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.233+0000 7f983af90740 1 client.25999 shutdown 2024-03-18T00:54:18.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.235+0000 7f983af90740 1 -- 172.21.15.49:0/3171426550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e573b0 msgr2=0x55fba2e61920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.235+0000 7f983af90740 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e573b0 0x55fba2e61920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f982009ba90 tx=0x7f982009ffa0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.236+0000 7f983af90740 1 -- 172.21.15.49:0/3171426550 shutdown_connections 2024-03-18T00:54:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.236+0000 7f983af90740 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55fba2edf840 0x55fba2ee1c60 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.236+0000 7f983af90740 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fba2e573b0 0x55fba2e61920 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.236+0000 7f983af90740 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fba2e52240 0x55fba2e60910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.236+0000 7f983af90740 1 --2- 172.21.15.49:0/3171426550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fba2e51850 0x55fba2e603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.236+0000 7f983af90740 1 -- 172.21.15.49:0/3171426550 >> 172.21.15.49:0/3171426550 conn(0x55fba2e4cdc0 msgr2=0x55fba2e508b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.236+0000 7f9819ffb640 0 client.25999 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:54:18.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.236+0000 7f983af90740 1 -- 172.21.15.49:0/3171426550 shutdown_connections 2024-03-18T00:54:18.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.236+0000 7f983af90740 1 -- 172.21.15.49:0/3171426550 wait complete. 2024-03-18T00:54:18.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.237+0000 7f983af90740 20 client.25999 trim_cache size 0 max 16384 2024-03-18T00:54:18.239 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:54:18.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.253+0000 7f98382c6640 1 -- 172.21.15.49:0/1972386033 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f981c0a2a00 msgr2=0x7f981c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.253+0000 7f98382c6640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f981c0a2a00 0x7f981c0a4ec0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9828005fd0 tx=0x7f98280077b0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.253+0000 7f98382c6640 1 -- 172.21.15.49:0/1972386033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9830078f20 msgr2=0x7f98301baa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.253+0000 7f98382c6640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9830078f20 0x7f98301baa80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f982000d720 tx=0x7f982000dbe0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.257+0000 7f98382c6640 1 -- 172.21.15.49:0/1972386033 shutdown_connections 2024-03-18T00:54:18.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.257+0000 7f98382c6640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f981c0a2a00 0x7f981c0a4ec0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.257+0000 7f98382c6640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9830078f20 0x7f98301baa80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.257+0000 7f98382c6640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9830076570 0x7f98301afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.257+0000 7f98382c6640 1 --2- 172.21.15.49:0/1972386033 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9830073690 0x7f98301af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.257+0000 7f98382c6640 1 -- 172.21.15.49:0/1972386033 >> 172.21.15.49:0/1972386033 conn(0x7f983006e640 msgr2=0x7f9830077420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.259+0000 7f98382c6640 1 -- 172.21.15.49:0/1972386033 shutdown_connections 2024-03-18T00:54:18.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.259+0000 7f98382c6640 1 -- 172.21.15.49:0/1972386033 wait complete. 2024-03-18T00:54:18.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:54:18.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.741+0000 7f2b8830c640 1 -- 172.21.15.49:0/3098382587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b8010b150 msgr2=0x7f2b8010d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.741+0000 7f2b8830c640 1 --2- 172.21.15.49:0/3098382587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b8010b150 0x7f2b8010d5c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009be0 tx=0x7f2b7802bf10 comp rx=0 tx=0).stop 2024-03-18T00:54:18.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.742+0000 7f2b8830c640 1 -- 172.21.15.49:0/3098382587 shutdown_connections 2024-03-18T00:54:18.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.742+0000 7f2b8830c640 1 --2- 172.21.15.49:0/3098382587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b8010b150 0x7f2b8010d5c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.742+0000 7f2b8830c640 1 --2- 172.21.15.49:0/3098382587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b8010db00 0x7f2b8010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.742+0000 7f2b8830c640 1 --2- 172.21.15.49:0/3098382587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b801087e0 0x7f2b8010ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.742+0000 7f2b8830c640 1 -- 172.21.15.49:0/3098382587 >> 172.21.15.49:0/3098382587 conn(0x7f2b8006e640 msgr2=0x7f2b80070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.749+0000 7f2b8830c640 1 -- 172.21.15.49:0/3098382587 shutdown_connections 2024-03-18T00:54:18.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.752+0000 7f2b8830c640 1 -- 172.21.15.49:0/3098382587 wait complete. 2024-03-18T00:54:18.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.753+0000 7f2b8830c640 1 Processor -- start 2024-03-18T00:54:18.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.755+0000 7f2b8830c640 1 -- start start 2024-03-18T00:54:18.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.755+0000 7f2b8830c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b801087e0 0x7f2b80119aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.755+0000 7f2b8830c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b8010b150 0x7f2b80119fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.755+0000 7f2b8830c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b8010db00 0x7f2b801180e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.755+0000 7f2b8830c640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2b80110900 con 0x7f2b8010b150 2024-03-18T00:54:18.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.755+0000 7f2b8830c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2b80110780 con 0x7f2b8010db00 2024-03-18T00:54:18.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.755+0000 7f2b8830c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2b80110a80 con 0x7f2b801087e0 2024-03-18T00:54:18.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.757+0000 7f2b858a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b8010b150 0x7f2b80119fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.757+0000 7f2b858a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b8010b150 0x7f2b80119fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47458/0 (socket says 172.21.15.49:47458) 2024-03-18T00:54:18.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.757+0000 7f2b858a7640 1 -- 172.21.15.49:0/1812326773 learned_addr learned my addr 172.21.15.49:0/1812326773 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.760+0000 7f2b858a7640 1 -- 172.21.15.49:0/1812326773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b801087e0 msgr2=0x7f2b80119aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:54:18.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.760+0000 7f2b858a7640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b801087e0 0x7f2b80119aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.760+0000 7f2b858a7640 1 -- 172.21.15.49:0/1812326773 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b8010db00 msgr2=0x7f2b801180e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.760+0000 7f2b858a7640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b8010db00 0x7f2b801180e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.760+0000 7f2b858a7640 1 -- 172.21.15.49:0/1812326773 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b80118860 con 0x7f2b8010b150 2024-03-18T00:54:18.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.760+0000 7f2b858a7640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b8010b150 0x7f2b80119fe0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2b7802c520 tx=0x7f2b780338c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.762+0000 7f2b777fe640 1 -- 172.21.15.49:0/1812326773 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78043020 con 0x7f2b8010b150 2024-03-18T00:54:18.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.762+0000 7f2b777fe640 1 -- 172.21.15.49:0/1812326773 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b78003c40 con 0x7f2b8010b150 2024-03-18T00:54:18.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.762+0000 7f2b777fe640 1 -- 172.21.15.49:0/1812326773 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78034890 con 0x7f2b8010b150 2024-03-18T00:54:18.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.763+0000 7f2b8830c640 1 -- 172.21.15.49:0/1812326773 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b80118af0 con 0x7f2b8010b150 2024-03-18T00:54:18.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.763+0000 7f2b8830c640 1 -- 172.21.15.49:0/1812326773 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b801b75e0 con 0x7f2b8010b150 2024-03-18T00:54:18.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.769+0000 7f2b8afcb740 1 Processor -- start 2024-03-18T00:54:18.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.769+0000 7f2b8afcb740 1 -- start start 2024-03-18T00:54:18.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.770+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56407989b850 0x56407989bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.770+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.770+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5640798a13b0 0x5640798a3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.770+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564079800490 con 0x5640798a13b0 2024-03-18T00:54:18.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.770+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5640797a8c90 con 0x56407989c240 2024-03-18T00:54:18.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.770+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5640797fc530 con 0x56407989b850 2024-03-18T00:54:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.772+0000 7f2b868a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56407989b850 0x56407989bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.772+0000 7f2b860a8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.772+0000 7f2b777fe640 1 -- 172.21.15.49:0/1812326773 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2b78046020 con 0x7f2b8010b150 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.772+0000 7f2b868a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56407989b850 0x56407989bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60350/0 (socket says 172.21.15.49:60350) 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.772+0000 7f2b868a9640 1 -- 172.21.15.49:0/752409960 learned_addr learned my addr 172.21.15.49:0/752409960 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.773+0000 7f2b777fe640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b6c0a2a00 0x7f2b6c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.774+0000 7f2b777fe640 1 -- 172.21.15.49:0/1812326773 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2b780f1480 con 0x7f2b8010b150 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.774+0000 7f2b868a9640 1 -- 172.21.15.49:0/752409960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 msgr2=0x5640798a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.774+0000 7f2b868a9640 1 --2- 172.21.15.49:0/752409960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.774+0000 7f2b868a9640 1 -- 172.21.15.49:0/752409960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5640798a13b0 msgr2=0x5640798a3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.774+0000 7f2b868a9640 1 --2- 172.21.15.49:0/752409960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5640798a13b0 0x5640798a3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.774+0000 7f2b868a9640 1 -- 172.21.15.49:0/752409960 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564079863500 con 0x56407989b850 2024-03-18T00:54:18.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.774+0000 7f2b868a9640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b6c0a2a00 0x7f2b6c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.775+0000 7f2b868a9640 1 --2- 172.21.15.49:0/752409960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56407989b850 0x56407989bc70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c004650 tx=0x7f2b7c00e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3696ead701cca28 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.778+0000 7f2b74bfe640 1 -- 172.21.15.49:0/752409960 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c012470 con 0x56407989b850 2024-03-18T00:54:18.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.778+0000 7f2b74bfe640 1 -- 172.21.15.49:0/752409960 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b7c012b70 con 0x56407989b850 2024-03-18T00:54:18.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.779+0000 7f2b8afcb740 1 -- 172.21.15.49:0/752409960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56407989b850 msgr2=0x56407989bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.779+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/752409960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56407989b850 0x56407989bc70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c004650 tx=0x7f2b7c00e720 comp rx=0 tx=0).stop 2024-03-18T00:54:18.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.782+0000 7f2b868a9640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b6c0a2a00 0x7f2b6c0a4ec0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009fd0 tx=0x7f2b7c011460 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.787+0000 7f2b8afcb740 1 -- 172.21.15.49:0/752409960 shutdown_connections 2024-03-18T00:54:18.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.787+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/752409960 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.787+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/752409960 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5640798a13b0 0x5640798a3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.787+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/752409960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56407989b850 0x56407989bc70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.787+0000 7f2b8afcb740 1 -- 172.21.15.49:0/752409960 >> 172.21.15.49:0/752409960 conn(0x564079896dc0 msgr2=0x5640798971c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.792+0000 7f2b8afcb740 1 -- 172.21.15.49:0/752409960 shutdown_connections 2024-03-18T00:54:18.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.797+0000 7f2b8afcb740 1 -- 172.21.15.49:0/752409960 wait complete. 2024-03-18T00:54:18.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.797+0000 7f2b8afcb740 1 -- start start 2024-03-18T00:54:18.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.797+0000 7f2b8afcb740 1 Processor -- start 2024-03-18T00:54:18.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.800+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989b850 0x5640798a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.801+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989c240 0x5640798a6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.801+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.801+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5640798a43b0 con 0x56407989c240 2024-03-18T00:54:18.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.801+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5640797bc230 con 0x56407989b850 2024-03-18T00:54:18.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.801+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5640798a4530 con 0x5640798a13b0 2024-03-18T00:54:18.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.801+0000 7f2b8afcb740 10 client.0 _ll_register_callbacks cb 0x5640797f8010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:18.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.801+0000 7f2b858a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989c240 0x5640798a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.801+0000 7f2b858a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989c240 0x5640798a6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47476/0 (socket says 172.21.15.49:47476) 2024-03-18T00:54:18.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.801+0000 7f2b858a7640 1 -- 172.21.15.49:0/3157929375 learned_addr learned my addr 172.21.15.49:0/3157929375 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.803+0000 7f2b868a9640 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.804+0000 7f2b860a8640 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989b850 0x5640798a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.806+0000 7f2b868a9640 1 -- 172.21.15.49:0/3157929375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989b850 msgr2=0x5640798a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.806+0000 7f2b868a9640 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989b850 0x5640798a5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.806+0000 7f2b868a9640 1 -- 172.21.15.49:0/3157929375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989c240 msgr2=0x5640798a6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.806+0000 7f2b868a9640 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989c240 0x5640798a6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.806+0000 7f2b868a9640 1 -- 172.21.15.49:0/3157929375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640797cb100 con 0x5640798a13b0 2024-03-18T00:54:18.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.811+0000 7f2b868a9640 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ad2b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005ab0 tx=0x7f2b7c015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64045ae2a3c5dbcf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.811+0000 7f2b69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:54:18.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.813+0000 7f2b69ffb640 1 -- 172.21.15.49:0/3157929375 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c032020 con 0x5640798a13b0 2024-03-18T00:54:18.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.814+0000 7f2b69ffb640 1 -- 172.21.15.49:0/3157929375 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2b7c0128a0 con 0x5640798a13b0 2024-03-18T00:54:18.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.815+0000 7f2b8afcb740 1 -- 172.21.15.49:0/3157929375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b7c012b70 con 0x5640798a13b0 2024-03-18T00:54:18.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.817+0000 7f2b69ffb640 1 -- 172.21.15.49:0/3157929375 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2b7c012450 con 0x5640798a13b0 2024-03-18T00:54:18.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.818+0000 7f2b8afcb740 10 client.26008 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:18.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.818+0000 7f2b8afcb740 1 -- 172.21.15.49:0/3157929375 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5640795143f0 con 0x5640798a13b0 2024-03-18T00:54:18.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.819+0000 7f2b69ffb640 1 -- 172.21.15.49:0/3157929375 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b7c012b70 con 0x5640798a13b0 2024-03-18T00:54:18.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.819+0000 7f2b69ffb640 1 -- 172.21.15.49:0/3157929375 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2b7c02e9d0 con 0x5640798a13b0 2024-03-18T00:54:18.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.821+0000 7f2b8afcb740 10 client.26008 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:18.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.821+0000 7f2b8afcb740 10 client.26008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:18.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.821+0000 7f2b8afcb740 20 client.26008 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:18.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.821+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5640798b1e90 0x5640798b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.821+0000 7f2b8afcb740 4 client.26008 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:18.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.821+0000 7f2b8afcb740 1 -- 172.21.15.49:0/3157929375 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56407980d9d0 con 0x5640798b1e90 2024-03-18T00:54:18.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.828+0000 7f2b860a8640 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5640798b1e90 0x5640798b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.840+0000 7f2b69ffb640 10 client.26008 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:18.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.840+0000 7f2b860a8640 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5640798b1e90 0x5640798b42f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ea14378c026882d server_cookie=62783fee7f7b33b1 in_seq=0 out_seq=0 2024-03-18T00:54:18.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.845+0000 7f2b69ffb640 1 -- 172.21.15.49:0/3157929375 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56407980d9d0 con 0x5640798b1e90 2024-03-18T00:54:18.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.846+0000 7f2b69ffb640 10 client.26008 handle_command_reply: tid=0 2024-03-18T00:54:18.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.846+0000 7f2b8afcb740 1 client.26008 shutdown 2024-03-18T00:54:18.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.852+0000 7f2b8afcb740 1 -- 172.21.15.49:0/3157929375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 msgr2=0x5640798ad2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.852+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ad2b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c005ab0 tx=0x7f2b7c015000 comp rx=0 tx=0).stop 2024-03-18T00:54:18.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.854+0000 7f2b8afcb740 1 -- 172.21.15.49:0/3157929375 shutdown_connections 2024-03-18T00:54:18.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.854+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5640798b1e90 0x5640798b42f0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.854+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ad2b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.854+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989c240 0x5640798a6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.854+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/3157929375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989b850 0x5640798a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.854+0000 7f2b8afcb740 1 -- 172.21.15.49:0/3157929375 >> 172.21.15.49:0/3157929375 conn(0x564079896dc0 msgr2=0x564079895fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.858+0000 7f2b69ffb640 0 client.26008 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:18.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.861+0000 7f2b8afcb740 1 -- 172.21.15.49:0/3157929375 shutdown_connections 2024-03-18T00:54:18.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.863+0000 7f2b8afcb740 1 -- 172.21.15.49:0/3157929375 wait complete. 2024-03-18T00:54:18.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.864+0000 7f2b8afcb740 20 client.26008 trim_cache size 0 max 16384 2024-03-18T00:54:18.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.905+0000 7f2b8afcb740 1 Processor -- start 2024-03-18T00:54:18.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.906+0000 7f2b8afcb740 1 -- start start 2024-03-18T00:54:18.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.906+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 0x56407989d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.906+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798ad270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.906+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x56407989dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.907+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400e690 con 0x56407989b850 2024-03-18T00:54:18.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.907+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2b640101d0 con 0x56407989c240 2024-03-18T00:54:18.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.907+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400ed60 con 0x5640798a13b0 2024-03-18T00:54:18.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.907+0000 7f2b858a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798ad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.907+0000 7f2b858a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798ad270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:35760/0 (socket says 172.21.15.49:35760) 2024-03-18T00:54:18.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.907+0000 7f2b858a7640 1 -- 172.21.15.49:0/1142364824 learned_addr learned my addr 172.21.15.49:0/1142364824 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.907+0000 7f2b868a9640 1 --2- 172.21.15.49:0/1142364824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x56407989dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.908+0000 7f2b868a9640 1 -- 172.21.15.49:0/1142364824 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 msgr2=0x5640798ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.908+0000 7f2b868a9640 1 --2- 172.21.15.49:0/1142364824 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798ad270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.908+0000 7f2b868a9640 1 -- 172.21.15.49:0/1142364824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 msgr2=0x56407989d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.908+0000 7f2b868a9640 1 --2- 172.21.15.49:0/1142364824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 0x56407989d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.908+0000 7f2b868a9640 1 -- 172.21.15.49:0/1142364824 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640795de7d0 con 0x5640798a13b0 2024-03-18T00:54:18.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.909+0000 7f2b868a9640 1 --2- 172.21.15.49:0/1142364824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x56407989dbe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c02ef10 tx=0x7f2b7c009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1eb204bbf820441 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.911+0000 7f2b6bfff640 1 -- 172.21.15.49:0/1142364824 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c032040 con 0x5640798a13b0 2024-03-18T00:54:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.912+0000 7f2b6bfff640 1 -- 172.21.15.49:0/1142364824 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b7c011e50 con 0x5640798a13b0 2024-03-18T00:54:18.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.912+0000 7f2b6bfff640 1 -- 172.21.15.49:0/1142364824 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c01a710 con 0x5640798a13b0 2024-03-18T00:54:18.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.912+0000 7f2b8afcb740 1 -- 172.21.15.49:0/1142364824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 msgr2=0x56407989dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.912+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/1142364824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x56407989dbe0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c02ef10 tx=0x7f2b7c009260 comp rx=0 tx=0).stop 2024-03-18T00:54:18.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.913+0000 7f2b8afcb740 1 -- 172.21.15.49:0/1142364824 shutdown_connections 2024-03-18T00:54:18.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.913+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/1142364824 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798ad270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.913+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/1142364824 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x56407989dbe0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.913+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/1142364824 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 0x56407989d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.913+0000 7f2b8afcb740 1 -- 172.21.15.49:0/1142364824 >> 172.21.15.49:0/1142364824 conn(0x564079896dc0 msgr2=0x5640798971a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.913+0000 7f2b8afcb740 1 -- 172.21.15.49:0/1142364824 shutdown_connections 2024-03-18T00:54:18.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.913+0000 7f2b8afcb740 1 -- 172.21.15.49:0/1142364824 wait complete. 2024-03-18T00:54:18.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.914+0000 7f2b8afcb740 1 -- start start 2024-03-18T00:54:18.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.914+0000 7f2b8afcb740 1 Processor -- start 2024-03-18T00:54:18.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.914+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 0x5640798a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.915+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798a9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.915+0000 7f2b8afcb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ac9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.915+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400fb00 con 0x56407989b850 2024-03-18T00:54:18.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.915+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400f430 con 0x56407989c240 2024-03-18T00:54:18.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.915+0000 7f2b8afcb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6400c670 con 0x5640798a13b0 2024-03-18T00:54:18.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.915+0000 7f2b8afcb740 10 client.0 _ll_register_callbacks cb 0x7f2b7c01d020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:18.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.917+0000 7f2b860a8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 0x5640798a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.917+0000 7f2b860a8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 0x5640798a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47482/0 (socket says 172.21.15.49:47482) 2024-03-18T00:54:18.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.917+0000 7f2b860a8640 1 -- 172.21.15.49:0/2463565556 learned_addr learned my addr 172.21.15.49:0/2463565556 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:18.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.917+0000 7f2b868a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ac9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.918+0000 7f2b858a7640 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798a9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.919+0000 7f2b868a9640 1 -- 172.21.15.49:0/2463565556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 msgr2=0x5640798a9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.919+0000 7f2b868a9640 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798a9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.919+0000 7f2b868a9640 1 -- 172.21.15.49:0/2463565556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 msgr2=0x5640798a94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:18.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.919+0000 7f2b868a9640 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 0x5640798a94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.919+0000 7f2b868a9640 1 -- 172.21.15.49:0/2463565556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640798ad010 con 0x5640798a13b0 2024-03-18T00:54:18.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.921+0000 7f2b868a9640 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ac9d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c011ab0 tx=0x7f2b7c0057d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e90e93ec53ec4769 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:18.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.921+0000 7f2b69ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:54:18.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.922+0000 7f2b69ffb640 1 -- 172.21.15.49:0/2463565556 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c032040 con 0x5640798a13b0 2024-03-18T00:54:18.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.922+0000 7f2b8afcb740 1 -- 172.21.15.49:0/2463565556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b7c011e50 con 0x5640798a13b0 2024-03-18T00:54:18.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.922+0000 7f2b69ffb640 1 -- 172.21.15.49:0/2463565556 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2b7c01a920 con 0x5640798a13b0 2024-03-18T00:54:18.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.923+0000 7f2b69ffb640 1 -- 172.21.15.49:0/2463565556 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2b7c0058f0 con 0x5640798a13b0 2024-03-18T00:54:18.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.924+0000 7f2b8afcb740 10 client.26014 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:18.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.924+0000 7f2b8afcb740 1 -- 172.21.15.49:0/2463565556 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564079916d20 con 0x5640798a13b0 2024-03-18T00:54:18.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.925+0000 7f2b69ffb640 1 -- 172.21.15.49:0/2463565556 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b7c011e50 con 0x5640798a13b0 2024-03-18T00:54:18.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.925+0000 7f2b69ffb640 1 -- 172.21.15.49:0/2463565556 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2b7c02e530 con 0x5640798a13b0 2024-03-18T00:54:18.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.925+0000 7f2b8afcb740 10 client.26014 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:18.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.925+0000 7f2b8afcb740 10 client.26014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:18.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.925+0000 7f2b8afcb740 20 client.26014 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:18.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.925+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564079917690 0x564079919ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:18.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.925+0000 7f2b8afcb740 4 client.26014 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:54:18.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.925+0000 7f2b8afcb740 1 -- 172.21.15.49:0/2463565556 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564079807d20 con 0x564079917690 2024-03-18T00:54:18.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.926+0000 7f2b860a8640 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564079917690 0x564079919ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:18.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.928+0000 7f2b69ffb640 10 client.26014 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:18.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.928+0000 7f2b860a8640 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564079917690 0x564079919ab0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=297321e1659edd80 server_cookie=22929c210446ab4d in_seq=0 out_seq=0 2024-03-18T00:54:18.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.932+0000 7f2b69ffb640 1 -- 172.21.15.49:0/2463565556 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564079807d20 con 0x564079917690 2024-03-18T00:54:18.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.932+0000 7f2b69ffb640 10 client.26014 handle_command_reply: tid=0 2024-03-18T00:54:18.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.932+0000 7f2b8afcb740 1 client.26014 shutdown 2024-03-18T00:54:18.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b8afcb740 1 -- 172.21.15.49:0/2463565556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 msgr2=0x5640798ac9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ac9d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c011ab0 tx=0x7f2b7c0057d0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b8afcb740 1 -- 172.21.15.49:0/2463565556 shutdown_connections 2024-03-18T00:54:18.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564079917690 0x564079919ab0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56407989c240 0x5640798a9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5640798a13b0 0x5640798ac9d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b8afcb740 1 --2- 172.21.15.49:0/2463565556 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56407989b850 0x5640798a94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b69ffb640 0 client.26014 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:18.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b8afcb740 1 -- 172.21.15.49:0/2463565556 >> 172.21.15.49:0/2463565556 conn(0x564079896dc0 msgr2=0x56407989d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.933+0000 7f2b8afcb740 1 -- 172.21.15.49:0/2463565556 shutdown_connections 2024-03-18T00:54:18.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.937+0000 7f2b8afcb740 1 -- 172.21.15.49:0/2463565556 wait complete. 2024-03-18T00:54:18.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.938+0000 7f2b8afcb740 20 client.26014 trim_cache size 0 max 16384 2024-03-18T00:54:18.940 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:54:18.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.947+0000 7f2b8830c640 1 -- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b6c0a2a00 msgr2=0x7f2b6c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.947+0000 7f2b8830c640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b6c0a2a00 0x7f2b6c0a4ec0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c009fd0 tx=0x7f2b7c011460 comp rx=0 tx=0).stop 2024-03-18T00:54:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.947+0000 7f2b8830c640 1 -- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b8010b150 msgr2=0x7f2b80119fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.947+0000 7f2b8830c640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b8010b150 0x7f2b80119fe0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2b7802c520 tx=0x7f2b780338c0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.947+0000 7f2b8830c640 1 -- 172.21.15.49:0/1812326773 shutdown_connections 2024-03-18T00:54:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.948+0000 7f2b8830c640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b6c0a2a00 0x7f2b6c0a4ec0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.948+0000 7f2b8830c640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b8010db00 0x7f2b801180e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.948+0000 7f2b8830c640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b8010b150 0x7f2b80119fe0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.948+0000 7f2b8830c640 1 --2- 172.21.15.49:0/1812326773 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b801087e0 0x7f2b80119aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:18.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.948+0000 7f2b8830c640 1 -- 172.21.15.49:0/1812326773 >> 172.21.15.49:0/1812326773 conn(0x7f2b8006e640 msgr2=0x7f2b8010ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:18.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.949+0000 7f2b8830c640 1 -- 172.21.15.49:0/1812326773 shutdown_connections 2024-03-18T00:54:18.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:18.949+0000 7f2b8830c640 1 -- 172.21.15.49:0/1812326773 wait complete. 2024-03-18T00:54:19.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:19 smithi049 ceph-mon[32130]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 849 MiB/s wr, 1.29k op/s 2024-03-18T00:54:19.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:19 smithi107 ceph-mon[36270]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 849 MiB/s wr, 1.29k op/s 2024-03-18T00:54:19.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:19 smithi106 ceph-mon[35899]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 849 MiB/s wr, 1.29k op/s 2024-03-18T00:54:19.982 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:54:19.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:54:19.982 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:54:20.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.398+0000 7fcfd811b640 1 -- 172.21.15.49:0/793686131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfd01099b0 msgr2=0x7fcfd0110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:20.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.398+0000 7fcfd811b640 1 --2- 172.21.15.49:0/793686131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfd01099b0 0x7fcfd0110240 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009be0 tx=0x7fcfc802bf10 comp rx=0 tx=0).stop 2024-03-18T00:54:20.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.399+0000 7fcfd811b640 1 -- 172.21.15.49:0/793686131 shutdown_connections 2024-03-18T00:54:20.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.399+0000 7fcfd811b640 1 --2- 172.21.15.49:0/793686131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfd01099b0 0x7fcfd0110240 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.399+0000 7fcfd811b640 1 --2- 172.21.15.49:0/793686131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd0108ff0 0x7fcfd0109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.399+0000 7fcfd811b640 1 --2- 172.21.15.49:0/793686131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfd0107e00 0x7fcfd0108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.399+0000 7fcfd811b640 1 -- 172.21.15.49:0/793686131 >> 172.21.15.49:0/793686131 conn(0x7fcfd0076da0 msgr2=0x7fcfd00791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:20.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.403+0000 7fcfd811b640 1 -- 172.21.15.49:0/793686131 shutdown_connections 2024-03-18T00:54:20.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.405+0000 7fcfd811b640 1 -- 172.21.15.49:0/793686131 wait complete. 2024-03-18T00:54:20.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.406+0000 7fcfd811b640 1 Processor -- start 2024-03-18T00:54:20.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.406+0000 7fcfd811b640 1 -- start start 2024-03-18T00:54:20.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.406+0000 7fcfd811b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfd0107e00 0x7fcfd00701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.406+0000 7fcfd811b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd0108ff0 0x7fcfd0070700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.406+0000 7fcfd811b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfd01099b0 0x7fcfd007dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.407+0000 7fcfd811b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0113ab0 con 0x7fcfd01099b0 2024-03-18T00:54:20.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.407+0000 7fcfd811b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0113930 con 0x7fcfd0107e00 2024-03-18T00:54:20.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.407+0000 7fcfd811b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0113c30 con 0x7fcfd0108ff0 2024-03-18T00:54:20.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.407+0000 7fcfd56b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd0108ff0 0x7fcfd0070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.407+0000 7fcfd56b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd0108ff0 0x7fcfd0070700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60394/0 (socket says 172.21.15.49:60394) 2024-03-18T00:54:20.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.407+0000 7fcfd56b6640 1 -- 172.21.15.49:0/434548510 learned_addr learned my addr 172.21.15.49:0/434548510 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:20.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.407+0000 7fcfd66b8640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfd01099b0 0x7fcfd007dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.408+0000 7fcfd56b6640 1 -- 172.21.15.49:0/434548510 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfd0107e00 msgr2=0x7fcfd00701c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:54:20.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.408+0000 7fcfd56b6640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfd0107e00 0x7fcfd00701c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.408+0000 7fcfd56b6640 1 -- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfd01099b0 msgr2=0x7fcfd007dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:20.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.408+0000 7fcfd56b6640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfd01099b0 0x7fcfd007dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.408+0000 7fcfd56b6640 1 -- 172.21.15.49:0/434548510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd007e510 con 0x7fcfd0108ff0 2024-03-18T00:54:20.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.410+0000 7fcfd56b6640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd0108ff0 0x7fcfd0070700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000e9f0 tx=0x7fcfc000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:20.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.413+0000 7fcfc6ffd640 1 -- 172.21.15.49:0/434548510 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc000cd80 con 0x7fcfd0108ff0 2024-03-18T00:54:20.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.413+0000 7fcfd811b640 1 -- 172.21.15.49:0/434548510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd007e800 con 0x7fcfd0108ff0 2024-03-18T00:54:20.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.413+0000 7fcfd811b640 1 -- 172.21.15.49:0/434548510 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd007eda0 con 0x7fcfd0108ff0 2024-03-18T00:54:20.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.414+0000 7fcfc6ffd640 1 -- 172.21.15.49:0/434548510 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc0010910 con 0x7fcfd0108ff0 2024-03-18T00:54:20.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.414+0000 7fcfc6ffd640 1 -- 172.21.15.49:0/434548510 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc0018990 con 0x7fcfd0108ff0 2024-03-18T00:54:20.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.420+0000 7fcfdade1740 1 Processor -- start 2024-03-18T00:54:20.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.422+0000 7fcfdade1740 1 -- start start 2024-03-18T00:54:20.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.422+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49edcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.422+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.422+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49ee24f0 0x555d49ee4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.422+0000 7fcfdade1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555d49e410d0 con 0x555d49edd380 2024-03-18T00:54:20.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.422+0000 7fcfdade1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555d49de9c70 con 0x555d49ee24f0 2024-03-18T00:54:20.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.422+0000 7fcfdade1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555d49e3d170 con 0x555d49edc990 2024-03-18T00:54:20.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.424+0000 7fcfc6ffd640 1 -- 172.21.15.49:0/434548510 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcfc0018b30 con 0x7fcfd0108ff0 2024-03-18T00:54:20.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.425+0000 7fcfd66b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49edcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.425+0000 7fcfd66b8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49edcdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60400/0 (socket says 172.21.15.49:60400) 2024-03-18T00:54:20.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.425+0000 7fcfd66b8640 1 -- 172.21.15.49:0/884274030 learned_addr learned my addr 172.21.15.49:0/884274030 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:20.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.426+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/884274030 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.426+0000 7fcfd56b6640 1 --2- 172.21.15.49:0/884274030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49ee24f0 0x555d49ee4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.426+0000 7fcfd66b8640 1 -- 172.21.15.49:0/884274030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49ee24f0 msgr2=0x555d49ee4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:20.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.426+0000 7fcfd66b8640 1 --2- 172.21.15.49:0/884274030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49ee24f0 0x555d49ee4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.426+0000 7fcfd66b8640 1 -- 172.21.15.49:0/884274030 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 msgr2=0x555d49ee1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:20.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.426+0000 7fcfd66b8640 1 --2- 172.21.15.49:0/884274030 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.426+0000 7fcfd66b8640 1 -- 172.21.15.49:0/884274030 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d49c1d7e0 con 0x555d49edc990 2024-03-18T00:54:20.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.427+0000 7fcfc6ffd640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfbc0a2900 0x7fcfbc0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.428+0000 7fcfc6ffd640 1 -- 172.21.15.49:0/434548510 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcfc00cee30 con 0x7fcfd0108ff0 2024-03-18T00:54:20.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.429+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfbc0a2900 0x7fcfbc0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.431+0000 7fcfd66b8640 1 --2- 172.21.15.49:0/884274030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49edcdb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8036040 tx=0x7fcfc8009280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a11a2600f303f015 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:20.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.432+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfbc0a2900 0x7fcfbc0a4dc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc005fd0 tx=0x7fcfcc00b530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:20.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.432+0000 7fcfbbfff640 1 -- 172.21.15.49:0/884274030 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8042070 con 0x555d49edc990 2024-03-18T00:54:20.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.432+0000 7fcfbbfff640 1 -- 172.21.15.49:0/884274030 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc8033d50 con 0x555d49edc990 2024-03-18T00:54:20.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.432+0000 7fcfbbfff640 1 -- 172.21.15.49:0/884274030 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc802cbe0 con 0x555d49edc990 2024-03-18T00:54:20.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.436+0000 7fcfdade1740 1 -- 172.21.15.49:0/884274030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 msgr2=0x555d49edcdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:20.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.436+0000 7fcfdade1740 1 --2- 172.21.15.49:0/884274030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49edcdb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8036040 tx=0x7fcfc8009280 comp rx=0 tx=0).stop 2024-03-18T00:54:20.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.439+0000 7fcfdade1740 1 -- 172.21.15.49:0/884274030 shutdown_connections 2024-03-18T00:54:20.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.439+0000 7fcfdade1740 1 --2- 172.21.15.49:0/884274030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49ee24f0 0x555d49ee4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.439+0000 7fcfdade1740 1 --2- 172.21.15.49:0/884274030 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.439+0000 7fcfdade1740 1 --2- 172.21.15.49:0/884274030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49edcdb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.439+0000 7fcfdade1740 1 -- 172.21.15.49:0/884274030 >> 172.21.15.49:0/884274030 conn(0x555d49ea4c50 msgr2=0x555d49ed8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:20.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.440+0000 7fcfdade1740 1 -- 172.21.15.49:0/884274030 shutdown_connections 2024-03-18T00:54:20.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.440+0000 7fcfdade1740 1 -- 172.21.15.49:0/884274030 wait complete. 2024-03-18T00:54:20.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.440+0000 7fcfdade1740 1 -- start start 2024-03-18T00:54:20.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.441+0000 7fcfdade1740 1 Processor -- start 2024-03-18T00:54:20.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.442+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49ee6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.442+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee73f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.442+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49ee24f0 0x555d49eee430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.443+0000 7fcfdade1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555d49ee5440 con 0x555d49edd380 2024-03-18T00:54:20.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.443+0000 7fcfdade1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555d49dfd0d0 con 0x555d49ee24f0 2024-03-18T00:54:20.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.443+0000 7fcfdade1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555d49ee55c0 con 0x555d49edc990 2024-03-18T00:54:20.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.444+0000 7fcfdade1740 10 client.0 _ll_register_callbacks cb 0x555d49dc6760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:20.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.445+0000 7fcfd56b6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.445+0000 7fcfd56b6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47526/0 (socket says 172.21.15.49:47526) 2024-03-18T00:54:20.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.445+0000 7fcfd56b6640 1 -- 172.21.15.49:0/1746373540 learned_addr learned my addr 172.21.15.49:0/1746373540 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:20.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.445+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49ee6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.445+0000 7fcfd56b6640 1 -- 172.21.15.49:0/1746373540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 msgr2=0x555d49ee6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:20.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.445+0000 7fcfd56b6640 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49ee6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.445+0000 7fcfd56b6640 1 -- 172.21.15.49:0/1746373540 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49ee24f0 msgr2=0x555d49eee430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:20.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.445+0000 7fcfd56b6640 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49ee24f0 0x555d49eee430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.445+0000 7fcfd56b6640 1 -- 172.21.15.49:0/1746373540 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d49c7df70 con 0x555d49edd380 2024-03-18T00:54:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.446+0000 7fcfd56b6640 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee73f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcfc009b6e0 tx=0x7fcfc009b0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e25b22a8113ad8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:20.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.447+0000 7fcfb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:20.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.448+0000 7fcfb97fa640 1 -- 172.21.15.49:0/1746373540 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc009f8a0 con 0x555d49edd380 2024-03-18T00:54:20.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.448+0000 7fcfb97fa640 1 -- 172.21.15.49:0/1746373540 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcfc009aba0 con 0x555d49edd380 2024-03-18T00:54:20.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.448+0000 7fcfdade1740 1 -- 172.21.15.49:0/1746373540 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfc8033d50 con 0x555d49edd380 2024-03-18T00:54:20.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.449+0000 7fcfb97fa640 1 -- 172.21.15.49:0/1746373540 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcfc00a3ef0 con 0x555d49edd380 2024-03-18T00:54:20.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.449+0000 7fcfdade1740 10 client.16473 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:20.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.449+0000 7fcfdade1740 1 -- 172.21.15.49:0/1746373540 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555d49b15170 con 0x555d49edd380 2024-03-18T00:54:20.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.450+0000 7fcfb97fa640 1 -- 172.21.15.49:0/1746373540 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc009ada0 con 0x555d49edd380 2024-03-18T00:54:20.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.450+0000 7fcfb97fa640 1 -- 172.21.15.49:0/1746373540 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcfc0084ce0 con 0x555d49edd380 2024-03-18T00:54:20.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.450+0000 7fcfdade1740 10 client.16473 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:20.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.450+0000 7fcfdade1740 10 client.16473 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:20.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.450+0000 7fcfdade1740 20 client.16473 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:20.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.450+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d49ef2920 0x555d49ef4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.451+0000 7fcfdade1740 4 client.16473 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:20.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.451+0000 7fcfdade1740 1 -- 172.21.15.49:0/1746373540 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fcfc8033d50 con 0x555d49ef2920 2024-03-18T00:54:20.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.452+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d49ef2920 0x555d49ef4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.467+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d49ef2920 0x555d49ef4d40 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29ccc1cc5c5c58ca server_cookie=c59ac230501e0d0e in_seq=0 out_seq=0 2024-03-18T00:54:20.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.467+0000 7fcfb97fa640 10 client.16473 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:20.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.473+0000 7fcfb97fa640 1 -- 172.21.15.49:0/1746373540 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fcfc8033d50 con 0x555d49ef2920 2024-03-18T00:54:20.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.473+0000 7fcfb97fa640 10 client.16473 handle_command_reply: tid=0 2024-03-18T00:54:20.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.473+0000 7fcfdade1740 1 client.16473 shutdown 2024-03-18T00:54:20.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.474+0000 7fcfdade1740 1 -- 172.21.15.49:0/1746373540 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 msgr2=0x555d49ee73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:20.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee73f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcfc009b6e0 tx=0x7fcfc009b0b0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfdade1740 1 -- 172.21.15.49:0/1746373540 shutdown_connections 2024-03-18T00:54:20.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d49ef2920 0x555d49ef4d40 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49ee24f0 0x555d49eee430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edd380 0x555d49ee73f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1746373540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edc990 0x555d49ee6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfdade1740 1 -- 172.21.15.49:0/1746373540 >> 172.21.15.49:0/1746373540 conn(0x555d49ea4c50 msgr2=0x555d49ed74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:20.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfb97fa640 0 client.16473 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:20.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfdade1740 1 -- 172.21.15.49:0/1746373540 shutdown_connections 2024-03-18T00:54:20.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.475+0000 7fcfdade1740 1 -- 172.21.15.49:0/1746373540 wait complete. 2024-03-18T00:54:20.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.476+0000 7fcfdade1740 20 client.16473 trim_cache size 0 max 16384 2024-03-18T00:54:20.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.521+0000 7fcfdade1740 1 Processor -- start 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.521+0000 7fcfdade1740 1 -- start start 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.522+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49edc990 0x555d49edbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.522+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edd380 0x555d49eee3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.522+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49ee24f0 0x555d49ee59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.522+0000 7fcfdade1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb400e5f0 con 0x555d49ee24f0 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.522+0000 7fcfdade1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb4010130 con 0x555d49edc990 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.522+0000 7fcfdade1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb400ecc0 con 0x555d49edd380 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.522+0000 7fcfd66b8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49ee24f0 0x555d49ee59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.522+0000 7fcfd66b8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49ee24f0 0x555d49ee59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47532/0 (socket says 172.21.15.49:47532) 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.522+0000 7fcfd66b8640 1 -- 172.21.15.49:0/1688150302 learned_addr learned my addr 172.21.15.49:0/1688150302 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.523+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/1688150302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49edc990 0x555d49edbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.523+0000 7fcfd66b8640 1 -- 172.21.15.49:0/1688150302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edd380 msgr2=0x555d49eee3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:20.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.523+0000 7fcfd66b8640 1 --2- 172.21.15.49:0/1688150302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edd380 0x555d49eee3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.523+0000 7fcfd66b8640 1 -- 172.21.15.49:0/1688150302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49edc990 msgr2=0x555d49edbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:20.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.523+0000 7fcfd66b8640 1 --2- 172.21.15.49:0/1688150302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49edc990 0x555d49edbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.523+0000 7fcfd66b8640 1 -- 172.21.15.49:0/1688150302 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d49ed88a0 con 0x555d49ee24f0 2024-03-18T00:54:20.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.523+0000 7fcfd66b8640 1 --2- 172.21.15.49:0/1688150302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49ee24f0 0x555d49ee59f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009370 tx=0x7fcfc8009480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ac37db245ef740 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:20.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.524+0000 7fcfbb7fe640 1 -- 172.21.15.49:0/1688150302 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8042040 con 0x555d49ee24f0 2024-03-18T00:54:20.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.524+0000 7fcfbb7fe640 1 -- 172.21.15.49:0/1688150302 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc8033d50 con 0x555d49ee24f0 2024-03-18T00:54:20.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.524+0000 7fcfbb7fe640 1 -- 172.21.15.49:0/1688150302 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc802cd90 con 0x555d49ee24f0 2024-03-18T00:54:20.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.525+0000 7fcfdade1740 1 -- 172.21.15.49:0/1688150302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49ee24f0 msgr2=0x555d49ee59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:20.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.525+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1688150302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49ee24f0 0x555d49ee59f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009370 tx=0x7fcfc8009480 comp rx=0 tx=0).stop 2024-03-18T00:54:20.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.525+0000 7fcfdade1740 1 -- 172.21.15.49:0/1688150302 shutdown_connections 2024-03-18T00:54:20.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.525+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1688150302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49ee24f0 0x555d49ee59f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.525+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1688150302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49edd380 0x555d49eee3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.525+0000 7fcfdade1740 1 --2- 172.21.15.49:0/1688150302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49edc990 0x555d49edbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.525+0000 7fcfdade1740 1 -- 172.21.15.49:0/1688150302 >> 172.21.15.49:0/1688150302 conn(0x555d49ea4c50 msgr2=0x555d49edb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:20.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.525+0000 7fcfdade1740 1 -- 172.21.15.49:0/1688150302 shutdown_connections 2024-03-18T00:54:20.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.525+0000 7fcfdade1740 1 -- 172.21.15.49:0/1688150302 wait complete. 2024-03-18T00:54:20.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.526+0000 7fcfdade1740 1 -- start start 2024-03-18T00:54:20.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.526+0000 7fcfdade1740 1 Processor -- start 2024-03-18T00:54:20.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.526+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edc990 0x555d49eec250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49edd380 0x555d49eec790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfdade1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49ee24f0 0x555d49eed7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfdade1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb400fa60 con 0x555d49edc990 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfd5eb7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edc990 0x555d49eec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfdade1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb400f390 con 0x555d49edd380 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfd5eb7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edc990 0x555d49eec250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47546/0 (socket says 172.21.15.49:47546) 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfd5eb7640 1 -- 172.21.15.49:0/2744222404 learned_addr learned my addr 172.21.15.49:0/2744222404 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfdade1740 1 -- 172.21.15.49:0/2744222404 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb400c5d0 con 0x555d49ee24f0 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfdade1740 10 client.0 _ll_register_callbacks cb 0x7fcfc009b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfd5eb7640 1 -- 172.21.15.49:0/2744222404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49ee24f0 msgr2=0x555d49eed7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:20.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49ee24f0 0x555d49eed7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfd5eb7640 1 -- 172.21.15.49:0/2744222404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49edd380 msgr2=0x555d49eec790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49edd380 0x555d49eec790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.527+0000 7fcfd5eb7640 1 -- 172.21.15.49:0/2744222404 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d49edc190 con 0x555d49edc990 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.528+0000 7fcfd5eb7640 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edc990 0x555d49eec250 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x555d49edddf0 tx=0x7fcfcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f86ef79c9b1c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.528+0000 7fcfb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.528+0000 7fcfb97fa640 1 -- 172.21.15.49:0/2744222404 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc027aa0 con 0x555d49edc990 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.528+0000 7fcfb97fa640 1 -- 172.21.15.49:0/2744222404 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcfcc027c40 con 0x555d49edc990 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.528+0000 7fcfdade1740 1 -- 172.21.15.49:0/2744222404 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfc8033d50 con 0x555d49edc990 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.528+0000 7fcfb97fa640 1 -- 172.21.15.49:0/2744222404 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcfcc036050 con 0x555d49edc990 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.528+0000 7fcfdade1740 10 client.16479 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.528+0000 7fcfdade1740 1 -- 172.21.15.49:0/2744222404 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555d49f59330 con 0x555d49edc990 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.529+0000 7fcfb97fa640 1 -- 172.21.15.49:0/2744222404 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc8033d50 con 0x555d49edc990 2024-03-18T00:54:20.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.529+0000 7fcfb97fa640 1 -- 172.21.15.49:0/2744222404 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcfcc027670 con 0x555d49edc990 2024-03-18T00:54:20.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.529+0000 7fcfdade1740 10 client.16479 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:20.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.529+0000 7fcfdade1740 10 client.16479 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:20.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.529+0000 7fcfdade1740 20 client.16479 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:20.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.529+0000 7fcfdade1740 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d49f59d90 0x555d49f5c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:20.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.529+0000 7fcfdade1740 4 client.16479 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:54:20.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.529+0000 7fcfdade1740 1 -- 172.21.15.49:0/2744222404 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555d49e4e610 con 0x555d49f59d90 2024-03-18T00:54:20.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.530+0000 7fcfd56b6640 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d49f59d90 0x555d49f5c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:20.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.533+0000 7fcfd56b6640 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d49f59d90 0x555d49f5c210 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=780c26116affb8f1 server_cookie=43dcdaac21be85ed in_seq=0 out_seq=0 2024-03-18T00:54:20.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:20.533+0000 7fcfb97fa640 10 client.16479 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfb97fa640 1 -- 172.21.15.49:0/2744222404 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555d49e4e610 con 0x555d49f59d90 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfb97fa640 10 client.16479 handle_command_reply: tid=0 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfdade1740 1 client.16479 shutdown 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfdade1740 1 -- 172.21.15.49:0/2744222404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edc990 msgr2=0x555d49eec250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfdade1740 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edc990 0x555d49eec250 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x555d49edddf0 tx=0x7fcfcc004730 comp rx=0 tx=0).stop 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfdade1740 1 -- 172.21.15.49:0/2744222404 shutdown_connections 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfdade1740 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555d49f59d90 0x555d49f5c210 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfdade1740 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555d49edd380 0x555d49eec790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfdade1740 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555d49ee24f0 0x555d49eed7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfdade1740 1 --2- 172.21.15.49:0/2744222404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555d49edc990 0x555d49eec250 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfdade1740 1 -- 172.21.15.49:0/2744222404 >> 172.21.15.49:0/2744222404 conn(0x555d49ea4c50 msgr2=0x555d49ed6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:21.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.264+0000 7fcfb97fa640 0 client.16479 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:21.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.268+0000 7fcfdade1740 1 -- 172.21.15.49:0/2744222404 shutdown_connections 2024-03-18T00:54:21.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.268+0000 7fcfdade1740 1 -- 172.21.15.49:0/2744222404 wait complete. 2024-03-18T00:54:21.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.268+0000 7fcfdade1740 20 client.16479 trim_cache size 0 max 16384 2024-03-18T00:54:21.279 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:54:21.279 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:54:21.279 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "53cfa523-445f-4c4a-a10a-11ab8c89a733", 2024-03-18T00:54:21.279 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:54:21.279 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.279+0000 7fcfd811b640 1 -- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfbc0a2900 msgr2=0x7fcfbc0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.280+0000 7fcfd811b640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfbc0a2900 0x7fcfbc0a4dc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc005fd0 tx=0x7fcfcc00b530 comp rx=0 tx=0).stop 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.280+0000 7fcfd811b640 1 -- 172.21.15.49:0/434548510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd0108ff0 msgr2=0x7fcfd0070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.280+0000 7fcfd811b640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd0108ff0 0x7fcfd0070700 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000e9f0 tx=0x7fcfc000eeb0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.280+0000 7fcfd811b640 1 -- 172.21.15.49:0/434548510 shutdown_connections 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.280+0000 7fcfd811b640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfbc0a2900 0x7fcfbc0a4dc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.280+0000 7fcfd811b640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcfd01099b0 0x7fcfd007dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.280+0000 7fcfd811b640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfd0108ff0 0x7fcfd0070700 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.280+0000 7fcfd811b640 1 --2- 172.21.15.49:0/434548510 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcfd0107e00 0x7fcfd00701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.280+0000 7fcfd811b640 1 -- 172.21.15.49:0/434548510 >> 172.21.15.49:0/434548510 conn(0x7fcfd0076da0 msgr2=0x7fcfd010c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:21.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.281+0000 7fcfd811b640 1 -- 172.21.15.49:0/434548510 shutdown_connections 2024-03-18T00:54:21.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.281+0000 7fcfd811b640 1 -- 172.21.15.49:0/434548510 wait complete. 2024-03-18T00:54:21.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:54:21.335 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:54:21.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:21 smithi049 ceph-mon[32130]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1008 MiB/s wr, 1.73k op/s 2024-03-18T00:54:21.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.634+0000 7fe86380b640 1 -- 172.21.15.49:0/2826567072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe85c1055d0 msgr2=0x7fe85c10be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.634+0000 7fe86380b640 1 --2- 172.21.15.49:0/2826567072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe85c1055d0 0x7fe85c10be60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe858009900 tx=0x7fe85802bc80 comp rx=0 tx=0).stop 2024-03-18T00:54:21.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.634+0000 7fe86380b640 1 -- 172.21.15.49:0/2826567072 shutdown_connections 2024-03-18T00:54:21.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.634+0000 7fe86380b640 1 --2- 172.21.15.49:0/2826567072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe85c1055d0 0x7fe85c10be60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.634+0000 7fe86380b640 1 --2- 172.21.15.49:0/2826567072 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe85c104c10 0x7fe85c105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.634+0000 7fe86380b640 1 --2- 172.21.15.49:0/2826567072 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe85c103a20 0x7fe85c103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.634+0000 7fe86380b640 1 -- 172.21.15.49:0/2826567072 >> 172.21.15.49:0/2826567072 conn(0x7fe85c0fef90 msgr2=0x7fe85c1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:21.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.640+0000 7fe86380b640 1 -- 172.21.15.49:0/2826567072 shutdown_connections 2024-03-18T00:54:21.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.645+0000 7fe86380b640 1 -- 172.21.15.49:0/2826567072 wait complete. 2024-03-18T00:54:21.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.646+0000 7fe86380b640 1 Processor -- start 2024-03-18T00:54:21.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.646+0000 7fe86380b640 1 -- start start 2024-03-18T00:54:21.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.646+0000 7fe86380b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe85c103a20 0x7fe85c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.646+0000 7fe86380b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe85c104c10 0x7fe85c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.646+0000 7fe86380b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe85c1055d0 0x7fe85c1a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.646+0000 7fe86380b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c10f800 con 0x7fe85c104c10 2024-03-18T00:54:21.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.646+0000 7fe86380b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c10f680 con 0x7fe85c103a20 2024-03-18T00:54:21.647 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.646+0000 7fe86380b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c10f980 con 0x7fe85c1055d0 2024-03-18T00:54:21.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.649+0000 7fe861da8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe85c1055d0 0x7fe85c1a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.649 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.650+0000 7fe860da6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe85c104c10 0x7fe85c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.650+0000 7fe860da6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe85c104c10 0x7fe85c19a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47554/0 (socket says 172.21.15.49:47554) 2024-03-18T00:54:21.650 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.650+0000 7fe860da6640 1 -- 172.21.15.49:0/1836516047 learned_addr learned my addr 172.21.15.49:0/1836516047 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:21.651 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.651+0000 7fe8615a7640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe85c103a20 0x7fe85c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.653+0000 7fe8615a7640 1 -- 172.21.15.49:0/1836516047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe85c1055d0 msgr2=0x7fe85c1a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.653+0000 7fe8615a7640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe85c1055d0 0x7fe85c1a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.653 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.653+0000 7fe8615a7640 1 -- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe85c104c10 msgr2=0x7fe85c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.654+0000 7fe8615a7640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe85c104c10 0x7fe85c19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.654+0000 7fe8615a7640 1 -- 172.21.15.49:0/1836516047 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c1a19d0 con 0x7fe85c103a20 2024-03-18T00:54:21.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.655+0000 7fe861da8640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe85c1055d0 0x7fe85c1a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:54:21.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.655+0000 7fe860da6640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe85c104c10 0x7fe85c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:54:21.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.655+0000 7fe8615a7640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe85c103a20 0x7fe85c199db0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe85400bdf0 tx=0x7fe854009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:21.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.658+0000 7fe8527fc640 1 -- 172.21.15.49:0/1836516047 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854019070 con 0x7fe85c103a20 2024-03-18T00:54:21.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.658+0000 7fe86380b640 1 -- 172.21.15.49:0/1836516047 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe85c1a1cc0 con 0x7fe85c103a20 2024-03-18T00:54:21.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.659+0000 7fe86380b640 1 -- 172.21.15.49:0/1836516047 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe85c1a21c0 con 0x7fe85c103a20 2024-03-18T00:54:21.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.659+0000 7fe8527fc640 1 -- 172.21.15.49:0/1836516047 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe854014920 con 0x7fe85c103a20 2024-03-18T00:54:21.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.659+0000 7fe8527fc640 1 -- 172.21.15.49:0/1836516047 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8540129b0 con 0x7fe85c103a20 2024-03-18T00:54:21.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.672+0000 7fe8663fc740 1 Processor -- start 2024-03-18T00:54:21.672 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.672+0000 7fe8527fc640 1 -- 172.21.15.49:0/1836516047 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe85401c070 con 0x7fe85c103a20 2024-03-18T00:54:21.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.673+0000 7fe8527fc640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8480a2a10 0x7fe8480a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.673+0000 7fe8663fc740 1 -- start start 2024-03-18T00:54:21.675 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.674+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec19c170 0x5638ec19c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.674+0000 7fe861da8640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8480a2a10 0x7fe8480a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.674+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.675+0000 7fe860da6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.675+0000 7fe860da6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47560/0 (socket says 172.21.15.49:47560) 2024-03-18T00:54:21.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.675+0000 7fe8527fc640 1 -- 172.21.15.49:0/1836516047 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe8540ce270 con 0x7fe85c103a20 2024-03-18T00:54:21.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.676+0000 7fe861da8640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8480a2a10 0x7fe8480a4ed0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe85802c760 tx=0x7fe858005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:21.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.680+0000 7fe860da6640 1 -- 172.21.15.49:0/3743391936 learned_addr learned my addr 172.21.15.49:0/3743391936 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:21.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.680+0000 7fe8615a7640 1 --2- 172.21.15.49:0/3743391936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec19c170 0x5638ec19c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.680+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3743391936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec1a1cd0 0x5638ec1a4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.680+0000 7fe8663fc740 1 -- 172.21.15.49:0/3743391936 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5638ec101500 con 0x5638ec19cb60 2024-03-18T00:54:21.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.680+0000 7fe8663fc740 1 -- 172.21.15.49:0/3743391936 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5638ec0a9be0 con 0x5638ec1a1cd0 2024-03-18T00:54:21.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.680+0000 7fe8663fc740 1 -- 172.21.15.49:0/3743391936 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5638ec0fd5a0 con 0x5638ec19c170 2024-03-18T00:54:21.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.689+0000 7fe860da6640 1 -- 172.21.15.49:0/3743391936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec19c170 msgr2=0x5638ec19c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:21.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.689+0000 7fe860da6640 1 --2- 172.21.15.49:0/3743391936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec19c170 0x5638ec19c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.689+0000 7fe860da6640 1 -- 172.21.15.49:0/3743391936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec1a1cd0 msgr2=0x5638ec1a4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:21.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.689+0000 7fe860da6640 1 --2- 172.21.15.49:0/3743391936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec1a1cd0 0x5638ec1a4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.689+0000 7fe860da6640 1 -- 172.21.15.49:0/3743391936 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638ec0bd4b0 con 0x5638ec19cb60 2024-03-18T00:54:21.693 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.691+0000 7fe860da6640 1 --2- 172.21.15.49:0/3743391936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a1660 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe84c0078c0 tx=0x7fe84c007d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b10a2018db5cd504 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:21.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.693+0000 7fe8477fe640 1 -- 172.21.15.49:0/3743391936 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c00d950 con 0x5638ec19cb60 2024-03-18T00:54:21.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.694+0000 7fe8477fe640 1 -- 172.21.15.49:0/3743391936 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe84c018070 con 0x5638ec19cb60 2024-03-18T00:54:21.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.694+0000 7fe8477fe640 1 -- 172.21.15.49:0/3743391936 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c012640 con 0x5638ec19cb60 2024-03-18T00:54:21.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.708+0000 7fe8663fc740 1 -- 172.21.15.49:0/3743391936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 msgr2=0x5638ec1a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.708+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3743391936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a1660 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe84c0078c0 tx=0x7fe84c007d80 comp rx=0 tx=0).stop 2024-03-18T00:54:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.709+0000 7fe8663fc740 1 -- 172.21.15.49:0/3743391936 shutdown_connections 2024-03-18T00:54:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.709+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3743391936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec1a1cd0 0x5638ec1a4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.709+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3743391936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a1660 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.709+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3743391936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec19c170 0x5638ec19c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.709+0000 7fe8663fc740 1 -- 172.21.15.49:0/3743391936 >> 172.21.15.49:0/3743391936 conn(0x5638ec1976e0 msgr2=0x5638ec197ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:21.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.710+0000 7fe8663fc740 1 -- 172.21.15.49:0/3743391936 shutdown_connections 2024-03-18T00:54:21.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.714+0000 7fe8663fc740 1 -- 172.21.15.49:0/3743391936 wait complete. 2024-03-18T00:54:21.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.715+0000 7fe8663fc740 1 -- start start 2024-03-18T00:54:21.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.715+0000 7fe8663fc740 1 Processor -- start 2024-03-18T00:54:21.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.716+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19c170 0x5638ec1a6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.717+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.717+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec1ada00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.717+0000 7fe860da6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.717+0000 7fe860da6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47562/0 (socket says 172.21.15.49:47562) 2024-03-18T00:54:21.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.717+0000 7fe860da6640 1 -- 172.21.15.49:0/3466751459 learned_addr learned my addr 172.21.15.49:0/3466751459 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:21.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.717+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5638ec1a4c80 con 0x5638ec19cb60 2024-03-18T00:54:21.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.717+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5638ec0bd160 con 0x5638ec19c170 2024-03-18T00:54:21.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.718+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5638ec1a4e00 con 0x5638ec1a1cd0 2024-03-18T00:54:21.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.718+0000 7fe8663fc740 10 client.0 _ll_register_callbacks cb 0x5638ec0a1150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:21.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.719+0000 7fe860da6640 1 -- 172.21.15.49:0/3466751459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 msgr2=0x5638ec1ada00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:21.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.719+0000 7fe860da6640 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec1ada00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.719+0000 7fe860da6640 1 -- 172.21.15.49:0/3466751459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19c170 msgr2=0x5638ec1a6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:21.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.719+0000 7fe860da6640 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19c170 0x5638ec1a6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.719+0000 7fe860da6640 1 -- 172.21.15.49:0/3466751459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638ebedf7d0 con 0x5638ec19cb60 2024-03-18T00:54:21.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.720+0000 7fe860da6640 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a69c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe84c017000 tx=0x7fe84c00df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734d79467975e065 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:21.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.722+0000 7fe844ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:21.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.722+0000 7fe844ff9640 1 -- 172.21.15.49:0/3466751459 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c013990 con 0x5638ec19cb60 2024-03-18T00:54:21.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.722+0000 7fe844ff9640 1 -- 172.21.15.49:0/3466751459 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe84c024420 con 0x5638ec19cb60 2024-03-18T00:54:21.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.722+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe84c018070 con 0x5638ec19cb60 2024-03-18T00:54:21.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.724+0000 7fe844ff9640 1 -- 172.21.15.49:0/3466751459 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe84c024660 con 0x5638ec19cb60 2024-03-18T00:54:21.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.724+0000 7fe8663fc740 10 client.16488 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:21.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.724+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5638ebe153f0 con 0x5638ec19cb60 2024-03-18T00:54:21.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.725+0000 7fe844ff9640 1 -- 172.21.15.49:0/3466751459 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe84c024ce0 con 0x5638ec19cb60 2024-03-18T00:54:21.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.725+0000 7fe844ff9640 1 -- 172.21.15.49:0/3466751459 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe84c01cad0 con 0x5638ec19cb60 2024-03-18T00:54:21.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.726+0000 7fe8663fc740 10 client.16488 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:21.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.726+0000 7fe8663fc740 10 client.16488 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:21.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.726+0000 7fe8663fc740 20 client.16488 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:21.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.726+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638ec1b1c10 0x5638ec1b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.726+0000 7fe8663fc740 4 client.16488 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:21.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.726+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe84c018070 con 0x5638ec1b1c10 2024-03-18T00:54:21.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.727+0000 7fe8615a7640 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638ec1b1c10 0x5638ec1b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.738+0000 7fe8615a7640 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638ec1b1c10 0x5638ec1b4060 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe6a85bb0d3d71ce server_cookie=aecb4ae00e17053c in_seq=0 out_seq=0 2024-03-18T00:54:21.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.738+0000 7fe844ff9640 10 client.16488 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.742+0000 7fe844ff9640 1 -- 172.21.15.49:0/3466751459 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe84c018070 con 0x5638ec1b1c10 2024-03-18T00:54:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.742+0000 7fe844ff9640 10 client.16488 handle_command_reply: tid=0 2024-03-18T00:54:21.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.742+0000 7fe8663fc740 1 client.16488 shutdown 2024-03-18T00:54:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.745+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 msgr2=0x5638ec1a69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.745+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a69c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe84c017000 tx=0x7fe84c00df50 comp rx=0 tx=0).stop 2024-03-18T00:54:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.745+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 shutdown_connections 2024-03-18T00:54:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.745+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638ec1b1c10 0x5638ec1b4060 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.745+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec1ada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.745+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1a69c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.745+0000 7fe8663fc740 1 --2- 172.21.15.49:0/3466751459 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19c170 0x5638ec1a6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.745+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 >> 172.21.15.49:0/3466751459 conn(0x5638ec1976e0 msgr2=0x5638ec196900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:21.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.745+0000 7fe844ff9640 0 client.16488 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.747+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 shutdown_connections 2024-03-18T00:54:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.747+0000 7fe8663fc740 1 -- 172.21.15.49:0/3466751459 wait complete. 2024-03-18T00:54:21.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.747+0000 7fe8663fc740 20 client.16488 trim_cache size 0 max 16384 2024-03-18T00:54:21.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.796+0000 7fe8663fc740 1 Processor -- start 2024-03-18T00:54:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.796+0000 7fe8663fc740 1 -- start start 2024-03-18T00:54:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.797+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19c170 0x5638ec1a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.797+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.797+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.797+0000 7fe8663fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe84000e610 con 0x5638ec19cb60 2024-03-18T00:54:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.797+0000 7fe8663fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe840010150 con 0x5638ec19c170 2024-03-18T00:54:21.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.797+0000 7fe8663fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe84000ece0 con 0x5638ec1a1cd0 2024-03-18T00:54:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.798+0000 7fe860da6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.798+0000 7fe8615a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19c170 0x5638ec1a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.798+0000 7fe860da6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47572/0 (socket says 172.21.15.49:47572) 2024-03-18T00:54:21.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.798+0000 7fe860da6640 1 -- 172.21.15.49:0/2164095126 learned_addr learned my addr 172.21.15.49:0/2164095126 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.798+0000 7fe860da6640 1 -- 172.21.15.49:0/2164095126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 msgr2=0x5638ec197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.799+0000 7fe860da6640 1 --2- 172.21.15.49:0/2164095126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.799+0000 7fe860da6640 1 -- 172.21.15.49:0/2164095126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19c170 msgr2=0x5638ec1a5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.799+0000 7fe860da6640 1 --2- 172.21.15.49:0/2164095126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19c170 0x5638ec1a5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.799+0000 7fe860da6640 1 -- 172.21.15.49:0/2164095126 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638ebdd50f0 con 0x5638ec19cb60 2024-03-18T00:54:21.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.799+0000 7fe860da6640 1 --2- 172.21.15.49:0/2164095126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1ad9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe84c023510 tx=0x7fe84c006ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac40a444decc4e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.800+0000 7fe846ffd640 1 -- 172.21.15.49:0/2164095126 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c033030 con 0x5638ec19cb60 2024-03-18T00:54:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.800+0000 7fe846ffd640 1 -- 172.21.15.49:0/2164095126 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe84c018070 con 0x5638ec19cb60 2024-03-18T00:54:21.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.800+0000 7fe846ffd640 1 -- 172.21.15.49:0/2164095126 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c013aa0 con 0x5638ec19cb60 2024-03-18T00:54:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.801+0000 7fe8663fc740 1 -- 172.21.15.49:0/2164095126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 msgr2=0x5638ec1ad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.801+0000 7fe8663fc740 1 --2- 172.21.15.49:0/2164095126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1ad9c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe84c023510 tx=0x7fe84c006ab0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.801+0000 7fe8663fc740 1 -- 172.21.15.49:0/2164095126 shutdown_connections 2024-03-18T00:54:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.802+0000 7fe8663fc740 1 --2- 172.21.15.49:0/2164095126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.802+0000 7fe8663fc740 1 --2- 172.21.15.49:0/2164095126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19cb60 0x5638ec1ad9c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.802+0000 7fe8663fc740 1 --2- 172.21.15.49:0/2164095126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19c170 0x5638ec1a5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.802+0000 7fe8663fc740 1 -- 172.21.15.49:0/2164095126 >> 172.21.15.49:0/2164095126 conn(0x5638ec1976e0 msgr2=0x5638ec1a5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.802+0000 7fe8663fc740 1 -- 172.21.15.49:0/2164095126 shutdown_connections 2024-03-18T00:54:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.802+0000 7fe8663fc740 1 -- 172.21.15.49:0/2164095126 wait complete. 2024-03-18T00:54:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.803+0000 7fe8663fc740 1 -- start start 2024-03-18T00:54:21.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.803+0000 7fe8663fc740 1 Processor -- start 2024-03-18T00:54:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19c170 0x5638ec1ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19cb60 0x5638ec1abb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8663fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec1aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8663fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe84000fa80 con 0x5638ec19c170 2024-03-18T00:54:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8663fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe84000f3b0 con 0x5638ec19cb60 2024-03-18T00:54:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8663fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe84000c5f0 con 0x5638ec1a1cd0 2024-03-18T00:54:21.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8663fc740 10 client.0 _ll_register_callbacks cb 0x7fe85400cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8615a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19c170 0x5638ec1ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8615a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19c170 0x5638ec1ab5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47586/0 (socket says 172.21.15.49:47586) 2024-03-18T00:54:21.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.804+0000 7fe8615a7640 1 -- 172.21.15.49:0/1353499856 learned_addr learned my addr 172.21.15.49:0/1353499856 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.805+0000 7fe861da8640 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec1aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.805+0000 7fe8615a7640 1 -- 172.21.15.49:0/1353499856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 msgr2=0x5638ec1aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.805+0000 7fe8615a7640 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec1aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.805+0000 7fe8615a7640 1 -- 172.21.15.49:0/1353499856 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19cb60 msgr2=0x5638ec1abb00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.805+0000 7fe8615a7640 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19cb60 0x5638ec1abb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.805+0000 7fe8615a7640 1 -- 172.21.15.49:0/1353499856 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638ec0c89d0 con 0x5638ec19c170 2024-03-18T00:54:21.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.805+0000 7fe861da8640 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec1aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:54:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.806+0000 7fe8615a7640 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19c170 0x5638ec1ab5c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe854000f80 tx=0x7fe854096ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd1b367016c34b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.806+0000 7fe844ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.806+0000 7fe844ff9640 1 -- 172.21.15.49:0/1353499856 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85409ab90 con 0x5638ec19c170 2024-03-18T00:54:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.806+0000 7fe844ff9640 1 -- 172.21.15.49:0/1353499856 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe8540cbe00 con 0x5638ec19c170 2024-03-18T00:54:21.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.806+0000 7fe8663fc740 1 -- 172.21.15.49:0/1353499856 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe84c018070 con 0x5638ec19c170 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe844ff9640 1 -- 172.21.15.49:0/1353499856 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe85409ece0 con 0x5638ec19c170 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe8663fc740 10 client.16494 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe8663fc740 1 -- 172.21.15.49:0/1353499856 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5638ec220b00 con 0x5638ec19c170 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe844ff9640 1 -- 172.21.15.49:0/1353499856 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe84c018070 con 0x5638ec19c170 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe844ff9640 1 -- 172.21.15.49:0/1353499856 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe8540ab030 con 0x5638ec19c170 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe8663fc740 10 client.16494 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe8663fc740 10 client.16494 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe8663fc740 20 client.16494 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe8663fc740 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638ec221500 0x5638ec223920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:21.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe8663fc740 4 client.16494 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:54:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.807+0000 7fe8663fc740 1 -- 172.21.15.49:0/1353499856 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5638ec10ea40 con 0x5638ec221500 2024-03-18T00:54:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.808+0000 7fe860da6640 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638ec221500 0x5638ec223920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:21.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.809+0000 7fe860da6640 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638ec221500 0x5638ec223920 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db294238f6d3c0c5 server_cookie=4db2f61c09acce64 in_seq=0 out_seq=0 2024-03-18T00:54:21.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.810+0000 7fe844ff9640 10 client.16494 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.813+0000 7fe844ff9640 1 -- 172.21.15.49:0/1353499856 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x5638ec10ea40 con 0x5638ec221500 2024-03-18T00:54:21.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.813+0000 7fe844ff9640 10 client.16494 handle_command_reply: tid=0 2024-03-18T00:54:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.814+0000 7fe8663fc740 1 client.16494 shutdown 2024-03-18T00:54:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.814+0000 7fe8663fc740 1 -- 172.21.15.49:0/1353499856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19c170 msgr2=0x5638ec1ab5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.815+0000 7fe8663fc740 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19c170 0x5638ec1ab5c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fe854000f80 tx=0x7fe854096ef0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe8663fc740 1 -- 172.21.15.49:0/1353499856 shutdown_connections 2024-03-18T00:54:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe8663fc740 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5638ec221500 0x5638ec223920 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe8663fc740 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5638ec19cb60 0x5638ec1abb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe8663fc740 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5638ec1a1cd0 0x5638ec1aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe8663fc740 1 --2- 172.21.15.49:0/1353499856 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5638ec19c170 0x5638ec1ab5c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe8663fc740 1 -- 172.21.15.49:0/1353499856 >> 172.21.15.49:0/1353499856 conn(0x5638ec1976e0 msgr2=0x5638ec19a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe844ff9640 0 client.16494 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe8663fc740 1 -- 172.21.15.49:0/1353499856 shutdown_connections 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe8663fc740 1 -- 172.21.15.49:0/1353499856 wait complete. 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.816+0000 7fe8663fc740 20 client.16494 trim_cache size 0 max 16384 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-18T00:54:21.819 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:54:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "53cfa523-445f-4c4a-a10a-11ab8c89a733": { 2024-03-18T00:54:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:54:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "53cfa523-445f-4c4a-a10a-11ab8c89a733", 2024-03-18T00:54:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:54:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:54:21.820 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:54:21.820 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.817+0000 7fe86380b640 1 -- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8480a2a10 msgr2=0x7fe8480a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.817+0000 7fe86380b640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8480a2a10 0x7fe8480a4ed0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe85802c760 tx=0x7fe858005fd0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.817+0000 7fe86380b640 1 -- 172.21.15.49:0/1836516047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe85c103a20 msgr2=0x7fe85c199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.817+0000 7fe86380b640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe85c103a20 0x7fe85c199db0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe85400bdf0 tx=0x7fe854009f70 comp rx=0 tx=0).stop 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.818+0000 7fe86380b640 1 -- 172.21.15.49:0/1836516047 shutdown_connections 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.818+0000 7fe86380b640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe8480a2a10 0x7fe8480a4ed0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.818+0000 7fe86380b640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe85c1055d0 0x7fe85c1a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.818+0000 7fe86380b640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe85c104c10 0x7fe85c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.818+0000 7fe86380b640 1 --2- 172.21.15.49:0/1836516047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe85c103a20 0x7fe85c199db0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:21.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.818+0000 7fe86380b640 1 -- 172.21.15.49:0/1836516047 >> 172.21.15.49:0/1836516047 conn(0x7fe85c0fef90 msgr2=0x7fe85c107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.818+0000 7fe86380b640 1 -- 172.21.15.49:0/1836516047 shutdown_connections 2024-03-18T00:54:21.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:21.818+0000 7fe86380b640 1 -- 172.21.15.49:0/1836516047 wait complete. 2024-03-18T00:54:21.847 INFO:tasks.cephfs.filesystem:scrub status for tag:53cfa523-445f-4c4a-a10a-11ab8c89a733 - {'path': '/', 'tag': '53cfa523-445f-4c4a-a10a-11ab8c89a733', 'options': 'recursive,force'} 2024-03-18T00:54:21.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:21 smithi107 ceph-mon[36270]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1008 MiB/s wr, 1.73k op/s 2024-03-18T00:54:21.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:21 smithi106 ceph-mon[35899]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1008 MiB/s wr, 1.73k op/s 2024-03-18T00:54:22.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:22 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:54:22.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:22 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:54:22.879 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:22 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:54:22.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:22 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:54:22.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:22 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:54:22.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:22 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:54:22.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:22 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:54:22.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:22 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:54:22.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:22 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:54:23.846 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:23 smithi106 ceph-mon[35899]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 819 MiB/s wr, 1.49k op/s 2024-03-18T00:54:23.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:23 smithi049 ceph-mon[32130]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 819 MiB/s wr, 1.49k op/s 2024-03-18T00:54:23.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:23 smithi107 ceph-mon[36270]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 819 MiB/s wr, 1.49k op/s 2024-03-18T00:54:24.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:24 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:54:24.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:24 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:54:24.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:24 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:54:24.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:24 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:54:24.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:24 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:54:24.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:24 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:54:25.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:25 smithi049 ceph-mon[32130]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1017 MiB/s wr, 2.09k op/s 2024-03-18T00:54:25.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:25 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:54:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:25 smithi107 ceph-mon[36270]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1017 MiB/s wr, 2.09k op/s 2024-03-18T00:54:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:25 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:54:25.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:25 smithi106 ceph-mon[35899]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1017 MiB/s wr, 2.09k op/s 2024-03-18T00:54:25.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:25 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:54:26.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:26 smithi049 ceph-mon[32130]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 837 MiB/s wr, 1.88k op/s 2024-03-18T00:54:26.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:26 smithi107 ceph-mon[36270]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 837 MiB/s wr, 1.88k op/s 2024-03-18T00:54:26.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:26 smithi106 ceph-mon[35899]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 837 MiB/s wr, 1.88k op/s 2024-03-18T00:54:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.380+0000 7fa374dba640 1 -- 172.21.15.49:0/2494592368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa370105390 msgr2=0x7fa370105810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.380+0000 7fa374dba640 1 --2- 172.21.15.49:0/2494592368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa370105390 0x7fa370105810 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa368009ca0 tx=0x7fa36802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.380+0000 7fa374dba640 1 -- 172.21.15.49:0/2494592368 shutdown_connections 2024-03-18T00:54:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.380+0000 7fa374dba640 1 --2- 172.21.15.49:0/2494592368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa370105390 0x7fa370105810 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.380+0000 7fa374dba640 1 --2- 172.21.15.49:0/2494592368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3701029e0 0x7fa370104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.380+0000 7fa374dba640 1 --2- 172.21.15.49:0/2494592368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa370100070 0x7fa3701024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.380+0000 7fa374dba640 1 -- 172.21.15.49:0/2494592368 >> 172.21.15.49:0/2494592368 conn(0x7fa3700fbc70 msgr2=0x7fa3700fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.381+0000 7fa374dba640 1 -- 172.21.15.49:0/2494592368 shutdown_connections 2024-03-18T00:54:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.381+0000 7fa374dba640 1 -- 172.21.15.49:0/2494592368 wait complete. 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.382+0000 7fa374dba640 1 Processor -- start 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.382+0000 7fa374dba640 1 -- start start 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.382+0000 7fa374dba640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa370100070 0x7fa37019e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.382+0000 7fa374dba640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3701029e0 0x7fa37019e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.382+0000 7fa374dba640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa370105390 0x7fa3701a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.382+0000 7fa374dba640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa370113ad0 con 0x7fa370100070 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.382+0000 7fa374dba640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa370113950 con 0x7fa370105390 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa374dba640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa370113c50 con 0x7fa3701029e0 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36ed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa370105390 0x7fa3701a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36ed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa370105390 0x7fa3701a5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:35888/0 (socket says 172.21.15.49:35888) 2024-03-18T00:54:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36ed9d640 1 -- 172.21.15.49:0/810316121 learned_addr learned my addr 172.21.15.49:0/810316121 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36e59c640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa370100070 0x7fa37019e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36dd9b640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3701029e0 0x7fa37019e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36ed9d640 1 -- 172.21.15.49:0/810316121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3701029e0 msgr2=0x7fa37019e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3701029e0 0x7fa37019e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36ed9d640 1 -- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa370100070 msgr2=0x7fa37019e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa370100070 0x7fa37019e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.383+0000 7fa36ed9d640 1 -- 172.21.15.49:0/810316121 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3701a5c70 con 0x7fa370105390 2024-03-18T00:54:27.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.385+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa370105390 0x7fa3701a5600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa36802c630 tx=0x7fa368032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:27.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.386+0000 7fa35f7fe640 1 -- 172.21.15.49:0/810316121 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368041020 con 0x7fa370105390 2024-03-18T00:54:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.386+0000 7fa35f7fe640 1 -- 172.21.15.49:0/810316121 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa368034cd0 con 0x7fa370105390 2024-03-18T00:54:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.386+0000 7fa35f7fe640 1 -- 172.21.15.49:0/810316121 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa368039040 con 0x7fa370105390 2024-03-18T00:54:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.386+0000 7fa374dba640 1 -- 172.21.15.49:0/810316121 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3701a5f00 con 0x7fa370105390 2024-03-18T00:54:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.386+0000 7fa374dba640 1 -- 172.21.15.49:0/810316121 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3701a62b0 con 0x7fa370105390 2024-03-18T00:54:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.389+0000 7fa35f7fe640 1 -- 172.21.15.49:0/810316121 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa368044020 con 0x7fa370105390 2024-03-18T00:54:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.390+0000 7fa37799d740 1 Processor -- start 2024-03-18T00:54:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa37799d740 1 -- start start 2024-03-18T00:54:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa35f7fe640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3580a29a0 0x7fa3580a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa37799d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8340 0x5584383b8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa37799d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383bd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa37799d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383bdea0 0x5584383c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa37799d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55843831da40 con 0x5584383b8340 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa37799d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5584382c6920 con 0x5584383b8d30 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa37799d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558438319ae0 con 0x5584383bdea0 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa35f7fe640 1 -- 172.21.15.49:0/810316121 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa3680f0a00 con 0x7fa370105390 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.391+0000 7fa36dd9b640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3580a29a0 0x7fa3580a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383bdea0 0x5584383c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383bdea0 0x5584383c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60490/0 (socket says 172.21.15.49:60490) 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36ed9d640 1 -- 172.21.15.49:0/3162622075 learned_addr learned my addr 172.21.15.49:0/3162622075 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36e59c640 1 --2- 172.21.15.49:0/3162622075 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8340 0x5584383b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36dd9b640 1 --2- 172.21.15.49:0/3162622075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36ed9d640 1 -- 172.21.15.49:0/3162622075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 msgr2=0x5584383bd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:27.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/3162622075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383bd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36ed9d640 1 -- 172.21.15.49:0/3162622075 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8340 msgr2=0x5584383b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:27.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/3162622075 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8340 0x5584383b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36ed9d640 1 -- 172.21.15.49:0/3162622075 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584382de460 con 0x5584383bdea0 2024-03-18T00:54:27.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.392+0000 7fa36dd9b640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3580a29a0 0x7fa3580a4e60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa37019f560 tx=0x7fa364009630 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:27.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.393+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/3162622075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383bdea0 0x5584383c0330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa3680bd2b0 tx=0x7fa3680b96f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e5b91c7e21345ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:27.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.393+0000 7fa35cff9640 1 -- 172.21.15.49:0/3162622075 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3680312a0 con 0x5584383bdea0 2024-03-18T00:54:27.393 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.393+0000 7fa35cff9640 1 -- 172.21.15.49:0/3162622075 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3680020b0 con 0x5584383bdea0 2024-03-18T00:54:27.394 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.393+0000 7fa35cff9640 1 -- 172.21.15.49:0/3162622075 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3680c1120 con 0x5584383bdea0 2024-03-18T00:54:27.394 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.394+0000 7fa37799d740 1 -- 172.21.15.49:0/3162622075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383bdea0 msgr2=0x5584383c0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.394+0000 7fa37799d740 1 --2- 172.21.15.49:0/3162622075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383bdea0 0x5584383c0330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa3680bd2b0 tx=0x7fa3680b96f0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.394+0000 7fa37799d740 1 -- 172.21.15.49:0/3162622075 shutdown_connections 2024-03-18T00:54:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.395+0000 7fa37799d740 1 --2- 172.21.15.49:0/3162622075 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383bd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.395+0000 7fa37799d740 1 --2- 172.21.15.49:0/3162622075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383bdea0 0x5584383c0330 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.395+0000 7fa37799d740 1 --2- 172.21.15.49:0/3162622075 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8340 0x5584383b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.395+0000 7fa37799d740 1 -- 172.21.15.49:0/3162622075 >> 172.21.15.49:0/3162622075 conn(0x5584383b38b0 msgr2=0x5584383b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:27.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.396+0000 7fa37799d740 1 -- 172.21.15.49:0/3162622075 shutdown_connections 2024-03-18T00:54:27.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.396+0000 7fa37799d740 1 -- 172.21.15.49:0/3162622075 wait complete. 2024-03-18T00:54:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.397+0000 7fa37799d740 1 -- start start 2024-03-18T00:54:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.397+0000 7fa37799d740 1 Processor -- start 2024-03-18T00:54:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.398+0000 7fa37799d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 0x5584383c2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.398+0000 7fa37799d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.398+0000 7fa37799d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383bdea0 0x5584383c9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.398+0000 7fa37799d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5584383c0c10 con 0x5584383bdea0 2024-03-18T00:54:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.398+0000 7fa37799d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5584382d9d70 con 0x5584383b8d30 2024-03-18T00:54:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.398+0000 7fa37799d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5584383c0d90 con 0x5584383b8340 2024-03-18T00:54:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.398+0000 7fa37799d740 10 client.0 _ll_register_callbacks cb 0x558438031a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.399+0000 7fa36dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.399+0000 7fa36e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 0x5584383c2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.400+0000 7fa36dd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383c28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:35896/0 (socket says 172.21.15.49:35896) 2024-03-18T00:54:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.400+0000 7fa36dd9b640 1 -- 172.21.15.49:0/1872207144 learned_addr learned my addr 172.21.15.49:0/1872207144 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.401+0000 7fa36e59c640 1 -- 172.21.15.49:0/1872207144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 msgr2=0x5584383c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.402+0000 7fa36e59c640 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383c28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.402+0000 7fa36e59c640 1 -- 172.21.15.49:0/1872207144 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383bdea0 msgr2=0x5584383c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.402+0000 7fa36e59c640 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383bdea0 0x5584383c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.402+0000 7fa36e59c640 1 -- 172.21.15.49:0/1872207144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558438032210 con 0x5584383b8340 2024-03-18T00:54:27.403 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.403+0000 7fa36e59c640 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 0x5584383c2360 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x5584383b9590 tx=0x7fa360011630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b6ca6ad66b44a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:27.403 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.404+0000 7fa355ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:54:27.403 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.404+0000 7fa355ffb640 1 -- 172.21.15.49:0/1872207144 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa360019070 con 0x5584383b8340 2024-03-18T00:54:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.404+0000 7fa37799d740 1 -- 172.21.15.49:0/1872207144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3680020b0 con 0x5584383b8340 2024-03-18T00:54:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.404+0000 7fa355ffb640 1 -- 172.21.15.49:0/1872207144 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa360015660 con 0x5584383b8340 2024-03-18T00:54:27.405 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.405+0000 7fa355ffb640 1 -- 172.21.15.49:0/1872207144 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa360025050 con 0x5584383b8340 2024-03-18T00:54:27.409 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.405+0000 7fa37799d740 10 client.26044 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:27.409 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.409+0000 7fa37799d740 1 -- 172.21.15.49:0/1872207144 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5584380fe620 con 0x5584383b8340 2024-03-18T00:54:27.414 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.412+0000 7fa355ffb640 1 -- 172.21.15.49:0/1872207144 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa3680020b0 con 0x5584383b8340 2024-03-18T00:54:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.412+0000 7fa355ffb640 1 -- 172.21.15.49:0/1872207144 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa360014ae0 con 0x5584383b8340 2024-03-18T00:54:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.414+0000 7fa37799d740 10 client.26044 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.414+0000 7fa37799d740 10 client.26044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:27.416 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.414+0000 7fa37799d740 20 client.26044 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:27.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.414+0000 7fa37799d740 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5584383ce060 0x5584383d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.414+0000 7fa37799d740 4 client.26044 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:27.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.414+0000 7fa37799d740 1 -- 172.21.15.49:0/1872207144 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55843832af80 con 0x5584383ce060 2024-03-18T00:54:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.419+0000 7fa36dd9b640 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5584383ce060 0x5584383d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.425 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.425+0000 7fa36dd9b640 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5584383ce060 0x5584383d04e0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d2cec22cbd48779 server_cookie=44367eca9aa3929a in_seq=0 out_seq=0 2024-03-18T00:54:27.428 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.428+0000 7fa355ffb640 10 client.26044 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:27.432 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.430+0000 7fa355ffb640 1 -- 172.21.15.49:0/1872207144 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55843832af80 con 0x5584383ce060 2024-03-18T00:54:27.433 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.431+0000 7fa355ffb640 10 client.26044 handle_command_reply: tid=0 2024-03-18T00:54:27.434 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.434+0000 7fa37799d740 1 client.26044 shutdown 2024-03-18T00:54:27.439 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.440+0000 7fa37799d740 1 -- 172.21.15.49:0/1872207144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 msgr2=0x5584383c2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:27.440 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.440+0000 7fa37799d740 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 0x5584383c2360 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x5584383b9590 tx=0x7fa360011630 comp rx=0 tx=0).stop 2024-03-18T00:54:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.441+0000 7fa37799d740 1 -- 172.21.15.49:0/1872207144 shutdown_connections 2024-03-18T00:54:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.441+0000 7fa37799d740 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5584383ce060 0x5584383d04e0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.441+0000 7fa37799d740 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383c28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.441+0000 7fa37799d740 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383bdea0 0x5584383c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.441+0000 7fa37799d740 1 --2- 172.21.15.49:0/1872207144 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 0x5584383c2360 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.441+0000 7fa37799d740 1 -- 172.21.15.49:0/1872207144 >> 172.21.15.49:0/1872207144 conn(0x5584383b38b0 msgr2=0x5584383b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.441+0000 7fa355ffb640 0 client.26044 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:27.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.443+0000 7fa37799d740 1 -- 172.21.15.49:0/1872207144 shutdown_connections 2024-03-18T00:54:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.444+0000 7fa37799d740 1 -- 172.21.15.49:0/1872207144 wait complete. 2024-03-18T00:54:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.444+0000 7fa37799d740 20 client.26044 trim_cache size 0 max 16384 2024-03-18T00:54:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.514+0000 7fa37799d740 1 Processor -- start 2024-03-18T00:54:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.515+0000 7fa37799d740 1 -- start start 2024-03-18T00:54:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.515+0000 7fa37799d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b38b0 0x5584383c2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.515+0000 7fa37799d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 0x5584383c9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.515+0000 7fa37799d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.515+0000 7fa37799d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701af330 con 0x5584383b38b0 2024-03-18T00:54:27.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.515+0000 7fa37799d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701b0e70 con 0x5584383b8d30 2024-03-18T00:54:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.515+0000 7fa37799d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701afa00 con 0x5584383b8340 2024-03-18T00:54:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.516+0000 7fa36e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b38b0 0x5584383c2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.516+0000 7fa36e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b38b0 0x5584383c2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47642/0 (socket says 172.21.15.49:47642) 2024-03-18T00:54:27.516 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.516+0000 7fa36e59c640 1 -- 172.21.15.49:0/1507339711 learned_addr learned my addr 172.21.15.49:0/1507339711 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:27.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.518+0000 7fa36dd9b640 1 --2- 172.21.15.49:0/1507339711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 0x5584383c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.518+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/1507339711 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.518+0000 7fa36e59c640 1 -- 172.21.15.49:0/1507339711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 msgr2=0x5584383c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.518+0000 7fa36e59c640 1 --2- 172.21.15.49:0/1507339711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 0x5584383c9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.518+0000 7fa36e59c640 1 -- 172.21.15.49:0/1507339711 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 msgr2=0x5584383b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:27.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.518+0000 7fa36e59c640 1 --2- 172.21.15.49:0/1507339711 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.518+0000 7fa36e59c640 1 -- 172.21.15.49:0/1507339711 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584383a5140 con 0x5584383b38b0 2024-03-18T00:54:27.522 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.522+0000 7fa36e59c640 1 --2- 172.21.15.49:0/1507339711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b38b0 0x5584383c2640 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa360014e80 tx=0x7fa360009410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b6897d0f20033d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:27.524 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.524+0000 7fa357fff640 1 -- 172.21.15.49:0/1507339711 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa360019040 con 0x5584383b38b0 2024-03-18T00:54:27.524 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.524+0000 7fa357fff640 1 -- 172.21.15.49:0/1507339711 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa36002e070 con 0x5584383b38b0 2024-03-18T00:54:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.526+0000 7fa37799d740 1 -- 172.21.15.49:0/1507339711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b38b0 msgr2=0x5584383c2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:27.526 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.526+0000 7fa37799d740 1 --2- 172.21.15.49:0/1507339711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b38b0 0x5584383c2640 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa360014e80 tx=0x7fa360009410 comp rx=0 tx=0).stop 2024-03-18T00:54:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.532+0000 7fa37799d740 1 -- 172.21.15.49:0/1507339711 shutdown_connections 2024-03-18T00:54:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.532+0000 7fa37799d740 1 --2- 172.21.15.49:0/1507339711 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8d30 0x5584383b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.532+0000 7fa37799d740 1 --2- 172.21.15.49:0/1507339711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b8340 0x5584383c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.532+0000 7fa37799d740 1 --2- 172.21.15.49:0/1507339711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b38b0 0x5584383c2640 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.532+0000 7fa37799d740 1 -- 172.21.15.49:0/1507339711 >> 172.21.15.49:0/1507339711 conn(0x5584383bd970 msgr2=0x5584383b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.534+0000 7fa37799d740 1 -- 172.21.15.49:0/1507339711 shutdown_connections 2024-03-18T00:54:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.538+0000 7fa37799d740 1 -- 172.21.15.49:0/1507339711 wait complete. 2024-03-18T00:54:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.539+0000 7fa37799d740 1 -- start start 2024-03-18T00:54:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.539+0000 7fa37799d740 1 Processor -- start 2024-03-18T00:54:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.539+0000 7fa37799d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b38b0 0x5584383c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.539+0000 7fa37799d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8340 0x5584383c5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.539+0000 7fa37799d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8d30 0x5584383c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.539+0000 7fa37799d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701b07a0 con 0x5584383b8d30 2024-03-18T00:54:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.539+0000 7fa37799d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701b00d0 con 0x5584383b8340 2024-03-18T00:54:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.539+0000 7fa37799d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa3701ad310 con 0x5584383b38b0 2024-03-18T00:54:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.539+0000 7fa37799d740 10 client.0 _ll_register_callbacks cb 0x7fa360009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.540+0000 7fa36ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8d30 0x5584383c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.540+0000 7fa36ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8d30 0x5584383c8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47644/0 (socket says 172.21.15.49:47644) 2024-03-18T00:54:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.540+0000 7fa36ed9d640 1 -- 172.21.15.49:0/3647041449 learned_addr learned my addr 172.21.15.49:0/3647041449 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.542+0000 7fa36e59c640 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b38b0 0x5584383c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.545+0000 7fa36ed9d640 1 -- 172.21.15.49:0/3647041449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b38b0 msgr2=0x5584383c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.545+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b38b0 0x5584383c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.545+0000 7fa36ed9d640 1 -- 172.21.15.49:0/3647041449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8340 msgr2=0x5584383c5e20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.545+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8340 0x5584383c5e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.545+0000 7fa36ed9d640 1 -- 172.21.15.49:0/3647041449 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584383c9350 con 0x5584383b8d30 2024-03-18T00:54:27.552 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.552+0000 7fa36ed9d640 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8d30 0x5584383c8d60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa3680c5590 tx=0x7fa368002130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1a235993a6ad24 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:27.556 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.554+0000 7fa355ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:27.567 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.563+0000 7fa355ffb640 1 -- 172.21.15.49:0/3647041449 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3680d5070 con 0x5584383b8d30 2024-03-18T00:54:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.563+0000 7fa355ffb640 1 -- 172.21.15.49:0/3647041449 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa3680dd0d0 con 0x5584383b8d30 2024-03-18T00:54:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.568+0000 7fa37799d740 1 -- 172.21.15.49:0/3647041449 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa36002e070 con 0x5584383b8d30 2024-03-18T00:54:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.572+0000 7fa355ffb640 1 -- 172.21.15.49:0/3647041449 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa3680dd2d0 con 0x5584383b8d30 2024-03-18T00:54:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.572+0000 7fa37799d740 10 client.16509 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.573+0000 7fa37799d740 1 -- 172.21.15.49:0/3647041449 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558438456e50 con 0x5584383b8d30 2024-03-18T00:54:27.577 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.577+0000 7fa355ffb640 1 -- 172.21.15.49:0/3647041449 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa36002e070 con 0x5584383b8d30 2024-03-18T00:54:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.577+0000 7fa355ffb640 1 -- 172.21.15.49:0/3647041449 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa36803c030 con 0x5584383b8d30 2024-03-18T00:54:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.577+0000 7fa37799d740 10 client.16509 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.577+0000 7fa37799d740 10 client.16509 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.577+0000 7fa37799d740 20 client.16509 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.577+0000 7fa37799d740 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5584383bdea0 0x5584384598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.577+0000 7fa37799d740 4 client.16509 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:54:27.578 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.577+0000 7fa37799d740 1 -- 172.21.15.49:0/3647041449 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5584383252d0 con 0x5584383bdea0 2024-03-18T00:54:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.584+0000 7fa36e59c640 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5584383bdea0 0x5584384598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.590+0000 7fa36e59c640 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5584383bdea0 0x5584384598b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a91d0ddcc0a6444 server_cookie=b71d2272d70f9743 in_seq=0 out_seq=0 2024-03-18T00:54:27.591 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:27.590+0000 7fa355ffb640 10 client.16509 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:29.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:28 smithi049 ceph-mon[32130]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 855 MiB/s wr, 2.04k op/s 2024-03-18T00:54:29.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:28 smithi107 ceph-mon[36270]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 855 MiB/s wr, 2.04k op/s 2024-03-18T00:54:29.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:28 smithi106 ceph-mon[35899]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 855 MiB/s wr, 2.04k op/s 2024-03-18T00:54:31.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:30 smithi049 ceph-mon[32130]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1010 MiB/s wr, 2.40k op/s 2024-03-18T00:54:31.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:30 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:54:31.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:30 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:54:31.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:30 smithi107 ceph-mon[36270]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1010 MiB/s wr, 2.40k op/s 2024-03-18T00:54:31.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:30 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:54:31.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:30 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:54:31.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:30 smithi106 ceph-mon[35899]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1010 MiB/s wr, 2.40k op/s 2024-03-18T00:54:31.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:30 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:54:31.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:30 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:54:33.133 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:32 smithi049 ceph-mon[32130]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 821 MiB/s wr, 1.96k op/s 2024-03-18T00:54:33.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:32 smithi107 ceph-mon[36270]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 821 MiB/s wr, 1.96k op/s 2024-03-18T00:54:33.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:32 smithi106 ceph-mon[35899]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 821 MiB/s wr, 1.96k op/s 2024-03-18T00:54:34.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:33 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:54:34.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:33 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:54:34.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:33 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:54:35.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:34 smithi049 ceph-mon[32130]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1014 MiB/s wr, 2.22k op/s 2024-03-18T00:54:35.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:34 smithi107 ceph-mon[36270]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1014 MiB/s wr, 2.22k op/s 2024-03-18T00:54:35.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:34 smithi106 ceph-mon[35899]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1014 MiB/s wr, 2.22k op/s 2024-03-18T00:54:35.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.302+0000 7fa355ffb640 1 -- 172.21.15.49:0/3647041449 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5584383252d0 con 0x5584383bdea0 2024-03-18T00:54:35.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.302+0000 7fa355ffb640 10 client.16509 handle_command_reply: tid=0 2024-03-18T00:54:35.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.302+0000 7fa37799d740 1 client.16509 shutdown 2024-03-18T00:54:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.302+0000 7fa37799d740 1 -- 172.21.15.49:0/3647041449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8d30 msgr2=0x5584383c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:35.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.302+0000 7fa37799d740 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8d30 0x5584383c8d60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fa3680c5590 tx=0x7fa368002130 comp rx=0 tx=0).stop 2024-03-18T00:54:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa37799d740 1 -- 172.21.15.49:0/3647041449 shutdown_connections 2024-03-18T00:54:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa37799d740 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5584383bdea0 0x5584384598b0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa37799d740 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5584383b8340 0x5584383c5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa37799d740 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5584383b8d30 0x5584383c8d60 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa37799d740 1 --2- 172.21.15.49:0/3647041449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5584383b38b0 0x5584383c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa37799d740 1 -- 172.21.15.49:0/3647041449 >> 172.21.15.49:0/3647041449 conn(0x5584383bd970 msgr2=0x5584383b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa355ffb640 0 client.16509 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa37799d740 1 -- 172.21.15.49:0/3647041449 shutdown_connections 2024-03-18T00:54:35.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa37799d740 1 -- 172.21.15.49:0/3647041449 wait complete. 2024-03-18T00:54:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.306+0000 7fa37799d740 20 client.16509 trim_cache size 0 max 16384 2024-03-18T00:54:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:54:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:54:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:54:35.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:54:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.313+0000 7fa374dba640 1 -- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3580a29a0 msgr2=0x7fa3580a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.313+0000 7fa374dba640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3580a29a0 0x7fa3580a4e60 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa37019f560 tx=0x7fa364009630 comp rx=0 tx=0).stop 2024-03-18T00:54:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.313+0000 7fa374dba640 1 -- 172.21.15.49:0/810316121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa370105390 msgr2=0x7fa3701a5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.313+0000 7fa374dba640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa370105390 0x7fa3701a5600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa36802c630 tx=0x7fa368032ea0 comp rx=0 tx=0).stop 2024-03-18T00:54:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.316+0000 7fa374dba640 1 -- 172.21.15.49:0/810316121 shutdown_connections 2024-03-18T00:54:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.316+0000 7fa374dba640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3580a29a0 0x7fa3580a4e60 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.316+0000 7fa374dba640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa370105390 0x7fa3701a5600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.316+0000 7fa374dba640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3701029e0 0x7fa37019e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.316+0000 7fa374dba640 1 --2- 172.21.15.49:0/810316121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa370100070 0x7fa37019e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.316+0000 7fa374dba640 1 -- 172.21.15.49:0/810316121 >> 172.21.15.49:0/810316121 conn(0x7fa3700fbc70 msgr2=0x7fa370100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.317+0000 7fa374dba640 1 -- 172.21.15.49:0/810316121 shutdown_connections 2024-03-18T00:54:35.317 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:35.317+0000 7fa374dba640 1 -- 172.21.15.49:0/810316121 wait complete. 2024-03-18T00:54:36.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:54:36.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:54:36.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:35 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:54:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:54:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:54:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:35 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:54:36.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:54:36.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:54:36.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:35 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:54:37.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:36 smithi049 ceph-mon[32130]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 829 MiB/s wr, 1.64k op/s 2024-03-18T00:54:37.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:36 smithi107 ceph-mon[36270]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 829 MiB/s wr, 1.64k op/s 2024-03-18T00:54:37.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:36 smithi106 ceph-mon[35899]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 829 MiB/s wr, 1.64k op/s 2024-03-18T00:54:39.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:38 smithi049 ceph-mon[32130]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 860 MiB/s wr, 1.74k op/s 2024-03-18T00:54:39.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:38 smithi107 ceph-mon[36270]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 860 MiB/s wr, 1.74k op/s 2024-03-18T00:54:39.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:38 smithi106 ceph-mon[35899]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 860 MiB/s wr, 1.74k op/s 2024-03-18T00:54:41.132 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:40 smithi049 ceph-mon[32130]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 999 MiB/s wr, 2.14k op/s 2024-03-18T00:54:41.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:40 smithi107 ceph-mon[36270]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 999 MiB/s wr, 2.14k op/s 2024-03-18T00:54:41.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:40 smithi106 ceph-mon[35899]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 999 MiB/s wr, 2.14k op/s 2024-03-18T00:54:43.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:42 smithi049 ceph-mon[32130]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 818 MiB/s wr, 1.94k op/s 2024-03-18T00:54:43.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:42 smithi107 ceph-mon[36270]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 818 MiB/s wr, 1.94k op/s 2024-03-18T00:54:43.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:42 smithi106 ceph-mon[35899]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 818 MiB/s wr, 1.94k op/s 2024-03-18T00:54:45.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:44 smithi049 ceph-mon[32130]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 964 MiB/s wr, 2.41k op/s 2024-03-18T00:54:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:44 smithi107 ceph-mon[36270]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 964 MiB/s wr, 2.41k op/s 2024-03-18T00:54:45.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:44 smithi106 ceph-mon[35899]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 964 MiB/s wr, 2.41k op/s 2024-03-18T00:54:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:46 smithi107 ceph-mon[36270]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 830 MiB/s wr, 2.15k op/s 2024-03-18T00:54:47.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:46 smithi106 ceph-mon[35899]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 830 MiB/s wr, 2.15k op/s 2024-03-18T00:54:47.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:46 smithi049 ceph-mon[32130]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 830 MiB/s wr, 2.15k op/s 2024-03-18T00:54:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.617+0000 7f3cae434640 1 -- 172.21.15.49:0/4240423030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8109950 msgr2=0x7f3ca8110270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.617+0000 7f3cae434640 1 --2- 172.21.15.49:0/4240423030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8109950 0x7f3ca8110270 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0009ca0 tx=0x7f3ca002c140 comp rx=0 tx=0).stop 2024-03-18T00:54:48.619 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.618+0000 7f3cae434640 1 -- 172.21.15.49:0/4240423030 shutdown_connections 2024-03-18T00:54:48.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.618+0000 7f3cae434640 1 --2- 172.21.15.49:0/4240423030 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8109950 0x7f3ca8110270 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.618+0000 7f3cae434640 1 --2- 172.21.15.49:0/4240423030 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ca8108f90 0x7f3ca8109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.618+0000 7f3cae434640 1 --2- 172.21.15.49:0/4240423030 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca8107e40 0x7f3ca8108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.618+0000 7f3cae434640 1 -- 172.21.15.49:0/4240423030 >> 172.21.15.49:0/4240423030 conn(0x7f3ca8076df0 msgr2=0x7f3ca8079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.622+0000 7f3cae434640 1 -- 172.21.15.49:0/4240423030 shutdown_connections 2024-03-18T00:54:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.624+0000 7f3cae434640 1 -- 172.21.15.49:0/4240423030 wait complete. 2024-03-18T00:54:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.625+0000 7f3cae434640 1 Processor -- start 2024-03-18T00:54:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.625+0000 7f3cae434640 1 -- start start 2024-03-18T00:54:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.626+0000 7f3cae434640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ca8107e40 0x7f3ca8070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.626+0000 7f3cae434640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8108f90 0x7f3ca80706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.626+0000 7f3cae434640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca8109950 0x7f3ca81ae250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.626+0000 7f3cae434640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca8113b40 con 0x7f3ca8107e40 2024-03-18T00:54:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.626+0000 7f3cae434640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca81139c0 con 0x7f3ca8109950 2024-03-18T00:54:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.626+0000 7f3cae434640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca8113cc0 con 0x7f3ca8108f90 2024-03-18T00:54:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.627+0000 7f3ca77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8108f90 0x7f3ca80706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.628+0000 7f3ca77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8108f90 0x7f3ca80706b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59432/0 (socket says 172.21.15.49:59432) 2024-03-18T00:54:48.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.628+0000 7f3ca77fe640 1 -- 172.21.15.49:0/1057226003 learned_addr learned my addr 172.21.15.49:0/1057226003 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:48.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.631+0000 7f3cac9d1640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca8109950 0x7f3ca81ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.633+0000 7f3ca77fe640 1 -- 172.21.15.49:0/1057226003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca8109950 msgr2=0x7f3ca81ae250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:48.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.633+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca8109950 0x7f3ca81ae250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.633+0000 7f3ca77fe640 1 -- 172.21.15.49:0/1057226003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ca8107e40 msgr2=0x7f3ca8070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:48.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.633+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ca8107e40 0x7f3ca8070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.633+0000 7f3ca77fe640 1 -- 172.21.15.49:0/1057226003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca81ae790 con 0x7f3ca8108f90 2024-03-18T00:54:48.639 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.636+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8108f90 0x7f3ca80706b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000e9c0 tx=0x7f3c9000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.638+0000 7f3ca57fa640 1 -- 172.21.15.49:0/1057226003 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9000cd80 con 0x7f3ca8108f90 2024-03-18T00:54:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.638+0000 7f3ca57fa640 1 -- 172.21.15.49:0/1057226003 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c900035f0 con 0x7f3ca8108f90 2024-03-18T00:54:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.638+0000 7f3ca57fa640 1 -- 172.21.15.49:0/1057226003 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c90010690 con 0x7f3ca8108f90 2024-03-18T00:54:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.639+0000 7f3cae434640 1 -- 172.21.15.49:0/1057226003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca81ae960 con 0x7f3ca8108f90 2024-03-18T00:54:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.640+0000 7f3cae434640 1 -- 172.21.15.49:0/1057226003 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca81aedd0 con 0x7f3ca8108f90 2024-03-18T00:54:48.649 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.647+0000 7f3cb1019740 1 Processor -- start 2024-03-18T00:54:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.650+0000 7f3cb1019740 1 -- start start 2024-03-18T00:54:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.650+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1a340 0x56413ce1a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.650+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce1f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.650+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 0x56413ce22330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.650+0000 7f3cb1019740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56413cd7fa40 con 0x56413ce1ad30 2024-03-18T00:54:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.650+0000 7f3cb1019740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56413cd28920 con 0x56413ce1a340 2024-03-18T00:54:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.650+0000 7f3cb1019740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56413cd7bae0 con 0x56413ce1fea0 2024-03-18T00:54:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.651+0000 7f3ca77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 0x56413ce22330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.651+0000 7f3ca77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 0x56413ce22330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59444/0 (socket says 172.21.15.49:59444) 2024-03-18T00:54:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.651+0000 7f3ca77fe640 1 -- 172.21.15.49:0/2804482503 learned_addr learned my addr 172.21.15.49:0/2804482503 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:48.655 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.653+0000 7f3ca77fe640 1 -- 172.21.15.49:0/2804482503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1a340 msgr2=0x56413ce1a760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:48.655 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.653+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/2804482503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1a340 0x56413ce1a760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.655 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.653+0000 7f3ca77fe640 1 -- 172.21.15.49:0/2804482503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 msgr2=0x56413ce1f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:48.655 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.653+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/2804482503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce1f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.655 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.653+0000 7f3ca77fe640 1 -- 172.21.15.49:0/2804482503 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413cd40460 con 0x56413ce1fea0 2024-03-18T00:54:48.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.655+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/2804482503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 0x56413ce22330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3c900d2db0 tx=0x7f3c900d74b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2fc670065adcc6e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:48.658 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.657+0000 7f3c9e7fc640 1 -- 172.21.15.49:0/2804482503 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c900e2070 con 0x56413ce1fea0 2024-03-18T00:54:48.658 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.657+0000 7f3c9e7fc640 1 -- 172.21.15.49:0/2804482503 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c900dd8e0 con 0x56413ce1fea0 2024-03-18T00:54:48.658 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.657+0000 7f3c9e7fc640 1 -- 172.21.15.49:0/2804482503 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c900dca10 con 0x56413ce1fea0 2024-03-18T00:54:48.659 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.658+0000 7f3ca57fa640 1 -- 172.21.15.49:0/1057226003 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3c900037a0 con 0x7f3ca8108f90 2024-03-18T00:54:48.661 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.659+0000 7f3ca57fa640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3c940a2ae0 0x7f3c940a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.661 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.660+0000 7f3ca57fa640 1 -- 172.21.15.49:0/1057226003 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3c900ceee0 con 0x7f3ca8108f90 2024-03-18T00:54:48.661 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.660+0000 7f3cb1019740 1 -- 172.21.15.49:0/2804482503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 msgr2=0x56413ce22330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:48.661 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.660+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2804482503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 0x56413ce22330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3c900d2db0 tx=0x7f3c900d74b0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.662 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.660+0000 7f3cac9d1640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3c940a2ae0 0x7f3c940a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.667 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.665+0000 7f3cac9d1640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3c940a2ae0 0x7f3c940a4fa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0006010 tx=0x7f3ca002e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.667+0000 7f3cb1019740 1 -- 172.21.15.49:0/2804482503 shutdown_connections 2024-03-18T00:54:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.667+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2804482503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 0x56413ce22330 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.667+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2804482503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce1f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.667+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2804482503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1a340 0x56413ce1a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.668 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.667+0000 7f3cb1019740 1 -- 172.21.15.49:0/2804482503 >> 172.21.15.49:0/2804482503 conn(0x56413ce158b0 msgr2=0x56413ce15cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:48.669 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.668+0000 7f3cb1019740 1 -- 172.21.15.49:0/2804482503 shutdown_connections 2024-03-18T00:54:48.674 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.673+0000 7f3cb1019740 1 -- 172.21.15.49:0/2804482503 wait complete. 2024-03-18T00:54:48.674 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.673+0000 7f3cb1019740 1 -- start start 2024-03-18T00:54:48.674 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.674+0000 7f3cb1019740 1 Processor -- start 2024-03-18T00:54:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.674+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 0x56413ce356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.674+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce35c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.674+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1fea0 0x56413ce3cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.674+0000 7f3cb1019740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56413ce33e60 con 0x56413ce1ad30 2024-03-18T00:54:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.674+0000 7f3cb1019740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56413cd3bd70 con 0x56413ce1fea0 2024-03-18T00:54:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.674+0000 7f3cb1019740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56413ce33fe0 con 0x56413ce1a340 2024-03-18T00:54:48.675 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.674+0000 7f3cb1019740 10 client.0 _ll_register_callbacks cb 0x56413ca93a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.675+0000 7f3ca77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce35c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.675+0000 7f3ca77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce35c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36852/0 (socket says 172.21.15.49:36852) 2024-03-18T00:54:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.675+0000 7f3ca77fe640 1 -- 172.21.15.49:0/2153565920 learned_addr learned my addr 172.21.15.49:0/2153565920 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.676+0000 7f3ca77fe640 1 -- 172.21.15.49:0/2153565920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 msgr2=0x56413ce356e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.676+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 0x56413ce356e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.676+0000 7f3ca77fe640 1 -- 172.21.15.49:0/2153565920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1fea0 msgr2=0x56413ce3cc60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:54:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.676+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1fea0 0x56413ce3cc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.676+0000 7f3ca77fe640 1 -- 172.21.15.49:0/2153565920 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56413ca94210 con 0x56413ce1ad30 2024-03-18T00:54:48.679 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.678+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce35c20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3c90097c60 tx=0x7f3c900dc410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94361ccb08629740 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:48.679 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.678+0000 7f3c877fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:48.681 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.679+0000 7f3c877fe640 1 -- 172.21.15.49:0/2153565920 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c9009b5d0 con 0x56413ce1ad30 2024-03-18T00:54:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.680+0000 7f3c877fe640 1 -- 172.21.15.49:0/2153565920 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3c9001fdf0 con 0x56413ce1ad30 2024-03-18T00:54:48.682 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.682+0000 7f3cb1019740 1 -- 172.21.15.49:0/2153565920 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c900dd8e0 con 0x56413ce1ad30 2024-03-18T00:54:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.682+0000 7f3c877fe640 1 -- 172.21.15.49:0/2153565920 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c9009b160 con 0x56413ce1ad30 2024-03-18T00:54:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.683+0000 7f3cb1019740 10 client.16515 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:48.683 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.683+0000 7f3cb1019740 1 -- 172.21.15.49:0/2153565920 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56413cb60620 con 0x56413ce1ad30 2024-03-18T00:54:48.685 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.684+0000 7f3c877fe640 1 -- 172.21.15.49:0/2153565920 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c900dd8e0 con 0x56413ce1ad30 2024-03-18T00:54:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.685+0000 7f3c877fe640 1 -- 172.21.15.49:0/2153565920 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3c900afbe0 con 0x56413ce1ad30 2024-03-18T00:54:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.685+0000 7f3cb1019740 10 client.16515 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.685+0000 7f3cb1019740 10 client.16515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:48.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.685+0000 7f3cb1019740 20 client.16515 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.685+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56413ce41470 0x56413ce438f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.685+0000 7f3cb1019740 4 client.16515 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:48.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.685+0000 7f3cb1019740 1 -- 172.21.15.49:0/2153565920 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56413cd8cf80 con 0x56413ce41470 2024-03-18T00:54:48.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.688+0000 7f3ca7fff640 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56413ce41470 0x56413ce438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.694+0000 7f3ca7fff640 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56413ce41470 0x56413ce438f0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=873cacc323f4d4a server_cookie=5847c7a33fe9c0ea in_seq=0 out_seq=0 2024-03-18T00:54:48.696 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.694+0000 7f3c877fe640 10 client.16515 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.706+0000 7f3c877fe640 1 -- 172.21.15.49:0/2153565920 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56413cd8cf80 con 0x56413ce41470 2024-03-18T00:54:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.706+0000 7f3c877fe640 10 client.16515 handle_command_reply: tid=0 2024-03-18T00:54:48.708 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.707+0000 7f3cb1019740 1 client.16515 shutdown 2024-03-18T00:54:48.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.714+0000 7f3cb1019740 1 -- 172.21.15.49:0/2153565920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 msgr2=0x56413ce35c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:48.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.714+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce35c20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f3c90097c60 tx=0x7f3c900dc410 comp rx=0 tx=0).stop 2024-03-18T00:54:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.718+0000 7f3cb1019740 1 -- 172.21.15.49:0/2153565920 shutdown_connections 2024-03-18T00:54:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.718+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56413ce41470 0x56413ce438f0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.719+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1fea0 0x56413ce3cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.719+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce35c20 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.719+0000 7f3cb1019740 1 --2- 172.21.15.49:0/2153565920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 0x56413ce356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.719+0000 7f3cb1019740 1 -- 172.21.15.49:0/2153565920 >> 172.21.15.49:0/2153565920 conn(0x56413ce158b0 msgr2=0x56413ce14bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.719+0000 7f3c877fe640 0 client.16515 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:48.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.720+0000 7f3cb1019740 1 -- 172.21.15.49:0/2153565920 shutdown_connections 2024-03-18T00:54:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.720+0000 7f3cb1019740 1 -- 172.21.15.49:0/2153565920 wait complete. 2024-03-18T00:54:48.723 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.721+0000 7f3cb1019740 20 client.16515 trim_cache size 0 max 16384 2024-03-18T00:54:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.759+0000 7f3cb1019740 1 Processor -- start 2024-03-18T00:54:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.760+0000 7f3cb1019740 1 -- start start 2024-03-18T00:54:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.760+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1a340 0x56413ce18b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.760+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1ad30 0x56413ce3cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.760+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 0x56413ce19050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.760+0000 7f3cb1019740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8800e610 con 0x56413ce1a340 2024-03-18T00:54:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.760+0000 7f3cb1019740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3c88010150 con 0x56413ce1ad30 2024-03-18T00:54:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.760+0000 7f3cb1019740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8800ece0 con 0x56413ce1fea0 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.760+0000 7f3ca7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1a340 0x56413ce18b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.760+0000 7f3ca7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1a340 0x56413ce18b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36856/0 (socket says 172.21.15.49:36856) 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.761+0000 7f3ca7fff640 1 -- 172.21.15.49:0/697191968 learned_addr learned my addr 172.21.15.49:0/697191968 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.761+0000 7f3ca7fff640 1 -- 172.21.15.49:0/697191968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 msgr2=0x56413ce19050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.761+0000 7f3ca7fff640 1 --2- 172.21.15.49:0/697191968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 0x56413ce19050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.761+0000 7f3ca7fff640 1 -- 172.21.15.49:0/697191968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1ad30 msgr2=0x56413ce3cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.761+0000 7f3ca7fff640 1 --2- 172.21.15.49:0/697191968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1ad30 0x56413ce3cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.761+0000 7f3ca7fff640 1 -- 172.21.15.49:0/697191968 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56413ce16300 con 0x56413ce1a340 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.761+0000 7f3ca7fff640 1 --2- 172.21.15.49:0/697191968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1a340 0x56413ce18b10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3c98004810 tx=0x7f3c98009a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96afd375880aa693 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:48.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.762+0000 7f3c9dffb640 1 -- 172.21.15.49:0/697191968 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98016110 con 0x56413ce1a340 2024-03-18T00:54:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.762+0000 7f3c9dffb640 1 -- 172.21.15.49:0/697191968 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c9801a030 con 0x56413ce1a340 2024-03-18T00:54:48.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.762+0000 7f3c9dffb640 1 -- 172.21.15.49:0/697191968 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98014d00 con 0x56413ce1a340 2024-03-18T00:54:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.763+0000 7f3cb1019740 1 -- 172.21.15.49:0/697191968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1a340 msgr2=0x56413ce18b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.763+0000 7f3cb1019740 1 --2- 172.21.15.49:0/697191968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1a340 0x56413ce18b10 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3c98004810 tx=0x7f3c98009a80 comp rx=0 tx=0).stop 2024-03-18T00:54:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.763+0000 7f3cb1019740 1 -- 172.21.15.49:0/697191968 shutdown_connections 2024-03-18T00:54:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.764+0000 7f3cb1019740 1 --2- 172.21.15.49:0/697191968 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1ad30 0x56413ce3cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.764+0000 7f3cb1019740 1 --2- 172.21.15.49:0/697191968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1fea0 0x56413ce19050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.764+0000 7f3cb1019740 1 --2- 172.21.15.49:0/697191968 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1a340 0x56413ce18b10 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.764+0000 7f3cb1019740 1 -- 172.21.15.49:0/697191968 >> 172.21.15.49:0/697191968 conn(0x56413ce158b0 msgr2=0x56413ce1c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.764+0000 7f3cb1019740 1 -- 172.21.15.49:0/697191968 shutdown_connections 2024-03-18T00:54:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.764+0000 7f3cb1019740 1 -- 172.21.15.49:0/697191968 wait complete. 2024-03-18T00:54:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.765+0000 7f3cb1019740 1 -- start start 2024-03-18T00:54:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.766+0000 7f3cb1019740 1 Processor -- start 2024-03-18T00:54:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.766+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 0x56413ce3a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.766+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce3a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.766+0000 7f3cb1019740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1fea0 0x56413ce3b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.766+0000 7f3cb1019740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8800fa80 con 0x56413ce1ad30 2024-03-18T00:54:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.766+0000 7f3cb1019740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8800f3b0 con 0x56413ce1fea0 2024-03-18T00:54:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.766+0000 7f3cb1019740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8800c5f0 con 0x56413ce1a340 2024-03-18T00:54:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.766+0000 7f3cb1019740 10 client.0 _ll_register_callbacks cb 0x7f3c900e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.767+0000 7f3ca77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce3a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.768 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.767+0000 7f3ca77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce3a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36868/0 (socket says 172.21.15.49:36868) 2024-03-18T00:54:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.767+0000 7f3ca77fe640 1 -- 172.21.15.49:0/4088288461 learned_addr learned my addr 172.21.15.49:0/4088288461 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.768+0000 7f3ca7fff640 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 0x56413ce3a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.768+0000 7f3ca77fe640 1 -- 172.21.15.49:0/4088288461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 msgr2=0x56413ce3a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.768+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 0x56413ce3a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.769 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.768+0000 7f3ca77fe640 1 -- 172.21.15.49:0/4088288461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1fea0 msgr2=0x56413ce3b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.768+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1fea0 0x56413ce3b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.768+0000 7f3ca77fe640 1 -- 172.21.15.49:0/4088288461 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56413ce3bc20 con 0x56413ce1ad30 2024-03-18T00:54:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.768+0000 7f3ca7fff640 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 0x56413ce3a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:54:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.768+0000 7f3ca77fe640 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce3a620 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3c90003e20 tx=0x7f3c900d7ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8219207a76e42241 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:48.770 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.768+0000 7f3c877fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.769+0000 7f3c877fe640 1 -- 172.21.15.49:0/4088288461 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c900b8060 con 0x56413ce1ad30 2024-03-18T00:54:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.769+0000 7f3c877fe640 1 -- 172.21.15.49:0/4088288461 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3c9001fdf0 con 0x56413ce1ad30 2024-03-18T00:54:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.769+0000 7f3cb1019740 1 -- 172.21.15.49:0/4088288461 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c9801a030 con 0x56413ce1ad30 2024-03-18T00:54:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.769+0000 7f3c877fe640 1 -- 172.21.15.49:0/4088288461 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3c9009b340 con 0x56413ce1ad30 2024-03-18T00:54:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.769+0000 7f3cb1019740 10 client.16521 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.769+0000 7f3cb1019740 1 -- 172.21.15.49:0/4088288461 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56413ce3cf30 con 0x56413ce1ad30 2024-03-18T00:54:48.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.769+0000 7f3c877fe640 1 -- 172.21.15.49:0/4088288461 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c9801a030 con 0x56413ce1ad30 2024-03-18T00:54:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.770+0000 7f3c877fe640 1 -- 172.21.15.49:0/4088288461 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f3c900a7020 con 0x56413ce1ad30 2024-03-18T00:54:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.770+0000 7f3cb1019740 10 client.16521 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.770+0000 7f3cb1019740 10 client.16521 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.770+0000 7f3cb1019740 20 client.16521 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.770+0000 7f3cb1019740 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56413cea7ce0 0x56413ceaa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.770+0000 7f3cb1019740 4 client.16521 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:54:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.770+0000 7f3cb1019740 1 -- 172.21.15.49:0/4088288461 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56413cd872d0 con 0x56413cea7ce0 2024-03-18T00:54:48.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.771+0000 7f3ca7fff640 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56413cea7ce0 0x56413ceaa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.774+0000 7f3c877fe640 10 client.16521 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:48.774+0000 7f3ca7fff640 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56413cea7ce0 0x56413ceaa100 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a7aa703eaedbd28 server_cookie=d8fe70d4ed4d0ba4 in_seq=0 out_seq=0 2024-03-18T00:54:49.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:48 smithi107 ceph-mon[36270]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 863 MiB/s wr, 2.16k op/s 2024-03-18T00:54:49.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:48 smithi106 ceph-mon[35899]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 863 MiB/s wr, 2.16k op/s 2024-03-18T00:54:49.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:48 smithi049 ceph-mon[32130]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 863 MiB/s wr, 2.16k op/s 2024-03-18T00:54:51.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:50 smithi107 ceph-mon[36270]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 946 MiB/s wr, 2.23k op/s 2024-03-18T00:54:51.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:50 smithi106 ceph-mon[35899]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 946 MiB/s wr, 2.23k op/s 2024-03-18T00:54:51.384 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:50 smithi049 ceph-mon[32130]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 946 MiB/s wr, 2.23k op/s 2024-03-18T00:54:51.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:54:51.848 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:54:52.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.252+0000 7fbcbab77640 1 -- 172.21.15.49:0/1169135700 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 msgr2=0x7fbcb4112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:52.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.252+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1169135700 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 0x7fbcb4112450 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbca4009900 tx=0x7fbca402bc40 comp rx=0 tx=0).stop 2024-03-18T00:54:52.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.265+0000 7fbcbab77640 1 -- 172.21.15.49:0/1169135700 shutdown_connections 2024-03-18T00:54:52.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.265+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1169135700 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbcb4109140 0x7fbcb410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.265+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1169135700 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 0x7fbcb4112450 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.265+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1169135700 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb41087e0 0x7fbcb4108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.265+0000 7fbcbab77640 1 -- 172.21.15.49:0/1169135700 >> 172.21.15.49:0/1169135700 conn(0x7fbcb406e640 msgr2=0x7fbcb4070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:52.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.265+0000 7fbcbab77640 1 -- 172.21.15.49:0/1169135700 shutdown_connections 2024-03-18T00:54:52.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.266+0000 7fbcbab77640 1 -- 172.21.15.49:0/1169135700 wait complete. 2024-03-18T00:54:52.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.267+0000 7fbcbab77640 1 Processor -- start 2024-03-18T00:54:52.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.267+0000 7fbcbab77640 1 -- start start 2024-03-18T00:54:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.268+0000 7fbcbab77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb41087e0 0x7fbcb41a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.268+0000 7fbcbab77640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbcb4109140 0x7fbcb41a6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.268+0000 7fbcbab77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 0x7fbcb41ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.268+0000 7fbcbab77640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb410abf0 con 0x7fbcb4110060 2024-03-18T00:54:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.268+0000 7fbcbab77640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb410aa70 con 0x7fbcb4109140 2024-03-18T00:54:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.268+0000 7fbcbab77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb410ad70 con 0x7fbcb41087e0 2024-03-18T00:54:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.268+0000 7fbcb9114640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 0x7fbcb41ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.269+0000 7fbcb9114640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 0x7fbcb41ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36880/0 (socket says 172.21.15.49:36880) 2024-03-18T00:54:52.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.269+0000 7fbcb9114640 1 -- 172.21.15.49:0/1668605675 learned_addr learned my addr 172.21.15.49:0/1668605675 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.269+0000 7fbcb8913640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb41087e0 0x7fbcb41a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.269+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbcb4109140 0x7fbcb41a6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.269+0000 7fbcb9114640 1 -- 172.21.15.49:0/1668605675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb41087e0 msgr2=0x7fbcb41a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.269+0000 7fbcb9114640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb41087e0 0x7fbcb41a6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.269+0000 7fbcb9114640 1 -- 172.21.15.49:0/1668605675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbcb4109140 msgr2=0x7fbcb41a6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.269+0000 7fbcb9114640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbcb4109140 0x7fbcb41a6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.269+0000 7fbcb9114640 1 -- 172.21.15.49:0/1668605675 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb41ae480 con 0x7fbcb4110060 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.270+0000 7fbcb9114640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 0x7fbcb41ade10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbca402c7c0 tx=0x7fbca4005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.271+0000 7fbcb1ffb640 1 -- 172.21.15.49:0/1668605675 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca4043020 con 0x7fbcb4110060 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.271+0000 7fbcb1ffb640 1 -- 172.21.15.49:0/1668605675 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbca4004350 con 0x7fbcb4110060 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.271+0000 7fbcb1ffb640 1 -- 172.21.15.49:0/1668605675 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca4005220 con 0x7fbcb4110060 2024-03-18T00:54:52.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.271+0000 7fbcbab77640 1 -- 172.21.15.49:0/1668605675 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb41ae710 con 0x7fbcb4110060 2024-03-18T00:54:52.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.271+0000 7fbcbab77640 1 -- 172.21.15.49:0/1668605675 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb41aec10 con 0x7fbcb4110060 2024-03-18T00:54:52.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.276+0000 7fbcbd839740 1 Processor -- start 2024-03-18T00:54:52.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.278+0000 7fbcb1ffb640 1 -- 172.21.15.49:0/1668605675 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbca4046020 con 0x7fbcb4110060 2024-03-18T00:54:52.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.279+0000 7fbcb1ffb640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbca00a2950 0x7fbca00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.281+0000 7fbcb1ffb640 1 -- 172.21.15.49:0/1668605675 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbca40bcb10 con 0x7fbcb4110060 2024-03-18T00:54:52.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.283+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbca00a2950 0x7fbca00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.286+0000 7fbcbd839740 1 -- start start 2024-03-18T00:54:52.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.286+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc624590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.286+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc629660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.286+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc62c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.286+0000 7fbcbd839740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559cdc589500 con 0x559cdc624b60 2024-03-18T00:54:52.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.286+0000 7fbcbd839740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559cdc531be0 con 0x559cdc624170 2024-03-18T00:54:52.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.286+0000 7fbcbd839740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559cdc5855a0 con 0x559cdc629cd0 2024-03-18T00:54:52.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.291+0000 7fbcb8913640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc624590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.291+0000 7fbcb9114640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc62c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.291+0000 7fbcb8913640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc624590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:38838/0 (socket says 172.21.15.49:38838) 2024-03-18T00:54:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.291+0000 7fbcb8913640 1 -- 172.21.15.49:0/292699707 learned_addr learned my addr 172.21.15.49:0/292699707 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:52.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.291+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/292699707 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc629660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.295+0000 7fbcb8913640 1 -- 172.21.15.49:0/292699707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 msgr2=0x559cdc62c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:52.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.295+0000 7fbcb8913640 1 --2- 172.21.15.49:0/292699707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc62c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.295+0000 7fbcb8913640 1 -- 172.21.15.49:0/292699707 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 msgr2=0x559cdc629660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:52.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.295+0000 7fbcb8913640 1 --2- 172.21.15.49:0/292699707 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc629660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.295+0000 7fbcb8913640 1 -- 172.21.15.49:0/292699707 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cdc5454b0 con 0x559cdc624170 2024-03-18T00:54:52.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.295+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbca00a2950 0x7fbca00a4e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c005fd0 tx=0x7fbc9c009470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:52.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.301+0000 7fbcb8913640 1 --2- 172.21.15.49:0/292699707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc624590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcac008e20 tx=0x7fbcac00d710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2918998577ac5e8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:52.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.304+0000 7fbcaaffd640 1 -- 172.21.15.49:0/292699707 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac00f830 con 0x559cdc624170 2024-03-18T00:54:52.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.304+0000 7fbcaaffd640 1 -- 172.21.15.49:0/292699707 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcac018480 con 0x559cdc624170 2024-03-18T00:54:52.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.304+0000 7fbcaaffd640 1 -- 172.21.15.49:0/292699707 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac00e5f0 con 0x559cdc624170 2024-03-18T00:54:52.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.308+0000 7fbcbd839740 1 -- 172.21.15.49:0/292699707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 msgr2=0x559cdc624590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:52.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.308+0000 7fbcbd839740 1 --2- 172.21.15.49:0/292699707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc624590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbcac008e20 tx=0x7fbcac00d710 comp rx=0 tx=0).stop 2024-03-18T00:54:52.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.309+0000 7fbcbd839740 1 -- 172.21.15.49:0/292699707 shutdown_connections 2024-03-18T00:54:52.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.309+0000 7fbcbd839740 1 --2- 172.21.15.49:0/292699707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc62c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.309+0000 7fbcbd839740 1 --2- 172.21.15.49:0/292699707 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc629660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.309+0000 7fbcbd839740 1 --2- 172.21.15.49:0/292699707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc624590 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.309+0000 7fbcbd839740 1 -- 172.21.15.49:0/292699707 >> 172.21.15.49:0/292699707 conn(0x559cdc61f6e0 msgr2=0x559cdc61fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:52.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.313+0000 7fbcbd839740 1 -- 172.21.15.49:0/292699707 shutdown_connections 2024-03-18T00:54:52.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.314+0000 7fbcbd839740 1 -- 172.21.15.49:0/292699707 wait complete. 2024-03-18T00:54:52.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.314+0000 7fbcbd839740 1 -- start start 2024-03-18T00:54:52.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.314+0000 7fbcbd839740 1 Processor -- start 2024-03-18T00:54:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.315+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc63f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.315+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc63fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.315+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc646cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.315+0000 7fbcbd839740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559cdc63dc90 con 0x559cdc624b60 2024-03-18T00:54:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.315+0000 7fbcbd839740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559cdc545160 con 0x559cdc624170 2024-03-18T00:54:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.315+0000 7fbcbd839740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559cdc63de10 con 0x559cdc629cd0 2024-03-18T00:54:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.315+0000 7fbcbd839740 10 client.0 _ll_register_callbacks cb 0x559cdc529150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:52.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.321+0000 7fbcb8913640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc63f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.321+0000 7fbcb8913640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc63f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:38850/0 (socket says 172.21.15.49:38850) 2024-03-18T00:54:52.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.321+0000 7fbcb8913640 1 -- 172.21.15.49:0/200012215 learned_addr learned my addr 172.21.15.49:0/200012215 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:52.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.322+0000 7fbcb9114640 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc646cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.322+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc63fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.324+0000 7fbcb3fff640 1 -- 172.21.15.49:0/200012215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 msgr2=0x559cdc646cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:52.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.324+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc646cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.324+0000 7fbcb3fff640 1 -- 172.21.15.49:0/200012215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 msgr2=0x559cdc63f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:52.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.324+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc63f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:52.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.324+0000 7fbcb3fff640 1 -- 172.21.15.49:0/200012215 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cdc3677d0 con 0x559cdc624b60 2024-03-18T00:54:52.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.329+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc63fcf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c005fa0 tx=0x7fbc9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a08863563248852 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:52.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.329+0000 7fbc93fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:54:52.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.333+0000 7fbc93fff640 1 -- 172.21.15.49:0/200012215 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c023070 con 0x559cdc624b60 2024-03-18T00:54:52.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.333+0000 7fbcbd839740 1 -- 172.21.15.49:0/200012215 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcac018480 con 0x559cdc624b60 2024-03-18T00:54:52.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.333+0000 7fbc93fff640 1 -- 172.21.15.49:0/200012215 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbc9c0109a0 con 0x559cdc624b60 2024-03-18T00:54:52.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.336+0000 7fbc93fff640 1 -- 172.21.15.49:0/200012215 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbc9c010ba0 con 0x559cdc624b60 2024-03-18T00:54:52.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.336+0000 7fbcbd839740 10 client.16533 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:52.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.336+0000 7fbcbd839740 1 -- 172.21.15.49:0/200012215 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559cdc29d3f0 con 0x559cdc624b60 2024-03-18T00:54:52.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.339+0000 7fbc93fff640 1 -- 172.21.15.49:0/200012215 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcac018480 con 0x559cdc624b60 2024-03-18T00:54:52.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.339+0000 7fbc93fff640 1 -- 172.21.15.49:0/200012215 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbc9c00f040 con 0x559cdc624b60 2024-03-18T00:54:52.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.341+0000 7fbcbd839740 10 client.16533 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:52.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.341+0000 7fbcbd839740 10 client.16533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:52.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.341+0000 7fbcbd839740 20 client.16533 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:52.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.341+0000 7fbcbd839740 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559cdc64b540 0x559cdc64d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:52.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.341+0000 7fbcbd839740 4 client.16533 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:54:52.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.341+0000 7fbcbd839740 1 -- 172.21.15.49:0/200012215 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559cdc596a40 con 0x559cdc64b540 2024-03-18T00:54:52.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.344+0000 7fbcb8913640 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559cdc64b540 0x559cdc64d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:52.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.348+0000 7fbcb8913640 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559cdc64b540 0x559cdc64d9e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b336cd0cd8cd561 server_cookie=6a8416b061cc9606 in_seq=0 out_seq=0 2024-03-18T00:54:52.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:52.348+0000 7fbc93fff640 10 client.16533 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:53.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:52 smithi107 ceph-mon[36270]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 822 MiB/s wr, 1.75k op/s 2024-03-18T00:54:53.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:52 smithi106 ceph-mon[35899]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 822 MiB/s wr, 1.75k op/s 2024-03-18T00:54:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:52 smithi049 ceph-mon[32130]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 822 MiB/s wr, 1.75k op/s 2024-03-18T00:54:55.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:54 smithi107 ceph-mon[36270]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 972 MiB/s wr, 1.75k op/s 2024-03-18T00:54:55.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:54 smithi106 ceph-mon[35899]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 972 MiB/s wr, 1.75k op/s 2024-03-18T00:54:55.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:54 smithi049 ceph-mon[32130]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 972 MiB/s wr, 1.75k op/s 2024-03-18T00:54:57.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:56 smithi107 ceph-mon[36270]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 887 MiB/s wr, 1.38k op/s 2024-03-18T00:54:57.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:56 smithi106 ceph-mon[35899]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 887 MiB/s wr, 1.38k op/s 2024-03-18T00:54:57.379 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:56 smithi049 ceph-mon[32130]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 887 MiB/s wr, 1.38k op/s 2024-03-18T00:54:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.369+0000 7f3c877fe640 1 -- 172.21.15.49:0/4088288461 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56413cd872d0 con 0x56413cea7ce0 2024-03-18T00:54:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.369+0000 7f3c877fe640 10 client.16521 handle_command_reply: tid=0 2024-03-18T00:54:58.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.369+0000 7f3cb1019740 1 client.16521 shutdown 2024-03-18T00:54:58.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.369+0000 7fbc93fff640 1 -- 172.21.15.49:0/200012215 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559cdc596a40 con 0x559cdc64b540 2024-03-18T00:54:58.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.369+0000 7fbc93fff640 10 client.16533 handle_command_reply: tid=0 2024-03-18T00:54:58.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.369+0000 7fbcbd839740 1 client.16533 shutdown 2024-03-18T00:54:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 -- 172.21.15.49:0/4088288461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 msgr2=0x56413ce3a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce3a620 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f3c90003e20 tx=0x7f3c900d7ad0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 -- 172.21.15.49:0/4088288461 shutdown_connections 2024-03-18T00:54:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56413cea7ce0 0x56413ceaa100 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56413ce1fea0 0x56413ce3b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56413ce1ad30 0x56413ce3a620 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 --2- 172.21.15.49:0/4088288461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56413ce1a340 0x56413ce3a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 -- 172.21.15.49:0/4088288461 >> 172.21.15.49:0/4088288461 conn(0x56413ce158b0 msgr2=0x56413ce1c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:58.374 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3c877fe640 0 client.16521 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:58.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 -- 172.21.15.49:0/200012215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 msgr2=0x559cdc63fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc63fcf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c005fa0 tx=0x7fbc9c0046c0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 -- 172.21.15.49:0/200012215 shutdown_connections 2024-03-18T00:54:58.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559cdc64b540 0x559cdc64d9e0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc646cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc63fcf0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 --2- 172.21.15.49:0/200012215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc63f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 -- 172.21.15.49:0/200012215 >> 172.21.15.49:0/200012215 conn(0x559cdc61f6e0 msgr2=0x559cdc61e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:58.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbc93fff640 0 client.16533 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 -- 172.21.15.49:0/4088288461 shutdown_connections 2024-03-18T00:54:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 1 -- 172.21.15.49:0/4088288461 wait complete. 2024-03-18T00:54:58.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 -- 172.21.15.49:0/200012215 shutdown_connections 2024-03-18T00:54:58.376 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.370+0000 7f3cb1019740 20 client.16521 trim_cache size 0 max 16384 2024-03-18T00:54:58.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 1 -- 172.21.15.49:0/200012215 wait complete. 2024-03-18T00:54:58.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.370+0000 7fbcbd839740 20 client.16533 trim_cache size 0 max 16384 2024-03-18T00:54:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:54:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:54:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:54:58.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:54:58.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.374+0000 7f3cae434640 1 -- 172.21.15.49:0/1057226003 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3c940a2ae0 msgr2=0x7f3c940a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.374+0000 7f3cae434640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3c940a2ae0 0x7f3c940a4fa0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3ca0006010 tx=0x7f3ca002e040 comp rx=0 tx=0).stop 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.374+0000 7f3cae434640 1 -- 172.21.15.49:0/1057226003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8108f90 msgr2=0x7f3ca80706b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.374+0000 7f3cae434640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8108f90 0x7f3ca80706b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000e9c0 tx=0x7f3c9000ee80 comp rx=0 tx=0).stop 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.375+0000 7f3cae434640 1 -- 172.21.15.49:0/1057226003 shutdown_connections 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.375+0000 7f3cae434640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3c940a2ae0 0x7f3c940a4fa0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.375+0000 7f3cae434640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3ca8109950 0x7f3ca81ae250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.375+0000 7f3cae434640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3ca8108f90 0x7f3ca80706b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.375+0000 7f3cae434640 1 --2- 172.21.15.49:0/1057226003 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3ca8107e40 0x7f3ca8070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.375+0000 7f3cae434640 1 -- 172.21.15.49:0/1057226003 >> 172.21.15.49:0/1057226003 conn(0x7f3ca8076df0 msgr2=0x7f3ca810c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.376+0000 7f3cae434640 1 -- 172.21.15.49:0/1057226003 shutdown_connections 2024-03-18T00:54:58.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:54:58.376+0000 7f3cae434640 1 -- 172.21.15.49:0/1057226003 wait complete. 2024-03-18T00:54:58.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.463+0000 7fbcbd839740 1 Processor -- start 2024-03-18T00:54:58.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.463+0000 7fbcbd839740 1 -- start start 2024-03-18T00:54:58.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.463+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc624170 0x559cdc63e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.464+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624b60 0x559cdc646c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.464+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc629cd0 0x559cdc63e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.464+0000 7fbcbd839740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9400e650 con 0x559cdc629cd0 2024-03-18T00:54:58.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.464+0000 7fbcbd839740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94010190 con 0x559cdc624b60 2024-03-18T00:54:58.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.466+0000 7fbcb3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624b60 0x559cdc646c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.466+0000 7fbcb8913640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc624170 0x559cdc63e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.466+0000 7fbcb8913640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc624170 0x559cdc63e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56892/0 (socket says 172.21.15.49:56892) 2024-03-18T00:54:58.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.466+0000 7fbcb8913640 1 -- 172.21.15.49:0/2509773501 learned_addr learned my addr 172.21.15.49:0/2509773501 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:58.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.466+0000 7fbcb9114640 1 --2- 172.21.15.49:0/2509773501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc629cd0 0x559cdc63e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.466+0000 7fbcbd839740 1 -- 172.21.15.49:0/2509773501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9400ed20 con 0x559cdc624170 2024-03-18T00:54:58.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.467+0000 7fbcb8913640 1 -- 172.21.15.49:0/2509773501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624b60 msgr2=0x559cdc646c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:58.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.467+0000 7fbcb8913640 1 --2- 172.21.15.49:0/2509773501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624b60 0x559cdc646c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.467+0000 7fbcb8913640 1 -- 172.21.15.49:0/2509773501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc629cd0 msgr2=0x559cdc63e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:58.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.467+0000 7fbcb8913640 1 --2- 172.21.15.49:0/2509773501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc629cd0 0x559cdc63e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.467+0000 7fbcb8913640 1 -- 172.21.15.49:0/2509773501 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559cdc25d0f0 con 0x559cdc624170 2024-03-18T00:54:58.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.468+0000 7fbcb8913640 1 --2- 172.21.15.49:0/2509773501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc624170 0x559cdc63e240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbcac008e20 tx=0x7fbcac018ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b35fab95ce7b346 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.476+0000 7fbcaa7fc640 1 -- 172.21.15.49:0/2509773501 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac0049c0 con 0x559cdc624170 2024-03-18T00:54:58.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.476+0000 7fbcaa7fc640 1 -- 172.21.15.49:0/2509773501 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcac027650 con 0x559cdc624170 2024-03-18T00:54:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.478+0000 7fbcbd839740 1 -- 172.21.15.49:0/2509773501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc624170 msgr2=0x559cdc63e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.478+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2509773501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc624170 0x559cdc63e240 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbcac008e20 tx=0x7fbcac018ce0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.487+0000 7fbcbd839740 1 -- 172.21.15.49:0/2509773501 shutdown_connections 2024-03-18T00:54:58.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.487+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2509773501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624b60 0x559cdc646c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.488+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2509773501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc629cd0 0x559cdc63e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.488+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2509773501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc624170 0x559cdc63e240 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.488+0000 7fbcbd839740 1 -- 172.21.15.49:0/2509773501 >> 172.21.15.49:0/2509773501 conn(0x559cdc61f6e0 msgr2=0x559cdc61fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:58.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.490+0000 7fbcbd839740 1 -- 172.21.15.49:0/2509773501 shutdown_connections 2024-03-18T00:54:58.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.490+0000 7fbcbd839740 1 -- 172.21.15.49:0/2509773501 wait complete. 2024-03-18T00:54:58.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.491+0000 7fbcbd839740 1 -- start start 2024-03-18T00:54:58.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.492+0000 7fbcbd839740 1 Processor -- start 2024-03-18T00:54:58.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.493+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc643d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.493+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc644270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.493+0000 7fbcbd839740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc645230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.494+0000 7fbcb3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc644270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.494+0000 7fbcb3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc644270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44626/0 (socket says 172.21.15.49:44626) 2024-03-18T00:54:58.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.494+0000 7fbcb3fff640 1 -- 172.21.15.49:0/2895776265 learned_addr learned my addr 172.21.15.49:0/2895776265 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:58.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.494+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9400fac0 con 0x559cdc624b60 2024-03-18T00:54:58.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.495+0000 7fbcb8913640 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc643d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.495+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9400f3f0 con 0x559cdc624170 2024-03-18T00:54:58.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.495+0000 7fbcb9114640 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc645230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.495+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9400c630 con 0x559cdc629cd0 2024-03-18T00:54:58.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.496+0000 7fbcbd839740 10 client.0 _ll_register_callbacks cb 0x7fbc9c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:54:58.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.497+0000 7fbcb8913640 1 -- 172.21.15.49:0/2895776265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 msgr2=0x559cdc645230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:58.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.497+0000 7fbcb8913640 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc645230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.497+0000 7fbcb8913640 1 -- 172.21.15.49:0/2895776265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 msgr2=0x559cdc644270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:54:58.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.497+0000 7fbcb8913640 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc644270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.497+0000 7fbcb8913640 1 -- 172.21.15.49:0/2895776265 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559cdc5509d0 con 0x559cdc624170 2024-03-18T00:54:58.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.501+0000 7fbcb8913640 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc643d30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x559cdc647f80 tx=0x7fbcac027800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d15da26cef3221c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:58.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.502+0000 7fbc93fff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:54:58.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.502+0000 7fbc93fff640 1 -- 172.21.15.49:0/2895776265 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcac018b10 con 0x559cdc624170 2024-03-18T00:54:58.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.502+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcac027650 con 0x559cdc624170 2024-03-18T00:54:58.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.502+0000 7fbc93fff640 1 -- 172.21.15.49:0/2895776265 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbcac0049c0 con 0x559cdc624170 2024-03-18T00:54:58.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.504+0000 7fbc93fff640 1 -- 172.21.15.49:0/2895776265 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbcac004c00 con 0x559cdc624170 2024-03-18T00:54:58.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.507+0000 7fbcbd839740 10 client.25988 fetch_fsmap learned FSMap version 27 2024-03-18T00:54:58.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.507+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559cdc646960 con 0x559cdc624170 2024-03-18T00:54:58.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.510+0000 7fbc93fff640 1 -- 172.21.15.49:0/2895776265 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcac027650 con 0x559cdc624170 2024-03-18T00:54:58.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.510+0000 7fbc93fff640 1 -- 172.21.15.49:0/2895776265 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fbcac005000 con 0x559cdc624170 2024-03-18T00:54:58.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.510+0000 7fbcbd839740 10 client.25988 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:54:58.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.510+0000 7fbcbd839740 10 client.25988 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:54:58.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.510+0000 7fbcbd839740 20 client.25988 populate_metadata read hostname 'smithi049' 2024-03-18T00:54:58.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.510+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559cdc6b1920 0x559cdc6b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.510+0000 7fbcbd839740 4 client.25988 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:54:58.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.510+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559cdc590d90 con 0x559cdc6b1920 2024-03-18T00:54:58.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.520+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559cdc6b1920 0x559cdc6b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.537+0000 7fbcb3fff640 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559cdc6b1920 0x559cdc6b3d10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbd03a1457ec1544 server_cookie=493699259a27c709 in_seq=0 out_seq=0 2024-03-18T00:54:58.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.537+0000 7fbc93fff640 10 client.25988 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:58.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.543+0000 7fbc93fff640 1 -- 172.21.15.49:0/2895776265 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559cdc590d90 con 0x559cdc6b1920 2024-03-18T00:54:58.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.543+0000 7fbc93fff640 10 client.25988 handle_command_reply: tid=0 2024-03-18T00:54:58.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.545+0000 7fbcbd839740 1 client.25988 shutdown 2024-03-18T00:54:58.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.552+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 msgr2=0x559cdc643d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.552+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc643d30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x559cdc647f80 tx=0x7fbcac027800 comp rx=0 tx=0).stop 2024-03-18T00:54:58.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.552+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 shutdown_connections 2024-03-18T00:54:58.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.552+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559cdc6b1920 0x559cdc6b3d10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.552+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559cdc629cd0 0x559cdc645230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.552+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559cdc624b60 0x559cdc644270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.552+0000 7fbcbd839740 1 --2- 172.21.15.49:0/2895776265 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559cdc624170 0x559cdc643d30 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x559cdc647f80 tx=0x7fbcac027800 comp rx=0 tx=0).stop 2024-03-18T00:54:58.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.552+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 >> 172.21.15.49:0/2895776265 conn(0x559cdc61f6e0 msgr2=0x559cdc622940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:58.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.552+0000 7fbc93fff640 0 client.25988 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:54:58.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.560+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 shutdown_connections 2024-03-18T00:54:58.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.567+0000 7fbcbd839740 1 -- 172.21.15.49:0/2895776265 wait complete. 2024-03-18T00:54:58.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.568+0000 7fbcbd839740 20 client.25988 trim_cache size 0 max 16384 2024-03-18T00:54:58.568 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:54:58.568 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:54:58.569 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:54:58.569 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:54:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.577+0000 7fbcbab77640 1 -- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbca00a2950 msgr2=0x7fbca00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.577+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbca00a2950 0x7fbca00a4e10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c005fd0 tx=0x7fbc9c009470 comp rx=0 tx=0).stop 2024-03-18T00:54:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.577+0000 7fbcbab77640 1 -- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 msgr2=0x7fbcb41ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.577+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 0x7fbcb41ade10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbca402c7c0 tx=0x7fbca4005dd0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.583+0000 7fbcbab77640 1 -- 172.21.15.49:0/1668605675 shutdown_connections 2024-03-18T00:54:58.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.583+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fbca00a2950 0x7fbca00a4e10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.583+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fbcb4109140 0x7fbcb41a6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.587 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.583+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fbcb4110060 0x7fbcb41ade10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.584+0000 7fbcbab77640 1 --2- 172.21.15.49:0/1668605675 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbcb41087e0 0x7fbcb41a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.584+0000 7fbcbab77640 1 -- 172.21.15.49:0/1668605675 >> 172.21.15.49:0/1668605675 conn(0x7fbcb406e640 msgr2=0x7fbcb4070310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:58.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.585+0000 7fbcbab77640 1 -- 172.21.15.49:0/1668605675 shutdown_connections 2024-03-18T00:54:58.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.588+0000 7fbcbab77640 1 -- 172.21.15.49:0/1668605675 wait complete. 2024-03-18T00:54:58.654 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:54:58.654 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:54:58.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.887+0000 7fafeded3640 1 -- 172.21.15.49:0/243116982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810adc0 msgr2=0x7fafe810d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.887+0000 7fafeded3640 1 --2- 172.21.15.49:0/243116982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810adc0 0x7fafe810d250 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009ca0 tx=0x7fafe002c140 comp rx=0 tx=0).stop 2024-03-18T00:54:58.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.889+0000 7fafeded3640 1 -- 172.21.15.49:0/243116982 shutdown_connections 2024-03-18T00:54:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.889+0000 7fafeded3640 1 --2- 172.21.15.49:0/243116982 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe810d8c0 0x7fafe8118720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.889+0000 7fafeded3640 1 --2- 172.21.15.49:0/243116982 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810adc0 0x7fafe810d250 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.889+0000 7fafeded3640 1 --2- 172.21.15.49:0/243116982 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafe8073600 0x7fafe810a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.889+0000 7fafeded3640 1 -- 172.21.15.49:0/243116982 >> 172.21.15.49:0/243116982 conn(0x7fafe806e530 msgr2=0x7fafe8070970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.891+0000 7fafeded3640 1 -- 172.21.15.49:0/243116982 shutdown_connections 2024-03-18T00:54:58.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.892+0000 7fafeded3640 1 -- 172.21.15.49:0/243116982 wait complete. 2024-03-18T00:54:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.893+0000 7fafeded3640 1 Processor -- start 2024-03-18T00:54:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.893+0000 7fafeded3640 1 -- start start 2024-03-18T00:54:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.894+0000 7fafeded3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe8073600 0x7fafe8115400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.894+0000 7fafeded3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafe810adc0 0x7fafe8115940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.894+0000 7fafeded3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810d8c0 0x7fafe81b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.894+0000 7fafeded3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fafe811c120 con 0x7fafe810adc0 2024-03-18T00:54:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.894+0000 7fafeded3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fafe811bfa0 con 0x7fafe8073600 2024-03-18T00:54:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.894+0000 7fafeded3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fafe811c2a0 con 0x7fafe810d8c0 2024-03-18T00:54:58.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.897+0000 7fafe7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810d8c0 0x7fafe81b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.897+0000 7fafe7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810d8c0 0x7fafe81b6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56908/0 (socket says 172.21.15.49:56908) 2024-03-18T00:54:58.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.897+0000 7fafe7fff640 1 -- 172.21.15.49:0/3634240065 learned_addr learned my addr 172.21.15.49:0/3634240065 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:54:58.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.897+0000 7fafe6ffd640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafe810adc0 0x7fafe8115940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.898+0000 7fafe77fe640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe8073600 0x7fafe8115400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.898+0000 7fafe7fff640 1 -- 172.21.15.49:0/3634240065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe8073600 msgr2=0x7fafe8115400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.899+0000 7fafe7fff640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe8073600 0x7fafe8115400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.899+0000 7fafe7fff640 1 -- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafe810adc0 msgr2=0x7fafe8115940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:58.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.899+0000 7fafe7fff640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafe810adc0 0x7fafe8115940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:58.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.899+0000 7fafe7fff640 1 -- 172.21.15.49:0/3634240065 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe81b6d50 con 0x7fafe810d8c0 2024-03-18T00:54:58.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.899+0000 7fafe7fff640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810d8c0 0x7fafe81b6810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fafd800e970 tx=0x7fafd800ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:58.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.904+0000 7fafe4ff9640 1 -- 172.21.15.49:0/3634240065 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd800ccc0 con 0x7fafe810d8c0 2024-03-18T00:54:58.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.904+0000 7fafe4ff9640 1 -- 172.21.15.49:0/3634240065 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd800ce60 con 0x7fafe810d8c0 2024-03-18T00:54:58.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.904+0000 7fafe4ff9640 1 -- 172.21.15.49:0/3634240065 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd8010490 con 0x7fafe810d8c0 2024-03-18T00:54:58.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.904+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe81b6f20 con 0x7fafe810d8c0 2024-03-18T00:54:58.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.904+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe81b7330 con 0x7fafe810d8c0 2024-03-18T00:54:58.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.910+0000 7fafe4ff9640 1 -- 172.21.15.49:0/3634240065 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fafd8005020 con 0x7fafe810d8c0 2024-03-18T00:54:58.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.910+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd40052c0 con 0x7fafe810d8c0 2024-03-18T00:54:58.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.914+0000 7fafe4ff9640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd00a2a00 0x7fafd00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:54:58.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.916+0000 7fafe4ff9640 1 -- 172.21.15.49:0/3634240065 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fafd80cee80 con 0x7fafe810d8c0 2024-03-18T00:54:58.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.917+0000 7fafe77fe640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd00a2a00 0x7fafd00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:54:58.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.918+0000 7fafe77fe640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd00a2a00 0x7fafd00a4ec0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fafe80739e0 tx=0x7fafdc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:54:58.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:58.924+0000 7fafe4ff9640 1 -- 172.21.15.49:0/3634240065 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fafd809bea0 con 0x7fafe810d8c0 2024-03-18T00:54:59.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:58 smithi106 ceph-mon[35899]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 864 MiB/s wr, 1.46k op/s 2024-03-18T00:54:59.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:58 smithi049 ceph-mon[32130]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 864 MiB/s wr, 1.46k op/s 2024-03-18T00:54:59.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:58 smithi107 ceph-mon[36270]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 864 MiB/s wr, 1.46k op/s 2024-03-18T00:54:59.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.735+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafd40055b0 con 0x7fafe810d8c0 2024-03-18T00:54:59.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.738+0000 7fafe4ff9640 1 -- 172.21.15.49:0/3634240065 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7fafd809b7a0 con 0x7fafe810d8c0 2024-03-18T00:54:59.738 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:54:59.739 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:53:25.016081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24470,24463,24481,24506,24499]},"id":1}]} 2024-03-18T00:54:59.739 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 27 2024-03-18T00:54:59.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.741+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd00a2a00 msgr2=0x7fafd00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:59.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.741+0000 7fafeded3640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd00a2a00 0x7fafd00a4ec0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fafe80739e0 tx=0x7fafdc005f90 comp rx=0 tx=0).stop 2024-03-18T00:54:59.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.741+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810d8c0 msgr2=0x7fafe81b6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:54:59.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.741+0000 7fafeded3640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810d8c0 0x7fafe81b6810 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fafd800e970 tx=0x7fafd800ed70 comp rx=0 tx=0).stop 2024-03-18T00:54:59.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.742+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 shutdown_connections 2024-03-18T00:54:59.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.742+0000 7fafeded3640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fafd00a2a00 0x7fafd00a4ec0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:59.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.742+0000 7fafeded3640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fafe810d8c0 0x7fafe81b6810 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:59.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.742+0000 7fafeded3640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fafe810adc0 0x7fafe8115940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:59.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.742+0000 7fafeded3640 1 --2- 172.21.15.49:0/3634240065 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fafe8073600 0x7fafe8115400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:54:59.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.742+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 >> 172.21.15.49:0/3634240065 conn(0x7fafe806e530 msgr2=0x7fafe806e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:54:59.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.742+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 shutdown_connections 2024-03-18T00:54:59.743 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:54:59.742+0000 7fafeded3640 1 -- 172.21.15.49:0/3634240065 wait complete. 2024-03-18T00:54:59.774 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:55:00.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:54:59 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3634240065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:55:00.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.199+0000 7f01a5cb1640 1 -- 172.21.15.49:0/147243178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01a0108eb0 msgr2=0x7f01a0109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:00.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.199+0000 7f01a5cb1640 1 --2- 172.21.15.49:0/147243178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01a0108eb0 0x7f01a0109330 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f018c009900 tx=0x7f018c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:55:00.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.208+0000 7f01a5cb1640 1 -- 172.21.15.49:0/147243178 shutdown_connections 2024-03-18T00:55:00.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.208+0000 7f01a5cb1640 1 --2- 172.21.15.49:0/147243178 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a0109870 0x7f01a010e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.208+0000 7f01a5cb1640 1 --2- 172.21.15.49:0/147243178 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01a0108eb0 0x7f01a0109330 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.208+0000 7f01a5cb1640 1 --2- 172.21.15.49:0/147243178 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01a0072bf0 0x7f01a0073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.208+0000 7f01a5cb1640 1 -- 172.21.15.49:0/147243178 >> 172.21.15.49:0/147243178 conn(0x7f01a006e6c0 msgr2=0x7f01a0070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:00.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.220+0000 7f01a5cb1640 1 -- 172.21.15.49:0/147243178 shutdown_connections 2024-03-18T00:55:00.222 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.222+0000 7f01a5cb1640 1 -- 172.21.15.49:0/147243178 wait complete. 2024-03-18T00:55:00.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.223+0000 7f01a5cb1640 1 Processor -- start 2024-03-18T00:55:00.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.223+0000 7f01a5cb1640 1 -- start start 2024-03-18T00:55:00.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.223+0000 7f01a5cb1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01a0072bf0 0x7f01a0118340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.223+0000 7f01a5cb1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01a0108eb0 0x7f01a0118880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.223+0000 7f01a5cb1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a0109870 0x7f01a0119750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.224+0000 7f01a5cb1640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f01a0110960 con 0x7f01a0108eb0 2024-03-18T00:55:00.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.224+0000 7f01a5cb1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f01a01107e0 con 0x7f01a0072bf0 2024-03-18T00:55:00.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.224+0000 7f01a5cb1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f01a0110ae0 con 0x7f01a0109870 2024-03-18T00:55:00.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.228+0000 7f01a54b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a0109870 0x7f01a0119750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.230+0000 7f01a54b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a0109870 0x7f01a0119750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56922/0 (socket says 172.21.15.49:56922) 2024-03-18T00:55:00.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.230+0000 7f01a54b0640 1 -- 172.21.15.49:0/2524762203 learned_addr learned my addr 172.21.15.49:0/2524762203 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:00.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.230+0000 7f01a4caf640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01a0072bf0 0x7f01a0118340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.235+0000 7f01a54b0640 1 -- 172.21.15.49:0/2524762203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01a0072bf0 msgr2=0x7f01a0118340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:00.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.235+0000 7f01a54b0640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01a0072bf0 0x7f01a0118340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.235+0000 7f01a54b0640 1 -- 172.21.15.49:0/2524762203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01a0108eb0 msgr2=0x7f01a0118880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:00.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.235+0000 7f01a54b0640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01a0108eb0 0x7f01a0118880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.235+0000 7f01a54b0640 1 -- 172.21.15.49:0/2524762203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a0119e80 con 0x7f01a0109870 2024-03-18T00:55:00.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.235+0000 7f01a4caf640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01a0072bf0 0x7f01a0118340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:55:00.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.240+0000 7f01a54b0640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a0109870 0x7f01a0119750 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f019800b990 tx=0x7f019800be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:00.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.243+0000 7f019dffb640 1 -- 172.21.15.49:0/2524762203 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198019070 con 0x7f01a0109870 2024-03-18T00:55:00.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.243+0000 7f019dffb640 1 -- 172.21.15.49:0/2524762203 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f019800cba0 con 0x7f01a0109870 2024-03-18T00:55:00.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.243+0000 7f019dffb640 1 -- 172.21.15.49:0/2524762203 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198007650 con 0x7f01a0109870 2024-03-18T00:55:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.245+0000 7f01a5cb1640 1 -- 172.21.15.49:0/2524762203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a011a170 con 0x7f01a0109870 2024-03-18T00:55:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.245+0000 7f01a5cb1640 1 -- 172.21.15.49:0/2524762203 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a01b75a0 con 0x7f01a0109870 2024-03-18T00:55:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.250+0000 7f01a9bd0740 1 Processor -- start 2024-03-18T00:55:00.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.256+0000 7f01a9bd0740 1 -- start start 2024-03-18T00:55:00.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.256+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1376e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.256+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 0x5632f137bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.256+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f137ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.256+0000 7f01a9bd0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5632f12db490 con 0x5632f1377430 2024-03-18T00:55:00.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.256+0000 7f01a9bd0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5632f1283c90 con 0x5632f1376a40 2024-03-18T00:55:00.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.256+0000 7f01a9bd0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5632f12d7530 con 0x5632f137c5a0 2024-03-18T00:55:00.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.257+0000 7f019dffb640 1 -- 172.21.15.49:0/2524762203 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0198014440 con 0x7f01a0109870 2024-03-18T00:55:00.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.257+0000 7f019ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1376e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.257+0000 7f019ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1376e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:60294/0 (socket says 172.21.15.49:60294) 2024-03-18T00:55:00.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.258+0000 7f019ffff640 1 -- 172.21.15.49:0/59489308 learned_addr learned my addr 172.21.15.49:0/59489308 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:00.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.258+0000 7f019ffff640 1 --2- 172.21.15.49:0/59489308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f137ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.259+0000 7f01a4caf640 1 --2- 172.21.15.49:0/59489308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 0x5632f137bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.260+0000 7f01a4caf640 1 -- 172.21.15.49:0/59489308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 msgr2=0x5632f137ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.260+0000 7f01a4caf640 1 --2- 172.21.15.49:0/59489308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f137ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.260+0000 7f01a4caf640 1 -- 172.21.15.49:0/59489308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 msgr2=0x5632f1376e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.260+0000 7f01a4caf640 1 --2- 172.21.15.49:0/59489308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1376e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.260+0000 7f01a4caf640 1 -- 172.21.15.49:0/59489308 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f133e500 con 0x5632f1377430 2024-03-18T00:55:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.260+0000 7f019dffb640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f01900a2a00 0x7f01900a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.260+0000 7f01a4caf640 1 --2- 172.21.15.49:0/59489308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 0x5632f137bf30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f01a0073940 tx=0x7f019400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce12d62135082c68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:00.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.261+0000 7f019dffb640 1 -- 172.21.15.49:0/2524762203 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f01980cdff0 con 0x7f01a0109870 2024-03-18T00:55:00.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.265+0000 7f01a5cb1640 1 -- 172.21.15.49:0/59489308 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019400d950 con 0x5632f1377430 2024-03-18T00:55:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.265+0000 7f01a5cb1640 1 -- 172.21.15.49:0/59489308 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0194019070 con 0x5632f1377430 2024-03-18T00:55:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.265+0000 7f01a5cb1640 1 -- 172.21.15.49:0/59489308 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01940123e0 con 0x5632f1377430 2024-03-18T00:55:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.267+0000 7f01a9bd0740 1 -- 172.21.15.49:0/59489308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 msgr2=0x5632f137bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.267+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/59489308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 0x5632f137bf30 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f01a0073940 tx=0x7f019400bc80 comp rx=0 tx=0).stop 2024-03-18T00:55:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.267+0000 7f01a9bd0740 1 -- 172.21.15.49:0/59489308 shutdown_connections 2024-03-18T00:55:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.267+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/59489308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f137ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.267+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/59489308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 0x5632f137bf30 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.267+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/59489308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1376e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.267+0000 7f01a9bd0740 1 -- 172.21.15.49:0/59489308 >> 172.21.15.49:0/59489308 conn(0x5632f13722a0 msgr2=0x5632f13726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:00.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.269+0000 7f019ffff640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f01900a2a00 0x7f01900a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.272+0000 7f01a9bd0740 1 -- 172.21.15.49:0/59489308 shutdown_connections 2024-03-18T00:55:00.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.273+0000 7f01a9bd0740 1 -- 172.21.15.49:0/59489308 wait complete. 2024-03-18T00:55:00.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.274+0000 7f01a9bd0740 1 -- start start 2024-03-18T00:55:00.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.274+0000 7f01a9bd0740 1 Processor -- start 2024-03-18T00:55:00.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.274+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1380fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.275+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1377430 0x5632f1381520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.275+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f1388560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.275+0000 7f01a9bd0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5632f137f550 con 0x5632f1376a40 2024-03-18T00:55:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.275+0000 7f01a9bd0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5632f1297230 con 0x5632f1377430 2024-03-18T00:55:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.275+0000 7f01a9bd0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5632f137f6d0 con 0x5632f137c5a0 2024-03-18T00:55:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.275+0000 7f01a9bd0740 10 client.0 _ll_register_callbacks cb 0x5632f12d3010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:00.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.276+0000 7f01a4caf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1380fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.276+0000 7f01a4caf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1380fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44684/0 (socket says 172.21.15.49:44684) 2024-03-18T00:55:00.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.276+0000 7f01a4caf640 1 -- 172.21.15.49:0/516091461 learned_addr learned my addr 172.21.15.49:0/516091461 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:00.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.278+0000 7f01a4caf640 1 -- 172.21.15.49:0/516091461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 msgr2=0x5632f1388560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:00.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.278+0000 7f01a4caf640 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f1388560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.278+0000 7f01a4caf640 1 -- 172.21.15.49:0/516091461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1377430 msgr2=0x5632f1381520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:00.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.280+0000 7f01a4caf640 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1377430 0x5632f1381520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.280+0000 7f01a4caf640 1 -- 172.21.15.49:0/516091461 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f12a6100 con 0x5632f1376a40 2024-03-18T00:55:00.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.290+0000 7f01a4caf640 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1380fe0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f019400d530 tx=0x7f01940147a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a316d19fd1dfeb15 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:00.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.294+0000 7f017ffff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:00.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.294+0000 7f017ffff640 1 -- 172.21.15.49:0/516091461 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194024070 con 0x5632f1376a40 2024-03-18T00:55:00.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.294+0000 7f01a9bd0740 1 -- 172.21.15.49:0/516091461 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0194019070 con 0x5632f1376a40 2024-03-18T00:55:00.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.295+0000 7f017ffff640 1 -- 172.21.15.49:0/516091461 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f019400dd60 con 0x5632f1376a40 2024-03-18T00:55:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.295+0000 7f019ffff640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f01900a2a00 0x7f01900a4ec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f018c036040 tx=0x7f018c02c8d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:00.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.295+0000 7f017ffff640 1 -- 172.21.15.49:0/516091461 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0194030050 con 0x5632f1376a40 2024-03-18T00:55:00.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.295+0000 7f01a9bd0740 10 client.16554 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:00.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.296+0000 7f01a9bd0740 1 -- 172.21.15.49:0/516091461 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5632f0fef3f0 con 0x5632f1376a40 2024-03-18T00:55:00.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.299+0000 7f017ffff640 1 -- 172.21.15.49:0/516091461 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0194019070 con 0x5632f1376a40 2024-03-18T00:55:00.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.299+0000 7f017ffff640 1 -- 172.21.15.49:0/516091461 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0194013070 con 0x5632f1376a40 2024-03-18T00:55:00.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.299+0000 7f01a9bd0740 10 client.16554 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:00.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.299+0000 7f01a9bd0740 10 client.16554 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:55:00.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.299+0000 7f01a9bd0740 20 client.16554 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:00.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.300+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5632f138d1d0 0x5632f138f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.300+0000 7f01a9bd0740 4 client.16554 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:00.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.300+0000 7f01a9bd0740 1 -- 172.21.15.49:0/516091461 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632f12e89d0 con 0x5632f138d1d0 2024-03-18T00:55:00.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.304+0000 7f019ffff640 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5632f138d1d0 0x5632f138f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:54:59 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3634240065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:55:00.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:54:59 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3634240065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:55:00.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.524+0000 7f019ffff640 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5632f138d1d0 0x5632f138f650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=32b4deeb365ba9e3 server_cookie=cc34bdc5045e0be6 in_seq=0 out_seq=0 2024-03-18T00:55:00.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.524+0000 7f017ffff640 10 client.16554 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:55:00.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.527+0000 7f017ffff640 1 -- 172.21.15.49:0/516091461 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5632f12e89d0 con 0x5632f138d1d0 2024-03-18T00:55:00.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.527+0000 7f017ffff640 10 client.16554 handle_command_reply: tid=0 2024-03-18T00:55:00.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 client.16554 shutdown 2024-03-18T00:55:00.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 -- 172.21.15.49:0/516091461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 msgr2=0x5632f1380fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1380fe0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f019400d530 tx=0x7f01940147a0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 -- 172.21.15.49:0/516091461 shutdown_connections 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5632f138d1d0 0x5632f138f650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1377430 0x5632f1381520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f1388560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/516091461 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1380fe0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f017ffff640 0 client.16554 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 -- 172.21.15.49:0/516091461 >> 172.21.15.49:0/516091461 conn(0x5632f13722a0 msgr2=0x5632f13714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 -- 172.21.15.49:0/516091461 shutdown_connections 2024-03-18T00:55:00.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.528+0000 7f01a9bd0740 1 -- 172.21.15.49:0/516091461 wait complete. 2024-03-18T00:55:00.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.529+0000 7f01a9bd0740 20 client.16554 trim_cache size 0 max 16384 2024-03-18T00:55:00.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.604+0000 7f01a9bd0740 1 Processor -- start 2024-03-18T00:55:00.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.605+0000 7f01a9bd0740 1 -- start start 2024-03-18T00:55:00.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.609+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1370c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.609+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1377430 0x5632f1388520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.610+0000 7f01a4caf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1370c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.610+0000 7f01a4caf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1370c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44696/0 (socket says 172.21.15.49:44696) 2024-03-18T00:55:00.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.610+0000 7f01a4caf640 1 -- 172.21.15.49:0/1854235733 learned_addr learned my addr 172.21.15.49:0/1854235733 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:00.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.610+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f1371170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.610+0000 7f01a9bd0740 1 -- 172.21.15.49:0/1854235733 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f01a01c02d0 con 0x5632f1376a40 2024-03-18T00:55:00.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.610+0000 7f01a9bd0740 1 -- 172.21.15.49:0/1854235733 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f01a01c1e10 con 0x5632f1377430 2024-03-18T00:55:00.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.610+0000 7f01a9bd0740 1 -- 172.21.15.49:0/1854235733 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f01a01c09a0 con 0x5632f137c5a0 2024-03-18T00:55:00.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.611+0000 7f01a54b0640 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f1371170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.613+0000 7f019ffff640 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1377430 0x5632f1388520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.613+0000 7f01a4caf640 1 -- 172.21.15.49:0/1854235733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 msgr2=0x5632f1371170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:00.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.613+0000 7f01a4caf640 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f1371170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.613+0000 7f01a4caf640 1 -- 172.21.15.49:0/1854235733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1377430 msgr2=0x5632f1388520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:00.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.613+0000 7f01a4caf640 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1377430 0x5632f1388520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.613+0000 7f01a4caf640 1 -- 172.21.15.49:0/1854235733 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632f10b97d0 con 0x5632f1376a40 2024-03-18T00:55:00.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.622+0000 7f01a4caf640 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1370c30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0194014460 tx=0x7f0194014490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3bc6b513f81ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:00.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.627+0000 7f018a7fc640 1 -- 172.21.15.49:0/1854235733 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019400a020 con 0x5632f1376a40 2024-03-18T00:55:00.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.627+0000 7f018a7fc640 1 -- 172.21.15.49:0/1854235733 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0194019040 con 0x5632f1376a40 2024-03-18T00:55:00.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.627+0000 7f018a7fc640 1 -- 172.21.15.49:0/1854235733 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194024070 con 0x5632f1376a40 2024-03-18T00:55:00.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.630+0000 7f01a9bd0740 1 -- 172.21.15.49:0/1854235733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 msgr2=0x5632f1370c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:00.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.630+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1370c30 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f0194014460 tx=0x7f0194014490 comp rx=0 tx=0).stop 2024-03-18T00:55:00.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.631+0000 7f01a9bd0740 1 -- 172.21.15.49:0/1854235733 shutdown_connections 2024-03-18T00:55:00.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.631+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1377430 0x5632f1388520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.631+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f1371170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.631+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/1854235733 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1376a40 0x5632f1370c30 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.631+0000 7f01a9bd0740 1 -- 172.21.15.49:0/1854235733 >> 172.21.15.49:0/1854235733 conn(0x5632f13722a0 msgr2=0x5632f137fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:00.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.632+0000 7f01a9bd0740 1 -- 172.21.15.49:0/1854235733 shutdown_connections 2024-03-18T00:55:00.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.632+0000 7f01a9bd0740 1 -- 172.21.15.49:0/1854235733 wait complete. 2024-03-18T00:55:00.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.634+0000 7f01a9bd0740 1 -- start start 2024-03-18T00:55:00.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.634+0000 7f01a9bd0740 1 Processor -- start 2024-03-18T00:55:00.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.635+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1388080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.636+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 0x5632f1384b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.636+0000 7f01a9bd0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f13850b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.636+0000 7f019ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 0x5632f1384b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.636+0000 7f01a9bd0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f01a01c1740 con 0x5632f1377430 2024-03-18T00:55:00.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.636+0000 7f01a9bd0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f01a01c1070 con 0x5632f1376a40 2024-03-18T00:55:00.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.637+0000 7f01a9bd0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f01a01be2b0 con 0x5632f137c5a0 2024-03-18T00:55:00.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.637+0000 7f01a9bd0740 10 client.0 _ll_register_callbacks cb 0x7f019400b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:00.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.638+0000 7f01a54b0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f13850b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.638+0000 7f01a4caf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1388080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.638+0000 7f01a4caf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1388080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:60312/0 (socket says 172.21.15.49:60312) 2024-03-18T00:55:00.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.638+0000 7f01a4caf640 1 -- 172.21.15.49:0/412730550 learned_addr learned my addr 172.21.15.49:0/412730550 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:00.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.639+0000 7f01a4caf640 1 -- 172.21.15.49:0/412730550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 msgr2=0x5632f13850b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:00.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.639+0000 7f01a4caf640 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f13850b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.639+0000 7f01a4caf640 1 -- 172.21.15.49:0/412730550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 msgr2=0x5632f1384b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:00.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.639+0000 7f01a4caf640 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 0x5632f1384b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.639+0000 7f01a4caf640 1 -- 172.21.15.49:0/412730550 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632f13886b0 con 0x5632f1376a40 2024-03-18T00:55:00.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.640+0000 7f01a4caf640 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1388080 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0194014430 tx=0x7f0194006d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c69dfc5641ce4f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:00.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.641+0000 7f017ffff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:55:00.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.641+0000 7f017ffff640 1 -- 172.21.15.49:0/412730550 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019400a030 con 0x5632f1376a40 2024-03-18T00:55:00.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.641+0000 7f01a9bd0740 1 -- 172.21.15.49:0/412730550 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0194019040 con 0x5632f1376a40 2024-03-18T00:55:00.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.645+0000 7f017ffff640 1 -- 172.21.15.49:0/412730550 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f019400db60 con 0x5632f1376a40 2024-03-18T00:55:00.652 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.651+0000 7f017ffff640 1 -- 172.21.15.49:0/412730550 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0194043050 con 0x5632f1376a40 2024-03-18T00:55:00.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.655+0000 7f01a9bd0740 10 client.26003 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:00.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.655+0000 7f01a9bd0740 1 -- 172.21.15.49:0/412730550 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5632f13f2480 con 0x5632f1376a40 2024-03-18T00:55:00.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.661+0000 7f017ffff640 1 -- 172.21.15.49:0/412730550 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0194019040 con 0x5632f1376a40 2024-03-18T00:55:00.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.662+0000 7f017ffff640 1 -- 172.21.15.49:0/412730550 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f019403f4f0 con 0x5632f1376a40 2024-03-18T00:55:00.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.662+0000 7f01a9bd0740 10 client.26003 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:00.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.662+0000 7f01a9bd0740 10 client.26003 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:55:00.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.662+0000 7f01a9bd0740 20 client.26003 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:00.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.662+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5632f13f2d90 0x5632f13f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:00.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.662+0000 7f01a9bd0740 4 client.26003 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:00.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.662+0000 7f01a9bd0740 1 -- 172.21.15.49:0/412730550 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632f12e2d20 con 0x5632f13f2d90 2024-03-18T00:55:00.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.665+0000 7f019ffff640 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5632f13f2d90 0x5632f13f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:00.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.679+0000 7f019ffff640 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5632f13f2d90 0x5632f13f51b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b277c6b4615091ee server_cookie=4deb2db85e61f75b in_seq=0 out_seq=0 2024-03-18T00:55:00.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.679+0000 7f017ffff640 10 client.26003 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:55:00.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.682+0000 7f017ffff640 1 -- 172.21.15.49:0/412730550 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632f12e2d20 con 0x5632f13f2d90 2024-03-18T00:55:00.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.682+0000 7f017ffff640 10 client.26003 handle_command_reply: tid=0 2024-03-18T00:55:00.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.682+0000 7f01a9bd0740 1 client.26003 shutdown 2024-03-18T00:55:00.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.686+0000 7f01a9bd0740 1 -- 172.21.15.49:0/412730550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 msgr2=0x5632f1388080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:00.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.686+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1388080 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0194014430 tx=0x7f0194006d90 comp rx=0 tx=0).stop 2024-03-18T00:55:00.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.688+0000 7f01a9bd0740 1 -- 172.21.15.49:0/412730550 shutdown_connections 2024-03-18T00:55:00.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.688+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x5632f13f2d90 0x5632f13f51b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.688+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5632f137c5a0 0x5632f13850b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.688+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5632f1377430 0x5632f1384b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.688+0000 7f01a9bd0740 1 --2- 172.21.15.49:0/412730550 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5632f1376a40 0x5632f1388080 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.688+0000 7f01a9bd0740 1 -- 172.21.15.49:0/412730550 >> 172.21.15.49:0/412730550 conn(0x5632f13722a0 msgr2=0x5632f1375d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:00.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.689+0000 7f017ffff640 0 client.26003 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:55:00.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.690+0000 7f01a9bd0740 1 -- 172.21.15.49:0/412730550 shutdown_connections 2024-03-18T00:55:00.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.692+0000 7f01a9bd0740 1 -- 172.21.15.49:0/412730550 wait complete. 2024-03-18T00:55:00.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.695+0000 7f01a9bd0740 20 client.26003 trim_cache size 0 max 16384 2024-03-18T00:55:00.696 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:00.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.706+0000 7f018b7fe640 1 -- 172.21.15.49:0/2524762203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f01900a2a00 msgr2=0x7f01900a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:00.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.707+0000 7f018b7fe640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f01900a2a00 0x7f01900a4ec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f018c036040 tx=0x7f018c02c8d0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.707+0000 7f018b7fe640 1 -- 172.21.15.49:0/2524762203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a0109870 msgr2=0x7f01a0119750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:00.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.707+0000 7f018b7fe640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a0109870 0x7f01a0119750 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f019800b990 tx=0x7f019800be50 comp rx=0 tx=0).stop 2024-03-18T00:55:00.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.707+0000 7f018b7fe640 1 -- 172.21.15.49:0/2524762203 shutdown_connections 2024-03-18T00:55:00.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.707+0000 7f018b7fe640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f01900a2a00 0x7f01900a4ec0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.707+0000 7f018b7fe640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f01a0109870 0x7f01a0119750 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.707+0000 7f018b7fe640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f01a0108eb0 0x7f01a0118880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.707+0000 7f018b7fe640 1 --2- 172.21.15.49:0/2524762203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f01a0072bf0 0x7f01a0118340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:00.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.707+0000 7f018b7fe640 1 -- 172.21.15.49:0/2524762203 >> 172.21.15.49:0/2524762203 conn(0x7f01a006e6c0 msgr2=0x7f01a010c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:00.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.711+0000 7f018b7fe640 1 -- 172.21.15.49:0/2524762203 shutdown_connections 2024-03-18T00:55:00.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:00.714+0000 7f018b7fe640 1 -- 172.21.15.49:0/2524762203 wait complete. 2024-03-18T00:55:00.790 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:55:01.100 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.100+0000 7f9192419640 1 -- 172.21.15.49:0/517676750 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f918c108f70 msgr2=0x7f918c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.100+0000 7f9192419640 1 --2- 172.21.15.49:0/517676750 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f918c108f70 0x7f918c1093f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9180009900 tx=0x7f918002bc80 comp rx=0 tx=0).stop 2024-03-18T00:55:01.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.108+0000 7f9192419640 1 -- 172.21.15.49:0/517676750 shutdown_connections 2024-03-18T00:55:01.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.108+0000 7f9192419640 1 --2- 172.21.15.49:0/517676750 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f918c108f70 0x7f918c1093f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.108+0000 7f9192419640 1 --2- 172.21.15.49:0/517676750 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f918c109930 0x7f918c110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.108+0000 7f9192419640 1 --2- 172.21.15.49:0/517676750 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f918c107e20 0x7f918c108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.108+0000 7f9192419640 1 -- 172.21.15.49:0/517676750 >> 172.21.15.49:0/517676750 conn(0x7f918c076db0 msgr2=0x7f918c0791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.121+0000 7f9192419640 1 -- 172.21.15.49:0/517676750 shutdown_connections 2024-03-18T00:55:01.129 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:00 smithi049 ceph-mon[32130]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 952 MiB/s wr, 2.06k op/s 2024-03-18T00:55:01.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.124+0000 7f9192419640 1 -- 172.21.15.49:0/517676750 wait complete. 2024-03-18T00:55:01.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.125+0000 7f9192419640 1 Processor -- start 2024-03-18T00:55:01.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.126+0000 7f9192419640 1 -- start start 2024-03-18T00:55:01.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.126+0000 7f9192419640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f918c107e20 0x7f918c072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.126+0000 7f9192419640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f918c108f70 0x7f918c073250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.127+0000 7f9192419640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f918c109930 0x7f918c071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.127+0000 7f9192419640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f918c06bc50 con 0x7f918c108f70 2024-03-18T00:55:01.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.127+0000 7f9192419640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f918c06bad0 con 0x7f918c107e20 2024-03-18T00:55:01.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.127+0000 7f9192419640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f918c06bdd0 con 0x7f918c109930 2024-03-18T00:55:01.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.129+0000 7f918b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f918c108f70 0x7f918c073250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.129+0000 7f918b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f918c108f70 0x7f918c073250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44722/0 (socket says 172.21.15.49:44722) 2024-03-18T00:55:01.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.129+0000 7f918b7fe640 1 -- 172.21.15.49:0/2073217284 learned_addr learned my addr 172.21.15.49:0/2073217284 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.129+0000 7f91909b6640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f918c109930 0x7f918c071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.130+0000 7f918b7fe640 1 -- 172.21.15.49:0/2073217284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f918c109930 msgr2=0x7f918c071390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.130+0000 7f918b7fe640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f918c109930 0x7f918c071390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.130+0000 7f918b7fe640 1 -- 172.21.15.49:0/2073217284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f918c107e20 msgr2=0x7f918c072d10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:55:01.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.130+0000 7f918b7fe640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f918c107e20 0x7f918c072d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.130+0000 7f918b7fe640 1 -- 172.21.15.49:0/2073217284 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f918c071b60 con 0x7f918c108f70 2024-03-18T00:55:01.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.131+0000 7f918b7fe640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f918c108f70 0x7f918c073250 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9180009850 tx=0x7f918002c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.131+0000 7f91897fa640 1 -- 172.21.15.49:0/2073217284 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180043020 con 0x7f918c108f70 2024-03-18T00:55:01.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.132+0000 7f91897fa640 1 -- 172.21.15.49:0/2073217284 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91800043d0 con 0x7f918c108f70 2024-03-18T00:55:01.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.132+0000 7f9192419640 1 -- 172.21.15.49:0/2073217284 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f918c071df0 con 0x7f918c108f70 2024-03-18T00:55:01.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.132+0000 7f9192419640 1 -- 172.21.15.49:0/2073217284 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f918c080060 con 0x7f918c108f70 2024-03-18T00:55:01.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.132+0000 7f91897fa640 1 -- 172.21.15.49:0/2073217284 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180003d40 con 0x7f918c108f70 2024-03-18T00:55:01.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.138+0000 7f9195009740 1 Processor -- start 2024-03-18T00:55:01.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.143+0000 7f9195009740 1 -- start start 2024-03-18T00:55:01.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.144+0000 7f9195009740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225862850 0x55f225862c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.144+0000 7f9195009740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225863240 0x55f225867d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.144+0000 7f9195009740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f2258683b0 0x55f22586a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.144+0000 7f9195009740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f2257c7490 con 0x55f2258683b0 2024-03-18T00:55:01.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.144+0000 7f9195009740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f22576fc90 con 0x55f225863240 2024-03-18T00:55:01.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.144+0000 7f9195009740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f2257c3530 con 0x55f225862850 2024-03-18T00:55:01.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.148+0000 7f91909b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225862850 0x55f225862c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.148+0000 7f91909b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225862850 0x55f225862c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56968/0 (socket says 172.21.15.49:56968) 2024-03-18T00:55:01.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.148+0000 7f91909b6640 1 -- 172.21.15.49:0/202738183 learned_addr learned my addr 172.21.15.49:0/202738183 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.148+0000 7f918bfff640 1 --2- 172.21.15.49:0/202738183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225863240 0x55f225867d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.156+0000 7f918b7fe640 1 --2- 172.21.15.49:0/202738183 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f2258683b0 0x55f22586a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.157+0000 7f91909b6640 1 -- 172.21.15.49:0/202738183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225863240 msgr2=0x55f225867d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.157+0000 7f91909b6640 1 --2- 172.21.15.49:0/202738183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225863240 0x55f225867d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.157+0000 7f91909b6640 1 -- 172.21.15.49:0/202738183 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f2258683b0 msgr2=0x55f22586a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.157+0000 7f91909b6640 1 --2- 172.21.15.49:0/202738183 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f2258683b0 0x55f22586a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.157+0000 7f91909b6640 1 -- 172.21.15.49:0/202738183 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f22582a500 con 0x55f225862850 2024-03-18T00:55:01.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.162+0000 7f91897fa640 1 -- 172.21.15.49:0/2073217284 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9180046020 con 0x7f918c108f70 2024-03-18T00:55:01.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.164+0000 7f91897fa640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f91740a2a00 0x7f91740a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.164+0000 7f91909b6640 1 --2- 172.21.15.49:0/202738183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225862850 0x55f225862c70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9184009f40 tx=0x7f918400c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=febcdf2cc8d06171 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.164+0000 7f91897fa640 1 -- 172.21.15.49:0/2073217284 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f91800f0e20 con 0x7f918c108f70 2024-03-18T00:55:01.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.165+0000 7f918bfff640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f91740a2a00 0x7f91740a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.167+0000 7f91727fc640 1 -- 172.21.15.49:0/202738183 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918401a070 con 0x55f225862850 2024-03-18T00:55:01.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.168+0000 7f91727fc640 1 -- 172.21.15.49:0/202738183 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f918400aad0 con 0x55f225862850 2024-03-18T00:55:01.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.169+0000 7f9195009740 1 -- 172.21.15.49:0/202738183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225862850 msgr2=0x55f225862c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.169+0000 7f9195009740 1 --2- 172.21.15.49:0/202738183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225862850 0x55f225862c70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9184009f40 tx=0x7f918400c890 comp rx=0 tx=0).stop 2024-03-18T00:55:01.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.169+0000 7f9195009740 1 -- 172.21.15.49:0/202738183 shutdown_connections 2024-03-18T00:55:01.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.169+0000 7f9195009740 1 --2- 172.21.15.49:0/202738183 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225863240 0x55f225867d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.169+0000 7f9195009740 1 --2- 172.21.15.49:0/202738183 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f2258683b0 0x55f22586a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.169+0000 7f9195009740 1 --2- 172.21.15.49:0/202738183 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225862850 0x55f225862c70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.169+0000 7f9195009740 1 -- 172.21.15.49:0/202738183 >> 172.21.15.49:0/202738183 conn(0x55f22585ddc0 msgr2=0x55f22585e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.169+0000 7f9195009740 1 -- 172.21.15.49:0/202738183 shutdown_connections 2024-03-18T00:55:01.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.176+0000 7f9195009740 1 -- 172.21.15.49:0/202738183 wait complete. 2024-03-18T00:55:01.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.177+0000 7f9195009740 1 -- start start 2024-03-18T00:55:01.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.178+0000 7f9195009740 1 Processor -- start 2024-03-18T00:55:01.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.181+0000 7f9195009740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f22586cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.181+0000 7f9195009740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f2258683b0 0x55f22586d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.181+0000 7f9195009740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 0x55f225874680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.181+0000 7f9195009740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f22586b3b0 con 0x55f225863240 2024-03-18T00:55:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.181+0000 7f9195009740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f225783230 con 0x55f225872290 2024-03-18T00:55:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.181+0000 7f9195009740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f22586b530 con 0x55f2258683b0 2024-03-18T00:55:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.181+0000 7f9195009740 10 client.0 _ll_register_callbacks cb 0x55f2257bf010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.182+0000 7f91909b6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 0x55f225874680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.182+0000 7f91909b6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 0x55f225874680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:60362/0 (socket says 172.21.15.49:60362) 2024-03-18T00:55:01.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.182+0000 7f91909b6640 1 -- 172.21.15.49:0/1236392171 learned_addr learned my addr 172.21.15.49:0/1236392171 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.183+0000 7f91909b6640 1 -- 172.21.15.49:0/1236392171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f2258683b0 msgr2=0x55f22586d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.183+0000 7f91909b6640 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f2258683b0 0x55f22586d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.183+0000 7f91909b6640 1 -- 172.21.15.49:0/1236392171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 msgr2=0x55f22586cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.183+0000 7f91909b6640 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f22586cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.183+0000 7f91909b6640 1 -- 172.21.15.49:0/1236392171 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f225792100 con 0x55f225872290 2024-03-18T00:55:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.185+0000 7f91909b6640 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 0x55f225874680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9184009a40 tx=0x7f91840158f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f59725af560532c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.185+0000 7f918bfff640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f91740a2a00 0x7f91740a4ec0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55f22585ed30 tx=0x7f917c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.185+0000 7f916b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:55:01.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.186+0000 7f916b7fe640 1 -- 172.21.15.49:0/1236392171 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9184024020 con 0x55f225872290 2024-03-18T00:55:01.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.186+0000 7f9195009740 1 -- 172.21.15.49:0/1236392171 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f2257d49d0 con 0x55f225872290 2024-03-18T00:55:01.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.186+0000 7f916b7fe640 1 -- 172.21.15.49:0/1236392171 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9184009b50 con 0x55f225872290 2024-03-18T00:55:01.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.189+0000 7f916b7fe640 1 -- 172.21.15.49:0/1236392171 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f918402f050 con 0x55f225872290 2024-03-18T00:55:01.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.190+0000 7f9195009740 10 client.26012 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:01.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.190+0000 7f9195009740 1 -- 172.21.15.49:0/1236392171 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f2254db3f0 con 0x55f225872290 2024-03-18T00:55:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.194+0000 7f916b7fe640 1 -- 172.21.15.49:0/1236392171 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f2257d49d0 con 0x55f225872290 2024-03-18T00:55:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.194+0000 7f916b7fe640 1 -- 172.21.15.49:0/1236392171 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9184004040 con 0x55f225872290 2024-03-18T00:55:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.194+0000 7f9195009740 10 client.26012 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.194+0000 7f9195009740 10 client.26012 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:55:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.194+0000 7f9195009740 20 client.26012 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.194+0000 7f9195009740 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55f2258793b0 0x55f22587b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.194+0000 7f9195009740 4 client.26012 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:01.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.194+0000 7f9195009740 1 -- 172.21.15.49:0/1236392171 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2257ced20 con 0x55f2258793b0 2024-03-18T00:55:01.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.196+0000 7f918bfff640 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55f2258793b0 0x55f22587b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.209+0000 7f918bfff640 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55f2258793b0 0x55f22587b7f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eb4b13ddeee21ba9 server_cookie=70c732918876ce7c in_seq=0 out_seq=0 2024-03-18T00:55:01.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.210+0000 7f916b7fe640 10 client.26012 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:55:01.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.215+0000 7f916b7fe640 1 -- 172.21.15.49:0/1236392171 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f2257ced20 con 0x55f2258793b0 2024-03-18T00:55:01.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.215+0000 7f916b7fe640 10 client.26012 handle_command_reply: tid=0 2024-03-18T00:55:01.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.215+0000 7f9195009740 1 client.26012 shutdown 2024-03-18T00:55:01.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.225+0000 7f9195009740 1 -- 172.21.15.49:0/1236392171 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 msgr2=0x55f225874680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.225+0000 7f9195009740 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 0x55f225874680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9184009a40 tx=0x7f91840158f0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.225+0000 7f9195009740 1 -- 172.21.15.49:0/1236392171 shutdown_connections 2024-03-18T00:55:01.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.225+0000 7f9195009740 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55f2258793b0 0x55f22587b7f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.226+0000 7f9195009740 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 0x55f225874680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.226+0000 7f9195009740 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f2258683b0 0x55f22586d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.226+0000 7f9195009740 1 --2- 172.21.15.49:0/1236392171 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f22586cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.226+0000 7f9195009740 1 -- 172.21.15.49:0/1236392171 >> 172.21.15.49:0/1236392171 conn(0x55f22585ddc0 msgr2=0x55f225862a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.226+0000 7f916b7fe640 0 client.26012 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:55:01.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.230+0000 7f9195009740 1 -- 172.21.15.49:0/1236392171 shutdown_connections 2024-03-18T00:55:01.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.237+0000 7f9195009740 1 -- 172.21.15.49:0/1236392171 wait complete. 2024-03-18T00:55:01.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.238+0000 7f9195009740 20 client.26012 trim_cache size 0 max 16384 2024-03-18T00:55:01.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.296+0000 7f9195009740 1 Processor -- start 2024-03-18T00:55:01.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.296+0000 7f9195009740 1 -- start start 2024-03-18T00:55:01.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.297+0000 7f9195009740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f225867c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.297+0000 7f9195009740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f2258683b0 0x55f22585e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.297+0000 7f9195009740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225872290 0x55f225864600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.297+0000 7f9195009740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f917800e690 con 0x55f225863240 2024-03-18T00:55:01.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.297+0000 7f9195009740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f91780101d0 con 0x55f2258683b0 2024-03-18T00:55:01.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.297+0000 7f9195009740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f917800ed60 con 0x55f225872290 2024-03-18T00:55:01.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.300+0000 7f91909b6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225872290 0x55f225864600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.301+0000 7f918bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f225867c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.301+0000 7f918bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f225867c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44738/0 (socket says 172.21.15.49:44738) 2024-03-18T00:55:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.301+0000 7f918bfff640 1 -- 172.21.15.49:0/2468311501 learned_addr learned my addr 172.21.15.49:0/2468311501 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.301+0000 7f918b7fe640 1 --2- 172.21.15.49:0/2468311501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f2258683b0 0x55f22585e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.307+0000 7f918bfff640 1 -- 172.21.15.49:0/2468311501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225872290 msgr2=0x55f225864600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.307+0000 7f918bfff640 1 --2- 172.21.15.49:0/2468311501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225872290 0x55f225864600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.307+0000 7f918bfff640 1 -- 172.21.15.49:0/2468311501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f2258683b0 msgr2=0x55f22585e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.307+0000 7f918bfff640 1 --2- 172.21.15.49:0/2468311501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f2258683b0 0x55f22585e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.307+0000 7f918bfff640 1 -- 172.21.15.49:0/2468311501 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2255a57d0 con 0x55f225863240 2024-03-18T00:55:01.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.311+0000 7f918bfff640 1 --2- 172.21.15.49:0/2468311501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f225867c30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55f225863640 tx=0x7f917c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6db33f58731696 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.312+0000 7f9171ffb640 1 -- 172.21.15.49:0/2468311501 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c02b070 con 0x55f225863240 2024-03-18T00:55:01.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.312+0000 7f9171ffb640 1 -- 172.21.15.49:0/2468311501 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f917c00fe20 con 0x55f225863240 2024-03-18T00:55:01.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.314+0000 7f9195009740 1 -- 172.21.15.49:0/2468311501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 msgr2=0x55f225867c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.314+0000 7f9195009740 1 --2- 172.21.15.49:0/2468311501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f225867c30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55f225863640 tx=0x7f917c012000 comp rx=0 tx=0).stop 2024-03-18T00:55:01.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.318+0000 7f9195009740 1 -- 172.21.15.49:0/2468311501 shutdown_connections 2024-03-18T00:55:01.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.318+0000 7f9195009740 1 --2- 172.21.15.49:0/2468311501 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f2258683b0 0x55f22585e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.318+0000 7f9195009740 1 --2- 172.21.15.49:0/2468311501 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f225872290 0x55f225864600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.318+0000 7f9195009740 1 --2- 172.21.15.49:0/2468311501 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f225867c30 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.318+0000 7f9195009740 1 -- 172.21.15.49:0/2468311501 >> 172.21.15.49:0/2468311501 conn(0x55f22585ddc0 msgr2=0x55f225862a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.321+0000 7f9195009740 1 -- 172.21.15.49:0/2468311501 shutdown_connections 2024-03-18T00:55:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.323+0000 7f9195009740 1 -- 172.21.15.49:0/2468311501 wait complete. 2024-03-18T00:55:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.324+0000 7f9195009740 1 -- start start 2024-03-18T00:55:01.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.324+0000 7f9195009740 1 Processor -- start 2024-03-18T00:55:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f9195009740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f22588c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f9195009740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f2258683b0 0x55f2258896d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f9195009740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 0x55f225889c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f9195009740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f917800fb00 con 0x55f225863240 2024-03-18T00:55:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f9195009740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f917800f430 con 0x55f225872290 2024-03-18T00:55:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f9195009740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f917800c670 con 0x55f2258683b0 2024-03-18T00:55:01.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f9195009740 10 client.0 _ll_register_callbacks cb 0x7f9184018070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f918bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f22588c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f918bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f22588c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44750/0 (socket says 172.21.15.49:44750) 2024-03-18T00:55:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.325+0000 7f918bfff640 1 -- 172.21.15.49:0/1458468966 learned_addr learned my addr 172.21.15.49:0/1458468966 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.326+0000 7f918b7fe640 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f2258683b0 0x55f2258896d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.326+0000 7f918bfff640 1 -- 172.21.15.49:0/1458468966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f2258683b0 msgr2=0x55f2258896d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.326+0000 7f918bfff640 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f2258683b0 0x55f2258896d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.326+0000 7f918bfff640 1 -- 172.21.15.49:0/1458468966 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 msgr2=0x55f225889c30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:01.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.326+0000 7f918bfff640 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 0x55f225889c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.326+0000 7f918bfff640 1 -- 172.21.15.49:0/1458468966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2258dd0e0 con 0x55f225863240 2024-03-18T00:55:01.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.327+0000 7f918bfff640 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f22588c820 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f917c005910 tx=0x7f917c00ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac67a17b5d999e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.329+0000 7f916b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.329+0000 7f916b7fe640 1 -- 172.21.15.49:0/1458468966 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917c02b070 con 0x55f225863240 2024-03-18T00:55:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.329+0000 7f9195009740 1 -- 172.21.15.49:0/1458468966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f917c00fe20 con 0x55f225863240 2024-03-18T00:55:01.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.329+0000 7f916b7fe640 1 -- 172.21.15.49:0/1458468966 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f917c00b2c0 con 0x55f225863240 2024-03-18T00:55:01.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.331+0000 7f916b7fe640 1 -- 172.21.15.49:0/1458468966 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f917c025250 con 0x55f225863240 2024-03-18T00:55:01.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.332+0000 7f9195009740 10 client.16572 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:01.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.332+0000 7f9195009740 1 -- 172.21.15.49:0/1458468966 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f2258de3b0 con 0x55f225863240 2024-03-18T00:55:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.333+0000 7f916b7fe640 1 -- 172.21.15.49:0/1458468966 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f917c00fe20 con 0x55f225863240 2024-03-18T00:55:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.334+0000 7f916b7fe640 1 -- 172.21.15.49:0/1458468966 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f917c03a8e0 con 0x55f225863240 2024-03-18T00:55:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.334+0000 7f9195009740 10 client.16572 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.334+0000 7f9195009740 10 client.16572 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:55:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.334+0000 7f9195009740 20 client.16572 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.334+0000 7f9195009740 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55f2258dec90 0x55f2258e10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.334+0000 7f9195009740 4 client.16572 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.334+0000 7f9195009740 1 -- 172.21.15.49:0/1458468966 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f22574c6a0 con 0x55f2258dec90 2024-03-18T00:55:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.335+0000 7f918b7fe640 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55f2258dec90 0x55f2258e10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.345+0000 7f918b7fe640 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55f2258dec90 0x55f2258e10b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d5fac5332d69c75e server_cookie=abd88a737b73dd59 in_seq=0 out_seq=0 2024-03-18T00:55:01.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.347+0000 7f916b7fe640 10 client.16572 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:55:01.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.347+0000 7f916b7fe640 1 -- 172.21.15.49:0/1458468966 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f22574c6a0 con 0x55f2258dec90 2024-03-18T00:55:01.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.347+0000 7f916b7fe640 10 client.16572 handle_command_reply: tid=0 2024-03-18T00:55:01.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.347+0000 7f9195009740 1 client.16572 shutdown 2024-03-18T00:55:01.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.351+0000 7f9195009740 1 -- 172.21.15.49:0/1458468966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 msgr2=0x55f22588c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.351+0000 7f9195009740 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f22588c820 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f917c005910 tx=0x7f917c00ba30 comp rx=0 tx=0).stop 2024-03-18T00:55:01.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.355+0000 7f9195009740 1 -- 172.21.15.49:0/1458468966 shutdown_connections 2024-03-18T00:55:01.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.355+0000 7f9195009740 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55f2258dec90 0x55f2258e10b0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.355+0000 7f9195009740 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f225872290 0x55f225889c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.355+0000 7f9195009740 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f2258683b0 0x55f2258896d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.355+0000 7f9195009740 1 --2- 172.21.15.49:0/1458468966 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f225863240 0x55f22588c820 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.355+0000 7f9195009740 1 -- 172.21.15.49:0/1458468966 >> 172.21.15.49:0/1458468966 conn(0x55f22585ddc0 msgr2=0x55f22585e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.356+0000 7f916b7fe640 0 client.16572 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:55:01.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.365+0000 7f9195009740 1 -- 172.21.15.49:0/1458468966 shutdown_connections 2024-03-18T00:55:01.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.370+0000 7f9195009740 1 -- 172.21.15.49:0/1458468966 wait complete. 2024-03-18T00:55:01.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.371+0000 7f9195009740 20 client.16572 trim_cache size 0 max 16384 2024-03-18T00:55:01.378 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 -- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f91740a2a00 msgr2=0x7f91740a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f91740a2a00 0x7f91740a4ec0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55f22585ed30 tx=0x7f917c005850 comp rx=0 tx=0).stop 2024-03-18T00:55:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 -- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f918c108f70 msgr2=0x7f918c073250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f918c108f70 0x7f918c073250 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9180009850 tx=0x7f918002c6b0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 -- 172.21.15.49:0/2073217284 shutdown_connections 2024-03-18T00:55:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f91740a2a00 0x7f91740a4ec0 secure :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55f22585ed30 tx=0x7f917c005850 comp rx=0 tx=0).stop 2024-03-18T00:55:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f918c109930 0x7f918c071390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f918c108f70 0x7f918c073250 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 --2- 172.21.15.49:0/2073217284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f918c107e20 0x7f918c072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.390+0000 7f9192419640 1 -- 172.21.15.49:0/2073217284 >> 172.21.15.49:0/2073217284 conn(0x7f918c076db0 msgr2=0x7f918c10c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.394+0000 7f9192419640 1 -- 172.21.15.49:0/2073217284 shutdown_connections 2024-03-18T00:55:01.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.397+0000 7f9192419640 1 -- 172.21.15.49:0/2073217284 wait complete. 2024-03-18T00:55:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:00 smithi107 ceph-mon[36270]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 952 MiB/s wr, 2.06k op/s 2024-03-18T00:55:01.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:55:01.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:00 smithi106 ceph-mon[35899]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 952 MiB/s wr, 2.06k op/s 2024-03-18T00:55:01.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.873+0000 7fb7b4f92640 1 -- 172.21.15.49:0/3788143032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0078de0 msgr2=0x7fb7b007b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.873+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/3788143032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0078de0 0x7fb7b007b270 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb7a40074e0 tx=0x7fb7a402c8b0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.873+0000 7fb7b4f92640 1 -- 172.21.15.49:0/3788143032 shutdown_connections 2024-03-18T00:55:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.873+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/3788143032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0076430 0x7fb7b00788a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.873+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/3788143032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0078de0 0x7fb7b007b270 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.873+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/3788143032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7b0073550 0x7fb7b0073970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.873+0000 7fb7b4f92640 1 -- 172.21.15.49:0/3788143032 >> 172.21.15.49:0/3788143032 conn(0x7fb7b006e6c0 msgr2=0x7fb7b0070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.874+0000 7fb7b4f92640 1 -- 172.21.15.49:0/3788143032 shutdown_connections 2024-03-18T00:55:01.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.875+0000 7fb7b4f92640 1 -- 172.21.15.49:0/3788143032 wait complete. 2024-03-18T00:55:01.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.875+0000 7fb7b4f92640 1 Processor -- start 2024-03-18T00:55:01.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.876+0000 7fb7b4f92640 1 -- start start 2024-03-18T00:55:01.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.876+0000 7fb7b4f92640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0073550 0x7fb7b012aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.876+0000 7fb7b4f92640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7b0076430 0x7fb7b012af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.876+0000 7fb7b4f92640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0078de0 0x7fb7b01318c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.876+0000 7fb7b4f92640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b0083010 con 0x7fb7b0073550 2024-03-18T00:55:01.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.876+0000 7fb7b4f92640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b0082e90 con 0x7fb7b0078de0 2024-03-18T00:55:01.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.876+0000 7fb7b4f92640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b0083190 con 0x7fb7b0076430 2024-03-18T00:55:01.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.877+0000 7fb7af7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0073550 0x7fb7b012aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.877+0000 7fb7af7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0073550 0x7fb7b012aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44770/0 (socket says 172.21.15.49:44770) 2024-03-18T00:55:01.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.877+0000 7fb7af7fe640 1 -- 172.21.15.49:0/4087888721 learned_addr learned my addr 172.21.15.49:0/4087888721 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.877+0000 7fb7affff640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0078de0 0x7fb7b01318c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.877+0000 7fb7af7fe640 1 -- 172.21.15.49:0/4087888721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7b0076430 msgr2=0x7fb7b012af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.877+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7b0076430 0x7fb7b012af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.877+0000 7fb7af7fe640 1 -- 172.21.15.49:0/4087888721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0078de0 msgr2=0x7fb7b01318c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.877+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0078de0 0x7fb7b01318c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.877+0000 7fb7af7fe640 1 -- 172.21.15.49:0/4087888721 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b0131f30 con 0x7fb7b0073550 2024-03-18T00:55:01.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.878+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0073550 0x7fb7b012aa10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000ba50 tx=0x7fb7a000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.878+0000 7fb7acff9640 1 -- 172.21.15.49:0/4087888721 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0019070 con 0x7fb7b0073550 2024-03-18T00:55:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.879+0000 7fb7acff9640 1 -- 172.21.15.49:0/4087888721 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7a000cbe0 con 0x7fb7b0073550 2024-03-18T00:55:01.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.879+0000 7fb7b4f92640 1 -- 172.21.15.49:0/4087888721 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b0132220 con 0x7fb7b0073550 2024-03-18T00:55:01.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.881+0000 7fb7b4f92640 1 -- 172.21.15.49:0/4087888721 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b01327f0 con 0x7fb7b0073550 2024-03-18T00:55:01.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.881+0000 7fb7acff9640 1 -- 172.21.15.49:0/4087888721 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a00076e0 con 0x7fb7b0073550 2024-03-18T00:55:01.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.884+0000 7fb7acff9640 1 -- 172.21.15.49:0/4087888721 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb7a001c070 con 0x7fb7b0073550 2024-03-18T00:55:01.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.885+0000 7fb7acff9640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7980a2880 0x7fb7980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.885+0000 7fb7acff9640 1 -- 172.21.15.49:0/4087888721 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb7a00d2060 con 0x7fb7b0073550 2024-03-18T00:55:01.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.885+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7980a2880 0x7fb7980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.887+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7980a2880 0x7fb7980a4d40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009710 tx=0x7fb7a8006c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.888+0000 7fb7b8de8740 1 Processor -- start 2024-03-18T00:55:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.889+0000 7fb7b8de8740 1 -- start start 2024-03-18T00:55:01.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.889+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.889+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880d8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.889+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d95a0 0x55b5880dba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.889+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b588038490 con 0x55b5880d4430 2024-03-18T00:55:01.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.889+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b587fe0c90 con 0x55b5880d95a0 2024-03-18T00:55:01.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.889+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b588034530 con 0x55b5880d3a40 2024-03-18T00:55:01.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7aeffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880d8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7aeffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880d8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44786/0 (socket says 172.21.15.49:44786) 2024-03-18T00:55:01.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7aeffd640 1 -- 172.21.15.49:0/3252451305 learned_addr learned my addr 172.21.15.49:0/3252451305 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/3252451305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7aeffd640 1 -- 172.21.15.49:0/3252451305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 msgr2=0x55b5880d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/3252451305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880d3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7aeffd640 1 -- 172.21.15.49:0/3252451305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d95a0 msgr2=0x55b5880dba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/3252451305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d95a0 0x55b5880dba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7aeffd640 1 -- 172.21.15.49:0/3252451305 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b58809b500 con 0x55b5880d4430 2024-03-18T00:55:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.890+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/3252451305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880d3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.891+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/3252451305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880d8f30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8015890 tx=0x7fb7a800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ea700d8d1c0701 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.891+0000 7fb795ffb640 1 -- 172.21.15.49:0/3252451305 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8023070 con 0x55b5880d4430 2024-03-18T00:55:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.891+0000 7fb795ffb640 1 -- 172.21.15.49:0/3252451305 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7a800be60 con 0x55b5880d4430 2024-03-18T00:55:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.891+0000 7fb795ffb640 1 -- 172.21.15.49:0/3252451305 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8013d20 con 0x55b5880d4430 2024-03-18T00:55:01.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.893+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3252451305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 msgr2=0x55b5880d8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.893+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3252451305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880d8f30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8015890 tx=0x7fb7a800e000 comp rx=0 tx=0).stop 2024-03-18T00:55:01.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.894+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3252451305 shutdown_connections 2024-03-18T00:55:01.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.894+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3252451305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d95a0 0x55b5880dba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.894+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3252451305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880d8f30 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.894+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3252451305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880d3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.894+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3252451305 >> 172.21.15.49:0/3252451305 conn(0x55b5880cefb0 msgr2=0x55b5880cf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.896+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3252451305 shutdown_connections 2024-03-18T00:55:01.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.896+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3252451305 wait complete. 2024-03-18T00:55:01.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.897+0000 7fb7b8de8740 1 -- start start 2024-03-18T00:55:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.897+0000 7fb7b8de8740 1 Processor -- start 2024-03-18T00:55:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.898+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d3a40 0x55b5880ddf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.898+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880de4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.898+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d95a0 0x55b5880e5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.898+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b5880dc550 con 0x55b5880d3a40 2024-03-18T00:55:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.898+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b587ff4230 con 0x55b5880d4430 2024-03-18T00:55:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.898+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b5880dc6d0 con 0x55b5880d95a0 2024-03-18T00:55:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.898+0000 7fb7af7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d3a40 0x55b5880ddf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.898+0000 7fb7af7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d3a40 0x55b5880ddf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44796/0 (socket says 172.21.15.49:44796) 2024-03-18T00:55:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.898+0000 7fb7af7fe640 1 -- 172.21.15.49:0/3564193167 learned_addr learned my addr 172.21.15.49:0/3564193167 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.899+0000 7fb7b8de8740 10 client.0 _ll_register_callbacks cb 0x55b588030010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.899+0000 7fb7affff640 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d95a0 0x55b5880e5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.899+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880de4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.899+0000 7fb7af7fe640 1 -- 172.21.15.49:0/3564193167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d95a0 msgr2=0x55b5880e5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.899+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d95a0 0x55b5880e5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.899+0000 7fb7af7fe640 1 -- 172.21.15.49:0/3564193167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 msgr2=0x55b5880de4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.899+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880de4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.899+0000 7fb7af7fe640 1 -- 172.21.15.49:0/3564193167 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b588003100 con 0x55b5880d3a40 2024-03-18T00:55:01.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.899+0000 7fb7affff640 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d95a0 0x55b5880e5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:55:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.900+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d3a40 0x55b5880ddf90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000ba20 tx=0x7fb7a009b020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331ecf490afafdd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.900+0000 7fb78effd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.900+0000 7fb78effd640 1 -- 172.21.15.49:0/3564193167 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a009ea70 con 0x55b5880d3a40 2024-03-18T00:55:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.900+0000 7fb78effd640 1 -- 172.21.15.49:0/3564193167 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb7a00cbb90 con 0x55b5880d3a40 2024-03-18T00:55:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.901+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3564193167 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7a800be60 con 0x55b5880d3a40 2024-03-18T00:55:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.901+0000 7fb78effd640 1 -- 172.21.15.49:0/3564193167 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb7a00b8050 con 0x55b5880d3a40 2024-03-18T00:55:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.901+0000 7fb7b8de8740 10 client.16584 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.901+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3564193167 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b587d4c3f0 con 0x55b5880d3a40 2024-03-18T00:55:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.902+0000 7fb78effd640 1 -- 172.21.15.49:0/3564193167 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7a00d1020 con 0x55b5880d3a40 2024-03-18T00:55:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.902+0000 7fb78effd640 1 -- 172.21.15.49:0/3564193167 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb7a0099d70 con 0x55b5880d3a40 2024-03-18T00:55:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.902+0000 7fb7b8de8740 10 client.16584 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.902+0000 7fb7b8de8740 10 client.16584 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:55:01.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.902+0000 7fb7b8de8740 20 client.16584 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:01.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.902+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55b5880e9c10 0x55b5880ec060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.902+0000 7fb7b8de8740 4 client.16584 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:01.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.902+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3564193167 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fb7a800be60 con 0x55b5880e9c10 2024-03-18T00:55:01.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.903+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55b5880e9c10 0x55b5880ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.904+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55b5880e9c10 0x55b5880ec060 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e0a7416886d51a48 server_cookie=b24437246ff2208 in_seq=0 out_seq=0 2024-03-18T00:55:01.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.904+0000 7fb78effd640 10 client.16584 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:55:01.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.905+0000 7fb78effd640 1 -- 172.21.15.49:0/3564193167 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fb7a800be60 con 0x55b5880e9c10 2024-03-18T00:55:01.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.905+0000 7fb78effd640 10 client.16584 handle_command_reply: tid=0 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.905+0000 7fb7b8de8740 1 client.16584 shutdown 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3564193167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d3a40 msgr2=0x55b5880ddf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d3a40 0x55b5880ddf90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000ba20 tx=0x7fb7a009b020 comp rx=0 tx=0).stop 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3564193167 shutdown_connections 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55b5880e9c10 0x55b5880ec060 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb78effd640 0 client.16584 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880de4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d95a0 0x55b5880e5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/3564193167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d3a40 0x55b5880ddf90 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3564193167 >> 172.21.15.49:0/3564193167 conn(0x55b5880cefb0 msgr2=0x55b5880ce140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3564193167 shutdown_connections 2024-03-18T00:55:01.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 1 -- 172.21.15.49:0/3564193167 wait complete. 2024-03-18T00:55:01.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.906+0000 7fb7b8de8740 20 client.16584 trim_cache size 0 max 16384 2024-03-18T00:55:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.928+0000 7fb7b8de8740 1 Processor -- start 2024-03-18T00:55:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.928+0000 7fb7b8de8740 1 -- start start 2024-03-18T00:55:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.928+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880d5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.928+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880d5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.928+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d95a0 0x55b5880e54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.928+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01d1330 con 0x55b5880d95a0 2024-03-18T00:55:01.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.928+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01d2e70 con 0x55b5880d4430 2024-03-18T00:55:01.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.928+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01d1a00 con 0x55b5880d3a40 2024-03-18T00:55:01.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.929+0000 7fb7aeffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880d5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.929+0000 7fb7aeffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880d5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:60430/0 (socket says 172.21.15.49:60430) 2024-03-18T00:55:01.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.929+0000 7fb7aeffd640 1 -- 172.21.15.49:0/534953217 learned_addr learned my addr 172.21.15.49:0/534953217 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.929+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880d5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.930+0000 7fb7aeffd640 1 -- 172.21.15.49:0/534953217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 msgr2=0x55b5880d5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.930+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880d5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.930+0000 7fb7aeffd640 1 -- 172.21.15.49:0/534953217 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d95a0 msgr2=0x55b5880e54a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:55:01.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.931+0000 7fb7affff640 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d95a0 0x55b5880e54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.932+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d95a0 0x55b5880e54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.932+0000 7fb7aeffd640 1 -- 172.21.15.49:0/534953217 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5880dd070 con 0x55b5880d4430 2024-03-18T00:55:01.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.932+0000 7fb7affff640 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d95a0 0x55b5880e54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:01.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.934+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880d5e70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8015890 tx=0x7fb7a80135b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b227680e29a601fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.936+0000 7fb7957fa640 1 -- 172.21.15.49:0/534953217 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8023040 con 0x55b5880d4430 2024-03-18T00:55:01.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.936+0000 7fb7957fa640 1 -- 172.21.15.49:0/534953217 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7a800be60 con 0x55b5880d4430 2024-03-18T00:55:01.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.936+0000 7fb7957fa640 1 -- 172.21.15.49:0/534953217 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8013d20 con 0x55b5880d4430 2024-03-18T00:55:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.938+0000 7fb7b8de8740 1 -- 172.21.15.49:0/534953217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 msgr2=0x55b5880d5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.938+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880d5e70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8015890 tx=0x7fb7a80135b0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.938+0000 7fb7b8de8740 1 -- 172.21.15.49:0/534953217 shutdown_connections 2024-03-18T00:55:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.938+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d4430 0x55b5880d5e70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.938+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d95a0 0x55b5880e54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.938+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/534953217 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880d5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.938+0000 7fb7b8de8740 1 -- 172.21.15.49:0/534953217 >> 172.21.15.49:0/534953217 conn(0x55b5880cefb0 msgr2=0x55b5880ddf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.938+0000 7fb7b8de8740 1 -- 172.21.15.49:0/534953217 shutdown_connections 2024-03-18T00:55:01.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.938+0000 7fb7b8de8740 1 -- 172.21.15.49:0/534953217 wait complete. 2024-03-18T00:55:01.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.939+0000 7fb7b8de8740 1 -- start start 2024-03-18T00:55:01.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.940+0000 7fb7b8de8740 1 Processor -- start 2024-03-18T00:55:01.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.940+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880e28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.940+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880e2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.941+0000 7fb7b8de8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d95a0 0x55b5880e3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.941+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01d27a0 con 0x55b5880d4430 2024-03-18T00:55:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.941+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01d20d0 con 0x55b5880d95a0 2024-03-18T00:55:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.941+0000 7fb7b8de8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b01cf310 con 0x55b5880d3a40 2024-03-18T00:55:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.941+0000 7fb7b8de8740 10 client.0 _ll_register_callbacks cb 0x7fb7a00cf030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.942+0000 7fb7af7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880e28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.942+0000 7fb7af7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880e28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:57062/0 (socket says 172.21.15.49:57062) 2024-03-18T00:55:01.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.942+0000 7fb7af7fe640 1 -- 172.21.15.49:0/289801210 learned_addr learned my addr 172.21.15.49:0/289801210 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:01.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.942+0000 7fb7affff640 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d95a0 0x55b5880e3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.942+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880e2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.943+0000 7fb7aeffd640 1 -- 172.21.15.49:0/289801210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 msgr2=0x55b5880e28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.943+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880e28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.943+0000 7fb7aeffd640 1 -- 172.21.15.49:0/289801210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d95a0 msgr2=0x55b5880e3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:01.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.943+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d95a0 0x55b5880e3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.943+0000 7fb7aeffd640 1 -- 172.21.15.49:0/289801210 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b587e167d0 con 0x55b5880d4430 2024-03-18T00:55:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.944+0000 7fb7aeffd640 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880e2df0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009480 tx=0x7fb7a8009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2cd262b49d861bc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.944+0000 7fb78effd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.945+0000 7fb78effd640 1 -- 172.21.15.49:0/289801210 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8039070 con 0x55b5880d4430 2024-03-18T00:55:01.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.945+0000 7fb78effd640 1 -- 172.21.15.49:0/289801210 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb7a8013a80 con 0x55b5880d4430 2024-03-18T00:55:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.945+0000 7fb7b8de8740 1 -- 172.21.15.49:0/289801210 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7a800be60 con 0x55b5880d4430 2024-03-18T00:55:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.945+0000 7fb78effd640 1 -- 172.21.15.49:0/289801210 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb7a8043050 con 0x55b5880d4430 2024-03-18T00:55:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.946+0000 7fb7b8de8740 10 client.16587 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:01.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.946+0000 7fb7b8de8740 1 -- 172.21.15.49:0/289801210 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b5880e5490 con 0x55b5880d4430 2024-03-18T00:55:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.950+0000 7fb78effd640 1 -- 172.21.15.49:0/289801210 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7a800be60 con 0x55b5880d4430 2024-03-18T00:55:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.950+0000 7fb78effd640 1 -- 172.21.15.49:0/289801210 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb7a802c460 con 0x55b5880d4430 2024-03-18T00:55:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.950+0000 7fb7b8de8740 10 client.16587 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:01.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.950+0000 7fb7b8de8740 10 client.16587 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:55:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.950+0000 7fb7b8de8740 20 client.16587 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.950+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55b5881500b0 0x55b5881524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.951+0000 7fb7b8de8740 4 client.16587 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:01.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.951+0000 7fb7b8de8740 1 -- 172.21.15.49:0/289801210 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5880459d0 con 0x55b5881500b0 2024-03-18T00:55:01.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.952+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55b5881500b0 0x55b5881524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:01.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.953+0000 7fb7af7fe640 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55b5881500b0 0x55b5881524a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eb846ff2d99578f6 server_cookie=ed7b314dafdaaea2 in_seq=0 out_seq=0 2024-03-18T00:55:01.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.953+0000 7fb78effd640 10 client.16587 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:55:01.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.954+0000 7fb78effd640 1 -- 172.21.15.49:0/289801210 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5880459d0 con 0x55b5881500b0 2024-03-18T00:55:01.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.954+0000 7fb78effd640 10 client.16587 handle_command_reply: tid=0 2024-03-18T00:55:01.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.954+0000 7fb7b8de8740 1 client.16587 shutdown 2024-03-18T00:55:01.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.955+0000 7fb7b8de8740 1 -- 172.21.15.49:0/289801210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 msgr2=0x55b5880e2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.955+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880e2df0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009480 tx=0x7fb7a8009fa0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.955+0000 7fb7b8de8740 1 -- 172.21.15.49:0/289801210 shutdown_connections 2024-03-18T00:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.956+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55b5881500b0 0x55b5881524a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.956+0000 7fb78effd640 0 client.16587 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.956+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b5880d95a0 0x55b5880e3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.956+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b5880d4430 0x55b5880e2df0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.956+0000 7fb7b8de8740 1 --2- 172.21.15.49:0/289801210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b5880d3a40 0x55b5880e28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.956+0000 7fb7b8de8740 1 -- 172.21.15.49:0/289801210 >> 172.21.15.49:0/289801210 conn(0x55b5880cefb0 msgr2=0x55b5880ddeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.956+0000 7fb7b8de8740 1 -- 172.21.15.49:0/289801210 shutdown_connections 2024-03-18T00:55:01.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.956+0000 7fb7b8de8740 1 -- 172.21.15.49:0/289801210 wait complete. 2024-03-18T00:55:01.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.957+0000 7fb7b8de8740 20 client.16587 trim_cache size 0 max 16384 2024-03-18T00:55:01.960 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:01.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.965+0000 7fb7b4f92640 1 -- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7980a2880 msgr2=0x7fb7980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.965+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7980a2880 0x7fb7980a4d40 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb7a8009710 tx=0x7fb7a8006c90 comp rx=0 tx=0).stop 2024-03-18T00:55:01.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.965+0000 7fb7b4f92640 1 -- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0073550 msgr2=0x7fb7b012aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:01.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.965+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0073550 0x7fb7b012aa10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000ba50 tx=0x7fb7a000bf10 comp rx=0 tx=0).stop 2024-03-18T00:55:01.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.969+0000 7fb7b4f92640 1 -- 172.21.15.49:0/4087888721 shutdown_connections 2024-03-18T00:55:01.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.969+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7980a2880 0x7fb7980a4d40 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.969+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb7b0078de0 0x7fb7b01318c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.969+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb7b0076430 0x7fb7b012af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.969+0000 7fb7b4f92640 1 --2- 172.21.15.49:0/4087888721 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb7b0073550 0x7fb7b012aa10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:01.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.969+0000 7fb7b4f92640 1 -- 172.21.15.49:0/4087888721 >> 172.21.15.49:0/4087888721 conn(0x7fb7b006e6c0 msgr2=0x7fb7b0076d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.972+0000 7fb7b4f92640 1 -- 172.21.15.49:0/4087888721 shutdown_connections 2024-03-18T00:55:01.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:01.972+0000 7fb7b4f92640 1 -- 172.21.15.49:0/4087888721 wait complete. 2024-03-18T00:55:02.022 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:55:02.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.409+0000 7f2946bc5640 1 -- 172.21.15.49:0/2297603762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29401017b0 msgr2=0x7f2940101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.409+0000 7f2946bc5640 1 --2- 172.21.15.49:0/2297603762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29401017b0 0x7f2940101c30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f2930009900 tx=0x7f293002bc80 comp rx=0 tx=0).stop 2024-03-18T00:55:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.409+0000 7f2946bc5640 1 -- 172.21.15.49:0/2297603762 shutdown_connections 2024-03-18T00:55:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.410+0000 7f2946bc5640 1 --2- 172.21.15.49:0/2297603762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2940102170 0x7f2940108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.410+0000 7f2946bc5640 1 --2- 172.21.15.49:0/2297603762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29401017b0 0x7f2940101c30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.410+0000 7f2946bc5640 1 --2- 172.21.15.49:0/2297603762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f29401005c0 0x7f29401009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.410+0000 7f2946bc5640 1 -- 172.21.15.49:0/2297603762 >> 172.21.15.49:0/2297603762 conn(0x7f29400fbb50 msgr2=0x7f29400fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.410+0000 7f2946bc5640 1 -- 172.21.15.49:0/2297603762 shutdown_connections 2024-03-18T00:55:02.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.410+0000 7f2946bc5640 1 -- 172.21.15.49:0/2297603762 wait complete. 2024-03-18T00:55:02.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.412+0000 7f2946bc5640 1 Processor -- start 2024-03-18T00:55:02.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.412+0000 7f2946bc5640 1 -- start start 2024-03-18T00:55:02.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.418+0000 7f2946bc5640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29401005c0 0x7f2940196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.418+0000 7f2946bc5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29401017b0 0x7f2940196e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.418+0000 7f2946bc5640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2940102170 0x7f2940068940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.418+0000 7f2946bc5640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f294010c310 con 0x7f29401017b0 2024-03-18T00:55:02.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.418+0000 7f2946bc5640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f294010c190 con 0x7f29401005c0 2024-03-18T00:55:02.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.418+0000 7f2946bc5640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f294010c490 con 0x7f2940102170 2024-03-18T00:55:02.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.419+0000 7f2945bc3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29401005c0 0x7f2940196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.419+0000 7f2945bc3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29401005c0 0x7f2940196920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:60458/0 (socket says 172.21.15.49:60458) 2024-03-18T00:55:02.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.419+0000 7f2945bc3640 1 -- 172.21.15.49:0/610005478 learned_addr learned my addr 172.21.15.49:0/610005478 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:02.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.421+0000 7f29453c2640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29401017b0 0x7f2940196e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.423+0000 7f29453c2640 1 -- 172.21.15.49:0/610005478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2940102170 msgr2=0x7f2940068940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.423+0000 7f29453c2640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2940102170 0x7f2940068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.423+0000 7f29453c2640 1 -- 172.21.15.49:0/610005478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29401005c0 msgr2=0x7f2940196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.423+0000 7f29453c2640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29401005c0 0x7f2940196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.423+0000 7f29453c2640 1 -- 172.21.15.49:0/610005478 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2940068fb0 con 0x7f29401017b0 2024-03-18T00:55:02.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.423+0000 7f2945bc3640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29401005c0 0x7f2940196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:55:02.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.424+0000 7f29453c2640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29401017b0 0x7f2940196e60 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f293002c1c0 tx=0x7f2930005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:02.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.426+0000 7f2936ffd640 1 -- 172.21.15.49:0/610005478 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930041070 con 0x7f29401017b0 2024-03-18T00:55:02.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.426+0000 7f2946bc5640 1 -- 172.21.15.49:0/610005478 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2940069240 con 0x7f29401017b0 2024-03-18T00:55:02.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.427+0000 7f2946bc5640 1 -- 172.21.15.49:0/610005478 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2940069730 con 0x7f29401017b0 2024-03-18T00:55:02.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.427+0000 7f2936ffd640 1 -- 172.21.15.49:0/610005478 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29300054e0 con 0x7f29401017b0 2024-03-18T00:55:02.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.427+0000 7f2936ffd640 1 -- 172.21.15.49:0/610005478 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930034440 con 0x7f29401017b0 2024-03-18T00:55:02.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.438+0000 7f294aa13740 1 Processor -- start 2024-03-18T00:55:02.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.442+0000 7f2936ffd640 1 -- 172.21.15.49:0/610005478 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2930044070 con 0x7f29401017b0 2024-03-18T00:55:02.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.445+0000 7f294aa13740 1 -- start start 2024-03-18T00:55:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.445+0000 7f294aa13740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa32e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.445+0000 7f294aa13740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa33430 0x55f8ffa37f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.445+0000 7f294aa13740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa385a0 0x55f8ffa3aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.445+0000 7f294aa13740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f8ff997490 con 0x55f8ffa32a40 2024-03-18T00:55:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.445+0000 7f294aa13740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f8ff93fc90 con 0x55f8ffa33430 2024-03-18T00:55:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.445+0000 7f294aa13740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f8ff993530 con 0x55f8ffa385a0 2024-03-18T00:55:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.444+0000 7f2936ffd640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f292c0a2950 0x7f292c0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.447+0000 7f2936ffd640 1 -- 172.21.15.49:0/610005478 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f29300f09f0 con 0x7f29401017b0 2024-03-18T00:55:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.448+0000 7f2945bc3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa32e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.448+0000 7f2945bc3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa32e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44838/0 (socket says 172.21.15.49:44838) 2024-03-18T00:55:02.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.448+0000 7f2945bc3640 1 -- 172.21.15.49:0/1752097012 learned_addr learned my addr 172.21.15.49:0/1752097012 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:02.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.450+0000 7f29453c2640 1 --2- 172.21.15.49:0/1752097012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa33430 0x55f8ffa37f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.450+0000 7f29463c4640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f292c0a2950 0x7f292c0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.450+0000 7f29463c4640 1 --2- 172.21.15.49:0/1752097012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa385a0 0x55f8ffa3aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.452+0000 7f29463c4640 1 -- 172.21.15.49:0/1752097012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa33430 msgr2=0x55f8ffa37f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:02.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.452+0000 7f29463c4640 1 --2- 172.21.15.49:0/1752097012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa33430 0x55f8ffa37f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.452+0000 7f29463c4640 1 -- 172.21.15.49:0/1752097012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 msgr2=0x55f8ffa32e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:02.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.452+0000 7f29463c4640 1 --2- 172.21.15.49:0/1752097012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa32e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.455 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.452+0000 7f29463c4640 1 -- 172.21.15.49:0/1752097012 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8ff9fa500 con 0x55f8ffa385a0 2024-03-18T00:55:02.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.454+0000 7f29463c4640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f292c0a2950 0x7f292c0a4e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f293c00bef0 tx=0x7f293c00b560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:02.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.459+0000 7f29463c4640 1 --2- 172.21.15.49:0/1752097012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa385a0 0x55f8ffa3aa30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f293c014c70 tx=0x7f293c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1fc9ce537514cf4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:02.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.463+0000 7f292bfff640 1 -- 172.21.15.49:0/1752097012 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c011cd0 con 0x55f8ffa385a0 2024-03-18T00:55:02.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.463+0000 7f292bfff640 1 -- 172.21.15.49:0/1752097012 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293c023070 con 0x55f8ffa385a0 2024-03-18T00:55:02.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.464+0000 7f294aa13740 1 -- 172.21.15.49:0/1752097012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa385a0 msgr2=0x55f8ffa3aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.464+0000 7f294aa13740 1 --2- 172.21.15.49:0/1752097012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa385a0 0x55f8ffa3aa30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f293c014c70 tx=0x7f293c017000 comp rx=0 tx=0).stop 2024-03-18T00:55:02.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.466+0000 7f294aa13740 1 -- 172.21.15.49:0/1752097012 shutdown_connections 2024-03-18T00:55:02.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.466+0000 7f294aa13740 1 --2- 172.21.15.49:0/1752097012 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa33430 0x55f8ffa37f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.466+0000 7f294aa13740 1 --2- 172.21.15.49:0/1752097012 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa385a0 0x55f8ffa3aa30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.466+0000 7f294aa13740 1 --2- 172.21.15.49:0/1752097012 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa32e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.466+0000 7f294aa13740 1 -- 172.21.15.49:0/1752097012 >> 172.21.15.49:0/1752097012 conn(0x55f8ffa2dfb0 msgr2=0x55f8ffa2e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:02.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.467+0000 7f294aa13740 1 -- 172.21.15.49:0/1752097012 shutdown_connections 2024-03-18T00:55:02.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.468+0000 7f294aa13740 1 -- 172.21.15.49:0/1752097012 wait complete. 2024-03-18T00:55:02.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.468+0000 7f294aa13740 1 -- start start 2024-03-18T00:55:02.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.468+0000 7f294aa13740 1 Processor -- start 2024-03-18T00:55:02.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.472+0000 7f294aa13740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa32a40 0x55f8ffa3ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.472+0000 7f294aa13740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa33430 0x55f8ffa3d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.472+0000 7f294aa13740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa385a0 0x55f8ffa44400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.472+0000 7f294aa13740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f8ffa3b540 con 0x55f8ffa33430 2024-03-18T00:55:02.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.472+0000 7f294aa13740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f8ff953230 con 0x55f8ffa385a0 2024-03-18T00:55:02.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.472+0000 7f294aa13740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f8ffa3b6c0 con 0x55f8ffa32a40 2024-03-18T00:55:02.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.472+0000 7f294aa13740 10 client.0 _ll_register_callbacks cb 0x55f8ff98f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:02.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.473+0000 7f29453c2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa33430 0x55f8ffa3d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.473+0000 7f29453c2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa33430 0x55f8ffa3d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44846/0 (socket says 172.21.15.49:44846) 2024-03-18T00:55:02.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.473+0000 7f29453c2640 1 -- 172.21.15.49:0/4203820477 learned_addr learned my addr 172.21.15.49:0/4203820477 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:02.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.474+0000 7f29453c2640 1 -- 172.21.15.49:0/4203820477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa32a40 msgr2=0x55f8ffa3ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:02.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.474+0000 7f29453c2640 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa32a40 0x55f8ffa3ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.474+0000 7f29453c2640 1 -- 172.21.15.49:0/4203820477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa385a0 msgr2=0x55f8ffa44400 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:02.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.474+0000 7f29453c2640 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa385a0 0x55f8ffa44400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.474+0000 7f29453c2640 1 -- 172.21.15.49:0/4203820477 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8ff962100 con 0x55f8ffa33430 2024-03-18T00:55:02.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.475+0000 7f29453c2640 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa33430 0x55f8ffa3d3c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f29300b97d0 tx=0x7f2930009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c786b0139bb0fc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:02.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.476+0000 7f29297fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:02.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.476+0000 7f29297fa640 1 -- 172.21.15.49:0/4203820477 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293004b050 con 0x55f8ffa33430 2024-03-18T00:55:02.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.476+0000 7f294aa13740 1 -- 172.21.15.49:0/4203820477 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f293c023070 con 0x55f8ffa33430 2024-03-18T00:55:02.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.476+0000 7f29297fa640 1 -- 172.21.15.49:0/4203820477 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f29300dd0b0 con 0x55f8ffa33430 2024-03-18T00:55:02.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.477+0000 7f29297fa640 1 -- 172.21.15.49:0/4203820477 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f29300bda70 con 0x55f8ffa33430 2024-03-18T00:55:02.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.477+0000 7f294aa13740 10 client.16599 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:02.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.477+0000 7f294aa13740 1 -- 172.21.15.49:0/4203820477 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8ff6ab3f0 con 0x55f8ffa33430 2024-03-18T00:55:02.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.478+0000 7f29297fa640 1 -- 172.21.15.49:0/4203820477 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293c023070 con 0x55f8ffa33430 2024-03-18T00:55:02.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.479+0000 7f29297fa640 1 -- 172.21.15.49:0/4203820477 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f29300bd5d0 con 0x55f8ffa33430 2024-03-18T00:55:02.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.479+0000 7f294aa13740 10 client.16599 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:02.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.479+0000 7f294aa13740 10 client.16599 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:55:02.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.479+0000 7f294aa13740 20 client.16599 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:02.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.479+0000 7f294aa13740 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f8ffa48ea0 0x55f8ffa4b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.479+0000 7f294aa13740 4 client.16599 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:02.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.479+0000 7f294aa13740 1 -- 172.21.15.49:0/4203820477 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8ff9a49d0 con 0x55f8ffa48ea0 2024-03-18T00:55:02.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.481+0000 7f2945bc3640 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f8ffa48ea0 0x55f8ffa4b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.483+0000 7f29297fa640 10 client.16599 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:55:02.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.483+0000 7f2945bc3640 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f8ffa48ea0 0x55f8ffa4b340 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d718e837ec0cbf4 server_cookie=69fbf567916a9b60 in_seq=0 out_seq=0 2024-03-18T00:55:02.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.485+0000 7f29297fa640 1 -- 172.21.15.49:0/4203820477 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f8ff9a49d0 con 0x55f8ffa48ea0 2024-03-18T00:55:02.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.485+0000 7f29297fa640 10 client.16599 handle_command_reply: tid=0 2024-03-18T00:55:02.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.485+0000 7f294aa13740 1 client.16599 shutdown 2024-03-18T00:55:02.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.486+0000 7f294aa13740 1 -- 172.21.15.49:0/4203820477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa33430 msgr2=0x55f8ffa3d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.486+0000 7f294aa13740 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa33430 0x55f8ffa3d3c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f29300b97d0 tx=0x7f2930009260 comp rx=0 tx=0).stop 2024-03-18T00:55:02.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.487+0000 7f294aa13740 1 -- 172.21.15.49:0/4203820477 shutdown_connections 2024-03-18T00:55:02.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.487+0000 7f294aa13740 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f8ffa48ea0 0x55f8ffa4b340 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.487+0000 7f294aa13740 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa385a0 0x55f8ffa44400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.487+0000 7f294aa13740 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa33430 0x55f8ffa3d3c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.487+0000 7f294aa13740 1 --2- 172.21.15.49:0/4203820477 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa32a40 0x55f8ffa3ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.487+0000 7f294aa13740 1 -- 172.21.15.49:0/4203820477 >> 172.21.15.49:0/4203820477 conn(0x55f8ffa2dfb0 msgr2=0x55f8ffa2d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:02.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.487+0000 7f29297fa640 0 client.16599 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:55:02.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.487+0000 7f294aa13740 1 -- 172.21.15.49:0/4203820477 shutdown_connections 2024-03-18T00:55:02.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.487+0000 7f294aa13740 1 -- 172.21.15.49:0/4203820477 wait complete. 2024-03-18T00:55:02.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.488+0000 7f294aa13740 20 client.16599 trim_cache size 0 max 16384 2024-03-18T00:55:02.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.524+0000 7f294aa13740 1 Processor -- start 2024-03-18T00:55:02.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.525+0000 7f294aa13740 1 -- start start 2024-03-18T00:55:02.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.525+0000 7f294aa13740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa32a40 0x55f8ffa347f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.526+0000 7f294aa13740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa33430 0x55f8ffa34d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.526+0000 7f294aa13740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa385a0 0x55f8ffa443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.527+0000 7f29453c2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa33430 0x55f8ffa34d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.527+0000 7f294aa13740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f292400e690 con 0x55f8ffa385a0 2024-03-18T00:55:02.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.527+0000 7f294aa13740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f29240101d0 con 0x55f8ffa32a40 2024-03-18T00:55:02.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.527+0000 7f294aa13740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f292400ed60 con 0x55f8ffa33430 2024-03-18T00:55:02.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.527+0000 7f2945bc3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa32a40 0x55f8ffa347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.528+0000 7f29463c4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa385a0 0x55f8ffa443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.528+0000 7f29463c4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa385a0 0x55f8ffa443c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44856/0 (socket says 172.21.15.49:44856) 2024-03-18T00:55:02.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.528+0000 7f29463c4640 1 -- 172.21.15.49:0/1517943022 learned_addr learned my addr 172.21.15.49:0/1517943022 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:02.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.529+0000 7f29463c4640 1 -- 172.21.15.49:0/1517943022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa33430 msgr2=0x55f8ffa34d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:02.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.529+0000 7f29463c4640 1 --2- 172.21.15.49:0/1517943022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa33430 0x55f8ffa34d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.529+0000 7f29463c4640 1 -- 172.21.15.49:0/1517943022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa32a40 msgr2=0x55f8ffa347f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:02.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.529+0000 7f29463c4640 1 --2- 172.21.15.49:0/1517943022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa32a40 0x55f8ffa347f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.529+0000 7f29463c4640 1 -- 172.21.15.49:0/1517943022 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8ff7757d0 con 0x55f8ffa385a0 2024-03-18T00:55:02.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.531+0000 7f29463c4640 1 --2- 172.21.15.49:0/1517943022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa385a0 0x55f8ffa443c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f293c00b860 tx=0x7f293c014360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b67ca6948953f3b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:02.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.532+0000 7f292b7fe640 1 -- 172.21.15.49:0/1517943022 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c012910 con 0x55f8ffa385a0 2024-03-18T00:55:02.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.532+0000 7f292b7fe640 1 -- 172.21.15.49:0/1517943022 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293c02c070 con 0x55f8ffa385a0 2024-03-18T00:55:02.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.532+0000 7f292b7fe640 1 -- 172.21.15.49:0/1517943022 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293c01eb50 con 0x55f8ffa385a0 2024-03-18T00:55:02.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.534+0000 7f294aa13740 1 -- 172.21.15.49:0/1517943022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa385a0 msgr2=0x55f8ffa443c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.534+0000 7f294aa13740 1 --2- 172.21.15.49:0/1517943022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa385a0 0x55f8ffa443c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f293c00b860 tx=0x7f293c014360 comp rx=0 tx=0).stop 2024-03-18T00:55:02.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.534+0000 7f294aa13740 1 -- 172.21.15.49:0/1517943022 shutdown_connections 2024-03-18T00:55:02.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.534+0000 7f294aa13740 1 --2- 172.21.15.49:0/1517943022 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa385a0 0x55f8ffa443c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.534+0000 7f294aa13740 1 --2- 172.21.15.49:0/1517943022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa33430 0x55f8ffa34d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.534+0000 7f294aa13740 1 --2- 172.21.15.49:0/1517943022 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa32a40 0x55f8ffa347f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.535+0000 7f294aa13740 1 -- 172.21.15.49:0/1517943022 >> 172.21.15.49:0/1517943022 conn(0x55f8ffa2dfb0 msgr2=0x55f8ffa3ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:02.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.535+0000 7f294aa13740 1 -- 172.21.15.49:0/1517943022 shutdown_connections 2024-03-18T00:55:02.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.535+0000 7f294aa13740 1 -- 172.21.15.49:0/1517943022 wait complete. 2024-03-18T00:55:02.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.536+0000 7f294aa13740 1 -- start start 2024-03-18T00:55:02.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.536+0000 7f294aa13740 1 Processor -- start 2024-03-18T00:55:02.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.538+0000 7f294aa13740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa43f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.538+0000 7f294aa13740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa33430 0x55f8ffa409f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.538+0000 7f294aa13740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa385a0 0x55f8ffa40f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.538+0000 7f294aa13740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f292400fb00 con 0x55f8ffa32a40 2024-03-18T00:55:02.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.538+0000 7f294aa13740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f292400f430 con 0x55f8ffa385a0 2024-03-18T00:55:02.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.538+0000 7f294aa13740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f292400c670 con 0x55f8ffa33430 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.538+0000 7f294aa13740 10 client.0 _ll_register_callbacks cb 0x7f29300b9610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.539+0000 7f2945bc3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa43f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.539+0000 7f2945bc3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa43f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44862/0 (socket says 172.21.15.49:44862) 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.539+0000 7f2945bc3640 1 -- 172.21.15.49:0/3127610416 learned_addr learned my addr 172.21.15.49:0/3127610416 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.539+0000 7f2945bc3640 1 -- 172.21.15.49:0/3127610416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa33430 msgr2=0x55f8ffa409f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.539+0000 7f2945bc3640 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa33430 0x55f8ffa409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.539+0000 7f2945bc3640 1 -- 172.21.15.49:0/3127610416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa385a0 msgr2=0x55f8ffa40f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.539+0000 7f2945bc3640 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa385a0 0x55f8ffa40f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.539+0000 7f2945bc3640 1 -- 172.21.15.49:0/3127610416 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8ffa44580 con 0x55f8ffa32a40 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.540+0000 7f2945bc3640 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa43f50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x55f8ffa33c90 tx=0x7f293800ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1d4e9473b184a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.540+0000 7f29297fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.540+0000 7f29297fa640 1 -- 172.21.15.49:0/3127610416 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293801ad70 con 0x55f8ffa32a40 2024-03-18T00:55:02.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.541+0000 7f294aa13740 1 -- 172.21.15.49:0/3127610416 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f293c02c070 con 0x55f8ffa32a40 2024-03-18T00:55:02.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.541+0000 7f29297fa640 1 -- 172.21.15.49:0/3127610416 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f293801c0b0 con 0x55f8ffa32a40 2024-03-18T00:55:02.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.542+0000 7f29297fa640 1 -- 172.21.15.49:0/3127610416 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f293801c2d0 con 0x55f8ffa32a40 2024-03-18T00:55:02.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.542+0000 7f294aa13740 10 client.16605 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:02.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.542+0000 7f294aa13740 1 -- 172.21.15.49:0/3127610416 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f8ffaae270 con 0x55f8ffa32a40 2024-03-18T00:55:02.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.542+0000 7f29297fa640 1 -- 172.21.15.49:0/3127610416 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f293c02c070 con 0x55f8ffa32a40 2024-03-18T00:55:02.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.544+0000 7f29297fa640 1 -- 172.21.15.49:0/3127610416 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2938025c80 con 0x55f8ffa32a40 2024-03-18T00:55:02.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.544+0000 7f294aa13740 10 client.16605 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:02.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.544+0000 7f294aa13740 10 client.16605 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:55:02.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.544+0000 7f294aa13740 20 client.16605 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:02.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.544+0000 7f294aa13740 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f8ffaaebd0 0x55f8ffab1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:02.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.544+0000 7f294aa13740 4 client.16605 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:02.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.544+0000 7f294aa13740 1 -- 172.21.15.49:0/3127610416 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f8ff99ed20 con 0x55f8ffaaebd0 2024-03-18T00:55:02.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.545+0000 7f29453c2640 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f8ffaaebd0 0x55f8ffab1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:02.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.552+0000 7f29297fa640 10 client.16605 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:55:02.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.552+0000 7f29453c2640 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f8ffaaebd0 0x55f8ffab1010 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=34b985d8ad1f751c server_cookie=de514c3a5d856cf1 in_seq=0 out_seq=0 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.553+0000 7f29297fa640 1 -- 172.21.15.49:0/3127610416 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f8ff99ed20 con 0x55f8ffaaebd0 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.553+0000 7f29297fa640 10 client.16605 handle_command_reply: tid=0 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.553+0000 7f294aa13740 1 client.16605 shutdown 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.553+0000 7f294aa13740 1 -- 172.21.15.49:0/3127610416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 msgr2=0x55f8ffa43f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.553+0000 7f294aa13740 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa43f50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x55f8ffa33c90 tx=0x7f293800ce40 comp rx=0 tx=0).stop 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f294aa13740 1 -- 172.21.15.49:0/3127610416 shutdown_connections 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f294aa13740 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55f8ffaaebd0 0x55f8ffab1010 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f294aa13740 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f8ffa385a0 0x55f8ffa40f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f294aa13740 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f8ffa33430 0x55f8ffa409f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f29297fa640 0 client.16605 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f294aa13740 1 --2- 172.21.15.49:0/3127610416 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f8ffa32a40 0x55f8ffa43f50 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f294aa13740 1 -- 172.21.15.49:0/3127610416 >> 172.21.15.49:0/3127610416 conn(0x55f8ffa2dfb0 msgr2=0x55f8ffa347f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:02.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f294aa13740 1 -- 172.21.15.49:0/3127610416 shutdown_connections 2024-03-18T00:55:02.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f294aa13740 1 -- 172.21.15.49:0/3127610416 wait complete. 2024-03-18T00:55:02.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.554+0000 7f294aa13740 20 client.16605 trim_cache size 0 max 16384 2024-03-18T00:55:02.555 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:02.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 -- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f292c0a2950 msgr2=0x7f292c0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f292c0a2950 0x7f292c0a4e10 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f293c00bef0 tx=0x7f293c00b560 comp rx=0 tx=0).stop 2024-03-18T00:55:02.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 -- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29401017b0 msgr2=0x7f2940196e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29401017b0 0x7f2940196e60 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f293002c1c0 tx=0x7f2930005d90 comp rx=0 tx=0).stop 2024-03-18T00:55:02.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 -- 172.21.15.49:0/610005478 shutdown_connections 2024-03-18T00:55:02.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f292c0a2950 0x7f292c0a4e10 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2940102170 0x7f2940068940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f29401017b0 0x7f2940196e60 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 --2- 172.21.15.49:0/610005478 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f29401005c0 0x7f2940196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:02.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 -- 172.21.15.49:0/610005478 >> 172.21.15.49:0/610005478 conn(0x7f29400fbb50 msgr2=0x7f2940104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:02.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 -- 172.21.15.49:0/610005478 shutdown_connections 2024-03-18T00:55:02.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.555+0000 7f2946bc5640 1 -- 172.21.15.49:0/610005478 wait complete. 2024-03-18T00:55:02.591 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:55:02.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.995+0000 7f05c1ac6640 1 -- 172.21.15.49:0/3272611250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05bc109930 msgr2=0x7f05bc110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:02.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:02.995+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/3272611250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05bc109930 0x7f05bc110250 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f05b4009be0 tx=0x7f05b402bef0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.001+0000 7f05c1ac6640 1 -- 172.21.15.49:0/3272611250 shutdown_connections 2024-03-18T00:55:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.001+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/3272611250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05bc108f70 0x7f05bc1093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.001+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/3272611250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05bc109930 0x7f05bc110250 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.001+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/3272611250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05bc107e20 0x7f05bc108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.001+0000 7f05c1ac6640 1 -- 172.21.15.49:0/3272611250 >> 172.21.15.49:0/3272611250 conn(0x7f05bc076db0 msgr2=0x7f05bc0791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:03.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.003+0000 7f05c1ac6640 1 -- 172.21.15.49:0/3272611250 shutdown_connections 2024-03-18T00:55:03.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.005+0000 7f05c1ac6640 1 -- 172.21.15.49:0/3272611250 wait complete. 2024-03-18T00:55:03.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.007+0000 7f05c1ac6640 1 Processor -- start 2024-03-18T00:55:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.007+0000 7f05c1ac6640 1 -- start start 2024-03-18T00:55:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.008+0000 7f05c1ac6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05bc107e20 0x7f05bc07fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.008+0000 7f05c1ac6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05bc108f70 0x7f05bc080430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.008+0000 7f05c1ac6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05bc109930 0x7f05bc081420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.008+0000 7f05c1ac6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc07a640 con 0x7f05bc108f70 2024-03-18T00:55:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.008+0000 7f05c1ac6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc07a4c0 con 0x7f05bc109930 2024-03-18T00:55:03.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.008+0000 7f05c1ac6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc07a7c0 con 0x7f05bc107e20 2024-03-18T00:55:03.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.009+0000 7f05bbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05bc109930 0x7f05bc081420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.009+0000 7f05bbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05bc109930 0x7f05bc081420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:60524/0 (socket says 172.21.15.49:60524) 2024-03-18T00:55:03.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.009+0000 7f05bbfff640 1 -- 172.21.15.49:0/4145712580 learned_addr learned my addr 172.21.15.49:0/4145712580 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:03.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.010+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05bc107e20 0x7f05bc07fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.013+0000 7f05baffd640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05bc108f70 0x7f05bc080430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.014+0000 7f05bbfff640 1 -- 172.21.15.49:0/4145712580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05bc107e20 msgr2=0x7f05bc07fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:03.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.014+0000 7f05bbfff640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05bc107e20 0x7f05bc07fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.014+0000 7f05bbfff640 1 -- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05bc108f70 msgr2=0x7f05bc080430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:03.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.014+0000 7f05bbfff640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05bc108f70 0x7f05bc080430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.014+0000 7f05bbfff640 1 -- 172.21.15.49:0/4145712580 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05bc081a90 con 0x7f05bc109930 2024-03-18T00:55:03.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.015+0000 7f05bbfff640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05bc109930 0x7f05bc081420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f05b402cc00 tx=0x7f05b4003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:03.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.016+0000 7f05b8ff9640 1 -- 172.21.15.49:0/4145712580 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b4043020 con 0x7f05bc109930 2024-03-18T00:55:03.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.016+0000 7f05b8ff9640 1 -- 172.21.15.49:0/4145712580 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05b4003d20 con 0x7f05bc109930 2024-03-18T00:55:03.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.016+0000 7f05b8ff9640 1 -- 172.21.15.49:0/4145712580 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b4034a00 con 0x7f05bc109930 2024-03-18T00:55:03.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.017+0000 7f05c1ac6640 1 -- 172.21.15.49:0/4145712580 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05bc081d20 con 0x7f05bc109930 2024-03-18T00:55:03.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.017+0000 7f05c1ac6640 1 -- 172.21.15.49:0/4145712580 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05bc082220 con 0x7f05bc109930 2024-03-18T00:55:03.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.023+0000 7f05c4792740 1 Processor -- start 2024-03-18T00:55:03.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.026+0000 7f05c4792740 1 -- start start 2024-03-18T00:55:03.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.026+0000 7f05c4792740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eea850 0x5579f1eeac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.027+0000 7f05c4792740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eeb240 0x5579f1eefd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.027+0000 7f05c4792740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1ef2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.027+0000 7f05c4792740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5579f1e4f490 con 0x5579f1ef03b0 2024-03-18T00:55:03.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.027+0000 7f05c4792740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5579f1df7c90 con 0x5579f1eea850 2024-03-18T00:55:03.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.027+0000 7f05c4792740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5579f1e4b530 con 0x5579f1eeb240 2024-03-18T00:55:03.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.028+0000 7f05baffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1ef2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.029+0000 7f05baffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1ef2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44894/0 (socket says 172.21.15.49:44894) 2024-03-18T00:55:03.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.029+0000 7f05baffd640 1 -- 172.21.15.49:0/3897698957 learned_addr learned my addr 172.21.15.49:0/3897698957 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:03.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.030+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/3897698957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eeb240 0x5579f1eefd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.030+0000 7f05baffd640 1 --2- 172.21.15.49:0/3897698957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eea850 0x5579f1eeac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.030+0000 7f05baffd640 1 -- 172.21.15.49:0/3897698957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eeb240 msgr2=0x5579f1eefd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.030+0000 7f05baffd640 1 --2- 172.21.15.49:0/3897698957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eeb240 0x5579f1eefd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.030+0000 7f05baffd640 1 -- 172.21.15.49:0/3897698957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eea850 msgr2=0x5579f1eeac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.030+0000 7f05baffd640 1 --2- 172.21.15.49:0/3897698957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eea850 0x5579f1eeac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.030+0000 7f05baffd640 1 -- 172.21.15.49:0/3897698957 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579f1eb2500 con 0x5579f1ef03b0 2024-03-18T00:55:03.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.030+0000 7f05baffd640 1 --2- 172.21.15.49:0/3897698957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1ef2840 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f05ac005fd0 tx=0x7f05ac00c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc198b7c91c66955 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:03.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.032+0000 7f05b8ff9640 1 -- 172.21.15.49:0/4145712580 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f05b4046020 con 0x7f05bc109930 2024-03-18T00:55:03.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.033+0000 7f05b8ff9640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05a40a28c0 0x7f05a40a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.033+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05a40a28c0 0x7f05a40a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.035+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05a40a28c0 0x7f05a40a4d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x5579f1eeb640 tx=0x7f05b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:03.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.035+0000 7f05a1ffb640 1 -- 172.21.15.49:0/3897698957 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac004230 con 0x5579f1ef03b0 2024-03-18T00:55:03.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.035+0000 7f05a1ffb640 1 -- 172.21.15.49:0/3897698957 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05ac0043d0 con 0x5579f1ef03b0 2024-03-18T00:55:03.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.035+0000 7f05a1ffb640 1 -- 172.21.15.49:0/3897698957 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac0035f0 con 0x5579f1ef03b0 2024-03-18T00:55:03.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.035+0000 7f05b8ff9640 1 -- 172.21.15.49:0/4145712580 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f05b40335c0 con 0x7f05bc109930 2024-03-18T00:55:03.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.036+0000 7f05c4792740 1 -- 172.21.15.49:0/3897698957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 msgr2=0x5579f1ef2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:03.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.037+0000 7f05c4792740 1 --2- 172.21.15.49:0/3897698957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1ef2840 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f05ac005fd0 tx=0x7f05ac00c5a0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.037+0000 7f05c4792740 1 -- 172.21.15.49:0/3897698957 shutdown_connections 2024-03-18T00:55:03.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.037+0000 7f05c4792740 1 --2- 172.21.15.49:0/3897698957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1ef2840 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.037+0000 7f05c4792740 1 --2- 172.21.15.49:0/3897698957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eeb240 0x5579f1eefd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.037+0000 7f05c4792740 1 --2- 172.21.15.49:0/3897698957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eea850 0x5579f1eeac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.038+0000 7f05c4792740 1 -- 172.21.15.49:0/3897698957 >> 172.21.15.49:0/3897698957 conn(0x5579f1ee5dc0 msgr2=0x5579f1ee61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:03.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.040+0000 7f05c4792740 1 -- 172.21.15.49:0/3897698957 shutdown_connections 2024-03-18T00:55:03.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.040+0000 7f05c4792740 1 -- 172.21.15.49:0/3897698957 wait complete. 2024-03-18T00:55:03.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.041+0000 7f05c4792740 1 -- start start 2024-03-18T00:55:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.042+0000 7f05c4792740 1 Processor -- start 2024-03-18T00:55:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.043+0000 7f05c4792740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.043+0000 7f05c4792740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.044+0000 7f05c4792740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1efc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.044+0000 7f05c4792740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5579f1ef3360 con 0x5579f1ef03b0 2024-03-18T00:55:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.044+0000 7f05c4792740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5579f1e0b230 con 0x5579f1eeb240 2024-03-18T00:55:03.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.044+0000 7f05c4792740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5579f1ef34e0 con 0x5579f1eea850 2024-03-18T00:55:03.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.044+0000 7f05bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.044+0000 7f05c4792740 10 client.0 _ll_register_callbacks cb 0x5579f1e47010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:03.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.045+0000 7f05baffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.045+0000 7f05bb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:57128/0 (socket says 172.21.15.49:57128) 2024-03-18T00:55:03.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.045+0000 7f05bb7fe640 1 -- 172.21.15.49:0/4201367061 learned_addr learned my addr 172.21.15.49:0/4201367061 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:03.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.045+0000 7f05bbfff640 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1efc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.046+0000 7f05bbfff640 1 -- 172.21.15.49:0/4201367061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 msgr2=0x5579f1ef4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.046+0000 7f05bbfff640 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.046+0000 7f05bbfff640 1 -- 172.21.15.49:0/4201367061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 msgr2=0x5579f1ef5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.046+0000 7f05bbfff640 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.046+0000 7f05bbfff640 1 -- 172.21.15.49:0/4201367061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579f1e1a100 con 0x5579f1ef03b0 2024-03-18T00:55:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.046+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:55:03.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.046+0000 7f05baffd640 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.047+0000 7f05bbfff640 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1efc190 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f05b40b87e0 tx=0x7f05b40b8810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec44cab47fa0ba2a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.047+0000 7f059affd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:03.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.048+0000 7f059affd640 1 -- 172.21.15.49:0/4201367061 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b40ba740 con 0x5579f1ef03b0 2024-03-18T00:55:03.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.048+0000 7f059affd640 1 -- 172.21.15.49:0/4201367061 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f05b40b9de0 con 0x5579f1ef03b0 2024-03-18T00:55:03.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.048+0000 7f05c4792740 1 -- 172.21.15.49:0/4201367061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05ac0043d0 con 0x5579f1ef03b0 2024-03-18T00:55:03.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.049+0000 7f059affd640 1 -- 172.21.15.49:0/4201367061 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f05b40ba740 con 0x5579f1ef03b0 2024-03-18T00:55:03.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.049+0000 7f05c4792740 10 client.16617 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:03.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.049+0000 7f05c4792740 1 -- 172.21.15.49:0/4201367061 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5579f1b633f0 con 0x5579f1ef03b0 2024-03-18T00:55:03.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.050+0000 7f059affd640 1 -- 172.21.15.49:0/4201367061 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05ac0043d0 con 0x5579f1ef03b0 2024-03-18T00:55:03.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.051+0000 7f059affd640 1 -- 172.21.15.49:0/4201367061 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f05b4050060 con 0x5579f1ef03b0 2024-03-18T00:55:03.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.052+0000 7f05c4792740 10 client.16617 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:03.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.052+0000 7f05c4792740 10 client.16617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:03.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.052+0000 7f05c4792740 20 client.16617 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:03.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.052+0000 7f05c4792740 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579f1f008e0 0x5579f1f02d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.052+0000 7f05c4792740 4 client.16617 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:03.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.052+0000 7f05c4792740 1 -- 172.21.15.49:0/4201367061 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579f1e5c9d0 con 0x5579f1f008e0 2024-03-18T00:55:03.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.054+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579f1f008e0 0x5579f1f02d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.058+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579f1f008e0 0x5579f1f02d10 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fc3146021d1b80a server_cookie=40bcde43dc561303 in_seq=0 out_seq=0 2024-03-18T00:55:03.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.058+0000 7f059affd640 10 client.16617 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:03.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.061+0000 7f059affd640 1 -- 172.21.15.49:0/4201367061 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5579f1e5c9d0 con 0x5579f1f008e0 2024-03-18T00:55:03.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.061+0000 7f059affd640 10 client.16617 handle_command_reply: tid=0 2024-03-18T00:55:03.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 client.16617 shutdown 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 -- 172.21.15.49:0/4201367061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 msgr2=0x5579f1efc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1efc190 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f05b40b87e0 tx=0x7f05b40b8810 comp rx=0 tx=0).stop 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 -- 172.21.15.49:0/4201367061 shutdown_connections 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579f1f008e0 0x5579f1f02d10 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f059affd640 0 client.16617 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1ef03b0 0x5579f1efc190 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 --2- 172.21.15.49:0/4201367061 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 -- 172.21.15.49:0/4201367061 >> 172.21.15.49:0/4201367061 conn(0x5579f1ee5dc0 msgr2=0x5579f1ee4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 -- 172.21.15.49:0/4201367061 shutdown_connections 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 1 -- 172.21.15.49:0/4201367061 wait complete. 2024-03-18T00:55:03.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.062+0000 7f05c4792740 20 client.16617 trim_cache size 0 max 16384 2024-03-18T00:55:03.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05c4792740 1 Processor -- start 2024-03-18T00:55:03.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05c4792740 1 -- start start 2024-03-18T00:55:03.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05c4792740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05c4792740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05c4792740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1ef03b0 0x5579f1efc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05c4792740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc1c01e0 con 0x5579f1eeb240 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05c4792740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc1c1d20 con 0x5579f1ef03b0 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05c4792740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc1c08b0 con 0x5579f1eea850 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05baffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05baffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44906/0 (socket says 172.21.15.49:44906) 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.087+0000 7f05baffd640 1 -- 172.21.15.49:0/3263506973 learned_addr learned my addr 172.21.15.49:0/3263506973 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/3263506973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05baffd640 1 -- 172.21.15.49:0/3263506973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 msgr2=0x5579f1ef4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05baffd640 1 --2- 172.21.15.49:0/3263506973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05baffd640 1 -- 172.21.15.49:0/3263506973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1ef03b0 msgr2=0x5579f1efc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05baffd640 1 --2- 172.21.15.49:0/3263506973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1ef03b0 0x5579f1efc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05baffd640 1 -- 172.21.15.49:0/3263506973 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579f1c2d7d0 con 0x5579f1eeb240 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/3263506973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:03.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05baffd640 1 --2- 172.21.15.49:0/3263506973 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00cbe0 tx=0x7f05ac005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda5d753af241099 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:03.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05a17fa640 1 -- 172.21.15.49:0/3263506973 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac004680 con 0x5579f1eeb240 2024-03-18T00:55:03.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05a17fa640 1 -- 172.21.15.49:0/3263506973 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05ac01b950 con 0x5579f1eeb240 2024-03-18T00:55:03.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.088+0000 7f05a17fa640 1 -- 172.21.15.49:0/3263506973 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac0127f0 con 0x5579f1eeb240 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.089+0000 7f05c4792740 1 -- 172.21.15.49:0/3263506973 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eeb240 msgr2=0x5579f1ef5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.089+0000 7f05c4792740 1 --2- 172.21.15.49:0/3263506973 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00cbe0 tx=0x7f05ac005eb0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.089+0000 7f05c4792740 1 -- 172.21.15.49:0/3263506973 shutdown_connections 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.089+0000 7f05c4792740 1 --2- 172.21.15.49:0/3263506973 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1ef03b0 0x5579f1efc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.089+0000 7f05c4792740 1 --2- 172.21.15.49:0/3263506973 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eeb240 0x5579f1ef5150 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.089+0000 7f05c4792740 1 --2- 172.21.15.49:0/3263506973 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1eea850 0x5579f1ef4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.090+0000 7f05c4792740 1 -- 172.21.15.49:0/3263506973 >> 172.21.15.49:0/3263506973 conn(0x5579f1ee5dc0 msgr2=0x5579f1ef4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.090+0000 7f05c4792740 1 -- 172.21.15.49:0/3263506973 shutdown_connections 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.090+0000 7f05c4792740 1 -- 172.21.15.49:0/3263506973 wait complete. 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.090+0000 7f05c4792740 1 -- start start 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.090+0000 7f05c4792740 1 Processor -- start 2024-03-18T00:55:03.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05c4792740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eea850 0x5579f1efc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05c4792740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 0x5579f1ef8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05c4792740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1ef03b0 0x5579f1ef91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05c4792740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc1c1650 con 0x5579f1eea850 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05c4792740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc1c0f80 con 0x5579f1eeb240 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05c4792740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f05bc1be270 con 0x5579f1ef03b0 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05c4792740 10 client.0 _ll_register_callbacks cb 0x7f05b404f4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05bb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eea850 0x5579f1efc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05bb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eea850 0x5579f1efc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44916/0 (socket says 172.21.15.49:44916) 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05bb7fe640 1 -- 172.21.15.49:0/3189706373 learned_addr learned my addr 172.21.15.49:0/3189706373 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05bb7fe640 1 -- 172.21.15.49:0/3189706373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1ef03b0 msgr2=0x5579f1ef91e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1ef03b0 0x5579f1ef91e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05bb7fe640 1 -- 172.21.15.49:0/3189706373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 msgr2=0x5579f1ef8bd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.091+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 0x5579f1ef8bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.092+0000 7f05bb7fe640 1 -- 172.21.15.49:0/3189706373 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579f1f64c80 con 0x5579f1eea850 2024-03-18T00:55:03.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.092+0000 7f05bb7fe640 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eea850 0x5579f1efc130 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009fc0 tx=0x7f05b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07729a4fcc5417d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:03.094 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.092+0000 7f059affd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.092+0000 7f059affd640 1 -- 172.21.15.49:0/3189706373 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b002a2a0 con 0x5579f1eea850 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.093+0000 7f059affd640 1 -- 172.21.15.49:0/3189706373 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f05b002a440 con 0x5579f1eea850 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.093+0000 7f05c4792740 1 -- 172.21.15.49:0/3189706373 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05ac01b950 con 0x5579f1eea850 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.093+0000 7f059affd640 1 -- 172.21.15.49:0/3189706373 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f05b0038070 con 0x5579f1eea850 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.093+0000 7f05c4792740 10 client.16623 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.093+0000 7f05c4792740 1 -- 172.21.15.49:0/3189706373 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5579f1f65fc0 con 0x5579f1eea850 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.094+0000 7f059affd640 1 -- 172.21.15.49:0/3189706373 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05b0026070 con 0x5579f1eea850 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.094+0000 7f059affd640 1 -- 172.21.15.49:0/3189706373 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f05b0004b80 con 0x5579f1eea850 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.094+0000 7f05c4792740 10 client.16623 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:03.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.094+0000 7f05c4792740 10 client.16623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:03.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.094+0000 7f05c4792740 20 client.16623 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:03.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.095+0000 7f05c4792740 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579f1f008e0 0x5579f1f689d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:03.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.095+0000 7f05c4792740 4 client.16623 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:03.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.095+0000 7f05c4792740 1 -- 172.21.15.49:0/3189706373 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f05ac01b950 con 0x5579f1f008e0 2024-03-18T00:55:03.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.102+0000 7f05baffd640 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579f1f008e0 0x5579f1f689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:03.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.107+0000 7f059affd640 10 client.16623 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:03.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.107+0000 7f05baffd640 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579f1f008e0 0x5579f1f689d0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aee687fb863bc1cb server_cookie=ea08270852582281 in_seq=0 out_seq=0 2024-03-18T00:55:03.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.110+0000 7f059affd640 1 -- 172.21.15.49:0/3189706373 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f05ac01b950 con 0x5579f1f008e0 2024-03-18T00:55:03.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.110+0000 7f059affd640 10 client.16623 handle_command_reply: tid=0 2024-03-18T00:55:03.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.110+0000 7f05c4792740 1 client.16623 shutdown 2024-03-18T00:55:03.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.110+0000 7f05c4792740 1 -- 172.21.15.49:0/3189706373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eea850 msgr2=0x5579f1efc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:03.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.110+0000 7f05c4792740 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eea850 0x5579f1efc130 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009fc0 tx=0x7f05b00046c0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f05c4792740 1 -- 172.21.15.49:0/3189706373 shutdown_connections 2024-03-18T00:55:03.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f05c4792740 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579f1f008e0 0x5579f1f689d0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f05c4792740 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579f1eeb240 0x5579f1ef8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f05c4792740 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5579f1ef03b0 0x5579f1ef91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f05c4792740 1 --2- 172.21.15.49:0/3189706373 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579f1eea850 0x5579f1efc130 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f059affd640 0 client.16623 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:03.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f05c4792740 1 -- 172.21.15.49:0/3189706373 >> 172.21.15.49:0/3189706373 conn(0x5579f1ee5dc0 msgr2=0x5579f1eec600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:03.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f05c4792740 1 -- 172.21.15.49:0/3189706373 shutdown_connections 2024-03-18T00:55:03.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f05c4792740 1 -- 172.21.15.49:0/3189706373 wait complete. 2024-03-18T00:55:03.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.111+0000 7f05c4792740 20 client.16623 trim_cache size 0 max 16384 2024-03-18T00:55:03.112 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:03.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 -- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05a40a28c0 msgr2=0x7f05a40a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:03.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05a40a28c0 0x7f05a40a4d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x5579f1eeb640 tx=0x7f05b0009260 comp rx=0 tx=0).stop 2024-03-18T00:55:03.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 -- 172.21.15.49:0/4145712580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05bc109930 msgr2=0x7f05bc081420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:03.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05bc109930 0x7f05bc081420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f05b402cc00 tx=0x7f05b4003970 comp rx=0 tx=0).stop 2024-03-18T00:55:03.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 -- 172.21.15.49:0/4145712580 shutdown_connections 2024-03-18T00:55:03.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05a40a28c0 0x7f05a40a4d80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05bc109930 0x7f05bc081420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05bc108f70 0x7f05bc080430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 --2- 172.21.15.49:0/4145712580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05bc107e20 0x7f05bc07fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:03.114 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.113+0000 7f05c1ac6640 1 -- 172.21.15.49:0/4145712580 >> 172.21.15.49:0/4145712580 conn(0x7f05bc076db0 msgr2=0x7f05bc10c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:03.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.116+0000 7f05c1ac6640 1 -- 172.21.15.49:0/4145712580 shutdown_connections 2024-03-18T00:55:03.116 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:03.116+0000 7f05c1ac6640 1 -- 172.21.15.49:0/4145712580 wait complete. 2024-03-18T00:55:03.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:02 smithi049 ceph-mon[32130]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 917 MiB/s wr, 2.16k op/s 2024-03-18T00:55:03.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:02 smithi107 ceph-mon[36270]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 917 MiB/s wr, 2.16k op/s 2024-03-18T00:55:03.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:02 smithi106 ceph-mon[35899]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 917 MiB/s wr, 2.16k op/s 2024-03-18T00:55:04.205 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:55:04.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:55:04.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:55:04.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.520+0000 7f7801c97640 1 -- 172.21.15.49:0/1615226442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc1022b0 msgr2=0x7f77fc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:04.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.520+0000 7f7801c97640 1 --2- 172.21.15.49:0/1615226442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc1022b0 0x7f77fc108b40 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f77f4009ca0 tx=0x7f77f402c140 comp rx=0 tx=0).stop 2024-03-18T00:55:04.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.520+0000 7f7801c97640 1 -- 172.21.15.49:0/1615226442 shutdown_connections 2024-03-18T00:55:04.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.521+0000 7f7801c97640 1 --2- 172.21.15.49:0/1615226442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc1022b0 0x7f77fc108b40 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.521+0000 7f7801c97640 1 --2- 172.21.15.49:0/1615226442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc1018f0 0x7f77fc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.521+0000 7f7801c97640 1 --2- 172.21.15.49:0/1615226442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc100700 0x7f77fc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.521+0000 7f7801c97640 1 -- 172.21.15.49:0/1615226442 >> 172.21.15.49:0/1615226442 conn(0x7f77fc0fbc90 msgr2=0x7f77fc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:04.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.521+0000 7f7801c97640 1 -- 172.21.15.49:0/1615226442 shutdown_connections 2024-03-18T00:55:04.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.521+0000 7f7801c97640 1 -- 172.21.15.49:0/1615226442 wait complete. 2024-03-18T00:55:04.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.524+0000 7f7801c97640 1 Processor -- start 2024-03-18T00:55:04.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.525+0000 7f7801c97640 1 -- start start 2024-03-18T00:55:04.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.527+0000 7f7801c97640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc100700 0x7f77fc1a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.527+0000 7f7801c97640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc1018f0 0x7f77fc1a2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.527+0000 7f7801c97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc1022b0 0x7f77fc1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.527+0000 7f7801c97640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc10c490 con 0x7f77fc100700 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.527+0000 7f7801c97640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc10c310 con 0x7f77fc1018f0 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.527+0000 7f7801c97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc10c610 con 0x7f77fc1022b0 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.527+0000 7f77fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc100700 0x7f77fc1a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.527+0000 7f77fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc100700 0x7f77fc1a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44944/0 (socket says 172.21.15.49:44944) 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.527+0000 7f77fb7fe640 1 -- 172.21.15.49:0/1795532338 learned_addr learned my addr 172.21.15.49:0/1795532338 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.528+0000 7f77fb7fe640 1 -- 172.21.15.49:0/1795532338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc1022b0 msgr2=0x7f77fc1a5420 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.528+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc1022b0 0x7f77fc1a5420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.528+0000 7f77fb7fe640 1 -- 172.21.15.49:0/1795532338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc1018f0 msgr2=0x7f77fc1a2ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.528+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc1018f0 0x7f77fc1a2ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.528+0000 7f77fb7fe640 1 -- 172.21.15.49:0/1795532338 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77fc1a5ce0 con 0x7f77fc100700 2024-03-18T00:55:04.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.528+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc100700 0x7f77fc1a0980 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00b470 tx=0x7f77ec00b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:04.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.528+0000 7f77f8ff9640 1 -- 172.21.15.49:0/1795532338 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec015020 con 0x7f77fc100700 2024-03-18T00:55:04.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.528+0000 7f77f8ff9640 1 -- 172.21.15.49:0/1795532338 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77ec011d70 con 0x7f77fc100700 2024-03-18T00:55:04.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.528+0000 7f7801c97640 1 -- 172.21.15.49:0/1795532338 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77fc1a5fd0 con 0x7f77fc100700 2024-03-18T00:55:04.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.529+0000 7f7801c97640 1 -- 172.21.15.49:0/1795532338 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77fc1a64a0 con 0x7f77fc100700 2024-03-18T00:55:04.529 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.529+0000 7f77f8ff9640 1 -- 172.21.15.49:0/1795532338 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec00f340 con 0x7f77fc100700 2024-03-18T00:55:04.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.534+0000 7f7804882740 1 Processor -- start 2024-03-18T00:55:04.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.534+0000 7f77f8ff9640 1 -- 172.21.15.49:0/1795532338 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f77ec018020 con 0x7f77fc100700 2024-03-18T00:55:04.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.534+0000 7f7804882740 1 -- start start 2024-03-18T00:55:04.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.535+0000 7f7804882740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab02990 0x559d5ab02db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.535+0000 7f7804882740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab07e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.535+0000 7f7804882740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab084f0 0x559d5ab0a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.535+0000 7f7804882740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559d5aa670d0 con 0x559d5ab03380 2024-03-18T00:55:04.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.535+0000 7f7804882740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559d5aa0fc70 con 0x559d5ab02990 2024-03-18T00:55:04.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.535+0000 7f7804882740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559d5aa63170 con 0x559d5ab084f0 2024-03-18T00:55:04.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.536+0000 7f77fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab07e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.536+0000 7f77faffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab084f0 0x559d5ab0a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.536+0000 7f77fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab07e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44948/0 (socket says 172.21.15.49:44948) 2024-03-18T00:55:04.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.536+0000 7f77fb7fe640 1 -- 172.21.15.49:0/2956928842 learned_addr learned my addr 172.21.15.49:0/2956928842 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:04.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.536+0000 7f77fb7fe640 1 -- 172.21.15.49:0/2956928842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab084f0 msgr2=0x559d5ab0a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:04.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.536+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/2956928842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab084f0 0x559d5ab0a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.536+0000 7f77fb7fe640 1 -- 172.21.15.49:0/2956928842 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab02990 msgr2=0x559d5ab02db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:04.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.536+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/2956928842 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab02990 0x559d5ab02db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.536+0000 7f77fb7fe640 1 -- 172.21.15.49:0/2956928842 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d5a8437e0 con 0x559d5ab03380 2024-03-18T00:55:04.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.537+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/2956928842 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab07e80 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0d4710 tx=0x7f77ec0d4bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d80bbc6e74f89a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:04.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.537+0000 7f77e1ffb640 1 -- 172.21.15.49:0/2956928842 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0e1070 con 0x559d5ab03380 2024-03-18T00:55:04.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.537+0000 7f77e1ffb640 1 -- 172.21.15.49:0/2956928842 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77ec0d4e20 con 0x559d5ab03380 2024-03-18T00:55:04.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.537+0000 7f77e1ffb640 1 -- 172.21.15.49:0/2956928842 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0d6b40 con 0x559d5ab03380 2024-03-18T00:55:04.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.537+0000 7f77f8ff9640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77e40a28d0 0x7f77e40a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.537+0000 7f77fbfff640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77e40a28d0 0x7f77e40a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.538+0000 7f77f8ff9640 1 -- 172.21.15.49:0/1795532338 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f77ec0ce4b0 con 0x7f77fc100700 2024-03-18T00:55:04.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.538+0000 7f7804882740 1 -- 172.21.15.49:0/2956928842 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 msgr2=0x559d5ab07e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:04.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.538+0000 7f7804882740 1 --2- 172.21.15.49:0/2956928842 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab07e80 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0d4710 tx=0x7f77ec0d4bd0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.539+0000 7f7804882740 1 -- 172.21.15.49:0/2956928842 shutdown_connections 2024-03-18T00:55:04.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.539+0000 7f7804882740 1 --2- 172.21.15.49:0/2956928842 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab084f0 0x559d5ab0a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.539+0000 7f7804882740 1 --2- 172.21.15.49:0/2956928842 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab07e80 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.539+0000 7f7804882740 1 --2- 172.21.15.49:0/2956928842 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab02990 0x559d5ab02db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.539+0000 7f7804882740 1 -- 172.21.15.49:0/2956928842 >> 172.21.15.49:0/2956928842 conn(0x559d5aacac50 msgr2=0x559d5aafe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:04.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.539+0000 7f77fbfff640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77e40a28d0 0x7f77e40a4d90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f77f402c750 tx=0x7f77f402c660 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:04.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.539+0000 7f7804882740 1 -- 172.21.15.49:0/2956928842 shutdown_connections 2024-03-18T00:55:04.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.539+0000 7f7804882740 1 -- 172.21.15.49:0/2956928842 wait complete. 2024-03-18T00:55:04.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.541+0000 7f7804882740 1 -- start start 2024-03-18T00:55:04.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.541+0000 7f7804882740 1 Processor -- start 2024-03-18T00:55:04.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.542+0000 7f7804882740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab02990 0x559d5ab0ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.542+0000 7f7804882740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab0d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.542+0000 7f7804882740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab084f0 0x559d5ab143f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.542+0000 7f7804882740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559d5ab0b4a0 con 0x559d5ab03380 2024-03-18T00:55:04.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.542+0000 7f7804882740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559d5aa230d0 con 0x559d5ab084f0 2024-03-18T00:55:04.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.542+0000 7f7804882740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559d5ab0b620 con 0x559d5ab02990 2024-03-18T00:55:04.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.542+0000 7f7804882740 10 client.0 _ll_register_callbacks cb 0x559d5a9ec760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:04.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.543+0000 7f77faffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab0d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.543+0000 7f77faffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab0d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44952/0 (socket says 172.21.15.49:44952) 2024-03-18T00:55:04.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.543+0000 7f77faffd640 1 -- 172.21.15.49:0/2534248002 learned_addr learned my addr 172.21.15.49:0/2534248002 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:04.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.543+0000 7f77faffd640 1 -- 172.21.15.49:0/2534248002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab02990 msgr2=0x559d5ab0ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.543+0000 7f77faffd640 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab02990 0x559d5ab0ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.543+0000 7f77faffd640 1 -- 172.21.15.49:0/2534248002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab084f0 msgr2=0x559d5ab143f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.543+0000 7f77faffd640 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab084f0 0x559d5ab143f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.543+0000 7f77faffd640 1 -- 172.21.15.49:0/2534248002 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d5a8a3f70 con 0x559d5ab03380 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.544+0000 7f77faffd640 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab0d3b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f77f000aed0 tx=0x7f77f000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd73f7f31b904c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.544+0000 7f77daffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.544+0000 7f77daffd640 1 -- 172.21.15.49:0/2534248002 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0015070 con 0x559d5ab03380 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.544+0000 7f77daffd640 1 -- 172.21.15.49:0/2534248002 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f77f00115f0 con 0x559d5ab03380 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.544+0000 7f7804882740 1 -- 172.21.15.49:0/2534248002 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77ec0d4e20 con 0x559d5ab03380 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.544+0000 7f77daffd640 1 -- 172.21.15.49:0/2534248002 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f77f0021050 con 0x559d5ab03380 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.544+0000 7f7804882740 10 client.16635 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:04.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.544+0000 7f7804882740 1 -- 172.21.15.49:0/2534248002 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559d5a73b170 con 0x559d5ab03380 2024-03-18T00:55:04.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.545+0000 7f77daffd640 1 -- 172.21.15.49:0/2534248002 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f0011cd0 con 0x559d5ab03380 2024-03-18T00:55:04.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.545+0000 7f77daffd640 1 -- 172.21.15.49:0/2534248002 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f77f0019b60 con 0x559d5ab03380 2024-03-18T00:55:04.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.545+0000 7f7804882740 10 client.16635 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:04.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.545+0000 7f7804882740 10 client.16635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:04.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.545+0000 7f7804882740 20 client.16635 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:04.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.545+0000 7f7804882740 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559d5ab189e0 0x559d5ab1ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.545+0000 7f7804882740 4 client.16635 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:04.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.545+0000 7f7804882740 1 -- 172.21.15.49:0/2534248002 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f77ec0d4e20 con 0x559d5ab189e0 2024-03-18T00:55:04.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.546+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559d5ab189e0 0x559d5ab1ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.551+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559d5ab189e0 0x559d5ab1ae40 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e50f4c348a6ea1c5 server_cookie=6a29d6c21900dad in_seq=0 out_seq=0 2024-03-18T00:55:04.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.551+0000 7f77daffd640 10 client.16635 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.556+0000 7f77daffd640 1 -- 172.21.15.49:0/2534248002 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f77ec0d4e20 con 0x559d5ab189e0 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.556+0000 7f77daffd640 10 client.16635 handle_command_reply: tid=0 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.556+0000 7f7804882740 1 client.16635 shutdown 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 -- 172.21.15.49:0/2534248002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 msgr2=0x559d5ab0d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab0d3b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f77f000aed0 tx=0x7f77f000c520 comp rx=0 tx=0).stop 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 -- 172.21.15.49:0/2534248002 shutdown_connections 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559d5ab189e0 0x559d5ab1ae40 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab084f0 0x559d5ab143f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab03380 0x559d5ab0d3b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 --2- 172.21.15.49:0/2534248002 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab02990 0x559d5ab0ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 -- 172.21.15.49:0/2534248002 >> 172.21.15.49:0/2534248002 conn(0x559d5aacac50 msgr2=0x559d5aafd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:04.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f77daffd640 0 client.16635 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:04.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 -- 172.21.15.49:0/2534248002 shutdown_connections 2024-03-18T00:55:04.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 1 -- 172.21.15.49:0/2534248002 wait complete. 2024-03-18T00:55:04.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.557+0000 7f7804882740 20 client.16635 trim_cache size 0 max 16384 2024-03-18T00:55:04.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.596+0000 7f7804882740 1 Processor -- start 2024-03-18T00:55:04.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.596+0000 7f7804882740 1 -- start start 2024-03-18T00:55:04.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.596+0000 7f7804882740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5aacac50 0x559d5ab0ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.596+0000 7f7804882740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab02990 0x559d5ab143b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.597+0000 7f7804882740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab03380 0x559d5ab0bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.597+0000 7f77faffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab02990 0x559d5ab143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.597+0000 7f77faffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab02990 0x559d5ab143b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44960/0 (socket says 172.21.15.49:44960) 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.597+0000 7f77faffd640 1 -- 172.21.15.49:0/2256807578 learned_addr learned my addr 172.21.15.49:0/2256807578 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.597+0000 7f7804882740 1 -- 172.21.15.49:0/2256807578 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f77e800e5f0 con 0x559d5ab02990 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.597+0000 7f7804882740 1 -- 172.21.15.49:0/2256807578 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8010130 con 0x559d5ab03380 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.597+0000 7f7804882740 1 -- 172.21.15.49:0/2256807578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f77e800ecc0 con 0x559d5aacac50 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.597+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/2256807578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5aacac50 0x559d5ab0ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.598+0000 7f77faffd640 1 -- 172.21.15.49:0/2256807578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5aacac50 msgr2=0x559d5ab0ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.598+0000 7f77faffd640 1 --2- 172.21.15.49:0/2256807578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5aacac50 0x559d5ab0ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.598+0000 7f77faffd640 1 -- 172.21.15.49:0/2256807578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab03380 msgr2=0x559d5ab0bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:04.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.598+0000 7f77faffd640 1 --2- 172.21.15.49:0/2256807578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab03380 0x559d5ab0bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.598+0000 7f77faffd640 1 -- 172.21.15.49:0/2256807578 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d5aaff050 con 0x559d5ab02990 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.598+0000 7f77faffd640 1 --2- 172.21.15.49:0/2256807578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab02990 0x559d5ab143b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f77f0000ea0 tx=0x7f77f000df10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f709ab075b5cce4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.598+0000 7f77e17fa640 1 -- 172.21.15.49:0/2256807578 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f002a070 con 0x559d5ab02990 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.598+0000 7f77e17fa640 1 -- 172.21.15.49:0/2256807578 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f001e030 con 0x559d5ab02990 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.598+0000 7f77e17fa640 1 -- 172.21.15.49:0/2256807578 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0015040 con 0x559d5ab02990 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.599+0000 7f7804882740 1 -- 172.21.15.49:0/2256807578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab02990 msgr2=0x559d5ab143b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.599+0000 7f7804882740 1 --2- 172.21.15.49:0/2256807578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab02990 0x559d5ab143b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f77f0000ea0 tx=0x7f77f000df10 comp rx=0 tx=0).stop 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.599+0000 7f7804882740 1 -- 172.21.15.49:0/2256807578 shutdown_connections 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.599+0000 7f7804882740 1 --2- 172.21.15.49:0/2256807578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab03380 0x559d5ab0bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.599+0000 7f7804882740 1 --2- 172.21.15.49:0/2256807578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5ab02990 0x559d5ab143b0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.599+0000 7f7804882740 1 --2- 172.21.15.49:0/2256807578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5aacac50 0x559d5ab0ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.599+0000 7f7804882740 1 -- 172.21.15.49:0/2256807578 >> 172.21.15.49:0/2256807578 conn(0x559d5aaf9910 msgr2=0x559d5aafd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:04.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.600+0000 7f7804882740 1 -- 172.21.15.49:0/2256807578 shutdown_connections 2024-03-18T00:55:04.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.600+0000 7f7804882740 1 -- 172.21.15.49:0/2256807578 wait complete. 2024-03-18T00:55:04.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.600+0000 7f7804882740 1 -- start start 2024-03-18T00:55:04.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.601+0000 7f7804882740 1 Processor -- start 2024-03-18T00:55:04.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.601+0000 7f7804882740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5aacac50 0x559d5ab140c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.601+0000 7f7804882740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab02990 0x559d5ab126d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.601+0000 7f7804882740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab03380 0x559d5ab12c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.601+0000 7f7804882740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f77e800fa60 con 0x559d5aacac50 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.601+0000 7f7804882740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f77e800f390 con 0x559d5ab02990 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.601+0000 7f7804882740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f77e800c5d0 con 0x559d5ab03380 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.601+0000 7f7804882740 10 client.0 _ll_register_callbacks cb 0x7f77f000aa90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5aacac50 0x559d5ab140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77fb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5aacac50 0x559d5ab140c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:44974/0 (socket says 172.21.15.49:44974) 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77fb7fe640 1 -- 172.21.15.49:0/3426679397 learned_addr learned my addr 172.21.15.49:0/3426679397 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77fb7fe640 1 -- 172.21.15.49:0/3426679397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab03380 msgr2=0x559d5ab12c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab03380 0x559d5ab12c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77fb7fe640 1 -- 172.21.15.49:0/3426679397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab02990 msgr2=0x559d5ab126d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:04.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab02990 0x559d5ab126d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:04.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77fb7fe640 1 -- 172.21.15.49:0/3426679397 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d5ab0dfa0 con 0x559d5aacac50 2024-03-18T00:55:04.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77fb7fe640 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5aacac50 0x559d5ab140c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f77ec09adc0 tx=0x7f77ec0d1aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963dd1ff9056d802 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:04.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.602+0000 7f77daffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:04.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.603+0000 7f77daffd640 1 -- 172.21.15.49:0/3426679397 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec09ac00 con 0x559d5aacac50 2024-03-18T00:55:04.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.603+0000 7f77daffd640 1 -- 172.21.15.49:0/3426679397 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f77ec0d4290 con 0x559d5aacac50 2024-03-18T00:55:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.603+0000 7f7804882740 1 -- 172.21.15.49:0/3426679397 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77f001e030 con 0x559d5aacac50 2024-03-18T00:55:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.603+0000 7f77daffd640 1 -- 172.21.15.49:0/3426679397 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f77ec0b8050 con 0x559d5aacac50 2024-03-18T00:55:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.604+0000 7f7804882740 10 client.16641 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.604+0000 7f7804882740 1 -- 172.21.15.49:0/3426679397 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559d5ab7fac0 con 0x559d5aacac50 2024-03-18T00:55:04.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.604+0000 7f77daffd640 1 -- 172.21.15.49:0/3426679397 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f001e030 con 0x559d5aacac50 2024-03-18T00:55:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.604+0000 7f77daffd640 1 -- 172.21.15.49:0/3426679397 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f77ec0974a0 con 0x559d5aacac50 2024-03-18T00:55:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.605+0000 7f7804882740 10 client.16641 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.605+0000 7f7804882740 10 client.16641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.605+0000 7f7804882740 20 client.16641 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.605+0000 7f7804882740 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559d5ab084f0 0x559d5ab82120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.605+0000 7f7804882740 4 client.16641 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:55:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.605+0000 7f7804882740 1 -- 172.21.15.49:0/3426679397 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559d5aa74610 con 0x559d5ab084f0 2024-03-18T00:55:04.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.606+0000 7f77faffd640 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559d5ab084f0 0x559d5ab82120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:04.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.613+0000 7f77daffd640 10 client.16641 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:04.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:04.613+0000 7f77faffd640 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559d5ab084f0 0x559d5ab82120 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3345f1cb9d26c75 server_cookie=9469f7f9a088be66 in_seq=0 out_seq=0 2024-03-18T00:55:05.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.232+0000 7f77daffd640 1 -- 172.21.15.49:0/3426679397 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559d5aa74610 con 0x559d5ab084f0 2024-03-18T00:55:05.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.232+0000 7f77daffd640 10 client.16641 handle_command_reply: tid=0 2024-03-18T00:55:05.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.234+0000 7f7804882740 1 client.16641 shutdown 2024-03-18T00:55:05.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.235+0000 7f7804882740 1 -- 172.21.15.49:0/3426679397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5aacac50 msgr2=0x559d5ab140c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.235+0000 7f7804882740 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5aacac50 0x559d5ab140c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f77ec09adc0 tx=0x7f77ec0d1aa0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.235+0000 7f7804882740 1 -- 172.21.15.49:0/3426679397 shutdown_connections 2024-03-18T00:55:05.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.235+0000 7f7804882740 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559d5ab084f0 0x559d5ab82120 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.235+0000 7f7804882740 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559d5ab02990 0x559d5ab126d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.235+0000 7f7804882740 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559d5ab03380 0x559d5ab12c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.235+0000 7f7804882740 1 --2- 172.21.15.49:0/3426679397 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559d5aacac50 0x559d5ab140c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.235+0000 7f7804882740 1 -- 172.21.15.49:0/3426679397 >> 172.21.15.49:0/3426679397 conn(0x559d5aaf9910 msgr2=0x559d5ab01540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:05.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.235+0000 7f77daffd640 0 client.16641 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:05.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.239+0000 7f7804882740 1 -- 172.21.15.49:0/3426679397 shutdown_connections 2024-03-18T00:55:05.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.239+0000 7f7804882740 1 -- 172.21.15.49:0/3426679397 wait complete. 2024-03-18T00:55:05.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.239+0000 7f7804882740 20 client.16641 trim_cache size 0 max 16384 2024-03-18T00:55:05.241 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:55:05.242 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:55:05.242 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "446b4af4-e182-4cb3-82b3-4db0a4e72548", 2024-03-18T00:55:05.242 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:55:05.242 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:55:05.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.244+0000 7f7801c97640 1 -- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77e40a28d0 msgr2=0x7f77e40a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.244+0000 7f7801c97640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77e40a28d0 0x7f77e40a4d90 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f77f402c750 tx=0x7f77f402c660 comp rx=0 tx=0).stop 2024-03-18T00:55:05.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.244+0000 7f7801c97640 1 -- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc100700 msgr2=0x7f77fc1a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.244+0000 7f7801c97640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc100700 0x7f77fc1a0980 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00b470 tx=0x7f77ec00b930 comp rx=0 tx=0).stop 2024-03-18T00:55:05.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.251+0000 7f7801c97640 1 -- 172.21.15.49:0/1795532338 shutdown_connections 2024-03-18T00:55:05.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.251+0000 7f7801c97640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77e40a28d0 0x7f77e40a4d90 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.251+0000 7f7801c97640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77fc1018f0 0x7f77fc1a2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.251+0000 7f7801c97640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77fc1022b0 0x7f77fc1a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.251+0000 7f7801c97640 1 --2- 172.21.15.49:0/1795532338 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77fc100700 0x7f77fc1a0980 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.251+0000 7f7801c97640 1 -- 172.21.15.49:0/1795532338 >> 172.21.15.49:0/1795532338 conn(0x7f77fc0fbc90 msgr2=0x7f77fc069b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:05.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.260+0000 7f7801c97640 1 -- 172.21.15.49:0/1795532338 shutdown_connections 2024-03-18T00:55:05.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.261+0000 7f7801c97640 1 -- 172.21.15.49:0/1795532338 wait complete. 2024-03-18T00:55:05.337 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:04 smithi049 ceph-mon[32130]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 971 MiB/s wr, 2.19k op/s 2024-03-18T00:55:05.344 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:55:05.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:55:05.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:04 smithi107 ceph-mon[36270]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 971 MiB/s wr, 2.19k op/s 2024-03-18T00:55:05.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:04 smithi106 ceph-mon[35899]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 971 MiB/s wr, 2.19k op/s 2024-03-18T00:55:05.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.758+0000 7f6095112640 1 -- 172.21.15.49:0/75117655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090072bf0 msgr2=0x7f6090073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.758+0000 7f6095112640 1 --2- 172.21.15.49:0/75117655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090072bf0 0x7f6090073010 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f6080009900 tx=0x7f608002bc50 comp rx=0 tx=0).stop 2024-03-18T00:55:05.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.763+0000 7f6095112640 1 -- 172.21.15.49:0/75117655 shutdown_connections 2024-03-18T00:55:05.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.763+0000 7f6095112640 1 --2- 172.21.15.49:0/75117655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6090109870 0x7f609010e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.763+0000 7f6095112640 1 --2- 172.21.15.49:0/75117655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6090108eb0 0x7f6090109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.763+0000 7f6095112640 1 --2- 172.21.15.49:0/75117655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090072bf0 0x7f6090073010 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.763+0000 7f6095112640 1 -- 172.21.15.49:0/75117655 >> 172.21.15.49:0/75117655 conn(0x7f609006e6c0 msgr2=0x7f6090070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:05.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.771+0000 7f6095112640 1 -- 172.21.15.49:0/75117655 shutdown_connections 2024-03-18T00:55:05.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.775+0000 7f6095112640 1 -- 172.21.15.49:0/75117655 wait complete. 2024-03-18T00:55:05.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.776+0000 7f6095112640 1 Processor -- start 2024-03-18T00:55:05.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.776+0000 7f6095112640 1 -- start start 2024-03-18T00:55:05.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.777+0000 7f6095112640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6090072bf0 0x7f6090119a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.777+0000 7f6095112640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6090108eb0 0x7f6090119f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.777+0000 7f6095112640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090109870 0x7f6090118040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.777+0000 7f6095112640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f6090110920 con 0x7f6090109870 2024-03-18T00:55:05.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.777+0000 7f6095112640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f60901107a0 con 0x7f6090108eb0 2024-03-18T00:55:05.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.777+0000 7f6095112640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6090110aa0 con 0x7f6090072bf0 2024-03-18T00:55:05.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.779+0000 7f6094911640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090109870 0x7f6090118040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.780+0000 7f6094911640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090109870 0x7f6090118040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45000/0 (socket says 172.21.15.49:45000) 2024-03-18T00:55:05.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.780+0000 7f6094911640 1 -- 172.21.15.49:0/805689514 learned_addr learned my addr 172.21.15.49:0/805689514 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:05.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.780+0000 7f608ffff640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6090072bf0 0x7f6090119a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.780+0000 7f608f7fe640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6090108eb0 0x7f6090119f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.783+0000 7f6094911640 1 -- 172.21.15.49:0/805689514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6090072bf0 msgr2=0x7f6090119a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.783+0000 7f6094911640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6090072bf0 0x7f6090119a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.783+0000 7f6094911640 1 -- 172.21.15.49:0/805689514 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6090108eb0 msgr2=0x7f6090119f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.783+0000 7f6094911640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6090108eb0 0x7f6090119f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.783+0000 7f6094911640 1 -- 172.21.15.49:0/805689514 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6090118720 con 0x7f6090109870 2024-03-18T00:55:05.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.784+0000 7f608f7fe640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6090108eb0 0x7f6090119f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:05.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.784+0000 7f6094911640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090109870 0x7f6090118040 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f608800eaa0 tx=0x7f608800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:05.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.788+0000 7f608d7fa640 1 -- 172.21.15.49:0/805689514 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608800cdb0 con 0x7f6090109870 2024-03-18T00:55:05.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.788+0000 7f608d7fa640 1 -- 172.21.15.49:0/805689514 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6088005020 con 0x7f6090109870 2024-03-18T00:55:05.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.788+0000 7f608d7fa640 1 -- 172.21.15.49:0/805689514 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60880116b0 con 0x7f6090109870 2024-03-18T00:55:05.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.789+0000 7f6095112640 1 -- 172.21.15.49:0/805689514 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6090118a10 con 0x7f6090109870 2024-03-18T00:55:05.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.789+0000 7f6095112640 1 -- 172.21.15.49:0/805689514 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60901b7500 con 0x7f6090109870 2024-03-18T00:55:05.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.794+0000 7f6099037740 1 Processor -- start 2024-03-18T00:55:05.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.794+0000 7f6099037740 1 -- start start 2024-03-18T00:55:05.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.794+0000 7f6099037740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x55796939ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.794+0000 7f6099037740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939d020 0x5579693a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.794+0000 7f6099037740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579693a2190 0x5579693a4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.794+0000 7f6099037740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557969301500 con 0x55796939d020 2024-03-18T00:55:05.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.794+0000 7f6099037740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5579692a9be0 con 0x5579693a2190 2024-03-18T00:55:05.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.794+0000 7f6099037740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5579692fd5a0 con 0x55796939c630 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.799+0000 7f608f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x55796939ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.802+0000 7f608f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x55796939ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:57236/0 (socket says 172.21.15.49:57236) 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.802+0000 7f608f7fe640 1 -- 172.21.15.49:0/417932866 learned_addr learned my addr 172.21.15.49:0/417932866 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.803+0000 7f608f7fe640 1 --2- 172.21.15.49:0/417932866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579693a2190 0x5579693a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.804+0000 7f608d7fa640 1 -- 172.21.15.49:0/805689514 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6088005560 con 0x7f6090109870 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.804+0000 7f608f7fe640 1 -- 172.21.15.49:0/417932866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579693a2190 msgr2=0x5579693a4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.804+0000 7f608f7fe640 1 --2- 172.21.15.49:0/417932866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579693a2190 0x5579693a4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.804+0000 7f608f7fe640 1 -- 172.21.15.49:0/417932866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939d020 msgr2=0x5579693a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.804+0000 7f608f7fe640 1 --2- 172.21.15.49:0/417932866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939d020 0x5579693a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.804+0000 7f608f7fe640 1 -- 172.21.15.49:0/417932866 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579692bd4b0 con 0x55796939c630 2024-03-18T00:55:05.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.805+0000 7f608d7fa640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60780a2980 0x7f60780a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.805+0000 7f608d7fa640 1 -- 172.21.15.49:0/805689514 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f60880cec60 con 0x7f6090109870 2024-03-18T00:55:05.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.807+0000 7f608ffff640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60780a2980 0x7f60780a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.807+0000 7f608f7fe640 1 --2- 172.21.15.49:0/417932866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x55796939ca50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6084013890 tx=0x7f608400b420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75e65030ef9e0258 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:05.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.808+0000 7f608ffff640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60780a2980 0x7f60780a4e40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f608002c260 tx=0x7f6080005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:05.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.808+0000 7f6075ffb640 1 -- 172.21.15.49:0/417932866 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608400e8f0 con 0x55796939c630 2024-03-18T00:55:05.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.808+0000 7f6075ffb640 1 -- 172.21.15.49:0/417932866 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60840113e0 con 0x55796939c630 2024-03-18T00:55:05.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.808+0000 7f6075ffb640 1 -- 172.21.15.49:0/417932866 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6084010630 con 0x55796939c630 2024-03-18T00:55:05.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.813+0000 7f6099037740 1 -- 172.21.15.49:0/417932866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 msgr2=0x55796939ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.813+0000 7f6099037740 1 --2- 172.21.15.49:0/417932866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x55796939ca50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6084013890 tx=0x7f608400b420 comp rx=0 tx=0).stop 2024-03-18T00:55:05.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.814+0000 7f6099037740 1 -- 172.21.15.49:0/417932866 shutdown_connections 2024-03-18T00:55:05.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.814+0000 7f6099037740 1 --2- 172.21.15.49:0/417932866 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579693a2190 0x5579693a4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.814+0000 7f6099037740 1 --2- 172.21.15.49:0/417932866 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939d020 0x5579693a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.814+0000 7f6099037740 1 --2- 172.21.15.49:0/417932866 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x55796939ca50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.814+0000 7f6099037740 1 -- 172.21.15.49:0/417932866 >> 172.21.15.49:0/417932866 conn(0x557969397ba0 msgr2=0x557969397fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:05.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.814+0000 7f6099037740 1 -- 172.21.15.49:0/417932866 shutdown_connections 2024-03-18T00:55:05.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.814+0000 7f6099037740 1 -- 172.21.15.49:0/417932866 wait complete. 2024-03-18T00:55:05.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.814+0000 7f6099037740 1 -- start start 2024-03-18T00:55:05.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.815+0000 7f6099037740 1 Processor -- start 2024-03-18T00:55:05.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.815+0000 7f6099037740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939c630 0x5579693a6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.815+0000 7f6099037740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 0x5579693a6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.815+0000 7f6099037740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579693a2190 0x5579693adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.815+0000 7f6099037740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5579693a5050 con 0x55796939c630 2024-03-18T00:55:05.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.815+0000 7f6099037740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5579692bd160 con 0x5579693a2190 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.815+0000 7f6099037740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5579693a51d0 con 0x55796939d020 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.815+0000 7f6099037740 10 client.0 _ll_register_callbacks cb 0x5579692a1150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.816+0000 7f608ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939c630 0x5579693a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.816+0000 7f608ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939c630 0x5579693a6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45014/0 (socket says 172.21.15.49:45014) 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.816+0000 7f608ffff640 1 -- 172.21.15.49:0/2917599221 learned_addr learned my addr 172.21.15.49:0/2917599221 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.816+0000 7f608ffff640 1 -- 172.21.15.49:0/2917599221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 msgr2=0x5579693a6e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.816+0000 7f608ffff640 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 0x5579693a6e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.816+0000 7f608ffff640 1 -- 172.21.15.49:0/2917599221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579693a2190 msgr2=0x5579693adeb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.816+0000 7f608ffff640 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579693a2190 0x5579693adeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.816+0000 7f608ffff640 1 -- 172.21.15.49:0/2917599221 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579690df7d0 con 0x55796939c630 2024-03-18T00:55:05.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.817+0000 7f608ffff640 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939c630 0x5579693a6930 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6080004cf0 tx=0x7f608002bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5104921ec21973aa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:05.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.830+0000 7f606f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:05.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.830+0000 7f606f7fe640 1 -- 172.21.15.49:0/2917599221 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080050060 con 0x55796939c630 2024-03-18T00:55:05.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.830+0000 7f6099037740 1 -- 172.21.15.49:0/2917599221 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60840113e0 con 0x55796939c630 2024-03-18T00:55:05.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.830+0000 7f606f7fe640 1 -- 172.21.15.49:0/2917599221 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6080003800 con 0x55796939c630 2024-03-18T00:55:05.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.833+0000 7f606f7fe640 1 -- 172.21.15.49:0/2917599221 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6080003a00 con 0x55796939c630 2024-03-18T00:55:05.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.836+0000 7f6099037740 10 client.16650 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:05.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.836+0000 7f6099037740 1 -- 172.21.15.49:0/2917599221 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5579690153f0 con 0x55796939c630 2024-03-18T00:55:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.841+0000 7f606f7fe640 1 -- 172.21.15.49:0/2917599221 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60840113e0 con 0x55796939c630 2024-03-18T00:55:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.841+0000 7f606f7fe640 1 -- 172.21.15.49:0/2917599221 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6080040480 con 0x55796939c630 2024-03-18T00:55:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.841+0000 7f6099037740 10 client.16650 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.841+0000 7f6099037740 10 client.16650 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.841+0000 7f6099037740 20 client.16650 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.841+0000 7f6099037740 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579693b24a0 0x5579693b4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.841+0000 7f6099037740 4 client.16650 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:05.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.841+0000 7f6099037740 1 -- 172.21.15.49:0/2917599221 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55796930ea40 con 0x5579693b24a0 2024-03-18T00:55:05.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.844+0000 7f608f7fe640 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579693b24a0 0x5579693b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.853+0000 7f606f7fe640 10 client.16650 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:05.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.853+0000 7f608f7fe640 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579693b24a0 0x5579693b4930 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=465da6c5c36a945d server_cookie=d670a357a0286a3e in_seq=0 out_seq=0 2024-03-18T00:55:05.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.858+0000 7f606f7fe640 1 -- 172.21.15.49:0/2917599221 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55796930ea40 con 0x5579693b24a0 2024-03-18T00:55:05.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.858+0000 7f606f7fe640 10 client.16650 handle_command_reply: tid=0 2024-03-18T00:55:05.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.858+0000 7f6099037740 1 client.16650 shutdown 2024-03-18T00:55:05.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.859+0000 7f6099037740 1 -- 172.21.15.49:0/2917599221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939c630 msgr2=0x5579693a6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.859+0000 7f6099037740 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939c630 0x5579693a6930 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6080004cf0 tx=0x7f608002bc50 comp rx=0 tx=0).stop 2024-03-18T00:55:05.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.859+0000 7f6099037740 1 -- 172.21.15.49:0/2917599221 shutdown_connections 2024-03-18T00:55:05.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.859+0000 7f6099037740 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5579693b24a0 0x5579693b4930 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.859+0000 7f6099037740 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5579693a2190 0x5579693adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.859+0000 7f6099037740 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 0x5579693a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.859+0000 7f6099037740 1 --2- 172.21.15.49:0/2917599221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55796939c630 0x5579693a6930 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.859+0000 7f6099037740 1 -- 172.21.15.49:0/2917599221 >> 172.21.15.49:0/2917599221 conn(0x557969397ba0 msgr2=0x557969396da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:05.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.859+0000 7f606f7fe640 0 client.16650 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:05.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.861+0000 7f6099037740 1 -- 172.21.15.49:0/2917599221 shutdown_connections 2024-03-18T00:55:05.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.861+0000 7f6099037740 1 -- 172.21.15.49:0/2917599221 wait complete. 2024-03-18T00:55:05.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.861+0000 7f6099037740 20 client.16650 trim_cache size 0 max 16384 2024-03-18T00:55:05.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.915+0000 7f6099037740 1 Processor -- start 2024-03-18T00:55:05.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.916+0000 7f6099037740 1 -- start start 2024-03-18T00:55:05.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.916+0000 7f6099037740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x5579693a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.916+0000 7f6099037740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55796939d020 0x5579693ade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.916+0000 7f6099037740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x55796939d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.916+0000 7f6099037740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f60901c02c0 con 0x5579693a2190 2024-03-18T00:55:05.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.916+0000 7f6099037740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f60901c1e00 con 0x55796939d020 2024-03-18T00:55:05.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.916+0000 7f6099037740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f60901c0990 con 0x55796939c630 2024-03-18T00:55:05.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.917+0000 7f6094911640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x55796939d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.917+0000 7f6094911640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x55796939d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45030/0 (socket says 172.21.15.49:45030) 2024-03-18T00:55:05.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.917+0000 7f6094911640 1 -- 172.21.15.49:0/1806481382 learned_addr learned my addr 172.21.15.49:0/1806481382 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:05.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.917+0000 7f608ffff640 1 --2- 172.21.15.49:0/1806481382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x5579693a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.918+0000 7f608f7fe640 1 --2- 172.21.15.49:0/1806481382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55796939d020 0x5579693ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.918+0000 7f608ffff640 1 -- 172.21.15.49:0/1806481382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55796939d020 msgr2=0x5579693ade70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:05.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.918+0000 7f608ffff640 1 --2- 172.21.15.49:0/1806481382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55796939d020 0x5579693ade70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.918+0000 7f608ffff640 1 -- 172.21.15.49:0/1806481382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 msgr2=0x55796939d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:05.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.918+0000 7f608ffff640 1 --2- 172.21.15.49:0/1806481382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x55796939d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.918+0000 7f608ffff640 1 -- 172.21.15.49:0/1806481382 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557968fd50f0 con 0x55796939c630 2024-03-18T00:55:05.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.919+0000 7f608ffff640 1 --2- 172.21.15.49:0/1806481382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x5579693a6dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6080040920 tx=0x7f6080041fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a859a6de8a640d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:05.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.920+0000 7f60757fa640 1 -- 172.21.15.49:0/1806481382 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608005a030 con 0x55796939c630 2024-03-18T00:55:05.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.920+0000 7f60757fa640 1 -- 172.21.15.49:0/1806481382 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6080047040 con 0x55796939c630 2024-03-18T00:55:05.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.920+0000 7f60757fa640 1 -- 172.21.15.49:0/1806481382 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080050040 con 0x55796939c630 2024-03-18T00:55:05.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.921+0000 7f6099037740 1 -- 172.21.15.49:0/1806481382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 msgr2=0x5579693a6dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.921+0000 7f6099037740 1 --2- 172.21.15.49:0/1806481382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x5579693a6dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f6080040920 tx=0x7f6080041fc0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.921+0000 7f6099037740 1 -- 172.21.15.49:0/1806481382 shutdown_connections 2024-03-18T00:55:05.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.921+0000 7f6099037740 1 --2- 172.21.15.49:0/1806481382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55796939d020 0x5579693ade70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.921+0000 7f6099037740 1 --2- 172.21.15.49:0/1806481382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x55796939d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.921+0000 7f6099037740 1 --2- 172.21.15.49:0/1806481382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939c630 0x5579693a6dd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.921+0000 7f6099037740 1 -- 172.21.15.49:0/1806481382 >> 172.21.15.49:0/1806481382 conn(0x557969397ba0 msgr2=0x55796939ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:05.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.921+0000 7f6099037740 1 -- 172.21.15.49:0/1806481382 shutdown_connections 2024-03-18T00:55:05.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.922+0000 7f6099037740 1 -- 172.21.15.49:0/1806481382 wait complete. 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.923+0000 7f6099037740 1 -- start start 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.923+0000 7f6099037740 1 Processor -- start 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.923+0000 7f6099037740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55796939c630 0x5579693ab5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.923+0000 7f6099037740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 0x5579693abb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.923+0000 7f6099037740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x5579693acb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.924+0000 7f6099037740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f60901c1730 con 0x5579693a2190 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.924+0000 7f6099037740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f60901c1060 con 0x55796939c630 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.924+0000 7f6099037740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f60901be2a0 con 0x55796939d020 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.924+0000 7f6099037740 10 client.0 _ll_register_callbacks cb 0x7f6080009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:05.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.924+0000 7f6094911640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x5579693acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.924+0000 7f6094911640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x5579693acb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:45036/0 (socket says 172.21.15.49:45036) 2024-03-18T00:55:05.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.924+0000 7f6094911640 1 -- 172.21.15.49:0/2998320205 learned_addr learned my addr 172.21.15.49:0/2998320205 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:05.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.926+0000 7f608f7fe640 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 0x5579693abb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.927+0000 7f608f7fe640 1 -- 172.21.15.49:0/2998320205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55796939c630 msgr2=0x5579693ab5d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:05.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.927+0000 7f608f7fe640 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55796939c630 0x5579693ab5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.928 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.928+0000 7f608f7fe640 1 -- 172.21.15.49:0/2998320205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 msgr2=0x5579693acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:05.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.928+0000 7f608f7fe640 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x5579693acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.928+0000 7f608f7fe640 1 -- 172.21.15.49:0/2998320205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579692c89d0 con 0x55796939d020 2024-03-18T00:55:05.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.930+0000 7f6094911640 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x5579693acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:55:05.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.933+0000 7f608f7fe640 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 0x5579693abb10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6084012040 tx=0x7f608400b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=102d289bc8d7aef server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:05.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.933+0000 7f606f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:55:05.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.935+0000 7f606f7fe640 1 -- 172.21.15.49:0/2998320205 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608402d020 con 0x55796939d020 2024-03-18T00:55:05.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.935+0000 7f606f7fe640 1 -- 172.21.15.49:0/2998320205 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6084029b20 con 0x55796939d020 2024-03-18T00:55:05.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.936+0000 7f6099037740 1 -- 172.21.15.49:0/2998320205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6080047040 con 0x55796939d020 2024-03-18T00:55:05.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.939+0000 7f606f7fe640 1 -- 172.21.15.49:0/2998320205 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f608400bca0 con 0x55796939d020 2024-03-18T00:55:05.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.939+0000 7f6099037740 10 client.26134 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:05.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.939+0000 7f6099037740 1 -- 172.21.15.49:0/2998320205 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5579693ae250 con 0x55796939d020 2024-03-18T00:55:05.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.940+0000 7f606f7fe640 1 -- 172.21.15.49:0/2998320205 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6080047040 con 0x55796939d020 2024-03-18T00:55:05.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.941+0000 7f606f7fe640 1 -- 172.21.15.49:0/2998320205 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f608400f040 con 0x55796939d020 2024-03-18T00:55:05.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.941+0000 7f6099037740 10 client.26134 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:05.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.941+0000 7f6099037740 10 client.26134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:05.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.941+0000 7f6099037740 20 client.26134 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:05.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.941+0000 7f6099037740 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557969418bf0 0x55796941b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:05.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.941+0000 7f6099037740 4 client.26134 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:55:05.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.941+0000 7f6099037740 1 -- 172.21.15.49:0/2998320205 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557969308d90 con 0x557969418bf0 2024-03-18T00:55:05.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.943+0000 7f608ffff640 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557969418bf0 0x55796941b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:05.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.947+0000 7f606f7fe640 10 client.26134 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:05.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.947+0000 7f608ffff640 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557969418bf0 0x55796941b010 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d624c66ab83df1a server_cookie=4c645088ea36b78f in_seq=0 out_seq=0 2024-03-18T00:55:05.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.954+0000 7f606f7fe640 1 -- 172.21.15.49:0/2998320205 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x557969308d90 con 0x557969418bf0 2024-03-18T00:55:05.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.954+0000 7f606f7fe640 10 client.26134 handle_command_reply: tid=0 2024-03-18T00:55:05.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.954+0000 7f6099037740 1 client.26134 shutdown 2024-03-18T00:55:05.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.956+0000 7f6099037740 1 -- 172.21.15.49:0/2998320205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 msgr2=0x5579693abb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.956+0000 7f6099037740 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 0x5579693abb10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6084012040 tx=0x7f608400b940 comp rx=0 tx=0).stop 2024-03-18T00:55:05.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.957+0000 7f6099037740 1 -- 172.21.15.49:0/2998320205 shutdown_connections 2024-03-18T00:55:05.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.957+0000 7f6099037740 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557969418bf0 0x55796941b010 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.957+0000 7f6099037740 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5579693a2190 0x5579693acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.957+0000 7f6099037740 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55796939d020 0x5579693abb10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.957+0000 7f6099037740 1 --2- 172.21.15.49:0/2998320205 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55796939c630 0x5579693ab5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.957+0000 7f606f7fe640 0 client.26134 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:05.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.957+0000 7f6099037740 1 -- 172.21.15.49:0/2998320205 >> 172.21.15.49:0/2998320205 conn(0x557969397ba0 msgr2=0x55796939ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:05.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.958+0000 7f6099037740 1 -- 172.21.15.49:0/2998320205 shutdown_connections 2024-03-18T00:55:05.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.958+0000 7f6099037740 1 -- 172.21.15.49:0/2998320205 wait complete. 2024-03-18T00:55:05.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.958+0000 7f6099037740 20 client.26134 trim_cache size 0 max 16384 2024-03-18T00:55:05.959 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:55:05.959 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-18T00:55:05.959 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:55:05.959 INFO:teuthology.orchestra.run.smithi049.stdout: "446b4af4-e182-4cb3-82b3-4db0a4e72548": { 2024-03-18T00:55:05.959 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:55:05.959 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "446b4af4-e182-4cb3-82b3-4db0a4e72548", 2024-03-18T00:55:05.960 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:55:05.960 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:55:05.960 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:55:05.960 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 -- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60780a2980 msgr2=0x7f60780a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60780a2980 0x7f60780a4e40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f608002c260 tx=0x7f6080005fd0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 -- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090109870 msgr2=0x7f6090118040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090109870 0x7f6090118040 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f608800eaa0 tx=0x7f608800ef60 comp rx=0 tx=0).stop 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 -- 172.21.15.49:0/805689514 shutdown_connections 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f60780a2980 0x7f60780a4e40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6090108eb0 0x7f6090119f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6090109870 0x7f6090118040 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 --2- 172.21.15.49:0/805689514 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6090072bf0 0x7f6090119a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.959+0000 7f6076ffd640 1 -- 172.21.15.49:0/805689514 >> 172.21.15.49:0/805689514 conn(0x7f609006e6c0 msgr2=0x7f609010c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:05.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.960+0000 7f6076ffd640 1 -- 172.21.15.49:0/805689514 shutdown_connections 2024-03-18T00:55:05.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:05.960+0000 7f6076ffd640 1 -- 172.21.15.49:0/805689514 wait complete. 2024-03-18T00:55:05.995 INFO:tasks.cephfs.filesystem:scrub status for tag:446b4af4-e182-4cb3-82b3-4db0a4e72548 - {'path': '/', 'tag': '446b4af4-e182-4cb3-82b3-4db0a4e72548', 'options': 'recursive,force'} 2024-03-18T00:55:07.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:07 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:55:07.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:07 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:07.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:07 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:55:07.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:07 smithi049 ceph-mon[32130]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 869 MiB/s wr, 2.05k op/s 2024-03-18T00:55:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:07 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:55:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:07 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:07 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:55:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:07 smithi107 ceph-mon[36270]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 869 MiB/s wr, 2.05k op/s 2024-03-18T00:55:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:07 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:55:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:07 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:07.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:07 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:55:07.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:07 smithi106 ceph-mon[35899]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 869 MiB/s wr, 2.05k op/s 2024-03-18T00:55:09.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:09 smithi049 ceph-mon[32130]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 861 MiB/s wr, 2.07k op/s 2024-03-18T00:55:09.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:09 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:55:09.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:09 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:09.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:09 smithi107 ceph-mon[36270]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 861 MiB/s wr, 2.07k op/s 2024-03-18T00:55:09.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:09 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:55:09.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:09 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:09.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:09 smithi106 ceph-mon[35899]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 861 MiB/s wr, 2.07k op/s 2024-03-18T00:55:09.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:09 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:55:09.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:09 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:11.383 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:11 smithi049 ceph-mon[32130]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 955 MiB/s wr, 2.21k op/s 2024-03-18T00:55:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:11 smithi107 ceph-mon[36270]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 955 MiB/s wr, 2.21k op/s 2024-03-18T00:55:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:11 smithi106 ceph-mon[35899]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 955 MiB/s wr, 2.21k op/s 2024-03-18T00:55:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.737+0000 7fccbe3f3640 1 -- 172.21.15.49:0/1705720264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb810db00 msgr2=0x7fccb810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.737+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/1705720264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb810db00 0x7fccb810df80 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fccac009900 tx=0x7fccac02bc80 comp rx=0 tx=0).stop 2024-03-18T00:55:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.737+0000 7fccbe3f3640 1 -- 172.21.15.49:0/1705720264 shutdown_connections 2024-03-18T00:55:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.737+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/1705720264 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb810db00 0x7fccb810df80 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.737+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/1705720264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb810b150 0x7fccb810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.737+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/1705720264 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccb81087e0 0x7fccb810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.737+0000 7fccbe3f3640 1 -- 172.21.15.49:0/1705720264 >> 172.21.15.49:0/1705720264 conn(0x7fccb806e820 msgr2=0x7fccb8070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.740+0000 7fccbe3f3640 1 -- 172.21.15.49:0/1705720264 shutdown_connections 2024-03-18T00:55:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.741+0000 7fccbe3f3640 1 -- 172.21.15.49:0/1705720264 wait complete. 2024-03-18T00:55:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.742+0000 7fccbe3f3640 1 Processor -- start 2024-03-18T00:55:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.742+0000 7fccbe3f3640 1 -- start start 2024-03-18T00:55:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.743+0000 7fccbe3f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb81087e0 0x7fccb8118170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.743+0000 7fccbe3f3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccb810b150 0x7fccb81186b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.743+0000 7fccbe3f3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb810db00 0x7fccb81196f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.743+0000 7fccbe3f3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fccb8110920 con 0x7fccb810db00 2024-03-18T00:55:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.743+0000 7fccbe3f3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fccb81107a0 con 0x7fccb810b150 2024-03-18T00:55:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.743+0000 7fccbe3f3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fccb8110aa0 con 0x7fccb81087e0 2024-03-18T00:55:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.744+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb81087e0 0x7fccb8118170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.744+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb81087e0 0x7fccb8118170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46702/0 (socket says 172.21.15.49:46702) 2024-03-18T00:55:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.744+0000 7fccb7fff640 1 -- 172.21.15.49:0/735853724 learned_addr learned my addr 172.21.15.49:0/735853724 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.744+0000 7fccbc990640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb810db00 0x7fccb81196f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.745+0000 7fccb7fff640 1 -- 172.21.15.49:0/735853724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccb810b150 msgr2=0x7fccb81186b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:55:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.745+0000 7fccb7fff640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccb810b150 0x7fccb81186b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.745+0000 7fccb7fff640 1 -- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb810db00 msgr2=0x7fccb81196f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.745+0000 7fccb7fff640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb810db00 0x7fccb81196f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.745+0000 7fccb7fff640 1 -- 172.21.15.49:0/735853724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccb8119d60 con 0x7fccb81087e0 2024-03-18T00:55:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.745+0000 7fccbc990640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb810db00 0x7fccb81196f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:55:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.748+0000 7fccb7fff640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb81087e0 0x7fccb8118170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcca800c890 tx=0x7fcca800cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:11.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.754+0000 7fccb57fa640 1 -- 172.21.15.49:0/735853724 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca8004220 con 0x7fccb81087e0 2024-03-18T00:55:11.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.754+0000 7fccb57fa640 1 -- 172.21.15.49:0/735853724 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcca80043c0 con 0x7fccb81087e0 2024-03-18T00:55:11.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.754+0000 7fccb57fa640 1 -- 172.21.15.49:0/735853724 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca80049e0 con 0x7fccb81087e0 2024-03-18T00:55:11.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.754+0000 7fccbe3f3640 1 -- 172.21.15.49:0/735853724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccb811a050 con 0x7fccb81087e0 2024-03-18T00:55:11.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.755+0000 7fccbe3f3640 1 -- 172.21.15.49:0/735853724 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccb81b75c0 con 0x7fccb81087e0 2024-03-18T00:55:11.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.759+0000 7fccc0fda740 1 Processor -- start 2024-03-18T00:55:11.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.759+0000 7fccc0fda740 1 -- start start 2024-03-18T00:55:11.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.760+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a64340 0x555707a64760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.760+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a69830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.760+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a69ea0 0x555707a6c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.760+0000 7fccc0fda740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5557079c9a40 con 0x555707a64d30 2024-03-18T00:55:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.760+0000 7fccc0fda740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555707972920 con 0x555707a69ea0 2024-03-18T00:55:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.760+0000 7fccc0fda740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5557079c5ae0 con 0x555707a64340 2024-03-18T00:55:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.761+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a69830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.761+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a69830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33272/0 (socket says 172.21.15.49:33272) 2024-03-18T00:55:11.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.761+0000 7fccb7fff640 1 -- 172.21.15.49:0/4176888414 learned_addr learned my addr 172.21.15.49:0/4176888414 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.764+0000 7fccb57fa640 1 -- 172.21.15.49:0/735853724 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcca80079b0 con 0x7fccb81087e0 2024-03-18T00:55:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.764+0000 7fccb7fff640 1 -- 172.21.15.49:0/4176888414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a64340 msgr2=0x555707a64760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.764+0000 7fccb7fff640 1 --2- 172.21.15.49:0/4176888414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a64340 0x555707a64760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.764+0000 7fccb7fff640 1 -- 172.21.15.49:0/4176888414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a69ea0 msgr2=0x555707a6c330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:11.771 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.764+0000 7fccb7fff640 1 --2- 172.21.15.49:0/4176888414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a69ea0 0x555707a6c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.764+0000 7fccb7fff640 1 -- 172.21.15.49:0/4176888414 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55570798a460 con 0x555707a64d30 2024-03-18T00:55:11.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.766+0000 7fccb57fa640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcca00a2a60 0x7fcca00a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.766+0000 7fccb57fa640 1 -- 172.21.15.49:0/735853724 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcca80d5910 con 0x7fccb81087e0 2024-03-18T00:55:11.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.771+0000 7fccbc990640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcca00a2a60 0x7fcca00a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.772+0000 7fccb7fff640 1 --2- 172.21.15.49:0/4176888414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a69830 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fcca80d33c0 tx=0x7fcca80d6550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5080d60932c8635 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:11.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.772+0000 7fccbc990640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcca00a2a60 0x7fcca00a4f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fccac005ab0 tx=0x7fccac004c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.775+0000 7fcc9e7fc640 1 -- 172.21.15.49:0/4176888414 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca809e4a0 con 0x555707a64d30 2024-03-18T00:55:11.775 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.776+0000 7fcc9e7fc640 1 -- 172.21.15.49:0/4176888414 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcca80a2990 con 0x555707a64d30 2024-03-18T00:55:11.776 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.776+0000 7fcc9e7fc640 1 -- 172.21.15.49:0/4176888414 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca80a03d0 con 0x555707a64d30 2024-03-18T00:55:11.778 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.778+0000 7fccc0fda740 1 -- 172.21.15.49:0/4176888414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 msgr2=0x555707a69830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.778+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4176888414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a69830 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fcca80d33c0 tx=0x7fcca80d6550 comp rx=0 tx=0).stop 2024-03-18T00:55:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.778+0000 7fccc0fda740 1 -- 172.21.15.49:0/4176888414 shutdown_connections 2024-03-18T00:55:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.779+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4176888414 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a69ea0 0x555707a6c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.779+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4176888414 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a69830 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.779+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4176888414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a64340 0x555707a64760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.779+0000 7fccc0fda740 1 -- 172.21.15.49:0/4176888414 >> 172.21.15.49:0/4176888414 conn(0x555707a5f8b0 msgr2=0x555707a5fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.785+0000 7fccc0fda740 1 -- 172.21.15.49:0/4176888414 shutdown_connections 2024-03-18T00:55:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.785+0000 7fccc0fda740 1 -- 172.21.15.49:0/4176888414 wait complete. 2024-03-18T00:55:11.785 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.786+0000 7fccc0fda740 1 -- start start 2024-03-18T00:55:11.786 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.786+0000 7fccc0fda740 1 Processor -- start 2024-03-18T00:55:11.786 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.787+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64340 0x555707a6e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.787 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.787+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a64d30 0x555707a6ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.787 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.787+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 0x555707a75c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.788+0000 7fccc0fda740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555707a6ce50 con 0x555707a64340 2024-03-18T00:55:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.788+0000 7fccc0fda740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555707985d70 con 0x555707a64d30 2024-03-18T00:55:11.788 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.789+0000 7fccc0fda740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555707a6cfd0 con 0x555707a69ea0 2024-03-18T00:55:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.789+0000 7fccc0fda740 10 client.0 _ll_register_callbacks cb 0x5557076dda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:11.789 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.790+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64340 0x555707a6e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.790+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64340 0x555707a6e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:33276/0 (socket says 172.21.15.49:33276) 2024-03-18T00:55:11.790 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.790+0000 7fccb7fff640 1 -- 172.21.15.49:0/3267423844 learned_addr learned my addr 172.21.15.49:0/3267423844 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.791+0000 7fccb7fff640 1 -- 172.21.15.49:0/3267423844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 msgr2=0x555707a75c30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.791+0000 7fccb7fff640 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 0x555707a75c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.791+0000 7fccb7fff640 1 -- 172.21.15.49:0/3267423844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a64d30 msgr2=0x555707a6ebf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.791+0000 7fccb7fff640 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a64d30 0x555707a6ebf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.791+0000 7fccb7fff640 1 -- 172.21.15.49:0/3267423844 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557076de210 con 0x555707a64340 2024-03-18T00:55:11.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.793+0000 7fccb7fff640 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64340 0x555707a6e6b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fcca80a2450 tx=0x7fcca8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70f15aa22750a1a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.801+0000 7fcc977fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:11.801 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.801+0000 7fcc977fe640 1 -- 172.21.15.49:0/3267423844 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca80a1bf0 con 0x555707a64340 2024-03-18T00:55:11.802 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.801+0000 7fcc977fe640 1 -- 172.21.15.49:0/3267423844 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcca809e870 con 0x555707a64340 2024-03-18T00:55:11.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.804+0000 7fccc0fda740 1 -- 172.21.15.49:0/3267423844 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcca80a2990 con 0x555707a64340 2024-03-18T00:55:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.810+0000 7fcc977fe640 1 -- 172.21.15.49:0/3267423844 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcca80a1dd0 con 0x555707a64340 2024-03-18T00:55:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.810+0000 7fccc0fda740 10 client.16668 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:11.810 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.810+0000 7fccc0fda740 1 -- 172.21.15.49:0/3267423844 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5557077aa620 con 0x555707a64340 2024-03-18T00:55:11.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.820+0000 7fcc977fe640 1 -- 172.21.15.49:0/3267423844 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcca80a2990 con 0x555707a64340 2024-03-18T00:55:11.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.822+0000 7fcc977fe640 1 -- 172.21.15.49:0/3267423844 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcca80a24f0 con 0x555707a64340 2024-03-18T00:55:11.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.822+0000 7fccc0fda740 10 client.16668 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:11.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.822+0000 7fccc0fda740 10 client.16668 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:11.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.822+0000 7fccc0fda740 20 client.16668 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:11.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.822+0000 7fccc0fda740 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555707a7a120 0x555707a7c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.822+0000 7fccc0fda740 4 client.16668 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:11.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.822+0000 7fccc0fda740 1 -- 172.21.15.49:0/3267423844 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557079d6f80 con 0x555707a7a120 2024-03-18T00:55:11.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.824+0000 7fccb77fe640 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555707a7a120 0x555707a7c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.828+0000 7fccb77fe640 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555707a7a120 0x555707a7c510 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=661dd5ce4909e889 server_cookie=d43b84c84e2a65a5 in_seq=0 out_seq=0 2024-03-18T00:55:11.828 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.829+0000 7fcc977fe640 10 client.16668 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.833+0000 7fcc977fe640 1 -- 172.21.15.49:0/3267423844 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5557079d6f80 con 0x555707a7a120 2024-03-18T00:55:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.833+0000 7fcc977fe640 10 client.16668 handle_command_reply: tid=0 2024-03-18T00:55:11.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.833+0000 7fccc0fda740 1 client.16668 shutdown 2024-03-18T00:55:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.834+0000 7fccc0fda740 1 -- 172.21.15.49:0/3267423844 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64340 msgr2=0x555707a6e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.834+0000 7fccc0fda740 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64340 0x555707a6e6b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fcca80a2450 tx=0x7fcca8000bc0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.834+0000 7fccc0fda740 1 -- 172.21.15.49:0/3267423844 shutdown_connections 2024-03-18T00:55:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.834+0000 7fccc0fda740 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555707a7a120 0x555707a7c510 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.834+0000 7fccc0fda740 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a64d30 0x555707a6ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.834+0000 7fccc0fda740 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 0x555707a75c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.834+0000 7fccc0fda740 1 --2- 172.21.15.49:0/3267423844 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64340 0x555707a6e6b0 secure :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fcca80a2450 tx=0x7fcca8000bc0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.834+0000 7fccc0fda740 1 -- 172.21.15.49:0/3267423844 >> 172.21.15.49:0/3267423844 conn(0x555707a5f8b0 msgr2=0x555707a5ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:11.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.834+0000 7fcc977fe640 0 client.16668 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.837+0000 7fccc0fda740 1 -- 172.21.15.49:0/3267423844 shutdown_connections 2024-03-18T00:55:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.837+0000 7fccc0fda740 1 -- 172.21.15.49:0/3267423844 wait complete. 2024-03-18T00:55:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.837+0000 7fccc0fda740 20 client.16668 trim_cache size 0 max 16384 2024-03-18T00:55:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.883+0000 7fccc0fda740 1 Processor -- start 2024-03-18T00:55:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.883+0000 7fccc0fda740 1 -- start start 2024-03-18T00:55:11.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.884+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a5f8b0 0x555707a63220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.884+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a75bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.884+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a69ea0 0x555707a5aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.884+0000 7fccc0fda740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fccb81c0450 con 0x555707a64d30 2024-03-18T00:55:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.884+0000 7fccc0fda740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fccb81c1f90 con 0x555707a69ea0 2024-03-18T00:55:11.887 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.884+0000 7fccc0fda740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fccb81c0b20 con 0x555707a5f8b0 2024-03-18T00:55:11.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.887+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a5f8b0 0x555707a63220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.887+0000 7fccb7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a5f8b0 0x555707a63220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46716/0 (socket says 172.21.15.49:46716) 2024-03-18T00:55:11.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.887+0000 7fccb7fff640 1 -- 172.21.15.49:0/223588988 learned_addr learned my addr 172.21.15.49:0/223588988 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:11.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.888+0000 7fccbc990640 1 --2- 172.21.15.49:0/223588988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a69ea0 0x555707a5aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.890+0000 7fccb7fff640 1 -- 172.21.15.49:0/223588988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a69ea0 msgr2=0x555707a5aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:11.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.890+0000 7fccb7fff640 1 --2- 172.21.15.49:0/223588988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a69ea0 0x555707a5aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.890+0000 7fccb7fff640 1 -- 172.21.15.49:0/223588988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 msgr2=0x555707a75bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.890+0000 7fccb7fff640 1 --2- 172.21.15.49:0/223588988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a75bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.892 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.890+0000 7fccb7fff640 1 -- 172.21.15.49:0/223588988 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555707a60620 con 0x555707a5f8b0 2024-03-18T00:55:11.896 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.895+0000 7fccb7fff640 1 --2- 172.21.15.49:0/223588988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a5f8b0 0x555707a63220 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcca8007590 tx=0x7fcca8005300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bda652776e49ccc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:11.907 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.901+0000 7fcc9dffb640 1 -- 172.21.15.49:0/223588988 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca80a2060 con 0x555707a5f8b0 2024-03-18T00:55:11.907 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.902+0000 7fcc9dffb640 1 -- 172.21.15.49:0/223588988 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcca80a2290 con 0x555707a5f8b0 2024-03-18T00:55:11.907 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.902+0000 7fcc9dffb640 1 -- 172.21.15.49:0/223588988 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca809e870 con 0x555707a5f8b0 2024-03-18T00:55:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.908+0000 7fccc0fda740 1 -- 172.21.15.49:0/223588988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a5f8b0 msgr2=0x555707a63220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.908+0000 7fccc0fda740 1 --2- 172.21.15.49:0/223588988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a5f8b0 0x555707a63220 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcca8007590 tx=0x7fcca8005300 comp rx=0 tx=0).stop 2024-03-18T00:55:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.908+0000 7fccc0fda740 1 -- 172.21.15.49:0/223588988 shutdown_connections 2024-03-18T00:55:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.908+0000 7fccc0fda740 1 --2- 172.21.15.49:0/223588988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a69ea0 0x555707a5aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.908+0000 7fccc0fda740 1 --2- 172.21.15.49:0/223588988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a75bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.908+0000 7fccc0fda740 1 --2- 172.21.15.49:0/223588988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a5f8b0 0x555707a63220 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.908+0000 7fccc0fda740 1 -- 172.21.15.49:0/223588988 >> 172.21.15.49:0/223588988 conn(0x555707a69a50 msgr2=0x555707a62b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.913+0000 7fccc0fda740 1 -- 172.21.15.49:0/223588988 shutdown_connections 2024-03-18T00:55:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.913+0000 7fccc0fda740 1 -- 172.21.15.49:0/223588988 wait complete. 2024-03-18T00:55:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.914+0000 7fccc0fda740 1 -- start start 2024-03-18T00:55:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.914+0000 7fccc0fda740 1 Processor -- start 2024-03-18T00:55:11.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.915+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a5f8b0 0x555707a736a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.915+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a73be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.915+0000 7fccc0fda740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 0x555707a74bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.915+0000 7fccc0fda740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fccb81c18c0 con 0x555707a64d30 2024-03-18T00:55:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.915+0000 7fccc0fda740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fccb81c11f0 con 0x555707a5f8b0 2024-03-18T00:55:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.915+0000 7fccc0fda740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fccb81be430 con 0x555707a69ea0 2024-03-18T00:55:11.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.915+0000 7fccc0fda740 10 client.0 _ll_register_callbacks cb 0x7fcca80d5ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.918+0000 7fccbc990640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 0x555707a74bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.918+0000 7fccbc990640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 0x555707a74bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46732/0 (socket says 172.21.15.49:46732) 2024-03-18T00:55:11.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.918+0000 7fccbc990640 1 -- 172.21.15.49:0/4281655765 learned_addr learned my addr 172.21.15.49:0/4281655765 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:11.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.919+0000 7fccb7fff640 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a5f8b0 0x555707a736a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.924+0000 7fccb7fff640 1 -- 172.21.15.49:0/4281655765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 msgr2=0x555707a74bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.924+0000 7fccb7fff640 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 0x555707a74bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.924+0000 7fccb7fff640 1 -- 172.21.15.49:0/4281655765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 msgr2=0x555707a73be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.924+0000 7fccb7fff640 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a73be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:11.925 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.924+0000 7fccb7fff640 1 -- 172.21.15.49:0/4281655765 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555707a751e0 con 0x555707a5f8b0 2024-03-18T00:55:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.931+0000 7fccb7fff640 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a5f8b0 0x555707a736a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcca8007590 tx=0x7fcca8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a24c7d040e284c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.932+0000 7fcc977fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:55:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.933+0000 7fcc977fe640 1 -- 172.21.15.49:0/4281655765 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcca80a2060 con 0x555707a5f8b0 2024-03-18T00:55:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.933+0000 7fcc977fe640 1 -- 172.21.15.49:0/4281655765 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcca80aa640 con 0x555707a5f8b0 2024-03-18T00:55:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.933+0000 7fccc0fda740 1 -- 172.21.15.49:0/4281655765 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5557079d12d0 con 0x555707a5f8b0 2024-03-18T00:55:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.938+0000 7fcc977fe640 1 -- 172.21.15.49:0/4281655765 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcca80a2990 con 0x555707a5f8b0 2024-03-18T00:55:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.941+0000 7fccc0fda740 10 client.26054 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.941+0000 7fccc0fda740 1 -- 172.21.15.49:0/4281655765 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555707a751e0 con 0x555707a5f8b0 2024-03-18T00:55:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.946+0000 7fcc977fe640 1 -- 172.21.15.49:0/4281655765 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5557079d12d0 con 0x555707a5f8b0 2024-03-18T00:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.946+0000 7fcc977fe640 1 -- 172.21.15.49:0/4281655765 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcca80a1150 con 0x555707a5f8b0 2024-03-18T00:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.946+0000 7fccc0fda740 10 client.26054 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.946+0000 7fccc0fda740 10 client.26054 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.946+0000 7fccc0fda740 20 client.26054 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.946+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555707ae0aa0 0x555707ae2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.946+0000 7fccc0fda740 4 client.26054 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:55:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.946+0000 7fccc0fda740 1 -- 172.21.15.49:0/4281655765 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55570794f770 con 0x555707ae0aa0 2024-03-18T00:55:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.951+0000 7fccb77fe640 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555707ae0aa0 0x555707ae2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:11.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.960+0000 7fcc977fe640 10 client.26054 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:11.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:11.960+0000 7fccb77fe640 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555707ae0aa0 0x555707ae2e90 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4aa6640f394dec81 server_cookie=c48fe46ff380b27d in_seq=0 out_seq=0 2024-03-18T00:55:13.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:13 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:13.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:13 smithi049 ceph-mon[32130]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 921 MiB/s wr, 1.75k op/s 2024-03-18T00:55:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:13 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:13 smithi107 ceph-mon[36270]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 921 MiB/s wr, 1.75k op/s 2024-03-18T00:55:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:13 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:13 smithi106 ceph-mon[35899]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 921 MiB/s wr, 1.75k op/s 2024-03-18T00:55:15.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:15 smithi049 ceph-mon[32130]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 936 MiB/s wr, 1.67k op/s 2024-03-18T00:55:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:15 smithi107 ceph-mon[36270]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 936 MiB/s wr, 1.67k op/s 2024-03-18T00:55:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:15 smithi106 ceph-mon[35899]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 936 MiB/s wr, 1.67k op/s 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.025+0000 7fcc977fe640 1 -- 172.21.15.49:0/4281655765 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55570794f770 con 0x555707ae0aa0 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.025+0000 7fcc977fe640 10 client.26054 handle_command_reply: tid=0 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.025+0000 7fccc0fda740 1 client.26054 shutdown 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 -- 172.21.15.49:0/4281655765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a5f8b0 msgr2=0x555707a736a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a5f8b0 0x555707a736a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcca8007590 tx=0x7fcca8000bc0 comp rx=0 tx=0).stop 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 -- 172.21.15.49:0/4281655765 shutdown_connections 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555707ae0aa0 0x555707ae2e90 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555707a69ea0 0x555707a74bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555707a64d30 0x555707a73be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:16.026 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fcc977fe640 0 client.26054 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 --2- 172.21.15.49:0/4281655765 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555707a5f8b0 0x555707a736a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 -- 172.21.15.49:0/4281655765 >> 172.21.15.49:0/4281655765 conn(0x555707a69a50 msgr2=0x555707a6e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 -- 172.21.15.49:0/4281655765 shutdown_connections 2024-03-18T00:55:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 1 -- 172.21.15.49:0/4281655765 wait complete. 2024-03-18T00:55:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:55:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:55:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:55:16.027 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:55:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.026+0000 7fccc0fda740 20 client.26054 trim_cache size 0 max 16384 2024-03-18T00:55:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.027+0000 7fccbe3f3640 1 -- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcca00a2a60 msgr2=0x7fcca00a4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:16.028 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.027+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcca00a2a60 0x7fcca00a4f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fccac005ab0 tx=0x7fccac004c80 comp rx=0 tx=0).stop 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.027+0000 7fccbe3f3640 1 -- 172.21.15.49:0/735853724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb81087e0 msgr2=0x7fccb8118170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.027+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb81087e0 0x7fccb8118170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcca800c890 tx=0x7fcca800cd50 comp rx=0 tx=0).stop 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.027+0000 7fccbe3f3640 1 -- 172.21.15.49:0/735853724 shutdown_connections 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.027+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcca00a2a60 0x7fcca00a4f20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.027+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccb810b150 0x7fccb81186b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.027+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccb810db00 0x7fccb81196f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.027+0000 7fccbe3f3640 1 --2- 172.21.15.49:0/735853724 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccb81087e0 0x7fccb8118170 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.028+0000 7fccbe3f3640 1 -- 172.21.15.49:0/735853724 >> 172.21.15.49:0/735853724 conn(0x7fccb806e820 msgr2=0x7fccb810b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.028+0000 7fccbe3f3640 1 -- 172.21.15.49:0/735853724 shutdown_connections 2024-03-18T00:55:16.029 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:16.028+0000 7fccbe3f3640 1 -- 172.21.15.49:0/735853724 wait complete. 2024-03-18T00:55:16.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:16 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:55:16.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:16 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:55:16.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:16 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:55:16.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:16 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:55:16.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:16 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:55:16.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:16 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:55:17.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:17 smithi049 ceph-mon[32130]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 884 MiB/s wr, 1.68k op/s 2024-03-18T00:55:17.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:17 smithi107 ceph-mon[36270]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 884 MiB/s wr, 1.68k op/s 2024-03-18T00:55:17.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:17 smithi106 ceph-mon[35899]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 884 MiB/s wr, 1.68k op/s 2024-03-18T00:55:19.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:19 smithi049 ceph-mon[32130]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 877 MiB/s wr, 1.72k op/s 2024-03-18T00:55:19.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:19 smithi107 ceph-mon[36270]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 877 MiB/s wr, 1.72k op/s 2024-03-18T00:55:19.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:19 smithi106 ceph-mon[35899]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 877 MiB/s wr, 1.72k op/s 2024-03-18T00:55:21.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:21 smithi049 ceph-mon[32130]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 954 MiB/s wr, 1.78k op/s 2024-03-18T00:55:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:21 smithi107 ceph-mon[36270]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 954 MiB/s wr, 1.78k op/s 2024-03-18T00:55:21.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:21 smithi106 ceph-mon[35899]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 954 MiB/s wr, 1.78k op/s 2024-03-18T00:55:23.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:23 smithi049 ceph-mon[32130]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 908 MiB/s wr, 1.76k op/s 2024-03-18T00:55:23.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:23 smithi107 ceph-mon[36270]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 908 MiB/s wr, 1.76k op/s 2024-03-18T00:55:23.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:23 smithi106 ceph-mon[35899]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 908 MiB/s wr, 1.76k op/s 2024-03-18T00:55:25.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:25 smithi107 ceph-mon[36270]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 931 MiB/s wr, 1.92k op/s 2024-03-18T00:55:25.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:25 smithi106 ceph-mon[35899]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 931 MiB/s wr, 1.92k op/s 2024-03-18T00:55:25.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:25 smithi049 ceph-mon[32130]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 931 MiB/s wr, 1.92k op/s 2024-03-18T00:55:27.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:27 smithi107 ceph-mon[36270]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 881 MiB/s wr, 1.75k op/s 2024-03-18T00:55:27.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:27 smithi106 ceph-mon[35899]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 881 MiB/s wr, 1.75k op/s 2024-03-18T00:55:27.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:27 smithi049 ceph-mon[32130]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 881 MiB/s wr, 1.75k op/s 2024-03-18T00:55:29.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.231+0000 7fd3f369a640 1 -- 172.21.15.49:0/1273134185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec111f20 msgr2=0x7fd3ec114330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:29.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.231+0000 7fd3f369a640 1 --2- 172.21.15.49:0/1273134185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec111f20 0x7fd3ec114330 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009ca0 tx=0x7fd3e402c0f0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.233+0000 7fd3f369a640 1 -- 172.21.15.49:0/1273134185 shutdown_connections 2024-03-18T00:55:29.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.233+0000 7fd3f369a640 1 --2- 172.21.15.49:0/1273134185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3ec10b150 0x7fd3ec10b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.233+0000 7fd3f369a640 1 --2- 172.21.15.49:0/1273134185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec111f20 0x7fd3ec114330 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.233+0000 7fd3f369a640 1 --2- 172.21.15.49:0/1273134185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ec1087e0 0x7fd3ec10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.233+0000 7fd3f369a640 1 -- 172.21.15.49:0/1273134185 >> 172.21.15.49:0/1273134185 conn(0x7fd3ec06e820 msgr2=0x7fd3ec070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:29.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.234+0000 7fd3f369a640 1 -- 172.21.15.49:0/1273134185 shutdown_connections 2024-03-18T00:55:29.235 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.235+0000 7fd3f369a640 1 -- 172.21.15.49:0/1273134185 wait complete. 2024-03-18T00:55:29.236 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.236+0000 7fd3f369a640 1 Processor -- start 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.236+0000 7fd3f369a640 1 -- start start 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f369a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3ec1087e0 0x7fd3ec1a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f369a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec10b150 0x7fd3ec1a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f369a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ec111f20 0x7fd3ec1ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f369a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec117f50 con 0x7fd3ec10b150 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f369a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec117dd0 con 0x7fd3ec1087e0 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f369a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1180d0 con 0x7fd3ec111f20 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f0c35640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec10b150 0x7fd3ec1a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f0c35640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec10b150 0x7fd3ec1a6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53796/0 (socket says 172.21.15.49:53796) 2024-03-18T00:55:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f0c35640 1 -- 172.21.15.49:0/3838884880 learned_addr learned my addr 172.21.15.49:0/3838884880 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.237+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ec111f20 0x7fd3ec1ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.238+0000 7fd3f0c35640 1 -- 172.21.15.49:0/3838884880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ec111f20 msgr2=0x7fd3ec1ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.238+0000 7fd3f0c35640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ec111f20 0x7fd3ec1ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.238+0000 7fd3f0c35640 1 -- 172.21.15.49:0/3838884880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3ec1087e0 msgr2=0x7fd3ec1a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.238+0000 7fd3f0c35640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3ec1087e0 0x7fd3ec1a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.238+0000 7fd3f0c35640 1 -- 172.21.15.49:0/3838884880 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ec1ae500 con 0x7fd3ec10b150 2024-03-18T00:55:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.238+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ec111f20 0x7fd3ec1ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:55:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.238+0000 7fd3f0c35640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec10b150 0x7fd3ec1a6e20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800b4a0 tx=0x7fd3e800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.239+0000 7fd3e27fc640 1 -- 172.21.15.49:0/3838884880 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8014070 con 0x7fd3ec10b150 2024-03-18T00:55:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.239+0000 7fd3e27fc640 1 -- 172.21.15.49:0/3838884880 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e8010af0 con 0x7fd3ec10b150 2024-03-18T00:55:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.239+0000 7fd3e27fc640 1 -- 172.21.15.49:0/3838884880 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8017d10 con 0x7fd3ec10b150 2024-03-18T00:55:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.239+0000 7fd3f369a640 1 -- 172.21.15.49:0/3838884880 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ec1ae7f0 con 0x7fd3ec10b150 2024-03-18T00:55:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.239+0000 7fd3f369a640 1 -- 172.21.15.49:0/3838884880 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ec1aed70 con 0x7fd3ec10b150 2024-03-18T00:55:29.248 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.249+0000 7fd3f6367740 1 Processor -- start 2024-03-18T00:55:29.249 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.249+0000 7fd3e27fc640 1 -- 172.21.15.49:0/3838884880 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd3e8017070 con 0x7fd3ec10b150 2024-03-18T00:55:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.249+0000 7fd3f6367740 1 -- start start 2024-03-18T00:55:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.250+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c65340 0x5569f5c65760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.250+0000 7fd3f1c37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c65340 0x5569f5c65760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.250+0000 7fd3f1c37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c65340 0x5569f5c65760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53808/0 (socket says 172.21.15.49:53808) 2024-03-18T00:55:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.250+0000 7fd3f1c37640 1 -- 172.21.15.49:0/647561670 learned_addr learned my addr 172.21.15.49:0/647561670 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.250+0000 7fd3f6367740 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c6a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.251 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.251+0000 7fd3e27fc640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3d80a2a10 0x7fd3d80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.252+0000 7fd3e27fc640 1 -- 172.21.15.49:0/3838884880 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd3e80cdd80 con 0x7fd3ec10b150 2024-03-18T00:55:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.252+0000 7fd3f0c35640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3d80a2a10 0x7fd3d80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.252+0000 7fd3f6367740 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c6aea0 0x5569f5c6d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.252+0000 7fd3f6367740 1 -- 172.21.15.49:0/647561670 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5569f5bcaa40 con 0x5569f5c65340 2024-03-18T00:55:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.252+0000 7fd3f6367740 1 -- 172.21.15.49:0/647561670 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5569f5b73920 con 0x5569f5c6aea0 2024-03-18T00:55:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.252+0000 7fd3f6367740 1 -- 172.21.15.49:0/647561670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5569f5bc6ae0 con 0x5569f5c65d30 2024-03-18T00:55:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.254+0000 7fd3f1436640 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c6a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.255+0000 7fd3f1436640 1 -- 172.21.15.49:0/647561670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c6aea0 msgr2=0x5569f5c6d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:55:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.255+0000 7fd3f1436640 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c6aea0 0x5569f5c6d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.255+0000 7fd3f1436640 1 -- 172.21.15.49:0/647561670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c65340 msgr2=0x5569f5c65760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.255+0000 7fd3f1436640 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c65340 0x5569f5c65760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.256+0000 7fd3f1436640 1 -- 172.21.15.49:0/647561670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569f5b8b460 con 0x5569f5c65d30 2024-03-18T00:55:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.257+0000 7fd3f1436640 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c6a830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009750 tx=0x7fd3dc009c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c543153bd944f087 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.258+0000 7fd3d77fe640 1 -- 172.21.15.49:0/647561670 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc015020 con 0x5569f5c65d30 2024-03-18T00:55:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.258+0000 7fd3f0c35640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3d80a2a10 0x7fd3d80a4ed0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80970d0 tx=0x7fd3e800bf90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.258+0000 7fd3d77fe640 1 -- 172.21.15.49:0/647561670 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc011420 con 0x5569f5c65d30 2024-03-18T00:55:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.258+0000 7fd3d77fe640 1 -- 172.21.15.49:0/647561670 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc009e60 con 0x5569f5c65d30 2024-03-18T00:55:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.260+0000 7fd3f6367740 1 -- 172.21.15.49:0/647561670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 msgr2=0x5569f5c6a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.260+0000 7fd3f6367740 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c6a830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc009750 tx=0x7fd3dc009c10 comp rx=0 tx=0).stop 2024-03-18T00:55:29.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.260+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c65340 0x5569f5c65760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.273+0000 7fd3f6367740 1 -- 172.21.15.49:0/647561670 shutdown_connections 2024-03-18T00:55:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.273+0000 7fd3f6367740 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c6aea0 0x5569f5c6d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.275 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.273+0000 7fd3f6367740 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c6a830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.273+0000 7fd3f6367740 1 --2- 172.21.15.49:0/647561670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c65340 0x5569f5c65760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.276 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.273+0000 7fd3f6367740 1 -- 172.21.15.49:0/647561670 >> 172.21.15.49:0/647561670 conn(0x5569f5c608b0 msgr2=0x5569f5c60cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:29.282 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.281+0000 7fd3f6367740 1 -- 172.21.15.49:0/647561670 shutdown_connections 2024-03-18T00:55:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.282+0000 7fd3f6367740 1 -- 172.21.15.49:0/647561670 wait complete. 2024-03-18T00:55:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.282+0000 7fd3f6367740 1 -- start start 2024-03-18T00:55:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.283+0000 7fd3f6367740 1 Processor -- start 2024-03-18T00:55:29.285 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.283+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c80780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.286 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.283+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c80cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.283+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c87c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.284+0000 7fd3f6367740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5569f5c6de50 con 0x5569f5c6aea0 2024-03-18T00:55:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.284+0000 7fd3f6367740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5569f5b86d70 con 0x5569f5c65340 2024-03-18T00:55:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.284+0000 7fd3f6367740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5569f5c6dfd0 con 0x5569f5c65d30 2024-03-18T00:55:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.285+0000 7fd3f6367740 10 client.0 _ll_register_callbacks cb 0x5569f58dea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.285+0000 7fd3f1c37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c87c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.285+0000 7fd3f1c37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c87c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53812/0 (socket says 172.21.15.49:53812) 2024-03-18T00:55:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.285+0000 7fd3f1c37640 1 -- 172.21.15.49:0/2826248674 learned_addr learned my addr 172.21.15.49:0/2826248674 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:29.287 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.287+0000 7fd3f1436640 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c80cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.292 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.288+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c80780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.292 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.288+0000 7fd3f1c37640 1 -- 172.21.15.49:0/2826248674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 msgr2=0x5569f5c80cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:29.292 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.288+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c80cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.288+0000 7fd3f1c37640 1 -- 172.21.15.49:0/2826248674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 msgr2=0x5569f5c80780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.288+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c80780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.288+0000 7fd3f1c37640 1 -- 172.21.15.49:0/2826248674 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569f58df210 con 0x5569f5c6aea0 2024-03-18T00:55:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.292+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c87c90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4037040 tx=0x7fd3e4009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b790d85dae996ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:29.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.292+0000 7fd3d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.295+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/2826248674 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4048060 con 0x5569f5c6aea0 2024-03-18T00:55:29.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.295+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/2826248674 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd3e4034640 con 0x5569f5c6aea0 2024-03-18T00:55:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.297+0000 7fd3f6367740 1 -- 172.21.15.49:0/2826248674 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3dc011420 con 0x5569f5c6aea0 2024-03-18T00:55:29.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.301+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/2826248674 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd3e4039040 con 0x5569f5c6aea0 2024-03-18T00:55:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.305+0000 7fd3f6367740 10 client.16677 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:29.308 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.305+0000 7fd3f6367740 1 -- 172.21.15.49:0/2826248674 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5569f59ab620 con 0x5569f5c6aea0 2024-03-18T00:55:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.310+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/2826248674 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc011420 con 0x5569f5c6aea0 2024-03-18T00:55:29.310 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.310+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/2826248674 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd3e403f040 con 0x5569f5c6aea0 2024-03-18T00:55:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.314+0000 7fd3f6367740 10 client.16677 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.314+0000 7fd3f6367740 10 client.16677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.314+0000 7fd3f6367740 20 client.16677 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.314+0000 7fd3f6367740 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5569f5c8c200 0x5569f5c8e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.314+0000 7fd3f6367740 4 client.16677 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:29.314 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.314+0000 7fd3f6367740 1 -- 172.21.15.49:0/2826248674 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569f5bd7f80 con 0x5569f5c8c200 2024-03-18T00:55:29.320 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.320+0000 7fd3f1436640 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5569f5c8c200 0x5569f5c8e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.344+0000 7fd3f1436640 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5569f5c8c200 0x5569f5c8e660 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87eae62cfdb1e6f8 server_cookie=fa03dac83e16b0b in_seq=0 out_seq=0 2024-03-18T00:55:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.344+0000 7fd3d4ff9640 10 client.16677 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:29.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.355+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/2826248674 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5569f5bd7f80 con 0x5569f5c8c200 2024-03-18T00:55:29.357 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.355+0000 7fd3d4ff9640 10 client.16677 handle_command_reply: tid=0 2024-03-18T00:55:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.357+0000 7fd3f6367740 1 client.16677 shutdown 2024-03-18T00:55:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.362+0000 7fd3f6367740 1 -- 172.21.15.49:0/2826248674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 msgr2=0x5569f5c87c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.362+0000 7fd3f6367740 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c87c90 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4037040 tx=0x7fd3e4009380 comp rx=0 tx=0).stop 2024-03-18T00:55:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.363+0000 7fd3f6367740 1 -- 172.21.15.49:0/2826248674 shutdown_connections 2024-03-18T00:55:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.364+0000 7fd3f6367740 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5569f5c8c200 0x5569f5c8e660 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.364+0000 7fd3f6367740 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c87c90 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.364+0000 7fd3f6367740 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c80cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.364+0000 7fd3f6367740 1 --2- 172.21.15.49:0/2826248674 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c80780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.364+0000 7fd3f6367740 1 -- 172.21.15.49:0/2826248674 >> 172.21.15.49:0/2826248674 conn(0x5569f5c608b0 msgr2=0x5569f5c5fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.364+0000 7fd3d4ff9640 0 client.16677 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.371+0000 7fd3f6367740 1 -- 172.21.15.49:0/2826248674 shutdown_connections 2024-03-18T00:55:29.379 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.378+0000 7fd3f6367740 1 -- 172.21.15.49:0/2826248674 wait complete. 2024-03-18T00:55:29.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.378+0000 7fd3f6367740 20 client.16677 trim_cache size 0 max 16384 2024-03-18T00:55:29.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.437+0000 7fd3f6367740 1 Processor -- start 2024-03-18T00:55:29.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:29 smithi106 ceph-mon[35899]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 863 MiB/s wr, 1.72k op/s 2024-03-18T00:55:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.448+0000 7fd3f6367740 1 -- start start 2024-03-18T00:55:29.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.449+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c6a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.449+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c87c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.449+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c67130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.450+0000 7fd3f6367740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1af4e0 con 0x5569f5c6aea0 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.450+0000 7fd3f6367740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1b1020 con 0x5569f5c65340 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.450+0000 7fd3f6367740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1afbb0 con 0x5569f5c65d30 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.450+0000 7fd3f1c37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c67130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.450+0000 7fd3f1c37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c67130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53824/0 (socket says 172.21.15.49:53824) 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.450+0000 7fd3f1c37640 1 -- 172.21.15.49:0/783836281 learned_addr learned my addr 172.21.15.49:0/783836281 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.450+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/783836281 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c6a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.450+0000 7fd3f1c37640 1 -- 172.21.15.49:0/783836281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 msgr2=0x5569f5c87c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.450+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/783836281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c87c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.452 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.451+0000 7fd3f1c37640 1 -- 172.21.15.49:0/783836281 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 msgr2=0x5569f5c6a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.451+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/783836281 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c6a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.451+0000 7fd3f1c37640 1 -- 172.21.15.49:0/783836281 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569f5c6dd30 con 0x5569f5c6aea0 2024-03-18T00:55:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.451+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/783836281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c67130 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40324c0 tx=0x7fd3e402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3199ec151a4db33f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:29.453 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.452+0000 7fd3d6ffd640 1 -- 172.21.15.49:0/783836281 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4048040 con 0x5569f5c6aea0 2024-03-18T00:55:29.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.452+0000 7fd3d6ffd640 1 -- 172.21.15.49:0/783836281 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e4032a10 con 0x5569f5c6aea0 2024-03-18T00:55:29.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.454+0000 7fd3f6367740 1 -- 172.21.15.49:0/783836281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 msgr2=0x5569f5c67130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:29.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.454+0000 7fd3f6367740 1 --2- 172.21.15.49:0/783836281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c67130 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40324c0 tx=0x7fd3e402c0f0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.455+0000 7fd3f6367740 1 -- 172.21.15.49:0/783836281 shutdown_connections 2024-03-18T00:55:29.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.455+0000 7fd3f6367740 1 --2- 172.21.15.49:0/783836281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c67130 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.455+0000 7fd3f6367740 1 --2- 172.21.15.49:0/783836281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c87c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.455+0000 7fd3f6367740 1 --2- 172.21.15.49:0/783836281 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c6a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.455+0000 7fd3f6367740 1 -- 172.21.15.49:0/783836281 >> 172.21.15.49:0/783836281 conn(0x5569f5c608b0 msgr2=0x5569f5c60c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:29.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.462+0000 7fd3f6367740 1 -- 172.21.15.49:0/783836281 shutdown_connections 2024-03-18T00:55:29.467 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.464+0000 7fd3f6367740 1 -- 172.21.15.49:0/783836281 wait complete. 2024-03-18T00:55:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.465+0000 7fd3f6367740 1 -- start start 2024-03-18T00:55:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.465+0000 7fd3f6367740 1 Processor -- start 2024-03-18T00:55:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.467+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c84f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.467+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c85450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.467+0000 7fd3f6367740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c86460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.467+0000 7fd3f6367740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1b0950 con 0x5569f5c6aea0 2024-03-18T00:55:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.467+0000 7fd3f6367740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1b0280 con 0x5569f5c65340 2024-03-18T00:55:29.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.467+0000 7fd3f6367740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ec1a42d0 con 0x5569f5c65d30 2024-03-18T00:55:29.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.467+0000 7fd3f6367740 10 client.0 _ll_register_callbacks cb 0x7fd3e4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:29.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.471+0000 7fd3f1436640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c85450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.471+0000 7fd3f1c37640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c84f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.471+0000 7fd3f1c37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c86460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.471+0000 7fd3f1c37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c86460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53834/0 (socket says 172.21.15.49:53834) 2024-03-18T00:55:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.471+0000 7fd3f1c37640 1 -- 172.21.15.49:0/3930523036 learned_addr learned my addr 172.21.15.49:0/3930523036 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.474+0000 7fd3f1c37640 1 -- 172.21.15.49:0/3930523036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 msgr2=0x5569f5c85450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.474+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c85450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.474+0000 7fd3f1c37640 1 -- 172.21.15.49:0/3930523036 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 msgr2=0x5569f5c84f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.474+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c84f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.474+0000 7fd3f1c37640 1 -- 172.21.15.49:0/3930523036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569f5c86a50 con 0x5569f5c6aea0 2024-03-18T00:55:29.475 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.475+0000 7fd3f1c37640 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c86460 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd3e402cd40 tx=0x7fd3e4043690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c347ae502101a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:29.476 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.475+0000 7fd3d4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:29.482 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.480+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/3930523036 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3e4048070 con 0x5569f5c6aea0 2024-03-18T00:55:29.482 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.480+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/3930523036 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd3e4041b70 con 0x5569f5c6aea0 2024-03-18T00:55:29.482 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.480+0000 7fd3f6367740 1 -- 172.21.15.49:0/3930523036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3e4032a10 con 0x5569f5c6aea0 2024-03-18T00:55:29.488 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.489+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/3930523036 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd3e4043480 con 0x5569f5c6aea0 2024-03-18T00:55:29.492 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.489+0000 7fd3f6367740 10 client.16683 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:29.492 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.489+0000 7fd3f6367740 1 -- 172.21.15.49:0/3930523036 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5569f5c87d60 con 0x5569f5c6aea0 2024-03-18T00:55:29.492 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.491+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/3930523036 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3e4032a10 con 0x5569f5c6aea0 2024-03-18T00:55:29.492 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.492+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/3930523036 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd3e403f070 con 0x5569f5c6aea0 2024-03-18T00:55:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.493+0000 7fd3f6367740 10 client.16683 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.493+0000 7fd3f6367740 10 client.16683 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.493+0000 7fd3f6367740 20 client.16683 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.493+0000 7fd3f6367740 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5569f5cf29f0 0x5569f5cf4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.493+0000 7fd3f6367740 4 client.16683 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:55:29.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.493+0000 7fd3f6367740 1 -- 172.21.15.49:0/3930523036 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5569f5bd22d0 con 0x5569f5cf29f0 2024-03-18T00:55:29.496 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.495+0000 7fd3f1436640 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5569f5cf29f0 0x5569f5cf4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.508+0000 7fd3d4ff9640 10 client.16683 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:29.508 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:29.508+0000 7fd3f1436640 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5569f5cf29f0 0x5569f5cf4e10 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c86ecc8ba2b5e0e8 server_cookie=aabe1e13a45b6877 in_seq=0 out_seq=0 2024-03-18T00:55:29.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:29 smithi049 ceph-mon[32130]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 863 MiB/s wr, 1.72k op/s 2024-03-18T00:55:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:29 smithi107 ceph-mon[36270]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 863 MiB/s wr, 1.72k op/s 2024-03-18T00:55:31.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:31 smithi106 ceph-mon[35899]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 949 MiB/s wr, 2.12k op/s 2024-03-18T00:55:31.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:31 smithi049 ceph-mon[32130]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 949 MiB/s wr, 2.12k op/s 2024-03-18T00:55:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:31 smithi107 ceph-mon[36270]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 949 MiB/s wr, 2.12k op/s 2024-03-18T00:55:33.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:33 smithi049 ceph-mon[32130]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 915 MiB/s wr, 2.36k op/s 2024-03-18T00:55:33.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:33 smithi107 ceph-mon[36270]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 915 MiB/s wr, 2.36k op/s 2024-03-18T00:55:33.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:33 smithi106 ceph-mon[35899]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 915 MiB/s wr, 2.36k op/s 2024-03-18T00:55:35.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:35 smithi049 ceph-mon[32130]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 894 MiB/s wr, 2.26k op/s 2024-03-18T00:55:35.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:35 smithi107 ceph-mon[36270]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 894 MiB/s wr, 2.26k op/s 2024-03-18T00:55:35.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:35 smithi106 ceph-mon[35899]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 894 MiB/s wr, 2.26k op/s 2024-03-18T00:55:35.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:55:35.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:55:36.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.935+0000 7f5a4c960640 1 -- 172.21.15.49:0/939536155 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a4810db00 msgr2=0x7f5a4810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:36.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.935+0000 7f5a4c960640 1 --2- 172.21.15.49:0/939536155 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a4810db00 0x7f5a4810df80 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f5a3801a910 tx=0x7f5a3803cca0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.937+0000 7f5a4c960640 1 -- 172.21.15.49:0/939536155 shutdown_connections 2024-03-18T00:55:36.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.937+0000 7f5a4c960640 1 --2- 172.21.15.49:0/939536155 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a4810b150 0x7f5a4810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.937+0000 7f5a4c960640 1 --2- 172.21.15.49:0/939536155 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a4810db00 0x7f5a4810df80 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.937+0000 7f5a4c960640 1 --2- 172.21.15.49:0/939536155 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a481087e0 0x7f5a4810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.937+0000 7f5a4c960640 1 -- 172.21.15.49:0/939536155 >> 172.21.15.49:0/939536155 conn(0x7f5a4806e640 msgr2=0x7f5a48070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:36.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.937+0000 7f5a4c960640 1 -- 172.21.15.49:0/939536155 shutdown_connections 2024-03-18T00:55:36.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.939+0000 7f5a4c960640 1 -- 172.21.15.49:0/939536155 wait complete. 2024-03-18T00:55:36.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.941+0000 7f5a4c960640 1 Processor -- start 2024-03-18T00:55:36.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.944+0000 7f5a4c960640 1 -- start start 2024-03-18T00:55:36.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.945+0000 7f5a4c960640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a481087e0 0x7f5a4819e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:36.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.945+0000 7f5a4c960640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a4810b150 0x7f5a4819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:36.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.945+0000 7f5a4c960640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a4810db00 0x7f5a481a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:36.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.945+0000 7f5a4c960640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5a48111ba0 con 0x7f5a481087e0 2024-03-18T00:55:36.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.945+0000 7f5a4c960640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5a48111a20 con 0x7f5a4810db00 2024-03-18T00:55:36.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.945+0000 7f5a4c960640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5a48111d20 con 0x7f5a4810b150 2024-03-18T00:55:36.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.946+0000 7f5a4659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a481087e0 0x7f5a4819e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:36.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.946+0000 7f5a4659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a481087e0 0x7f5a4819e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53850/0 (socket says 172.21.15.49:53850) 2024-03-18T00:55:36.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.946+0000 7f5a4659c640 1 -- 172.21.15.49:0/1147149631 learned_addr learned my addr 172.21.15.49:0/1147149631 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:36.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.950+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a4810b150 0x7f5a4819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:36.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.950+0000 7f5a46d9d640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a4810db00 0x7f5a481a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:36.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.951+0000 7f5a46d9d640 1 -- 172.21.15.49:0/1147149631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a4810b150 msgr2=0x7f5a4819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:36.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.951+0000 7f5a46d9d640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a4810b150 0x7f5a4819e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.951+0000 7f5a46d9d640 1 -- 172.21.15.49:0/1147149631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a481087e0 msgr2=0x7f5a4819e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:36.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.951+0000 7f5a46d9d640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a481087e0 0x7f5a4819e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.951+0000 7f5a46d9d640 1 -- 172.21.15.49:0/1147149631 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a481a5d50 con 0x7f5a4810db00 2024-03-18T00:55:36.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.953+0000 7f5a46d9d640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a4810db00 0x7f5a481a56e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a38047040 tx=0x7f5a380045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:36.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.962+0000 7f5a377fe640 1 -- 172.21.15.49:0/1147149631 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3803dba0 con 0x7f5a4810db00 2024-03-18T00:55:36.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.963+0000 7f5a4c960640 1 -- 172.21.15.49:0/1147149631 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a481a5fe0 con 0x7f5a4810db00 2024-03-18T00:55:36.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.963+0000 7f5a4c960640 1 -- 172.21.15.49:0/1147149631 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a481a64c0 con 0x7f5a4810db00 2024-03-18T00:55:36.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.968+0000 7f5a4f61f740 1 Processor -- start 2024-03-18T00:55:36.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.968+0000 7f5a377fe640 1 -- 172.21.15.49:0/1147149631 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a38053070 con 0x7f5a4810db00 2024-03-18T00:55:36.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.969+0000 7f5a377fe640 1 -- 172.21.15.49:0/1147149631 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3804e770 con 0x7f5a4810db00 2024-03-18T00:55:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.971+0000 7f5a4f61f740 1 -- start start 2024-03-18T00:55:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.972+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddaf9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.972+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddafe660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.972+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 0x55f5ddb01160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.972+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f5dda5e500 con 0x55f5ddafecd0 2024-03-18T00:55:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.972+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f5dda06be0 con 0x55f5ddaf9170 2024-03-18T00:55:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.972+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f5dda5a5a0 con 0x55f5ddaf9b60 2024-03-18T00:55:36.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.974+0000 7f5a4659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddafe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:36.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.974+0000 7f5a4659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddafe660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46830/0 (socket says 172.21.15.49:46830) 2024-03-18T00:55:36.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.974+0000 7f5a4659c640 1 -- 172.21.15.49:0/2175520320 learned_addr learned my addr 172.21.15.49:0/2175520320 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:36.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.977+0000 7f5a4659c640 1 -- 172.21.15.49:0/2175520320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 msgr2=0x55f5ddaf9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:36.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.977+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddaf9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:36.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.977+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 0x55f5ddb01160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:36.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.977+0000 7f5a4659c640 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddaf9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.977+0000 7f5a4659c640 1 -- 172.21.15.49:0/2175520320 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 msgr2=0x55f5ddb01160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:36.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.977+0000 7f5a4659c640 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 0x55f5ddb01160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.977 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.977+0000 7f5a4659c640 1 -- 172.21.15.49:0/2175520320 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5dda1a4b0 con 0x55f5ddaf9b60 2024-03-18T00:55:36.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.977+0000 7f5a377fe640 1 -- 172.21.15.49:0/1147149631 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5a380035f0 con 0x7f5a4810db00 2024-03-18T00:55:36.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.979+0000 7f5a4659c640 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddafe660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009b00 tx=0x7f5a3c00e1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=423acfc1bd515cf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:36.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.980+0000 7f5a377fe640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a300a2a10 0x7f5a300a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:36.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.980+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 0x55f5ddb01160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:36.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.981+0000 7f5a377fe640 1 -- 172.21.15.49:0/1147149631 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5a38101690 con 0x7f5a4810db00 2024-03-18T00:55:36.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.982+0000 7f5a34ff9640 1 -- 172.21.15.49:0/2175520320 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c017070 con 0x55f5ddaf9b60 2024-03-18T00:55:36.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.982+0000 7f5a34ff9640 1 -- 172.21.15.49:0/2175520320 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a3c00ec10 con 0x55f5ddaf9b60 2024-03-18T00:55:36.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.985+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2175520320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 msgr2=0x55f5ddafe660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:36.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.985+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddafe660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009b00 tx=0x7f5a3c00e1d0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.988+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a300a2a10 0x7f5a300a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:36.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.990+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2175520320 shutdown_connections 2024-03-18T00:55:36.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.990+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 0x55f5ddb01160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.990+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddafe660 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.990+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2175520320 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddaf9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:36.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.990+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2175520320 >> 172.21.15.49:0/2175520320 conn(0x55f5ddaf46e0 msgr2=0x55f5ddaf4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:37.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:36.999+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2175520320 shutdown_connections 2024-03-18T00:55:37.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.002+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a300a2a10 0x7f5a300a4ed0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009440 tx=0x7f5a4000c040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:37.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.005+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2175520320 wait complete. 2024-03-18T00:55:37.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.005+0000 7f5a4f61f740 1 -- start start 2024-03-18T00:55:37.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.006+0000 7f5a4f61f740 1 Processor -- start 2024-03-18T00:55:37.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.011+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb036c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.011+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb03c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.011+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 0x55f5ddb0abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.011+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f5ddb01cd0 con 0x55f5ddafecd0 2024-03-18T00:55:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.011+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f5dda1a160 con 0x55f5ddaf9170 2024-03-18T00:55:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.011+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f5ddb01e50 con 0x55f5ddaf9b60 2024-03-18T00:55:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.011+0000 7f5a4f61f740 10 client.0 _ll_register_callbacks cb 0x55f5dd9fe150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.013+0000 7f5a4659c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.013+0000 7f5a4659c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb036c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:43234/0 (socket says 172.21.15.49:43234) 2024-03-18T00:55:37.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.013+0000 7f5a4659c640 1 -- 172.21.15.49:0/2924787 learned_addr learned my addr 172.21.15.49:0/2924787 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:37.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.014+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb03c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.014+0000 7f5a4659c640 1 -- 172.21.15.49:0/2924787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 msgr2=0x55f5ddb03c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:37.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.014+0000 7f5a4659c640 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb03c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.014+0000 7f5a4659c640 1 -- 172.21.15.49:0/2924787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 msgr2=0x55f5ddb0abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:37.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.014+0000 7f5a4659c640 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 0x55f5ddb0abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.014+0000 7f5a4659c640 1 -- 172.21.15.49:0/2924787 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5dd83c7d0 con 0x55f5ddaf9170 2024-03-18T00:55:37.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.016+0000 7f5a4659c640 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb036c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00e6d0 tx=0x7f5a3c012b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21f4352347c8eb3e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:37.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.020+0000 7f5a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:55:37.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.020+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2924787 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c017040 con 0x55f5ddaf9170 2024-03-18T00:55:37.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.020+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2924787 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a3c00ec10 con 0x55f5ddaf9170 2024-03-18T00:55:37.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.020+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2924787 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5a3c012e20 con 0x55f5ddaf9170 2024-03-18T00:55:37.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.024+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2924787 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a3c004030 con 0x55f5ddaf9170 2024-03-18T00:55:37.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.026+0000 7f5a4f61f740 10 client.26069 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:37.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.026+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2924787 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f5dd7723f0 con 0x55f5ddaf9170 2024-03-18T00:55:37.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.029+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2924787 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a3c00ec10 con 0x55f5ddaf9170 2024-03-18T00:55:37.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.041+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2924787 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5a3c023020 con 0x55f5ddaf9170 2024-03-18T00:55:37.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.042+0000 7f5a4f61f740 10 client.26069 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:37.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.042+0000 7f5a4f61f740 10 client.26069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:37.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.042+0000 7f5a4f61f740 20 client.26069 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:37.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.042+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f5ddb0f230 0x55f5ddb116d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.042+0000 7f5a4f61f740 4 client.26069 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:37.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.042+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2924787 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5dda6ba40 con 0x55f5ddb0f230 2024-03-18T00:55:37.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.044+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f5ddb0f230 0x55f5ddb116d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.055+0000 7f5a2dffb640 10 client.26069 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:37.054 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.055+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f5ddb0f230 0x55f5ddb116d0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfe0b4a23366d49b server_cookie=f3a86b691da9936b in_seq=0 out_seq=0 2024-03-18T00:55:37.168 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.166+0000 7fd3d4ff9640 1 -- 172.21.15.49:0/3930523036 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5569f5bd22d0 con 0x5569f5cf29f0 2024-03-18T00:55:37.168 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.166+0000 7fd3d4ff9640 10 client.16683 handle_command_reply: tid=0 2024-03-18T00:55:37.170 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.166+0000 7fd3f6367740 1 client.16683 shutdown 2024-03-18T00:55:37.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.168+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2924787 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f5dda6ba40 con 0x55f5ddb0f230 2024-03-18T00:55:37.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.168+0000 7f5a2dffb640 10 client.26069 handle_command_reply: tid=0 2024-03-18T00:55:37.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.168+0000 7f5a4f61f740 1 client.26069 shutdown 2024-03-18T00:55:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.168+0000 7fd3f6367740 1 -- 172.21.15.49:0/3930523036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 msgr2=0x5569f5c86460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.168+0000 7fd3f6367740 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c86460 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd3e402cd40 tx=0x7fd3e4043690 comp rx=0 tx=0).stop 2024-03-18T00:55:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.168+0000 7fd3f6367740 1 -- 172.21.15.49:0/3930523036 shutdown_connections 2024-03-18T00:55:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.168+0000 7fd3f6367740 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5569f5cf29f0 0x5569f5cf4e10 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.168+0000 7fd3f6367740 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5569f5c6aea0 0x5569f5c86460 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.168+0000 7fd3f6367740 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5569f5c65d30 0x5569f5c85450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.168+0000 7fd3f6367740 1 --2- 172.21.15.49:0/3930523036 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5569f5c65340 0x5569f5c84f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.169+0000 7fd3f6367740 1 -- 172.21.15.49:0/3930523036 >> 172.21.15.49:0/3930523036 conn(0x5569f5c608b0 msgr2=0x5569f5c67130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:37.171 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.169+0000 7fd3d4ff9640 0 client.16683 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:37.172 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.171+0000 7fd3f6367740 1 -- 172.21.15.49:0/3930523036 shutdown_connections 2024-03-18T00:55:37.173 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.171+0000 7fd3f6367740 1 -- 172.21.15.49:0/3930523036 wait complete. 2024-03-18T00:55:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.172+0000 7fd3f6367740 20 client.16683 trim_cache size 0 max 16384 2024-03-18T00:55:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:55:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:55:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:55:37.174 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:55:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.175+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2924787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 msgr2=0x55f5ddb036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.175+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb036c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00e6d0 tx=0x7f5a3c012b20 comp rx=0 tx=0).stop 2024-03-18T00:55:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.178+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2924787 shutdown_connections 2024-03-18T00:55:37.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.178+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f5ddb0f230 0x55f5ddb116d0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.178+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddafecd0 0x55f5ddb0abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.178+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb03c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.178+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2924787 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb036c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.178+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2924787 >> 172.21.15.49:0/2924787 conn(0x55f5ddaf46e0 msgr2=0x55f5ddaf3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.178+0000 7f5a2dffb640 0 client.26069 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:37.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.186+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2924787 shutdown_connections 2024-03-18T00:55:37.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.186+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2924787 wait complete. 2024-03-18T00:55:37.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.187+0000 7f5a4f61f740 20 client.26069 trim_cache size 0 max 16384 2024-03-18T00:55:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.191+0000 7fd3f369a640 1 -- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3d80a2a10 msgr2=0x7fd3d80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.191+0000 7fd3f369a640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3d80a2a10 0x7fd3d80a4ed0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80970d0 tx=0x7fd3e800bf90 comp rx=0 tx=0).stop 2024-03-18T00:55:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.193+0000 7fd3f369a640 1 -- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec10b150 msgr2=0x7fd3ec1a6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:37.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.193+0000 7fd3f369a640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec10b150 0x7fd3ec1a6e20 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd3e800b4a0 tx=0x7fd3e800b960 comp rx=0 tx=0).stop 2024-03-18T00:55:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.196+0000 7fd3f369a640 1 -- 172.21.15.49:0/3838884880 shutdown_connections 2024-03-18T00:55:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.196+0000 7fd3f369a640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3d80a2a10 0x7fd3d80a4ed0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.196+0000 7fd3f369a640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd3ec111f20 0x7fd3ec1ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.196+0000 7fd3f369a640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3ec10b150 0x7fd3ec1a6e20 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.196+0000 7fd3f369a640 1 --2- 172.21.15.49:0/3838884880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3ec1087e0 0x7fd3ec1a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.196+0000 7fd3f369a640 1 -- 172.21.15.49:0/3838884880 >> 172.21.15.49:0/3838884880 conn(0x7fd3ec06e820 msgr2=0x7fd3ec112cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:37.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.207+0000 7fd3f369a640 1 -- 172.21.15.49:0/3838884880 shutdown_connections 2024-03-18T00:55:37.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:37.207+0000 7fd3f369a640 1 -- 172.21.15.49:0/3838884880 wait complete. 2024-03-18T00:55:37.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.263+0000 7f5a4f61f740 1 Processor -- start 2024-03-18T00:55:37.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.264+0000 7f5a4f61f740 1 -- start start 2024-03-18T00:55:37.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.264+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.264+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb0ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.265+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 0x55f5ddaf8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.265+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2800e640 con 0x55f5ddaf9b60 2024-03-18T00:55:37.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.265+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5a28010180 con 0x55f5ddaf9170 2024-03-18T00:55:37.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.265+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2800ed10 con 0x55f5ddafecd0 2024-03-18T00:55:37.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.269+0000 7f5a46d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 0x55f5ddaf8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.269+0000 7f5a4659c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.269+0000 7f5a45d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb0ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.269+0000 7f5a45d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb0ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53862/0 (socket says 172.21.15.49:53862) 2024-03-18T00:55:37.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.269+0000 7f5a45d9b640 1 -- 172.21.15.49:0/1767462686 learned_addr learned my addr 172.21.15.49:0/1767462686 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:37.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.271+0000 7f5a4659c640 1 -- 172.21.15.49:0/1767462686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 msgr2=0x55f5ddaf8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:37.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.271+0000 7f5a4659c640 1 --2- 172.21.15.49:0/1767462686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 0x55f5ddaf8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.271+0000 7f5a4659c640 1 -- 172.21.15.49:0/1767462686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 msgr2=0x55f5ddb0ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:37.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.271+0000 7f5a4659c640 1 --2- 172.21.15.49:0/1767462686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb0ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.271+0000 7f5a4659c640 1 -- 172.21.15.49:0/1767462686 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f5dd7320f0 con 0x55f5ddaf9170 2024-03-18T00:55:37.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.283+0000 7f5a4659c640 1 --2- 172.21.15.49:0/1767462686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb039a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009a20 tx=0x7f5a3c009300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e1517409d00734e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:37.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.286+0000 7f5a2ffff640 1 -- 172.21.15.49:0/1767462686 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c034070 con 0x55f5ddaf9170 2024-03-18T00:55:37.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.286+0000 7f5a2ffff640 1 -- 172.21.15.49:0/1767462686 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a3c013e50 con 0x55f5ddaf9170 2024-03-18T00:55:37.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.286+0000 7f5a2ffff640 1 -- 172.21.15.49:0/1767462686 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c017030 con 0x55f5ddaf9170 2024-03-18T00:55:37.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.288+0000 7f5a4f61f740 1 -- 172.21.15.49:0/1767462686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 msgr2=0x55f5ddb039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:37.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.288+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/1767462686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb039a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009a20 tx=0x7f5a3c009300 comp rx=0 tx=0).stop 2024-03-18T00:55:37.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.289+0000 7f5a4f61f740 1 -- 172.21.15.49:0/1767462686 shutdown_connections 2024-03-18T00:55:37.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.289+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/1767462686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 0x55f5ddaf8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.289+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/1767462686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb0ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.289+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/1767462686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb039a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.289+0000 7f5a4f61f740 1 -- 172.21.15.49:0/1767462686 >> 172.21.15.49:0/1767462686 conn(0x55f5ddaf46e0 msgr2=0x55f5ddaf7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:37.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.307+0000 7f5a4f61f740 1 -- 172.21.15.49:0/1767462686 shutdown_connections 2024-03-18T00:55:37.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.307+0000 7f5a4f61f740 1 -- 172.21.15.49:0/1767462686 wait complete. 2024-03-18T00:55:37.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.308+0000 7f5a4f61f740 1 -- start start 2024-03-18T00:55:37.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.308+0000 7f5a4f61f740 1 Processor -- start 2024-03-18T00:55:37.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.309+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb08400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.309+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb08940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.310+0000 7f5a4f61f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 0x55f5ddb09950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.310+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2800fab0 con 0x55f5ddaf9b60 2024-03-18T00:55:37.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.310+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2800f3e0 con 0x55f5ddaf9170 2024-03-18T00:55:37.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.310+0000 7f5a4f61f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2800c620 con 0x55f5ddafecd0 2024-03-18T00:55:37.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.310+0000 7f5a4f61f740 10 client.0 _ll_register_callbacks cb 0x7f5a3c009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:37.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.312+0000 7f5a45d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb08940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.312+0000 7f5a45d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb08940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53872/0 (socket says 172.21.15.49:53872) 2024-03-18T00:55:37.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.312+0000 7f5a45d9b640 1 -- 172.21.15.49:0/2171256936 learned_addr learned my addr 172.21.15.49:0/2171256936 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:37.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.312+0000 7f5a4659c640 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb08400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.312+0000 7f5a46d9d640 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 0x55f5ddb09950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.317+0000 7f5a45d9b640 1 -- 172.21.15.49:0/2171256936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 msgr2=0x55f5ddb09950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.317+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 0x55f5ddb09950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.317+0000 7f5a45d9b640 1 -- 172.21.15.49:0/2171256936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 msgr2=0x55f5ddb08400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.317+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb08400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.317+0000 7f5a45d9b640 1 -- 172.21.15.49:0/2171256936 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5dda259d0 con 0x55f5ddaf9b60 2024-03-18T00:55:37.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.322+0000 7f5a45d9b640 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb08940 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5a40016ad0 tx=0x7f5a400093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f422036ffc373 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:37.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.334+0000 7f5a2dffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:37.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.334+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2171256936 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40024c90 con 0x55f5ddaf9b60 2024-03-18T00:55:37.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.334+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2171256936 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5a400260b0 con 0x55f5ddaf9b60 2024-03-18T00:55:37.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.334+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2171256936 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a3c013e50 con 0x55f5ddaf9b60 2024-03-18T00:55:37.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.341+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2171256936 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5a40024e30 con 0x55f5ddaf9b60 2024-03-18T00:55:37.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.341+0000 7f5a4f61f740 10 client.16695 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:37.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.341+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2171256936 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f5ddb756a0 con 0x55f5ddaf9b60 2024-03-18T00:55:37.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.349+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2171256936 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a40026790 con 0x55f5ddaf9b60 2024-03-18T00:55:37.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.353+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2171256936 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5a4002e420 con 0x55f5ddaf9b60 2024-03-18T00:55:37.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.353+0000 7f5a4f61f740 10 client.16695 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:37.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.353+0000 7f5a4f61f740 10 client.16695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:37.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.353+0000 7f5a4f61f740 20 client.16695 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:37.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.353+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f5ddb76080 0x55f5ddb784a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.353+0000 7f5a4f61f740 4 client.16695 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:55:37.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.353+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2171256936 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f5a3c013e50 con 0x55f5ddb76080 2024-03-18T00:55:37.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.354+0000 7f5a4659c640 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f5ddb76080 0x55f5ddb784a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.364+0000 7f5a4659c640 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f5ddb76080 0x55f5ddb784a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f94840780805f7d server_cookie=b2f64dd7c50cda7 in_seq=0 out_seq=0 2024-03-18T00:55:37.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.366+0000 7f5a2dffb640 10 client.16695 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:37.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.366+0000 7f5a2dffb640 1 -- 172.21.15.49:0/2171256936 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f5a3c013e50 con 0x55f5ddb76080 2024-03-18T00:55:37.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.366+0000 7f5a2dffb640 10 client.16695 handle_command_reply: tid=0 2024-03-18T00:55:37.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.367+0000 7f5a4f61f740 1 client.16695 shutdown 2024-03-18T00:55:37.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.373+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2171256936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 msgr2=0x55f5ddb08940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:37.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.374+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb08940 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f5a40016ad0 tx=0x7f5a400093d0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.378+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2171256936 shutdown_connections 2024-03-18T00:55:37.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.378+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f5ddb76080 0x55f5ddb784a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.378+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f5ddafecd0 0x55f5ddb09950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.378+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f5ddaf9b60 0x55f5ddb08940 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.378+0000 7f5a4f61f740 1 --2- 172.21.15.49:0/2171256936 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f5ddaf9170 0x55f5ddb08400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.378+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2171256936 >> 172.21.15.49:0/2171256936 conn(0x55f5ddaf46e0 msgr2=0x55f5ddaf81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:37.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.378+0000 7f5a2dffb640 0 client.16695 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:37.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.380+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2171256936 shutdown_connections 2024-03-18T00:55:37.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.381+0000 7f5a4f61f740 1 -- 172.21.15.49:0/2171256936 wait complete. 2024-03-18T00:55:37.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.381+0000 7f5a4f61f740 20 client.16695 trim_cache size 0 max 16384 2024-03-18T00:55:37.385 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:55:37.385 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:55:37.385 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:55:37.385 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:55:37.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.394+0000 7f5a4c960640 1 -- 172.21.15.49:0/1147149631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a300a2a10 msgr2=0x7f5a300a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:37.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.394+0000 7f5a4c960640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a300a2a10 0x7f5a300a4ed0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009440 tx=0x7f5a4000c040 comp rx=0 tx=0).stop 2024-03-18T00:55:37.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.394+0000 7f5a4c960640 1 -- 172.21.15.49:0/1147149631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a4810db00 msgr2=0x7f5a481a56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:37.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.394+0000 7f5a4c960640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a4810db00 0x7f5a481a56e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a38047040 tx=0x7f5a380045e0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.397+0000 7f5a4c960640 1 -- 172.21.15.49:0/1147149631 shutdown_connections 2024-03-18T00:55:37.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.397+0000 7f5a4c960640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a300a2a10 0x7f5a300a4ed0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.397+0000 7f5a4c960640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a4810db00 0x7f5a481a56e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.397+0000 7f5a4c960640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a4810b150 0x7f5a4819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.397+0000 7f5a4c960640 1 --2- 172.21.15.49:0/1147149631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a481087e0 0x7f5a4819e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.397+0000 7f5a4c960640 1 -- 172.21.15.49:0/1147149631 >> 172.21.15.49:0/1147149631 conn(0x7f5a4806e640 msgr2=0x7f5a4810b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:37.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.398+0000 7f5a4c960640 1 -- 172.21.15.49:0/1147149631 shutdown_connections 2024-03-18T00:55:37.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.398+0000 7f5a4c960640 1 -- 172.21.15.49:0/1147149631 wait complete. 2024-03-18T00:55:37.456 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:55:37.456 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:55:37.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:37 smithi049 ceph-mon[32130]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 871 MiB/s wr, 2.06k op/s 2024-03-18T00:55:37.617 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:37 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:55:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:37 smithi107 ceph-mon[36270]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 871 MiB/s wr, 2.06k op/s 2024-03-18T00:55:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:37 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:55:37.750 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:37 smithi106 ceph-mon[35899]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 871 MiB/s wr, 2.06k op/s 2024-03-18T00:55:37.750 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:37 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:55:37.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.941+0000 7fccf564b640 1 -- 172.21.15.49:0/2415498290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccf0073600 msgr2=0x7fccf010a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:37.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.941+0000 7fccf564b640 1 --2- 172.21.15.49:0/2415498290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccf0073600 0x7fccf010a7f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009900 tx=0x7fcce002bc60 comp rx=0 tx=0).stop 2024-03-18T00:55:37.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.942+0000 7fccf564b640 1 -- 172.21.15.49:0/2415498290 shutdown_connections 2024-03-18T00:55:37.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.942+0000 7fccf564b640 1 --2- 172.21.15.49:0/2415498290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf010d8c0 0x7fccf0118720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.942+0000 7fccf564b640 1 --2- 172.21.15.49:0/2415498290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf010adc0 0x7fccf010d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.942+0000 7fccf564b640 1 --2- 172.21.15.49:0/2415498290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccf0073600 0x7fccf010a7f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.942+0000 7fccf564b640 1 -- 172.21.15.49:0/2415498290 >> 172.21.15.49:0/2415498290 conn(0x7fccf006e530 msgr2=0x7fccf0070970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:37.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.945+0000 7fccf564b640 1 -- 172.21.15.49:0/2415498290 shutdown_connections 2024-03-18T00:55:37.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.947+0000 7fccf564b640 1 -- 172.21.15.49:0/2415498290 wait complete. 2024-03-18T00:55:37.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.948+0000 7fccf564b640 1 Processor -- start 2024-03-18T00:55:37.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.949+0000 7fccf564b640 1 -- start start 2024-03-18T00:55:37.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.949+0000 7fccf564b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccf0073600 0x7fccf0114ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.949+0000 7fccf564b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf010adc0 0x7fccf0115410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.949+0000 7fccf564b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf010d8c0 0x7fccf01b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.949+0000 7fccf564b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fccf011c200 con 0x7fccf010adc0 2024-03-18T00:55:37.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.949+0000 7fccf564b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fccf011c080 con 0x7fccf0073600 2024-03-18T00:55:37.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.949+0000 7fccf564b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fccf011c380 con 0x7fccf010d8c0 2024-03-18T00:55:37.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.950+0000 7fccef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf010d8c0 0x7fccf01b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.950+0000 7fccef7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf010d8c0 0x7fccf01b6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58932/0 (socket says 172.21.15.49:58932) 2024-03-18T00:55:37.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.950+0000 7fccef7fe640 1 -- 172.21.15.49:0/1308776199 learned_addr learned my addr 172.21.15.49:0/1308776199 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:37.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.951+0000 7fccef7fe640 1 -- 172.21.15.49:0/1308776199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccf0073600 msgr2=0x7fccf0114ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:55:37.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.951+0000 7fccef7fe640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccf0073600 0x7fccf0114ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.951+0000 7fccef7fe640 1 -- 172.21.15.49:0/1308776199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf010adc0 msgr2=0x7fccf0115410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:55:37.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.951+0000 7fccef7fe640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf010adc0 0x7fccf0115410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:37.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.951+0000 7fccef7fe640 1 -- 172.21.15.49:0/1308776199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf01b6e50 con 0x7fccf010d8c0 2024-03-18T00:55:37.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.952+0000 7fccef7fe640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf010d8c0 0x7fccf01b6910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcce800b990 tx=0x7fcce800be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:37.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.954+0000 7fccd7fff640 1 -- 172.21.15.49:0/1308776199 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce8019070 con 0x7fccf010d8c0 2024-03-18T00:55:37.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.954+0000 7fccd7fff640 1 -- 172.21.15.49:0/1308776199 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcce800cba0 con 0x7fccf010d8c0 2024-03-18T00:55:37.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.955+0000 7fccd7fff640 1 -- 172.21.15.49:0/1308776199 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce80076e0 con 0x7fccf010d8c0 2024-03-18T00:55:37.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.955+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf01b7020 con 0x7fccf010d8c0 2024-03-18T00:55:37.957 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.955+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf01b7490 con 0x7fccf010d8c0 2024-03-18T00:55:37.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.958+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf0071030 con 0x7fccf010d8c0 2024-03-18T00:55:37.976 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.977+0000 7fccd7fff640 1 -- 172.21.15.49:0/1308776199 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcce801c070 con 0x7fccf010d8c0 2024-03-18T00:55:37.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.978+0000 7fccd7fff640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd80a29d0 0x7fccd80a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.978+0000 7fccd7fff640 1 -- 172.21.15.49:0/1308776199 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcce80ce0c0 con 0x7fccf010d8c0 2024-03-18T00:55:37.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.979+0000 7fccd7fff640 1 -- 172.21.15.49:0/1308776199 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcce80ce580 con 0x7fccf010d8c0 2024-03-18T00:55:37.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:37.984+0000 7fcceeffd640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd80a29d0 0x7fccd80a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:37.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.000+0000 7fcceeffd640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd80a29d0 0x7fccd80a4e90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcce0036040 tx=0x7fcce0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:38.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.520+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccf002d1b0 con 0x7fccf010d8c0 2024-03-18T00:55:38.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.524+0000 7fccd7fff640 1 -- 172.21.15.49:0/1308776199 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7fcce809b160 con 0x7fccf010d8c0 2024-03-18T00:55:38.525 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:55:38.527 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:53:25.016081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24470,24463,24481,24506,24499]},"id":1}]} 2024-03-18T00:55:38.527 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 27 2024-03-18T00:55:38.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.529+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd80a29d0 msgr2=0x7fccd80a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:38.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.529+0000 7fccf564b640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd80a29d0 0x7fccd80a4e90 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcce0036040 tx=0x7fcce0005fd0 comp rx=0 tx=0).stop 2024-03-18T00:55:38.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.530+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf010d8c0 msgr2=0x7fccf01b6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:38.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.530+0000 7fccf564b640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf010d8c0 0x7fccf01b6910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcce800b990 tx=0x7fcce800be50 comp rx=0 tx=0).stop 2024-03-18T00:55:38.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.533+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 shutdown_connections 2024-03-18T00:55:38.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.533+0000 7fccf564b640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fccd80a29d0 0x7fccd80a4e90 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:38.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.533+0000 7fccf564b640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fccf010d8c0 0x7fccf01b6910 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:38.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.533+0000 7fccf564b640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fccf010adc0 0x7fccf0115410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:38.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.533+0000 7fccf564b640 1 --2- 172.21.15.49:0/1308776199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fccf0073600 0x7fccf0114ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:38.535 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.533+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 >> 172.21.15.49:0/1308776199 conn(0x7fccf006e530 msgr2=0x7fccf0109a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:38.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.536+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 shutdown_connections 2024-03-18T00:55:38.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:38.537+0000 7fccf564b640 1 -- 172.21.15.49:0/1308776199 wait complete. 2024-03-18T00:55:38.587 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:55:39.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.147+0000 7ff555819640 1 -- 172.21.15.49:0/1577987375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 msgr2=0x7ff550108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.147+0000 7ff555819640 1 --2- 172.21.15.49:0/1577987375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 0x7ff550108b60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff548009ca0 tx=0x7ff54802c120 comp rx=0 tx=0).stop 2024-03-18T00:55:39.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.148+0000 7ff555819640 1 -- 172.21.15.49:0/1577987375 shutdown_connections 2024-03-18T00:55:39.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.148+0000 7ff555819640 1 --2- 172.21.15.49:0/1577987375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff550101910 0x7ff550101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.148+0000 7ff555819640 1 --2- 172.21.15.49:0/1577987375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 0x7ff550108b60 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.148+0000 7ff555819640 1 --2- 172.21.15.49:0/1577987375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff550100720 0x7ff550100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.148+0000 7ff555819640 1 -- 172.21.15.49:0/1577987375 >> 172.21.15.49:0/1577987375 conn(0x7ff5500fbc90 msgr2=0x7ff5500fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.148+0000 7ff555819640 1 -- 172.21.15.49:0/1577987375 shutdown_connections 2024-03-18T00:55:39.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.149+0000 7ff555819640 1 -- 172.21.15.49:0/1577987375 wait complete. 2024-03-18T00:55:39.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.150+0000 7ff555819640 1 Processor -- start 2024-03-18T00:55:39.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.150+0000 7ff555819640 1 -- start start 2024-03-18T00:55:39.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.150+0000 7ff555819640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff550100720 0x7ff55019e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.150+0000 7ff555819640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff550101910 0x7ff55019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.150+0000 7ff555819640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 0x7ff5501a5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.150+0000 7ff555819640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff55010c4b0 con 0x7ff5501022d0 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.150+0000 7ff555819640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff55010c330 con 0x7ff550101910 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.150+0000 7ff555819640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff55010c630 con 0x7ff550100720 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.151+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 0x7ff5501a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.151+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 0x7ff5501a5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54184/0 (socket says 172.21.15.49:54184) 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.151+0000 7ff54f7fe640 1 -- 172.21.15.49:0/800877813 learned_addr learned my addr 172.21.15.49:0/800877813 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.151+0000 7ff54f7fe640 1 -- 172.21.15.49:0/800877813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff550100720 msgr2=0x7ff55019e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.151+0000 7ff54f7fe640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff550100720 0x7ff55019e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.151+0000 7ff54f7fe640 1 -- 172.21.15.49:0/800877813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff550101910 msgr2=0x7ff55019ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.151+0000 7ff54f7fe640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff550101910 0x7ff55019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.152+0000 7ff54f7fe640 1 -- 172.21.15.49:0/800877813 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5501a5ef0 con 0x7ff5501022d0 2024-03-18T00:55:39.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.152+0000 7ff54f7fe640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 0x7ff5501a5880 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff548037040 tx=0x7ff5480324f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.153+0000 7ff554817640 1 -- 172.21.15.49:0/800877813 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548039040 con 0x7ff5501022d0 2024-03-18T00:55:39.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.153+0000 7ff554817640 1 -- 172.21.15.49:0/800877813 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff548044070 con 0x7ff5501022d0 2024-03-18T00:55:39.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.153+0000 7ff555819640 1 -- 172.21.15.49:0/800877813 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5501a6180 con 0x7ff5501022d0 2024-03-18T00:55:39.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.153+0000 7ff554817640 1 -- 172.21.15.49:0/800877813 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff548004610 con 0x7ff5501022d0 2024-03-18T00:55:39.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.153+0000 7ff555819640 1 -- 172.21.15.49:0/800877813 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5501a6740 con 0x7ff5501022d0 2024-03-18T00:55:39.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.162+0000 7ff554817640 1 -- 172.21.15.49:0/800877813 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff548003970 con 0x7ff5501022d0 2024-03-18T00:55:39.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.163+0000 7ff558409740 1 Processor -- start 2024-03-18T00:55:39.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.163+0000 7ff554817640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff53c0a2880 0x7ff53c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.164+0000 7ff554817640 1 -- 172.21.15.49:0/800877813 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff5480f13a0 con 0x7ff5501022d0 2024-03-18T00:55:39.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.164+0000 7ff547fff640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff53c0a2880 0x7ff53c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.164+0000 7ff558409740 1 -- start start 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff558409740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564868289850 0x564868289c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff558409740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x56486828ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff558409740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486828f3b0 0x564868291840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff558409740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5648681ee490 con 0x56486828f3b0 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff558409740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x564868196c90 con 0x56486828a240 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff558409740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5648681ea530 con 0x564868289850 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486828f3b0 0x564868291840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff54f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486828f3b0 0x564868291840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54198/0 (socket says 172.21.15.49:54198) 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff54f7fe640 1 -- 172.21.15.49:0/2335994190 learned_addr learned my addr 172.21.15.49:0/2335994190 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.165+0000 7ff54effd640 1 --2- 172.21.15.49:0/2335994190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564868289850 0x564868289c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.166+0000 7ff54f7fe640 1 -- 172.21.15.49:0/2335994190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564868289850 msgr2=0x564868289c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.166+0000 7ff54f7fe640 1 --2- 172.21.15.49:0/2335994190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564868289850 0x564868289c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.166+0000 7ff54f7fe640 1 -- 172.21.15.49:0/2335994190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 msgr2=0x56486828ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.166+0000 7ff54f7fe640 1 --2- 172.21.15.49:0/2335994190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x56486828ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.166+0000 7ff54f7fe640 1 -- 172.21.15.49:0/2335994190 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564868251500 con 0x56486828f3b0 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.166+0000 7ff547fff640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff53c0a2880 0x7ff53c0a4d40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff550108c90 tx=0x7ff538006c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.166+0000 7ff54effd640 1 --2- 172.21.15.49:0/2335994190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564868289850 0x564868289c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.166+0000 7ff54f7fe640 1 --2- 172.21.15.49:0/2335994190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486828f3b0 0x564868291840 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff5480ba690 tx=0x7ff548003f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643d300d494839d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.166+0000 7ff545ffb640 1 -- 172.21.15.49:0/2335994190 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480be3c0 con 0x56486828f3b0 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.167+0000 7ff545ffb640 1 -- 172.21.15.49:0/2335994190 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5480098b0 con 0x56486828f3b0 2024-03-18T00:55:39.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.167+0000 7ff545ffb640 1 -- 172.21.15.49:0/2335994190 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5480bc050 con 0x56486828f3b0 2024-03-18T00:55:39.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.168+0000 7ff558409740 1 -- 172.21.15.49:0/2335994190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486828f3b0 msgr2=0x564868291840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.168+0000 7ff558409740 1 --2- 172.21.15.49:0/2335994190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486828f3b0 0x564868291840 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7ff5480ba690 tx=0x7ff548003f50 comp rx=0 tx=0).stop 2024-03-18T00:55:39.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.169+0000 7ff558409740 1 -- 172.21.15.49:0/2335994190 shutdown_connections 2024-03-18T00:55:39.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.169+0000 7ff558409740 1 --2- 172.21.15.49:0/2335994190 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x56486828ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.169+0000 7ff558409740 1 --2- 172.21.15.49:0/2335994190 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486828f3b0 0x564868291840 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.169+0000 7ff558409740 1 --2- 172.21.15.49:0/2335994190 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564868289850 0x564868289c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.169+0000 7ff558409740 1 -- 172.21.15.49:0/2335994190 >> 172.21.15.49:0/2335994190 conn(0x564868284dc0 msgr2=0x5648682851c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.170+0000 7ff558409740 1 -- 172.21.15.49:0/2335994190 shutdown_connections 2024-03-18T00:55:39.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.170+0000 7ff558409740 1 -- 172.21.15.49:0/2335994190 wait complete. 2024-03-18T00:55:39.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.171+0000 7ff558409740 1 -- start start 2024-03-18T00:55:39.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.171+0000 7ff558409740 1 Processor -- start 2024-03-18T00:55:39.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff558409740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564868289850 0x5648682939c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff558409740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x564868293f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff558409740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x56486829af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff558409740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564868292120 con 0x564868289850 2024-03-18T00:55:39.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff558409740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5648681aa230 con 0x56486828a240 2024-03-18T00:55:39.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff558409740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5648682922a0 con 0x56486828f3b0 2024-03-18T00:55:39.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff54effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564868289850 0x5648682939c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff54effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564868289850 0x5648682939c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54200/0 (socket says 172.21.15.49:54200) 2024-03-18T00:55:39.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff54effd640 1 -- 172.21.15.49:0/4726945 learned_addr learned my addr 172.21.15.49:0/4726945 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff558409740 10 client.0 _ll_register_callbacks cb 0x5648681e6010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:39.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff54effd640 1 -- 172.21.15.49:0/4726945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 msgr2=0x56486829af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff54effd640 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x56486829af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff54effd640 1 -- 172.21.15.49:0/4726945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 msgr2=0x564868293f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.172+0000 7ff54effd640 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x564868293f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.173+0000 7ff54effd640 1 -- 172.21.15.49:0/4726945 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648681b9100 con 0x564868289850 2024-03-18T00:55:39.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.173+0000 7ff54effd640 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564868289850 0x5648682939c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff54000afd0 tx=0x7ff54000c5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4c53fac90c31c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.173+0000 7ff52effd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:39.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.173+0000 7ff52effd640 1 -- 172.21.15.49:0/4726945 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff540015070 con 0x564868289850 2024-03-18T00:55:39.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.173+0000 7ff52effd640 1 -- 172.21.15.49:0/4726945 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff5400115f0 con 0x564868289850 2024-03-18T00:55:39.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.173+0000 7ff558409740 1 -- 172.21.15.49:0/4726945 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5480098b0 con 0x564868289850 2024-03-18T00:55:39.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff52effd640 1 -- 172.21.15.49:0/4726945 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff540021050 con 0x564868289850 2024-03-18T00:55:39.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff558409740 10 client.16707 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:39.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff558409740 1 -- 172.21.15.49:0/4726945 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564867f023f0 con 0x564868289850 2024-03-18T00:55:39.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff52effd640 1 -- 172.21.15.49:0/4726945 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff540011cd0 con 0x564868289850 2024-03-18T00:55:39.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff52effd640 1 -- 172.21.15.49:0/4726945 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff540019b40 con 0x564868289850 2024-03-18T00:55:39.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff558409740 10 client.16707 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:39.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff558409740 10 client.16707 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:55:39.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff558409740 20 client.16707 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:39.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff558409740 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56486829f7e0 0x5648682a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff558409740 4 client.16707 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:39.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.174+0000 7ff558409740 1 -- 172.21.15.49:0/4726945 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff5480098b0 con 0x56486829f7e0 2024-03-18T00:55:39.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.177+0000 7ff547fff640 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56486829f7e0 0x5648682a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.180+0000 7ff547fff640 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56486829f7e0 0x5648682a1c80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=99f1dd74443f0aa1 server_cookie=6a44c2d2e793ff3e in_seq=0 out_seq=0 2024-03-18T00:55:39.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.180+0000 7ff52effd640 10 client.16707 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.181+0000 7ff52effd640 1 -- 172.21.15.49:0/4726945 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7ff5480098b0 con 0x56486829f7e0 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.181+0000 7ff52effd640 10 client.16707 handle_command_reply: tid=0 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff558409740 1 client.16707 shutdown 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff558409740 1 -- 172.21.15.49:0/4726945 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564868289850 msgr2=0x5648682939c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff558409740 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564868289850 0x5648682939c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff54000afd0 tx=0x7ff54000c5e0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff558409740 1 -- 172.21.15.49:0/4726945 shutdown_connections 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff558409740 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56486829f7e0 0x5648682a1c80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff52effd640 0 client.16707 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff558409740 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x564868293f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff558409740 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x56486829af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff558409740 1 --2- 172.21.15.49:0/4726945 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564868289850 0x5648682939c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.182+0000 7ff558409740 1 -- 172.21.15.49:0/4726945 >> 172.21.15.49:0/4726945 conn(0x564868284dc0 msgr2=0x564868283fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.183+0000 7ff558409740 1 -- 172.21.15.49:0/4726945 shutdown_connections 2024-03-18T00:55:39.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.183+0000 7ff558409740 1 -- 172.21.15.49:0/4726945 wait complete. 2024-03-18T00:55:39.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.183+0000 7ff558409740 20 client.16707 trim_cache size 0 max 16384 2024-03-18T00:55:39.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.221+0000 7ff558409740 1 Processor -- start 2024-03-18T00:55:39.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.222+0000 7ff558409740 1 -- start start 2024-03-18T00:55:39.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.222+0000 7ff558409740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x564868280110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.223+0000 7ff558409740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x56486829af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.223+0000 7ff558409740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486829f7e0 0x5648682851a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.223 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.223+0000 7ff558409740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501af630 con 0x56486829f7e0 2024-03-18T00:55:39.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.223+0000 7ff558409740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501b1170 con 0x56486828a240 2024-03-18T00:55:39.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.223+0000 7ff54effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x564868280110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.223+0000 7ff54effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x564868280110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47566/0 (socket says 172.21.15.49:47566) 2024-03-18T00:55:39.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.223+0000 7ff54effd640 1 -- 172.21.15.49:0/94742437 learned_addr learned my addr 172.21.15.49:0/94742437 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.224+0000 7ff547fff640 1 --2- 172.21.15.49:0/94742437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x56486829af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.224+0000 7ff558409740 1 -- 172.21.15.49:0/94742437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501afd00 con 0x56486828f3b0 2024-03-18T00:55:39.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.225+0000 7ff547fff640 1 -- 172.21.15.49:0/94742437 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 msgr2=0x564868280110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.225+0000 7ff547fff640 1 --2- 172.21.15.49:0/94742437 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x564868280110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.225+0000 7ff547fff640 1 -- 172.21.15.49:0/94742437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486829f7e0 msgr2=0x5648682851a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.225+0000 7ff547fff640 1 --2- 172.21.15.49:0/94742437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486829f7e0 0x5648682851a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.225+0000 7ff547fff640 1 -- 172.21.15.49:0/94742437 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564867fcc7d0 con 0x56486828f3b0 2024-03-18T00:55:39.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.232+0000 7ff547fff640 1 --2- 172.21.15.49:0/94742437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x56486829af00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x564868294ea0 tx=0x7ff5380047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee125ca236b9732a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.233+0000 7ff5457fa640 1 -- 172.21.15.49:0/94742437 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff538013950 con 0x56486828f3b0 2024-03-18T00:55:39.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.233+0000 7ff5457fa640 1 -- 172.21.15.49:0/94742437 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff538021050 con 0x56486828f3b0 2024-03-18T00:55:39.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.233+0000 7ff5457fa640 1 -- 172.21.15.49:0/94742437 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff538020740 con 0x56486828f3b0 2024-03-18T00:55:39.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.235+0000 7ff558409740 1 -- 172.21.15.49:0/94742437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 msgr2=0x56486829af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.235+0000 7ff558409740 1 --2- 172.21.15.49:0/94742437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x56486829af00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x564868294ea0 tx=0x7ff5380047c0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.235+0000 7ff558409740 1 -- 172.21.15.49:0/94742437 shutdown_connections 2024-03-18T00:55:39.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.235+0000 7ff558409740 1 --2- 172.21.15.49:0/94742437 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486829f7e0 0x5648682851a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.235+0000 7ff558409740 1 --2- 172.21.15.49:0/94742437 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x56486829af00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.235+0000 7ff558409740 1 --2- 172.21.15.49:0/94742437 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x564868280110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.235+0000 7ff558409740 1 -- 172.21.15.49:0/94742437 >> 172.21.15.49:0/94742437 conn(0x564868284dc0 msgr2=0x5648682888b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.239+0000 7ff558409740 1 -- 172.21.15.49:0/94742437 shutdown_connections 2024-03-18T00:55:39.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.243+0000 7ff558409740 1 -- 172.21.15.49:0/94742437 wait complete. 2024-03-18T00:55:39.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.243+0000 7ff558409740 1 -- start start 2024-03-18T00:55:39.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.244+0000 7ff558409740 1 Processor -- start 2024-03-18T00:55:39.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.244+0000 7ff558409740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x5648682985d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.244+0000 7ff558409740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x564868298b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.244+0000 7ff558409740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486829f7e0 0x564868299b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.244+0000 7ff558409740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501b0aa0 con 0x56486829f7e0 2024-03-18T00:55:39.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.244+0000 7ff558409740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501b03d0 con 0x56486828a240 2024-03-18T00:55:39.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.244+0000 7ff558409740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff5501ad610 con 0x56486828f3b0 2024-03-18T00:55:39.246 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.245+0000 7ff558409740 10 client.0 _ll_register_callbacks cb 0x7ff54000a0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:39.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.248+0000 7ff547fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x564868298b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.250+0000 7ff547fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x564868298b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58982/0 (socket says 172.21.15.49:58982) 2024-03-18T00:55:39.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.250+0000 7ff547fff640 1 -- 172.21.15.49:0/1743882253 learned_addr learned my addr 172.21.15.49:0/1743882253 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.255+0000 7ff547fff640 1 -- 172.21.15.49:0/1743882253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 msgr2=0x5648682985d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:39.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.255+0000 7ff547fff640 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x5648682985d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.255+0000 7ff547fff640 1 -- 172.21.15.49:0/1743882253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486829f7e0 msgr2=0x564868299b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.255+0000 7ff547fff640 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486829f7e0 0x564868299b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.255+0000 7ff547fff640 1 -- 172.21.15.49:0/1743882253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56486829a110 con 0x56486828f3b0 2024-03-18T00:55:39.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.263+0000 7ff547fff640 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x564868298b10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff538004790 tx=0x7ff53800b590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63eda75eff59ad35 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.265+0000 7ff52effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:55:39.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.268+0000 7ff52effd640 1 -- 172.21.15.49:0/1743882253 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff538038060 con 0x56486828f3b0 2024-03-18T00:55:39.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.268+0000 7ff558409740 1 -- 172.21.15.49:0/1743882253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff538021050 con 0x56486828f3b0 2024-03-18T00:55:39.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.268+0000 7ff52effd640 1 -- 172.21.15.49:0/1743882253 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff538013b60 con 0x56486828f3b0 2024-03-18T00:55:39.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.280+0000 7ff52effd640 1 -- 172.21.15.49:0/1743882253 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff538013d60 con 0x56486828f3b0 2024-03-18T00:55:39.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.283+0000 7ff558409740 10 client.26176 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:39.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.283+0000 7ff558409740 1 -- 172.21.15.49:0/1743882253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564868316bd0 con 0x56486828f3b0 2024-03-18T00:55:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.493+0000 7ff52effd640 1 -- 172.21.15.49:0/1743882253 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff538021050 con 0x56486828f3b0 2024-03-18T00:55:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.494+0000 7ff52effd640 1 -- 172.21.15.49:0/1743882253 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff538012070 con 0x56486828f3b0 2024-03-18T00:55:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.494+0000 7ff558409740 10 client.26176 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.494+0000 7ff558409740 10 client.26176 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:55:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.494+0000 7ff558409740 20 client.26176 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.494+0000 7ff558409740 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x564868317400 0x564868319820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.494+0000 7ff558409740 4 client.26176 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.494+0000 7ff558409740 1 -- 172.21.15.49:0/1743882253 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5648681fb9d0 con 0x564868317400 2024-03-18T00:55:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.497+0000 7ff54effd640 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x564868317400 0x564868319820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.515+0000 7ff54effd640 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x564868317400 0x564868319820 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7e6928d85a4e61fe server_cookie=ae1b28e439c93daf in_seq=0 out_seq=0 2024-03-18T00:55:39.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.518+0000 7ff52effd640 10 client.26176 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:55:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.521+0000 7ff52effd640 1 -- 172.21.15.49:0/1743882253 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5648681fb9d0 con 0x564868317400 2024-03-18T00:55:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.521+0000 7ff52effd640 10 client.26176 handle_command_reply: tid=0 2024-03-18T00:55:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.523+0000 7ff558409740 1 client.26176 shutdown 2024-03-18T00:55:39.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.526+0000 7ff558409740 1 -- 172.21.15.49:0/1743882253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 msgr2=0x564868298b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.526+0000 7ff558409740 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x564868298b10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff538004790 tx=0x7ff53800b590 comp rx=0 tx=0).stop 2024-03-18T00:55:39.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.526+0000 7ff558409740 1 -- 172.21.15.49:0/1743882253 shutdown_connections 2024-03-18T00:55:39.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.526+0000 7ff558409740 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x564868317400 0x564868319820 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.526+0000 7ff558409740 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56486829f7e0 0x564868299b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.526+0000 7ff558409740 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56486828f3b0 0x564868298b10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.526+0000 7ff558409740 1 --2- 172.21.15.49:0/1743882253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56486828a240 0x5648682985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.526+0000 7ff558409740 1 -- 172.21.15.49:0/1743882253 >> 172.21.15.49:0/1743882253 conn(0x564868284dc0 msgr2=0x5648682888b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.526+0000 7ff52effd640 0 client.26176 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:55:39.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.527+0000 7ff558409740 1 -- 172.21.15.49:0/1743882253 shutdown_connections 2024-03-18T00:55:39.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.527+0000 7ff558409740 1 -- 172.21.15.49:0/1743882253 wait complete. 2024-03-18T00:55:39.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.527+0000 7ff558409740 20 client.26176 trim_cache size 0 max 16384 2024-03-18T00:55:39.528 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.531+0000 7ff555819640 1 -- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff53c0a2880 msgr2=0x7ff53c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.531+0000 7ff555819640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff53c0a2880 0x7ff53c0a4d40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff550108c90 tx=0x7ff538006c40 comp rx=0 tx=0).stop 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.531+0000 7ff555819640 1 -- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 msgr2=0x7ff5501a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.531+0000 7ff555819640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 0x7ff5501a5880 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff548037040 tx=0x7ff5480324f0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.532+0000 7ff555819640 1 -- 172.21.15.49:0/800877813 shutdown_connections 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.532+0000 7ff555819640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff53c0a2880 0x7ff53c0a4d40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.532+0000 7ff555819640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff550101910 0x7ff55019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.532+0000 7ff555819640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff5501022d0 0x7ff5501a5880 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.532+0000 7ff555819640 1 --2- 172.21.15.49:0/800877813 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff550100720 0x7ff55019e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.532+0000 7ff555819640 1 -- 172.21.15.49:0/800877813 >> 172.21.15.49:0/800877813 conn(0x7ff5500fbc90 msgr2=0x7ff550069aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.532+0000 7ff555819640 1 -- 172.21.15.49:0/800877813 shutdown_connections 2024-03-18T00:55:39.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.532+0000 7ff555819640 1 -- 172.21.15.49:0/800877813 wait complete. 2024-03-18T00:55:39.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:39 smithi049 ceph-mon[32130]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 850 MiB/s wr, 2.13k op/s 2024-03-18T00:55:39.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:39 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1308776199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:55:39.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:55:39.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:55:39.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:39 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:55:39.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:55:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:39 smithi107 ceph-mon[36270]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 850 MiB/s wr, 2.13k op/s 2024-03-18T00:55:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:39 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1308776199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:55:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:55:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:55:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:39 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:55:39.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.741+0000 7f9695bc6640 1 -- 172.21.15.49:0/559870210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9690102300 msgr2=0x7f96901046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.741+0000 7f9695bc6640 1 --2- 172.21.15.49:0/559870210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9690102300 0x7f96901046f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9680009860 tx=0x7f968002bb90 comp rx=0 tx=0).stop 2024-03-18T00:55:39.741 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.742+0000 7f9695bc6640 1 -- 172.21.15.49:0/559870210 shutdown_connections 2024-03-18T00:55:39.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.742+0000 7f9695bc6640 1 --2- 172.21.15.49:0/559870210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9690104c30 0x7f9690107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.742+0000 7f9695bc6640 1 --2- 172.21.15.49:0/559870210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9690107560 0x7f9690109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.742+0000 7f9695bc6640 1 --2- 172.21.15.49:0/559870210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9690102300 0x7f96901046f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.742+0000 7f9695bc6640 1 -- 172.21.15.49:0/559870210 >> 172.21.15.49:0/559870210 conn(0x7f96900fbc90 msgr2=0x7f96900fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.744+0000 7f9695bc6640 1 -- 172.21.15.49:0/559870210 shutdown_connections 2024-03-18T00:55:39.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.744+0000 7f9695bc6640 1 -- 172.21.15.49:0/559870210 wait complete. 2024-03-18T00:55:39.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.745+0000 7f9695bc6640 1 Processor -- start 2024-03-18T00:55:39.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.746+0000 7f9695bc6640 1 -- start start 2024-03-18T00:55:39.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.746+0000 7f9695bc6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9690102300 0x7f9690101c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.746+0000 7f9695bc6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9690104c30 0x7f96901002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.747+0000 7f9695bc6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9690107560 0x7f96901007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.747+0000 7f9695bc6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f969010d510 con 0x7f9690107560 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.747+0000 7f9695bc6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f969010d390 con 0x7f9690102300 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.747+0000 7f9695bc6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f969010d690 con 0x7f9690104c30 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.747+0000 7f968effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9690104c30 0x7f96901002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.747+0000 7f968effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9690104c30 0x7f96901002b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59004/0 (socket says 172.21.15.49:59004) 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.747+0000 7f968effd640 1 -- 172.21.15.49:0/1891544359 learned_addr learned my addr 172.21.15.49:0/1891544359 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.747+0000 7f968ffff640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9690107560 0x7f96901007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.747+0000 7f968f7fe640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9690102300 0x7f9690101c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.748+0000 7f968ffff640 1 -- 172.21.15.49:0/1891544359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9690104c30 msgr2=0x7f96901002b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.748+0000 7f968ffff640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9690104c30 0x7f96901002b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.748+0000 7f968ffff640 1 -- 172.21.15.49:0/1891544359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9690102300 msgr2=0x7f9690101c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.748+0000 7f968ffff640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9690102300 0x7f9690101c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.748+0000 7f968ffff640 1 -- 172.21.15.49:0/1891544359 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9690100d30 con 0x7f9690107560 2024-03-18T00:55:39.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.748+0000 7f968ffff640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9690107560 0x7f96901007f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f968800c5f0 tx=0x7f968800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.748+0000 7f968cff9640 1 -- 172.21.15.49:0/1891544359 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688003970 con 0x7f9690107560 2024-03-18T00:55:39.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.749+0000 7f968cff9640 1 -- 172.21.15.49:0/1891544359 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9688004020 con 0x7f9690107560 2024-03-18T00:55:39.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.749+0000 7f9695bc6640 1 -- 172.21.15.49:0/1891544359 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9690101020 con 0x7f9690107560 2024-03-18T00:55:39.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.749+0000 7f9695bc6640 1 -- 172.21.15.49:0/1891544359 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96901a67c0 con 0x7f9690107560 2024-03-18T00:55:39.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.750+0000 7f968cff9640 1 -- 172.21.15.49:0/1891544359 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968800b970 con 0x7f9690107560 2024-03-18T00:55:39.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.755+0000 7f96987b1740 1 Processor -- start 2024-03-18T00:55:39.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.756+0000 7f968cff9640 1 -- 172.21.15.49:0/1891544359 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f968801d020 con 0x7f9690107560 2024-03-18T00:55:39.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.757+0000 7f96987b1740 1 -- start start 2024-03-18T00:55:39.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.757+0000 7f96987b1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac80c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.757+0000 7f96987b1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac85d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.757+0000 7f96987b1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac863b0 0x55ee5ac88840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.757+0000 7f96987b1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ee5abe5490 con 0x55ee5ac81240 2024-03-18T00:55:39.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.757+0000 7f96987b1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ee5ab8dc90 con 0x55ee5ac80850 2024-03-18T00:55:39.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.757+0000 7f96987b1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ee5abe1530 con 0x55ee5ac863b0 2024-03-18T00:55:39.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.758+0000 7f968cff9640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f96780a2880 0x7f96780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.758+0000 7f968cff9640 1 -- 172.21.15.49:0/1891544359 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f96880ce9d0 con 0x7f9690107560 2024-03-18T00:55:39.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.759+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac85d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.759+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac85d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54260/0 (socket says 172.21.15.49:54260) 2024-03-18T00:55:39.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.759+0000 7f968f7fe640 1 -- 172.21.15.49:0/3203883322 learned_addr learned my addr 172.21.15.49:0/3203883322 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.763+0000 7f968effd640 1 --2- 172.21.15.49:0/3203883322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac80c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.763+0000 7f968effd640 1 --2- 172.21.15.49:0/3203883322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac863b0 0x55ee5ac88840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.763+0000 7f968ffff640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f96780a2880 0x7f96780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.768+0000 7f968effd640 1 -- 172.21.15.49:0/3203883322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac863b0 msgr2=0x55ee5ac88840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.768+0000 7f968effd640 1 --2- 172.21.15.49:0/3203883322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac863b0 0x55ee5ac88840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.768+0000 7f968effd640 1 -- 172.21.15.49:0/3203883322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 msgr2=0x55ee5ac85d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.768+0000 7f968effd640 1 --2- 172.21.15.49:0/3203883322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac85d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.768+0000 7f968effd640 1 -- 172.21.15.49:0/3203883322 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5ac48500 con 0x55ee5ac80850 2024-03-18T00:55:39.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.770+0000 7f968effd640 1 --2- 172.21.15.49:0/3203883322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac80c70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f96901019e0 tx=0x7f9684009340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=125ccb774af14b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.771+0000 7f9675ffb640 1 -- 172.21.15.49:0/3203883322 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9684015580 con 0x55ee5ac80850 2024-03-18T00:55:39.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.771+0000 7f9675ffb640 1 -- 172.21.15.49:0/3203883322 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f968401d070 con 0x55ee5ac80850 2024-03-18T00:55:39.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.771+0000 7f9675ffb640 1 -- 172.21.15.49:0/3203883322 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9684018810 con 0x55ee5ac80850 2024-03-18T00:55:39.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.775+0000 7f96987b1740 1 -- 172.21.15.49:0/3203883322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 msgr2=0x55ee5ac80c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.775+0000 7f96987b1740 1 --2- 172.21.15.49:0/3203883322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac80c70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f96901019e0 tx=0x7f9684009340 comp rx=0 tx=0).stop 2024-03-18T00:55:39.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.783+0000 7f96987b1740 1 -- 172.21.15.49:0/3203883322 shutdown_connections 2024-03-18T00:55:39.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.783+0000 7f96987b1740 1 --2- 172.21.15.49:0/3203883322 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac863b0 0x55ee5ac88840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.783+0000 7f96987b1740 1 --2- 172.21.15.49:0/3203883322 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac85d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.783+0000 7f96987b1740 1 --2- 172.21.15.49:0/3203883322 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac80c70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.783+0000 7f96987b1740 1 -- 172.21.15.49:0/3203883322 >> 172.21.15.49:0/3203883322 conn(0x55ee5ac7bdc0 msgr2=0x55ee5ac7c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.789+0000 7f968ffff640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f96780a2880 0x7f96780a4d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9688000f80 tx=0x7f968800b460 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.789+0000 7f96987b1740 1 -- 172.21.15.49:0/3203883322 shutdown_connections 2024-03-18T00:55:39.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.794+0000 7f96987b1740 1 -- 172.21.15.49:0/3203883322 wait complete. 2024-03-18T00:55:39.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.794+0000 7f96987b1740 1 -- start start 2024-03-18T00:55:39.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.795+0000 7f96987b1740 1 Processor -- start 2024-03-18T00:55:39.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.802+0000 7f96987b1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac80850 0x55ee5ac8ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.802+0000 7f96987b1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac8b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.802+0000 7f96987b1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac863b0 0x55ee5ac92310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.802+0000 7f96987b1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ee5ac89300 con 0x55ee5ac81240 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.802+0000 7f96987b1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ee5aba1230 con 0x55ee5ac863b0 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.802+0000 7f96987b1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ee5ac89480 con 0x55ee5ac80850 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.802+0000 7f96987b1740 10 client.0 _ll_register_callbacks cb 0x55ee5abdd010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.804+0000 7f968effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac863b0 0x55ee5ac92310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.804+0000 7f968effd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac863b0 0x55ee5ac92310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47602/0 (socket says 172.21.15.49:47602) 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.804+0000 7f968effd640 1 -- 172.21.15.49:0/465206981 learned_addr learned my addr 172.21.15.49:0/465206981 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.804+0000 7f968effd640 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac80850 0x55ee5ac8ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.805+0000 7f968f7fe640 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac8b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.806+0000 7f968f7fe640 1 -- 172.21.15.49:0/465206981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac80850 msgr2=0x55ee5ac8ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.806+0000 7f968f7fe640 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac80850 0x55ee5ac8ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.806+0000 7f968f7fe640 1 -- 172.21.15.49:0/465206981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac863b0 msgr2=0x55ee5ac92310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.806+0000 7f968f7fe640 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac863b0 0x55ee5ac92310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.806+0000 7f968f7fe640 1 -- 172.21.15.49:0/465206981 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5abb0100 con 0x55ee5ac81240 2024-03-18T00:55:39.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.806+0000 7f968f7fe640 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac8b2d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9680000c00 tx=0x7f96800324a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fd5325f0cc278 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.807+0000 7f966effd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:39.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.808+0000 7f966effd640 1 -- 172.21.15.49:0/465206981 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680043070 con 0x55ee5ac81240 2024-03-18T00:55:39.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.808+0000 7f96987b1740 1 -- 172.21.15.49:0/465206981 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f968401d070 con 0x55ee5ac81240 2024-03-18T00:55:39.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.808+0000 7f966effd640 1 -- 172.21.15.49:0/465206981 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9680004040 con 0x55ee5ac81240 2024-03-18T00:55:39.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.809+0000 7f966effd640 1 -- 172.21.15.49:0/465206981 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f968004f050 con 0x55ee5ac81240 2024-03-18T00:55:39.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.812+0000 7f96987b1740 10 client.16719 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:39.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.812+0000 7f96987b1740 1 -- 172.21.15.49:0/465206981 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee5a8f93f0 con 0x55ee5ac81240 2024-03-18T00:55:39.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.814+0000 7f966effd640 1 -- 172.21.15.49:0/465206981 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f968401d070 con 0x55ee5ac81240 2024-03-18T00:55:39.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.815+0000 7f966effd640 1 -- 172.21.15.49:0/465206981 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f96800326d0 con 0x55ee5ac81240 2024-03-18T00:55:39.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.815+0000 7f96987b1740 10 client.16719 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:39.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.815+0000 7f96987b1740 10 client.16719 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:55:39.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.815+0000 7f96987b1740 20 client.16719 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:39.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.815+0000 7f96987b1740 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55ee5aca7f90 0x55ee5acaa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.815+0000 7f96987b1740 4 client.16719 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:39.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.815+0000 7f96987b1740 1 -- 172.21.15.49:0/465206981 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee5abf29d0 con 0x55ee5aca7f90 2024-03-18T00:55:39.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.818+0000 7f968effd640 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55ee5aca7f90 0x55ee5acaa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.822 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.821+0000 7f968effd640 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55ee5aca7f90 0x55ee5acaa3a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6986dfde8590dd7 server_cookie=b4db28c4ed395ea6 in_seq=0 out_seq=0 2024-03-18T00:55:39.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.834+0000 7f966effd640 10 client.16719 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:55:39.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.834+0000 7f966effd640 1 -- 172.21.15.49:0/465206981 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee5abf29d0 con 0x55ee5aca7f90 2024-03-18T00:55:39.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.834+0000 7f966effd640 10 client.16719 handle_command_reply: tid=0 2024-03-18T00:55:39.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.834+0000 7f96987b1740 1 client.16719 shutdown 2024-03-18T00:55:39.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.835+0000 7f96987b1740 1 -- 172.21.15.49:0/465206981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 msgr2=0x55ee5ac8b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.835+0000 7f96987b1740 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac8b2d0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9680000c00 tx=0x7f96800324a0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.838+0000 7f96987b1740 1 -- 172.21.15.49:0/465206981 shutdown_connections 2024-03-18T00:55:39.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.838+0000 7f96987b1740 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55ee5aca7f90 0x55ee5acaa3a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.838+0000 7f96987b1740 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac863b0 0x55ee5ac92310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.838+0000 7f96987b1740 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac81240 0x55ee5ac8b2d0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.838+0000 7f96987b1740 1 --2- 172.21.15.49:0/465206981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac80850 0x55ee5ac8ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.838+0000 7f96987b1740 1 -- 172.21.15.49:0/465206981 >> 172.21.15.49:0/465206981 conn(0x55ee5ac7bdc0 msgr2=0x55ee5ac7afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.843+0000 7f966effd640 0 client.16719 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:55:39.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.844+0000 7f96987b1740 1 -- 172.21.15.49:0/465206981 shutdown_connections 2024-03-18T00:55:39.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.844+0000 7f96987b1740 1 -- 172.21.15.49:0/465206981 wait complete. 2024-03-18T00:55:39.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.844+0000 7f96987b1740 20 client.16719 trim_cache size 0 max 16384 2024-03-18T00:55:39.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.878+0000 7f96987b1740 1 Processor -- start 2024-03-18T00:55:39.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.878+0000 7f96987b1740 1 -- start start 2024-03-18T00:55:39.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.878+0000 7f96987b1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac8a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.879+0000 7f96987b1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 0x55ee5ac922d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.879+0000 7f96987b1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.879+0000 7f96987b1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f96901a6e00 con 0x55ee5ac863b0 2024-03-18T00:55:39.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.879+0000 7f96987b1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f96901a8940 con 0x55ee5ac80850 2024-03-18T00:55:39.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.879+0000 7f96987b1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96901a74d0 con 0x55ee5ac81240 2024-03-18T00:55:39.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.881+0000 7f968effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.881+0000 7f968effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54274/0 (socket says 172.21.15.49:54274) 2024-03-18T00:55:39.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.881+0000 7f968effd640 1 -- 172.21.15.49:0/2598091176 learned_addr learned my addr 172.21.15.49:0/2598091176 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.881+0000 7f968effd640 1 --2- 172.21.15.49:0/2598091176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac8a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.881+0000 7f968effd640 1 -- 172.21.15.49:0/2598091176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 msgr2=0x55ee5ac922d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.881+0000 7f968effd640 1 --2- 172.21.15.49:0/2598091176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 0x55ee5ac922d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.881+0000 7f968effd640 1 -- 172.21.15.49:0/2598091176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 msgr2=0x55ee5ac8a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.881+0000 7f968effd640 1 --2- 172.21.15.49:0/2598091176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac8a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.881+0000 7f968effd640 1 -- 172.21.15.49:0/2598091176 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5a9c37d0 con 0x55ee5ac863b0 2024-03-18T00:55:39.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.882+0000 7f968effd640 1 --2- 172.21.15.49:0/2598091176 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8ad90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9684009620 tx=0x7f9684008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a845394f9058f946 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.883+0000 7f96757fa640 1 -- 172.21.15.49:0/2598091176 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9684032070 con 0x55ee5ac863b0 2024-03-18T00:55:39.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.883+0000 7f96757fa640 1 -- 172.21.15.49:0/2598091176 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f968401d070 con 0x55ee5ac863b0 2024-03-18T00:55:39.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.883+0000 7f96757fa640 1 -- 172.21.15.49:0/2598091176 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96840042b0 con 0x55ee5ac863b0 2024-03-18T00:55:39.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.884+0000 7f96987b1740 1 -- 172.21.15.49:0/2598091176 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 msgr2=0x55ee5ac8ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.884+0000 7f96987b1740 1 --2- 172.21.15.49:0/2598091176 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8ad90 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9684009620 tx=0x7f9684008040 comp rx=0 tx=0).stop 2024-03-18T00:55:39.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.884+0000 7f96987b1740 1 -- 172.21.15.49:0/2598091176 shutdown_connections 2024-03-18T00:55:39.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.884+0000 7f96987b1740 1 --2- 172.21.15.49:0/2598091176 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8ad90 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.884+0000 7f96987b1740 1 --2- 172.21.15.49:0/2598091176 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 0x55ee5ac922d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.884+0000 7f96987b1740 1 --2- 172.21.15.49:0/2598091176 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac8a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.884+0000 7f96987b1740 1 -- 172.21.15.49:0/2598091176 >> 172.21.15.49:0/2598091176 conn(0x55ee5ac7bdc0 msgr2=0x55ee5ac7c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.885+0000 7f96987b1740 1 -- 172.21.15.49:0/2598091176 shutdown_connections 2024-03-18T00:55:39.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.885+0000 7f96987b1740 1 -- 172.21.15.49:0/2598091176 wait complete. 2024-03-18T00:55:39.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.885+0000 7f96987b1740 1 -- start start 2024-03-18T00:55:39.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.886+0000 7f96987b1740 1 Processor -- start 2024-03-18T00:55:39.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.886+0000 7f96987b1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac92180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.886+0000 7f96987b1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 0x55ee5ac8ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.886+0000 7f96987b1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.886+0000 7f96987b1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f96901a8270 con 0x55ee5ac863b0 2024-03-18T00:55:39.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.886+0000 7f96987b1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f96901a7ba0 con 0x55ee5ac80850 2024-03-18T00:55:39.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.886+0000 7f96987b1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f96901a27c0 con 0x55ee5ac81240 2024-03-18T00:55:39.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.887+0000 7f96987b1740 10 client.0 _ll_register_callbacks cb 0x7f96800096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:39.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.887+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 0x55ee5ac8ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.887+0000 7f968f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 0x55ee5ac8ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59042/0 (socket says 172.21.15.49:59042) 2024-03-18T00:55:39.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.887+0000 7f968f7fe640 1 -- 172.21.15.49:0/1813153793 learned_addr learned my addr 172.21.15.49:0/1813153793 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:39.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.888+0000 7f968effd640 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.888+0000 7f968effd640 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac92180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.889+0000 7f968f7fe640 1 -- 172.21.15.49:0/1813153793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 msgr2=0x55ee5ac92180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.889+0000 7f968f7fe640 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac92180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.889+0000 7f968f7fe640 1 -- 172.21.15.49:0/1813153793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 msgr2=0x55ee5ac8f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.889+0000 7f968f7fe640 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.889+0000 7f968f7fe640 1 -- 172.21.15.49:0/1813153793 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5ad0c2d0 con 0x55ee5ac81240 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.889+0000 7f968f7fe640 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 0x55ee5ac8ec20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f968002c0d0 tx=0x7f96800349a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=547a68fd8f1d6f07 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.890+0000 7f966effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.890+0000 7f966effd640 1 -- 172.21.15.49:0/1813153793 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680058060 con 0x55ee5ac81240 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.890+0000 7f96987b1740 1 -- 172.21.15.49:0/1813153793 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f968401d070 con 0x55ee5ac81240 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.890+0000 7f966effd640 1 -- 172.21.15.49:0/1813153793 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9680032640 con 0x55ee5ac81240 2024-03-18T00:55:39.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.890+0000 7f966effd640 1 -- 172.21.15.49:0/1813153793 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9680043040 con 0x55ee5ac81240 2024-03-18T00:55:39.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.892+0000 7f96987b1740 10 client.26191 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:39.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.892+0000 7f96987b1740 1 -- 172.21.15.49:0/1813153793 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee5ad0d590 con 0x55ee5ac81240 2024-03-18T00:55:39.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.896+0000 7f966effd640 1 -- 172.21.15.49:0/1813153793 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f968401d070 con 0x55ee5ac81240 2024-03-18T00:55:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.896+0000 7f966effd640 1 -- 172.21.15.49:0/1813153793 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f968003f570 con 0x55ee5ac81240 2024-03-18T00:55:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.896+0000 7f96987b1740 10 client.26191 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.896+0000 7f96987b1740 10 client.26191 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:55:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.896+0000 7f96987b1740 20 client.26191 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.896+0000 7f96987b1740 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55ee5ad0de90 0x55ee5ad102b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.896+0000 7f96987b1740 4 client.26191 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:39.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.896+0000 7f96987b1740 1 -- 172.21.15.49:0/1813153793 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee5abecd20 con 0x55ee5ad0de90 2024-03-18T00:55:39.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.899+0000 7f968effd640 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55ee5ad0de90 0x55ee5ad102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.904+0000 7f968effd640 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55ee5ad0de90 0x55ee5ad102b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e93509648d26e5e6 server_cookie=dd83827f53ce6b10 in_seq=0 out_seq=0 2024-03-18T00:55:39.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.904+0000 7f966effd640 10 client.26191 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:55:39.907 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:39.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f966effd640 1 -- 172.21.15.49:0/1813153793 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee5abecd20 con 0x55ee5ad0de90 2024-03-18T00:55:39.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f966effd640 10 client.26191 handle_command_reply: tid=0 2024-03-18T00:55:39.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 client.26191 shutdown 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 -- 172.21.15.49:0/1813153793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 msgr2=0x55ee5ac8ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 0x55ee5ac8ec20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f968002c0d0 tx=0x7f96800349a0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 -- 172.21.15.49:0/1813153793 shutdown_connections 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55ee5ad0de90 0x55ee5ad102b0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee5ac863b0 0x55ee5ac8f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee5ac81240 0x55ee5ac8ec20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 --2- 172.21.15.49:0/1813153793 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee5ac80850 0x55ee5ac92180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 -- 172.21.15.49:0/1813153793 >> 172.21.15.49:0/1813153793 conn(0x55ee5ac7bdc0 msgr2=0x55ee5ac827d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f966effd640 0 client.26191 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 -- 172.21.15.49:0/1813153793 shutdown_connections 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.906+0000 7f96987b1740 1 -- 172.21.15.49:0/1813153793 wait complete. 2024-03-18T00:55:39.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.907+0000 7f96987b1740 20 client.26191 trim_cache size 0 max 16384 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.908+0000 7f9695bc6640 1 -- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f96780a2880 msgr2=0x7f96780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.908+0000 7f9695bc6640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f96780a2880 0x7f96780a4d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9688000f80 tx=0x7f968800b460 comp rx=0 tx=0).stop 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.908+0000 7f9695bc6640 1 -- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9690107560 msgr2=0x7f96901007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.908+0000 7f9695bc6640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9690107560 0x7f96901007f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f968800c5f0 tx=0x7f968800ef90 comp rx=0 tx=0).stop 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.909+0000 7f9695bc6640 1 -- 172.21.15.49:0/1891544359 shutdown_connections 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.909+0000 7f9695bc6640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f96780a2880 0x7f96780a4d40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.909+0000 7f9695bc6640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9690107560 0x7f96901007f0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.909+0000 7f9695bc6640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9690104c30 0x7f96901002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.909+0000 7f9695bc6640 1 --2- 172.21.15.49:0/1891544359 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9690102300 0x7f9690101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:39.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.909+0000 7f9695bc6640 1 -- 172.21.15.49:0/1891544359 >> 172.21.15.49:0/1891544359 conn(0x7f96900fbc90 msgr2=0x7f9690102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:39.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.909+0000 7f9695bc6640 1 -- 172.21.15.49:0/1891544359 shutdown_connections 2024-03-18T00:55:39.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:39.909+0000 7f9695bc6640 1 -- 172.21.15.49:0/1891544359 wait complete. 2024-03-18T00:55:39.941 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:55:40.173 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:39 smithi106 ceph-mon[35899]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 850 MiB/s wr, 2.13k op/s 2024-03-18T00:55:40.173 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:39 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1308776199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:55:40.173 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:55:40.173 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:55:40.173 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:39 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:55:40.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.309+0000 7ff869af0640 1 -- 172.21.15.49:0/2139245449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864072bf0 msgr2=0x7ff864073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:40.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.309+0000 7ff869af0640 1 --2- 172.21.15.49:0/2139245449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864072bf0 0x7ff864073010 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7ff854009900 tx=0x7ff85402bc50 comp rx=0 tx=0).stop 2024-03-18T00:55:40.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.313+0000 7ff869af0640 1 -- 172.21.15.49:0/2139245449 shutdown_connections 2024-03-18T00:55:40.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.313+0000 7ff869af0640 1 --2- 172.21.15.49:0/2139245449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff864109870 0x7ff86410e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.313+0000 7ff869af0640 1 --2- 172.21.15.49:0/2139245449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff864108eb0 0x7ff864109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.313+0000 7ff869af0640 1 --2- 172.21.15.49:0/2139245449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864072bf0 0x7ff864073010 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.313+0000 7ff869af0640 1 -- 172.21.15.49:0/2139245449 >> 172.21.15.49:0/2139245449 conn(0x7ff86406e6c0 msgr2=0x7ff864070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:40.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.319+0000 7ff869af0640 1 -- 172.21.15.49:0/2139245449 shutdown_connections 2024-03-18T00:55:40.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.319+0000 7ff869af0640 1 -- 172.21.15.49:0/2139245449 wait complete. 2024-03-18T00:55:40.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.320+0000 7ff869af0640 1 Processor -- start 2024-03-18T00:55:40.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.320+0000 7ff869af0640 1 -- start start 2024-03-18T00:55:40.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.321+0000 7ff869af0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff864072bf0 0x7ff8641a6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.321+0000 7ff869af0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864108eb0 0x7ff8641a6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.321+0000 7ff869af0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff864109870 0x7ff8641addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.321+0000 7ff869af0640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff86411c200 con 0x7ff864108eb0 2024-03-18T00:55:40.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.321+0000 7ff869af0640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff86411c080 con 0x7ff864109870 2024-03-18T00:55:40.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.321+0000 7ff869af0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff86411c380 con 0x7ff864072bf0 2024-03-18T00:55:40.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.322+0000 7ff8692ef640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff864109870 0x7ff8641addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.322+0000 7ff8692ef640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff864109870 0x7ff8641addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47628/0 (socket says 172.21.15.49:47628) 2024-03-18T00:55:40.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.322+0000 7ff8692ef640 1 -- 172.21.15.49:0/2651482766 learned_addr learned my addr 172.21.15.49:0/2651482766 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:40.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.323+0000 7ff8692ef640 1 -- 172.21.15.49:0/2651482766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff864072bf0 msgr2=0x7ff8641a6840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:55:40.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.323+0000 7ff8692ef640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff864072bf0 0x7ff8641a6840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.323+0000 7ff8692ef640 1 -- 172.21.15.49:0/2651482766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864108eb0 msgr2=0x7ff8641a6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:40.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.323+0000 7ff8692ef640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864108eb0 0x7ff8641a6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.323+0000 7ff8692ef640 1 -- 172.21.15.49:0/2651482766 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8641ae460 con 0x7ff864109870 2024-03-18T00:55:40.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.324+0000 7ff8692ef640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff864109870 0x7ff8641addf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff85c00eaa0 tx=0x7ff85c00ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:40.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.324+0000 7ff861ffb640 1 -- 172.21.15.49:0/2651482766 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c00cdb0 con 0x7ff864109870 2024-03-18T00:55:40.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.324+0000 7ff861ffb640 1 -- 172.21.15.49:0/2651482766 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff85c005020 con 0x7ff864109870 2024-03-18T00:55:40.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.324+0000 7ff861ffb640 1 -- 172.21.15.49:0/2651482766 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85c0116b0 con 0x7ff864109870 2024-03-18T00:55:40.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.326+0000 7ff869af0640 1 -- 172.21.15.49:0/2651482766 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8641ae750 con 0x7ff864109870 2024-03-18T00:55:40.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.326+0000 7ff869af0640 1 -- 172.21.15.49:0/2651482766 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8641aec50 con 0x7ff864109870 2024-03-18T00:55:40.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.334+0000 7ff861ffb640 1 -- 172.21.15.49:0/2651482766 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff85c0051d0 con 0x7ff864109870 2024-03-18T00:55:40.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.338+0000 7ff861ffb640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff84c0a2980 0x7ff84c0a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.339+0000 7ff86da10740 1 Processor -- start 2024-03-18T00:55:40.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.339+0000 7ff868aee640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff84c0a2980 0x7ff84c0a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.340+0000 7ff861ffb640 1 -- 172.21.15.49:0/2651482766 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff85c0ce8f0 con 0x7ff864109870 2024-03-18T00:55:40.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.341+0000 7ff86da10740 1 -- start start 2024-03-18T00:55:40.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.341+0000 7ff86da10740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 0x5598aad92e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.341+0000 7ff86da10740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad93430 0x5598aad97f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.341+0000 7ff86da10740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad985a0 0x5598aad9aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.341+0000 7ff86da10740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5598aacf7490 con 0x5598aad92a40 2024-03-18T00:55:40.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.341+0000 7ff86da10740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5598aac9fc90 con 0x5598aad985a0 2024-03-18T00:55:40.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.341+0000 7ff86da10740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5598aacf3530 con 0x5598aad93430 2024-03-18T00:55:40.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.343+0000 7ff863fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad93430 0x5598aad97f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.343+0000 7ff863fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad93430 0x5598aad97f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59064/0 (socket says 172.21.15.49:59064) 2024-03-18T00:55:40.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.343+0000 7ff863fff640 1 -- 172.21.15.49:0/3571303381 learned_addr learned my addr 172.21.15.49:0/3571303381 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:40.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.343+0000 7ff8692ef640 1 --2- 172.21.15.49:0/3571303381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad985a0 0x5598aad9aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.343+0000 7ff863fff640 1 -- 172.21.15.49:0/3571303381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad985a0 msgr2=0x5598aad9aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:40.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.343+0000 7ff863fff640 1 --2- 172.21.15.49:0/3571303381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad985a0 0x5598aad9aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.343+0000 7ff863fff640 1 -- 172.21.15.49:0/3571303381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 msgr2=0x5598aad92e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:40.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.343+0000 7ff863fff640 1 --2- 172.21.15.49:0/3571303381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 0x5598aad92e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.343+0000 7ff863fff640 1 -- 172.21.15.49:0/3571303381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598aad5a500 con 0x5598aad93430 2024-03-18T00:55:40.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.344+0000 7ff863fff640 1 --2- 172.21.15.49:0/3571303381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad93430 0x5598aad97f30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff8580097b0 tx=0x7ff858009c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78e077cd4e328e23 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:40.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.345+0000 7ff868aee640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff84c0a2980 0x7ff84c0a4e40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff854036040 tx=0x7ff854000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:40.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.346+0000 7ff84affd640 1 -- 172.21.15.49:0/3571303381 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858015020 con 0x5598aad93430 2024-03-18T00:55:40.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.346+0000 7ff84affd640 1 -- 172.21.15.49:0/3571303381 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff858011420 con 0x5598aad93430 2024-03-18T00:55:40.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.347+0000 7ff86da10740 1 -- 172.21.15.49:0/3571303381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad93430 msgr2=0x5598aad97f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:40.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.347+0000 7ff86da10740 1 --2- 172.21.15.49:0/3571303381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad93430 0x5598aad97f30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff8580097b0 tx=0x7ff858009c70 comp rx=0 tx=0).stop 2024-03-18T00:55:40.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.350+0000 7ff86da10740 1 -- 172.21.15.49:0/3571303381 shutdown_connections 2024-03-18T00:55:40.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.350+0000 7ff86da10740 1 --2- 172.21.15.49:0/3571303381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad985a0 0x5598aad9aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.350+0000 7ff86da10740 1 --2- 172.21.15.49:0/3571303381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad93430 0x5598aad97f30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.350+0000 7ff86da10740 1 --2- 172.21.15.49:0/3571303381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 0x5598aad92e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.350+0000 7ff86da10740 1 -- 172.21.15.49:0/3571303381 >> 172.21.15.49:0/3571303381 conn(0x5598aad8dfb0 msgr2=0x5598aad8e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:40.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.352+0000 7ff86da10740 1 -- 172.21.15.49:0/3571303381 shutdown_connections 2024-03-18T00:55:40.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.352+0000 7ff86da10740 1 -- 172.21.15.49:0/3571303381 wait complete. 2024-03-18T00:55:40.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.353+0000 7ff86da10740 1 -- start start 2024-03-18T00:55:40.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.353+0000 7ff86da10740 1 Processor -- start 2024-03-18T00:55:40.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.353+0000 7ff86da10740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad92a40 0x5598aad9d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.353+0000 7ff86da10740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad9d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.353+0000 7ff86da10740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad985a0 0x5598aada46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.353+0000 7ff86da10740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5598aad9b5a0 con 0x5598aad985a0 2024-03-18T00:55:40.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.353+0000 7ff86da10740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5598aacb3230 con 0x5598aad93430 2024-03-18T00:55:40.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.353+0000 7ff86da10740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5598aad9b720 con 0x5598aad92a40 2024-03-18T00:55:40.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.354+0000 7ff86da10740 10 client.0 _ll_register_callbacks cb 0x5598aacef010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:40.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.354+0000 7ff863fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad9d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.354+0000 7ff863fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad9d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47660/0 (socket says 172.21.15.49:47660) 2024-03-18T00:55:40.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.354+0000 7ff863fff640 1 -- 172.21.15.49:0/3842820284 learned_addr learned my addr 172.21.15.49:0/3842820284 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:40.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.354+0000 7ff8692ef640 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad985a0 0x5598aada46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.354+0000 7ff863fff640 1 -- 172.21.15.49:0/3842820284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad92a40 msgr2=0x5598aad9d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:40.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.354+0000 7ff863fff640 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad92a40 0x5598aad9d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.354+0000 7ff863fff640 1 -- 172.21.15.49:0/3842820284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad985a0 msgr2=0x5598aada46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:40.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.355+0000 7ff863fff640 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad985a0 0x5598aada46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.355+0000 7ff863fff640 1 -- 172.21.15.49:0/3842820284 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598aacc2100 con 0x5598aad93430 2024-03-18T00:55:40.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.361+0000 7ff863fff640 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad9d690 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5598aad93ea0 tx=0x7ff85800bb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ebc4dd4fe434238 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:40.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.363+0000 7ff843fff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:55:40.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.371+0000 7ff843fff640 1 -- 172.21.15.49:0/3842820284 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858015040 con 0x5598aad93430 2024-03-18T00:55:40.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.371+0000 7ff843fff640 1 -- 172.21.15.49:0/3842820284 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff858010a00 con 0x5598aad93430 2024-03-18T00:55:40.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.371+0000 7ff86da10740 1 -- 172.21.15.49:0/3842820284 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5598aad049d0 con 0x5598aad93430 2024-03-18T00:55:40.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.375+0000 7ff843fff640 1 -- 172.21.15.49:0/3842820284 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff858028050 con 0x5598aad93430 2024-03-18T00:55:40.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.375+0000 7ff86da10740 10 client.26102 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:40.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.375+0000 7ff86da10740 1 -- 172.21.15.49:0/3842820284 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5598aaa0b3f0 con 0x5598aad93430 2024-03-18T00:55:40.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.377+0000 7ff843fff640 1 -- 172.21.15.49:0/3842820284 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5598aad049d0 con 0x5598aad93430 2024-03-18T00:55:40.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.377+0000 7ff843fff640 1 -- 172.21.15.49:0/3842820284 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff858023c70 con 0x5598aad93430 2024-03-18T00:55:40.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.377+0000 7ff86da10740 10 client.26102 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:40.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.377+0000 7ff86da10740 10 client.26102 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:55:40.377 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.377+0000 7ff86da10740 20 client.26102 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:40.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.377+0000 7ff86da10740 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5598aada9390 0x5598aadab7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.377+0000 7ff86da10740 4 client.26102 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:40.378 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.377+0000 7ff86da10740 1 -- 172.21.15.49:0/3842820284 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5598aacfed20 con 0x5598aada9390 2024-03-18T00:55:40.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.379+0000 7ff868aee640 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5598aada9390 0x5598aadab7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.383+0000 7ff868aee640 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5598aada9390 0x5598aadab7f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d05973187fc434c9 server_cookie=2856bf2a0ce66cd1 in_seq=0 out_seq=0 2024-03-18T00:55:40.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.383+0000 7ff843fff640 10 client.26102 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:55:40.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.385+0000 7ff843fff640 1 -- 172.21.15.49:0/3842820284 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5598aacfed20 con 0x5598aada9390 2024-03-18T00:55:40.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.385+0000 7ff843fff640 10 client.26102 handle_command_reply: tid=0 2024-03-18T00:55:40.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.387+0000 7ff86da10740 1 client.26102 shutdown 2024-03-18T00:55:40.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.388+0000 7ff86da10740 1 -- 172.21.15.49:0/3842820284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 msgr2=0x5598aad9d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:40.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.388+0000 7ff86da10740 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad9d690 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x5598aad93ea0 tx=0x7ff85800bb00 comp rx=0 tx=0).stop 2024-03-18T00:55:40.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.393+0000 7ff86da10740 1 -- 172.21.15.49:0/3842820284 shutdown_connections 2024-03-18T00:55:40.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.393+0000 7ff86da10740 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5598aada9390 0x5598aadab7f0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.393+0000 7ff86da10740 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad9d690 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.393+0000 7ff86da10740 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad985a0 0x5598aada46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.393+0000 7ff86da10740 1 --2- 172.21.15.49:0/3842820284 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad92a40 0x5598aad9d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.393+0000 7ff843fff640 0 client.26102 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:55:40.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.393+0000 7ff86da10740 1 -- 172.21.15.49:0/3842820284 >> 172.21.15.49:0/3842820284 conn(0x5598aad8dfb0 msgr2=0x5598aad8d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:40.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.396+0000 7ff86da10740 1 -- 172.21.15.49:0/3842820284 shutdown_connections 2024-03-18T00:55:40.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.397+0000 7ff86da10740 1 -- 172.21.15.49:0/3842820284 wait complete. 2024-03-18T00:55:40.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.398+0000 7ff86da10740 20 client.26102 trim_cache size 0 max 16384 2024-03-18T00:55:40.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.433+0000 7ff86da10740 1 Processor -- start 2024-03-18T00:55:40.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.433+0000 7ff86da10740 1 -- start start 2024-03-18T00:55:40.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.434+0000 7ff86da10740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 0x5598aad94ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.435+0000 7ff86da10740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad95020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.435+0000 7ff86da10740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.436+0000 7ff868aee640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 0x5598aad94ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.436+0000 7ff868aee640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 0x5598aad94ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54320/0 (socket says 172.21.15.49:54320) 2024-03-18T00:55:40.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.436+0000 7ff868aee640 1 -- 172.21.15.49:0/960051047 learned_addr learned my addr 172.21.15.49:0/960051047 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:40.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.436+0000 7ff86da10740 1 -- 172.21.15.49:0/960051047 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b7b30 con 0x5598aad92a40 2024-03-18T00:55:40.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.436+0000 7ff863fff640 1 --2- 172.21.15.49:0/960051047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad95020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.436+0000 7ff8692ef640 1 --2- 172.21.15.49:0/960051047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.436+0000 7ff86da10740 1 -- 172.21.15.49:0/960051047 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b9670 con 0x5598aad93430 2024-03-18T00:55:40.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.436+0000 7ff86da10740 1 -- 172.21.15.49:0/960051047 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b8200 con 0x5598aad985a0 2024-03-18T00:55:40.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.440+0000 7ff868aee640 1 -- 172.21.15.49:0/960051047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 msgr2=0x5598aada4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:40.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.440+0000 7ff868aee640 1 --2- 172.21.15.49:0/960051047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.440+0000 7ff868aee640 1 -- 172.21.15.49:0/960051047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 msgr2=0x5598aad95020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:40.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.440+0000 7ff868aee640 1 --2- 172.21.15.49:0/960051047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad95020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.441+0000 7ff868aee640 1 -- 172.21.15.49:0/960051047 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598aaad57d0 con 0x5598aad92a40 2024-03-18T00:55:40.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.441+0000 7ff868aee640 1 --2- 172.21.15.49:0/960051047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 0x5598aad94ae0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff85402c730 tx=0x7ff854009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38c7899685586f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:40.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.447+0000 7ff84a7fc640 1 -- 172.21.15.49:0/960051047 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854047040 con 0x5598aad92a40 2024-03-18T00:55:40.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.447+0000 7ff84a7fc640 1 -- 172.21.15.49:0/960051047 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8540054e0 con 0x5598aad92a40 2024-03-18T00:55:40.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.449+0000 7ff86da10740 1 -- 172.21.15.49:0/960051047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 msgr2=0x5598aad94ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:40.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.449+0000 7ff86da10740 1 --2- 172.21.15.49:0/960051047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 0x5598aad94ae0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff85402c730 tx=0x7ff854009260 comp rx=0 tx=0).stop 2024-03-18T00:55:40.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.450+0000 7ff86da10740 1 -- 172.21.15.49:0/960051047 shutdown_connections 2024-03-18T00:55:40.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.450+0000 7ff86da10740 1 --2- 172.21.15.49:0/960051047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aad93430 0x5598aad95020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.450+0000 7ff86da10740 1 --2- 172.21.15.49:0/960051047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.450+0000 7ff86da10740 1 --2- 172.21.15.49:0/960051047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad92a40 0x5598aad94ae0 secure :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff85402c730 tx=0x7ff854009260 comp rx=0 tx=0).stop 2024-03-18T00:55:40.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.450+0000 7ff86da10740 1 -- 172.21.15.49:0/960051047 >> 172.21.15.49:0/960051047 conn(0x5598aad8dfb0 msgr2=0x5598aad88fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:40.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.457+0000 7ff86da10740 1 -- 172.21.15.49:0/960051047 shutdown_connections 2024-03-18T00:55:40.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.459+0000 7ff86da10740 1 -- 172.21.15.49:0/960051047 wait complete. 2024-03-18T00:55:40.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.460+0000 7ff86da10740 1 -- start start 2024-03-18T00:55:40.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.460+0000 7ff86da10740 1 Processor -- start 2024-03-18T00:55:40.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.462+0000 7ff86da10740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad93430 0x5598aada1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.462+0000 7ff86da10740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.462+0000 7ff86da10740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aae0b710 0x5598aae0db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.462+0000 7ff86da10740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b8fa0 con 0x5598aad93430 2024-03-18T00:55:40.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.463+0000 7ff86da10740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b88d0 con 0x5598aae0b710 2024-03-18T00:55:40.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.463+0000 7ff86da10740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff8641b5b10 con 0x5598aad985a0 2024-03-18T00:55:40.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.463+0000 7ff86da10740 10 client.0 _ll_register_callbacks cb 0x7ff858009600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:40.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.464+0000 7ff8692ef640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aae0b710 0x5598aae0db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.466+0000 7ff863fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.466+0000 7ff863fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59094/0 (socket says 172.21.15.49:59094) 2024-03-18T00:55:40.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.466+0000 7ff863fff640 1 -- 172.21.15.49:0/2916356150 learned_addr learned my addr 172.21.15.49:0/2916356150 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:40.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.469+0000 7ff868aee640 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad93430 0x5598aada1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.469+0000 7ff863fff640 1 -- 172.21.15.49:0/2916356150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aae0b710 msgr2=0x5598aae0db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:40.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.469+0000 7ff863fff640 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aae0b710 0x5598aae0db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.469+0000 7ff863fff640 1 -- 172.21.15.49:0/2916356150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad93430 msgr2=0x5598aada1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:40.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.469+0000 7ff863fff640 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad93430 0x5598aada1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.469+0000 7ff863fff640 1 -- 172.21.15.49:0/2916356150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5598aada48b0 con 0x5598aad985a0 2024-03-18T00:55:40.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.471+0000 7ff863fff640 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada1940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff858000c00 tx=0x7ff8580109b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6625984243c97a19 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:40.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.473+0000 7ff843fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:55:40.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.473+0000 7ff843fff640 1 -- 172.21.15.49:0/2916356150 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff858031020 con 0x5598aad985a0 2024-03-18T00:55:40.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.473+0000 7ff843fff640 1 -- 172.21.15.49:0/2916356150 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff8580116e0 con 0x5598aad985a0 2024-03-18T00:55:40.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.473+0000 7ff86da10740 1 -- 172.21.15.49:0/2916356150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8540054e0 con 0x5598aad985a0 2024-03-18T00:55:40.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.475+0000 7ff843fff640 1 -- 172.21.15.49:0/2916356150 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff858015040 con 0x5598aad985a0 2024-03-18T00:55:40.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.478+0000 7ff86da10740 10 client.26200 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:40.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.478+0000 7ff86da10740 1 -- 172.21.15.49:0/2916356150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5598aae0f020 con 0x5598aad985a0 2024-03-18T00:55:40.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.478+0000 7ff843fff640 1 -- 172.21.15.49:0/2916356150 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8540054e0 con 0x5598aad985a0 2024-03-18T00:55:40.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.478+0000 7ff843fff640 1 -- 172.21.15.49:0/2916356150 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff858034020 con 0x5598aad985a0 2024-03-18T00:55:40.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.481+0000 7ff86da10740 10 client.26200 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:40.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.481+0000 7ff86da10740 10 client.26200 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:55:40.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.481+0000 7ff86da10740 20 client.26200 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:40.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.481+0000 7ff86da10740 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5598aae0f910 0x5598aae11d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:40.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.481+0000 7ff86da10740 4 client.26200 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:40.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.481+0000 7ff86da10740 1 -- 172.21.15.49:0/2916356150 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5598aac7c6a0 con 0x5598aae0f910 2024-03-18T00:55:40.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.484+0000 7ff868aee640 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5598aae0f910 0x5598aae11d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:40.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.499+0000 7ff843fff640 10 client.26200 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:55:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.499+0000 7ff868aee640 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5598aae0f910 0x5598aae11d00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6f13fb4c556083cc server_cookie=a030e14cd16df71e in_seq=0 out_seq=0 2024-03-18T00:55:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.501+0000 7ff843fff640 1 -- 172.21.15.49:0/2916356150 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5598aac7c6a0 con 0x5598aae0f910 2024-03-18T00:55:40.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.501+0000 7ff843fff640 10 client.26200 handle_command_reply: tid=0 2024-03-18T00:55:40.502 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.502+0000 7ff86da10740 1 client.26200 shutdown 2024-03-18T00:55:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.503+0000 7ff86da10740 1 -- 172.21.15.49:0/2916356150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 msgr2=0x5598aada1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.503+0000 7ff86da10740 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada1940 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff858000c00 tx=0x7ff8580109b0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.503+0000 7ff86da10740 1 -- 172.21.15.49:0/2916356150 shutdown_connections 2024-03-18T00:55:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.503+0000 7ff86da10740 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x5598aae0f910 0x5598aae11d00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.503+0000 7ff86da10740 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5598aae0b710 0x5598aae0db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.503+0000 7ff86da10740 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598aad985a0 0x5598aada1940 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.503+0000 7ff86da10740 1 --2- 172.21.15.49:0/2916356150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5598aad93430 0x5598aada1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.503+0000 7ff843fff640 0 client.26200 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:55:40.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.503+0000 7ff86da10740 1 -- 172.21.15.49:0/2916356150 >> 172.21.15.49:0/2916356150 conn(0x5598aad8dfb0 msgr2=0x5598aad94ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:40.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.504+0000 7ff86da10740 1 -- 172.21.15.49:0/2916356150 shutdown_connections 2024-03-18T00:55:40.504 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.504+0000 7ff86da10740 1 -- 172.21.15.49:0/2916356150 wait complete. 2024-03-18T00:55:40.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.506+0000 7ff86da10740 20 client.26200 trim_cache size 0 max 16384 2024-03-18T00:55:40.508 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:40.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.521+0000 7ff869af0640 1 -- 172.21.15.49:0/2651482766 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff84c0a2980 msgr2=0x7ff84c0a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:40.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.521+0000 7ff869af0640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff84c0a2980 0x7ff84c0a4e40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff854036040 tx=0x7ff854000bc0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.521+0000 7ff869af0640 1 -- 172.21.15.49:0/2651482766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff864109870 msgr2=0x7ff8641addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:40.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.521+0000 7ff869af0640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff864109870 0x7ff8641addf0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff85c00eaa0 tx=0x7ff85c00ef60 comp rx=0 tx=0).stop 2024-03-18T00:55:40.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.525+0000 7ff869af0640 1 -- 172.21.15.49:0/2651482766 shutdown_connections 2024-03-18T00:55:40.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.525+0000 7ff869af0640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff84c0a2980 0x7ff84c0a4e40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.525+0000 7ff869af0640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff864109870 0x7ff8641addf0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.525+0000 7ff869af0640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff864108eb0 0x7ff8641a6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.525+0000 7ff869af0640 1 --2- 172.21.15.49:0/2651482766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff864072bf0 0x7ff8641a6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:40.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.525+0000 7ff869af0640 1 -- 172.21.15.49:0/2651482766 >> 172.21.15.49:0/2651482766 conn(0x7ff86406e6c0 msgr2=0x7ff864070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:40.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.531+0000 7ff869af0640 1 -- 172.21.15.49:0/2651482766 shutdown_connections 2024-03-18T00:55:40.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:40.532+0000 7ff869af0640 1 -- 172.21.15.49:0/2651482766 wait complete. 2024-03-18T00:55:40.575 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:55:41.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.115+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3123000185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03a8101170 msgr2=0x7f03a8105a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:41.118 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.115+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3123000185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03a8101170 0x7f03a8105a30 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009ca0 tx=0x7f03a002c100 comp rx=0 tx=0).stop 2024-03-18T00:55:41.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.116+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3123000185 shutdown_connections 2024-03-18T00:55:41.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.116+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3123000185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03a8101170 0x7f03a8105a30 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.116+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3123000185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a81007d0 0x7f03a8100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.116+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3123000185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03a80ff6a0 0x7f03a80ffac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.116+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3123000185 >> 172.21.15.49:0/3123000185 conn(0x7f03a80fafa0 msgr2=0x7f03a80fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:41.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.116+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3123000185 shutdown_connections 2024-03-18T00:55:41.119 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.117+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3123000185 wait complete. 2024-03-18T00:55:41.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.152+0000 7f03ae8c6640 1 Processor -- start 2024-03-18T00:55:41.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.156+0000 7f03ae8c6640 1 -- start start 2024-03-18T00:55:41.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.157+0000 7f03ae8c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a80ff6a0 0x7f03a8070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.157+0000 7f03ae8c6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03a81007d0 0x7f03a8070640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.157+0000 7f03ae8c6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03a8101170 0x7f03a81ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.157+0000 7f03ae8c6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f03a8113ac0 con 0x7f03a8101170 2024-03-18T00:55:41.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.157+0000 7f03ae8c6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f03a8113940 con 0x7f03a81007d0 2024-03-18T00:55:41.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.157+0000 7f03ae8c6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f03a8113c40 con 0x7f03a80ff6a0 2024-03-18T00:55:41.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.159+0000 7f03ace63640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03a8101170 0x7f03a81ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.159+0000 7f03a77fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03a81007d0 0x7f03a8070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.159+0000 7f03ace63640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03a8101170 0x7f03a81ae1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54338/0 (socket says 172.21.15.49:54338) 2024-03-18T00:55:41.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.159+0000 7f03a7fff640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a80ff6a0 0x7f03a8070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.159+0000 7f03ace63640 1 -- 172.21.15.49:0/3581988861 learned_addr learned my addr 172.21.15.49:0/3581988861 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:41.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.160+0000 7f03a77fe640 1 -- 172.21.15.49:0/3581988861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a80ff6a0 msgr2=0x7f03a8070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:41.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.160+0000 7f03a77fe640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a80ff6a0 0x7f03a8070100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.160+0000 7f03a77fe640 1 -- 172.21.15.49:0/3581988861 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03a8101170 msgr2=0x7f03a81ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:41.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.160+0000 7f03a77fe640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03a8101170 0x7f03a81ae1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.160+0000 7f03a77fe640 1 -- 172.21.15.49:0/3581988861 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a80737c0 con 0x7f03a81007d0 2024-03-18T00:55:41.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.169+0000 7f03a77fe640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03a81007d0 0x7f03a8070640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b3f0 tx=0x7f039c00b7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:41.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.182+0000 7f03a57fa640 1 -- 172.21.15.49:0/3581988861 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c015020 con 0x7f03a81007d0 2024-03-18T00:55:41.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.183+0000 7f03a57fa640 1 -- 172.21.15.49:0/3581988861 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f039c011d10 con 0x7f03a81007d0 2024-03-18T00:55:41.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.183+0000 7f03a57fa640 1 -- 172.21.15.49:0/3581988861 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c010e60 con 0x7f03a81007d0 2024-03-18T00:55:41.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.183+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3581988861 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a81ae7d0 con 0x7f03a81007d0 2024-03-18T00:55:41.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.184+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3581988861 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a81aed50 con 0x7f03a81007d0 2024-03-18T00:55:41.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.209+0000 7f03a57fa640 1 -- 172.21.15.49:0/3581988861 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f039c018020 con 0x7f03a81007d0 2024-03-18T00:55:41.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.214+0000 7f03a57fa640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f03900a29d0 0x7f03900a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.214+0000 7f03a57fa640 1 -- 172.21.15.49:0/3581988861 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f039c0cf2e0 con 0x7f03a81007d0 2024-03-18T00:55:41.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.216+0000 7f03a7fff640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f03900a29d0 0x7f03900a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.312+0000 7f03a7fff640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f03900a29d0 0x7f03900a4e90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f039800abe0 tx=0x7f039800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:41.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.323+0000 7f03b1592740 1 Processor -- start 2024-03-18T00:55:41.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.323+0000 7f03b1592740 1 -- start start 2024-03-18T00:55:41.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.324+0000 7f03b1592740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963987860 0x560963987c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.324+0000 7f03b1592740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963988250 0x56096398cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.324+0000 7f03b1592740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x56096398f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.324+0000 7f03b1592740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5609638db490 con 0x56096398d3c0 2024-03-18T00:55:41.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.324+0000 7f03b1592740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560963883c90 con 0x560963987860 2024-03-18T00:55:41.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.324+0000 7f03b1592740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5609638d7530 con 0x560963988250 2024-03-18T00:55:41.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.325+0000 7f03a7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963987860 0x560963987c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.326+0000 7f03ace63640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x56096398f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.326+0000 7f03ace63640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x56096398f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54352/0 (socket says 172.21.15.49:54352) 2024-03-18T00:55:41.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.326+0000 7f03ace63640 1 -- 172.21.15.49:0/2648977894 learned_addr learned my addr 172.21.15.49:0/2648977894 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:41.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.328+0000 7f03a77fe640 1 --2- 172.21.15.49:0/2648977894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963988250 0x56096398cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.328+0000 7f03ace63640 1 -- 172.21.15.49:0/2648977894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963988250 msgr2=0x56096398cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:41.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.328+0000 7f03ace63640 1 --2- 172.21.15.49:0/2648977894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963988250 0x56096398cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.328+0000 7f03ace63640 1 -- 172.21.15.49:0/2648977894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963987860 msgr2=0x560963987c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:41.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.328+0000 7f03ace63640 1 --2- 172.21.15.49:0/2648977894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963987860 0x560963987c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.328+0000 7f03ace63640 1 -- 172.21.15.49:0/2648977894 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56096393e500 con 0x56096398d3c0 2024-03-18T00:55:41.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.330+0000 7f03ace63640 1 --2- 172.21.15.49:0/2648977894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x56096398f850 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009800 tx=0x7f03a0009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28db6e991906f06f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:41.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.331+0000 7f038e7fc640 1 -- 172.21.15.49:0/2648977894 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a0043020 con 0x56096398d3c0 2024-03-18T00:55:41.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.331+0000 7f038e7fc640 1 -- 172.21.15.49:0/2648977894 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a0003970 con 0x56096398d3c0 2024-03-18T00:55:41.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.331+0000 7f038e7fc640 1 -- 172.21.15.49:0/2648977894 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a0039040 con 0x56096398d3c0 2024-03-18T00:55:41.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.333+0000 7f03b1592740 1 -- 172.21.15.49:0/2648977894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 msgr2=0x56096398f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:41.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.333+0000 7f03b1592740 1 --2- 172.21.15.49:0/2648977894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x56096398f850 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009800 tx=0x7f03a0009830 comp rx=0 tx=0).stop 2024-03-18T00:55:41.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.333+0000 7f03b1592740 1 -- 172.21.15.49:0/2648977894 shutdown_connections 2024-03-18T00:55:41.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.333+0000 7f03b1592740 1 --2- 172.21.15.49:0/2648977894 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x56096398f850 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.333+0000 7f03b1592740 1 --2- 172.21.15.49:0/2648977894 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963988250 0x56096398cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.333+0000 7f03b1592740 1 --2- 172.21.15.49:0/2648977894 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963987860 0x560963987c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.333+0000 7f03b1592740 1 -- 172.21.15.49:0/2648977894 >> 172.21.15.49:0/2648977894 conn(0x560963971dc0 msgr2=0x5609639721c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:41.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.335+0000 7f03b1592740 1 -- 172.21.15.49:0/2648977894 shutdown_connections 2024-03-18T00:55:41.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.336+0000 7f03b1592740 1 -- 172.21.15.49:0/2648977894 wait complete. 2024-03-18T00:55:41.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.390+0000 7f03b1592740 1 -- start start 2024-03-18T00:55:41.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.395+0000 7f03b1592740 1 Processor -- start 2024-03-18T00:55:41.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.395+0000 7f03b1592740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 0x560963991d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.395+0000 7f03b1592740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963992260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.395+0000 7f03b1592740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56096398d3c0 0x5609639992a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.395+0000 7f03b1592740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x560963990310 con 0x560963988250 2024-03-18T00:55:41.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.395+0000 7f03b1592740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560963897230 con 0x56096398d3c0 2024-03-18T00:55:41.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.395+0000 7f03b1592740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560963990490 con 0x560963987860 2024-03-18T00:55:41.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.396+0000 7f03a77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963992260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.396+0000 7f03a77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963992260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54354/0 (socket says 172.21.15.49:54354) 2024-03-18T00:55:41.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.396+0000 7f03a77fe640 1 -- 172.21.15.49:0/1084299006 learned_addr learned my addr 172.21.15.49:0/1084299006 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:41.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.397+0000 7f03a77fe640 1 -- 172.21.15.49:0/1084299006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 msgr2=0x560963991d20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:41.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.397+0000 7f03a77fe640 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 0x560963991d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.397+0000 7f03a77fe640 1 -- 172.21.15.49:0/1084299006 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56096398d3c0 msgr2=0x5609639992a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:41.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.397+0000 7f03a77fe640 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56096398d3c0 0x5609639992a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.397+0000 7f03a77fe640 1 -- 172.21.15.49:0/1084299006 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609638a6100 con 0x560963988250 2024-03-18T00:55:41.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.398+0000 7f03a77fe640 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963992260 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f039c00bcf0 tx=0x7f039c09c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c021d15ad800f56 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:41.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.399+0000 7f03877fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:41.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.399+0000 7f03877fe640 1 -- 172.21.15.49:0/1084299006 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c0b3020 con 0x560963988250 2024-03-18T00:55:41.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.399+0000 7f03877fe640 1 -- 172.21.15.49:0/1084299006 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f039c098270 con 0x560963988250 2024-03-18T00:55:41.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.404+0000 7f03b1592740 10 client.0 _ll_register_callbacks cb 0x5609638d3010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:41.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.404+0000 7f03b1592740 1 -- 172.21.15.49:0/1084299006 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03a0003970 con 0x560963988250 2024-03-18T00:55:41.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.405+0000 7f03877fe640 1 -- 172.21.15.49:0/1084299006 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f039c09fd50 con 0x560963988250 2024-03-18T00:55:41.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.405+0000 7f03b1592740 10 client.16749 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:41.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.405+0000 7f03b1592740 1 -- 172.21.15.49:0/1084299006 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5609635ef3f0 con 0x560963988250 2024-03-18T00:55:41.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.407+0000 7f03877fe640 1 -- 172.21.15.49:0/1084299006 <== mon.0 v2:172.21.15.49:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f039c0a3f10 con 0x560963988250 2024-03-18T00:55:41.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.407+0000 7f03b1592740 10 client.16749 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:41.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.407+0000 7f03b1592740 10 client.16749 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:55:41.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.407+0000 7f03b1592740 20 client.16749 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:41.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.407+0000 7f03b1592740 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x56096399dab0 0x56096399fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.407+0000 7f03b1592740 4 client.16749 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:41.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.407+0000 7f03b1592740 1 -- 172.21.15.49:0/1084299006 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5609638e89d0 con 0x56096399dab0 2024-03-18T00:55:41.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.410+0000 7f03a7fff640 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x56096399dab0 0x56096399fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.417+0000 7f03877fe640 10 client.16749 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:55:41.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.417+0000 7f03a7fff640 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x56096399dab0 0x56096399fea0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74cde682207ae7a2 server_cookie=b84c97547796849 in_seq=0 out_seq=0 2024-03-18T00:55:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03877fe640 1 -- 172.21.15.49:0/1084299006 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5609638e89d0 con 0x56096399dab0 2024-03-18T00:55:41.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03877fe640 10 client.16749 handle_command_reply: tid=0 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 client.16749 shutdown 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 -- 172.21.15.49:0/1084299006 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 msgr2=0x560963992260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963992260 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f039c00bcf0 tx=0x7f039c09c3c0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 -- 172.21.15.49:0/1084299006 shutdown_connections 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x56096399dab0 0x56096399fea0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56096398d3c0 0x5609639992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963992260 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 --2- 172.21.15.49:0/1084299006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 0x560963991d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 -- 172.21.15.49:0/1084299006 >> 172.21.15.49:0/1084299006 conn(0x560963971dc0 msgr2=0x560963970fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03877fe640 0 client.16749 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 -- 172.21.15.49:0/1084299006 shutdown_connections 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.419+0000 7f03b1592740 1 -- 172.21.15.49:0/1084299006 wait complete. 2024-03-18T00:55:41.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.420+0000 7f03b1592740 20 client.16749 trim_cache size 0 max 16384 2024-03-18T00:55:41.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03b1592740 1 Processor -- start 2024-03-18T00:55:41.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03b1592740 1 -- start start 2024-03-18T00:55:41.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03b1592740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 0x560963970750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03b1592740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963988250 0x560963999260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03b1592740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x560963970c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03b1592740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81af400 con 0x56096398d3c0 2024-03-18T00:55:41.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03b1592740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81b0f40 con 0x560963988250 2024-03-18T00:55:41.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03b1592740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81afad0 con 0x560963987860 2024-03-18T00:55:41.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03ace63640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x560963970c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03ace63640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x560963970c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54358/0 (socket says 172.21.15.49:54358) 2024-03-18T00:55:41.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03ace63640 1 -- 172.21.15.49:0/3214009447 learned_addr learned my addr 172.21.15.49:0/3214009447 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:41.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.459+0000 7f03a77fe640 1 --2- 172.21.15.49:0/3214009447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963988250 0x560963999260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.460+0000 7f03ace63640 1 -- 172.21.15.49:0/3214009447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 msgr2=0x560963970750 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:41.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.460+0000 7f03ace63640 1 --2- 172.21.15.49:0/3214009447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 0x560963970750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.460+0000 7f03ace63640 1 -- 172.21.15.49:0/3214009447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963988250 msgr2=0x560963999260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:41.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.460+0000 7f03ace63640 1 --2- 172.21.15.49:0/3214009447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963988250 0x560963999260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.460+0000 7f03ace63640 1 -- 172.21.15.49:0/3214009447 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609636b97d0 con 0x56096398d3c0 2024-03-18T00:55:41.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.460+0000 7f03ace63640 1 --2- 172.21.15.49:0/3214009447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x560963970c90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f03a00098a0 tx=0x7f03a0005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbca06e2aa813 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:41.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.460+0000 7f038dffb640 1 -- 172.21.15.49:0/3214009447 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a004d070 con 0x56096398d3c0 2024-03-18T00:55:41.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.460+0000 7f038dffb640 1 -- 172.21.15.49:0/3214009447 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a003f470 con 0x56096398d3c0 2024-03-18T00:55:41.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.460+0000 7f038dffb640 1 -- 172.21.15.49:0/3214009447 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a0039070 con 0x56096398d3c0 2024-03-18T00:55:41.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.461+0000 7f03b1592740 1 -- 172.21.15.49:0/3214009447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 msgr2=0x560963970c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:41.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.461+0000 7f03b1592740 1 --2- 172.21.15.49:0/3214009447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x560963970c90 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f03a00098a0 tx=0x7f03a0005de0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.462+0000 7f03b1592740 1 -- 172.21.15.49:0/3214009447 shutdown_connections 2024-03-18T00:55:41.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.462+0000 7f03b1592740 1 --2- 172.21.15.49:0/3214009447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560963988250 0x560963999260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.462+0000 7f03b1592740 1 --2- 172.21.15.49:0/3214009447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56096398d3c0 0x560963970c90 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.462+0000 7f03b1592740 1 --2- 172.21.15.49:0/3214009447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 0x560963970750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.462+0000 7f03b1592740 1 -- 172.21.15.49:0/3214009447 >> 172.21.15.49:0/3214009447 conn(0x560963971dc0 msgr2=0x560963989750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:41.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.462+0000 7f03b1592740 1 -- 172.21.15.49:0/3214009447 shutdown_connections 2024-03-18T00:55:41.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.462+0000 7f03b1592740 1 -- 172.21.15.49:0/3214009447 wait complete. 2024-03-18T00:55:41.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.462+0000 7f03b1592740 1 -- start start 2024-03-18T00:55:41.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.463+0000 7f03b1592740 1 Processor -- start 2024-03-18T00:55:41.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.463+0000 7f03b1592740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 0x560963996940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03b1592740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963996e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03b1592740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56096398d3c0 0x560963997e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03b1592740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81b0870 con 0x560963988250 2024-03-18T00:55:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03b1592740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f03a81b01a0 con 0x56096398d3c0 2024-03-18T00:55:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03b1592740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f03a810ddc0 con 0x560963987860 2024-03-18T00:55:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03a77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963996e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03b1592740 10 client.0 _ll_register_callbacks cb 0x7f039c097ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03a77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963996e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54374/0 (socket says 172.21.15.49:54374) 2024-03-18T00:55:41.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03a77fe640 1 -- 172.21.15.49:0/1280413583 learned_addr learned my addr 172.21.15.49:0/1280413583 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:41.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03a77fe640 1 -- 172.21.15.49:0/1280413583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 msgr2=0x560963996940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:41.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.464+0000 7f03a77fe640 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 0x560963996940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.465+0000 7f03a77fe640 1 -- 172.21.15.49:0/1280413583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56096398d3c0 msgr2=0x560963997e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.465+0000 7f03a77fe640 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56096398d3c0 0x560963997e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.465+0000 7f03a77fe640 1 -- 172.21.15.49:0/1280413583 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560963998430 con 0x560963988250 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.465+0000 7f03a77fe640 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963996e80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f039c011f70 tx=0x7f039c0ccc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9676fd0ce1e532a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.465+0000 7f03877fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.466+0000 7f03877fe640 1 -- 172.21.15.49:0/1280413583 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c0b3040 con 0x560963988250 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.466+0000 7f03877fe640 1 -- 172.21.15.49:0/1280413583 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f039c09b260 con 0x560963988250 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.466+0000 7f03b1592740 1 -- 172.21.15.49:0/1280413583 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f03a003f470 con 0x560963988250 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.466+0000 7f03877fe640 1 -- 172.21.15.49:0/1280413583 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f039c09b560 con 0x560963988250 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.466+0000 7f03b1592740 10 client.16755 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:41.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.467+0000 7f03b1592740 1 -- 172.21.15.49:0/1280413583 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560963a03da0 con 0x560963988250 2024-03-18T00:55:41.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.467+0000 7f03877fe640 1 -- 172.21.15.49:0/1280413583 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a0003970 con 0x560963988250 2024-03-18T00:55:41.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.467+0000 7f03877fe640 1 -- 172.21.15.49:0/1280413583 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f039c07cbb0 con 0x560963988250 2024-03-18T00:55:41.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.468+0000 7f03b1592740 10 client.16755 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:41.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.468+0000 7f03b1592740 10 client.16755 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:55:41.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.468+0000 7f03b1592740 20 client.16755 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:41.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.468+0000 7f03b1592740 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x560963a045b0 0x560963a069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:41.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.468+0000 7f03b1592740 4 client.16755 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:41.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.468+0000 7f03b1592740 1 -- 172.21.15.49:0/1280413583 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f03a003f470 con 0x560963a045b0 2024-03-18T00:55:41.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.470+0000 7f03a7fff640 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x560963a045b0 0x560963a069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:41.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.473+0000 7f03877fe640 10 client.16755 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:55:41.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.473+0000 7f03a7fff640 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x560963a045b0 0x560963a069a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa62f54964223fb6 server_cookie=d69c551108bbb68a in_seq=0 out_seq=0 2024-03-18T00:55:41.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03877fe640 1 -- 172.21.15.49:0/1280413583 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f03a003f470 con 0x560963a045b0 2024-03-18T00:55:41.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03877fe640 10 client.16755 handle_command_reply: tid=0 2024-03-18T00:55:41.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 client.16755 shutdown 2024-03-18T00:55:41.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 -- 172.21.15.49:0/1280413583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 msgr2=0x560963996e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:41.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963996e80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f039c011f70 tx=0x7f039c0ccc30 comp rx=0 tx=0).stop 2024-03-18T00:55:41.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 -- 172.21.15.49:0/1280413583 shutdown_connections 2024-03-18T00:55:41.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x560963a045b0 0x560963a069a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03877fe640 0 client.16755 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:55:41.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56096398d3c0 0x560963997e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560963988250 0x560963996e80 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 --2- 172.21.15.49:0/1280413583 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560963987860 0x560963996940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 -- 172.21.15.49:0/1280413583 >> 172.21.15.49:0/1280413583 conn(0x560963971dc0 msgr2=0x560963975020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:41.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.474+0000 7f03b1592740 1 -- 172.21.15.49:0/1280413583 shutdown_connections 2024-03-18T00:55:41.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.475+0000 7f03b1592740 1 -- 172.21.15.49:0/1280413583 wait complete. 2024-03-18T00:55:41.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.475+0000 7f03b1592740 20 client.16755 trim_cache size 0 max 16384 2024-03-18T00:55:41.477 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:41.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.476+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3581988861 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f03900a29d0 msgr2=0x7f03900a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:41.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.476+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f03900a29d0 0x7f03900a4e90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f039800abe0 tx=0x7f039800a2d0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.476+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3581988861 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03a81007d0 msgr2=0x7f03a8070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:41.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.476+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03a81007d0 0x7f03a8070640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b3f0 tx=0x7f039c00b7f0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.476+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3581988861 shutdown_connections 2024-03-18T00:55:41.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.476+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f03900a29d0 0x7f03900a4e90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.477+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f03a81007d0 0x7f03a8070640 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.477+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f03a8101170 0x7f03a81ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.477+0000 7f03ae8c6640 1 --2- 172.21.15.49:0/3581988861 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a80ff6a0 0x7f03a8070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:41.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.477+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3581988861 >> 172.21.15.49:0/3581988861 conn(0x7f03a80fafa0 msgr2=0x7f03a81019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:41.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.477+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3581988861 shutdown_connections 2024-03-18T00:55:41.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:41.477+0000 7f03ae8c6640 1 -- 172.21.15.49:0/3581988861 wait complete. 2024-03-18T00:55:41.540 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:41 smithi049 ceph-mon[32130]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 941 MiB/s wr, 2.42k op/s 2024-03-18T00:55:41.542 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:55:41.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:41 smithi107 ceph-mon[36270]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 941 MiB/s wr, 2.42k op/s 2024-03-18T00:55:41.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:41 smithi106 ceph-mon[35899]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 941 MiB/s wr, 2.42k op/s 2024-03-18T00:55:42.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.069+0000 7fadc63d3640 1 -- 172.21.15.49:0/577552722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0102a00 msgr2=0x7fadc0102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:42.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.069+0000 7fadc63d3640 1 --2- 172.21.15.49:0/577552722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0102a00 0x7fadc0102e80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fadb4009900 tx=0x7fadb402bc30 comp rx=0 tx=0).stop 2024-03-18T00:55:42.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.070+0000 7fadc63d3640 1 -- 172.21.15.49:0/577552722 shutdown_connections 2024-03-18T00:55:42.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.070+0000 7fadc63d3640 1 --2- 172.21.15.49:0/577552722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadc01033c0 0x7fadc0109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.070+0000 7fadc63d3640 1 --2- 172.21.15.49:0/577552722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0102a00 0x7fadc0102e80 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.070+0000 7fadc63d3640 1 --2- 172.21.15.49:0/577552722 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc0101810 0x7fadc0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.070+0000 7fadc63d3640 1 -- 172.21.15.49:0/577552722 >> 172.21.15.49:0/577552722 conn(0x7fadc00fcd80 msgr2=0x7fadc00ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:42.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.070+0000 7fadc63d3640 1 -- 172.21.15.49:0/577552722 shutdown_connections 2024-03-18T00:55:42.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.070+0000 7fadc63d3640 1 -- 172.21.15.49:0/577552722 wait complete. 2024-03-18T00:55:42.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.071+0000 7fadc63d3640 1 Processor -- start 2024-03-18T00:55:42.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.071+0000 7fadc63d3640 1 -- start start 2024-03-18T00:55:42.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.071+0000 7fadc63d3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0101810 0x7fadc019e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.071+0000 7fadc63d3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc0102a00 0x7fadc019e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.072+0000 7fadc63d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadc01033c0 0x7fadc01a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.072+0000 7fadc63d3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fadc010d500 con 0x7fadc0101810 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.072+0000 7fadc63d3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fadc010d380 con 0x7fadc0102a00 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.072+0000 7fadc63d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fadc010d680 con 0x7fadc01033c0 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.072+0000 7fadbffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0101810 0x7fadc019e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.072+0000 7fadbffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0101810 0x7fadc019e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54400/0 (socket says 172.21.15.49:54400) 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.072+0000 7fadbffff640 1 -- 172.21.15.49:0/2175302034 learned_addr learned my addr 172.21.15.49:0/2175302034 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.072+0000 7fadbf7fe640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc0102a00 0x7fadc019e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadbffff640 1 -- 172.21.15.49:0/2175302034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadc01033c0 msgr2=0x7fadc01a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadbffff640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadc01033c0 0x7fadc01a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadbffff640 1 -- 172.21.15.49:0/2175302034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc0102a00 msgr2=0x7fadc019e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadbffff640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc0102a00 0x7fadc019e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadbffff640 1 -- 172.21.15.49:0/2175302034 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc01a5d70 con 0x7fadc0101810 2024-03-18T00:55:42.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadbffff640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0101810 0x7fadc019e150 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fadb000b450 tx=0x7fadb000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:42.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadbd7fa640 1 -- 172.21.15.49:0/2175302034 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0015020 con 0x7fadc0101810 2024-03-18T00:55:42.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadbd7fa640 1 -- 172.21.15.49:0/2175302034 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb0011d70 con 0x7fadc0101810 2024-03-18T00:55:42.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadbd7fa640 1 -- 172.21.15.49:0/2175302034 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb000f340 con 0x7fadc0101810 2024-03-18T00:55:42.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadc63d3640 1 -- 172.21.15.49:0/2175302034 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc01a6060 con 0x7fadc0101810 2024-03-18T00:55:42.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.073+0000 7fadc63d3640 1 -- 172.21.15.49:0/2175302034 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc01a65d0 con 0x7fadc0101810 2024-03-18T00:55:42.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadc8fb7740 1 Processor -- start 2024-03-18T00:55:42.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadc8fb7740 1 -- start start 2024-03-18T00:55:42.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd0f850 0x55fdbdd0fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd10240 0x55fdbdd14d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd17840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55fdbdc74490 con 0x55fdbdd153b0 2024-03-18T00:55:42.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55fdbdc1cc90 con 0x55fdbdd10240 2024-03-18T00:55:42.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fdbdc70530 con 0x55fdbdd0f850 2024-03-18T00:55:42.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadbf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd17840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadbf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd17840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54414/0 (socket says 172.21.15.49:54414) 2024-03-18T00:55:42.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.077+0000 7fadbf7fe640 1 -- 172.21.15.49:0/123940371 learned_addr learned my addr 172.21.15.49:0/123940371 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:42.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.078+0000 7fadbd7fa640 1 -- 172.21.15.49:0/2175302034 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fadb0018020 con 0x7fadc0101810 2024-03-18T00:55:42.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.078+0000 7fadbf7fe640 1 -- 172.21.15.49:0/123940371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd0f850 msgr2=0x55fdbdd0fc70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:42.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.078+0000 7fadbf7fe640 1 --2- 172.21.15.49:0/123940371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd0f850 0x55fdbdd0fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.078+0000 7fadbf7fe640 1 -- 172.21.15.49:0/123940371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd10240 msgr2=0x55fdbdd14d40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.078+0000 7fadbf7fe640 1 --2- 172.21.15.49:0/123940371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd10240 0x55fdbdd14d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.078+0000 7fadbf7fe640 1 -- 172.21.15.49:0/123940371 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdbdcd7500 con 0x55fdbdd153b0 2024-03-18T00:55:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.078+0000 7fadbf7fe640 1 --2- 172.21.15.49:0/123940371 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd17840 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fadb4036040 tx=0x7fadb40095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca896d2edebedd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.078+0000 7fadbd7fa640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fada80a2880 0x7fada80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.079+0000 7fada67fc640 1 -- 172.21.15.49:0/123940371 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb4042070 con 0x55fdbdd153b0 2024-03-18T00:55:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.079+0000 7fada67fc640 1 -- 172.21.15.49:0/123940371 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb402cb80 con 0x55fdbdd153b0 2024-03-18T00:55:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.079+0000 7fada67fc640 1 -- 172.21.15.49:0/123940371 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb4005280 con 0x55fdbdd153b0 2024-03-18T00:55:42.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.079+0000 7fadc4970640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fada80a2880 0x7fada80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.080+0000 7fadbd7fa640 1 -- 172.21.15.49:0/2175302034 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fadb00ce5c0 con 0x7fadc0101810 2024-03-18T00:55:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.080+0000 7fadc4970640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fada80a2880 0x7fada80a4d40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fadb8005fd0 tx=0x7fadb80071c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.080+0000 7fadc8fb7740 1 -- 172.21.15.49:0/123940371 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 msgr2=0x55fdbdd17840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.080+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/123940371 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd17840 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fadb4036040 tx=0x7fadb40095e0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.081+0000 7fadc8fb7740 1 -- 172.21.15.49:0/123940371 shutdown_connections 2024-03-18T00:55:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.081+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/123940371 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd10240 0x55fdbdd14d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.081+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/123940371 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd17840 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.081+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/123940371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd0f850 0x55fdbdd0fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.081+0000 7fadc8fb7740 1 -- 172.21.15.49:0/123940371 >> 172.21.15.49:0/123940371 conn(0x55fdbdd0adc0 msgr2=0x55fdbdd0b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:42.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.082+0000 7fadc8fb7740 1 -- 172.21.15.49:0/123940371 shutdown_connections 2024-03-18T00:55:42.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.083+0000 7fadc8fb7740 1 -- 172.21.15.49:0/123940371 wait complete. 2024-03-18T00:55:42.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.083+0000 7fadc8fb7740 1 -- start start 2024-03-18T00:55:42.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.083+0000 7fadc8fb7740 1 Processor -- start 2024-03-18T00:55:42.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd0f850 0x55fdbdd19e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd1a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd213a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55fdbdd18300 con 0x55fdbdd153b0 2024-03-18T00:55:42.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55fdbdc30230 con 0x55fdbdd0f850 2024-03-18T00:55:42.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fdbdd18480 con 0x55fdbdd10240 2024-03-18T00:55:42.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc8fb7740 10 client.0 _ll_register_callbacks cb 0x55fdbdc6c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:42.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc4970640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd213a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc4970640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd213a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54424/0 (socket says 172.21.15.49:54424) 2024-03-18T00:55:42.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.084+0000 7fadc4970640 1 -- 172.21.15.49:0/347847378 learned_addr learned my addr 172.21.15.49:0/347847378 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:42.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.085+0000 7fadc4970640 1 -- 172.21.15.49:0/347847378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 msgr2=0x55fdbdd1a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:42.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.085+0000 7fadbf7fe640 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd1a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.085+0000 7fadc4970640 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd1a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.085+0000 7fadc4970640 1 -- 172.21.15.49:0/347847378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd0f850 msgr2=0x55fdbdd19e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:42.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.085+0000 7fadc4970640 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd0f850 0x55fdbdd19e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.085+0000 7fadc4970640 1 -- 172.21.15.49:0/347847378 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdbdc3f100 con 0x55fdbdd153b0 2024-03-18T00:55:42.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.085+0000 7fadbf7fe640 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd1a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:42.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.085+0000 7fadc4970640 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd213a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fadb80042b0 tx=0x7fadb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ddb1ab5c0034813 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:42.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.086+0000 7fad9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:42.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.086+0000 7fad9f7fe640 1 -- 172.21.15.49:0/347847378 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb800f9d0 con 0x55fdbdd153b0 2024-03-18T00:55:42.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.086+0000 7fad9f7fe640 1 -- 172.21.15.49:0/347847378 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fadb800fb70 con 0x55fdbdd153b0 2024-03-18T00:55:42.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.086+0000 7fadc8fb7740 1 -- 172.21.15.49:0/347847378 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadb402cb80 con 0x55fdbdd153b0 2024-03-18T00:55:42.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.086+0000 7fad9f7fe640 1 -- 172.21.15.49:0/347847378 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fadb800fd70 con 0x55fdbdd153b0 2024-03-18T00:55:42.090 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.086+0000 7fadc8fb7740 10 client.16767 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:42.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.087+0000 7fadc8fb7740 1 -- 172.21.15.49:0/347847378 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fdbd9883f0 con 0x55fdbdd153b0 2024-03-18T00:55:42.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.087+0000 7fad9f7fe640 1 -- 172.21.15.49:0/347847378 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb402cb80 con 0x55fdbdd153b0 2024-03-18T00:55:42.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.087+0000 7fad9f7fe640 1 -- 172.21.15.49:0/347847378 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fadb8025030 con 0x55fdbdd153b0 2024-03-18T00:55:42.092 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.087+0000 7fadc8fb7740 10 client.16767 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:42.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.087+0000 7fadc8fb7740 10 client.16767 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:42.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.087+0000 7fadc8fb7740 20 client.16767 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:42.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.087+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fdbdd371f0 0x55fdbdd396b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.087+0000 7fadc8fb7740 4 client.16767 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:42.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.087+0000 7fadc8fb7740 1 -- 172.21.15.49:0/347847378 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdbdc819d0 con 0x55fdbdd371f0 2024-03-18T00:55:42.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.090+0000 7fadbffff640 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fdbdd371f0 0x55fdbdd396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.092+0000 7fadbffff640 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fdbdd371f0 0x55fdbdd396b0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=840e8a001cd43700 server_cookie=db9f707892a7dfbf in_seq=0 out_seq=0 2024-03-18T00:55:42.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.093+0000 7fad9f7fe640 10 client.16767 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:42.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.096+0000 7fad9f7fe640 1 -- 172.21.15.49:0/347847378 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fdbdc819d0 con 0x55fdbdd371f0 2024-03-18T00:55:42.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.096+0000 7fad9f7fe640 10 client.16767 handle_command_reply: tid=0 2024-03-18T00:55:42.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.096+0000 7fadc8fb7740 1 client.16767 shutdown 2024-03-18T00:55:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.097+0000 7fadc8fb7740 1 -- 172.21.15.49:0/347847378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 msgr2=0x55fdbdd213a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.097+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd213a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fadb80042b0 tx=0x7fadb8004a40 comp rx=0 tx=0).stop 2024-03-18T00:55:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.097+0000 7fadc8fb7740 1 -- 172.21.15.49:0/347847378 shutdown_connections 2024-03-18T00:55:42.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.097+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fdbdd371f0 0x55fdbdd396b0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.097+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd153b0 0x55fdbdd213a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.097+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd1a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.097+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/347847378 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd0f850 0x55fdbdd19e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.097+0000 7fadc8fb7740 1 -- 172.21.15.49:0/347847378 >> 172.21.15.49:0/347847378 conn(0x55fdbdd0adc0 msgr2=0x55fdbdd09fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:42.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.097+0000 7fad9f7fe640 0 client.16767 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:42.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.098+0000 7fadc8fb7740 1 -- 172.21.15.49:0/347847378 shutdown_connections 2024-03-18T00:55:42.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.098+0000 7fadc8fb7740 1 -- 172.21.15.49:0/347847378 wait complete. 2024-03-18T00:55:42.099 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.098+0000 7fadc8fb7740 20 client.16767 trim_cache size 0 max 16384 2024-03-18T00:55:42.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.203+0000 7fadc8fb7740 1 Processor -- start 2024-03-18T00:55:42.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.207+0000 7fadc8fb7740 1 -- start start 2024-03-18T00:55:42.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.207+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd117e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.208+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd21360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.208+0000 7fadbffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.208+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 0x55fdbdd11d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.208+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01a6c40 con 0x55fdbdd0f850 2024-03-18T00:55:42.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.208+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01a8780 con 0x55fdbdd153b0 2024-03-18T00:55:42.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.208+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01a7310 con 0x55fdbdd10240 2024-03-18T00:55:42.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.209+0000 7fadbffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd117e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54432/0 (socket says 172.21.15.49:54432) 2024-03-18T00:55:42.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.209+0000 7fadbffff640 1 -- 172.21.15.49:0/3901395762 learned_addr learned my addr 172.21.15.49:0/3901395762 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:42.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.209+0000 7fadbffff640 1 -- 172.21.15.49:0/3901395762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 msgr2=0x55fdbdd21360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:42.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.209+0000 7fadbffff640 1 --2- 172.21.15.49:0/3901395762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd21360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.209+0000 7fadbffff640 1 -- 172.21.15.49:0/3901395762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 msgr2=0x55fdbdd11d20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:42.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.209+0000 7fadbffff640 1 --2- 172.21.15.49:0/3901395762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 0x55fdbdd11d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.209+0000 7fadbffff640 1 -- 172.21.15.49:0/3901395762 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdbda527d0 con 0x55fdbdd0f850 2024-03-18T00:55:42.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.210+0000 7fadbffff640 1 --2- 172.21.15.49:0/3901395762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd117e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fadb009aed0 tx=0x7fadb0010970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e20e4ff4ac6375 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:42.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.217+0000 7fada5ffb640 1 -- 172.21.15.49:0/3901395762 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb009f090 con 0x55fdbdd0f850 2024-03-18T00:55:42.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.217+0000 7fada5ffb640 1 -- 172.21.15.49:0/3901395762 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb00a36e0 con 0x55fdbdd0f850 2024-03-18T00:55:42.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.219+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3901395762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 msgr2=0x55fdbdd117e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:42.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.219+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3901395762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd117e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fadb009aed0 tx=0x7fadb0010970 comp rx=0 tx=0).stop 2024-03-18T00:55:42.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.224+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3901395762 shutdown_connections 2024-03-18T00:55:42.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.224+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3901395762 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 0x55fdbdd11d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.225+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3901395762 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd21360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.225+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3901395762 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd117e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.225+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3901395762 >> 172.21.15.49:0/3901395762 conn(0x55fdbdd0adc0 msgr2=0x55fdbdd19e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:42.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.234+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3901395762 shutdown_connections 2024-03-18T00:55:42.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.235+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3901395762 wait complete. 2024-03-18T00:55:42.235 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.235+0000 7fadc8fb7740 1 -- start start 2024-03-18T00:55:42.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.236+0000 7fadc8fb7740 1 Processor -- start 2024-03-18T00:55:42.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.236+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd210c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.236+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd1db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.236+0000 7fadc8fb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 0x55fdbdd1e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.236+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01a80b0 con 0x55fdbdd0f850 2024-03-18T00:55:42.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.236+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fadc01a79e0 con 0x55fdbdd153b0 2024-03-18T00:55:42.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.236+0000 7fadc8fb7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fadc0071d30 con 0x55fdbdd10240 2024-03-18T00:55:42.237 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.236+0000 7fadc8fb7740 10 client.0 _ll_register_callbacks cb 0x7fadb8014040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:42.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.238+0000 7fadbffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd210c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.238+0000 7fadbffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd210c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54434/0 (socket says 172.21.15.49:54434) 2024-03-18T00:55:42.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.238+0000 7fadbffff640 1 -- 172.21.15.49:0/3102445203 learned_addr learned my addr 172.21.15.49:0/3102445203 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:42.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.239+0000 7fadbf7fe640 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd1db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.239 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.239+0000 7fadc4970640 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 0x55fdbdd1e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.241+0000 7fadc4970640 1 -- 172.21.15.49:0/3102445203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 msgr2=0x55fdbdd1db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:42.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.241+0000 7fadc4970640 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd1db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.241+0000 7fadc4970640 1 -- 172.21.15.49:0/3102445203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 msgr2=0x55fdbdd210c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:42.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.241+0000 7fadc4970640 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd210c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.241+0000 7fadc4970640 1 -- 172.21.15.49:0/3102445203 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdbdd1e740 con 0x55fdbdd153b0 2024-03-18T00:55:42.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.246+0000 7fadc4970640 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 0x55fdbdd1e0a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fadb801d850 tx=0x7fadb800ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34d839d86c1b3b96 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:42.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.246+0000 7fad9f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:55:42.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.248+0000 7fad9f7fe640 1 -- 172.21.15.49:0/3102445203 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8035020 con 0x55fdbdd153b0 2024-03-18T00:55:42.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.248+0000 7fad9f7fe640 1 -- 172.21.15.49:0/3102445203 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fadb8028450 con 0x55fdbdd153b0 2024-03-18T00:55:42.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.254+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3102445203 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadb00a36e0 con 0x55fdbdd153b0 2024-03-18T00:55:42.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.259+0000 7fad9f7fe640 1 -- 172.21.15.49:0/3102445203 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fadb8028650 con 0x55fdbdd153b0 2024-03-18T00:55:42.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.260+0000 7fadc8fb7740 10 client.26123 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:42.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.260+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3102445203 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fdbddad720 con 0x55fdbdd153b0 2024-03-18T00:55:42.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.268+0000 7fad9f7fe640 1 -- 172.21.15.49:0/3102445203 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb00a36e0 con 0x55fdbdd153b0 2024-03-18T00:55:42.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.269+0000 7fad9f7fe640 1 -- 172.21.15.49:0/3102445203 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fadb8025050 con 0x55fdbdd153b0 2024-03-18T00:55:42.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.269+0000 7fadc8fb7740 10 client.26123 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:42.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.269+0000 7fadc8fb7740 10 client.26123 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:42.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.269+0000 7fadc8fb7740 20 client.26123 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:42.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.269+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fdbddae030 0x55fdbddb0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:42.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.269+0000 7fadc8fb7740 4 client.26123 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:55:42.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.269+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3102445203 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fdbdc7bd20 con 0x55fdbddae030 2024-03-18T00:55:42.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.273+0000 7fadbffff640 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fdbddae030 0x55fdbddb0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:42.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.288+0000 7fadbffff640 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fdbddae030 0x55fdbddb0450 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32bec81e4b8e6d6 server_cookie=d9bfbdac8f7f8935 in_seq=0 out_seq=0 2024-03-18T00:55:42.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.290+0000 7fad9f7fe640 10 client.26123 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:42.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.293+0000 7fad9f7fe640 1 -- 172.21.15.49:0/3102445203 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fdbdc7bd20 con 0x55fdbddae030 2024-03-18T00:55:42.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.294+0000 7fad9f7fe640 10 client.26123 handle_command_reply: tid=0 2024-03-18T00:55:42.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.294+0000 7fadc8fb7740 1 client.26123 shutdown 2024-03-18T00:55:42.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.295+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3102445203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 msgr2=0x55fdbdd1e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:42.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.295+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 0x55fdbdd1e0a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fadb801d850 tx=0x7fadb800ca50 comp rx=0 tx=0).stop 2024-03-18T00:55:42.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.295+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3102445203 shutdown_connections 2024-03-18T00:55:42.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.295+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fdbddae030 0x55fdbddb0450 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.295+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fdbdd153b0 0x55fdbdd1e0a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.296+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fdbdd10240 0x55fdbdd1db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.296+0000 7fadc8fb7740 1 --2- 172.21.15.49:0/3102445203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fdbdd0f850 0x55fdbdd210c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.296+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3102445203 >> 172.21.15.49:0/3102445203 conn(0x55fdbdd0adc0 msgr2=0x55fdbdd19de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:42.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.297+0000 7fad9f7fe640 0 client.26123 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:42.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.303+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3102445203 shutdown_connections 2024-03-18T00:55:42.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.303+0000 7fadc8fb7740 1 -- 172.21.15.49:0/3102445203 wait complete. 2024-03-18T00:55:42.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.303+0000 7fadc8fb7740 20 client.26123 trim_cache size 0 max 16384 2024-03-18T00:55:42.307 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:55:42.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.313+0000 7fadc63d3640 1 -- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fada80a2880 msgr2=0x7fada80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:42.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.313+0000 7fadc63d3640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fada80a2880 0x7fada80a4d40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fadb8005fd0 tx=0x7fadb80071c0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.313+0000 7fadc63d3640 1 -- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0101810 msgr2=0x7fadc019e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:42.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.313+0000 7fadc63d3640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0101810 0x7fadc019e150 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fadb000b450 tx=0x7fadb000b910 comp rx=0 tx=0).stop 2024-03-18T00:55:42.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.318+0000 7fadc63d3640 1 -- 172.21.15.49:0/2175302034 shutdown_connections 2024-03-18T00:55:42.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.318+0000 7fadc63d3640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fada80a2880 0x7fada80a4d40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.318+0000 7fadc63d3640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fadc0102a00 0x7fadc019e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.318+0000 7fadc63d3640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fadc01033c0 0x7fadc01a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.318+0000 7fadc63d3640 1 --2- 172.21.15.49:0/2175302034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fadc0101810 0x7fadc019e150 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:42.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.318+0000 7fadc63d3640 1 -- 172.21.15.49:0/2175302034 >> 172.21.15.49:0/2175302034 conn(0x7fadc00fcd80 msgr2=0x7fadc0103c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:42.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.320+0000 7fadc63d3640 1 -- 172.21.15.49:0/2175302034 shutdown_connections 2024-03-18T00:55:42.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:42.321+0000 7fadc63d3640 1 -- 172.21.15.49:0/2175302034 wait complete. 2024-03-18T00:55:43.368 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:55:43.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:55:43.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:55:43.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:43 smithi049 ceph-mon[32130]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 894 MiB/s wr, 2.18k op/s 2024-03-18T00:55:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:43 smithi107 ceph-mon[36270]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 894 MiB/s wr, 2.18k op/s 2024-03-18T00:55:43.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:43 smithi106 ceph-mon[35899]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 894 MiB/s wr, 2.18k op/s 2024-03-18T00:55:43.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.805+0000 7f6d4af00640 1 -- 172.21.15.49:0/117719105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 msgr2=0x7f6d441056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:43.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.805+0000 7f6d4af00640 1 --2- 172.21.15.49:0/117719105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 0x7f6d441056d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009900 tx=0x7f6d3402bc60 comp rx=0 tx=0).stop 2024-03-18T00:55:43.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.806+0000 7f6d4af00640 1 -- 172.21.15.49:0/117719105 shutdown_connections 2024-03-18T00:55:43.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.806+0000 7f6d4af00640 1 --2- 172.21.15.49:0/117719105 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 0x7f6d441056d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.806+0000 7f6d4af00640 1 --2- 172.21.15.49:0/117719105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d441028a0 0x7f6d44104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.806+0000 7f6d4af00640 1 --2- 172.21.15.49:0/117719105 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d440fff30 0x7f6d44102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.806+0000 7f6d4af00640 1 -- 172.21.15.49:0/117719105 >> 172.21.15.49:0/117719105 conn(0x7f6d440fbb50 msgr2=0x7f6d440fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:43.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.808+0000 7f6d4af00640 1 -- 172.21.15.49:0/117719105 shutdown_connections 2024-03-18T00:55:43.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.810+0000 7f6d4af00640 1 -- 172.21.15.49:0/117719105 wait complete. 2024-03-18T00:55:43.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.811+0000 7f6d4af00640 1 Processor -- start 2024-03-18T00:55:43.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.812+0000 7f6d4af00640 1 -- start start 2024-03-18T00:55:43.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.812+0000 7f6d4af00640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d440fff30 0x7f6d44110070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.812+0000 7f6d4af00640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d441028a0 0x7f6d441105b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.812+0000 7f6d4af00640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 0x7f6d4410e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.812+0000 7f6d4af00640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f6d44106fa0 con 0x7f6d441028a0 2024-03-18T00:55:43.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.812+0000 7f6d4af00640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6d44106e20 con 0x7f6d44105250 2024-03-18T00:55:43.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.812+0000 7f6d4af00640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6d44107120 con 0x7f6d440fff30 2024-03-18T00:55:43.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.813+0000 7f6d4a6ff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 0x7f6d4410e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.813+0000 7f6d4a6ff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 0x7f6d4410e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:47794/0 (socket says 172.21.15.49:47794) 2024-03-18T00:55:43.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.813+0000 7f6d4a6ff640 1 -- 172.21.15.49:0/3186885796 learned_addr learned my addr 172.21.15.49:0/3186885796 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:43.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.814+0000 7f6d4a6ff640 1 -- 172.21.15.49:0/3186885796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d440fff30 msgr2=0x7f6d44110070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:55:43.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.814+0000 7f6d4a6ff640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d440fff30 0x7f6d44110070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.814+0000 7f6d4a6ff640 1 -- 172.21.15.49:0/3186885796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d441028a0 msgr2=0x7f6d441105b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:43.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.814+0000 7f6d4a6ff640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d441028a0 0x7f6d441105b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.814+0000 7f6d4a6ff640 1 -- 172.21.15.49:0/3186885796 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d4410ee30 con 0x7f6d44105250 2024-03-18T00:55:43.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.815+0000 7f6d4a6ff640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 0x7f6d4410e6b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009850 tx=0x7f6d3402ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:43.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.822+0000 7f6d3affd640 1 -- 172.21.15.49:0/3186885796 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d34041020 con 0x7f6d44105250 2024-03-18T00:55:43.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.822+0000 7f6d4af00640 1 -- 172.21.15.49:0/3186885796 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d4410f0c0 con 0x7f6d44105250 2024-03-18T00:55:43.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.822+0000 7f6d4af00640 1 -- 172.21.15.49:0/3186885796 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d4406b200 con 0x7f6d44105250 2024-03-18T00:55:43.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.822+0000 7f6d3affd640 1 -- 172.21.15.49:0/3186885796 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d3402ee40 con 0x7f6d44105250 2024-03-18T00:55:43.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.822+0000 7f6d3affd640 1 -- 172.21.15.49:0/3186885796 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3402ca20 con 0x7f6d44105250 2024-03-18T00:55:43.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.829+0000 7f6d3affd640 1 -- 172.21.15.49:0/3186885796 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6d34044020 con 0x7f6d44105250 2024-03-18T00:55:43.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.829+0000 7f6d4ee2f740 1 Processor -- start 2024-03-18T00:55:43.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.830+0000 7f6d4ee2f740 1 -- start start 2024-03-18T00:55:43.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.830+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a2582b80 0x5558a2582fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.830+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2583570 0x5558a2588070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.830+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a25886e0 0x5558a258ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.830+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5558a24e70d0 con 0x5558a25886e0 2024-03-18T00:55:43.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.830+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5558a248fc70 con 0x5558a2582b80 2024-03-18T00:55:43.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.830+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5558a24e3170 con 0x5558a2583570 2024-03-18T00:55:43.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.833+0000 7f6d49efe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2583570 0x5558a2588070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.833+0000 7f6d496fd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a25886e0 0x5558a258ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.833+0000 7f6d496fd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a25886e0 0x5558a258ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54460/0 (socket says 172.21.15.49:54460) 2024-03-18T00:55:43.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.833+0000 7f6d496fd640 1 -- 172.21.15.49:0/205121812 learned_addr learned my addr 172.21.15.49:0/205121812 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:43.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.834+0000 7f6d496fd640 1 --2- 172.21.15.49:0/205121812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a2582b80 0x5558a2582fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.835+0000 7f6d496fd640 1 -- 172.21.15.49:0/205121812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2583570 msgr2=0x5558a2588070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:43.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.835+0000 7f6d496fd640 1 --2- 172.21.15.49:0/205121812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2583570 0x5558a2588070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.835+0000 7f6d496fd640 1 -- 172.21.15.49:0/205121812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a2582b80 msgr2=0x5558a2582fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:43.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.835+0000 7f6d496fd640 1 --2- 172.21.15.49:0/205121812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a2582b80 0x5558a2582fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.835+0000 7f6d496fd640 1 -- 172.21.15.49:0/205121812 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558a22c37e0 con 0x5558a25886e0 2024-03-18T00:55:43.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.835+0000 7f6d3affd640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6d300a2a00 0x7f6d300a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.836+0000 7f6d4a6ff640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6d300a2a00 0x7f6d300a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.836+0000 7f6d496fd640 1 --2- 172.21.15.49:0/205121812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a25886e0 0x5558a258ab70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x5558a2583dd0 tx=0x7f6d4000c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67862be39567cde server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:43.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.837+0000 7f6d2ffff640 1 -- 172.21.15.49:0/205121812 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d40004220 con 0x5558a25886e0 2024-03-18T00:55:43.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.837+0000 7f6d2ffff640 1 -- 172.21.15.49:0/205121812 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d400043c0 con 0x5558a25886e0 2024-03-18T00:55:43.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.837+0000 7f6d2ffff640 1 -- 172.21.15.49:0/205121812 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d400099e0 con 0x5558a25886e0 2024-03-18T00:55:43.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.838+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/205121812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a25886e0 msgr2=0x5558a258ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:43.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.838+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/205121812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a25886e0 0x5558a258ab70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x5558a2583dd0 tx=0x7f6d4000c270 comp rx=0 tx=0).stop 2024-03-18T00:55:43.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.841+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/205121812 shutdown_connections 2024-03-18T00:55:43.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.841+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/205121812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a25886e0 0x5558a258ab70 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.841+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/205121812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2583570 0x5558a2588070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.841+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/205121812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a2582b80 0x5558a2582fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.841+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/205121812 >> 172.21.15.49:0/205121812 conn(0x5558a254ac50 msgr2=0x5558a257e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:43.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.842+0000 7f6d3affd640 1 -- 172.21.15.49:0/3186885796 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6d340f0670 con 0x7f6d44105250 2024-03-18T00:55:43.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.842+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/205121812 shutdown_connections 2024-03-18T00:55:43.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.843+0000 7f6d4a6ff640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6d300a2a00 0x7f6d300a4ec0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6d340bcc90 tx=0x7f6d340b8af0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:43.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.843+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/205121812 wait complete. 2024-03-18T00:55:43.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.843+0000 7f6d4ee2f740 1 -- start start 2024-03-18T00:55:43.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.843+0000 7f6d4ee2f740 1 Processor -- start 2024-03-18T00:55:43.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.845+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a258d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.845+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a258d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.845+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a2594740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.845+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5558a258b690 con 0x5558a2583570 2024-03-18T00:55:43.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.845+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5558a24a30d0 con 0x5558a25886e0 2024-03-18T00:55:43.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.845+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5558a258b810 con 0x5558a2582b80 2024-03-18T00:55:43.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.845+0000 7f6d4ee2f740 10 client.0 _ll_register_callbacks cb 0x5558a246c760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:43.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.845+0000 7f6d49efe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a258d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.846+0000 7f6d49efe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a258d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54468/0 (socket says 172.21.15.49:54468) 2024-03-18T00:55:43.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.846+0000 7f6d49efe640 1 -- 172.21.15.49:0/1194541606 learned_addr learned my addr 172.21.15.49:0/1194541606 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:43.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.847+0000 7f6d49efe640 1 -- 172.21.15.49:0/1194541606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 msgr2=0x5558a258d1c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:43.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.847+0000 7f6d49efe640 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a258d1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.847+0000 7f6d49efe640 1 -- 172.21.15.49:0/1194541606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 msgr2=0x5558a2594740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:43.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.847+0000 7f6d49efe640 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a2594740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.847+0000 7f6d49efe640 1 -- 172.21.15.49:0/1194541606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558a2323f70 con 0x5558a2583570 2024-03-18T00:55:43.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.847+0000 7f6d49efe640 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a258d700 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00c960 tx=0x7f6d3c00ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898cf4fae41d9a31 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:43.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.847+0000 7f6d2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:43.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.851+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/1194541606 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c01a060 con 0x5558a2583570 2024-03-18T00:55:43.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.851+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/1194541606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d400043c0 con 0x5558a2583570 2024-03-18T00:55:43.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.852+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/1194541606 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6d3c0078f0 con 0x5558a2583570 2024-03-18T00:55:43.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.855+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/1194541606 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d3c007af0 con 0x5558a2583570 2024-03-18T00:55:43.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.861+0000 7f6d4ee2f740 10 client.16779 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:43.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.861+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/1194541606 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5558a21bb170 con 0x5558a2583570 2024-03-18T00:55:43.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.867+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/1194541606 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d400043c0 con 0x5558a2583570 2024-03-18T00:55:43.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.867+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/1194541606 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6d3c00a400 con 0x5558a2583570 2024-03-18T00:55:43.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.867+0000 7f6d4ee2f740 10 client.16779 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:43.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.867+0000 7f6d4ee2f740 10 client.16779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:43.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.867+0000 7f6d4ee2f740 20 client.16779 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:43.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.867+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a2599210 0x5558a259b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.867+0000 7f6d4ee2f740 4 client.16779 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:43.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.867+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/1194541606 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558a24f4610 con 0x5558a2599210 2024-03-18T00:55:43.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.869+0000 7f6d496fd640 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a2599210 0x5558a259b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.871+0000 7f6d496fd640 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a2599210 0x5558a259b650 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14dd396cbc55b790 server_cookie=ccf426f7ae874cc3 in_seq=0 out_seq=0 2024-03-18T00:55:43.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.873+0000 7f6d2d7fa640 10 client.16779 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:43.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.879+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/1194541606 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5558a24f4610 con 0x5558a2599210 2024-03-18T00:55:43.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.879+0000 7f6d2d7fa640 10 client.16779 handle_command_reply: tid=0 2024-03-18T00:55:43.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.880+0000 7f6d4ee2f740 1 client.16779 shutdown 2024-03-18T00:55:43.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.888+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/1194541606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 msgr2=0x5558a258d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:43.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.888+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a258d700 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00c960 tx=0x7f6d3c00ce20 comp rx=0 tx=0).stop 2024-03-18T00:55:43.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.890+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/1194541606 shutdown_connections 2024-03-18T00:55:43.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.890+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a2599210 0x5558a259b650 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.890+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a2594740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.890+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a258d700 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.890+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/1194541606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a258d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.890+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/1194541606 >> 172.21.15.49:0/1194541606 conn(0x5558a254ac50 msgr2=0x5558a257d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:43.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.892+0000 7f6d2d7fa640 0 client.16779 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:43.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.892+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/1194541606 shutdown_connections 2024-03-18T00:55:43.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.895+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/1194541606 wait complete. 2024-03-18T00:55:43.896 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.896+0000 7f6d4ee2f740 20 client.16779 trim_cache size 0 max 16384 2024-03-18T00:55:43.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.949+0000 7f6d4ee2f740 1 Processor -- start 2024-03-18T00:55:43.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.949+0000 7f6d4ee2f740 1 -- start start 2024-03-18T00:55:43.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.950+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a258c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.950+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a2594700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.950+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a258c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.950+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2800e650 con 0x5558a2583570 2024-03-18T00:55:43.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.950+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6d28010190 con 0x5558a25886e0 2024-03-18T00:55:43.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.950+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2800ed20 con 0x5558a2582b80 2024-03-18T00:55:43.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.952+0000 7f6d496fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a258c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.952+0000 7f6d496fd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a258c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59230/0 (socket says 172.21.15.49:59230) 2024-03-18T00:55:43.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.952+0000 7f6d496fd640 1 -- 172.21.15.49:0/3266213582 learned_addr learned my addr 172.21.15.49:0/3266213582 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:43.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.952+0000 7f6d49efe640 1 --2- 172.21.15.49:0/3266213582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a2594700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.953+0000 7f6d49efe640 1 -- 172.21.15.49:0/3266213582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 msgr2=0x5558a258c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:43.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.953+0000 7f6d49efe640 1 --2- 172.21.15.49:0/3266213582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a258c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.953+0000 7f6d49efe640 1 -- 172.21.15.49:0/3266213582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 msgr2=0x5558a258c9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:43.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.953+0000 7f6d49efe640 1 --2- 172.21.15.49:0/3266213582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a258c9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.953+0000 7f6d49efe640 1 -- 172.21.15.49:0/3266213582 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558a2578ab0 con 0x5558a2583570 2024-03-18T00:55:43.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.954+0000 7f6d49efe640 1 --2- 172.21.15.49:0/3266213582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a2594700 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00aa10 tx=0x7f6d3c0049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32815c884bbf5010 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:43.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.955+0000 7f6d2f7fe640 1 -- 172.21.15.49:0/3266213582 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c00ae60 con 0x5558a2583570 2024-03-18T00:55:43.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.955+0000 7f6d2f7fe640 1 -- 172.21.15.49:0/3266213582 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d3c0097a0 con 0x5558a2583570 2024-03-18T00:55:43.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.955+0000 7f6d2f7fe640 1 -- 172.21.15.49:0/3266213582 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c016960 con 0x5558a2583570 2024-03-18T00:55:43.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.958+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/3266213582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 msgr2=0x5558a2594700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:43.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.958+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/3266213582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a2594700 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00aa10 tx=0x7f6d3c0049c0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.959+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/3266213582 shutdown_connections 2024-03-18T00:55:43.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.959+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/3266213582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a258c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.959+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/3266213582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a2594700 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.959+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/3266213582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a258c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.959+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/3266213582 >> 172.21.15.49:0/3266213582 conn(0x5558a254ac50 msgr2=0x5558a257e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:43.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.963+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/3266213582 shutdown_connections 2024-03-18T00:55:43.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.963+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/3266213582 wait complete. 2024-03-18T00:55:43.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.963+0000 7f6d4ee2f740 1 -- start start 2024-03-18T00:55:43.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.964+0000 7f6d4ee2f740 1 Processor -- start 2024-03-18T00:55:43.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.965+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a2591e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.965+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a25923d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.965+0000 7f6d4ee2f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a2593390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.965+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2800fac0 con 0x5558a2583570 2024-03-18T00:55:43.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.965+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2800f3f0 con 0x5558a25886e0 2024-03-18T00:55:43.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.965+0000 7f6d4ee2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2800c630 con 0x5558a2582b80 2024-03-18T00:55:43.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.965+0000 7f6d4ee2f740 10 client.0 _ll_register_callbacks cb 0x7f6d3c0095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:43.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.966+0000 7f6d49efe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a25923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.966+0000 7f6d49efe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a25923d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54496/0 (socket says 172.21.15.49:54496) 2024-03-18T00:55:43.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.966+0000 7f6d49efe640 1 -- 172.21.15.49:0/4069950586 learned_addr learned my addr 172.21.15.49:0/4069950586 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:43.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.968+0000 7f6d496fd640 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a2591e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.969+0000 7f6d496fd640 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a2593390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.969+0000 7f6d49efe640 1 -- 172.21.15.49:0/4069950586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 msgr2=0x5558a2591e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:43.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.969+0000 7f6d49efe640 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a2591e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.969+0000 7f6d49efe640 1 -- 172.21.15.49:0/4069950586 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 msgr2=0x5558a2593390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:43.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.969+0000 7f6d49efe640 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a2593390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:43.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.969+0000 7f6d49efe640 1 -- 172.21.15.49:0/4069950586 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558a2593980 con 0x5558a2583570 2024-03-18T00:55:43.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.971+0000 7f6d49efe640 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a25923d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00a990 tx=0x7f6d3c00ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e36f7437c9ebed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:43.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.971+0000 7f6d2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:43.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.974+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/4069950586 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c0078f0 con 0x5558a2583570 2024-03-18T00:55:43.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.975+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/4069950586 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6d3c016690 con 0x5558a2583570 2024-03-18T00:55:43.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.978+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/4069950586 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d3c0097a0 con 0x5558a2583570 2024-03-18T00:55:43.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.978+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/4069950586 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d3c039050 con 0x5558a2583570 2024-03-18T00:55:43.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.982+0000 7f6d4ee2f740 10 client.16785 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:43.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.982+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/4069950586 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5558a25ff5f0 con 0x5558a2583570 2024-03-18T00:55:43.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.984+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/4069950586 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d3c0097a0 con 0x5558a2583570 2024-03-18T00:55:43.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.984+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/4069950586 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6d3c027020 con 0x5558a2583570 2024-03-18T00:55:43.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.984+0000 7f6d4ee2f740 10 client.16785 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:43.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.984+0000 7f6d4ee2f740 10 client.16785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:43.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.984+0000 7f6d4ee2f740 20 client.16785 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:43.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.984+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a25ffee0 0x5558a2602300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:43.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.984+0000 7f6d4ee2f740 4 client.16785 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:55:43.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.984+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/4069950586 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5558a24ee960 con 0x5558a25ffee0 2024-03-18T00:55:43.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.986+0000 7f6d496fd640 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a25ffee0 0x5558a2602300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:43.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.993+0000 7f6d2d7fa640 10 client.16785 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:43.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:43.993+0000 7f6d496fd640 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a25ffee0 0x5558a2602300 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf259b52ff5d3040 server_cookie=d6f57cf041e89c01 in_seq=0 out_seq=0 2024-03-18T00:55:45.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:45 smithi049 ceph-mon[32130]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 863 MiB/s wr, 1.98k op/s 2024-03-18T00:55:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:45 smithi107 ceph-mon[36270]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 863 MiB/s wr, 1.98k op/s 2024-03-18T00:55:45.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:45 smithi106 ceph-mon[35899]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 863 MiB/s wr, 1.98k op/s 2024-03-18T00:55:46.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.336+0000 7f6d2d7fa640 1 -- 172.21.15.49:0/4069950586 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5558a24ee960 con 0x5558a25ffee0 2024-03-18T00:55:46.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.336+0000 7f6d2d7fa640 10 client.16785 handle_command_reply: tid=0 2024-03-18T00:55:46.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.336+0000 7f6d4ee2f740 1 client.16785 shutdown 2024-03-18T00:55:46.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/4069950586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 msgr2=0x5558a25923d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a25923d0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00a990 tx=0x7f6d3c00ce20 comp rx=0 tx=0).stop 2024-03-18T00:55:46.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/4069950586 shutdown_connections 2024-03-18T00:55:46.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5558a25ffee0 0x5558a2602300 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5558a25886e0 0x5558a2593390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d2d7fa640 0 client.16785 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5558a2583570 0x5558a25923d0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 --2- 172.21.15.49:0/4069950586 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5558a2582b80 0x5558a2591e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/4069950586 >> 172.21.15.49:0/4069950586 conn(0x5558a254ac50 msgr2=0x5558a258c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/4069950586 shutdown_connections 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 1 -- 172.21.15.49:0/4069950586 wait complete. 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.337+0000 7f6d4ee2f740 20 client.16785 trim_cache size 0 max 16384 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "6352c583-5f5d-4103-966f-b363adb5d999", 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:55:46.339 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:55:46.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 -- 172.21.15.49:0/3186885796 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6d300a2a00 msgr2=0x7f6d300a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6d300a2a00 0x7f6d300a4ec0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6d340bcc90 tx=0x7f6d340b8af0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 -- 172.21.15.49:0/3186885796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 msgr2=0x7f6d4410e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 0x7f6d4410e6b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009850 tx=0x7f6d3402ed50 comp rx=0 tx=0).stop 2024-03-18T00:55:46.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 -- 172.21.15.49:0/3186885796 shutdown_connections 2024-03-18T00:55:46.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f6d300a2a00 0x7f6d300a4ec0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f6d44105250 0x7f6d4410e6b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6d441028a0 0x7f6d441105b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 --2- 172.21.15.49:0/3186885796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d440fff30 0x7f6d44110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 -- 172.21.15.49:0/3186885796 >> 172.21.15.49:0/3186885796 conn(0x7f6d440fbb50 msgr2=0x7f6d440fbf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:46.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.338+0000 7f6d4af00640 1 -- 172.21.15.49:0/3186885796 shutdown_connections 2024-03-18T00:55:46.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.339+0000 7f6d4af00640 1 -- 172.21.15.49:0/3186885796 wait complete. 2024-03-18T00:55:46.368 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:55:46.369 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:55:46.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.845+0000 7f61a27fc640 1 -- 172.21.15.49:0/2819379235 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619002cca0 con 0x7f619c09d540 2024-03-18T00:55:46.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.846+0000 7f61aa1d3640 1 -- 172.21.15.49:0/2819379235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f619c09d540 msgr2=0x7f619c09d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.846+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/2819379235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f619c09d540 0x7f619c09d9c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6190009900 tx=0x7f619002bc50 comp rx=0 tx=0).stop 2024-03-18T00:55:46.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.847+0000 7f61aa1d3640 1 -- 172.21.15.49:0/2819379235 shutdown_connections 2024-03-18T00:55:46.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.847+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/2819379235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c09df00 0x7f619c0a4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.847+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/2819379235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f619c09d540 0x7f619c09d9c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.847+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/2819379235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f619c09c3a0 0x7f619c09c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.847+0000 7f61aa1d3640 1 -- 172.21.15.49:0/2819379235 >> 172.21.15.49:0/2819379235 conn(0x7f619c00b510 msgr2=0x7f619c09a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:46.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.847+0000 7f61aa1d3640 1 -- 172.21.15.49:0/2819379235 shutdown_connections 2024-03-18T00:55:46.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.847+0000 7f61aa1d3640 1 -- 172.21.15.49:0/2819379235 wait complete. 2024-03-18T00:55:46.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.849+0000 7f61aa1d3640 1 Processor -- start 2024-03-18T00:55:46.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.849+0000 7f61aa1d3640 1 -- start start 2024-03-18T00:55:46.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.850+0000 7f61aa1d3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f619c09c3a0 0x7f619c1327c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.850+0000 7f61aa1d3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f619c09d540 0x7f619c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.851+0000 7f61aa1d3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c09df00 0x7f619c139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.851+0000 7f61aa1d3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f619c0a8130 con 0x7f619c09d540 2024-03-18T00:55:46.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.851+0000 7f61aa1d3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f619c0a7fb0 con 0x7f619c09c3a0 2024-03-18T00:55:46.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.851+0000 7f61aa1d3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f619c0a82b0 con 0x7f619c09df00 2024-03-18T00:55:46.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.851+0000 7f61a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c09df00 0x7f619c139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.852+0000 7f61a3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c09df00 0x7f619c139d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59246/0 (socket says 172.21.15.49:59246) 2024-03-18T00:55:46.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.852+0000 7f61a3fff640 1 -- 172.21.15.49:0/1976785893 learned_addr learned my addr 172.21.15.49:0/1976785893 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:46.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.852+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f619c09d540 0x7f619c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.852+0000 7f61a37fe640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f619c09c3a0 0x7f619c1327c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.852+0000 7f61a3fff640 1 -- 172.21.15.49:0/1976785893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f619c09c3a0 msgr2=0x7f619c1327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.852+0000 7f61a3fff640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f619c09c3a0 0x7f619c1327c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.852+0000 7f61a3fff640 1 -- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f619c09d540 msgr2=0x7f619c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.852+0000 7f61a3fff640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f619c09d540 0x7f619c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.852+0000 7f61a3fff640 1 -- 172.21.15.49:0/1976785893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619c13a3e0 con 0x7f619c09df00 2024-03-18T00:55:46.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.854+0000 7f61a3fff640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c09df00 0x7f619c139d70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f61a4066f10 tx=0x7f61a4078820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:46.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.855+0000 7f61a0ff9640 1 -- 172.21.15.49:0/1976785893 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4084070 con 0x7f619c09df00 2024-03-18T00:55:46.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.855+0000 7f61a0ff9640 1 -- 172.21.15.49:0/1976785893 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61a4062920 con 0x7f619c09df00 2024-03-18T00:55:46.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.855+0000 7f61a0ff9640 1 -- 172.21.15.49:0/1976785893 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a407d380 con 0x7f619c09df00 2024-03-18T00:55:46.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.855+0000 7f61aa1d3640 1 -- 172.21.15.49:0/1976785893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619c13a6d0 con 0x7f619c09df00 2024-03-18T00:55:46.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.855+0000 7f61aa1d3640 1 -- 172.21.15.49:0/1976785893 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f619c13aba0 con 0x7f619c09df00 2024-03-18T00:55:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.861+0000 7f61acdb7740 1 Processor -- start 2024-03-18T00:55:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.862+0000 7f61a0ff9640 1 -- 172.21.15.49:0/1976785893 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f61a4087070 con 0x7f619c09df00 2024-03-18T00:55:46.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.862+0000 7f61acdb7740 1 -- start start 2024-03-18T00:55:46.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.863+0000 7f61a0ff9640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61940a2880 0x7f61940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.863+0000 7f61a0ff9640 1 -- 172.21.15.49:0/1976785893 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f61a4139b60 con 0x7f619c09df00 2024-03-18T00:55:46.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.863+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d47360 0x55ab90d47780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.863+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61940a2880 0x7f61940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.863+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47d50 0x55ab90d4c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.864+0000 7f61a2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47d50 0x55ab90d4c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.864+0000 7f61a2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47d50 0x55ab90d4c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54524/0 (socket says 172.21.15.49:54524) 2024-03-18T00:55:46.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.864+0000 7f61a2ffd640 1 -- 172.21.15.49:0/641975711 learned_addr learned my addr 172.21.15.49:0/641975711 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.864+0000 7f61acdb7740 1 --2- 172.21.15.49:0/641975711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d4cec0 0x55ab90d4f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.864+0000 7f61acdb7740 1 -- 172.21.15.49:0/641975711 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ab90cac500 con 0x55ab90d47d50 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.864+0000 7f61acdb7740 1 -- 172.21.15.49:0/641975711 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ab90c54be0 con 0x55ab90d47360 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.864+0000 7f61acdb7740 1 -- 172.21.15.49:0/641975711 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ab90ca85a0 con 0x55ab90d4cec0 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.864+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61940a2880 0x7f61940a4d40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6190006010 tx=0x7f6190004f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.865+0000 7f61a2ffd640 1 -- 172.21.15.49:0/641975711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d4cec0 msgr2=0x55ab90d4f350 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.865+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/641975711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d4cec0 0x55ab90d4f350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.865+0000 7f61a2ffd640 1 -- 172.21.15.49:0/641975711 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d47360 msgr2=0x55ab90d47780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.865+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/641975711 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d47360 0x55ab90d47780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.865 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.865+0000 7f61a2ffd640 1 -- 172.21.15.49:0/641975711 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab90c684b0 con 0x55ab90d47d50 2024-03-18T00:55:46.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.865+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/641975711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47d50 0x55ab90d4c850 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6190005f70 tx=0x7f6190046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b98a400de0c0798e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:46.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.865+0000 7f618dffb640 1 -- 172.21.15.49:0/641975711 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61900046c0 con 0x55ab90d47d50 2024-03-18T00:55:46.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.866+0000 7f618dffb640 1 -- 172.21.15.49:0/641975711 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6190050070 con 0x55ab90d47d50 2024-03-18T00:55:46.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.866+0000 7f618dffb640 1 -- 172.21.15.49:0/641975711 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6190003830 con 0x55ab90d47d50 2024-03-18T00:55:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.867+0000 7f61acdb7740 1 -- 172.21.15.49:0/641975711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47d50 msgr2=0x55ab90d4c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.867+0000 7f61acdb7740 1 --2- 172.21.15.49:0/641975711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47d50 0x55ab90d4c850 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6190005f70 tx=0x7f6190046000 comp rx=0 tx=0).stop 2024-03-18T00:55:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.868+0000 7f61acdb7740 1 -- 172.21.15.49:0/641975711 shutdown_connections 2024-03-18T00:55:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.868+0000 7f61acdb7740 1 --2- 172.21.15.49:0/641975711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d4cec0 0x55ab90d4f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.868+0000 7f61acdb7740 1 --2- 172.21.15.49:0/641975711 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47d50 0x55ab90d4c850 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.868+0000 7f61acdb7740 1 --2- 172.21.15.49:0/641975711 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d47360 0x55ab90d47780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.868+0000 7f61acdb7740 1 -- 172.21.15.49:0/641975711 >> 172.21.15.49:0/641975711 conn(0x55ab90d428d0 msgr2=0x55ab90d42cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.869+0000 7f61acdb7740 1 -- 172.21.15.49:0/641975711 shutdown_connections 2024-03-18T00:55:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.869+0000 7f61acdb7740 1 -- 172.21.15.49:0/641975711 wait complete. 2024-03-18T00:55:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.870+0000 7f61acdb7740 1 -- start start 2024-03-18T00:55:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.870+0000 7f61acdb7740 1 Processor -- start 2024-03-18T00:55:46.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.871+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d47360 0x55ab90d51700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.871+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d51c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.871+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d4cec0 0x55ab90d58c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.872+0000 7f61acdb7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ab90d4fe60 con 0x55ab90d4cec0 2024-03-18T00:55:46.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.872+0000 7f61acdb7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ab90c68160 con 0x55ab90d47360 2024-03-18T00:55:46.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.872+0000 7f61a2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d51c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.872+0000 7f61a37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d47360 0x55ab90d51700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.872+0000 7f61a2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d51c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59262/0 (socket says 172.21.15.49:59262) 2024-03-18T00:55:46.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.872+0000 7f61a2ffd640 1 -- 172.21.15.49:0/2520101980 learned_addr learned my addr 172.21.15.49:0/2520101980 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:46.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.872+0000 7f61acdb7740 1 -- 172.21.15.49:0/2520101980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ab90d4ffe0 con 0x55ab90d47d50 2024-03-18T00:55:46.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.873+0000 7f61acdb7740 10 client.0 _ll_register_callbacks cb 0x55ab90c4c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:46.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.873+0000 7f61a3fff640 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d4cec0 0x55ab90d58c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.873+0000 7f61a2ffd640 1 -- 172.21.15.49:0/2520101980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d47360 msgr2=0x55ab90d51700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:46.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.874+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d47360 0x55ab90d51700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.874+0000 7f61a2ffd640 1 -- 172.21.15.49:0/2520101980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d4cec0 msgr2=0x55ab90d58c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:46.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.874+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d4cec0 0x55ab90d58c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.874+0000 7f61a2ffd640 1 -- 172.21.15.49:0/2520101980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab90a8a7d0 con 0x55ab90d47d50 2024-03-18T00:55:46.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.874+0000 7f61a3fff640 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d4cec0 0x55ab90d58c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:55:46.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.874+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d51c40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f619002c7d0 tx=0x7f6190005700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd7b16b4e2369295 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:46.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.874+0000 7f6182ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:55:46.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.875+0000 7f6182ffd640 1 -- 172.21.15.49:0/2520101980 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6190058070 con 0x55ab90d47d50 2024-03-18T00:55:46.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.875+0000 7f6182ffd640 1 -- 172.21.15.49:0/2520101980 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6190003a00 con 0x55ab90d47d50 2024-03-18T00:55:46.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.875+0000 7f61acdb7740 1 -- 172.21.15.49:0/2520101980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6190050070 con 0x55ab90d47d50 2024-03-18T00:55:46.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.876+0000 7f6182ffd640 1 -- 172.21.15.49:0/2520101980 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6190064050 con 0x55ab90d47d50 2024-03-18T00:55:46.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.876+0000 7f61acdb7740 10 client.26221 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:46.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.876+0000 7f61acdb7740 1 -- 172.21.15.49:0/2520101980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ab909c03f0 con 0x55ab90d47d50 2024-03-18T00:55:46.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.877+0000 7f6182ffd640 1 -- 172.21.15.49:0/2520101980 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6190047070 con 0x55ab90d47d50 2024-03-18T00:55:46.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.877+0000 7f6182ffd640 1 -- 172.21.15.49:0/2520101980 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6190040b70 con 0x55ab90d47d50 2024-03-18T00:55:46.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.877+0000 7f61acdb7740 10 client.26221 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:46.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.877+0000 7f61acdb7740 10 client.26221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:46.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.877+0000 7f61acdb7740 20 client.26221 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:46.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.877+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ab90d5cdc0 0x55ab90d5f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.877+0000 7f61acdb7740 4 client.26221 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:46.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.877+0000 7f61acdb7740 1 -- 172.21.15.49:0/2520101980 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6190050070 con 0x55ab90d5cdc0 2024-03-18T00:55:46.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.881+0000 7f61a37fe640 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ab90d5cdc0 0x55ab90d5f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.884+0000 7f6182ffd640 10 client.26221 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:46.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.884+0000 7f61a37fe640 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ab90d5cdc0 0x55ab90d5f1b0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d08ae2e95a62f16 server_cookie=abf6f10c92bdd0b2 in_seq=0 out_seq=0 2024-03-18T00:55:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.890+0000 7f6182ffd640 1 -- 172.21.15.49:0/2520101980 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f6190050070 con 0x55ab90d5cdc0 2024-03-18T00:55:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.890+0000 7f6182ffd640 10 client.26221 handle_command_reply: tid=0 2024-03-18T00:55:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.890+0000 7f61acdb7740 1 client.26221 shutdown 2024-03-18T00:55:46.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.891+0000 7f61acdb7740 1 -- 172.21.15.49:0/2520101980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 msgr2=0x55ab90d51c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.891+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d51c40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f619002c7d0 tx=0x7f6190005700 comp rx=0 tx=0).stop 2024-03-18T00:55:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.892+0000 7f61acdb7740 1 -- 172.21.15.49:0/2520101980 shutdown_connections 2024-03-18T00:55:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.892+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ab90d5cdc0 0x55ab90d5f1b0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.892+0000 7f6182ffd640 0 client.26221 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.892+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d4cec0 0x55ab90d58c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.892+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d51c40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.892+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2520101980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d47360 0x55ab90d51700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.892+0000 7f61acdb7740 1 -- 172.21.15.49:0/2520101980 >> 172.21.15.49:0/2520101980 conn(0x55ab90d428d0 msgr2=0x55ab90d41b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.892+0000 7f61acdb7740 1 -- 172.21.15.49:0/2520101980 shutdown_connections 2024-03-18T00:55:46.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.893+0000 7f61acdb7740 1 -- 172.21.15.49:0/2520101980 wait complete. 2024-03-18T00:55:46.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.893+0000 7f61acdb7740 20 client.26221 trim_cache size 0 max 16384 2024-03-18T00:55:46.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.932+0000 7f61acdb7740 1 Processor -- start 2024-03-18T00:55:46.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.932+0000 7f61acdb7740 1 -- start start 2024-03-18T00:55:46.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.932+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47360 0x55ab90d41280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.932+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d58c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.932+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 0x55ab90d417c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.933+0000 7f61acdb7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f619c143980 con 0x55ab90d47360 2024-03-18T00:55:46.932 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.933+0000 7f61acdb7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f619c1454c0 con 0x55ab90d4cec0 2024-03-18T00:55:46.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.933+0000 7f61acdb7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f619c144050 con 0x55ab90d47d50 2024-03-18T00:55:46.933 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47360 0x55ab90d41280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47360 0x55ab90d41280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54548/0 (socket says 172.21.15.49:54548) 2024-03-18T00:55:46.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a37fe640 1 -- 172.21.15.49:0/2637456419 learned_addr learned my addr 172.21.15.49:0/2637456419 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:46.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/2637456419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d58c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a3fff640 1 --2- 172.21.15.49:0/2637456419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 0x55ab90d417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a37fe640 1 -- 172.21.15.49:0/2637456419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 msgr2=0x55ab90d58c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:46.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a37fe640 1 --2- 172.21.15.49:0/2637456419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d58c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a37fe640 1 -- 172.21.15.49:0/2637456419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 msgr2=0x55ab90d417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:46.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a37fe640 1 --2- 172.21.15.49:0/2637456419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 0x55ab90d417c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.934+0000 7f61a37fe640 1 -- 172.21.15.49:0/2637456419 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab909800f0 con 0x55ab90d47360 2024-03-18T00:55:46.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.935+0000 7f61a37fe640 1 --2- 172.21.15.49:0/2637456419 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47360 0x55ab90d41280 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6198004790 tx=0x7f619800b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda7d5de14f83215 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:46.935 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.935+0000 7f618d7fa640 1 -- 172.21.15.49:0/2637456419 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198021070 con 0x55ab90d47360 2024-03-18T00:55:46.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.936+0000 7f618d7fa640 1 -- 172.21.15.49:0/2637456419 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6198015b40 con 0x55ab90d47360 2024-03-18T00:55:46.936 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f618d7fa640 1 -- 172.21.15.49:0/2637456419 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6198024b70 con 0x55ab90d47360 2024-03-18T00:55:46.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f61acdb7740 1 -- 172.21.15.49:0/2637456419 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47360 msgr2=0x55ab90d41280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2637456419 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47360 0x55ab90d41280 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6198004790 tx=0x7f619800b720 comp rx=0 tx=0).stop 2024-03-18T00:55:46.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f61acdb7740 1 -- 172.21.15.49:0/2637456419 shutdown_connections 2024-03-18T00:55:46.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2637456419 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 0x55ab90d417c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2637456419 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d58c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f61acdb7740 1 --2- 172.21.15.49:0/2637456419 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d47360 0x55ab90d41280 secure :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f6198004790 tx=0x7f619800b720 comp rx=0 tx=0).stop 2024-03-18T00:55:46.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f61acdb7740 1 -- 172.21.15.49:0/2637456419 >> 172.21.15.49:0/2637456419 conn(0x55ab90d428d0 msgr2=0x55ab90d51700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:46.937 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f61acdb7740 1 -- 172.21.15.49:0/2637456419 shutdown_connections 2024-03-18T00:55:46.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.937+0000 7f61acdb7740 1 -- 172.21.15.49:0/2637456419 wait complete. 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.938+0000 7f61acdb7740 1 -- start start 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.938+0000 7f61acdb7740 1 Processor -- start 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.938+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d55dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.938+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 0x55ab90d56310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.938+0000 7f61acdb7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d5cdc0 0x55ab90d57260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.938+0000 7f61acdb7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f619c144df0 con 0x55ab90d5cdc0 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.938+0000 7f61acdb7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f619c144720 con 0x55ab90d4cec0 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.938+0000 7f61acdb7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f619c141960 con 0x55ab90d47d50 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61acdb7740 10 client.0 _ll_register_callbacks cb 0x7f6190009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61a3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d5cdc0 0x55ab90d57260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61a3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d5cdc0 0x55ab90d57260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54556/0 (socket says 172.21.15.49:54556) 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61a3fff640 1 -- 172.21.15.49:0/3694147817 learned_addr learned my addr 172.21.15.49:0/3694147817 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:46.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61a2ffd640 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 0x55ab90d56310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.940 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61a3fff640 1 -- 172.21.15.49:0/3694147817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 msgr2=0x55ab90d55dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61a3fff640 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d55dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61a3fff640 1 -- 172.21.15.49:0/3694147817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 msgr2=0x55ab90d56310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61a3fff640 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 0x55ab90d56310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.939+0000 7f61a3fff640 1 -- 172.21.15.49:0/3694147817 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab90c739d0 con 0x55ab90d5cdc0 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.940+0000 7f61a3fff640 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d5cdc0 0x55ab90d57260 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f61a406b1d0 tx=0x7f61a40673e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbbafa5b922c50d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.940+0000 7f6182ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.940+0000 7f6182ffd640 1 -- 172.21.15.49:0/3694147817 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4106be0 con 0x55ab90d5cdc0 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.940+0000 7f61acdb7740 1 -- 172.21.15.49:0/3694147817 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6198015b40 con 0x55ab90d5cdc0 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.940+0000 7f6182ffd640 1 -- 172.21.15.49:0/3694147817 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f61a407f480 con 0x55ab90d5cdc0 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.940+0000 7f6182ffd640 1 -- 172.21.15.49:0/3694147817 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f61a41064e0 con 0x55ab90d5cdc0 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.940+0000 7f61acdb7740 10 client.16803 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.940+0000 7f61acdb7740 1 -- 172.21.15.49:0/3694147817 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ab90d58a00 con 0x55ab90d5cdc0 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.940+0000 7f6182ffd640 1 -- 172.21.15.49:0/3694147817 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6198015b40 con 0x55ab90d5cdc0 2024-03-18T00:55:46.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.941+0000 7f6182ffd640 1 -- 172.21.15.49:0/3694147817 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f61a4105930 con 0x55ab90d5cdc0 2024-03-18T00:55:46.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.941+0000 7f61acdb7740 10 client.16803 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:46.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.941+0000 7f61acdb7740 10 client.16803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:46.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.941+0000 7f61acdb7740 20 client.16803 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:46.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.941+0000 7f61acdb7740 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ab90dc3190 0x55ab90dc55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:46.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.941+0000 7f61acdb7740 4 client.16803 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:55:46.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.941+0000 7f61acdb7740 1 -- 172.21.15.49:0/3694147817 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ab90cb9a40 con 0x55ab90dc3190 2024-03-18T00:55:46.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.942+0000 7f61a37fe640 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ab90dc3190 0x55ab90dc55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:46.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.944+0000 7f61a37fe640 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ab90dc3190 0x55ab90dc55b0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a02c8f1e9a20abce server_cookie=3b7fb1030a7e7432 in_seq=0 out_seq=0 2024-03-18T00:55:46.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.944+0000 7f6182ffd640 10 client.16803 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:46.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f6182ffd640 1 -- 172.21.15.49:0/3694147817 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55ab90cb9a40 con 0x55ab90dc3190 2024-03-18T00:55:46.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f6182ffd640 10 client.16803 handle_command_reply: tid=0 2024-03-18T00:55:46.949 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f61acdb7740 1 client.16803 shutdown 2024-03-18T00:55:46.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f61acdb7740 1 -- 172.21.15.49:0/3694147817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d5cdc0 msgr2=0x55ab90d57260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f61acdb7740 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d5cdc0 0x55ab90d57260 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f61a406b1d0 tx=0x7f61a40673e0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f61acdb7740 1 -- 172.21.15.49:0/3694147817 shutdown_connections 2024-03-18T00:55:46.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f61acdb7740 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ab90dc3190 0x55ab90dc55b0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f61acdb7740 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ab90d4cec0 0x55ab90d56310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f61acdb7740 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ab90d5cdc0 0x55ab90d57260 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f61acdb7740 1 --2- 172.21.15.49:0/3694147817 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ab90d47d50 0x55ab90d55dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f61acdb7740 1 -- 172.21.15.49:0/3694147817 >> 172.21.15.49:0/3694147817 conn(0x55ab90d428d0 msgr2=0x55ab90d41b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.950+0000 7f6182ffd640 0 client.16803 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.951+0000 7f61acdb7740 1 -- 172.21.15.49:0/3694147817 shutdown_connections 2024-03-18T00:55:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.951+0000 7f61acdb7740 1 -- 172.21.15.49:0/3694147817 wait complete. 2024-03-18T00:55:46.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.951+0000 7f61acdb7740 20 client.16803 trim_cache size 0 max 16384 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (3 inodes in the stack)", 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout: "6352c583-5f5d-4103-966f-b363adb5d999": { 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "6352c583-5f5d-4103-966f-b363adb5d999", 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:55:46.952 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:55:46.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 -- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61940a2880 msgr2=0x7f61940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61940a2880 0x7f61940a4d40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6190006010 tx=0x7f6190004f10 comp rx=0 tx=0).stop 2024-03-18T00:55:46.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 -- 172.21.15.49:0/1976785893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c09df00 msgr2=0x7f619c139d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:46.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c09df00 0x7f619c139d70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f61a4066f10 tx=0x7f61a4078820 comp rx=0 tx=0).stop 2024-03-18T00:55:46.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 -- 172.21.15.49:0/1976785893 shutdown_connections 2024-03-18T00:55:46.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61940a2880 0x7f61940a4d40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f619c09df00 0x7f619c139d70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f619c09d540 0x7f619c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 --2- 172.21.15.49:0/1976785893 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f619c09c3a0 0x7f619c1327c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:46.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.952+0000 7f61aa1d3640 1 -- 172.21.15.49:0/1976785893 >> 172.21.15.49:0/1976785893 conn(0x7f619c00b510 msgr2=0x7f619c0a0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:46.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.953+0000 7f61aa1d3640 1 -- 172.21.15.49:0/1976785893 shutdown_connections 2024-03-18T00:55:46.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:55:46.953+0000 7f61aa1d3640 1 -- 172.21.15.49:0/1976785893 wait complete. 2024-03-18T00:55:46.977 INFO:tasks.cephfs.filesystem:scrub status for tag:6352c583-5f5d-4103-966f-b363adb5d999 - {'path': '/', 'tag': '6352c583-5f5d-4103-966f-b363adb5d999', 'options': 'recursive,force'} 2024-03-18T00:55:47.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:47 smithi049 ceph-mon[32130]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 896 MiB/s wr, 2.32k op/s 2024-03-18T00:55:47.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:47 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:55:47.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:47 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:47.321 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:47 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:55:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:47 smithi107 ceph-mon[36270]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 896 MiB/s wr, 2.32k op/s 2024-03-18T00:55:47.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:47 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:55:47.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:47 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:47.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:47 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:55:47.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:47 smithi106 ceph-mon[35899]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 896 MiB/s wr, 2.32k op/s 2024-03-18T00:55:47.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:47 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:55:47.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:47 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:47.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:47 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:55:49.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:49 smithi049 ceph-mon[32130]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 839 MiB/s wr, 2.31k op/s 2024-03-18T00:55:49.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:49 smithi107 ceph-mon[36270]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 839 MiB/s wr, 2.31k op/s 2024-03-18T00:55:49.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:49 smithi106 ceph-mon[35899]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 839 MiB/s wr, 2.31k op/s 2024-03-18T00:55:50.617 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.617+0000 7fd9f9227640 1 -- 172.21.15.49:0/37642170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f41012a0 msgr2=0x7fd9f4108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.617+0000 7fd9f9227640 1 --2- 172.21.15.49:0/37642170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f41012a0 0x7fd9f4108b60 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009ca0 tx=0x7fd9ec02c120 comp rx=0 tx=0).stop 2024-03-18T00:55:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.617+0000 7fd9f9227640 1 -- 172.21.15.49:0/37642170 shutdown_connections 2024-03-18T00:55:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.617+0000 7fd9f9227640 1 --2- 172.21.15.49:0/37642170 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9f4100900 0x7fd9f4100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.617+0000 7fd9f9227640 1 --2- 172.21.15.49:0/37642170 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f41012a0 0x7fd9f4108b60 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.617+0000 7fd9f9227640 1 --2- 172.21.15.49:0/37642170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f4101cb0 0x7fd9f41020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.617+0000 7fd9f9227640 1 -- 172.21.15.49:0/37642170 >> 172.21.15.49:0/37642170 conn(0x7fd9f40fb430 msgr2=0x7fd9f40fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.617+0000 7fd9f9227640 1 -- 172.21.15.49:0/37642170 shutdown_connections 2024-03-18T00:55:50.618 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.617+0000 7fd9f9227640 1 -- 172.21.15.49:0/37642170 wait complete. 2024-03-18T00:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.618+0000 7fd9f9227640 1 Processor -- start 2024-03-18T00:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f9227640 1 -- start start 2024-03-18T00:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f9227640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f4100900 0x7fd9f4199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f9227640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9f41012a0 0x7fd9f419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f9227640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f4101cb0 0x7fd9f41a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f9227640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f410d500 con 0x7fd9f4100900 2024-03-18T00:55:50.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f9227640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f410d380 con 0x7fd9f41012a0 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f9227640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f410d680 con 0x7fd9f4101cb0 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f2d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f4100900 0x7fd9f4199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f2d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f4100900 0x7fd9f4199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52258/0 (socket says 172.21.15.49:52258) 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/2556627207 learned_addr learned my addr 172.21.15.49:0/2556627207 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/2556627207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f4101cb0 msgr2=0x7fd9f41a1210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f4101cb0 0x7fd9f41a1210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/2556627207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9f41012a0 msgr2=0x7fd9f419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9f41012a0 0x7fd9f419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.619+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/2556627207 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f41a18b0 con 0x7fd9f4100900 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.620+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f4100900 0x7fd9f4199c70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400e9f0 tx=0x7fd9e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.620+0000 7fd9dbfff640 1 -- 172.21.15.49:0/2556627207 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e400cd80 con 0x7fd9f4100900 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.620+0000 7fd9f9227640 1 -- 172.21.15.49:0/2556627207 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f41a1ba0 con 0x7fd9f4100900 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.620+0000 7fd9dbfff640 1 -- 172.21.15.49:0/2556627207 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9e4005500 con 0x7fd9f4100900 2024-03-18T00:55:50.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.620+0000 7fd9dbfff640 1 -- 172.21.15.49:0/2556627207 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4010690 con 0x7fd9f4100900 2024-03-18T00:55:50.622 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.620+0000 7fd9f9227640 1 -- 172.21.15.49:0/2556627207 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f41a20f0 con 0x7fd9f4100900 2024-03-18T00:55:50.623 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.624+0000 7fd9dbfff640 1 -- 172.21.15.49:0/2556627207 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd9e40046c0 con 0x7fd9f4100900 2024-03-18T00:55:50.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.624+0000 7fd9fbe15740 1 Processor -- start 2024-03-18T00:55:50.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.624+0000 7fd9fbe15740 1 -- start start 2024-03-18T00:55:50.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f29760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29d30 0x557a99f2e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9dbfff640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd9dc0a2920 0x7fd9dc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f2eea0 0x557a99f31330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9fbe15740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557a99e8ea40 con 0x557a99f29340 2024-03-18T00:55:50.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9fbe15740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557a99e37920 con 0x557a99f2eea0 2024-03-18T00:55:50.626 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9fbe15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557a99e8aae0 con 0x557a99f29d30 2024-03-18T00:55:50.626 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9f359e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f29760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.626 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9f259c640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd9dc0a2920 0x7fd9dc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.626 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9f359e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f29760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52260/0 (socket says 172.21.15.49:52260) 2024-03-18T00:55:50.626 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9f359e640 1 -- 172.21.15.49:0/966009857 learned_addr learned my addr 172.21.15.49:0/966009857 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:50.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:50 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:55:50.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:50 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9f359e640 1 -- 172.21.15.49:0/966009857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29d30 msgr2=0x557a99f2e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:50.629 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9f359e640 1 --2- 172.21.15.49:0/966009857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29d30 0x557a99f2e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9f359e640 1 -- 172.21.15.49:0/966009857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f2eea0 msgr2=0x557a99f31330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9f359e640 1 --2- 172.21.15.49:0/966009857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f2eea0 0x557a99f31330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.625+0000 7fd9f359e640 1 -- 172.21.15.49:0/966009857 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a99e4f460 con 0x557a99f29340 2024-03-18T00:55:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.626+0000 7fd9f359e640 1 --2- 172.21.15.49:0/966009857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f29760 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec005e30 tx=0x7fd9ec0323f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d09d1350aec780 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.626+0000 7fd9d97fa640 1 -- 172.21.15.49:0/966009857 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec041020 con 0x557a99f29340 2024-03-18T00:55:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.626+0000 7fd9d97fa640 1 -- 172.21.15.49:0/966009857 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9ec032d00 con 0x557a99f29340 2024-03-18T00:55:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.626+0000 7fd9d97fa640 1 -- 172.21.15.49:0/966009857 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec039040 con 0x557a99f29340 2024-03-18T00:55:50.630 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.626+0000 7fd9dbfff640 1 -- 172.21.15.49:0/2556627207 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd9e40cefa0 con 0x7fd9f4100900 2024-03-18T00:55:50.631 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9fbe15740 1 -- 172.21.15.49:0/966009857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 msgr2=0x557a99f29760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:50.631 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/966009857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f29760 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec005e30 tx=0x7fd9ec0323f0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.631 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9f259c640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd9dc0a2920 0x7fd9dc0a4de0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd9f419b170 tx=0x7fd9e8006c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:50.631 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9fbe15740 1 -- 172.21.15.49:0/966009857 shutdown_connections 2024-03-18T00:55:50.631 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/966009857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f2eea0 0x557a99f31330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.631 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/966009857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29d30 0x557a99f2e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.631 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/966009857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f29760 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.631 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9fbe15740 1 -- 172.21.15.49:0/966009857 >> 172.21.15.49:0/966009857 conn(0x557a99f248b0 msgr2=0x557a99f24cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:50.631 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9fbe15740 1 -- 172.21.15.49:0/966009857 shutdown_connections 2024-03-18T00:55:50.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.627+0000 7fd9fbe15740 1 -- 172.21.15.49:0/966009857 wait complete. 2024-03-18T00:55:50.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.628+0000 7fd9fbe15740 1 -- start start 2024-03-18T00:55:50.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.628+0000 7fd9fbe15740 1 Processor -- start 2024-03-18T00:55:50.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.628+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29340 0x557a99f336b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f33bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f2eea0 0x557a99f3ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9fbe15740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557a99f31df0 con 0x557a99f2eea0 2024-03-18T00:55:50.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9fbe15740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557a99e4ad70 con 0x557a99f29d30 2024-03-18T00:55:50.632 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9fbe15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557a99f31f70 con 0x557a99f29340 2024-03-18T00:55:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9fbe15740 10 client.0 _ll_register_callbacks cb 0x557a99ba2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9f359e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f2eea0 0x557a99f3ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9f359e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f2eea0 0x557a99f3ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52274/0 (socket says 172.21.15.49:52274) 2024-03-18T00:55:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9f359e640 1 -- 172.21.15.49:0/2937915084 learned_addr learned my addr 172.21.15.49:0/2937915084 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9f359e640 1 -- 172.21.15.49:0/2937915084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29340 msgr2=0x557a99f336b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.629+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29340 0x557a99f336b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.630+0000 7fd9f359e640 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29340 0x557a99f336b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.633 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.630+0000 7fd9f359e640 1 -- 172.21.15.49:0/2937915084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 msgr2=0x557a99f33bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.630+0000 7fd9f359e640 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f33bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.630+0000 7fd9f359e640 1 -- 172.21.15.49:0/2937915084 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a99ba3210 con 0x557a99f2eea0 2024-03-18T00:55:50.634 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.630+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29340 0x557a99f336b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:55:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.630+0000 7fd9f359e640 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f2eea0 0x557a99f3ac30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec02c660 tx=0x7fd9ec009990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed53f43c904d968 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.630+0000 7fd9d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:50.635 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.631+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/2937915084 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec041030 con 0x557a99f2eea0 2024-03-18T00:55:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.631+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/2937915084 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd9ec003f50 con 0x557a99f2eea0 2024-03-18T00:55:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.631+0000 7fd9fbe15740 1 -- 172.21.15.49:0/2937915084 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9ec032d00 con 0x557a99f2eea0 2024-03-18T00:55:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.631+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/2937915084 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd9ec055050 con 0x557a99f2eea0 2024-03-18T00:55:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.631+0000 7fd9fbe15740 10 client.16815 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.631+0000 7fd9fbe15740 1 -- 172.21.15.49:0/2937915084 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a99c6f620 con 0x557a99f2eea0 2024-03-18T00:55:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.632+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/2937915084 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9ec032d00 con 0x557a99f2eea0 2024-03-18T00:55:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.632+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/2937915084 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd9ec03c030 con 0x557a99f2eea0 2024-03-18T00:55:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.633+0000 7fd9fbe15740 10 client.16815 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.633+0000 7fd9fbe15740 10 client.16815 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.633+0000 7fd9fbe15740 20 client.16815 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.633+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557a99f3f170 0x557a99f41560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.633+0000 7fd9fbe15740 4 client.16815 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:55:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.633+0000 7fd9fbe15740 1 -- 172.21.15.49:0/2937915084 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a99e9bf80 con 0x557a99f3f170 2024-03-18T00:55:50.637 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.634+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557a99f3f170 0x557a99f41560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.638+0000 7fd9d6ff5640 10 client.16815 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:50.638 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.638+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557a99f3f170 0x557a99f41560 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcdbd9ff981ed982 server_cookie=f0fce20790c6f5ff in_seq=0 out_seq=0 2024-03-18T00:55:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.645+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/2937915084 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557a99e9bf80 con 0x557a99f3f170 2024-03-18T00:55:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.645+0000 7fd9d6ff5640 10 client.16815 handle_command_reply: tid=0 2024-03-18T00:55:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.645+0000 7fd9fbe15740 1 client.16815 shutdown 2024-03-18T00:55:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 -- 172.21.15.49:0/2937915084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f2eea0 msgr2=0x557a99f3ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f2eea0 0x557a99f3ac30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec02c660 tx=0x7fd9ec009990 comp rx=0 tx=0).stop 2024-03-18T00:55:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 -- 172.21.15.49:0/2937915084 shutdown_connections 2024-03-18T00:55:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557a99f3f170 0x557a99f41560 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.646 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f33bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f2eea0 0x557a99f3ac30 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/2937915084 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f29340 0x557a99f336b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 -- 172.21.15.49:0/2937915084 >> 172.21.15.49:0/2937915084 conn(0x557a99f248b0 msgr2=0x557a99f23bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9d6ff5640 0 client.16815 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 -- 172.21.15.49:0/2937915084 shutdown_connections 2024-03-18T00:55:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 1 -- 172.21.15.49:0/2937915084 wait complete. 2024-03-18T00:55:50.647 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.646+0000 7fd9fbe15740 20 client.16815 trim_cache size 0 max 16384 2024-03-18T00:55:50.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:50 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:55:50.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:50 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:50.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:50 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:55:50.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:50 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.718+0000 7fd9fbe15740 1 Processor -- start 2024-03-18T00:55:50.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.718+0000 7fd9fbe15740 1 -- start start 2024-03-18T00:55:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.719+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f23320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.719+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f3abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.719+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 0x557a99f23860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.719+0000 7fd9fbe15740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f41ab010 con 0x557a99f29340 2024-03-18T00:55:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.719+0000 7fd9fbe15740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f41acb50 con 0x557a99f29d30 2024-03-18T00:55:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.719+0000 7fd9fbe15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f41ab6e0 con 0x557a99f2eea0 2024-03-18T00:55:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.719+0000 7fd9f2d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f23320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.719+0000 7fd9f2d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f23320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52276/0 (socket says 172.21.15.49:52276) 2024-03-18T00:55:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.719+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/1203029463 learned_addr learned my addr 172.21.15.49:0/1203029463 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.720+0000 7fd9f259c640 1 --2- 172.21.15.49:0/1203029463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f3abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.720+0000 7fd9f359e640 1 --2- 172.21.15.49:0/1203029463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 0x557a99f23860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.720+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/1203029463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 msgr2=0x557a99f23860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:50.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.720+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/1203029463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 0x557a99f23860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.720+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/1203029463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 msgr2=0x557a99f3abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:50.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.720+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/1203029463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f3abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.720+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/1203029463 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a99f28f80 con 0x557a99f29340 2024-03-18T00:55:50.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.721+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/1203029463 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f23320 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400c6a0 tx=0x7fd9e409b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e9a57c3338ff50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:50.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.721+0000 7fd9d8ff9640 1 -- 172.21.15.49:0/1203029463 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e409c020 con 0x557a99f29340 2024-03-18T00:55:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.722+0000 7fd9d8ff9640 1 -- 172.21.15.49:0/1203029463 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9e409b920 con 0x557a99f29340 2024-03-18T00:55:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.722+0000 7fd9d8ff9640 1 -- 172.21.15.49:0/1203029463 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4097f90 con 0x557a99f29340 2024-03-18T00:55:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.723+0000 7fd9fbe15740 1 -- 172.21.15.49:0/1203029463 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 msgr2=0x557a99f23320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:50.722 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.723+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/1203029463 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f23320 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400c6a0 tx=0x7fd9e409b7a0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.723 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.723+0000 7fd9fbe15740 1 -- 172.21.15.49:0/1203029463 shutdown_connections 2024-03-18T00:55:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.723+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/1203029463 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f3abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.723+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/1203029463 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 0x557a99f23860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.724+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/1203029463 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f23320 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.724+0000 7fd9fbe15740 1 -- 172.21.15.49:0/1203029463 >> 172.21.15.49:0/1203029463 conn(0x557a99f248b0 msgr2=0x557a99f336b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.724+0000 7fd9fbe15740 1 -- 172.21.15.49:0/1203029463 shutdown_connections 2024-03-18T00:55:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.724+0000 7fd9fbe15740 1 -- 172.21.15.49:0/1203029463 wait complete. 2024-03-18T00:55:50.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.725+0000 7fd9fbe15740 1 -- start start 2024-03-18T00:55:50.725 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.726+0000 7fd9fbe15740 1 Processor -- start 2024-03-18T00:55:50.726 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.726+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f38510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f38a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9f2d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f38510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9fbe15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 0x557a99f39a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9f2d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f38510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:52280/0 (socket says 172.21.15.49:52280) 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/3299494569 learned_addr learned my addr 172.21.15.49:0/3299494569 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f41ac480 con 0x557a99f29340 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f41abdb0 con 0x557a99f29d30 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f41a8ff0 con 0x557a99f2eea0 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9fbe15740 10 client.0 _ll_register_callbacks cb 0x7fd9e4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:55:50.727 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9f359e640 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 0x557a99f39a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/3299494569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 msgr2=0x557a99f39a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:55:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 0x557a99f39a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/3299494569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 msgr2=0x557a99f38a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:55:50.728 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f38a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.727+0000 7fd9f2d9d640 1 -- 172.21.15.49:0/3299494569 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a99f3a050 con 0x557a99f29340 2024-03-18T00:55:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.728+0000 7fd9f2d9d640 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f38510 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd9e40097d0 tx=0x7fd9e409b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4519766700f43c40 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:55:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.728+0000 7fd9d6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:55:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.728+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/3299494569 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e409fa70 con 0x557a99f29340 2024-03-18T00:55:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.729+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/3299494569 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd9e40ba110 con 0x557a99f29340 2024-03-18T00:55:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.729+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9e409b920 con 0x557a99f29340 2024-03-18T00:55:50.742 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.742+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/3299494569 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd9e40a40c0 con 0x557a99f29340 2024-03-18T00:55:50.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.743+0000 7fd9fbe15740 10 client.16821 fetch_fsmap learned FSMap version 27 2024-03-18T00:55:50.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.743+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a99fa55c0 con 0x557a99f29340 2024-03-18T00:55:50.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.748+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/3299494569 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9e409b920 con 0x557a99f29340 2024-03-18T00:55:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.749+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/3299494569 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd9e40861d0 con 0x557a99f29340 2024-03-18T00:55:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.750+0000 7fd9fbe15740 10 client.16821 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:55:50.749 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.750+0000 7fd9fbe15740 10 client.16821 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:55:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.750+0000 7fd9fbe15740 20 client.16821 populate_metadata read hostname 'smithi049' 2024-03-18T00:55:50.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.750+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557a99fa5ec0 0x557a99fa82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:55:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.751+0000 7fd9fbe15740 4 client.16821 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:55:50.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.751+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a99e962d0 con 0x557a99fa5ec0 2024-03-18T00:55:50.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.755+0000 7fd9f259c640 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557a99fa5ec0 0x557a99fa82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:55:50.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.761+0000 7fd9f259c640 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557a99fa5ec0 0x557a99fa82e0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc528d144e1165da server_cookie=5b9ed184c1210406 in_seq=0 out_seq=0 2024-03-18T00:55:50.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:50.765+0000 7fd9d6ff5640 10 client.16821 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:51.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:51 smithi049 ceph-mon[32130]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 929 MiB/s wr, 2.34k op/s 2024-03-18T00:55:51.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:51 smithi107 ceph-mon[36270]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 929 MiB/s wr, 2.34k op/s 2024-03-18T00:55:51.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:51 smithi106 ceph-mon[35899]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 929 MiB/s wr, 2.34k op/s 2024-03-18T00:55:53.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:53 smithi107 ceph-mon[36270]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 901 MiB/s wr, 2.07k op/s 2024-03-18T00:55:53.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:53 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:53.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:53 smithi106 ceph-mon[35899]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 901 MiB/s wr, 2.07k op/s 2024-03-18T00:55:53.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:53 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:53.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:53 smithi049 ceph-mon[32130]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 901 MiB/s wr, 2.07k op/s 2024-03-18T00:55:53.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:53 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:55:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:55 smithi107 ceph-mon[36270]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 890 MiB/s wr, 1.96k op/s 2024-03-18T00:55:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:55 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:55:55.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:55 smithi106 ceph-mon[35899]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 890 MiB/s wr, 1.96k op/s 2024-03-18T00:55:55.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:55 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:55:55.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:55 smithi049 ceph-mon[32130]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 890 MiB/s wr, 1.96k op/s 2024-03-18T00:55:55.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:55 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:55:56.488 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.487+0000 7fd9d6ff5640 1 -- 172.21.15.49:0/3299494569 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a99e962d0 con 0x557a99fa5ec0 2024-03-18T00:55:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.487+0000 7fd9d6ff5640 10 client.16821 handle_command_reply: tid=0 2024-03-18T00:55:56.489 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.487+0000 7fd9fbe15740 1 client.16821 shutdown 2024-03-18T00:55:56.490 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.489+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 msgr2=0x557a99f38510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.489+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f38510 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fd9e40097d0 tx=0x7fd9e409b4e0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.489+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 shutdown_connections 2024-03-18T00:55:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.489+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557a99fa5ec0 0x557a99fa82e0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.489+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557a99f29d30 0x557a99f38a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.489+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557a99f2eea0 0x557a99f39a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.489+0000 7fd9fbe15740 1 --2- 172.21.15.49:0/3299494569 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557a99f29340 0x557a99f38510 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.489+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 >> 172.21.15.49:0/3299494569 conn(0x557a99f248b0 msgr2=0x557a99f27b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:56.491 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.489+0000 7fd9d6ff5640 0 client.16821 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:55:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.490+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 shutdown_connections 2024-03-18T00:55:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.491+0000 7fd9fbe15740 1 -- 172.21.15.49:0/3299494569 wait complete. 2024-03-18T00:55:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.491+0000 7fd9fbe15740 20 client.16821 trim_cache size 0 max 16384 2024-03-18T00:55:56.492 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:55:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:55:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:55:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:55:56.493 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.492+0000 7fd9f9227640 1 -- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd9dc0a2920 msgr2=0x7fd9dc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.492+0000 7fd9f9227640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd9dc0a2920 0x7fd9dc0a4de0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd9f419b170 tx=0x7fd9e8006c40 comp rx=0 tx=0).stop 2024-03-18T00:55:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.493+0000 7fd9f9227640 1 -- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f4100900 msgr2=0x7fd9f4199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:55:56.494 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.493+0000 7fd9f9227640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f4100900 0x7fd9f4199c70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400e9f0 tx=0x7fd9e400edf0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.493+0000 7fd9f9227640 1 -- 172.21.15.49:0/2556627207 shutdown_connections 2024-03-18T00:55:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.494+0000 7fd9f9227640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd9dc0a2920 0x7fd9dc0a4de0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.494+0000 7fd9f9227640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd9f41012a0 0x7fd9f419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.494+0000 7fd9f9227640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd9f4101cb0 0x7fd9f41a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.494+0000 7fd9f9227640 1 --2- 172.21.15.49:0/2556627207 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd9f4100900 0x7fd9f4199c70 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:55:56.495 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.494+0000 7fd9f9227640 1 -- 172.21.15.49:0/2556627207 >> 172.21.15.49:0/2556627207 conn(0x7fd9f40fb430 msgr2=0x7fd9f4109520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:55:56.498 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.498+0000 7fd9f9227640 1 -- 172.21.15.49:0/2556627207 shutdown_connections 2024-03-18T00:55:56.499 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:55:56.498+0000 7fd9f9227640 1 -- 172.21.15.49:0/2556627207 wait complete. 2024-03-18T00:55:56.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:56 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:55:56.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:56 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:55:56.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:56 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:55:57.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:57 smithi106 ceph-mon[35899]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 923 MiB/s wr, 2.00k op/s 2024-03-18T00:55:57.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:57 smithi049 ceph-mon[32130]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 923 MiB/s wr, 2.00k op/s 2024-03-18T00:55:57.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:57 smithi107 ceph-mon[36270]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 923 MiB/s wr, 2.00k op/s 2024-03-18T00:55:59.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:55:59 smithi049 ceph-mon[32130]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 868 MiB/s wr, 1.70k op/s 2024-03-18T00:55:59.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:55:59 smithi107 ceph-mon[36270]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 868 MiB/s wr, 1.70k op/s 2024-03-18T00:55:59.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:55:59 smithi106 ceph-mon[35899]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 868 MiB/s wr, 1.70k op/s 2024-03-18T00:56:01.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:01 smithi049 ceph-mon[32130]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 951 MiB/s wr, 1.93k op/s 2024-03-18T00:56:01.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:01 smithi107 ceph-mon[36270]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 951 MiB/s wr, 1.93k op/s 2024-03-18T00:56:01.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:01 smithi106 ceph-mon[35899]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 951 MiB/s wr, 1.93k op/s 2024-03-18T00:56:03.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:03 smithi049 ceph-mon[32130]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 923 MiB/s wr, 1.83k op/s 2024-03-18T00:56:03.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:03 smithi107 ceph-mon[36270]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 923 MiB/s wr, 1.83k op/s 2024-03-18T00:56:03.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:03 smithi106 ceph-mon[35899]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 923 MiB/s wr, 1.83k op/s 2024-03-18T00:56:05.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:05 smithi049 ceph-mon[32130]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 886 MiB/s wr, 1.79k op/s 2024-03-18T00:56:05.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:05 smithi107 ceph-mon[36270]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 886 MiB/s wr, 1.79k op/s 2024-03-18T00:56:05.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:05 smithi106 ceph-mon[35899]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 886 MiB/s wr, 1.79k op/s 2024-03-18T00:56:07.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:07 smithi049 ceph-mon[32130]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 911 MiB/s wr, 1.83k op/s 2024-03-18T00:56:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:07 smithi107 ceph-mon[36270]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 911 MiB/s wr, 1.83k op/s 2024-03-18T00:56:07.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:07 smithi106 ceph-mon[35899]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 911 MiB/s wr, 1.83k op/s 2024-03-18T00:56:09.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:09 smithi049 ceph-mon[32130]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 856 MiB/s wr, 1.62k op/s 2024-03-18T00:56:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:09 smithi107 ceph-mon[36270]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 856 MiB/s wr, 1.62k op/s 2024-03-18T00:56:09.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:09 smithi106 ceph-mon[35899]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 856 MiB/s wr, 1.62k op/s 2024-03-18T00:56:09.957 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.957+0000 7f9345d9b640 1 -- 172.21.15.49:0/1883104735 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340034bf0 con 0x7f934810c220 2024-03-18T00:56:09.957 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.958+0000 7f934d014640 1 -- 172.21.15.49:0/1883104735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810c220 msgr2=0x7f934810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:09.957 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.958+0000 7f934d014640 1 --2- 172.21.15.49:0/1883104735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810c220 0x7f934810e610 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9340009ca0 tx=0x7f934002e350 comp rx=0 tx=0).stop 2024-03-18T00:56:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.958+0000 7f934d014640 1 -- 172.21.15.49:0/1883104735 shutdown_connections 2024-03-18T00:56:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.958+0000 7f934d014640 1 --2- 172.21.15.49:0/1883104735 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f934810eba0 0x7f9348111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.958+0000 7f934d014640 1 --2- 172.21.15.49:0/1883104735 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810c220 0x7f934810e610 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.958+0000 7f934d014640 1 --2- 172.21.15.49:0/1883104735 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93481098f0 0x7f934810bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.958 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.958+0000 7f934d014640 1 -- 172.21.15.49:0/1883104735 >> 172.21.15.49:0/1883104735 conn(0x7f934806e820 msgr2=0x7f9348070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.958+0000 7f934d014640 1 -- 172.21.15.49:0/1883104735 shutdown_connections 2024-03-18T00:56:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.958+0000 7f934d014640 1 -- 172.21.15.49:0/1883104735 wait complete. 2024-03-18T00:56:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.959+0000 7f934d014640 1 Processor -- start 2024-03-18T00:56:09.959 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.959+0000 7f934d014640 1 -- start start 2024-03-18T00:56:09.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.960+0000 7f934d014640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93481098f0 0x7f93481a2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.960+0000 7f934d014640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f934810c220 0x7f93481a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.960+0000 7f934d014640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810eba0 0x7f93481a9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.960+0000 7f934d014640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9348113be0 con 0x7f934810eba0 2024-03-18T00:56:09.960 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.960+0000 7f934d014640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9348113a60 con 0x7f934810c220 2024-03-18T00:56:09.961 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.960+0000 7f934d014640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9348113d60 con 0x7f93481098f0 2024-03-18T00:56:09.961 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.961+0000 7f934759e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810eba0 0x7f93481a9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:09.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.962+0000 7f934759e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810eba0 0x7f93481a9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46894/0 (socket says 172.21.15.49:46894) 2024-03-18T00:56:09.962 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.962+0000 7f934759e640 1 -- 172.21.15.49:0/2563662718 learned_addr learned my addr 172.21.15.49:0/2563662718 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:09.963 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.962+0000 7f934659c640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f934810c220 0x7f93481a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:09.963 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.962+0000 7f9346d9d640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93481098f0 0x7f93481a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:09.963 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.963+0000 7f934759e640 1 -- 172.21.15.49:0/2563662718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93481098f0 msgr2=0x7f93481a2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:09.963 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.963+0000 7f934759e640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93481098f0 0x7f93481a2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.963 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.963+0000 7f934759e640 1 -- 172.21.15.49:0/2563662718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f934810c220 msgr2=0x7f93481a2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:09.963 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.963+0000 7f934759e640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f934810c220 0x7f93481a2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.963 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.963+0000 7f934759e640 1 -- 172.21.15.49:0/2563662718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93481aa2b0 con 0x7f934810eba0 2024-03-18T00:56:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.963+0000 7f934759e640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810eba0 0x7f93481a9c40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f933c00b3f0 tx=0x7f933c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.964+0000 7f932ffff640 1 -- 172.21.15.49:0/2563662718 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c015020 con 0x7f934810eba0 2024-03-18T00:56:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.964+0000 7f932ffff640 1 -- 172.21.15.49:0/2563662718 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f933c011d10 con 0x7f934810eba0 2024-03-18T00:56:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.964+0000 7f934d014640 1 -- 172.21.15.49:0/2563662718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93481aa5a0 con 0x7f934810eba0 2024-03-18T00:56:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.964+0000 7f932ffff640 1 -- 172.21.15.49:0/2563662718 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c00f2e0 con 0x7f934810eba0 2024-03-18T00:56:09.964 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.964+0000 7f934d014640 1 -- 172.21.15.49:0/2563662718 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93481aab70 con 0x7f934810eba0 2024-03-18T00:56:09.970 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.970+0000 7f932ffff640 1 -- 172.21.15.49:0/2563662718 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f933c018020 con 0x7f934810eba0 2024-03-18T00:56:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.970+0000 7f934fbfd740 1 Processor -- start 2024-03-18T00:56:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.970+0000 7f934fbfd740 1 -- start start 2024-03-18T00:56:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.970+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f992b50 0x55ee1f992f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.970+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f998040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.970+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f9986b0 0x55ee1f99ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.970+0000 7f934fbfd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ee1f8efa40 con 0x55ee1f993540 2024-03-18T00:56:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.970+0000 7f934fbfd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ee1f898920 con 0x55ee1f9986b0 2024-03-18T00:56:09.971 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.970+0000 7f934fbfd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ee1f8ebae0 con 0x55ee1f992b50 2024-03-18T00:56:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.971+0000 7f9346d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f998040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.971+0000 7f9346d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f998040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46902/0 (socket says 172.21.15.49:46902) 2024-03-18T00:56:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.971+0000 7f9346d9d640 1 -- 172.21.15.49:0/3257663103 learned_addr learned my addr 172.21.15.49:0/3257663103 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.972+0000 7f9346d9d640 1 -- 172.21.15.49:0/3257663103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f992b50 msgr2=0x55ee1f992f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.972+0000 7f9346d9d640 1 --2- 172.21.15.49:0/3257663103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f992b50 0x55ee1f992f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.972+0000 7f9346d9d640 1 -- 172.21.15.49:0/3257663103 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f9986b0 msgr2=0x55ee1f99ab40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:09.972 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.972+0000 7f9346d9d640 1 --2- 172.21.15.49:0/3257663103 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f9986b0 0x55ee1f99ab40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.972+0000 7f9346d9d640 1 -- 172.21.15.49:0/3257663103 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee1f8b0460 con 0x55ee1f993540 2024-03-18T00:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.972+0000 7f932ffff640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93300a29f0 0x7f93300a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.972+0000 7f932ffff640 1 -- 172.21.15.49:0/2563662718 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f933c0ce420 con 0x7f934810eba0 2024-03-18T00:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.972+0000 7f934659c640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93300a29f0 0x7f93300a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:09.973 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.973+0000 7f9346d9d640 1 --2- 172.21.15.49:0/3257663103 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f998040 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f933800b820 tx=0x7f933800bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ac831938f6623a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.973+0000 7f932d7fa640 1 -- 172.21.15.49:0/3257663103 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933800d950 con 0x55ee1f993540 2024-03-18T00:56:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.973+0000 7f932d7fa640 1 -- 172.21.15.49:0/3257663103 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9338019070 con 0x55ee1f993540 2024-03-18T00:56:09.974 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.973+0000 7f932d7fa640 1 -- 172.21.15.49:0/3257663103 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338014e60 con 0x55ee1f993540 2024-03-18T00:56:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.976+0000 7f934659c640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93300a29f0 0x7f93300a4eb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9340000f80 tx=0x7f9340034ef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:09.975 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.976+0000 7f934fbfd740 1 -- 172.21.15.49:0/3257663103 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 msgr2=0x55ee1f998040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.976+0000 7f934fbfd740 1 --2- 172.21.15.49:0/3257663103 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f998040 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f933800b820 tx=0x7f933800bce0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.976+0000 7f934fbfd740 1 -- 172.21.15.49:0/3257663103 shutdown_connections 2024-03-18T00:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.976+0000 7f934fbfd740 1 --2- 172.21.15.49:0/3257663103 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f9986b0 0x55ee1f99ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.976+0000 7f934fbfd740 1 --2- 172.21.15.49:0/3257663103 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f998040 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.976+0000 7f934fbfd740 1 --2- 172.21.15.49:0/3257663103 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f992b50 0x55ee1f992f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.976 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.976+0000 7f934fbfd740 1 -- 172.21.15.49:0/3257663103 >> 172.21.15.49:0/3257663103 conn(0x55ee1f98e0c0 msgr2=0x55ee1f98e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:09.977 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.977+0000 7f934fbfd740 1 -- 172.21.15.49:0/3257663103 shutdown_connections 2024-03-18T00:56:09.977 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.977+0000 7f934fbfd740 1 -- 172.21.15.49:0/3257663103 wait complete. 2024-03-18T00:56:09.978 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.978+0000 7f934fbfd740 1 -- start start 2024-03-18T00:56:09.979 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.978+0000 7f934fbfd740 1 Processor -- start 2024-03-18T00:56:09.980 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.979+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f992b50 0x55ee1f99cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.979+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f993540 0x55ee1f99d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.979+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.979+0000 7f934fbfd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55ee1f99b660 con 0x55ee1f9986b0 2024-03-18T00:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.979+0000 7f934fbfd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55ee1f8abd70 con 0x55ee1f993540 2024-03-18T00:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.979+0000 7f934fbfd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55ee1f99b7e0 con 0x55ee1f992b50 2024-03-18T00:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.979+0000 7f934fbfd740 10 client.0 _ll_register_callbacks cb 0x55ee1f603a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.981+0000 7f934759e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.981+0000 7f934759e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46906/0 (socket says 172.21.15.49:46906) 2024-03-18T00:56:09.981 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.981+0000 7f934759e640 1 -- 172.21.15.49:0/1446965729 learned_addr learned my addr 172.21.15.49:0/1446965729 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.982+0000 7f934659c640 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f993540 0x55ee1f99d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.983+0000 7f9346d9d640 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f992b50 0x55ee1f99cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.983+0000 7f934759e640 1 -- 172.21.15.49:0/1446965729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f992b50 msgr2=0x55ee1f99cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.983+0000 7f934759e640 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f992b50 0x55ee1f99cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.983+0000 7f934759e640 1 -- 172.21.15.49:0/1446965729 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f993540 msgr2=0x55ee1f99d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.983+0000 7f934759e640 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f993540 0x55ee1f99d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.984+0000 7f934759e640 1 -- 172.21.15.49:0/1446965729 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee1f604210 con 0x55ee1f9986b0 2024-03-18T00:56:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.984+0000 7f934759e640 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4440 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f933c0a2fb0 tx=0x7f933c09a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5873da685a41fd09 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.984+0000 7f932aff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:09.985 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.985+0000 7f932aff5640 1 -- 172.21.15.49:0/1446965729 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c09ee90 con 0x55ee1f9986b0 2024-03-18T00:56:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.985+0000 7f934fbfd740 1 -- 172.21.15.49:0/1446965729 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9338019070 con 0x55ee1f9986b0 2024-03-18T00:56:09.986 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.986+0000 7f932aff5640 1 -- 172.21.15.49:0/1446965729 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f933c0104f0 con 0x55ee1f9986b0 2024-03-18T00:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.988+0000 7f932aff5640 1 -- 172.21.15.49:0/1446965729 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f933c0a34e0 con 0x55ee1f9986b0 2024-03-18T00:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.988+0000 7f934fbfd740 10 client.16833 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:09.988 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.988+0000 7f934fbfd740 1 -- 172.21.15.49:0/1446965729 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee1f6d0620 con 0x55ee1f9986b0 2024-03-18T00:56:09.990 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.990+0000 7f932aff5640 1 -- 172.21.15.49:0/1446965729 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9338019070 con 0x55ee1f9986b0 2024-03-18T00:56:09.991 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.991+0000 7f932aff5640 1 -- 172.21.15.49:0/1446965729 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f933c096a90 con 0x55ee1f9986b0 2024-03-18T00:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.993+0000 7f934fbfd740 10 client.16833 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.993+0000 7f934fbfd740 10 client.16833 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.993+0000 7f934fbfd740 20 client.16833 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.993+0000 7f934fbfd740 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ee1f9a90b0 0x55ee1f9ab570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:09.993 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.993+0000 7f934fbfd740 4 client.16833 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:09.994 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.993+0000 7f934fbfd740 1 -- 172.21.15.49:0/1446965729 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee1f8fcf80 con 0x55ee1f9a90b0 2024-03-18T00:56:09.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.995+0000 7f9346d9d640 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ee1f9a90b0 0x55ee1f9ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.998+0000 7f9346d9d640 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ee1f9a90b0 0x55ee1f9ab570 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d759183cd7454ae server_cookie=8b21e558902aef57 in_seq=0 out_seq=0 2024-03-18T00:56:09.998 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:09.998+0000 7f932aff5640 10 client.16833 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.005+0000 7f932aff5640 1 -- 172.21.15.49:0/1446965729 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee1f8fcf80 con 0x55ee1f9a90b0 2024-03-18T00:56:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.005+0000 7f932aff5640 10 client.16833 handle_command_reply: tid=0 2024-03-18T00:56:10.004 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.005+0000 7f934fbfd740 1 client.16833 shutdown 2024-03-18T00:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f934fbfd740 1 -- 172.21.15.49:0/1446965729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 msgr2=0x55ee1f9a4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f934fbfd740 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4440 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f933c0a2fb0 tx=0x7f933c09a6a0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f934fbfd740 1 -- 172.21.15.49:0/1446965729 shutdown_connections 2024-03-18T00:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f934fbfd740 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ee1f9a90b0 0x55ee1f9ab570 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f934fbfd740 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f993540 0x55ee1f99d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f934fbfd740 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4440 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f934fbfd740 1 --2- 172.21.15.49:0/1446965729 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f992b50 0x55ee1f99cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f934fbfd740 1 -- 172.21.15.49:0/1446965729 >> 172.21.15.49:0/1446965729 conn(0x55ee1f98e0c0 msgr2=0x55ee1f984bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:10.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f932aff5640 0 client.16833 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.006+0000 7f934fbfd740 1 -- 172.21.15.49:0/1446965729 shutdown_connections 2024-03-18T00:56:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.007+0000 7f934fbfd740 1 -- 172.21.15.49:0/1446965729 wait complete. 2024-03-18T00:56:10.007 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.007+0000 7f934fbfd740 20 client.16833 trim_cache size 0 max 16384 2024-03-18T00:56:10.038 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.038+0000 7f934fbfd740 1 Processor -- start 2024-03-18T00:56:10.038 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.039+0000 7f934fbfd740 1 -- start start 2024-03-18T00:56:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.039+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f992b50 0x55ee1f99cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.039+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f993540 0x55ee1f99d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.039+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.039+0000 7f934fbfd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f93481af5e0 con 0x55ee1f9986b0 2024-03-18T00:56:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.039+0000 7f934fbfd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f93481b1120 con 0x55ee1f992b50 2024-03-18T00:56:10.039 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.039+0000 7f934fbfd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f93481afcb0 con 0x55ee1f993540 2024-03-18T00:56:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.039+0000 7f934759e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.040+0000 7f934759e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46922/0 (socket says 172.21.15.49:46922) 2024-03-18T00:56:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.040+0000 7f934759e640 1 -- 172.21.15.49:0/762368489 learned_addr learned my addr 172.21.15.49:0/762368489 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:10.040 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.040+0000 7f9346d9d640 1 --2- 172.21.15.49:0/762368489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f992b50 0x55ee1f99cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.040+0000 7f934659c640 1 --2- 172.21.15.49:0/762368489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f993540 0x55ee1f99d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.040+0000 7f934759e640 1 -- 172.21.15.49:0/762368489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f993540 msgr2=0x55ee1f99d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.040+0000 7f934759e640 1 --2- 172.21.15.49:0/762368489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f993540 0x55ee1f99d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.040+0000 7f934759e640 1 -- 172.21.15.49:0/762368489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f992b50 msgr2=0x55ee1f99cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.040+0000 7f934759e640 1 --2- 172.21.15.49:0/762368489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f992b50 0x55ee1f99cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.040+0000 7f934759e640 1 -- 172.21.15.49:0/762368489 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee1f993dd0 con 0x55ee1f9986b0 2024-03-18T00:56:10.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.041+0000 7f934759e640 1 --2- 172.21.15.49:0/762368489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4400 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f933c096eb0 tx=0x7f933c011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5e68986ed0a248 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.041+0000 7f932cff9640 1 -- 172.21.15.49:0/762368489 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c084080 con 0x55ee1f9986b0 2024-03-18T00:56:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.041+0000 7f932cff9640 1 -- 172.21.15.49:0/762368489 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f933c09ee90 con 0x55ee1f9986b0 2024-03-18T00:56:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.041+0000 7f932cff9640 1 -- 172.21.15.49:0/762368489 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c099c50 con 0x55ee1f9986b0 2024-03-18T00:56:10.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.043+0000 7f934fbfd740 1 -- 172.21.15.49:0/762368489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 msgr2=0x55ee1f9a4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:10.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.043+0000 7f934fbfd740 1 --2- 172.21.15.49:0/762368489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4400 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f933c096eb0 tx=0x7f933c011f70 comp rx=0 tx=0).stop 2024-03-18T00:56:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.043+0000 7f934fbfd740 1 -- 172.21.15.49:0/762368489 shutdown_connections 2024-03-18T00:56:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.043+0000 7f934fbfd740 1 --2- 172.21.15.49:0/762368489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f9986b0 0x55ee1f9a4400 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.043+0000 7f934fbfd740 1 --2- 172.21.15.49:0/762368489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f993540 0x55ee1f99d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.043+0000 7f934fbfd740 1 --2- 172.21.15.49:0/762368489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f992b50 0x55ee1f99cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.043+0000 7f934fbfd740 1 -- 172.21.15.49:0/762368489 >> 172.21.15.49:0/762368489 conn(0x55ee1f98e0c0 msgr2=0x55ee1f980a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.044+0000 7f934fbfd740 1 -- 172.21.15.49:0/762368489 shutdown_connections 2024-03-18T00:56:10.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.044+0000 7f934fbfd740 1 -- 172.21.15.49:0/762368489 wait complete. 2024-03-18T00:56:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.044+0000 7f934fbfd740 1 -- start start 2024-03-18T00:56:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.044+0000 7f934fbfd740 1 Processor -- start 2024-03-18T00:56:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.045+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f992b50 0x55ee1f9a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.045+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f9a1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.045+0000 7f934fbfd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f9986b0 0x55ee1fa0d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.045+0000 7f934fbfd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f93481b0a50 con 0x55ee1f993540 2024-03-18T00:56:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.045+0000 7f934fbfd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f93481b0380 con 0x55ee1f992b50 2024-03-18T00:56:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.045+0000 7f934fbfd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f93481ad5c0 con 0x55ee1f9986b0 2024-03-18T00:56:10.045 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.045+0000 7f934fbfd740 10 client.0 _ll_register_callbacks cb 0x7f933c011ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:10.046 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.046+0000 7f934659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f9a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:10.046 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.046+0000 7f934659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f9a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46932/0 (socket says 172.21.15.49:46932) 2024-03-18T00:56:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.046+0000 7f934659c640 1 -- 172.21.15.49:0/490815369 learned_addr learned my addr 172.21.15.49:0/490815369 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.046+0000 7f934759e640 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f9986b0 0x55ee1fa0d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.047+0000 7f934659c640 1 -- 172.21.15.49:0/490815369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f9986b0 msgr2=0x55ee1fa0d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.047+0000 7f934659c640 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f9986b0 0x55ee1fa0d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.047+0000 7f934659c640 1 -- 172.21.15.49:0/490815369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f992b50 msgr2=0x55ee1f9a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.047+0000 7f934659c640 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f992b50 0x55ee1f9a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:10.047 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.047+0000 7f934659c640 1 -- 172.21.15.49:0/490815369 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee1f9a1dd0 con 0x55ee1f993540 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.047+0000 7f934659c640 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f9a1730 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9340009bf0 tx=0x7f93400095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64ce6b4ac493d2ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.047+0000 7f932aff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.047+0000 7f932aff5640 1 -- 172.21.15.49:0/490815369 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340051060 con 0x55ee1f993540 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.048+0000 7f934fbfd740 1 -- 172.21.15.49:0/490815369 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f933c09ee90 con 0x55ee1f993540 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.048+0000 7f932aff5640 1 -- 172.21.15.49:0/490815369 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9340004040 con 0x55ee1f993540 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.048+0000 7f932aff5640 1 -- 172.21.15.49:0/490815369 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f934003b040 con 0x55ee1f993540 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.048+0000 7f934fbfd740 10 client.16839 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.048+0000 7f934fbfd740 1 -- 172.21.15.49:0/490815369 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ee1f986510 con 0x55ee1f993540 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.048+0000 7f932aff5640 1 -- 172.21.15.49:0/490815369 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f933c09ee90 con 0x55ee1f993540 2024-03-18T00:56:10.049 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.049+0000 7f932aff5640 1 -- 172.21.15.49:0/490815369 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9340041830 con 0x55ee1f993540 2024-03-18T00:56:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.049+0000 7f934fbfd740 10 client.16839 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.049+0000 7f934fbfd740 10 client.16839 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.049+0000 7f934fbfd740 20 client.16839 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.049+0000 7f934fbfd740 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ee1f986ea0 0x55ee1f9892c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.049+0000 7f934fbfd740 4 client.16839 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:56:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.049+0000 7f934fbfd740 1 -- 172.21.15.49:0/490815369 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee1f8f72d0 con 0x55ee1f986ea0 2024-03-18T00:56:10.050 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.050+0000 7f9346d9d640 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ee1f986ea0 0x55ee1f9892c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:10.052 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.052+0000 7f932aff5640 10 client.16839 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:10.052 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:10.052+0000 7f9346d9d640 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ee1f986ea0 0x55ee1f9892c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8e4ce7cc7d1161a server_cookie=c2e748fcb137a6e6 in_seq=0 out_seq=0 2024-03-18T00:56:11.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:11 smithi049 ceph-mon[32130]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 942 MiB/s wr, 1.55k op/s 2024-03-18T00:56:11.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:11 smithi107 ceph-mon[36270]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 942 MiB/s wr, 1.55k op/s 2024-03-18T00:56:11.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:11 smithi106 ceph-mon[35899]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 942 MiB/s wr, 1.55k op/s 2024-03-18T00:56:13.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:13 smithi049 ceph-mon[32130]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 921 MiB/s wr, 1.37k op/s 2024-03-18T00:56:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:13 smithi107 ceph-mon[36270]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 921 MiB/s wr, 1.37k op/s 2024-03-18T00:56:13.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:13 smithi106 ceph-mon[35899]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 921 MiB/s wr, 1.37k op/s 2024-03-18T00:56:15.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:15 smithi049 ceph-mon[32130]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 890 MiB/s wr, 1.56k op/s 2024-03-18T00:56:15.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:15 smithi107 ceph-mon[36270]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 890 MiB/s wr, 1.56k op/s 2024-03-18T00:56:15.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:15 smithi106 ceph-mon[35899]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 890 MiB/s wr, 1.56k op/s 2024-03-18T00:56:16.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:56:16.978 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:56:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.389+0000 7f932aff5640 1 -- 172.21.15.49:0/490815369 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee1f8f72d0 con 0x55ee1f986ea0 2024-03-18T00:56:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.389+0000 7f932aff5640 10 client.16839 handle_command_reply: tid=0 2024-03-18T00:56:17.389 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.389+0000 7f934fbfd740 1 client.16839 shutdown 2024-03-18T00:56:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f934fbfd740 1 -- 172.21.15.49:0/490815369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 msgr2=0x55ee1f9a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f934fbfd740 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f9a1730 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9340009bf0 tx=0x7f93400095c0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f934fbfd740 1 -- 172.21.15.49:0/490815369 shutdown_connections 2024-03-18T00:56:17.390 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f934fbfd740 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55ee1f986ea0 0x55ee1f9892c0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f934fbfd740 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55ee1f9986b0 0x55ee1fa0d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f934fbfd740 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55ee1f993540 0x55ee1f9a1730 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f934fbfd740 1 --2- 172.21.15.49:0/490815369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55ee1f992b50 0x55ee1f9a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f934fbfd740 1 -- 172.21.15.49:0/490815369 >> 172.21.15.49:0/490815369 conn(0x55ee1f98e0c0 msgr2=0x55ee1f99ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f932aff5640 0 client.16839 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.390+0000 7f934fbfd740 1 -- 172.21.15.49:0/490815369 shutdown_connections 2024-03-18T00:56:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.391+0000 7f934fbfd740 1 -- 172.21.15.49:0/490815369 wait complete. 2024-03-18T00:56:17.391 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.391+0000 7f934fbfd740 20 client.16839 trim_cache size 0 max 16384 2024-03-18T00:56:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:56:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:56:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:56:17.392 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:56:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.394+0000 7f934d014640 1 -- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93300a29f0 msgr2=0x7f93300a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.394 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.394+0000 7f934d014640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93300a29f0 0x7f93300a4eb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9340000f80 tx=0x7f9340034ef0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.394+0000 7f934d014640 1 -- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810eba0 msgr2=0x7f93481a9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.394+0000 7f934d014640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810eba0 0x7f93481a9c40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f933c00b3f0 tx=0x7f933c00b8b0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.395+0000 7f934d014640 1 -- 172.21.15.49:0/2563662718 shutdown_connections 2024-03-18T00:56:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.395+0000 7f934d014640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f93300a29f0 0x7f93300a4eb0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.395 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.395+0000 7f934d014640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f934810c220 0x7f93481a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.395+0000 7f934d014640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f934810eba0 0x7f93481a9c40 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.395+0000 7f934d014640 1 --2- 172.21.15.49:0/2563662718 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f93481098f0 0x7f93481a2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.395+0000 7f934d014640 1 -- 172.21.15.49:0/2563662718 >> 172.21.15.49:0/2563662718 conn(0x7f934806e820 msgr2=0x7f9348109cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.395+0000 7f934d014640 1 -- 172.21.15.49:0/2563662718 shutdown_connections 2024-03-18T00:56:17.396 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:17.396+0000 7f934d014640 1 -- 172.21.15.49:0/2563662718 wait complete. 2024-03-18T00:56:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.460+0000 7f52e2227640 1 -- 172.21.15.49:0/1506137298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 msgr2=0x7f52dc101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.460+0000 7f52e2227640 1 --2- 172.21.15.49:0/1506137298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 0x7f52dc101d50 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009900 tx=0x7f52d002bc30 comp rx=0 tx=0).stop 2024-03-18T00:56:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.460+0000 7f52e2227640 1 -- 172.21.15.49:0/1506137298 shutdown_connections 2024-03-18T00:56:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.460+0000 7f52e2227640 1 --2- 172.21.15.49:0/1506137298 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc102290 0x7f52dc108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.460+0000 7f52e2227640 1 --2- 172.21.15.49:0/1506137298 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 0x7f52dc101d50 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.460+0000 7f52e2227640 1 --2- 172.21.15.49:0/1506137298 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52dc1006e0 0x7f52dc100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.460+0000 7f52e2227640 1 -- 172.21.15.49:0/1506137298 >> 172.21.15.49:0/1506137298 conn(0x7f52dc0fbc70 msgr2=0x7f52dc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:17.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.460+0000 7f52e2227640 1 -- 172.21.15.49:0/1506137298 shutdown_connections 2024-03-18T00:56:17.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.463+0000 7f52e2227640 1 -- 172.21.15.49:0/1506137298 wait complete. 2024-03-18T00:56:17.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.464+0000 7f52e2227640 1 Processor -- start 2024-03-18T00:56:17.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.464+0000 7f52e2227640 1 -- start start 2024-03-18T00:56:17.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52e2227640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc1006e0 0x7f52dc197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52e2227640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 0x7f52dc1981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52e2227640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52dc102290 0x7f52dc19f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52daffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 0x7f52dc1981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52daffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 0x7f52dc1981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46948/0 (socket says 172.21.15.49:46948) 2024-03-18T00:56:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52daffd640 1 -- 172.21.15.49:0/2272962853 learned_addr learned my addr 172.21.15.49:0/2272962853 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc10c4a0 con 0x7f52dc1018d0 2024-03-18T00:56:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc10c320 con 0x7f52dc102290 2024-03-18T00:56:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc10c620 con 0x7f52dc1006e0 2024-03-18T00:56:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52db7fe640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc1006e0 0x7f52dc197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.465+0000 7f52dbfff640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52dc102290 0x7f52dc19f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.466+0000 7f52dbfff640 1 -- 172.21.15.49:0/2272962853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc1006e0 msgr2=0x7f52dc197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.466+0000 7f52dbfff640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc1006e0 0x7f52dc197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.466+0000 7f52dbfff640 1 -- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 msgr2=0x7f52dc1981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.466+0000 7f52dbfff640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 0x7f52dc1981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.466+0000 7f52dbfff640 1 -- 172.21.15.49:0/2272962853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52dc102d40 con 0x7f52dc102290 2024-03-18T00:56:17.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.467+0000 7f52dbfff640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52dc102290 0x7f52dc19f1c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f52d400be20 tx=0x7f52d4009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:17.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.468+0000 7f52d8ff9640 1 -- 172.21.15.49:0/2272962853 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4019070 con 0x7f52dc102290 2024-03-18T00:56:17.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.469+0000 7f52d8ff9640 1 -- 172.21.15.49:0/2272962853 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52d400cd70 con 0x7f52dc102290 2024-03-18T00:56:17.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.470+0000 7f52d8ff9640 1 -- 172.21.15.49:0/2272962853 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d4012880 con 0x7f52dc102290 2024-03-18T00:56:17.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.478+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52dc103030 con 0x7f52dc102290 2024-03-18T00:56:17.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.479+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52dc103500 con 0x7f52dc102290 2024-03-18T00:56:17.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.480+0000 7f52daffd640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 0x7f52dc1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:56:17.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.488+0000 7f52e4e15740 1 Processor -- start 2024-03-18T00:56:17.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.491+0000 7f52e4e15740 1 -- start start 2024-03-18T00:56:17.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.492+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a82590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.492+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a87660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.492+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a87cd0 0x558d69a8a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.493+0000 7f52e4e15740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558d699e7500 con 0x558d69a82b60 2024-03-18T00:56:17.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.493+0000 7f52e4e15740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558d6998fbe0 con 0x558d69a82170 2024-03-18T00:56:17.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.493+0000 7f52daffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a82590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.493 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.493+0000 7f52e4e15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558d699e35a0 con 0x558d69a87cd0 2024-03-18T00:56:17.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.493+0000 7f52d8ff9640 1 -- 172.21.15.49:0/2272962853 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f52d4012a20 con 0x7f52dc102290 2024-03-18T00:56:17.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.493+0000 7f52daffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a82590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:38190/0 (socket says 172.21.15.49:38190) 2024-03-18T00:56:17.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.493+0000 7f52daffd640 1 -- 172.21.15.49:0/265249029 learned_addr learned my addr 172.21.15.49:0/265249029 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.502+0000 7f52d8ff9640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52c40a28b0 0x7f52c40a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.503+0000 7f52d8ff9640 1 -- 172.21.15.49:0/2272962853 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f52d40ce120 con 0x7f52dc102290 2024-03-18T00:56:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.504+0000 7f52dbfff640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52c40a28b0 0x7f52c40a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.506 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.505+0000 7f52daffd640 1 --2- 172.21.15.49:0/265249029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a87cd0 0x558d69a8a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.508+0000 7f52daffd640 1 -- 172.21.15.49:0/265249029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a87cd0 msgr2=0x558d69a8a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.508+0000 7f52daffd640 1 --2- 172.21.15.49:0/265249029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a87cd0 0x558d69a8a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.508+0000 7f52daffd640 1 -- 172.21.15.49:0/265249029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 msgr2=0x558d69a87660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.508+0000 7f52daffd640 1 --2- 172.21.15.49:0/265249029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a87660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.508+0000 7f52daffd640 1 -- 172.21.15.49:0/265249029 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d699a34b0 con 0x558d69a82170 2024-03-18T00:56:17.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.518+0000 7f52dbfff640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52c40a28b0 0x7f52c40a4d70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f52d4097380 tx=0x7f52d400e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:17.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.519+0000 7f52daffd640 1 --2- 172.21.15.49:0/265249029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a82590 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52d002c780 tx=0x7f52d0009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9948c06a89be12b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:17.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.525+0000 7f52c1ffb640 1 -- 172.21.15.49:0/265249029 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0048070 con 0x558d69a82170 2024-03-18T00:56:17.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.525+0000 7f52c1ffb640 1 -- 172.21.15.49:0/265249029 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52d00054e0 con 0x558d69a82170 2024-03-18T00:56:17.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.525+0000 7f52c1ffb640 1 -- 172.21.15.49:0/265249029 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0004bf0 con 0x558d69a82170 2024-03-18T00:56:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.529+0000 7f52e4e15740 1 -- 172.21.15.49:0/265249029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 msgr2=0x558d69a82590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.530 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.529+0000 7f52e4e15740 1 --2- 172.21.15.49:0/265249029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a82590 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f52d002c780 tx=0x7f52d0009610 comp rx=0 tx=0).stop 2024-03-18T00:56:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.530+0000 7f52e4e15740 1 -- 172.21.15.49:0/265249029 shutdown_connections 2024-03-18T00:56:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.530+0000 7f52e4e15740 1 --2- 172.21.15.49:0/265249029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a87cd0 0x558d69a8a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.530+0000 7f52e4e15740 1 --2- 172.21.15.49:0/265249029 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a87660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.530+0000 7f52e4e15740 1 --2- 172.21.15.49:0/265249029 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a82590 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.530+0000 7f52e4e15740 1 -- 172.21.15.49:0/265249029 >> 172.21.15.49:0/265249029 conn(0x558d69a7d6e0 msgr2=0x558d69a7dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:17.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.537+0000 7f52e4e15740 1 -- 172.21.15.49:0/265249029 shutdown_connections 2024-03-18T00:56:17.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.539+0000 7f52e4e15740 1 -- 172.21.15.49:0/265249029 wait complete. 2024-03-18T00:56:17.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.540+0000 7f52e4e15740 1 -- start start 2024-03-18T00:56:17.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.540+0000 7f52e4e15740 1 Processor -- start 2024-03-18T00:56:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.543+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a82170 0x558d69a8c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.543+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.543+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a87cd0 0x558d69a93ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.543+0000 7f52e4e15740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558d69a8ac20 con 0x558d69a82b60 2024-03-18T00:56:17.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.543+0000 7f52e4e15740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558d699a3160 con 0x558d69a87cd0 2024-03-18T00:56:17.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.543+0000 7f52e4e15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558d69a8ada0 con 0x558d69a82170 2024-03-18T00:56:17.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.544+0000 7f52e4e15740 10 client.0 _ll_register_callbacks cb 0x558d69987150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:17.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.544+0000 7f52db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.544+0000 7f52db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:46976/0 (socket says 172.21.15.49:46976) 2024-03-18T00:56:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.544+0000 7f52db7fe640 1 -- 172.21.15.49:0/4024004525 learned_addr learned my addr 172.21.15.49:0/4024004525 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.545+0000 7f52db7fe640 1 -- 172.21.15.49:0/4024004525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a82170 msgr2=0x558d69a8c720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:56:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.545+0000 7f52db7fe640 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a82170 0x558d69a8c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.545+0000 7f52db7fe640 1 -- 172.21.15.49:0/4024004525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a87cd0 msgr2=0x558d69a93ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:56:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.545+0000 7f52db7fe640 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a87cd0 0x558d69a93ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.545+0000 7f52db7fe640 1 -- 172.21.15.49:0/4024004525 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d697c57d0 con 0x558d69a82b60 2024-03-18T00:56:17.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.548+0000 7f52db7fe640 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f52dc101730 tx=0x7f52cc00e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57543a185814b4c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:17.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.549+0000 7f52baffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:17.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.549+0000 7f52baffd640 1 -- 172.21.15.49:0/4024004525 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52cc017070 con 0x558d69a82b60 2024-03-18T00:56:17.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.550+0000 7f52e4e15740 1 -- 172.21.15.49:0/4024004525 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52d00054e0 con 0x558d69a82b60 2024-03-18T00:56:17.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.550+0000 7f52baffd640 1 -- 172.21.15.49:0/4024004525 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f52cc013680 con 0x558d69a82b60 2024-03-18T00:56:17.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.556+0000 7f52baffd640 1 -- 172.21.15.49:0/4024004525 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f52cc023050 con 0x558d69a82b60 2024-03-18T00:56:17.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.558+0000 7f52e4e15740 10 client.16845 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:17.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.558+0000 7f52e4e15740 1 -- 172.21.15.49:0/4024004525 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558d696fb3f0 con 0x558d69a82b60 2024-03-18T00:56:17.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.559+0000 7f52baffd640 1 -- 172.21.15.49:0/4024004525 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52d00054e0 con 0x558d69a82b60 2024-03-18T00:56:17.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.560+0000 7f52baffd640 1 -- 172.21.15.49:0/4024004525 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f52cc012a50 con 0x558d69a82b60 2024-03-18T00:56:17.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.563+0000 7f52e4e15740 10 client.16845 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:17.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.563+0000 7f52e4e15740 10 client.16845 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:17.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.563+0000 7f52e4e15740 20 client.16845 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:17.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.563+0000 7f52e4e15740 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558d69a98240 0x558d69a9a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.563+0000 7f52e4e15740 4 client.16845 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:17.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.563+0000 7f52e4e15740 1 -- 172.21.15.49:0/4024004525 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d699f4a40 con 0x558d69a98240 2024-03-18T00:56:17.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.567+0000 7f52daffd640 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558d69a98240 0x558d69a9a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.587+0000 7f52baffd640 10 client.16845 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:17.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.587+0000 7f52daffd640 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558d69a98240 0x558d69a9a660 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66882907505c2270 server_cookie=ae7188de05ddea4c in_seq=0 out_seq=0 2024-03-18T00:56:17.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.599+0000 7f52baffd640 1 -- 172.21.15.49:0/4024004525 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558d699f4a40 con 0x558d69a98240 2024-03-18T00:56:17.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.599+0000 7f52baffd640 10 client.16845 handle_command_reply: tid=0 2024-03-18T00:56:17.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.600+0000 7f52e4e15740 1 client.16845 shutdown 2024-03-18T00:56:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.605+0000 7f52e4e15740 1 -- 172.21.15.49:0/4024004525 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 msgr2=0x558d69a8cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.605+0000 7f52e4e15740 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f52dc101730 tx=0x7f52cc00e8e0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.605+0000 7f52e4e15740 1 -- 172.21.15.49:0/4024004525 shutdown_connections 2024-03-18T00:56:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.605+0000 7f52e4e15740 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558d69a98240 0x558d69a9a660 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.605+0000 7f52e4e15740 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a87cd0 0x558d69a93ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.605+0000 7f52e4e15740 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.605+0000 7f52e4e15740 1 --2- 172.21.15.49:0/4024004525 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a82170 0x558d69a8c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.605+0000 7f52e4e15740 1 -- 172.21.15.49:0/4024004525 >> 172.21.15.49:0/4024004525 conn(0x558d69a7d6e0 msgr2=0x558d69a7c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:17.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.606+0000 7f52baffd640 0 client.16845 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:17.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.610+0000 7f52e4e15740 1 -- 172.21.15.49:0/4024004525 shutdown_connections 2024-03-18T00:56:17.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.611+0000 7f52e4e15740 1 -- 172.21.15.49:0/4024004525 wait complete. 2024-03-18T00:56:17.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.611+0000 7f52e4e15740 20 client.16845 trim_cache size 0 max 16384 2024-03-18T00:56:17.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.699+0000 7f52e4e15740 1 Processor -- start 2024-03-18T00:56:17.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.700+0000 7f52e4e15740 1 -- start start 2024-03-18T00:56:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.701+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a8c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.701+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.701+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a87cd0 0x558d69a93c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.701+0000 7f52e4e15740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc1aaf50 con 0x558d69a82b60 2024-03-18T00:56:17.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.701+0000 7f52e4e15740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc1aca90 con 0x558d69a82170 2024-03-18T00:56:17.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.701+0000 7f52e4e15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc1ab620 con 0x558d69a87cd0 2024-03-18T00:56:17.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.702+0000 7f52db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.702+0000 7f52db7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38598/0 (socket says 172.21.15.49:38598) 2024-03-18T00:56:17.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.702+0000 7f52db7fe640 1 -- 172.21.15.49:0/2776302963 learned_addr learned my addr 172.21.15.49:0/2776302963 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:17.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.703+0000 7f52daffd640 1 --2- 172.21.15.49:0/2776302963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a8c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.705+0000 7f52db7fe640 1 -- 172.21.15.49:0/2776302963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a87cd0 msgr2=0x558d69a93c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:17.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.705+0000 7f52db7fe640 1 --2- 172.21.15.49:0/2776302963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a87cd0 0x558d69a93c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.705+0000 7f52db7fe640 1 -- 172.21.15.49:0/2776302963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 msgr2=0x558d69a8c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:17.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.705+0000 7f52db7fe640 1 --2- 172.21.15.49:0/2776302963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a8c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.705+0000 7f52db7fe640 1 -- 172.21.15.49:0/2776302963 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d696bb0f0 con 0x558d69a82b60 2024-03-18T00:56:17.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.705+0000 7f52db7fe640 1 --2- 172.21.15.49:0/2776302963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009d70 tx=0x7f52cc01b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89060f9c35bfaf30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.714+0000 7f52c17fa640 1 -- 172.21.15.49:0/2776302963 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52cc017050 con 0x558d69a82b60 2024-03-18T00:56:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.714+0000 7f52c17fa640 1 -- 172.21.15.49:0/2776302963 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52cc006d00 con 0x558d69a82b60 2024-03-18T00:56:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.714+0000 7f52c17fa640 1 -- 172.21.15.49:0/2776302963 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52cc0138e0 con 0x558d69a82b60 2024-03-18T00:56:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.716+0000 7f52e4e15740 1 -- 172.21.15.49:0/2776302963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 msgr2=0x558d69a8cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.716+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2776302963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009d70 tx=0x7f52cc01b670 comp rx=0 tx=0).stop 2024-03-18T00:56:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.717+0000 7f52e4e15740 1 -- 172.21.15.49:0/2776302963 shutdown_connections 2024-03-18T00:56:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.717+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2776302963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a87cd0 0x558d69a93c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.717+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2776302963 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a82b60 0x558d69a8cc60 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.717+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2776302963 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a8c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.717+0000 7f52e4e15740 1 -- 172.21.15.49:0/2776302963 >> 172.21.15.49:0/2776302963 conn(0x558d69a7d6e0 msgr2=0x558d69a84240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.719+0000 7f52e4e15740 1 -- 172.21.15.49:0/2776302963 shutdown_connections 2024-03-18T00:56:17.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.719+0000 7f52e4e15740 1 -- 172.21.15.49:0/2776302963 wait complete. 2024-03-18T00:56:17.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.720+0000 7f52e4e15740 1 -- start start 2024-03-18T00:56:17.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.721+0000 7f52e4e15740 1 Processor -- start 2024-03-18T00:56:17.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.722+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a93b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.722+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a82b60 0x558d69a905d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.723+0000 7f52e4e15740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a87cd0 0x558d69a90b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.723+0000 7f52e4e15740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc1ac3c0 con 0x558d69a87cd0 2024-03-18T00:56:17.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.723+0000 7f52e4e15740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc1abcf0 con 0x558d69a82170 2024-03-18T00:56:17.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.723+0000 7f52e4e15740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc1a8f30 con 0x558d69a82b60 2024-03-18T00:56:17.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.723+0000 7f52e4e15740 10 client.0 _ll_register_callbacks cb 0x7f52cc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:17.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.724+0000 7f52daffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a87cd0 0x558d69a90b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.724+0000 7f52daffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a87cd0 0x558d69a90b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38600/0 (socket says 172.21.15.49:38600) 2024-03-18T00:56:17.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.724+0000 7f52daffd640 1 -- 172.21.15.49:0/2770610191 learned_addr learned my addr 172.21.15.49:0/2770610191 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:17.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.726+0000 7f52db7fe640 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a82b60 0x558d69a905d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.726+0000 7f52daffd640 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a93b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.726+0000 7f52daffd640 1 -- 172.21.15.49:0/2770610191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a82b60 msgr2=0x558d69a905d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:17.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.726+0000 7f52daffd640 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a82b60 0x558d69a905d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.726+0000 7f52daffd640 1 -- 172.21.15.49:0/2770610191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 msgr2=0x558d69a93b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:17.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.726+0000 7f52daffd640 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a93b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.726+0000 7f52daffd640 1 -- 172.21.15.49:0/2770610191 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d699ae9d0 con 0x558d69a87cd0 2024-03-18T00:56:17.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.728+0000 7f52daffd640 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a87cd0 0x558d69a90b10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f52d002c820 tx=0x7f52d0045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a480173d1a876f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:17.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.729+0000 7f52baffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:17.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.729+0000 7f52baffd640 1 -- 172.21.15.49:0/2770610191 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d0048040 con 0x558d69a87cd0 2024-03-18T00:56:17.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.729+0000 7f52baffd640 1 -- 172.21.15.49:0/2770610191 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f52d002ca80 con 0x558d69a87cd0 2024-03-18T00:56:17.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.729+0000 7f52e4e15740 1 -- 172.21.15.49:0/2770610191 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558d699eed90 con 0x558d69a87cd0 2024-03-18T00:56:17.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.730+0000 7f52baffd640 1 -- 172.21.15.49:0/2770610191 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f52d0061050 con 0x558d69a87cd0 2024-03-18T00:56:17.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.730+0000 7f52e4e15740 10 client.16851 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:17.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.730+0000 7f52e4e15740 1 -- 172.21.15.49:0/2770610191 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x558d69b05d80 con 0x558d69a87cd0 2024-03-18T00:56:17.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.731+0000 7f52baffd640 1 -- 172.21.15.49:0/2770610191 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558d699eed90 con 0x558d69a87cd0 2024-03-18T00:56:17.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.731+0000 7f52baffd640 1 -- 172.21.15.49:0/2770610191 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f52d0004b00 con 0x558d69a87cd0 2024-03-18T00:56:17.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.731+0000 7f52e4e15740 10 client.16851 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:17.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.731+0000 7f52e4e15740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:17.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.732+0000 7f52e4e15740 20 client.16851 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:17.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.732+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558d69b06280 0x558d69b08670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:17.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.732+0000 7f52e4e15740 4 client.16851 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:56:17.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.732+0000 7f52e4e15740 1 -- 172.21.15.49:0/2770610191 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558d6996c540 con 0x558d69b06280 2024-03-18T00:56:17.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.736+0000 7f52db7fe640 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558d69b06280 0x558d69b08670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:17.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.746+0000 7f52db7fe640 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558d69b06280 0x558d69b08670 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cec6e4898c73d0d6 server_cookie=4392581c6b651800 in_seq=0 out_seq=0 2024-03-18T00:56:17.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.747+0000 7f52baffd640 10 client.16851 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:17.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.748+0000 7f52baffd640 1 -- 172.21.15.49:0/2770610191 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558d6996c540 con 0x558d69b06280 2024-03-18T00:56:17.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.748+0000 7f52baffd640 10 client.16851 handle_command_reply: tid=0 2024-03-18T00:56:17.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.750+0000 7f52e4e15740 1 client.16851 shutdown 2024-03-18T00:56:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.762+0000 7f52e4e15740 1 -- 172.21.15.49:0/2770610191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a87cd0 msgr2=0x558d69a90b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.762+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a87cd0 0x558d69a90b10 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f52d002c820 tx=0x7f52d0045040 comp rx=0 tx=0).stop 2024-03-18T00:56:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.762+0000 7f52e4e15740 1 -- 172.21.15.49:0/2770610191 shutdown_connections 2024-03-18T00:56:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.762+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558d69b06280 0x558d69b08670 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.762+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558d69a87cd0 0x558d69a90b10 secure :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f52d002c820 tx=0x7f52d0045040 comp rx=0 tx=0).stop 2024-03-18T00:56:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.762+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558d69a82b60 0x558d69a905d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.762+0000 7f52e4e15740 1 --2- 172.21.15.49:0/2770610191 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558d69a82170 0x558d69a93b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.762+0000 7f52e4e15740 1 -- 172.21.15.49:0/2770610191 >> 172.21.15.49:0/2770610191 conn(0x558d69a7d6e0 msgr2=0x558d69a84240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.762+0000 7f52baffd640 0 client.16851 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.764+0000 7f52e4e15740 1 -- 172.21.15.49:0/2770610191 shutdown_connections 2024-03-18T00:56:17.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.764+0000 7f52e4e15740 1 -- 172.21.15.49:0/2770610191 wait complete. 2024-03-18T00:56:17.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.765+0000 7f52e4e15740 20 client.16851 trim_cache size 0 max 16384 2024-03-18T00:56:17.769 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:56:17.769 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:56:17.769 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:56:17.769 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:56:17.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52c40a28b0 msgr2=0x7f52c40a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52c40a28b0 0x7f52c40a4d70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f52d4097380 tx=0x7f52d400e040 comp rx=0 tx=0).stop 2024-03-18T00:56:17.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52dc102290 msgr2=0x7f52dc19f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:17.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52dc102290 0x7f52dc19f1c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f52d400be20 tx=0x7f52d4009ed0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 shutdown_connections 2024-03-18T00:56:17.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f52c40a28b0 0x7f52c40a4d70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52dc102290 0x7f52dc19f1c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f52dc1018d0 0x7f52dc1981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 --2- 172.21.15.49:0/2272962853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52dc1006e0 0x7f52dc197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:17.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.771+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 >> 172.21.15.49:0/2272962853 conn(0x7f52dc0fbc70 msgr2=0x7f52dc068940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:17.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.772+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 shutdown_connections 2024-03-18T00:56:17.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:17.774+0000 7f52e2227640 1 -- 172.21.15.49:0/2272962853 wait complete. 2024-03-18T00:56:17.806 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:56:17.807 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:56:17.857 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:17 smithi049 ceph-mon[32130]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 882 MiB/s wr, 1.78k op/s 2024-03-18T00:56:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:17 smithi107 ceph-mon[36270]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 882 MiB/s wr, 1.78k op/s 2024-03-18T00:56:17.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:17 smithi106 ceph-mon[35899]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 882 MiB/s wr, 1.78k op/s 2024-03-18T00:56:18.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.143+0000 7f32b8bd4640 1 -- 172.21.15.49:0/257094057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4109930 msgr2=0x7f32b4110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:18.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.143+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/257094057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4109930 0x7f32b4110250 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009900 tx=0x7f32a802bc30 comp rx=0 tx=0).stop 2024-03-18T00:56:18.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.143+0000 7f32b8bd4640 1 -- 172.21.15.49:0/257094057 shutdown_connections 2024-03-18T00:56:18.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.143+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/257094057 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32b4108f70 0x7f32b41093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.143+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/257094057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4109930 0x7f32b4110250 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.143+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/257094057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32b4107e20 0x7f32b4108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.143+0000 7f32b8bd4640 1 -- 172.21.15.49:0/257094057 >> 172.21.15.49:0/257094057 conn(0x7f32b4076df0 msgr2=0x7f32b4079210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:18.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.144+0000 7f32b8bd4640 1 -- 172.21.15.49:0/257094057 shutdown_connections 2024-03-18T00:56:18.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.144+0000 7f32b8bd4640 1 -- 172.21.15.49:0/257094057 wait complete. 2024-03-18T00:56:18.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.146+0000 7f32b8bd4640 1 Processor -- start 2024-03-18T00:56:18.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.146+0000 7f32b8bd4640 1 -- start start 2024-03-18T00:56:18.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.147+0000 7f32b8bd4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4107e20 0x7f32b419df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:18.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.147+0000 7f32b8bd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32b4108f70 0x7f32b419e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:18.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.147+0000 7f32b8bd4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32b4109930 0x7f32b41a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:18.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.147+0000 7f32b8bd4640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f32b4113aa0 con 0x7f32b4107e20 2024-03-18T00:56:18.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.147+0000 7f32b8bd4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f32b4113920 con 0x7f32b4109930 2024-03-18T00:56:18.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.147+0000 7f32b8bd4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f32b4113c20 con 0x7f32b4108f70 2024-03-18T00:56:18.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.147+0000 7f32b259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4107e20 0x7f32b419df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:18.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.147+0000 7f32b259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4107e20 0x7f32b419df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38614/0 (socket says 172.21.15.49:38614) 2024-03-18T00:56:18.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.147+0000 7f32b259c640 1 -- 172.21.15.49:0/4237984548 learned_addr learned my addr 172.21.15.49:0/4237984548 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:18.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.148+0000 7f32b259c640 1 -- 172.21.15.49:0/4237984548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32b4108f70 msgr2=0x7f32b419e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:18.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.148+0000 7f32b259c640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32b4108f70 0x7f32b419e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.148+0000 7f32b259c640 1 -- 172.21.15.49:0/4237984548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32b4109930 msgr2=0x7f32b41a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:18.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.148+0000 7f32b259c640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32b4109930 0x7f32b41a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.148+0000 7f32b259c640 1 -- 172.21.15.49:0/4237984548 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b41a5b80 con 0x7f32b4107e20 2024-03-18T00:56:18.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.149+0000 7f32b259c640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4107e20 0x7f32b419df60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f32a400c920 tx=0x7f32a400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:18.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.150+0000 7f32a37fe640 1 -- 172.21.15.49:0/4237984548 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a4007c70 con 0x7f32b4107e20 2024-03-18T00:56:18.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.150+0000 7f32a37fe640 1 -- 172.21.15.49:0/4237984548 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32a4007e10 con 0x7f32b4107e20 2024-03-18T00:56:18.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.152+0000 7f32a37fe640 1 -- 172.21.15.49:0/4237984548 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32a4018890 con 0x7f32b4107e20 2024-03-18T00:56:18.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.152+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b41a5e70 con 0x7f32b4107e20 2024-03-18T00:56:18.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.153+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b41a6220 con 0x7f32b4107e20 2024-03-18T00:56:18.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.160+0000 7f32a37fe640 1 -- 172.21.15.49:0/4237984548 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f32a4018a30 con 0x7f32b4107e20 2024-03-18T00:56:18.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.162+0000 7f32a37fe640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f329c0a2830 0x7f329c0a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:18.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.163+0000 7f32b1d9b640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f329c0a2830 0x7f329c0a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:18.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.164+0000 7f32a37fe640 1 -- 172.21.15.49:0/4237984548 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f32a4098f30 con 0x7f32b4107e20 2024-03-18T00:56:18.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.166+0000 7f32b1d9b640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f329c0a2830 0x7f329c0a4cf0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f32b419f470 tx=0x7f32ac0071e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:18.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.166+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b4068a20 con 0x7f32b4107e20 2024-03-18T00:56:18.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.225+0000 7f32a37fe640 1 -- 172.21.15.49:0/4237984548 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f32a401d070 con 0x7f32b4107e20 2024-03-18T00:56:18.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.906+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f32b410f060 con 0x7f32b4107e20 2024-03-18T00:56:18.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.907+0000 7f32a37fe640 1 -- 172.21.15.49:0/4237984548 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f32b410f060 con 0x7f32b4107e20 2024-03-18T00:56:18.907 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:56:18.907 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:53:25.016081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24470,24463,24481,24506,24499]},"id":1}]} 2024-03-18T00:56:18.907 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 27 2024-03-18T00:56:18.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.910+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f329c0a2830 msgr2=0x7f329c0a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:18.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.911+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f329c0a2830 0x7f329c0a4cf0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f32b419f470 tx=0x7f32ac0071e0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.911+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4107e20 msgr2=0x7f32b419df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:18.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.911+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4107e20 0x7f32b419df60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f32a400c920 tx=0x7f32a400cde0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.911+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 shutdown_connections 2024-03-18T00:56:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.911+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f329c0a2830 0x7f329c0a4cf0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.911+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f32b4109930 0x7f32b41a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.911+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f32b4108f70 0x7f32b419e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.911+0000 7f32b8bd4640 1 --2- 172.21.15.49:0/4237984548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f32b4107e20 0x7f32b419df60 secure :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f32a400c920 tx=0x7f32a400cde0 comp rx=0 tx=0).stop 2024-03-18T00:56:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.911+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 >> 172.21.15.49:0/4237984548 conn(0x7f32b4076df0 msgr2=0x7f32b410c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.912+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 shutdown_connections 2024-03-18T00:56:18.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:18.912+0000 7f32b8bd4640 1 -- 172.21.15.49:0/4237984548 wait complete. 2024-03-18T00:56:18.979 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:56:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.378+0000 7f74e59ce640 1 -- 172.21.15.49:0/2870330358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74e0108eb0 msgr2=0x7f74e0109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.378+0000 7f74e59ce640 1 --2- 172.21.15.49:0/2870330358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74e0108eb0 0x7f74e0109330 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f74d4009900 tx=0x7f74d402bc80 comp rx=0 tx=0).stop 2024-03-18T00:56:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.381+0000 7f74e59ce640 1 -- 172.21.15.49:0/2870330358 shutdown_connections 2024-03-18T00:56:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.381+0000 7f74e59ce640 1 --2- 172.21.15.49:0/2870330358 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74e0109870 0x7f74e010e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.381+0000 7f74e59ce640 1 --2- 172.21.15.49:0/2870330358 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74e0108eb0 0x7f74e0109330 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.381+0000 7f74e59ce640 1 --2- 172.21.15.49:0/2870330358 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74e0072bf0 0x7f74e0073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.381+0000 7f74e59ce640 1 -- 172.21.15.49:0/2870330358 >> 172.21.15.49:0/2870330358 conn(0x7f74e006e6c0 msgr2=0x7f74e0070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:19.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.385+0000 7f74e59ce640 1 -- 172.21.15.49:0/2870330358 shutdown_connections 2024-03-18T00:56:19.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.387+0000 7f74e59ce640 1 -- 172.21.15.49:0/2870330358 wait complete. 2024-03-18T00:56:19.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.388+0000 7f74e59ce640 1 Processor -- start 2024-03-18T00:56:19.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.388+0000 7f74e59ce640 1 -- start start 2024-03-18T00:56:19.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.389+0000 7f74e59ce640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74e0072bf0 0x7f74e01a6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.389+0000 7f74e59ce640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74e0108eb0 0x7f74e01a6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.389+0000 7f74e59ce640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74e0109870 0x7f74e0079a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.389+0000 7f74e59ce640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f74e011c200 con 0x7f74e0108eb0 2024-03-18T00:56:19.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.389+0000 7f74e59ce640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f74e011c080 con 0x7f74e0072bf0 2024-03-18T00:56:19.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.389+0000 7f74e59ce640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f74e011c380 con 0x7f74e0109870 2024-03-18T00:56:19.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.391+0000 7f74e51cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74e0109870 0x7f74e0079a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.391+0000 7f74e51cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74e0109870 0x7f74e0079a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:43916/0 (socket says 172.21.15.49:43916) 2024-03-18T00:56:19.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.391+0000 7f74e51cd640 1 -- 172.21.15.49:0/3545745744 learned_addr learned my addr 172.21.15.49:0/3545745744 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:19.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.392+0000 7f74e51cd640 1 -- 172.21.15.49:0/3545745744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74e0072bf0 msgr2=0x7f74e01a6800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:56:19.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.392+0000 7f74e51cd640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74e0072bf0 0x7f74e01a6800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.392+0000 7f74e51cd640 1 -- 172.21.15.49:0/3545745744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74e0108eb0 msgr2=0x7f74e01a6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:19.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.392+0000 7f74e51cd640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74e0108eb0 0x7f74e01a6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.392+0000 7f74e51cd640 1 -- 172.21.15.49:0/3545745744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e007a0e0 con 0x7f74e0109870 2024-03-18T00:56:19.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.393+0000 7f74e51cd640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74e0109870 0x7f74e0079a40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f74d800cad0 tx=0x7f74d800cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:19.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.393+0000 7f74ddffb640 1 -- 172.21.15.49:0/3545745744 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d800ae20 con 0x7f74e0109870 2024-03-18T00:56:19.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.393+0000 7f74ddffb640 1 -- 172.21.15.49:0/3545745744 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d8005020 con 0x7f74e0109870 2024-03-18T00:56:19.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.393+0000 7f74ddffb640 1 -- 172.21.15.49:0/3545745744 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d800f6f0 con 0x7f74e0109870 2024-03-18T00:56:19.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.394+0000 7f74e59ce640 1 -- 172.21.15.49:0/3545745744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e007a3d0 con 0x7f74e0109870 2024-03-18T00:56:19.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.394+0000 7f74e59ce640 1 -- 172.21.15.49:0/3545745744 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e007a8d0 con 0x7f74e0109870 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.414+0000 7f74e9816740 1 Processor -- start 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.414+0000 7f74e9816740 1 -- start start 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.414+0000 7f74e9816740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5afea40 0x55f0a5afee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.414+0000 7f74ddffb640 1 -- 172.21.15.49:0/3545745744 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f74d80051d0 con 0x7f74e0109870 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.414+0000 7f74e9816740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 0x55f0a5b03f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.414+0000 7f74e9816740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 0x55f0a5b06a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.414+0000 7f74e9816740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f0a5a63490 con 0x55f0a5b045a0 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.414+0000 7f74e9816740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f0a5a0bc90 con 0x55f0a5afea40 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.414+0000 7f74e9816740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f0a5a5f530 con 0x55f0a5aff430 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.416+0000 7f74ddffb640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f74c80a2a00 0x7f74c80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.416+0000 7f74dffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5afea40 0x55f0a5afee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.416+0000 7f74dffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5afea40 0x55f0a5afee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59680/0 (socket says 172.21.15.49:59680) 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.416+0000 7f74dffff640 1 -- 172.21.15.49:0/1659149200 learned_addr learned my addr 172.21.15.49:0/1659149200 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.416+0000 7f74dffff640 1 --2- 172.21.15.49:0/1659149200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 0x55f0a5b06a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.416+0000 7f74ddffb640 1 -- 172.21.15.49:0/3545745744 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f74d80ceb00 con 0x7f74e0109870 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.417+0000 7f74e51cd640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f74c80a2a00 0x7f74c80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.417+0000 7f74e49cc640 1 --2- 172.21.15.49:0/1659149200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 0x55f0a5b03f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.420+0000 7f74dffff640 1 -- 172.21.15.49:0/1659149200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 msgr2=0x55f0a5b03f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.420+0000 7f74dffff640 1 --2- 172.21.15.49:0/1659149200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 0x55f0a5b03f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.420+0000 7f74dffff640 1 -- 172.21.15.49:0/1659149200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5afea40 msgr2=0x55f0a5afee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.420+0000 7f74dffff640 1 --2- 172.21.15.49:0/1659149200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5afea40 0x55f0a5afee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.420+0000 7f74dffff640 1 -- 172.21.15.49:0/1659149200 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a5ac6500 con 0x55f0a5b045a0 2024-03-18T00:56:19.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.424+0000 7f74e51cd640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f74c80a2a00 0x7f74c80a4ec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f74d8097880 tx=0x7f74d8095f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:19.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.426+0000 7f74dffff640 1 --2- 172.21.15.49:0/1659149200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 0x55f0a5b06a30 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f74d402c930 tx=0x7f74d4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f53cefd9126419f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:19.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.434+0000 7f74c6ffd640 1 -- 172.21.15.49:0/1659149200 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4005000 con 0x55f0a5b045a0 2024-03-18T00:56:19.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.434+0000 7f74c6ffd640 1 -- 172.21.15.49:0/1659149200 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d4048070 con 0x55f0a5b045a0 2024-03-18T00:56:19.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.435+0000 7f74e9816740 1 -- 172.21.15.49:0/1659149200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 msgr2=0x55f0a5b06a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:19.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.435+0000 7f74e9816740 1 --2- 172.21.15.49:0/1659149200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 0x55f0a5b06a30 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f74d402c930 tx=0x7f74d4005fd0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.435+0000 7f74e9816740 1 -- 172.21.15.49:0/1659149200 shutdown_connections 2024-03-18T00:56:19.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.435+0000 7f74e9816740 1 --2- 172.21.15.49:0/1659149200 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 0x55f0a5b06a30 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.435+0000 7f74e9816740 1 --2- 172.21.15.49:0/1659149200 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 0x55f0a5b03f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.435+0000 7f74e9816740 1 --2- 172.21.15.49:0/1659149200 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5afea40 0x55f0a5afee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.435+0000 7f74e9816740 1 -- 172.21.15.49:0/1659149200 >> 172.21.15.49:0/1659149200 conn(0x55f0a5af9fb0 msgr2=0x55f0a5afa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:19.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.442+0000 7f74e9816740 1 -- 172.21.15.49:0/1659149200 shutdown_connections 2024-03-18T00:56:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.442+0000 7f74e9816740 1 -- 172.21.15.49:0/1659149200 wait complete. 2024-03-18T00:56:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.443+0000 7f74e9816740 1 -- start start 2024-03-18T00:56:19.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.443+0000 7f74e9816740 1 Processor -- start 2024-03-18T00:56:19.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.444+0000 7f74e9816740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5afea40 0x55f0a5b08f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.444+0000 7f74e9816740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5aff430 0x55f0a5b094d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.444+0000 7f74e9816740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 0x55f0a5b10510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.444+0000 7f74e9816740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f0a5b07540 con 0x55f0a5b045a0 2024-03-18T00:56:19.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.444+0000 7f74e9816740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f0a5a1f230 con 0x55f0a5aff430 2024-03-18T00:56:19.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.444+0000 7f74e9816740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f0a5b076c0 con 0x55f0a5afea40 2024-03-18T00:56:19.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.444+0000 7f74e9816740 10 client.0 _ll_register_callbacks cb 0x55f0a5a5b010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:19.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.445+0000 7f74dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5afea40 0x55f0a5b08f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.445+0000 7f74dffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5afea40 0x55f0a5b08f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:43932/0 (socket says 172.21.15.49:43932) 2024-03-18T00:56:19.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.445+0000 7f74dffff640 1 -- 172.21.15.49:0/600390059 learned_addr learned my addr 172.21.15.49:0/600390059 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:19.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.446+0000 7f74dffff640 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 0x55f0a5b10510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.446+0000 7f74dffff640 1 -- 172.21.15.49:0/600390059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5aff430 msgr2=0x55f0a5b094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:19.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.446+0000 7f74dffff640 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5aff430 0x55f0a5b094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.446+0000 7f74dffff640 1 -- 172.21.15.49:0/600390059 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 msgr2=0x55f0a5b10510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:19.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.446+0000 7f74dffff640 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 0x55f0a5b10510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.446+0000 7f74dffff640 1 -- 172.21.15.49:0/600390059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0a5a2e100 con 0x55f0a5afea40 2024-03-18T00:56:19.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.447+0000 7f74dffff640 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5afea40 0x55f0a5b08f90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f74d4009850 tx=0x7f74d402e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12124ded972b6357 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:19.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.449+0000 7f74bffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:56:19.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.450+0000 7f74bffff640 1 -- 172.21.15.49:0/600390059 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4005550 con 0x55f0a5afea40 2024-03-18T00:56:19.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.450+0000 7f74bffff640 1 -- 172.21.15.49:0/600390059 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f74d40056f0 con 0x55f0a5afea40 2024-03-18T00:56:19.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.450+0000 7f74e9816740 1 -- 172.21.15.49:0/600390059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74d4048070 con 0x55f0a5afea40 2024-03-18T00:56:19.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.452+0000 7f74bffff640 1 -- 172.21.15.49:0/600390059 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f74d4004030 con 0x55f0a5afea40 2024-03-18T00:56:19.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.452+0000 7f74e9816740 10 client.26260 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:19.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.452+0000 7f74e9816740 1 -- 172.21.15.49:0/600390059 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f0a57773f0 con 0x55f0a5afea40 2024-03-18T00:56:19.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.453+0000 7f74bffff640 1 -- 172.21.15.49:0/600390059 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d4048070 con 0x55f0a5afea40 2024-03-18T00:56:19.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.455+0000 7f74bffff640 1 -- 172.21.15.49:0/600390059 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f74d40435f0 con 0x55f0a5afea40 2024-03-18T00:56:19.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.455+0000 7f74e9816740 10 client.26260 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:19.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.455+0000 7f74e9816740 10 client.26260 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:56:19.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.455+0000 7f74e9816740 20 client.26260 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:19.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.455+0000 7f74e9816740 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f0a5b25dd0 0x55f0a5b28210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.455+0000 7f74e9816740 4 client.26260 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:19.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.455+0000 7f74e9816740 1 -- 172.21.15.49:0/600390059 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0a5a709d0 con 0x55f0a5b25dd0 2024-03-18T00:56:19.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.458+0000 7f74e49cc640 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f0a5b25dd0 0x55f0a5b28210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.466+0000 7f74e49cc640 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f0a5b25dd0 0x55f0a5b28210 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ff27c45a1afce986 server_cookie=55e7c1446ce54a04 in_seq=0 out_seq=0 2024-03-18T00:56:19.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.466+0000 7f74bffff640 10 client.26260 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:56:19.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.472+0000 7f74bffff640 1 -- 172.21.15.49:0/600390059 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f0a5a709d0 con 0x55f0a5b25dd0 2024-03-18T00:56:19.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.472+0000 7f74bffff640 10 client.26260 handle_command_reply: tid=0 2024-03-18T00:56:19.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.472+0000 7f74e9816740 1 client.26260 shutdown 2024-03-18T00:56:19.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.478+0000 7f74e9816740 1 -- 172.21.15.49:0/600390059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5afea40 msgr2=0x55f0a5b08f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:19.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.478+0000 7f74e9816740 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5afea40 0x55f0a5b08f90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f74d4009850 tx=0x7f74d402e040 comp rx=0 tx=0).stop 2024-03-18T00:56:19.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.479+0000 7f74e9816740 1 -- 172.21.15.49:0/600390059 shutdown_connections 2024-03-18T00:56:19.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.479+0000 7f74e9816740 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f0a5b25dd0 0x55f0a5b28210 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.479+0000 7f74e9816740 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5aff430 0x55f0a5b094d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.479+0000 7f74e9816740 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5b045a0 0x55f0a5b10510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.479+0000 7f74e9816740 1 --2- 172.21.15.49:0/600390059 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5afea40 0x55f0a5b08f90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.479+0000 7f74e9816740 1 -- 172.21.15.49:0/600390059 >> 172.21.15.49:0/600390059 conn(0x55f0a5af9fb0 msgr2=0x55f0a5af9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:19.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.479+0000 7f74bffff640 0 client.26260 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:56:19.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.479+0000 7f74e9816740 1 -- 172.21.15.49:0/600390059 shutdown_connections 2024-03-18T00:56:19.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.482+0000 7f74e9816740 1 -- 172.21.15.49:0/600390059 wait complete. 2024-03-18T00:56:19.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.482+0000 7f74e9816740 20 client.26260 trim_cache size 0 max 16384 2024-03-18T00:56:19.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.541+0000 7f74e9816740 1 Processor -- start 2024-03-18T00:56:19.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.541+0000 7f74e9816740 1 -- start start 2024-03-18T00:56:19.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.542+0000 7f74e9816740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b092b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.542+0000 7f74e9816740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5aff430 0x55f0a5af88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.542+0000 7f74e9816740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5b045a0 0x55f0a5af8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.543+0000 7f74e9816740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f74e007afc0 con 0x55f0a5afea40 2024-03-18T00:56:19.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.543+0000 7f74e9816740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f74e007cb00 con 0x55f0a5aff430 2024-03-18T00:56:19.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.543+0000 7f74e9816740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f74e007b690 con 0x55f0a5b045a0 2024-03-18T00:56:19.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.546+0000 7f74e49cc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5aff430 0x55f0a5af88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.549+0000 7f74dffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b092b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.549+0000 7f74dffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b092b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38634/0 (socket says 172.21.15.49:38634) 2024-03-18T00:56:19.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.549+0000 7f74dffff640 1 -- 172.21.15.49:0/3520391799 learned_addr learned my addr 172.21.15.49:0/3520391799 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:19.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.549+0000 7f74dffff640 1 --2- 172.21.15.49:0/3520391799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5b045a0 0x55f0a5af8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.550+0000 7f74dffff640 1 -- 172.21.15.49:0/3520391799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5b045a0 msgr2=0x55f0a5af8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:19.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.550+0000 7f74dffff640 1 --2- 172.21.15.49:0/3520391799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5b045a0 0x55f0a5af8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.550+0000 7f74dffff640 1 -- 172.21.15.49:0/3520391799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5aff430 msgr2=0x55f0a5af88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:19.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.550+0000 7f74dffff640 1 --2- 172.21.15.49:0/3520391799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5aff430 0x55f0a5af88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.550+0000 7f74dffff640 1 -- 172.21.15.49:0/3520391799 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0a58417d0 con 0x55f0a5afea40 2024-03-18T00:56:19.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.551+0000 7f74dffff640 1 --2- 172.21.15.49:0/3520391799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b092b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f74d402c930 tx=0x7f74d4005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43286b4318cb7d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:19.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.551+0000 7f74c67fc640 1 -- 172.21.15.49:0/3520391799 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d40435f0 con 0x55f0a5afea40 2024-03-18T00:56:19.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.551+0000 7f74c67fc640 1 -- 172.21.15.49:0/3520391799 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d4048040 con 0x55f0a5afea40 2024-03-18T00:56:19.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.551+0000 7f74c67fc640 1 -- 172.21.15.49:0/3520391799 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4004230 con 0x55f0a5afea40 2024-03-18T00:56:19.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.553+0000 7f74e9816740 1 -- 172.21.15.49:0/3520391799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 msgr2=0x55f0a5b092b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:19.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.553+0000 7f74e9816740 1 --2- 172.21.15.49:0/3520391799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b092b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f74d402c930 tx=0x7f74d4005fd0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.553+0000 7f74e9816740 1 -- 172.21.15.49:0/3520391799 shutdown_connections 2024-03-18T00:56:19.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.553+0000 7f74e9816740 1 --2- 172.21.15.49:0/3520391799 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5aff430 0x55f0a5af88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.553+0000 7f74e9816740 1 --2- 172.21.15.49:0/3520391799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5b045a0 0x55f0a5af8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.553+0000 7f74e9816740 1 --2- 172.21.15.49:0/3520391799 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b092b0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.553+0000 7f74e9816740 1 -- 172.21.15.49:0/3520391799 >> 172.21.15.49:0/3520391799 conn(0x55f0a5af9fb0 msgr2=0x55f0a5b08ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:19.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.556+0000 7f74e9816740 1 -- 172.21.15.49:0/3520391799 shutdown_connections 2024-03-18T00:56:19.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.556+0000 7f74e9816740 1 -- 172.21.15.49:0/3520391799 wait complete. 2024-03-18T00:56:19.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.557+0000 7f74e9816740 1 -- start start 2024-03-18T00:56:19.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.557+0000 7f74e9816740 1 Processor -- start 2024-03-18T00:56:19.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.559+0000 7f74e9816740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b2ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.559+0000 7f74e9816740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 0x55f0a5b2c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.559+0000 7f74e9816740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5b045a0 0x55f0a5b2c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.559+0000 7f74e9816740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f74e007c430 con 0x55f0a5afea40 2024-03-18T00:56:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.559+0000 7f74e9816740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f74e007bd60 con 0x55f0a5b045a0 2024-03-18T00:56:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.559+0000 7f74e9816740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f74e0116b60 con 0x55f0a5aff430 2024-03-18T00:56:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.559+0000 7f74e9816740 10 client.0 _ll_register_callbacks cb 0x7f74d402ca10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.560+0000 7f74dffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b2ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.560+0000 7f74dffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b2ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38636/0 (socket says 172.21.15.49:38636) 2024-03-18T00:56:19.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.560+0000 7f74dffff640 1 -- 172.21.15.49:0/495247447 learned_addr learned my addr 172.21.15.49:0/495247447 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:19.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.561+0000 7f74e49cc640 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 0x55f0a5b2c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.561+0000 7f74dffff640 1 -- 172.21.15.49:0/495247447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 msgr2=0x55f0a5b2c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:19.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.561+0000 7f74dffff640 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 0x55f0a5b2c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.561+0000 7f74dffff640 1 -- 172.21.15.49:0/495247447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5b045a0 msgr2=0x55f0a5b2c940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:19.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.561+0000 7f74dffff640 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5b045a0 0x55f0a5b2c940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.561+0000 7f74dffff640 1 -- 172.21.15.49:0/495247447 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0a5b2d170 con 0x55f0a5afea40 2024-03-18T00:56:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.562+0000 7f74dffff640 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b2ddf0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f74d4043b00 tx=0x7f74d40097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af02a26f6fe4c370 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.562+0000 7f74bffff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:19.563 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.563+0000 7f74bffff640 1 -- 172.21.15.49:0/495247447 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d40043d0 con 0x55f0a5afea40 2024-03-18T00:56:19.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.563+0000 7f74e9816740 1 -- 172.21.15.49:0/495247447 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74d4048040 con 0x55f0a5afea40 2024-03-18T00:56:19.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.563+0000 7f74bffff640 1 -- 172.21.15.49:0/495247447 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f74d4005350 con 0x55f0a5afea40 2024-03-18T00:56:19.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.564+0000 7f74bffff640 1 -- 172.21.15.49:0/495247447 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f74d4005570 con 0x55f0a5afea40 2024-03-18T00:56:19.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.564+0000 7f74e9816740 10 client.16872 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:19.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.564+0000 7f74e9816740 1 -- 172.21.15.49:0/495247447 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f0a5b8b110 con 0x55f0a5afea40 2024-03-18T00:56:19.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.566+0000 7f74bffff640 1 -- 172.21.15.49:0/495247447 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74d4048040 con 0x55f0a5afea40 2024-03-18T00:56:19.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.567+0000 7f74bffff640 1 -- 172.21.15.49:0/495247447 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f74d4060450 con 0x55f0a5afea40 2024-03-18T00:56:19.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.568+0000 7f74e9816740 10 client.16872 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:19.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.568+0000 7f74e9816740 10 client.16872 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:56:19.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.568+0000 7f74e9816740 20 client.16872 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:19.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.568+0000 7f74e9816740 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f0a5b8b880 0x55f0a5b8dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:19.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.568+0000 7f74e9816740 4 client.16872 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:56:19.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.568+0000 7f74e9816740 1 -- 172.21.15.49:0/495247447 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f0a5a6ad20 con 0x55f0a5b8b880 2024-03-18T00:56:19.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.569+0000 7f74e49cc640 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f0a5b8b880 0x55f0a5b8dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.579+0000 7f74e49cc640 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f0a5b8b880 0x55f0a5b8dca0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2211ec1104fb9629 server_cookie=a19cc90a7234edda in_seq=0 out_seq=0 2024-03-18T00:56:19.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.579+0000 7f74bffff640 10 client.16872 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:56:19.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.585+0000 7f74bffff640 1 -- 172.21.15.49:0/495247447 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f0a5a6ad20 con 0x55f0a5b8b880 2024-03-18T00:56:19.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.585+0000 7f74bffff640 10 client.16872 handle_command_reply: tid=0 2024-03-18T00:56:19.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.585+0000 7f74e9816740 1 client.16872 shutdown 2024-03-18T00:56:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.586+0000 7f74e9816740 1 -- 172.21.15.49:0/495247447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 msgr2=0x55f0a5b2ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.586+0000 7f74e9816740 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b2ddf0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f74d4043b00 tx=0x7f74d40097a0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.586+0000 7f74e9816740 1 -- 172.21.15.49:0/495247447 shutdown_connections 2024-03-18T00:56:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.586+0000 7f74e9816740 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f0a5b8b880 0x55f0a5b8dca0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.586+0000 7f74e9816740 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f0a5b045a0 0x55f0a5b2c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.587+0000 7f74e9816740 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f0a5aff430 0x55f0a5b2c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.587+0000 7f74e9816740 1 --2- 172.21.15.49:0/495247447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f0a5afea40 0x55f0a5b2ddf0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.587+0000 7f74e9816740 1 -- 172.21.15.49:0/495247447 >> 172.21.15.49:0/495247447 conn(0x55f0a5af9fb0 msgr2=0x55f0a5af88b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:19.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.589+0000 7f74bffff640 0 client.16872 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:56:19.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.593+0000 7f74e9816740 1 -- 172.21.15.49:0/495247447 shutdown_connections 2024-03-18T00:56:19.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.596+0000 7f74e9816740 1 -- 172.21.15.49:0/495247447 wait complete. 2024-03-18T00:56:19.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.597+0000 7f74e9816740 20 client.16872 trim_cache size 0 max 16384 2024-03-18T00:56:19.598 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:56:19.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.598+0000 7f74e59ce640 1 -- 172.21.15.49:0/3545745744 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f74c80a2a00 msgr2=0x7f74c80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:19.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.598+0000 7f74e59ce640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f74c80a2a00 0x7f74c80a4ec0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f74d8097880 tx=0x7f74d8095f90 comp rx=0 tx=0).stop 2024-03-18T00:56:19.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.598+0000 7f74e59ce640 1 -- 172.21.15.49:0/3545745744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74e0109870 msgr2=0x7f74e0079a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:19.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.598+0000 7f74e59ce640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74e0109870 0x7f74e0079a40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f74d800cad0 tx=0x7f74d800cf90 comp rx=0 tx=0).stop 2024-03-18T00:56:19.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.599+0000 7f74e59ce640 1 -- 172.21.15.49:0/3545745744 shutdown_connections 2024-03-18T00:56:19.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.599+0000 7f74e59ce640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f74c80a2a00 0x7f74c80a4ec0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.599+0000 7f74e59ce640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74e0109870 0x7f74e0079a40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.599+0000 7f74e59ce640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f74e0108eb0 0x7f74e01a6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.599+0000 7f74e59ce640 1 --2- 172.21.15.49:0/3545745744 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74e0072bf0 0x7f74e01a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:19.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.599+0000 7f74e59ce640 1 -- 172.21.15.49:0/3545745744 >> 172.21.15.49:0/3545745744 conn(0x7f74e006e6c0 msgr2=0x7f74e0070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:19.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.602+0000 7f74e59ce640 1 -- 172.21.15.49:0/3545745744 shutdown_connections 2024-03-18T00:56:19.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:19.603+0000 7f74e59ce640 1 -- 172.21.15.49:0/3545745744 wait complete. 2024-03-18T00:56:19.636 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:56:19.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:19 smithi049 ceph-mon[32130]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 862 MiB/s wr, 1.97k op/s 2024-03-18T00:56:19.878 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:19 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/4237984548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:56:19.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:19 smithi107 ceph-mon[36270]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 862 MiB/s wr, 1.97k op/s 2024-03-18T00:56:19.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:19 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/4237984548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:56:19.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:19 smithi106 ceph-mon[35899]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 862 MiB/s wr, 1.97k op/s 2024-03-18T00:56:19.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:19 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/4237984548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:56:20.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.002+0000 7fb21bfff640 1 -- 172.21.15.49:0/1444690592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb21c0ff560 msgr2=0x7fb21c0ff980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.002+0000 7fb21bfff640 1 --2- 172.21.15.49:0/1444690592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb21c0ff560 0x7fb21c0ff980 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb21401acb0 tx=0x7fb21403d170 comp rx=0 tx=0).stop 2024-03-18T00:56:20.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.004+0000 7fb21bfff640 1 -- 172.21.15.49:0/1444690592 shutdown_connections 2024-03-18T00:56:20.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.004+0000 7fb21bfff640 1 --2- 172.21.15.49:0/1444690592 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb21c100690 0x7fb21c100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.004+0000 7fb21bfff640 1 --2- 172.21.15.49:0/1444690592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c101030 0x7fb21c1058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.004+0000 7fb21bfff640 1 --2- 172.21.15.49:0/1444690592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb21c0ff560 0x7fb21c0ff980 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.004+0000 7fb21bfff640 1 -- 172.21.15.49:0/1444690592 >> 172.21.15.49:0/1444690592 conn(0x7fb21c0fae60 msgr2=0x7fb21c0fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.005+0000 7fb21bfff640 1 -- 172.21.15.49:0/1444690592 shutdown_connections 2024-03-18T00:56:20.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.005+0000 7fb21bfff640 1 -- 172.21.15.49:0/1444690592 wait complete. 2024-03-18T00:56:20.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.007+0000 7fb21bfff640 1 Processor -- start 2024-03-18T00:56:20.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.007+0000 7fb21bfff640 1 -- start start 2024-03-18T00:56:20.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.009+0000 7fb21bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb21c0ff560 0x7fb21c195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.009+0000 7fb21bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c100690 0x7fb21c195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.009+0000 7fb21bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb21c101030 0x7fb21c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.010+0000 7fb21bfff640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c109290 con 0x7fb21c0ff560 2024-03-18T00:56:20.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.010+0000 7fb21bfff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c109110 con 0x7fb21c101030 2024-03-18T00:56:20.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.010+0000 7fb21bfff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c109410 con 0x7fb21c100690 2024-03-18T00:56:20.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.010+0000 7fb21a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c100690 0x7fb21c195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.011+0000 7fb21a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c100690 0x7fb21c195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:43966/0 (socket says 172.21.15.49:43966) 2024-03-18T00:56:20.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.011+0000 7fb21a7fc640 1 -- 172.21.15.49:0/3665027375 learned_addr learned my addr 172.21.15.49:0/3665027375 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.011+0000 7fb21affd640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb21c0ff560 0x7fb21c195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.012+0000 7fb21affd640 1 -- 172.21.15.49:0/3665027375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c100690 msgr2=0x7fb21c195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.012+0000 7fb21affd640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c100690 0x7fb21c195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.012+0000 7fb21affd640 1 -- 172.21.15.49:0/3665027375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb21c101030 msgr2=0x7fb21c19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.012+0000 7fb21affd640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb21c101030 0x7fb21c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.012+0000 7fb21affd640 1 -- 172.21.15.49:0/3665027375 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21c19d420 con 0x7fb21c0ff560 2024-03-18T00:56:20.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.013+0000 7fb21affd640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb21c0ff560 0x7fb21c195800 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb21403d780 tx=0x7fb214045440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.014+0000 7fb213fff640 1 -- 172.21.15.49:0/3665027375 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21404a040 con 0x7fb21c0ff560 2024-03-18T00:56:20.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.015+0000 7fb213fff640 1 -- 172.21.15.49:0/3665027375 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb214055070 con 0x7fb21c0ff560 2024-03-18T00:56:20.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.015+0000 7fb213fff640 1 -- 172.21.15.49:0/3665027375 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214050b60 con 0x7fb21c0ff560 2024-03-18T00:56:20.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.015+0000 7fb21bfff640 1 -- 172.21.15.49:0/3665027375 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb21c19d6b0 con 0x7fb21c0ff560 2024-03-18T00:56:20.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.015+0000 7fb21bfff640 1 -- 172.21.15.49:0/3665027375 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb21c19db20 con 0x7fb21c0ff560 2024-03-18T00:56:20.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.024+0000 7fb213fff640 1 -- 172.21.15.49:0/3665027375 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb214050d00 con 0x7fb21c0ff560 2024-03-18T00:56:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.025+0000 7fb213fff640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb2080a28d0 0x7fb2080a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.026+0000 7fb224249740 1 Processor -- start 2024-03-18T00:56:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.026+0000 7fb224249740 1 -- start start 2024-03-18T00:56:20.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.026+0000 7fb213fff640 1 -- 172.21.15.49:0/3665027375 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb2141014e0 con 0x7fb21c0ff560 2024-03-18T00:56:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.027+0000 7fb224249740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.027+0000 7fb224249740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 0x55fa24164f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.027+0000 7fb224249740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 0x55fa24167a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.028+0000 7fb224249740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55fa240c4490 con 0x55fa24160430 2024-03-18T00:56:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.028+0000 7fb224249740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55fa2406cc90 con 0x55fa2415fa40 2024-03-18T00:56:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.028+0000 7fb224249740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fa240c0530 con 0x55fa241655a0 2024-03-18T00:56:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.028+0000 7fb21a7fc640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb2080a28d0 0x7fb2080a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.029+0000 7fb21affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.029+0000 7fb21affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2415fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59736/0 (socket says 172.21.15.49:59736) 2024-03-18T00:56:20.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.029+0000 7fb21affd640 1 -- 172.21.15.49:0/3087441631 learned_addr learned my addr 172.21.15.49:0/3087441631 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.029 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.029+0000 7fb21b7fe640 1 --2- 172.21.15.49:0/3087441631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 0x55fa24167a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.030+0000 7fb21affd640 1 -- 172.21.15.49:0/3087441631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 msgr2=0x55fa24167a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.030+0000 7fb21affd640 1 --2- 172.21.15.49:0/3087441631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 0x55fa24167a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.030+0000 7fb21affd640 1 -- 172.21.15.49:0/3087441631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 msgr2=0x55fa24164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.030+0000 7fb21affd640 1 --2- 172.21.15.49:0/3087441631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 0x55fa24164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.030+0000 7fb21affd640 1 -- 172.21.15.49:0/3087441631 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa24127500 con 0x55fa2415fa40 2024-03-18T00:56:20.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.030+0000 7fb21b7fe640 1 --2- 172.21.15.49:0/3087441631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 0x55fa24167a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:56:20.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.031+0000 7fb21affd640 1 --2- 172.21.15.49:0/3087441631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2415fe60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb2140d60d0 tx=0x7fb214000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e0be543c5c378b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.031 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.032+0000 7fb2117fa640 1 -- 172.21.15.49:0/3087441631 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2140ce560 con 0x55fa2415fa40 2024-03-18T00:56:20.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.032+0000 7fb2117fa640 1 -- 172.21.15.49:0/3087441631 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2140cde60 con 0x55fa2415fa40 2024-03-18T00:56:20.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.032+0000 7fb2117fa640 1 -- 172.21.15.49:0/3087441631 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2141010d0 con 0x55fa2415fa40 2024-03-18T00:56:20.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.034+0000 7fb224249740 1 -- 172.21.15.49:0/3087441631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 msgr2=0x55fa2415fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.034+0000 7fb224249740 1 --2- 172.21.15.49:0/3087441631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2415fe60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb2140d60d0 tx=0x7fb214000f40 comp rx=0 tx=0).stop 2024-03-18T00:56:20.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.034+0000 7fb224249740 1 -- 172.21.15.49:0/3087441631 shutdown_connections 2024-03-18T00:56:20.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.034+0000 7fb224249740 1 --2- 172.21.15.49:0/3087441631 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 0x55fa24167a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.034+0000 7fb224249740 1 --2- 172.21.15.49:0/3087441631 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 0x55fa24164f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.034+0000 7fb224249740 1 --2- 172.21.15.49:0/3087441631 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2415fe60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.034+0000 7fb224249740 1 -- 172.21.15.49:0/3087441631 >> 172.21.15.49:0/3087441631 conn(0x55fa2415afb0 msgr2=0x55fa2415b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.035+0000 7fb224249740 1 -- 172.21.15.49:0/3087441631 shutdown_connections 2024-03-18T00:56:20.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.036+0000 7fb21a7fc640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb2080a28d0 0x7fb2080a4d90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x55fa24160830 tx=0x7fb20400b890 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.036+0000 7fb224249740 1 -- 172.21.15.49:0/3087441631 wait complete. 2024-03-18T00:56:20.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.037+0000 7fb224249740 1 -- start start 2024-03-18T00:56:20.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.037+0000 7fb224249740 1 Processor -- start 2024-03-18T00:56:20.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.037+0000 7fb224249740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa2415fa40 0x55fa24169cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.037+0000 7fb224249740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa24160430 0x55fa2416a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.037+0000 7fb224249740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa241655a0 0x55fa24171230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.037+0000 7fb224249740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55fa24168310 con 0x55fa2415fa40 2024-03-18T00:56:20.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.037+0000 7fb224249740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55fa24080230 con 0x55fa241655a0 2024-03-18T00:56:20.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.037+0000 7fb224249740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fa24168490 con 0x55fa24160430 2024-03-18T00:56:20.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.037+0000 7fb224249740 10 client.0 _ll_register_callbacks cb 0x55fa240bc010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:20.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.039+0000 7fb21b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa241655a0 0x55fa24171230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.039+0000 7fb21a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa24160430 0x55fa2416a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.039+0000 7fb21b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa241655a0 0x55fa24171230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59740/0 (socket says 172.21.15.49:59740) 2024-03-18T00:56:20.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.040+0000 7fb21b7fe640 1 -- 172.21.15.49:0/178880927 learned_addr learned my addr 172.21.15.49:0/178880927 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.040+0000 7fb21affd640 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa2415fa40 0x55fa24169cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.040+0000 7fb21affd640 1 -- 172.21.15.49:0/178880927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa24160430 msgr2=0x55fa2416a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.040+0000 7fb21affd640 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa24160430 0x55fa2416a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.040+0000 7fb21affd640 1 -- 172.21.15.49:0/178880927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa241655a0 msgr2=0x55fa24171230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.040+0000 7fb21affd640 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa241655a0 0x55fa24171230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.040+0000 7fb21affd640 1 -- 172.21.15.49:0/178880927 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2408f100 con 0x55fa2415fa40 2024-03-18T00:56:20.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.041+0000 7fb21affd640 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa2415fa40 0x55fa24169cc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x55fa241681a0 tx=0x7fb21403d170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c45e1dd0f05dee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.043+0000 7fb2027fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:20.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.044+0000 7fb2027fc640 1 -- 172.21.15.49:0/178880927 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2141012d0 con 0x55fa2415fa40 2024-03-18T00:56:20.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.044+0000 7fb224249740 1 -- 172.21.15.49:0/178880927 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2140cde60 con 0x55fa2415fa40 2024-03-18T00:56:20.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.045+0000 7fb2027fc640 1 -- 172.21.15.49:0/178880927 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb2140d6170 con 0x55fa2415fa40 2024-03-18T00:56:20.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.045+0000 7fb2027fc640 1 -- 172.21.15.49:0/178880927 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb2140cbf00 con 0x55fa2415fa40 2024-03-18T00:56:20.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.048+0000 7fb224249740 10 client.16881 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:20.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.048+0000 7fb224249740 1 -- 172.21.15.49:0/178880927 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fa23dd83f0 con 0x55fa2415fa40 2024-03-18T00:56:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.050+0000 7fb2027fc640 1 -- 172.21.15.49:0/178880927 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2140cde60 con 0x55fa2415fa40 2024-03-18T00:56:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.050+0000 7fb2027fc640 1 -- 172.21.15.49:0/178880927 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb2140cd2b0 con 0x55fa2415fa40 2024-03-18T00:56:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.050+0000 7fb224249740 10 client.16881 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.050+0000 7fb224249740 10 client.16881 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:56:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.050+0000 7fb224249740 20 client.16881 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.050+0000 7fb224249740 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55fa241867b0 0x55fa24188ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.050+0000 7fb224249740 4 client.16881 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:20.050 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.050+0000 7fb224249740 1 -- 172.21.15.49:0/178880927 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa240d19d0 con 0x55fa241867b0 2024-03-18T00:56:20.052 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.052+0000 7fb21a7fc640 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55fa241867b0 0x55fa24188ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.056+0000 7fb21a7fc640 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55fa241867b0 0x55fa24188ba0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b938d0551d0ac466 server_cookie=4442a757d6e77268 in_seq=0 out_seq=0 2024-03-18T00:56:20.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.056+0000 7fb2027fc640 10 client.16881 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:56:20.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.063+0000 7fb2027fc640 1 -- 172.21.15.49:0/178880927 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fa240d19d0 con 0x55fa241867b0 2024-03-18T00:56:20.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.064+0000 7fb2027fc640 10 client.16881 handle_command_reply: tid=0 2024-03-18T00:56:20.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.064+0000 7fb224249740 1 client.16881 shutdown 2024-03-18T00:56:20.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.065+0000 7fb224249740 1 -- 172.21.15.49:0/178880927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa2415fa40 msgr2=0x55fa24169cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.065+0000 7fb224249740 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa2415fa40 0x55fa24169cc0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x55fa241681a0 tx=0x7fb21403d170 comp rx=0 tx=0).stop 2024-03-18T00:56:20.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.066+0000 7fb224249740 1 -- 172.21.15.49:0/178880927 shutdown_connections 2024-03-18T00:56:20.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.066+0000 7fb224249740 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55fa241867b0 0x55fa24188ba0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.066+0000 7fb224249740 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa241655a0 0x55fa24171230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.066+0000 7fb224249740 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa24160430 0x55fa2416a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.066+0000 7fb224249740 1 --2- 172.21.15.49:0/178880927 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa2415fa40 0x55fa24169cc0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.066 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.066+0000 7fb2027fc640 0 client.16881 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:56:20.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.066+0000 7fb224249740 1 -- 172.21.15.49:0/178880927 >> 172.21.15.49:0/178880927 conn(0x55fa2415afb0 msgr2=0x55fa2415a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.066+0000 7fb224249740 1 -- 172.21.15.49:0/178880927 shutdown_connections 2024-03-18T00:56:20.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.066+0000 7fb224249740 1 -- 172.21.15.49:0/178880927 wait complete. 2024-03-18T00:56:20.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.067+0000 7fb224249740 20 client.16881 trim_cache size 0 max 16384 2024-03-18T00:56:20.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.106+0000 7fb224249740 1 Processor -- start 2024-03-18T00:56:20.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.106+0000 7fb224249740 1 -- start start 2024-03-18T00:56:20.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.107+0000 7fb224249740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa24169f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.107+0000 7fb224249740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa24160430 0x55fa24171100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.107+0000 7fb224249740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa241655a0 0x55fa24168710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb224249740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c19e280 con 0x55fa241655a0 2024-03-18T00:56:20.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb224249740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c19fdc0 con 0x55fa2415fa40 2024-03-18T00:56:20.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb224249740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c19e950 con 0x55fa24160430 2024-03-18T00:56:20.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb21b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa241655a0 0x55fa24168710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb21b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa241655a0 0x55fa24168710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38706/0 (socket says 172.21.15.49:38706) 2024-03-18T00:56:20.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb21b7fe640 1 -- 172.21.15.49:0/1508295045 learned_addr learned my addr 172.21.15.49:0/1508295045 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb21affd640 1 --2- 172.21.15.49:0/1508295045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa24169f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb21b7fe640 1 -- 172.21.15.49:0/1508295045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa24160430 msgr2=0x55fa24171100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb21b7fe640 1 --2- 172.21.15.49:0/1508295045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa24160430 0x55fa24171100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb21b7fe640 1 -- 172.21.15.49:0/1508295045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 msgr2=0x55fa24169f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb21b7fe640 1 --2- 172.21.15.49:0/1508295045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa24169f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.108+0000 7fb21b7fe640 1 -- 172.21.15.49:0/1508295045 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa23ea27d0 con 0x55fa241655a0 2024-03-18T00:56:20.109 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.109+0000 7fb21b7fe640 1 --2- 172.21.15.49:0/1508295045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa241655a0 0x55fa24168710 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00dac0 tx=0x7fb20c00df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba6ff4732e92840 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.110+0000 7fb210ff9640 1 -- 172.21.15.49:0/1508295045 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c00fd80 con 0x55fa241655a0 2024-03-18T00:56:20.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.110+0000 7fb210ff9640 1 -- 172.21.15.49:0/1508295045 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb20c0035f0 con 0x55fa241655a0 2024-03-18T00:56:20.111 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.110+0000 7fb210ff9640 1 -- 172.21.15.49:0/1508295045 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c0196c0 con 0x55fa241655a0 2024-03-18T00:56:20.112 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.113+0000 7fb224249740 1 -- 172.21.15.49:0/1508295045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa241655a0 msgr2=0x55fa24168710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.113 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.113+0000 7fb224249740 1 --2- 172.21.15.49:0/1508295045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa241655a0 0x55fa24168710 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00dac0 tx=0x7fb20c00df80 comp rx=0 tx=0).stop 2024-03-18T00:56:20.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.115+0000 7fb224249740 1 -- 172.21.15.49:0/1508295045 shutdown_connections 2024-03-18T00:56:20.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.115+0000 7fb224249740 1 --2- 172.21.15.49:0/1508295045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa241655a0 0x55fa24168710 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.115+0000 7fb224249740 1 --2- 172.21.15.49:0/1508295045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa24160430 0x55fa24171100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.115+0000 7fb224249740 1 --2- 172.21.15.49:0/1508295045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa24169f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.115+0000 7fb224249740 1 -- 172.21.15.49:0/1508295045 >> 172.21.15.49:0/1508295045 conn(0x55fa2415afb0 msgr2=0x55fa2415a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.123+0000 7fb224249740 1 -- 172.21.15.49:0/1508295045 shutdown_connections 2024-03-18T00:56:20.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.123+0000 7fb224249740 1 -- 172.21.15.49:0/1508295045 wait complete. 2024-03-18T00:56:20.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.124+0000 7fb224249740 1 -- start start 2024-03-18T00:56:20.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.125+0000 7fb224249740 1 Processor -- start 2024-03-18T00:56:20.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.125+0000 7fb224249740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2416e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.126 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.126+0000 7fb224249740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 0x55fa2416e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.127+0000 7fb224249740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 0x55fa2416f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.127 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.127+0000 7fb224249740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c19f6f0 con 0x55fa24160430 2024-03-18T00:56:20.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.127+0000 7fb224249740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c19f020 con 0x55fa2415fa40 2024-03-18T00:56:20.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.128+0000 7fb21a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 0x55fa2416e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.128+0000 7fb21a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 0x55fa2416e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38712/0 (socket says 172.21.15.49:38712) 2024-03-18T00:56:20.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.128+0000 7fb21a7fc640 1 -- 172.21.15.49:0/2428242655 learned_addr learned my addr 172.21.15.49:0/2428242655 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.129+0000 7fb21affd640 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2416e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.129+0000 7fb224249740 1 -- 172.21.15.49:0/2428242655 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c071e50 con 0x55fa241655a0 2024-03-18T00:56:20.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.130+0000 7fb224249740 10 client.0 _ll_register_callbacks cb 0x7fb2140cd9a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:20.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.130+0000 7fb21b7fe640 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 0x55fa2416f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.132+0000 7fb21a7fc640 1 -- 172.21.15.49:0/2428242655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 msgr2=0x55fa2416f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.132+0000 7fb21a7fc640 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 0x55fa2416f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.132+0000 7fb21a7fc640 1 -- 172.21.15.49:0/2428242655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 msgr2=0x55fa2416e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.132+0000 7fb21a7fc640 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2416e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.132+0000 7fb21a7fc640 1 -- 172.21.15.49:0/2428242655 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa2416fe00 con 0x55fa24160430 2024-03-18T00:56:20.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.133+0000 7fb21a7fc640 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 0x55fa2416e8e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb204005fd0 tx=0x7fb2040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f41397c89460742 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.133+0000 7fb2027fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:20.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.134+0000 7fb2027fc640 1 -- 172.21.15.49:0/2428242655 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204005110 con 0x55fa24160430 2024-03-18T00:56:20.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.134+0000 7fb224249740 1 -- 172.21.15.49:0/2428242655 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb20c0035f0 con 0x55fa24160430 2024-03-18T00:56:20.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.137+0000 7fb2027fc640 1 -- 172.21.15.49:0/2428242655 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb2040052b0 con 0x55fa24160430 2024-03-18T00:56:20.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.138+0000 7fb2027fc640 1 -- 172.21.15.49:0/2428242655 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb2040054d0 con 0x55fa24160430 2024-03-18T00:56:20.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.138+0000 7fb224249740 10 client.16887 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:20.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.138+0000 7fb224249740 1 -- 172.21.15.49:0/2428242655 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55fa24171200 con 0x55fa24160430 2024-03-18T00:56:20.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.141+0000 7fb2027fc640 1 -- 172.21.15.49:0/2428242655 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb204012e50 con 0x55fa24160430 2024-03-18T00:56:20.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.141+0000 7fb2027fc640 1 -- 172.21.15.49:0/2428242655 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb204019040 con 0x55fa24160430 2024-03-18T00:56:20.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.142+0000 7fb224249740 10 client.16887 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:20.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.142+0000 7fb224249740 10 client.16887 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:56:20.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.142+0000 7fb224249740 20 client.16887 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:20.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.142+0000 7fb224249740 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55fa241ecdd0 0x55fa241ef1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.142+0000 7fb224249740 4 client.16887 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:56:20.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.142+0000 7fb224249740 1 -- 172.21.15.49:0/2428242655 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb20c0035f0 con 0x55fa241ecdd0 2024-03-18T00:56:20.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.143+0000 7fb21affd640 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55fa241ecdd0 0x55fa241ef1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.149+0000 7fb21affd640 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55fa241ecdd0 0x55fa241ef1f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8e70f24bf8a54b9f server_cookie=897e6dd4f0c21726 in_seq=0 out_seq=0 2024-03-18T00:56:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.149+0000 7fb2027fc640 10 client.16887 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:56:20.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.149+0000 7fb2027fc640 1 -- 172.21.15.49:0/2428242655 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb20c0035f0 con 0x55fa241ecdd0 2024-03-18T00:56:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.149+0000 7fb2027fc640 10 client.16887 handle_command_reply: tid=0 2024-03-18T00:56:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.149+0000 7fb224249740 1 client.16887 shutdown 2024-03-18T00:56:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.150+0000 7fb224249740 1 -- 172.21.15.49:0/2428242655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 msgr2=0x55fa2416e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.150+0000 7fb224249740 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 0x55fa2416e8e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb204005fd0 tx=0x7fb2040046c0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.151+0000 7fb224249740 1 -- 172.21.15.49:0/2428242655 shutdown_connections 2024-03-18T00:56:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.151+0000 7fb224249740 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55fa241ecdd0 0x55fa241ef1f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.152+0000 7fb224249740 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa241655a0 0x55fa2416f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.152+0000 7fb224249740 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa24160430 0x55fa2416e8e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.152+0000 7fb224249740 1 --2- 172.21.15.49:0/2428242655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa2415fa40 0x55fa2416e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.152+0000 7fb224249740 1 -- 172.21.15.49:0/2428242655 >> 172.21.15.49:0/2428242655 conn(0x55fa2415afb0 msgr2=0x55fa2415a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.152+0000 7fb2027fc640 0 client.16887 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:56:20.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.152+0000 7fb224249740 1 -- 172.21.15.49:0/2428242655 shutdown_connections 2024-03-18T00:56:20.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.152+0000 7fb224249740 1 -- 172.21.15.49:0/2428242655 wait complete. 2024-03-18T00:56:20.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.152+0000 7fb224249740 20 client.16887 trim_cache size 0 max 16384 2024-03-18T00:56:20.154 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:56:20.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.155+0000 7fb21bfff640 1 -- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb2080a28d0 msgr2=0x7fb2080a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.155+0000 7fb21bfff640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb2080a28d0 0x7fb2080a4d90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x55fa24160830 tx=0x7fb20400b890 comp rx=0 tx=0).stop 2024-03-18T00:56:20.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.155+0000 7fb21bfff640 1 -- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb21c0ff560 msgr2=0x7fb21c195800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.155+0000 7fb21bfff640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb21c0ff560 0x7fb21c195800 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb21403d780 tx=0x7fb214045440 comp rx=0 tx=0).stop 2024-03-18T00:56:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.155+0000 7fb21bfff640 1 -- 172.21.15.49:0/3665027375 shutdown_connections 2024-03-18T00:56:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.156+0000 7fb21bfff640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb2080a28d0 0x7fb2080a4d90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.156+0000 7fb21bfff640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb21c101030 0x7fb21c19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.156+0000 7fb21bfff640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb21c100690 0x7fb21c195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.156+0000 7fb21bfff640 1 --2- 172.21.15.49:0/3665027375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb21c0ff560 0x7fb21c195800 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.156+0000 7fb21bfff640 1 -- 172.21.15.49:0/3665027375 >> 172.21.15.49:0/3665027375 conn(0x7fb21c0fae60 msgr2=0x7fb21c1038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.156+0000 7fb21bfff640 1 -- 172.21.15.49:0/3665027375 shutdown_connections 2024-03-18T00:56:20.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.156+0000 7fb21bfff640 1 -- 172.21.15.49:0/3665027375 wait complete. 2024-03-18T00:56:20.240 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:56:20.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.696+0000 7fa12e774640 1 -- 172.21.15.49:0/461811664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa12810b150 msgr2=0x7fa12810d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.696+0000 7fa12e774640 1 --2- 172.21.15.49:0/461811664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa12810b150 0x7fa12810d5c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009900 tx=0x7fa11c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:56:20.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.697+0000 7fa12e774640 1 -- 172.21.15.49:0/461811664 shutdown_connections 2024-03-18T00:56:20.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.697+0000 7fa12e774640 1 --2- 172.21.15.49:0/461811664 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa12810b150 0x7fa12810d5c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.697+0000 7fa12e774640 1 --2- 172.21.15.49:0/461811664 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa12810db00 0x7fa12810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.697+0000 7fa12e774640 1 --2- 172.21.15.49:0/461811664 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1281087e0 0x7fa12810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.697+0000 7fa12e774640 1 -- 172.21.15.49:0/461811664 >> 172.21.15.49:0/461811664 conn(0x7fa12806e640 msgr2=0x7fa128070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.701+0000 7fa12e774640 1 -- 172.21.15.49:0/461811664 shutdown_connections 2024-03-18T00:56:20.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.701+0000 7fa12e774640 1 -- 172.21.15.49:0/461811664 wait complete. 2024-03-18T00:56:20.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.702+0000 7fa12e774640 1 Processor -- start 2024-03-18T00:56:20.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.703+0000 7fa12e774640 1 -- start start 2024-03-18T00:56:20.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.703+0000 7fa12e774640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1281087e0 0x7fa1281a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.703+0000 7fa12e774640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa12810b150 0x7fa1281a6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.704+0000 7fa12e774640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa12810db00 0x7fa1281adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.704+0000 7fa12e774640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa12811c3a0 con 0x7fa12810b150 2024-03-18T00:56:20.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.704+0000 7fa12e774640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa12811c220 con 0x7fa12810db00 2024-03-18T00:56:20.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.704+0000 7fa12e774640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa12811c520 con 0x7fa1281087e0 2024-03-18T00:56:20.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.705+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa12810b150 0x7fa1281a6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.705+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa12810b150 0x7fa1281a6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38734/0 (socket says 172.21.15.49:38734) 2024-03-18T00:56:20.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.705+0000 7fa1277fe640 1 -- 172.21.15.49:0/3562533174 learned_addr learned my addr 172.21.15.49:0/3562533174 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.707+0000 7fa12cd11640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa12810db00 0x7fa1281adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.707+0000 7fa127fff640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1281087e0 0x7fa1281a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.711+0000 7fa1277fe640 1 -- 172.21.15.49:0/3562533174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1281087e0 msgr2=0x7fa1281a69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.711+0000 7fa1277fe640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1281087e0 0x7fa1281a69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.711+0000 7fa1277fe640 1 -- 172.21.15.49:0/3562533174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa12810db00 msgr2=0x7fa1281adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.711+0000 7fa1277fe640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa12810db00 0x7fa1281adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.711+0000 7fa1277fe640 1 -- 172.21.15.49:0/3562533174 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa128110040 con 0x7fa12810b150 2024-03-18T00:56:20.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.712+0000 7fa1277fe640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa12810b150 0x7fa1281a6f20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa11c02c830 tx=0x7fa11c0045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.714+0000 7fa1257fa640 1 -- 172.21.15.49:0/3562533174 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0052d0 con 0x7fa12810b150 2024-03-18T00:56:20.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.715+0000 7fa1257fa640 1 -- 172.21.15.49:0/3562533174 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa11c044070 con 0x7fa12810b150 2024-03-18T00:56:20.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.716+0000 7fa12e774640 1 -- 172.21.15.49:0/3562533174 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1281102d0 con 0x7fa12810b150 2024-03-18T00:56:20.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.716+0000 7fa1257fa640 1 -- 172.21.15.49:0/3562533174 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c03f760 con 0x7fa12810b150 2024-03-18T00:56:20.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.717+0000 7fa12e774640 1 -- 172.21.15.49:0/3562533174 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa128110790 con 0x7fa12810b150 2024-03-18T00:56:20.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.728+0000 7fa1257fa640 1 -- 172.21.15.49:0/3562533174 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa11c03d040 con 0x7fa12810b150 2024-03-18T00:56:20.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.730+0000 7fa1257fa640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1100a2860 0x7fa1100a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.730+0000 7fa1257fa640 1 -- 172.21.15.49:0/3562533174 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa11c0effb0 con 0x7fa12810b150 2024-03-18T00:56:20.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.735+0000 7fa127fff640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1100a2860 0x7fa1100a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.737+0000 7fa131436740 1 Processor -- start 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.743+0000 7fa131436740 1 -- start start 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.743+0000 7fa131436740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ed8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.743+0000 7fa131436740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574eddd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.743+0000 7fa131436740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ede3b0 0x562574ee0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.743+0000 7fa131436740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x562574e3d490 con 0x562574ed8850 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.743+0000 7fa131436740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x562574de5c90 con 0x562574ed9240 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.743+0000 7fa131436740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562574e39530 con 0x562574ede3b0 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.744+0000 7fa127fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ed8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.744+0000 7fa127fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ed8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38740/0 (socket says 172.21.15.49:38740) 2024-03-18T00:56:20.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.744+0000 7fa127fff640 1 -- 172.21.15.49:0/2859803352 learned_addr learned my addr 172.21.15.49:0/2859803352 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.745+0000 7fa127fff640 1 -- 172.21.15.49:0/2859803352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ede3b0 msgr2=0x562574ee0840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:20.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.745+0000 7fa127fff640 1 --2- 172.21.15.49:0/2859803352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ede3b0 0x562574ee0840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.745+0000 7fa127fff640 1 -- 172.21.15.49:0/2859803352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 msgr2=0x562574eddd40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:20.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.745+0000 7fa127fff640 1 --2- 172.21.15.49:0/2859803352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574eddd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.745+0000 7fa127fff640 1 -- 172.21.15.49:0/2859803352 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562574ea0500 con 0x562574ed8850 2024-03-18T00:56:20.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.750+0000 7fa127fff640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1100a2860 0x7fa1100a4d20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa12000cd70 tx=0x7fa12000c860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.751+0000 7fa127fff640 1 --2- 172.21.15.49:0/2859803352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ed8c70 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa120007f80 tx=0x7fa12001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160db784e4e073b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.752+0000 7fa10e7fc640 1 -- 172.21.15.49:0/2859803352 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa120025600 con 0x562574ed8850 2024-03-18T00:56:20.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.757+0000 7fa10e7fc640 1 -- 172.21.15.49:0/2859803352 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa120016040 con 0x562574ed8850 2024-03-18T00:56:20.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.761+0000 7fa10e7fc640 1 -- 172.21.15.49:0/2859803352 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12001a380 con 0x562574ed8850 2024-03-18T00:56:20.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.767+0000 7fa131436740 1 -- 172.21.15.49:0/2859803352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 msgr2=0x562574ed8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.767+0000 7fa131436740 1 --2- 172.21.15.49:0/2859803352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ed8c70 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa120007f80 tx=0x7fa12001a000 comp rx=0 tx=0).stop 2024-03-18T00:56:20.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.775+0000 7fa131436740 1 -- 172.21.15.49:0/2859803352 shutdown_connections 2024-03-18T00:56:20.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.775+0000 7fa131436740 1 --2- 172.21.15.49:0/2859803352 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574eddd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.775+0000 7fa131436740 1 --2- 172.21.15.49:0/2859803352 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ede3b0 0x562574ee0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.775+0000 7fa131436740 1 --2- 172.21.15.49:0/2859803352 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ed8c70 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.775+0000 7fa131436740 1 -- 172.21.15.49:0/2859803352 >> 172.21.15.49:0/2859803352 conn(0x562574ed3dc0 msgr2=0x562574ed41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.785+0000 7fa131436740 1 -- 172.21.15.49:0/2859803352 shutdown_connections 2024-03-18T00:56:20.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.789+0000 7fa131436740 1 -- 172.21.15.49:0/2859803352 wait complete. 2024-03-18T00:56:20.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.790+0000 7fa131436740 1 -- start start 2024-03-18T00:56:20.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.790+0000 7fa131436740 1 Processor -- start 2024-03-18T00:56:20.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.791+0000 7fa131436740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ee2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.791+0000 7fa131436740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed9240 0x562574ee3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.791+0000 7fa131436740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ede3b0 0x562574eea320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.791+0000 7fa131436740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x562574ee1300 con 0x562574ed8850 2024-03-18T00:56:20.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.791+0000 7fa131436740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x562574df9230 con 0x562574ede3b0 2024-03-18T00:56:20.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.791+0000 7fa131436740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562574ee1480 con 0x562574ed9240 2024-03-18T00:56:20.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.791+0000 7fa131436740 10 client.0 _ll_register_callbacks cb 0x562574e35010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:20.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.793+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed9240 0x562574ee3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.793+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed9240 0x562574ee3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44058/0 (socket says 172.21.15.49:44058) 2024-03-18T00:56:20.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.793+0000 7fa1277fe640 1 -- 172.21.15.49:0/1339861253 learned_addr learned my addr 172.21.15.49:0/1339861253 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.794+0000 7fa12cd11640 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ede3b0 0x562574eea320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.794+0000 7fa127fff640 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ee2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.795+0000 7fa1277fe640 1 -- 172.21.15.49:0/1339861253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ede3b0 msgr2=0x562574eea320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.795+0000 7fa1277fe640 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ede3b0 0x562574eea320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.796+0000 7fa1277fe640 1 -- 172.21.15.49:0/1339861253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 msgr2=0x562574ee2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.796+0000 7fa1277fe640 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ee2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.796+0000 7fa1277fe640 1 -- 172.21.15.49:0/1339861253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562574e08100 con 0x562574ed9240 2024-03-18T00:56:20.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.802+0000 7fa1277fe640 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed9240 0x562574ee3350 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0b92a0 tx=0x7fa11c0bc2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88fb1ba197bbf93c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.805+0000 7fa1077fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:56:20.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.805+0000 7fa1077fe640 1 -- 172.21.15.49:0/1339861253 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0d5060 con 0x562574ed9240 2024-03-18T00:56:20.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.805+0000 7fa1077fe640 1 -- 172.21.15.49:0/1339861253 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa11c005470 con 0x562574ed9240 2024-03-18T00:56:20.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.805+0000 7fa131436740 1 -- 172.21.15.49:0/1339861253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa120016040 con 0x562574ed9240 2024-03-18T00:56:20.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.810+0000 7fa1077fe640 1 -- 172.21.15.49:0/1339861253 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa11c0da050 con 0x562574ed9240 2024-03-18T00:56:20.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.811+0000 7fa131436740 10 client.26281 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:20.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.811+0000 7fa131436740 1 -- 172.21.15.49:0/1339861253 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562574b513f0 con 0x562574ed9240 2024-03-18T00:56:20.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.815+0000 7fa1077fe640 1 -- 172.21.15.49:0/1339861253 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa120016040 con 0x562574ed9240 2024-03-18T00:56:20.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.815+0000 7fa1077fe640 1 -- 172.21.15.49:0/1339861253 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa11c0c4be0 con 0x562574ed9240 2024-03-18T00:56:20.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.819+0000 7fa131436740 10 client.26281 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:20.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.819+0000 7fa131436740 10 client.26281 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:56:20.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.819+0000 7fa131436740 20 client.26281 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:20.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.819+0000 7fa131436740 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562574eeee10 0x562574ef1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.819+0000 7fa131436740 4 client.26281 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:20.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.819+0000 7fa131436740 1 -- 172.21.15.49:0/1339861253 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562574e4a9d0 con 0x562574eeee10 2024-03-18T00:56:20.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.827+0000 7fa127fff640 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562574eeee10 0x562574ef1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.828+0000 7fa127fff640 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562574eeee10 0x562574ef1200 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ba06b12c329bc85f server_cookie=5a671a6a6453859d in_seq=0 out_seq=0 2024-03-18T00:56:20.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.828+0000 7fa1077fe640 10 client.26281 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:56:20.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.833+0000 7fa1077fe640 1 -- 172.21.15.49:0/1339861253 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562574e4a9d0 con 0x562574eeee10 2024-03-18T00:56:20.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.833+0000 7fa1077fe640 10 client.26281 handle_command_reply: tid=0 2024-03-18T00:56:20.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.833+0000 7fa131436740 1 client.26281 shutdown 2024-03-18T00:56:20.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.835+0000 7fa131436740 1 -- 172.21.15.49:0/1339861253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed9240 msgr2=0x562574ee3350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.835+0000 7fa131436740 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed9240 0x562574ee3350 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0b92a0 tx=0x7fa11c0bc2f0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.840+0000 7fa131436740 1 -- 172.21.15.49:0/1339861253 shutdown_connections 2024-03-18T00:56:20.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.840+0000 7fa131436740 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562574eeee10 0x562574ef1200 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.840+0000 7fa131436740 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ede3b0 0x562574eea320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.840+0000 7fa131436740 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed9240 0x562574ee3350 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.840+0000 7fa131436740 1 --2- 172.21.15.49:0/1339861253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ed8850 0x562574ee2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.840+0000 7fa131436740 1 -- 172.21.15.49:0/1339861253 >> 172.21.15.49:0/1339861253 conn(0x562574ed3dc0 msgr2=0x562574ed2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.840+0000 7fa1077fe640 0 client.26281 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:56:20.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.842+0000 7fa131436740 1 -- 172.21.15.49:0/1339861253 shutdown_connections 2024-03-18T00:56:20.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.842+0000 7fa131436740 1 -- 172.21.15.49:0/1339861253 wait complete. 2024-03-18T00:56:20.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.842+0000 7fa131436740 20 client.26281 trim_cache size 0 max 16384 2024-03-18T00:56:20.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.902+0000 7fa131436740 1 Processor -- start 2024-03-18T00:56:20.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.903+0000 7fa131436740 1 -- start start 2024-03-18T00:56:20.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.904+0000 7fa131436740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed8850 0x562574ecee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.905+0000 7fa131436740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574eea2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.905+0000 7fa131436740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574eda9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.906+0000 7fa131436740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281c02d0 con 0x562574ede3b0 2024-03-18T00:56:20.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.906+0000 7fa131436740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281c1e10 con 0x562574ed9240 2024-03-18T00:56:20.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.906+0000 7fa131436740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281c09a0 con 0x562574ed8850 2024-03-18T00:56:20.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.906+0000 7fa12cd11640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574eda9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.906+0000 7fa12cd11640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574eda9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38756/0 (socket says 172.21.15.49:38756) 2024-03-18T00:56:20.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.906+0000 7fa12cd11640 1 -- 172.21.15.49:0/4180800081 learned_addr learned my addr 172.21.15.49:0/4180800081 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.907+0000 7fa1277fe640 1 --2- 172.21.15.49:0/4180800081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574eea2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.907+0000 7fa12cd11640 1 -- 172.21.15.49:0/4180800081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed8850 msgr2=0x562574ecee70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:20.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.907+0000 7fa12cd11640 1 --2- 172.21.15.49:0/4180800081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed8850 0x562574ecee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.907+0000 7fa12cd11640 1 -- 172.21.15.49:0/4180800081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 msgr2=0x562574eea2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.907+0000 7fa12cd11640 1 --2- 172.21.15.49:0/4180800081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574eea2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.907+0000 7fa12cd11640 1 -- 172.21.15.49:0/4180800081 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562574c1b7d0 con 0x562574ede3b0 2024-03-18T00:56:20.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.907+0000 7fa1277fe640 1 --2- 172.21.15.49:0/4180800081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574eea2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:56:20.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.908+0000 7fa12cd11640 1 --2- 172.21.15.49:0/4180800081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574eda9c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa11800b640 tx=0x7fa11800bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823a1bd8610de58f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.910+0000 7fa10dffb640 1 -- 172.21.15.49:0/4180800081 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11801a020 con 0x562574ede3b0 2024-03-18T00:56:20.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.910+0000 7fa10dffb640 1 -- 172.21.15.49:0/4180800081 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa118010c70 con 0x562574ede3b0 2024-03-18T00:56:20.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.912+0000 7fa131436740 1 -- 172.21.15.49:0/4180800081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 msgr2=0x562574eda9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.912+0000 7fa131436740 1 --2- 172.21.15.49:0/4180800081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574eda9c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fa11800b640 tx=0x7fa11800bb00 comp rx=0 tx=0).stop 2024-03-18T00:56:20.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.915+0000 7fa131436740 1 -- 172.21.15.49:0/4180800081 shutdown_connections 2024-03-18T00:56:20.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.915+0000 7fa131436740 1 --2- 172.21.15.49:0/4180800081 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574eea2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.915+0000 7fa131436740 1 --2- 172.21.15.49:0/4180800081 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574eda9c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.915+0000 7fa131436740 1 --2- 172.21.15.49:0/4180800081 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed8850 0x562574ecee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.915 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.915+0000 7fa131436740 1 -- 172.21.15.49:0/4180800081 >> 172.21.15.49:0/4180800081 conn(0x562574ed3dc0 msgr2=0x562574ee1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.919 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.920+0000 7fa131436740 1 -- 172.21.15.49:0/4180800081 shutdown_connections 2024-03-18T00:56:20.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.923+0000 7fa131436740 1 -- 172.21.15.49:0/4180800081 wait complete. 2024-03-18T00:56:20.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.923+0000 7fa131436740 1 -- start start 2024-03-18T00:56:20.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.924+0000 7fa131436740 1 Processor -- start 2024-03-18T00:56:20.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.924+0000 7fa131436740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed8850 0x562574ee7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.924+0000 7fa131436740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574ee7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.924+0000 7fa131436740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574f530e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.924+0000 7fa131436740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281c1740 con 0x562574ede3b0 2024-03-18T00:56:20.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.924+0000 7fa131436740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281c1070 con 0x562574ed9240 2024-03-18T00:56:20.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.924+0000 7fa131436740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1281174f0 con 0x562574ed8850 2024-03-18T00:56:20.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.924+0000 7fa131436740 10 client.0 _ll_register_callbacks cb 0x7fa11c02c990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:20.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.925+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574ee7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.925+0000 7fa1277fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574ee7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59826/0 (socket says 172.21.15.49:59826) 2024-03-18T00:56:20.925 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.925+0000 7fa1277fe640 1 -- 172.21.15.49:0/1939062164 learned_addr learned my addr 172.21.15.49:0/1939062164 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:20.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.926+0000 7fa12cd11640 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574f530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.926+0000 7fa1277fe640 1 -- 172.21.15.49:0/1939062164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed8850 msgr2=0x562574ee7010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:56:20.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.926+0000 7fa1277fe640 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed8850 0x562574ee7010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.926+0000 7fa1277fe640 1 -- 172.21.15.49:0/1939062164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 msgr2=0x562574f530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:20.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.926+0000 7fa1277fe640 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574f530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.926+0000 7fa1277fe640 1 -- 172.21.15.49:0/1939062164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562574eea5b0 con 0x562574ed9240 2024-03-18T00:56:20.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.926+0000 7fa12cd11640 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574f530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:56:20.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.930+0000 7fa1277fe640 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574ee7550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0c4b40 tx=0x7fa11c0bade0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b50d3a6e1ac531 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:20.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.931+0000 7fa1077fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:56:20.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.934+0000 7fa1077fe640 1 -- 172.21.15.49:0/1939062164 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c0d5040 con 0x562574ed9240 2024-03-18T00:56:20.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.934+0000 7fa131436740 1 -- 172.21.15.49:0/1939062164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa118010c70 con 0x562574ed9240 2024-03-18T00:56:20.934 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.934+0000 7fa1077fe640 1 -- 172.21.15.49:0/1939062164 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa11c0bbf20 con 0x562574ed9240 2024-03-18T00:56:20.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.939+0000 7fa1077fe640 1 -- 172.21.15.49:0/1939062164 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa11c0bc410 con 0x562574ed9240 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.940+0000 7fa131436740 10 client.26192 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.941+0000 7fa131436740 1 -- 172.21.15.49:0/1939062164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562574f545d0 con 0x562574ed9240 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.942+0000 7fa1077fe640 1 -- 172.21.15.49:0/1939062164 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa118010c70 con 0x562574ed9240 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.942+0000 7fa1077fe640 1 -- 172.21.15.49:0/1939062164 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa11c0bb790 con 0x562574ed9240 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.942+0000 7fa131436740 10 client.26192 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.942+0000 7fa131436740 10 client.26192 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.942+0000 7fa131436740 20 client.26192 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.942+0000 7fa131436740 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562574f54d60 0x562574f57180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.942+0000 7fa131436740 4 client.26192 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:56:20.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.942+0000 7fa131436740 1 -- 172.21.15.49:0/1939062164 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa118016e50 con 0x562574f54d60 2024-03-18T00:56:20.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.944+0000 7fa127fff640 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562574f54d60 0x562574f57180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:20.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.946+0000 7fa127fff640 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562574f54d60 0x562574f57180 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94f43dc30a3707b3 server_cookie=bf04d94032959f46 in_seq=0 out_seq=0 2024-03-18T00:56:20.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.947+0000 7fa1077fe640 10 client.26192 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:56:20.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.947+0000 7fa1077fe640 1 -- 172.21.15.49:0/1939062164 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa118016e50 con 0x562574f54d60 2024-03-18T00:56:20.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.947+0000 7fa1077fe640 10 client.26192 handle_command_reply: tid=0 2024-03-18T00:56:20.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.947+0000 7fa131436740 1 client.26192 shutdown 2024-03-18T00:56:20.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.955+0000 7fa131436740 1 -- 172.21.15.49:0/1939062164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 msgr2=0x562574ee7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.955+0000 7fa131436740 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574ee7550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa11c0c4b40 tx=0x7fa11c0bade0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.959+0000 7fa131436740 1 -- 172.21.15.49:0/1939062164 shutdown_connections 2024-03-18T00:56:20.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.959+0000 7fa131436740 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562574f54d60 0x562574f57180 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.959+0000 7fa131436740 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562574ed9240 0x562574ee7550 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.959+0000 7fa131436740 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562574ede3b0 0x562574f530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.959+0000 7fa131436740 1 --2- 172.21.15.49:0/1939062164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562574ed8850 0x562574ee7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.959+0000 7fa131436740 1 -- 172.21.15.49:0/1939062164 >> 172.21.15.49:0/1939062164 conn(0x562574ed3dc0 msgr2=0x562574ecee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.959+0000 7fa1077fe640 0 client.26192 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:56:20.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.962+0000 7fa131436740 1 -- 172.21.15.49:0/1939062164 shutdown_connections 2024-03-18T00:56:20.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.962+0000 7fa131436740 1 -- 172.21.15.49:0/1939062164 wait complete. 2024-03-18T00:56:20.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.964+0000 7fa131436740 20 client.26192 trim_cache size 0 max 16384 2024-03-18T00:56:20.965 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:56:20.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.967+0000 7fa12e774640 1 -- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1100a2860 msgr2=0x7fa1100a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.967+0000 7fa12e774640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1100a2860 0x7fa1100a4d20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa12000cd70 tx=0x7fa12000c860 comp rx=0 tx=0).stop 2024-03-18T00:56:20.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.967+0000 7fa12e774640 1 -- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa12810b150 msgr2=0x7fa1281a6f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:20.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.967+0000 7fa12e774640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa12810b150 0x7fa1281a6f20 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa11c02c830 tx=0x7fa11c0045c0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.971+0000 7fa12e774640 1 -- 172.21.15.49:0/3562533174 shutdown_connections 2024-03-18T00:56:20.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.971+0000 7fa12e774640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1100a2860 0x7fa1100a4d20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.971+0000 7fa12e774640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa12810db00 0x7fa1281adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.971+0000 7fa12e774640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa12810b150 0x7fa1281a6f20 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.971+0000 7fa12e774640 1 --2- 172.21.15.49:0/3562533174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1281087e0 0x7fa1281a69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:20.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.971+0000 7fa12e774640 1 -- 172.21.15.49:0/3562533174 >> 172.21.15.49:0/3562533174 conn(0x7fa12806e640 msgr2=0x7fa12810a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:20.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.972+0000 7fa12e774640 1 -- 172.21.15.49:0/3562533174 shutdown_connections 2024-03-18T00:56:20.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:20.972+0000 7fa12e774640 1 -- 172.21.15.49:0/3562533174 wait complete. 2024-03-18T00:56:21.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:56:21.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.413+0000 7f834b2bf640 1 -- 172.21.15.49:0/377795328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8344109890 msgr2=0x7f834410e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:21.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.413+0000 7f834b2bf640 1 --2- 172.21.15.49:0/377795328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8344109890 0x7f834410e150 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8340009900 tx=0x7f834002bd10 comp rx=0 tx=0).stop 2024-03-18T00:56:21.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.414+0000 7f834b2bf640 1 -- 172.21.15.49:0/377795328 shutdown_connections 2024-03-18T00:56:21.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.414+0000 7f834b2bf640 1 --2- 172.21.15.49:0/377795328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8344109890 0x7f834410e150 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.414+0000 7f834b2bf640 1 --2- 172.21.15.49:0/377795328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8344108ef0 0x7f8344109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.414+0000 7f834b2bf640 1 --2- 172.21.15.49:0/377795328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8344072bc0 0x7f8344072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.414+0000 7f834b2bf640 1 -- 172.21.15.49:0/377795328 >> 172.21.15.49:0/377795328 conn(0x7f834406e6c0 msgr2=0x7f8344070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:21.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.416+0000 7f834b2bf640 1 -- 172.21.15.49:0/377795328 shutdown_connections 2024-03-18T00:56:21.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.416+0000 7f834b2bf640 1 -- 172.21.15.49:0/377795328 wait complete. 2024-03-18T00:56:21.419 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.417+0000 7f834b2bf640 1 Processor -- start 2024-03-18T00:56:21.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.418+0000 7f834b2bf640 1 -- start start 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.419+0000 7f834b2bf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8344072bc0 0x7f8344116e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.419+0000 7f834b2bf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8344108ef0 0x7f8344117370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.419+0000 7f834b2bf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8344109890 0x7f83441b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.419+0000 7f834b2bf640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f834411c260 con 0x7f8344072bc0 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.419+0000 7f834b2bf640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f834411c0e0 con 0x7f8344108ef0 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.419+0000 7f834b2bf640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f834411c3e0 con 0x7f8344109890 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.420+0000 7f834985c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8344109890 0x7f83441b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.420+0000 7f834985c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8344109890 0x7f83441b6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44088/0 (socket says 172.21.15.49:44088) 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.420+0000 7f834985c640 1 -- 172.21.15.49:0/2203334578 learned_addr learned my addr 172.21.15.49:0/2203334578 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:21.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.422+0000 7f834885a640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8344108ef0 0x7f8344117370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.426+0000 7f834905b640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8344072bc0 0x7f8344116e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.426+0000 7f834985c640 1 -- 172.21.15.49:0/2203334578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8344108ef0 msgr2=0x7f8344117370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:21.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.426+0000 7f834985c640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8344108ef0 0x7f8344117370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.426+0000 7f834985c640 1 -- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8344072bc0 msgr2=0x7f8344116e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:21.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.426+0000 7f834985c640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8344072bc0 0x7f8344116e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.426+0000 7f834985c640 1 -- 172.21.15.49:0/2203334578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834411a4f0 con 0x7f8344109890 2024-03-18T00:56:21.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.430+0000 7f834985c640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8344109890 0x7f83441b6970 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8340004bc0 tx=0x7f834002ead0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:21.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.431+0000 7f833a7fc640 1 -- 172.21.15.49:0/2203334578 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340041020 con 0x7f8344109890 2024-03-18T00:56:21.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.431+0000 7f833a7fc640 1 -- 172.21.15.49:0/2203334578 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f834002c6b0 con 0x7f8344109890 2024-03-18T00:56:21.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.431+0000 7f833a7fc640 1 -- 172.21.15.49:0/2203334578 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340004ff0 con 0x7f8344109890 2024-03-18T00:56:21.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.436+0000 7f834b2bf640 1 -- 172.21.15.49:0/2203334578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83441b6eb0 con 0x7f8344109890 2024-03-18T00:56:21.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.437+0000 7f834b2bf640 1 -- 172.21.15.49:0/2203334578 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83441b7260 con 0x7f8344109890 2024-03-18T00:56:21.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.448+0000 7f833a7fc640 1 -- 172.21.15.49:0/2203334578 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8340044020 con 0x7f8344109890 2024-03-18T00:56:21.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.449+0000 7f833a7fc640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f83300a2800 0x7f83300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.449+0000 7f833a7fc640 1 -- 172.21.15.49:0/2203334578 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f83400efda0 con 0x7f8344109890 2024-03-18T00:56:21.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.452+0000 7f834905b640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f83300a2800 0x7f83300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.453 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.453+0000 7f834905b640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f83300a2800 0x7f83300a4cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8334006fd0 tx=0x7f8334008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:21.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.454+0000 7f834df89740 1 Processor -- start 2024-03-18T00:56:21.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.454+0000 7f834df89740 1 -- start start 2024-03-18T00:56:21.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.454+0000 7f834df89740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b3ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.454+0000 7f834df89740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b3ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.454+0000 7f834df89740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b42a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.454+0000 7f834df89740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a305a9f490 con 0x55a305b3b430 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.454+0000 7f834df89740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a305a47c90 con 0x55a305b405a0 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.454+0000 7f834df89740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a305a9b530 con 0x55a305b3aa40 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.455+0000 7f834985c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b42a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.455+0000 7f834985c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b42a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59868/0 (socket says 172.21.15.49:59868) 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.455+0000 7f834985c640 1 -- 172.21.15.49:0/2139163533 learned_addr learned my addr 172.21.15.49:0/2139163533 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.456+0000 7f834885a640 1 --2- 172.21.15.49:0/2139163533 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b3ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.456+0000 7f834885a640 1 -- 172.21.15.49:0/2139163533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 msgr2=0x55a305b3ae60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.456+0000 7f834885a640 1 --2- 172.21.15.49:0/2139163533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b3ae60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.456+0000 7f834885a640 1 -- 172.21.15.49:0/2139163533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 msgr2=0x55a305b42a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:21.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.456+0000 7f834885a640 1 --2- 172.21.15.49:0/2139163533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b42a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.456+0000 7f834885a640 1 -- 172.21.15.49:0/2139163533 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a305b02500 con 0x55a305b3b430 2024-03-18T00:56:21.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.459+0000 7f834885a640 1 --2- 172.21.15.49:0/2139163533 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b3ff30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f833c005fd0 tx=0x7f833c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67563d06002967fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:21.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.461+0000 7f832f7fe640 1 -- 172.21.15.49:0/2139163533 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c011440 con 0x55a305b3b430 2024-03-18T00:56:21.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.461+0000 7f832f7fe640 1 -- 172.21.15.49:0/2139163533 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c011ac0 con 0x55a305b3b430 2024-03-18T00:56:21.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.461+0000 7f832f7fe640 1 -- 172.21.15.49:0/2139163533 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c019ca0 con 0x55a305b3b430 2024-03-18T00:56:21.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.465+0000 7f834df89740 1 -- 172.21.15.49:0/2139163533 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 msgr2=0x55a305b3ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:21.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.465+0000 7f834df89740 1 --2- 172.21.15.49:0/2139163533 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b3ff30 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f833c005fd0 tx=0x7f833c00f730 comp rx=0 tx=0).stop 2024-03-18T00:56:21.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.466+0000 7f834df89740 1 -- 172.21.15.49:0/2139163533 shutdown_connections 2024-03-18T00:56:21.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.466+0000 7f834df89740 1 --2- 172.21.15.49:0/2139163533 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b42a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.466+0000 7f834df89740 1 --2- 172.21.15.49:0/2139163533 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b3ff30 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.466+0000 7f834df89740 1 --2- 172.21.15.49:0/2139163533 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b3ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.466+0000 7f834df89740 1 -- 172.21.15.49:0/2139163533 >> 172.21.15.49:0/2139163533 conn(0x55a305b35fb0 msgr2=0x55a305b363b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:21.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.472+0000 7f834df89740 1 -- 172.21.15.49:0/2139163533 shutdown_connections 2024-03-18T00:56:21.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.473+0000 7f834df89740 1 -- 172.21.15.49:0/2139163533 wait complete. 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.473+0000 7f834df89740 1 -- start start 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.473+0000 7f834df89740 1 Processor -- start 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834df89740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b44fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834df89740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b45500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834df89740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b4c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834df89740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a305b43550 con 0x55a305b3b430 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834df89740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a305a5b230 con 0x55a305b405a0 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834df89740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a305b436d0 con 0x55a305b3aa40 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834df89740 10 client.0 _ll_register_callbacks cb 0x55a305a97010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834885a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b45500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834885a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b45500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38806/0 (socket says 172.21.15.49:38806) 2024-03-18T00:56:21.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.474+0000 7f834885a640 1 -- 172.21.15.49:0/2295917622 learned_addr learned my addr 172.21.15.49:0/2295917622 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:21.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.475+0000 7f834885a640 1 -- 172.21.15.49:0/2295917622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 msgr2=0x55a305b44fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:21.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.475+0000 7f834885a640 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b44fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.475+0000 7f834885a640 1 -- 172.21.15.49:0/2295917622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 msgr2=0x55a305b4c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:21.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.475+0000 7f834885a640 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b4c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.475+0000 7f834885a640 1 -- 172.21.15.49:0/2295917622 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a305a6a100 con 0x55a305b3b430 2024-03-18T00:56:21.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.476+0000 7f834885a640 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b45500 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f833c000f80 tx=0x7f833c012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2c51a8669934cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:21.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.476+0000 7f832cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:21.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.477+0000 7f832cff9640 1 -- 172.21.15.49:0/2295917622 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c004c50 con 0x55a305b3b430 2024-03-18T00:56:21.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.477+0000 7f832cff9640 1 -- 172.21.15.49:0/2295917622 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f833c004df0 con 0x55a305b3b430 2024-03-18T00:56:21.477 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.477+0000 7f834df89740 1 -- 172.21.15.49:0/2295917622 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f833c011ac0 con 0x55a305b3b430 2024-03-18T00:56:21.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.478+0000 7f832cff9640 1 -- 172.21.15.49:0/2295917622 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f833c0199e0 con 0x55a305b3b430 2024-03-18T00:56:21.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.478+0000 7f834df89740 10 client.16911 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:21.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.478+0000 7f834df89740 1 -- 172.21.15.49:0/2295917622 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a3057b33f0 con 0x55a305b3b430 2024-03-18T00:56:21.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.480+0000 7f832cff9640 1 -- 172.21.15.49:0/2295917622 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c011ac0 con 0x55a305b3b430 2024-03-18T00:56:21.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.480+0000 7f832cff9640 1 -- 172.21.15.49:0/2295917622 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f833c0052f0 con 0x55a305b3b430 2024-03-18T00:56:21.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.480+0000 7f834df89740 10 client.16911 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:21.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.480+0000 7f834df89740 10 client.16911 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:56:21.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.480+0000 7f834df89740 20 client.16911 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:21.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.480+0000 7f834df89740 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a305b50ae0 0x55a305b52ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.480+0000 7f834df89740 4 client.16911 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:21.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.480+0000 7f834df89740 1 -- 172.21.15.49:0/2295917622 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a305aac9d0 con 0x55a305b50ae0 2024-03-18T00:56:21.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.481+0000 7f834905b640 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a305b50ae0 0x55a305b52ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.486+0000 7f834905b640 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a305b50ae0 0x55a305b52ed0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2c27b4e09509dafc server_cookie=68921cc3fef6502a in_seq=0 out_seq=0 2024-03-18T00:56:21.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.486+0000 7f832cff9640 10 client.16911 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:56:21.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.489+0000 7f832cff9640 1 -- 172.21.15.49:0/2295917622 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a305aac9d0 con 0x55a305b50ae0 2024-03-18T00:56:21.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.489+0000 7f832cff9640 10 client.16911 handle_command_reply: tid=0 2024-03-18T00:56:21.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.489+0000 7f834df89740 1 client.16911 shutdown 2024-03-18T00:56:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 -- 172.21.15.49:0/2295917622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 msgr2=0x55a305b45500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b45500 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f833c000f80 tx=0x7f833c012eb0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 -- 172.21.15.49:0/2295917622 shutdown_connections 2024-03-18T00:56:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a305b50ae0 0x55a305b52ed0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b4c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3b430 0x55a305b45500 secure :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f833c000f80 tx=0x7f833c012eb0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 --2- 172.21.15.49:0/2295917622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b44fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 -- 172.21.15.49:0/2295917622 >> 172.21.15.49:0/2295917622 conn(0x55a305b35fb0 msgr2=0x55a305b35140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f832cff9640 0 client.16911 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:56:21.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 -- 172.21.15.49:0/2295917622 shutdown_connections 2024-03-18T00:56:21.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.490+0000 7f834df89740 1 -- 172.21.15.49:0/2295917622 wait complete. 2024-03-18T00:56:21.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.491+0000 7f834df89740 20 client.16911 trim_cache size 0 max 16384 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.535+0000 7f834df89740 1 Processor -- start 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834df89740 1 -- start start 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834df89740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b44fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834df89740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b45500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834df89740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b4c500 0x55a305b3c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834df89740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f83441259c0 con 0x55a305b4c500 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834df89740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8344127500 con 0x55a305b405a0 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834df89740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8344126090 con 0x55a305b3aa40 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834905b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b44fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834905b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b44fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44096/0 (socket says 172.21.15.49:44096) 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834905b640 1 -- 172.21.15.49:0/1733767557 learned_addr learned my addr 172.21.15.49:0/1733767557 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:21.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.536+0000 7f834885a640 1 --2- 172.21.15.49:0/1733767557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b45500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.537+0000 7f834885a640 1 -- 172.21.15.49:0/1733767557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 msgr2=0x55a305b44fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:21.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.537+0000 7f834885a640 1 --2- 172.21.15.49:0/1733767557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b44fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.537+0000 7f834885a640 1 -- 172.21.15.49:0/1733767557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b4c500 msgr2=0x55a305b3c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:21.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.537+0000 7f834885a640 1 --2- 172.21.15.49:0/1733767557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b4c500 0x55a305b3c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.537+0000 7f834885a640 1 -- 172.21.15.49:0/1733767557 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a30587d7d0 con 0x55a305b405a0 2024-03-18T00:56:21.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.537+0000 7f834885a640 1 --2- 172.21.15.49:0/1733767557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b45500 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f833c021000 tx=0x7f833c009a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7cbb0587c1e5f980 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:21.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.547+0000 7f832effd640 1 -- 172.21.15.49:0/1733767557 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833c019770 con 0x55a305b405a0 2024-03-18T00:56:21.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.547+0000 7f832effd640 1 -- 172.21.15.49:0/1733767557 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c028070 con 0x55a305b405a0 2024-03-18T00:56:21.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.549+0000 7f834df89740 1 -- 172.21.15.49:0/1733767557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 msgr2=0x55a305b45500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:21.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.549+0000 7f834df89740 1 --2- 172.21.15.49:0/1733767557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b45500 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f833c021000 tx=0x7f833c009a40 comp rx=0 tx=0).stop 2024-03-18T00:56:21.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.552+0000 7f834df89740 1 -- 172.21.15.49:0/1733767557 shutdown_connections 2024-03-18T00:56:21.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.552+0000 7f834df89740 1 --2- 172.21.15.49:0/1733767557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b45500 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.552+0000 7f834df89740 1 --2- 172.21.15.49:0/1733767557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b4c500 0x55a305b3c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.552+0000 7f834df89740 1 --2- 172.21.15.49:0/1733767557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b3aa40 0x55a305b44fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.553+0000 7f834df89740 1 -- 172.21.15.49:0/1733767557 >> 172.21.15.49:0/1733767557 conn(0x55a305b35fb0 msgr2=0x55a305b30fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:21.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.553+0000 7f834df89740 1 -- 172.21.15.49:0/1733767557 shutdown_connections 2024-03-18T00:56:21.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.553+0000 7f834df89740 1 -- 172.21.15.49:0/1733767557 wait complete. 2024-03-18T00:56:21.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.554+0000 7f834df89740 1 -- start start 2024-03-18T00:56:21.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.554+0000 7f834df89740 1 Processor -- start 2024-03-18T00:56:21.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.555+0000 7f834df89740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3aa40 0x55a305b49490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.555+0000 7f834df89740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b499d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.555+0000 7f834df89740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b4c500 0x55a305b4a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.555+0000 7f834df89740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8344126e30 con 0x55a305b3aa40 2024-03-18T00:56:21.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.555+0000 7f834df89740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8344126760 con 0x55a305b405a0 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.555+0000 7f834df89740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f83441239c0 con 0x55a305b4c500 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.555+0000 7f834df89740 10 client.0 _ll_register_callbacks cb 0x7f833c009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.557+0000 7f834985c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b4c500 0x55a305b4a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.558+0000 7f834885a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.558+0000 7f834905b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3aa40 0x55a305b49490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.558+0000 7f834905b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3aa40 0x55a305b49490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38836/0 (socket says 172.21.15.49:38836) 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.558+0000 7f834905b640 1 -- 172.21.15.49:0/2639350565 learned_addr learned my addr 172.21.15.49:0/2639350565 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.559+0000 7f834905b640 1 -- 172.21.15.49:0/2639350565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b4c500 msgr2=0x55a305b4a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.559+0000 7f834905b640 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b4c500 0x55a305b4a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.559+0000 7f834905b640 1 -- 172.21.15.49:0/2639350565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 msgr2=0x55a305b499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.559+0000 7f834905b640 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b499d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.559+0000 7f834905b640 1 -- 172.21.15.49:0/2639350565 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a305b4aef0 con 0x55a305b3aa40 2024-03-18T00:56:21.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.559+0000 7f834905b640 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3aa40 0x55a305b49490 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f833400dc00 tx=0x7f833400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a4ceb5df6d9dd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:21.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.561+0000 7f832cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:21.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.561+0000 7f832cff9640 1 -- 172.21.15.49:0/2639350565 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83340160d0 con 0x55a305b3aa40 2024-03-18T00:56:21.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.561+0000 7f832cff9640 1 -- 172.21.15.49:0/2639350565 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f83340169d0 con 0x55a305b3aa40 2024-03-18T00:56:21.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.561+0000 7f834df89740 1 -- 172.21.15.49:0/2639350565 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f833c028070 con 0x55a305b3aa40 2024-03-18T00:56:21.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.562+0000 7f832cff9640 1 -- 172.21.15.49:0/2639350565 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8334016270 con 0x55a305b3aa40 2024-03-18T00:56:21.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.564+0000 7f834df89740 10 client.16914 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:21.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.564+0000 7f834df89740 1 -- 172.21.15.49:0/2639350565 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a305b4c290 con 0x55a305b3aa40 2024-03-18T00:56:21.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.568+0000 7f832cff9640 1 -- 172.21.15.49:0/2639350565 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f833c028070 con 0x55a305b3aa40 2024-03-18T00:56:21.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.568+0000 7f832cff9640 1 -- 172.21.15.49:0/2639350565 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8334013070 con 0x55a305b3aa40 2024-03-18T00:56:21.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.568+0000 7f834df89740 10 client.16914 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:21.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.568+0000 7f834df89740 10 client.16914 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:56:21.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.568+0000 7f834df89740 20 client.16914 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:21.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.568+0000 7f834df89740 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a305bb7560 0x55a305bb9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:21.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.568+0000 7f834df89740 4 client.16914 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:56:21.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.568+0000 7f834df89740 1 -- 172.21.15.49:0/2639350565 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a305aa6d20 con 0x55a305bb7560 2024-03-18T00:56:21.575 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.573+0000 7f834885a640 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a305bb7560 0x55a305bb9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:21.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.586+0000 7f834885a640 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a305bb7560 0x55a305bb9980 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a681805a662021a5 server_cookie=6018e5ef84ddb08c in_seq=0 out_seq=0 2024-03-18T00:56:21.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.586+0000 7f832cff9640 10 client.16914 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:56:21.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.589+0000 7f832cff9640 1 -- 172.21.15.49:0/2639350565 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a305aa6d20 con 0x55a305bb7560 2024-03-18T00:56:21.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.589+0000 7f832cff9640 10 client.16914 handle_command_reply: tid=0 2024-03-18T00:56:21.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.590+0000 7f834df89740 1 client.16914 shutdown 2024-03-18T00:56:21.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.597+0000 7f834df89740 1 -- 172.21.15.49:0/2639350565 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3aa40 msgr2=0x55a305b49490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:21.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.597+0000 7f834df89740 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3aa40 0x55a305b49490 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f833400dc00 tx=0x7f833400dc30 comp rx=0 tx=0).stop 2024-03-18T00:56:21.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.602+0000 7f834df89740 1 -- 172.21.15.49:0/2639350565 shutdown_connections 2024-03-18T00:56:21.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.602+0000 7f834df89740 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a305bb7560 0x55a305bb9980 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.602+0000 7f834df89740 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a305b405a0 0x55a305b499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.602+0000 7f834df89740 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a305b4c500 0x55a305b4a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.602+0000 7f834df89740 1 --2- 172.21.15.49:0/2639350565 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a305b3aa40 0x55a305b49490 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.602+0000 7f834df89740 1 -- 172.21.15.49:0/2639350565 >> 172.21.15.49:0/2639350565 conn(0x55a305b35fb0 msgr2=0x55a305b44fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:21.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.603+0000 7f832cff9640 0 client.16914 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:56:21.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.607+0000 7f834df89740 1 -- 172.21.15.49:0/2639350565 shutdown_connections 2024-03-18T00:56:21.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.608+0000 7f834df89740 1 -- 172.21.15.49:0/2639350565 wait complete. 2024-03-18T00:56:21.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.608+0000 7f834df89740 20 client.16914 trim_cache size 0 max 16384 2024-03-18T00:56:21.612 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:56:21.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.616+0000 7f834b2bf640 1 -- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f83300a2800 msgr2=0x7f83300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:21.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.616+0000 7f834b2bf640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f83300a2800 0x7f83300a4cc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8334006fd0 tx=0x7f8334008040 comp rx=0 tx=0).stop 2024-03-18T00:56:21.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.616+0000 7f834b2bf640 1 -- 172.21.15.49:0/2203334578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8344109890 msgr2=0x7f83441b6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:21.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.616+0000 7f834b2bf640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8344109890 0x7f83441b6970 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8340004bc0 tx=0x7f834002ead0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.621+0000 7f834b2bf640 1 -- 172.21.15.49:0/2203334578 shutdown_connections 2024-03-18T00:56:21.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.621+0000 7f834b2bf640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f83300a2800 0x7f83300a4cc0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.621+0000 7f834b2bf640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8344108ef0 0x7f8344117370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.621+0000 7f834b2bf640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8344109890 0x7f83441b6970 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.621+0000 7f834b2bf640 1 --2- 172.21.15.49:0/2203334578 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8344072bc0 0x7f8344116e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:21.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.621+0000 7f834b2bf640 1 -- 172.21.15.49:0/2203334578 >> 172.21.15.49:0/2203334578 conn(0x7f834406e6c0 msgr2=0x7f834410c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:21.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.628+0000 7f834b2bf640 1 -- 172.21.15.49:0/2203334578 shutdown_connections 2024-03-18T00:56:21.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:21.629+0000 7f834b2bf640 1 -- 172.21.15.49:0/2203334578 wait complete. 2024-03-18T00:56:21.678 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:56:21.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:21 smithi049 ceph-mon[32130]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 934 MiB/s wr, 2.16k op/s 2024-03-18T00:56:21.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:21 smithi107 ceph-mon[36270]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 934 MiB/s wr, 2.16k op/s 2024-03-18T00:56:21.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:21 smithi106 ceph-mon[35899]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 934 MiB/s wr, 2.16k op/s 2024-03-18T00:56:22.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.176+0000 7f2434bff640 1 -- 172.21.15.49:0/3486460695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2430072bf0 msgr2=0x7f2430073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:22.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.176+0000 7f2434bff640 1 --2- 172.21.15.49:0/3486460695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2430072bf0 0x7f2430073010 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2420009860 tx=0x7f242002bb90 comp rx=0 tx=0).stop 2024-03-18T00:56:22.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.182+0000 7f2434bff640 1 -- 172.21.15.49:0/3486460695 shutdown_connections 2024-03-18T00:56:22.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.182+0000 7f2434bff640 1 --2- 172.21.15.49:0/3486460695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2430108eb0 0x7f2430109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.182+0000 7f2434bff640 1 --2- 172.21.15.49:0/3486460695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2430109870 0x7f243010e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.182+0000 7f2434bff640 1 --2- 172.21.15.49:0/3486460695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2430072bf0 0x7f2430073010 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.182+0000 7f2434bff640 1 -- 172.21.15.49:0/3486460695 >> 172.21.15.49:0/3486460695 conn(0x7f243006e6c0 msgr2=0x7f2430070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:22.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.183+0000 7f2434bff640 1 -- 172.21.15.49:0/3486460695 shutdown_connections 2024-03-18T00:56:22.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.183+0000 7f2434bff640 1 -- 172.21.15.49:0/3486460695 wait complete. 2024-03-18T00:56:22.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.185+0000 7f2434bff640 1 Processor -- start 2024-03-18T00:56:22.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.186+0000 7f2434bff640 1 -- start start 2024-03-18T00:56:22.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.186+0000 7f2434bff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2430072bf0 0x7f2430114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.187+0000 7f2434bff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2430108eb0 0x7f2430115430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.187+0000 7f242f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2430072bf0 0x7f2430114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.187+0000 7f242f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2430072bf0 0x7f2430114ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38856/0 (socket says 172.21.15.49:38856) 2024-03-18T00:56:22.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.187+0000 7f242f7fe640 1 -- 172.21.15.49:0/546714618 learned_addr learned my addr 172.21.15.49:0/546714618 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:22.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.187+0000 7f2434bff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2430109870 0x7f24301b6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.187+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f243011c180 con 0x7f2430072bf0 2024-03-18T00:56:22.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.187+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f243011c000 con 0x7f2430109870 2024-03-18T00:56:22.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.187+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f243011c300 con 0x7f2430108eb0 2024-03-18T00:56:22.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.188+0000 7f2426ffd640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2430108eb0 0x7f2430115430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.189+0000 7f242ffff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2430109870 0x7f24301b6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.189+0000 7f242f7fe640 1 -- 172.21.15.49:0/546714618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2430108eb0 msgr2=0x7f2430115430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:22.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.189+0000 7f242f7fe640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2430108eb0 0x7f2430115430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.189+0000 7f242f7fe640 1 -- 172.21.15.49:0/546714618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2430109870 msgr2=0x7f24301b6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:22.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.189+0000 7f242f7fe640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2430109870 0x7f24301b6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.189+0000 7f242f7fe640 1 -- 172.21.15.49:0/546714618 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24301b6db0 con 0x7f2430072bf0 2024-03-18T00:56:22.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.190+0000 7f242f7fe640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2430072bf0 0x7f2430114ef0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2420009780 tx=0x7f24200324d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:22.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.191+0000 7f242d7fa640 1 -- 172.21.15.49:0/546714618 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420039040 con 0x7f2430072bf0 2024-03-18T00:56:22.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.191+0000 7f242d7fa640 1 -- 172.21.15.49:0/546714618 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2420064070 con 0x7f2430072bf0 2024-03-18T00:56:22.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.191+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2430073a10 con 0x7f2430072bf0 2024-03-18T00:56:22.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.192+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2430073e40 con 0x7f2430072bf0 2024-03-18T00:56:22.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.192+0000 7f242d7fa640 1 -- 172.21.15.49:0/546714618 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2420004920 con 0x7f2430072bf0 2024-03-18T00:56:22.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.199+0000 7f2438a52740 1 Processor -- start 2024-03-18T00:56:22.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.199+0000 7f2438a52740 1 -- start start 2024-03-18T00:56:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.199+0000 7f2438a52740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc77e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.199+0000 7f242d7fa640 1 -- 172.21.15.49:0/546714618 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f242002c530 con 0x7f2430072bf0 2024-03-18T00:56:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.199+0000 7f2438a52740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc78430 0x55e36bc7cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.199+0000 7f2438a52740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc7fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.199+0000 7f2438a52740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e36bbdc490 con 0x55e36bc77a40 2024-03-18T00:56:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.199+0000 7f2438a52740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e36bb84c90 con 0x55e36bc7d5a0 2024-03-18T00:56:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.199+0000 7f2438a52740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e36bbd8530 con 0x55e36bc78430 2024-03-18T00:56:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.200+0000 7f242ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc77e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.200+0000 7f242ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc77e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38872/0 (socket says 172.21.15.49:38872) 2024-03-18T00:56:22.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.200+0000 7f242ffff640 1 -- 172.21.15.49:0/1018881880 learned_addr learned my addr 172.21.15.49:0/1018881880 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:22.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.201+0000 7f242d7fa640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f241c0a2a00 0x7f241c0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.201+0000 7f242ffff640 1 -- 172.21.15.49:0/1018881880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc78430 msgr2=0x55e36bc7cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.201+0000 7f242ffff640 1 --2- 172.21.15.49:0/1018881880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc78430 0x55e36bc7cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.201+0000 7f242ffff640 1 -- 172.21.15.49:0/1018881880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 msgr2=0x55e36bc7fa30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.201+0000 7f242ffff640 1 --2- 172.21.15.49:0/1018881880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc7fa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.201+0000 7f242ffff640 1 -- 172.21.15.49:0/1018881880 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e36bc3f500 con 0x55e36bc77a40 2024-03-18T00:56:22.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.202+0000 7f242d7fa640 1 -- 172.21.15.49:0/546714618 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2420112430 con 0x7f2430072bf0 2024-03-18T00:56:22.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.204+0000 7f242ffff640 1 --2- 172.21.15.49:0/1018881880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc77e60 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f242800f480 tx=0x7f242800f940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edaf940c0d86b569 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:22.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.208+0000 7f242ffff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f241c0a2a00 0x7f241c0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.210+0000 7f2425ffb640 1 -- 172.21.15.49:0/1018881880 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242801e070 con 0x55e36bc77a40 2024-03-18T00:56:22.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.210+0000 7f2425ffb640 1 -- 172.21.15.49:0/1018881880 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242800cd40 con 0x55e36bc77a40 2024-03-18T00:56:22.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.211+0000 7f2425ffb640 1 -- 172.21.15.49:0/1018881880 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242800d640 con 0x55e36bc77a40 2024-03-18T00:56:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.213+0000 7f242ffff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f241c0a2a00 0x7f241c0a4ec0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f242801b000 tx=0x7f2428018e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.213+0000 7f2438a52740 1 -- 172.21.15.49:0/1018881880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 msgr2=0x55e36bc77e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.213+0000 7f2438a52740 1 --2- 172.21.15.49:0/1018881880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc77e60 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f242800f480 tx=0x7f242800f940 comp rx=0 tx=0).stop 2024-03-18T00:56:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.213+0000 7f2438a52740 1 -- 172.21.15.49:0/1018881880 shutdown_connections 2024-03-18T00:56:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.213+0000 7f2438a52740 1 --2- 172.21.15.49:0/1018881880 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc7fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.214+0000 7f2438a52740 1 --2- 172.21.15.49:0/1018881880 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc78430 0x55e36bc7cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.214+0000 7f2438a52740 1 --2- 172.21.15.49:0/1018881880 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc77e60 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.214+0000 7f2438a52740 1 -- 172.21.15.49:0/1018881880 >> 172.21.15.49:0/1018881880 conn(0x55e36bc72fb0 msgr2=0x55e36bc733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:22.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.220+0000 7f2438a52740 1 -- 172.21.15.49:0/1018881880 shutdown_connections 2024-03-18T00:56:22.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.220+0000 7f2438a52740 1 -- 172.21.15.49:0/1018881880 wait complete. 2024-03-18T00:56:22.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.221+0000 7f2438a52740 1 -- start start 2024-03-18T00:56:22.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.221+0000 7f2438a52740 1 Processor -- start 2024-03-18T00:56:22.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.222+0000 7f2438a52740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc81e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.222+0000 7f2438a52740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 0x55e36bc823a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.222+0000 7f2438a52740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.222+0000 7f2438a52740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e36bc805a0 con 0x55e36bc78430 2024-03-18T00:56:22.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.222+0000 7f2438a52740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e36bb98230 con 0x55e36bc7d5a0 2024-03-18T00:56:22.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.222+0000 7f2438a52740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e36bc80720 con 0x55e36bc77a40 2024-03-18T00:56:22.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.222+0000 7f2438a52740 10 client.0 _ll_register_callbacks cb 0x55e36bbd4010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:22.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.226+0000 7f242ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.228+0000 7f242ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59944/0 (socket says 172.21.15.49:59944) 2024-03-18T00:56:22.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.228+0000 7f242ffff640 1 -- 172.21.15.49:0/988864390 learned_addr learned my addr 172.21.15.49:0/988864390 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:22.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.228+0000 7f2426ffd640 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 0x55e36bc823a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.228+0000 7f242f7fe640 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc81e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.230+0000 7f2426ffd640 1 -- 172.21.15.49:0/988864390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 msgr2=0x55e36bc81e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:22.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.230+0000 7f2426ffd640 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc81e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.230+0000 7f2426ffd640 1 -- 172.21.15.49:0/988864390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 msgr2=0x55e36bc893e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:22.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.230+0000 7f2426ffd640 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.230+0000 7f2426ffd640 1 -- 172.21.15.49:0/988864390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e36bba7100 con 0x55e36bc78430 2024-03-18T00:56:22.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.232+0000 7f2426ffd640 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 0x55e36bc823a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2430116400 tx=0x7f2418011b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1881365d36fd06 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:22.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.233+0000 7f240effd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:22.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.233+0000 7f240effd640 1 -- 172.21.15.49:0/988864390 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241800fcd0 con 0x55e36bc78430 2024-03-18T00:56:22.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.235+0000 7f240effd640 1 -- 172.21.15.49:0/988864390 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2418004030 con 0x55e36bc78430 2024-03-18T00:56:22.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.235+0000 7f2438a52740 1 -- 172.21.15.49:0/988864390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f242800cd40 con 0x55e36bc78430 2024-03-18T00:56:22.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.236+0000 7f240effd640 1 -- 172.21.15.49:0/988864390 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f241800fe70 con 0x55e36bc78430 2024-03-18T00:56:22.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.239+0000 7f2438a52740 10 client.16926 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:22.240 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.239+0000 7f2438a52740 1 -- 172.21.15.49:0/988864390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e36b8f03f0 con 0x55e36bc78430 2024-03-18T00:56:22.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.241+0000 7f240effd640 1 -- 172.21.15.49:0/988864390 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242800cd40 con 0x55e36bc78430 2024-03-18T00:56:22.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.241+0000 7f240effd640 1 -- 172.21.15.49:0/988864390 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2418019510 con 0x55e36bc78430 2024-03-18T00:56:22.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.243+0000 7f2438a52740 10 client.16926 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:22.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.243+0000 7f2438a52740 10 client.16926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:22.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.243+0000 7f2438a52740 20 client.16926 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:22.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.243+0000 7f2438a52740 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e36bc8e160 0x55e36bc90590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.243+0000 7f2438a52740 4 client.16926 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:22.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.243+0000 7f2438a52740 1 -- 172.21.15.49:0/988864390 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e36bbe99d0 con 0x55e36bc8e160 2024-03-18T00:56:22.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.248+0000 7f242f7fe640 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e36bc8e160 0x55e36bc90590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.267+0000 7f242f7fe640 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e36bc8e160 0x55e36bc90590 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=192b6a9b7e1abeeb server_cookie=4ac191fba9a9387a in_seq=0 out_seq=0 2024-03-18T00:56:22.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.267+0000 7f240effd640 10 client.16926 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:22.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.272+0000 7f240effd640 1 -- 172.21.15.49:0/988864390 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e36bbe99d0 con 0x55e36bc8e160 2024-03-18T00:56:22.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.272+0000 7f240effd640 10 client.16926 handle_command_reply: tid=0 2024-03-18T00:56:22.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.272+0000 7f2438a52740 1 client.16926 shutdown 2024-03-18T00:56:22.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.289+0000 7f2438a52740 1 -- 172.21.15.49:0/988864390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 msgr2=0x55e36bc823a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:22.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.289+0000 7f2438a52740 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 0x55e36bc823a0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f2430116400 tx=0x7f2418011b50 comp rx=0 tx=0).stop 2024-03-18T00:56:22.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.290+0000 7f2438a52740 1 -- 172.21.15.49:0/988864390 shutdown_connections 2024-03-18T00:56:22.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.290+0000 7f2438a52740 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e36bc8e160 0x55e36bc90590 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.290+0000 7f2438a52740 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.290+0000 7f2438a52740 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 0x55e36bc823a0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.290+0000 7f2438a52740 1 --2- 172.21.15.49:0/988864390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc81e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.290+0000 7f2438a52740 1 -- 172.21.15.49:0/988864390 >> 172.21.15.49:0/988864390 conn(0x55e36bc72fb0 msgr2=0x55e36bc72140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:22.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.291+0000 7f240effd640 0 client.16926 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:22.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.291+0000 7f2438a52740 1 -- 172.21.15.49:0/988864390 shutdown_connections 2024-03-18T00:56:22.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.291+0000 7f2438a52740 1 -- 172.21.15.49:0/988864390 wait complete. 2024-03-18T00:56:22.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.296+0000 7f2438a52740 20 client.16926 trim_cache size 0 max 16384 2024-03-18T00:56:22.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.388+0000 7f2438a52740 1 Processor -- start 2024-03-18T00:56:22.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.391+0000 7f2438a52740 1 -- start start 2024-03-18T00:56:22.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.391+0000 7f2438a52740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc81e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.391+0000 7f2438a52740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc78430 0x55e36bc823a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.391+0000 7f2438a52740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.391+0000 7f2438a52740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f243007ccf0 con 0x55e36bc77a40 2024-03-18T00:56:22.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.391+0000 7f2438a52740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f243007e830 con 0x55e36bc7d5a0 2024-03-18T00:56:22.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.391+0000 7f2438a52740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f243007d3c0 con 0x55e36bc78430 2024-03-18T00:56:22.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.392+0000 7f242f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc81e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.392+0000 7f242f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc81e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38886/0 (socket says 172.21.15.49:38886) 2024-03-18T00:56:22.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.392+0000 7f242f7fe640 1 -- 172.21.15.49:0/223643083 learned_addr learned my addr 172.21.15.49:0/223643083 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:22.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.393+0000 7f242ffff640 1 --2- 172.21.15.49:0/223643083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.393+0000 7f2426ffd640 1 --2- 172.21.15.49:0/223643083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc78430 0x55e36bc823a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.395+0000 7f242ffff640 1 -- 172.21.15.49:0/223643083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc78430 msgr2=0x55e36bc823a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:22.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.395+0000 7f242ffff640 1 --2- 172.21.15.49:0/223643083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc78430 0x55e36bc823a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.395+0000 7f242ffff640 1 -- 172.21.15.49:0/223643083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 msgr2=0x55e36bc81e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:22.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.395+0000 7f242ffff640 1 --2- 172.21.15.49:0/223643083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc81e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.395+0000 7f242ffff640 1 -- 172.21.15.49:0/223643083 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e36b9ba7d0 con 0x55e36bc7d5a0 2024-03-18T00:56:22.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.400+0000 7f242ffff640 1 --2- 172.21.15.49:0/223643083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f242800fea0 tx=0x7f2428027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e527a98d895122a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:22.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.404+0000 7f24257fa640 1 -- 172.21.15.49:0/223643083 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242801e030 con 0x55e36bc7d5a0 2024-03-18T00:56:22.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.404+0000 7f24257fa640 1 -- 172.21.15.49:0/223643083 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242802a070 con 0x55e36bc7d5a0 2024-03-18T00:56:22.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.404+0000 7f24257fa640 1 -- 172.21.15.49:0/223643083 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f242800f2d0 con 0x55e36bc7d5a0 2024-03-18T00:56:22.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.408+0000 7f2438a52740 1 -- 172.21.15.49:0/223643083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 msgr2=0x55e36bc893a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:22.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.408+0000 7f2438a52740 1 --2- 172.21.15.49:0/223643083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f242800fea0 tx=0x7f2428027000 comp rx=0 tx=0).stop 2024-03-18T00:56:22.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.411+0000 7f2438a52740 1 -- 172.21.15.49:0/223643083 shutdown_connections 2024-03-18T00:56:22.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.411+0000 7f2438a52740 1 --2- 172.21.15.49:0/223643083 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bc893a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.411+0000 7f2438a52740 1 --2- 172.21.15.49:0/223643083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc78430 0x55e36bc823a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.411+0000 7f2438a52740 1 --2- 172.21.15.49:0/223643083 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc77a40 0x55e36bc81e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.411+0000 7f2438a52740 1 -- 172.21.15.49:0/223643083 >> 172.21.15.49:0/223643083 conn(0x55e36bc72fb0 msgr2=0x55e36bc73390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:22.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.419+0000 7f2438a52740 1 -- 172.21.15.49:0/223643083 shutdown_connections 2024-03-18T00:56:22.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.422+0000 7f2438a52740 1 -- 172.21.15.49:0/223643083 wait complete. 2024-03-18T00:56:22.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.423+0000 7f2438a52740 1 -- start start 2024-03-18T00:56:22.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.424+0000 7f2438a52740 1 Processor -- start 2024-03-18T00:56:22.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.424+0000 7f2438a52740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc861e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.424+0000 7f2438a52740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 0x55e36bc86720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.425+0000 7f2438a52740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bcf24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.425+0000 7f2438a52740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f243007e160 con 0x55e36bc78430 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.425+0000 7f2438a52740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f243007da90 con 0x55e36bc7d5a0 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.425+0000 7f2438a52740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f243007acd0 con 0x55e36bc77a40 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.425+0000 7f2438a52740 10 client.0 _ll_register_callbacks cb 0x7f24180097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.426+0000 7f242f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc861e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.426+0000 7f242f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc861e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44150/0 (socket says 172.21.15.49:44150) 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.426+0000 7f242f7fe640 1 -- 172.21.15.49:0/2643235279 learned_addr learned my addr 172.21.15.49:0/2643235279 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:22.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.426+0000 7f242ffff640 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bcf24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.429+0000 7f242f7fe640 1 -- 172.21.15.49:0/2643235279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 msgr2=0x55e36bcf24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:22.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.429+0000 7f242f7fe640 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bcf24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.429+0000 7f242f7fe640 1 -- 172.21.15.49:0/2643235279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 msgr2=0x55e36bc86720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:22.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.429+0000 7f242f7fe640 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 0x55e36bc86720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.429+0000 7f242f7fe640 1 -- 172.21.15.49:0/2643235279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e36bc86dc0 con 0x55e36bc77a40 2024-03-18T00:56:22.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.435+0000 7f242f7fe640 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc861e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f24200dece0 tx=0x7f24200debd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5675780a1a63302c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:22.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.437+0000 7f240effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:56:22.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.438+0000 7f240effd640 1 -- 172.21.15.49:0/2643235279 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24200ded50 con 0x55e36bc77a40 2024-03-18T00:56:22.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.438+0000 7f240effd640 1 -- 172.21.15.49:0/2643235279 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2420005150 con 0x55e36bc77a40 2024-03-18T00:56:22.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.438+0000 7f2438a52740 1 -- 172.21.15.49:0/2643235279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f242802a070 con 0x55e36bc77a40 2024-03-18T00:56:22.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.442+0000 7f240effd640 1 -- 172.21.15.49:0/2643235279 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f24200e2ea0 con 0x55e36bc77a40 2024-03-18T00:56:22.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.443+0000 7f2438a52740 10 client.26305 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:22.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.443+0000 7f2438a52740 1 -- 172.21.15.49:0/2643235279 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e36bcf3ba0 con 0x55e36bc77a40 2024-03-18T00:56:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.447+0000 7f240effd640 1 -- 172.21.15.49:0/2643235279 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f242802a070 con 0x55e36bc77a40 2024-03-18T00:56:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.447+0000 7f240effd640 1 -- 172.21.15.49:0/2643235279 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f24200defb0 con 0x55e36bc77a40 2024-03-18T00:56:22.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.448+0000 7f2438a52740 10 client.26305 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.448+0000 7f2438a52740 10 client.26305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.448+0000 7f2438a52740 20 client.26305 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.448+0000 7f2438a52740 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e36bcf45f0 0x55e36bcf6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.448+0000 7f2438a52740 4 client.26305 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:56:22.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.448+0000 7f2438a52740 1 -- 172.21.15.49:0/2643235279 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e36bbe3d20 con 0x55e36bcf45f0 2024-03-18T00:56:22.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.451+0000 7f2426ffd640 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e36bcf45f0 0x55e36bcf6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:22.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.468+0000 7f2426ffd640 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e36bcf45f0 0x55e36bcf6a10 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fc594d1aa469813 server_cookie=d0de050afcc57cea in_seq=0 out_seq=0 2024-03-18T00:56:22.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.468+0000 7f240effd640 10 client.26305 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:22.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.472+0000 7f240effd640 1 -- 172.21.15.49:0/2643235279 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e36bbe3d20 con 0x55e36bcf45f0 2024-03-18T00:56:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.472+0000 7f240effd640 10 client.26305 handle_command_reply: tid=0 2024-03-18T00:56:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.472+0000 7f2438a52740 1 client.26305 shutdown 2024-03-18T00:56:22.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.478+0000 7f2438a52740 1 -- 172.21.15.49:0/2643235279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 msgr2=0x55e36bc861e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:22.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.478+0000 7f2438a52740 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc861e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f24200dece0 tx=0x7f24200debd0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.479+0000 7f2438a52740 1 -- 172.21.15.49:0/2643235279 shutdown_connections 2024-03-18T00:56:22.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.479+0000 7f2438a52740 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55e36bcf45f0 0x55e36bcf6a10 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.479+0000 7f2438a52740 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e36bc7d5a0 0x55e36bcf24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.479+0000 7f2438a52740 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e36bc78430 0x55e36bc86720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.479+0000 7f2438a52740 1 --2- 172.21.15.49:0/2643235279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e36bc77a40 0x55e36bc861e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.479+0000 7f2438a52740 1 -- 172.21.15.49:0/2643235279 >> 172.21.15.49:0/2643235279 conn(0x55e36bc72fb0 msgr2=0x55e36bc797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:22.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.479+0000 7f240effd640 0 client.26305 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:22.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.479+0000 7f2438a52740 1 -- 172.21.15.49:0/2643235279 shutdown_connections 2024-03-18T00:56:22.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.480+0000 7f2438a52740 1 -- 172.21.15.49:0/2643235279 wait complete. 2024-03-18T00:56:22.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.480+0000 7f2438a52740 20 client.26305 trim_cache size 0 max 16384 2024-03-18T00:56:22.482 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:56:22.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.488+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f241c0a2a00 msgr2=0x7f241c0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:22.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.488+0000 7f2434bff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f241c0a2a00 0x7f241c0a4ec0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f242801b000 tx=0x7f2428018e30 comp rx=0 tx=0).stop 2024-03-18T00:56:22.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.488+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2430072bf0 msgr2=0x7f2430114ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:22.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.488+0000 7f2434bff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2430072bf0 0x7f2430114ef0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f2420009780 tx=0x7f24200324d0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.491+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 shutdown_connections 2024-03-18T00:56:22.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.491+0000 7f2434bff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f241c0a2a00 0x7f241c0a4ec0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.491+0000 7f2434bff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2430109870 0x7f24301b6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.491+0000 7f2434bff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2430108eb0 0x7f2430115430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.491+0000 7f2434bff640 1 --2- 172.21.15.49:0/546714618 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2430072bf0 0x7f2430114ef0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:22.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.491+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 >> 172.21.15.49:0/546714618 conn(0x7f243006e6c0 msgr2=0x7f2430070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:22.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.498+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 shutdown_connections 2024-03-18T00:56:22.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:22.500+0000 7f2434bff640 1 -- 172.21.15.49:0/546714618 wait complete. 2024-03-18T00:56:23.556 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:56:23.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:56:23.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:56:23.877 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:23 smithi049 ceph-mon[32130]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 909 MiB/s wr, 2.21k op/s 2024-03-18T00:56:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.907+0000 7f5be4d15640 1 -- 172.21.15.49:0/2547762329 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0072bf0 msgr2=0x7f5be0073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:23.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.907+0000 7f5be4d15640 1 --2- 172.21.15.49:0/2547762329 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0072bf0 0x7f5be0073010 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009ca0 tx=0x7f5bd802c120 comp rx=0 tx=0).stop 2024-03-18T00:56:23.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.909+0000 7f5be4d15640 1 -- 172.21.15.49:0/2547762329 shutdown_connections 2024-03-18T00:56:23.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.909+0000 7f5be4d15640 1 --2- 172.21.15.49:0/2547762329 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be0108eb0 0x7f5be0109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.909+0000 7f5be4d15640 1 --2- 172.21.15.49:0/2547762329 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0109870 0x7f5be010e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.909+0000 7f5be4d15640 1 --2- 172.21.15.49:0/2547762329 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0072bf0 0x7f5be0073010 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.909+0000 7f5be4d15640 1 -- 172.21.15.49:0/2547762329 >> 172.21.15.49:0/2547762329 conn(0x7f5be006e6c0 msgr2=0x7f5be0070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:23.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.915+0000 7f5be4d15640 1 -- 172.21.15.49:0/2547762329 shutdown_connections 2024-03-18T00:56:23.921 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.918+0000 7f5be4d15640 1 -- 172.21.15.49:0/2547762329 wait complete. 2024-03-18T00:56:23.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.919+0000 7f5be4d15640 1 Processor -- start 2024-03-18T00:56:23.922 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.919+0000 7f5be4d15640 1 -- start start 2024-03-18T00:56:23.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.919+0000 7f5be4d15640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0072bf0 0x7f5be0119ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.919+0000 7f5be4d15640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0108eb0 0x7f5be011a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.919+0000 7f5be4d15640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be0109870 0x7f5be0118190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.923 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.919+0000 7f5be4d15640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5be0110960 con 0x7f5be0108eb0 2024-03-18T00:56:23.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.919+0000 7f5be4d15640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5be01107e0 con 0x7f5be0109870 2024-03-18T00:56:23.924 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.919+0000 7f5be4d15640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5be0110ae0 con 0x7f5be0072bf0 2024-03-18T00:56:23.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.921+0000 7f5bdeffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0108eb0 0x7f5be011a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:23.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.921+0000 7f5bdeffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0108eb0 0x7f5be011a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38914/0 (socket says 172.21.15.49:38914) 2024-03-18T00:56:23.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.921+0000 7f5bdeffd640 1 -- 172.21.15.49:0/4084382338 learned_addr learned my addr 172.21.15.49:0/4084382338 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:23.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.924+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0072bf0 0x7f5be0119ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:23.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.925+0000 7f5bdeffd640 1 -- 172.21.15.49:0/4084382338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0072bf0 msgr2=0x7f5be0119ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:23.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.925+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0072bf0 0x7f5be0119ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.925+0000 7f5bdeffd640 1 -- 172.21.15.49:0/4084382338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be0109870 msgr2=0x7f5be0118190 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:56:23.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.925+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be0109870 0x7f5be0118190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.926 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.925+0000 7f5bdeffd640 1 -- 172.21.15.49:0/4084382338 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be0118990 con 0x7f5be0108eb0 2024-03-18T00:56:23.927 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.925+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0108eb0 0x7f5be011a000 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000c890 tx=0x7f5bd000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:23.929 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.929+0000 7f5bdcff9640 1 -- 172.21.15.49:0/4084382338 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0004220 con 0x7f5be0108eb0 2024-03-18T00:56:23.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:23 smithi107 ceph-mon[36270]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 909 MiB/s wr, 2.21k op/s 2024-03-18T00:56:23.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.929+0000 7f5bdcff9640 1 -- 172.21.15.49:0/4084382338 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd00043c0 con 0x7f5be0108eb0 2024-03-18T00:56:23.930 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.929+0000 7f5be4d15640 1 -- 172.21.15.49:0/4084382338 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be0118c80 con 0x7f5be0108eb0 2024-03-18T00:56:23.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.929+0000 7f5bdcff9640 1 -- 172.21.15.49:0/4084382338 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd00049e0 con 0x7f5be0108eb0 2024-03-18T00:56:23.931 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.929+0000 7f5be4d15640 1 -- 172.21.15.49:0/4084382338 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be0079ca0 con 0x7f5be0108eb0 2024-03-18T00:56:23.938 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.935+0000 7f5be8b64740 1 Processor -- start 2024-03-18T00:56:23.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.936+0000 7f5be8b64740 1 -- start start 2024-03-18T00:56:23.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.936+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd931f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.936+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd937040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.937+0000 7f5bdf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd937040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:23.939 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.937+0000 7f5bdf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd937040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38918/0 (socket says 172.21.15.49:38918) 2024-03-18T00:56:23.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.938+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd9376b0 0x563fdd939b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.938+0000 7f5be8b64740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563fdd8960d0 con 0x563fdd932540 2024-03-18T00:56:23.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.938+0000 7f5be8b64740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563fdd83ec70 con 0x563fdd931b50 2024-03-18T00:56:23.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.938+0000 7f5be8b64740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563fdd892170 con 0x563fdd9376b0 2024-03-18T00:56:23.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.938+0000 7f5bdf7fe640 1 -- 172.21.15.49:0/734074370 learned_addr learned my addr 172.21.15.49:0/734074370 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:23.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.938+0000 7f5bdffff640 1 --2- 172.21.15.49:0/734074370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd931f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:23.941 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.939+0000 7f5bdffff640 1 -- 172.21.15.49:0/734074370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd9376b0 msgr2=0x563fdd939b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:23.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.939+0000 7f5bdffff640 1 --2- 172.21.15.49:0/734074370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd9376b0 0x563fdd939b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.940+0000 7f5bdffff640 1 -- 172.21.15.49:0/734074370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 msgr2=0x563fdd937040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:23.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.940+0000 7f5bdffff640 1 --2- 172.21.15.49:0/734074370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd937040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.940+0000 7f5bdffff640 1 -- 172.21.15.49:0/734074370 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fdd6727e0 con 0x563fdd931b50 2024-03-18T00:56:23.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.940+0000 7f5bdcff9640 1 -- 172.21.15.49:0/4084382338 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5bd0004c40 con 0x7f5be0108eb0 2024-03-18T00:56:23.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.941+0000 7f5bdffff640 1 --2- 172.21.15.49:0/734074370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd931f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009600 tx=0x7f5bd4009ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dc2cd504964056 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:23.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:23 smithi106 ceph-mon[35899]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 909 MiB/s wr, 2.21k op/s 2024-03-18T00:56:23.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.943+0000 7f5bdcff9640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bc80a2a00 0x7f5bc80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.944+0000 7f5bdcff9640 1 -- 172.21.15.49:0/4084382338 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5bd00ce690 con 0x7f5be0108eb0 2024-03-18T00:56:23.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.945+0000 7f5bc5ffb640 1 -- 172.21.15.49:0/734074370 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4015020 con 0x563fdd931b50 2024-03-18T00:56:23.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.945+0000 7f5bc5ffb640 1 -- 172.21.15.49:0/734074370 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd4011d10 con 0x563fdd931b50 2024-03-18T00:56:23.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.948+0000 7f5be8b64740 1 -- 172.21.15.49:0/734074370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 msgr2=0x563fdd931f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:23.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.948+0000 7f5be8b64740 1 --2- 172.21.15.49:0/734074370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd931f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009600 tx=0x7f5bd4009ac0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.945+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bc80a2a00 0x7f5bc80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:23.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.955+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bc80a2a00 0x7f5bc80a4ec0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x563fdd932940 tx=0x7f5bd8032430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:23.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.960+0000 7f5be8b64740 1 -- 172.21.15.49:0/734074370 shutdown_connections 2024-03-18T00:56:23.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.960+0000 7f5be8b64740 1 --2- 172.21.15.49:0/734074370 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd9376b0 0x563fdd939b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.960+0000 7f5be8b64740 1 --2- 172.21.15.49:0/734074370 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd937040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.960+0000 7f5be8b64740 1 --2- 172.21.15.49:0/734074370 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd931f70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.960+0000 7f5be8b64740 1 -- 172.21.15.49:0/734074370 >> 172.21.15.49:0/734074370 conn(0x563fdd8f9c50 msgr2=0x563fdd92d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:23.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.968+0000 7f5be8b64740 1 -- 172.21.15.49:0/734074370 shutdown_connections 2024-03-18T00:56:23.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.969+0000 7f5be8b64740 1 -- 172.21.15.49:0/734074370 wait complete. 2024-03-18T00:56:23.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.969+0000 7f5be8b64740 1 -- start start 2024-03-18T00:56:23.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.970+0000 7f5be8b64740 1 Processor -- start 2024-03-18T00:56:23.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.970+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd93c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.971+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd932540 0x563fdd93c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.971+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd9376b0 0x563fdd9436c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:23.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.971+0000 7f5be8b64740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563fdd93a6b0 con 0x563fdd9376b0 2024-03-18T00:56:23.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.971+0000 7f5be8b64740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563fdd8520d0 con 0x563fdd931b50 2024-03-18T00:56:23.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.971+0000 7f5be8b64740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563fdd93a830 con 0x563fdd932540 2024-03-18T00:56:23.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.971+0000 7f5be8b64740 10 client.0 _ll_register_callbacks cb 0x563fdd81b760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:23.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.971+0000 7f5bdffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd9376b0 0x563fdd9436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:23.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.971+0000 7f5bdffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd9376b0 0x563fdd9436c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38926/0 (socket says 172.21.15.49:38926) 2024-03-18T00:56:23.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.971+0000 7f5bdffff640 1 -- 172.21.15.49:0/173816100 learned_addr learned my addr 172.21.15.49:0/173816100 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:23.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.973+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd93c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:23.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.974+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd932540 0x563fdd93c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:23.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.979+0000 7f5bdf7fe640 1 -- 172.21.15.49:0/173816100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd932540 msgr2=0x563fdd93c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:23.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.979+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd932540 0x563fdd93c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.979+0000 7f5bdf7fe640 1 -- 172.21.15.49:0/173816100 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd9376b0 msgr2=0x563fdd9436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:23.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.979+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd9376b0 0x563fdd9436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:23.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.979+0000 7f5bdf7fe640 1 -- 172.21.15.49:0/173816100 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fdd6d2f70 con 0x563fdd931b50 2024-03-18T00:56:23.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.991+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd93c140 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5bd802c730 tx=0x7f5bd80095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83d0acfa62d84104 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:23.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.993+0000 7f5bbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:56:23.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.993+0000 7f5bbeffd640 1 -- 172.21.15.49:0/173816100 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8045070 con 0x563fdd931b50 2024-03-18T00:56:23.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.994+0000 7f5be8b64740 1 -- 172.21.15.49:0/173816100 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd4011d10 con 0x563fdd931b50 2024-03-18T00:56:23.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.994+0000 7f5bbeffd640 1 -- 172.21.15.49:0/173816100 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5bd80041c0 con 0x563fdd931b50 2024-03-18T00:56:23.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:23.997+0000 7f5bbeffd640 1 -- 172.21.15.49:0/173816100 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5bd8039040 con 0x563fdd931b50 2024-03-18T00:56:24.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.001+0000 7f5be8b64740 10 client.26219 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:24.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.001+0000 7f5be8b64740 1 -- 172.21.15.49:0/173816100 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563fdd56a170 con 0x563fdd931b50 2024-03-18T00:56:24.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.009+0000 7f5bbeffd640 1 -- 172.21.15.49:0/173816100 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd4011d10 con 0x563fdd931b50 2024-03-18T00:56:24.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.009+0000 7f5bbeffd640 1 -- 172.21.15.49:0/173816100 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5bd802dce0 con 0x563fdd931b50 2024-03-18T00:56:24.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.010+0000 7f5be8b64740 10 client.26219 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:24.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.010+0000 7f5be8b64740 10 client.26219 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:24.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.010+0000 7f5be8b64740 20 client.26219 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:24.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.010+0000 7f5be8b64740 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fdd947bb0 0x563fdd94a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:24.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.010+0000 7f5be8b64740 4 client.26219 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:24.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.010+0000 7f5be8b64740 1 -- 172.21.15.49:0/173816100 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fdd8a3610 con 0x563fdd947bb0 2024-03-18T00:56:24.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.017+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fdd947bb0 0x563fdd94a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:24.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.023+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fdd947bb0 0x563fdd94a010 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92610c5ef9649227 server_cookie=33db45d3321e2221 in_seq=0 out_seq=0 2024-03-18T00:56:24.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.024+0000 7f5bbeffd640 10 client.26219 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:24.033 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.032+0000 7f5bbeffd640 1 -- 172.21.15.49:0/173816100 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563fdd8a3610 con 0x563fdd947bb0 2024-03-18T00:56:24.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.032+0000 7f5bbeffd640 10 client.26219 handle_command_reply: tid=0 2024-03-18T00:56:24.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.032+0000 7f5be8b64740 1 client.26219 shutdown 2024-03-18T00:56:24.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 -- 172.21.15.49:0/173816100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 msgr2=0x563fdd93c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:24.034 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd93c140 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5bd802c730 tx=0x7f5bd80095c0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 -- 172.21.15.49:0/173816100 shutdown_connections 2024-03-18T00:56:24.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fdd947bb0 0x563fdd94a010 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd9376b0 0x563fdd9436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.035 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd932540 0x563fdd93c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 --2- 172.21.15.49:0/173816100 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd931b50 0x563fdd93c140 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 -- 172.21.15.49:0/173816100 >> 172.21.15.49:0/173816100 conn(0x563fdd8f9c50 msgr2=0x563fdd92c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5bbeffd640 0 client.26219 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 -- 172.21.15.49:0/173816100 shutdown_connections 2024-03-18T00:56:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.033+0000 7f5be8b64740 1 -- 172.21.15.49:0/173816100 wait complete. 2024-03-18T00:56:24.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.034+0000 7f5be8b64740 20 client.26219 trim_cache size 0 max 16384 2024-03-18T00:56:24.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.069+0000 7f5be8b64740 1 Processor -- start 2024-03-18T00:56:24.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.069+0000 7f5be8b64740 1 -- start start 2024-03-18T00:56:24.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.069+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd931b50 0x563fdd930e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.069+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd932540 0x563fdd943680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.069+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd9376b0 0x563fdd94f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.069+0000 7f5be8b64740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc00e640 con 0x563fdd931b50 2024-03-18T00:56:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.069+0000 7f5be8b64740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc010180 con 0x563fdd932540 2024-03-18T00:56:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.069+0000 7f5be8b64740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc00ed10 con 0x563fdd9376b0 2024-03-18T00:56:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.070+0000 7f5bdf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd931b50 0x563fdd930e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.070+0000 7f5bdf7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd931b50 0x563fdd930e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38940/0 (socket says 172.21.15.49:38940) 2024-03-18T00:56:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.070+0000 7f5bdf7fe640 1 -- 172.21.15.49:0/1427453008 learned_addr learned my addr 172.21.15.49:0/1427453008 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:24.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.070+0000 7f5bdf7fe640 1 -- 172.21.15.49:0/1427453008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd9376b0 msgr2=0x563fdd94f8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:24.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.070+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/1427453008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd9376b0 0x563fdd94f8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.070+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/1427453008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd932540 0x563fdd943680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:24.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.070+0000 7f5bdf7fe640 1 -- 172.21.15.49:0/1427453008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd932540 msgr2=0x563fdd943680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:24.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.070+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/1427453008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd932540 0x563fdd943680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.070+0000 7f5bdf7fe640 1 -- 172.21.15.49:0/1427453008 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fdd931350 con 0x563fdd931b50 2024-03-18T00:56:24.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.071+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/1427453008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd931b50 0x563fdd930e10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8033000 tx=0x7f5bd802c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5324b9f653c840a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:24.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.073+0000 7f5bc57fa640 1 -- 172.21.15.49:0/1427453008 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8063020 con 0x563fdd931b50 2024-03-18T00:56:24.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.073+0000 7f5bc57fa640 1 -- 172.21.15.49:0/1427453008 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd8051070 con 0x563fdd931b50 2024-03-18T00:56:24.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.073+0000 7f5bc57fa640 1 -- 172.21.15.49:0/1427453008 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd8045040 con 0x563fdd931b50 2024-03-18T00:56:24.075 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.075+0000 7f5be8b64740 1 -- 172.21.15.49:0/1427453008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd931b50 msgr2=0x563fdd930e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:24.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.075+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1427453008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd931b50 0x563fdd930e10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8033000 tx=0x7f5bd802c120 comp rx=0 tx=0).stop 2024-03-18T00:56:24.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.075+0000 7f5be8b64740 1 -- 172.21.15.49:0/1427453008 shutdown_connections 2024-03-18T00:56:24.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.075+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1427453008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd932540 0x563fdd943680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.075+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1427453008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd9376b0 0x563fdd94f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.075+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1427453008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd931b50 0x563fdd930e10 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.075+0000 7f5be8b64740 1 -- 172.21.15.49:0/1427453008 >> 172.21.15.49:0/1427453008 conn(0x563fdd8f9c50 msgr2=0x563fdd930700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:24.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.076+0000 7f5be8b64740 1 -- 172.21.15.49:0/1427453008 shutdown_connections 2024-03-18T00:56:24.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.076+0000 7f5be8b64740 1 -- 172.21.15.49:0/1427453008 wait complete. 2024-03-18T00:56:24.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.076+0000 7f5be8b64740 1 -- start start 2024-03-18T00:56:24.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.077+0000 7f5be8b64740 1 Processor -- start 2024-03-18T00:56:24.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd931b50 0x563fdd941110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:24.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd941650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:24.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5be8b64740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd9376b0 0x563fdd942660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:24.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5be8b64740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc00fab0 con 0x563fdd932540 2024-03-18T00:56:24.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5be8b64740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc00f3e0 con 0x563fdd9376b0 2024-03-18T00:56:24.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5be8b64740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5bcc00c620 con 0x563fdd931b50 2024-03-18T00:56:24.078 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5be8b64740 10 client.0 _ll_register_callbacks cb 0x7f5bd8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:24.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdeffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd941650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:24.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdeffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd941650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38946/0 (socket says 172.21.15.49:38946) 2024-03-18T00:56:24.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdeffd640 1 -- 172.21.15.49:0/1634865082 learned_addr learned my addr 172.21.15.49:0/1634865082 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:24.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdeffd640 1 -- 172.21.15.49:0/1634865082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd931b50 msgr2=0x563fdd941110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:24.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd931b50 0x563fdd941110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:24.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd931b50 0x563fdd941110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdeffd640 1 -- 172.21.15.49:0/1634865082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd9376b0 msgr2=0x563fdd942660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:24.079 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd9376b0 0x563fdd942660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdeffd640 1 -- 172.21.15.49:0/1634865082 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fdd942c50 con 0x563fdd932540 2024-03-18T00:56:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.078+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd931b50 0x563fdd941110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:56:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.079+0000 7f5bdeffd640 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd941650 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0097410 tx=0x7f5bd00ce040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b99f67123a7e0d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.079+0000 7f5bbeffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.079+0000 7f5bbeffd640 1 -- 172.21.15.49:0/1634865082 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd001d040 con 0x563fdd932540 2024-03-18T00:56:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.079+0000 7f5be8b64740 1 -- 172.21.15.49:0/1634865082 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd8051070 con 0x563fdd932540 2024-03-18T00:56:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.079+0000 7f5bbeffd640 1 -- 172.21.15.49:0/1634865082 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5bd00cc220 con 0x563fdd932540 2024-03-18T00:56:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.079+0000 7f5bbeffd640 1 -- 172.21.15.49:0/1634865082 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5bd00a3750 con 0x563fdd932540 2024-03-18T00:56:24.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.079+0000 7f5be8b64740 10 client.16947 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:24.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.079+0000 7f5be8b64740 1 -- 172.21.15.49:0/1634865082 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563fdd9ae360 con 0x563fdd932540 2024-03-18T00:56:24.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.080+0000 7f5bbeffd640 1 -- 172.21.15.49:0/1634865082 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd8051070 con 0x563fdd932540 2024-03-18T00:56:24.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.080+0000 7f5bbeffd640 1 -- 172.21.15.49:0/1634865082 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5bd009e040 con 0x563fdd932540 2024-03-18T00:56:24.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.080+0000 7f5be8b64740 10 client.16947 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:24.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.080+0000 7f5be8b64740 10 client.16947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:24.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.080+0000 7f5be8b64740 20 client.16947 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:24.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.080+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fdd9aed10 0x563fdd9b1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:24.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.080+0000 7f5be8b64740 4 client.16947 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:56:24.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.080+0000 7f5be8b64740 1 -- 172.21.15.49:0/1634865082 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563fdd89d960 con 0x563fdd9aed10 2024-03-18T00:56:24.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.081+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fdd9aed10 0x563fdd9b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:24.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.086+0000 7f5bbeffd640 10 client.16947 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:24.086 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.086+0000 7f5bdf7fe640 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fdd9aed10 0x563fdd9b1130 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=868af3e2fef9fe2b server_cookie=c9df54fe230feafb in_seq=0 out_seq=0 2024-03-18T00:56:24.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.943+0000 7f5bbeffd640 1 -- 172.21.15.49:0/1634865082 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563fdd89d960 con 0x563fdd9aed10 2024-03-18T00:56:24.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.943+0000 7f5bbeffd640 10 client.16947 handle_command_reply: tid=0 2024-03-18T00:56:24.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.943+0000 7f5be8b64740 1 client.16947 shutdown 2024-03-18T00:56:24.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.947+0000 7f5be8b64740 1 -- 172.21.15.49:0/1634865082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 msgr2=0x563fdd941650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:24.947 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.948+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd941650 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0097410 tx=0x7f5bd00ce040 comp rx=0 tx=0).stop 2024-03-18T00:56:24.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5be8b64740 1 -- 172.21.15.49:0/1634865082 shutdown_connections 2024-03-18T00:56:24.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fdd9aed10 0x563fdd9b1130 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fdd9376b0 0x563fdd942660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fdd932540 0x563fdd941650 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5be8b64740 1 --2- 172.21.15.49:0/1634865082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fdd931b50 0x563fdd941110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5be8b64740 1 -- 172.21.15.49:0/1634865082 >> 172.21.15.49:0/1634865082 conn(0x563fdd8f9c50 msgr2=0x563fdd93bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:24.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5bbeffd640 0 client.16947 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:24.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5be8b64740 1 -- 172.21.15.49:0/1634865082 shutdown_connections 2024-03-18T00:56:24.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5be8b64740 1 -- 172.21.15.49:0/1634865082 wait complete. 2024-03-18T00:56:24.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.951+0000 7f5be8b64740 20 client.16947 trim_cache size 0 max 16384 2024-03-18T00:56:24.958 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:56:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:56:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "a6f56545-7b60-4703-be56-ded402dc5c70", 2024-03-18T00:56:24.958 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:56:24.958 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:56:24.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 -- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bc80a2a00 msgr2=0x7f5bc80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:24.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bc80a2a00 0x7f5bc80a4ec0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x563fdd932940 tx=0x7f5bd8032430 comp rx=0 tx=0).stop 2024-03-18T00:56:24.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 -- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0108eb0 msgr2=0x7f5be011a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:24.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0108eb0 0x7f5be011a000 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000c890 tx=0x7f5bd000cd50 comp rx=0 tx=0).stop 2024-03-18T00:56:24.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 -- 172.21.15.49:0/4084382338 shutdown_connections 2024-03-18T00:56:24.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bc80a2a00 0x7f5bc80a4ec0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be0109870 0x7f5be0118190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be0108eb0 0x7f5be011a000 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 --2- 172.21.15.49:0/4084382338 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be0072bf0 0x7f5be0119ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:24.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.962+0000 7f5be4d15640 1 -- 172.21.15.49:0/4084382338 >> 172.21.15.49:0/4084382338 conn(0x7f5be006e6c0 msgr2=0x7f5be010c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:24.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.968+0000 7f5be4d15640 1 -- 172.21.15.49:0/4084382338 shutdown_connections 2024-03-18T00:56:24.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:24.969+0000 7f5be4d15640 1 -- 172.21.15.49:0/4084382338 wait complete. 2024-03-18T00:56:25.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:56:25.019 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:56:25.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.585+0000 7f9732644640 1 -- 172.21.15.49:0/27517887 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f972c1029e0 msgr2=0x7f972c102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:25.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.585+0000 7f9732644640 1 --2- 172.21.15.49:0/27517887 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f972c1029e0 0x7f972c102e60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f971c009900 tx=0x7f971c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:56:25.588 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.588+0000 7f9732644640 1 -- 172.21.15.49:0/27517887 shutdown_connections 2024-03-18T00:56:25.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.588+0000 7f9732644640 1 --2- 172.21.15.49:0/27517887 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f972c1033a0 0x7f972c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.588+0000 7f9732644640 1 --2- 172.21.15.49:0/27517887 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f972c1029e0 0x7f972c102e60 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.588+0000 7f9732644640 1 --2- 172.21.15.49:0/27517887 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f972c1017f0 0x7f972c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.588+0000 7f9732644640 1 -- 172.21.15.49:0/27517887 >> 172.21.15.49:0/27517887 conn(0x7f972c0fcd40 msgr2=0x7f972c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:25.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.588+0000 7f9732644640 1 -- 172.21.15.49:0/27517887 shutdown_connections 2024-03-18T00:56:25.589 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.590+0000 7f9732644640 1 -- 172.21.15.49:0/27517887 wait complete. 2024-03-18T00:56:25.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.591+0000 7f9732644640 1 Processor -- start 2024-03-18T00:56:25.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.592+0000 7f9732644640 1 -- start start 2024-03-18T00:56:25.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.593+0000 7f9732644640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f972c1017f0 0x7f972c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.593+0000 7f9732644640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f972c1029e0 0x7f972c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.593+0000 7f9732644640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f972c1033a0 0x7f972c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.593+0000 7f9732644640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f972c10d5b0 con 0x7f972c1033a0 2024-03-18T00:56:25.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.594+0000 7f9732644640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f972c10d430 con 0x7f972c1017f0 2024-03-18T00:56:25.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.594+0000 7f9732644640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f972c10d730 con 0x7f972c1029e0 2024-03-18T00:56:25.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.594+0000 7f972bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f972c1017f0 0x7f972c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.595+0000 7f97237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f972c1029e0 0x7f972c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.595+0000 7f97237fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f972c1029e0 0x7f972c197f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44222/0 (socket says 172.21.15.49:44222) 2024-03-18T00:56:25.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.595+0000 7f97237fe640 1 -- 172.21.15.49:0/222571670 learned_addr learned my addr 172.21.15.49:0/222571670 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:25.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.596+0000 7f97237fe640 1 -- 172.21.15.49:0/222571670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f972c1017f0 msgr2=0x7f972c197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:25.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.596+0000 7f97237fe640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f972c1017f0 0x7f972c197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.596+0000 7f97237fe640 1 -- 172.21.15.49:0/222571670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f972c1033a0 msgr2=0x7f972c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:25.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.596+0000 7f97237fe640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f972c1033a0 0x7f972c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.596+0000 7f97237fe640 1 -- 172.21.15.49:0/222571670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f972c19f660 con 0x7f972c1029e0 2024-03-18T00:56:25.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.600+0000 7f97237fe640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f972c1029e0 0x7f972c197f80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f971c036040 tx=0x7f971c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:25.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.601+0000 7f9729ffb640 1 -- 172.21.15.49:0/222571670 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c043020 con 0x7f972c1029e0 2024-03-18T00:56:25.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.601+0000 7f9732644640 1 -- 172.21.15.49:0/222571670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972c19f8f0 con 0x7f972c1029e0 2024-03-18T00:56:25.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.601+0000 7f9729ffb640 1 -- 172.21.15.49:0/222571670 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f971c004490 con 0x7f972c1029e0 2024-03-18T00:56:25.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.601+0000 7f9729ffb640 1 -- 172.21.15.49:0/222571670 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971c03e6b0 con 0x7f972c1029e0 2024-03-18T00:56:25.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.601+0000 7f9732644640 1 -- 172.21.15.49:0/222571670 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f972c19fe90 con 0x7f972c1029e0 2024-03-18T00:56:25.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.606+0000 7f973522e740 1 Processor -- start 2024-03-18T00:56:25.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.606+0000 7f973522e740 1 -- start start 2024-03-18T00:56:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.607+0000 7f9729ffb640 1 -- 172.21.15.49:0/222571670 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f971c046020 con 0x7f972c1029e0 2024-03-18T00:56:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.607+0000 7f973522e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 0x55d9c4b889a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.607+0000 7f973522e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b8da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.607+0000 7f973522e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b90570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.607+0000 7f973522e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d9c4ae9500 con 0x55d9c4b8e0e0 2024-03-18T00:56:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.607+0000 7f973522e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d9c4a91be0 con 0x55d9c4b88f70 2024-03-18T00:56:25.607 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.607+0000 7f973522e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d9c4ae55a0 con 0x55d9c4b88580 2024-03-18T00:56:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.608+0000 7f9729ffb640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f970c0a28d0 0x7f970c0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.608+0000 7f97237fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b90570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.608+0000 7f97237fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b90570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38966/0 (socket says 172.21.15.49:38966) 2024-03-18T00:56:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.608+0000 7f97237fe640 1 -- 172.21.15.49:0/3323170517 learned_addr learned my addr 172.21.15.49:0/3323170517 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.608+0000 7f9729ffb640 1 -- 172.21.15.49:0/222571670 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f971c0f1380 con 0x7f972c1029e0 2024-03-18T00:56:25.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.608+0000 7f9730be1640 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 0x55d9c4b889a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.608+0000 7f9730be1640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f970c0a28d0 0x7f970c0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.608+0000 7f972bfff640 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b8da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.609+0000 7f9730be1640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f970c0a28d0 0x7f970c0a4d90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9724008fd0 tx=0x7f972400b6f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:25.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.609+0000 7f9730be1640 1 -- 172.21.15.49:0/3323170517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 msgr2=0x55d9c4b8da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:25.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.609+0000 7f9730be1640 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b8da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.609+0000 7f9730be1640 1 -- 172.21.15.49:0/3323170517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 msgr2=0x55d9c4b90570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:25.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.609+0000 7f9730be1640 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b90570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.609+0000 7f9730be1640 1 -- 172.21.15.49:0/3323170517 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c4aa54b0 con 0x55d9c4b88580 2024-03-18T00:56:25.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.609+0000 7f972bfff640 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b8da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:56:25.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.610+0000 7f9730be1640 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 0x55d9c4b889a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f972400be50 tx=0x7f9724006000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3463e17ac141d798 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:25.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.610+0000 7f97227fc640 1 -- 172.21.15.49:0/3323170517 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97240060d0 con 0x55d9c4b88580 2024-03-18T00:56:25.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.611+0000 7f97227fc640 1 -- 172.21.15.49:0/3323170517 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f972401d070 con 0x55d9c4b88580 2024-03-18T00:56:25.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.611+0000 7f97227fc640 1 -- 172.21.15.49:0/3323170517 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9724019a80 con 0x55d9c4b88580 2024-03-18T00:56:25.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.613+0000 7f973522e740 1 -- 172.21.15.49:0/3323170517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 msgr2=0x55d9c4b889a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:25.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.613+0000 7f973522e740 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 0x55d9c4b889a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f972400be50 tx=0x7f9724006000 comp rx=0 tx=0).stop 2024-03-18T00:56:25.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:25 smithi049 ceph-mon[32130]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 869 MiB/s wr, 2.29k op/s 2024-03-18T00:56:25.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:25 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:56:25.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:25 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:56:25.613 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:25 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:56:25.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.613+0000 7f973522e740 1 -- 172.21.15.49:0/3323170517 shutdown_connections 2024-03-18T00:56:25.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.613+0000 7f973522e740 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b8da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.613+0000 7f973522e740 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b90570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.613+0000 7f973522e740 1 --2- 172.21.15.49:0/3323170517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 0x55d9c4b889a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.613+0000 7f973522e740 1 -- 172.21.15.49:0/3323170517 >> 172.21.15.49:0/3323170517 conn(0x55d9c4b7f6e0 msgr2=0x55d9c4b7fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.613+0000 7f973522e740 1 -- 172.21.15.49:0/3323170517 shutdown_connections 2024-03-18T00:56:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.613+0000 7f973522e740 1 -- 172.21.15.49:0/3323170517 wait complete. 2024-03-18T00:56:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.615+0000 7f973522e740 1 -- start start 2024-03-18T00:56:25.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.615+0000 7f973522e740 1 Processor -- start 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.616+0000 7f973522e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88580 0x55d9c4b92880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.616+0000 7f973522e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b92dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.616+0000 7f973522e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b99e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.616+0000 7f973522e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d9c4b90fd0 con 0x55d9c4b88580 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.616+0000 7f973522e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d9c4aa5160 con 0x55d9c4b88f70 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.616+0000 7f973522e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d9c4b91150 con 0x55d9c4b8e0e0 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.616+0000 7f973522e740 10 client.0 _ll_register_callbacks cb 0x55d9c4a89150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.617+0000 7f972bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88580 0x55d9c4b92880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.617+0000 7f972bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88580 0x55d9c4b92880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38982/0 (socket says 172.21.15.49:38982) 2024-03-18T00:56:25.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.617+0000 7f972bfff640 1 -- 172.21.15.49:0/2862421344 learned_addr learned my addr 172.21.15.49:0/2862421344 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:25.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.618+0000 7f972bfff640 1 -- 172.21.15.49:0/2862421344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b8e0e0 msgr2=0x55d9c4b99e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:25.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.618+0000 7f972bfff640 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b99e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.618+0000 7f972bfff640 1 -- 172.21.15.49:0/2862421344 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 msgr2=0x55d9c4b92dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:25.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.618+0000 7f972bfff640 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b92dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.618+0000 7f972bfff640 1 -- 172.21.15.49:0/2862421344 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c48c77d0 con 0x55d9c4b88580 2024-03-18T00:56:25.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.619+0000 7f972bfff640 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88580 0x55d9c4b92880 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f971400bf30 tx=0x7f971400f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f0c0e5d84d85dcd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:25.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.619+0000 7f971b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:25.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.619+0000 7f971b7fe640 1 -- 172.21.15.49:0/2862421344 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714018020 con 0x55d9c4b88580 2024-03-18T00:56:25.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.619+0000 7f973522e740 1 -- 172.21.15.49:0/2862421344 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f972401d070 con 0x55d9c4b88580 2024-03-18T00:56:25.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.619+0000 7f971b7fe640 1 -- 172.21.15.49:0/2862421344 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9714014660 con 0x55d9c4b88580 2024-03-18T00:56:25.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.620+0000 7f971b7fe640 1 -- 172.21.15.49:0/2862421344 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9714023050 con 0x55d9c4b88580 2024-03-18T00:56:25.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.620+0000 7f973522e740 10 client.16956 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:25.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.620+0000 7f973522e740 1 -- 172.21.15.49:0/2862421344 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d9c47fd3f0 con 0x55d9c4b88580 2024-03-18T00:56:25.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.621+0000 7f971b7fe640 1 -- 172.21.15.49:0/2862421344 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f972401d070 con 0x55d9c4b88580 2024-03-18T00:56:25.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.622+0000 7f971b7fe640 1 -- 172.21.15.49:0/2862421344 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9714013aa0 con 0x55d9c4b88580 2024-03-18T00:56:25.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.622+0000 7f973522e740 10 client.16956 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:25.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.622+0000 7f973522e740 10 client.16956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:25.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.622+0000 7f973522e740 20 client.16956 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:25.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.622+0000 7f973522e740 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9c4b87210 0x55d9c4b876c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.622+0000 7f973522e740 4 client.16956 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:25.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.622+0000 7f973522e740 1 -- 172.21.15.49:0/2862421344 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9c4af6a40 con 0x55d9c4b87210 2024-03-18T00:56:25.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.624+0000 7f97237fe640 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9c4b87210 0x55d9c4b876c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.628+0000 7f97237fe640 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9c4b87210 0x55d9c4b876c0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a819017d645c5e server_cookie=ba25a0809004f2c6 in_seq=0 out_seq=0 2024-03-18T00:56:25.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.628+0000 7f971b7fe640 10 client.16956 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:25.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f971b7fe640 1 -- 172.21.15.49:0/2862421344 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d9c4af6a40 con 0x55d9c4b87210 2024-03-18T00:56:25.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f971b7fe640 10 client.16956 handle_command_reply: tid=0 2024-03-18T00:56:25.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f973522e740 1 client.16956 shutdown 2024-03-18T00:56:25.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f973522e740 1 -- 172.21.15.49:0/2862421344 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88580 msgr2=0x55d9c4b92880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:25.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f973522e740 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88580 0x55d9c4b92880 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f971400bf30 tx=0x7f971400f670 comp rx=0 tx=0).stop 2024-03-18T00:56:25.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f973522e740 1 -- 172.21.15.49:0/2862421344 shutdown_connections 2024-03-18T00:56:25.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f973522e740 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9c4b87210 0x55d9c4b876c0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f973522e740 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b92dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.633 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f973522e740 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b99e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f973522e740 1 --2- 172.21.15.49:0/2862421344 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88580 0x55d9c4b92880 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f973522e740 1 -- 172.21.15.49:0/2862421344 >> 172.21.15.49:0/2862421344 conn(0x55d9c4b7f6e0 msgr2=0x55d9c4b7e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:25.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.633+0000 7f971b7fe640 0 client.16956 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:25.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.635+0000 7f973522e740 1 -- 172.21.15.49:0/2862421344 shutdown_connections 2024-03-18T00:56:25.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.635+0000 7f973522e740 1 -- 172.21.15.49:0/2862421344 wait complete. 2024-03-18T00:56:25.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.636+0000 7f973522e740 20 client.16956 trim_cache size 0 max 16384 2024-03-18T00:56:25.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.717+0000 7f973522e740 1 Processor -- start 2024-03-18T00:56:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.720+0000 7f973522e740 1 -- start start 2024-03-18T00:56:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.720+0000 7f973522e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 0x55d9c4b91b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.720+0000 7f973522e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b99dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.720+0000 7f973522e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b92090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.720+0000 7f973522e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f972c1a6a20 con 0x55d9c4b8e0e0 2024-03-18T00:56:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.720+0000 7f973522e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f972c1a8560 con 0x55d9c4b88f70 2024-03-18T00:56:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.720+0000 7f973522e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f972c1a70f0 con 0x55d9c4b88580 2024-03-18T00:56:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.721+0000 7f9730be1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b92090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.721+0000 7f9730be1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b92090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38986/0 (socket says 172.21.15.49:38986) 2024-03-18T00:56:25.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.721+0000 7f9730be1640 1 -- 172.21.15.49:0/2478640213 learned_addr learned my addr 172.21.15.49:0/2478640213 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:25.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.724+0000 7f972bfff640 1 --2- 172.21.15.49:0/2478640213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 0x55d9c4b91b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.726+0000 7f97237fe640 1 --2- 172.21.15.49:0/2478640213 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b99dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.728+0000 7f9730be1640 1 -- 172.21.15.49:0/2478640213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 msgr2=0x55d9c4b91b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.728+0000 7f9730be1640 1 --2- 172.21.15.49:0/2478640213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 0x55d9c4b91b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.728+0000 7f9730be1640 1 -- 172.21.15.49:0/2478640213 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 msgr2=0x55d9c4b99dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.728+0000 7f9730be1640 1 --2- 172.21.15.49:0/2478640213 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b99dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.728+0000 7f9730be1640 1 -- 172.21.15.49:0/2478640213 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9c47bd0f0 con 0x55d9c4b8e0e0 2024-03-18T00:56:25.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.733+0000 7f9730be1640 1 --2- 172.21.15.49:0/2478640213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b92090 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9724023040 tx=0x7f972400d0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd83ef484b991aa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:25.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.737+0000 7f9721ffb640 1 -- 172.21.15.49:0/2478640213 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f972402e070 con 0x55d9c4b8e0e0 2024-03-18T00:56:25.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.737+0000 7f9721ffb640 1 -- 172.21.15.49:0/2478640213 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f972401d050 con 0x55d9c4b8e0e0 2024-03-18T00:56:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.738+0000 7f973522e740 1 -- 172.21.15.49:0/2478640213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 msgr2=0x55d9c4b92090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:25.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.738+0000 7f973522e740 1 --2- 172.21.15.49:0/2478640213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b92090 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9724023040 tx=0x7f972400d0f0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.740+0000 7f973522e740 1 -- 172.21.15.49:0/2478640213 shutdown_connections 2024-03-18T00:56:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.740+0000 7f973522e740 1 --2- 172.21.15.49:0/2478640213 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88f70 0x55d9c4b99dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.740+0000 7f973522e740 1 --2- 172.21.15.49:0/2478640213 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b92090 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.740+0000 7f973522e740 1 --2- 172.21.15.49:0/2478640213 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b88580 0x55d9c4b91b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.740+0000 7f973522e740 1 -- 172.21.15.49:0/2478640213 >> 172.21.15.49:0/2478640213 conn(0x55d9c4b7f6e0 msgr2=0x55d9c4b831d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:25.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.745+0000 7f973522e740 1 -- 172.21.15.49:0/2478640213 shutdown_connections 2024-03-18T00:56:25.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.751+0000 7f973522e740 1 -- 172.21.15.49:0/2478640213 wait complete. 2024-03-18T00:56:25.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.751+0000 7f973522e740 1 -- start start 2024-03-18T00:56:25.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.752+0000 7f973522e740 1 Processor -- start 2024-03-18T00:56:25.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.753+0000 7f973522e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88580 0x55d9c4b952d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.753+0000 7f973522e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88f70 0x55d9c4b95810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.753+0000 7f973522e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b98830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.753+0000 7f973522e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f972c1a7e90 con 0x55d9c4b88f70 2024-03-18T00:56:25.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.753+0000 7f973522e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f972c1a77c0 con 0x55d9c4b88580 2024-03-18T00:56:25.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.753+0000 7f973522e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f972c1a4a00 con 0x55d9c4b8e0e0 2024-03-18T00:56:25.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.753+0000 7f973522e740 10 client.0 _ll_register_callbacks cb 0x7f971400b240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:25.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.754+0000 7f97237fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88f70 0x55d9c4b95810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.754+0000 7f97237fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88f70 0x55d9c4b95810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38990/0 (socket says 172.21.15.49:38990) 2024-03-18T00:56:25.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.754+0000 7f97237fe640 1 -- 172.21.15.49:0/463424517 learned_addr learned my addr 172.21.15.49:0/463424517 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:25.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.755+0000 7f9730be1640 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b98830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.757+0000 7f972bfff640 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88580 0x55d9c4b952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.758+0000 7f972bfff640 1 -- 172.21.15.49:0/463424517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b8e0e0 msgr2=0x55d9c4b98830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:25.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.758+0000 7f972bfff640 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b98830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.758+0000 7f972bfff640 1 -- 172.21.15.49:0/463424517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88f70 msgr2=0x55d9c4b95810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:25.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.758+0000 7f972bfff640 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88f70 0x55d9c4b95810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.758+0000 7f972bfff640 1 -- 172.21.15.49:0/463424517 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9c4ab09d0 con 0x55d9c4b88580 2024-03-18T00:56:25.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.759+0000 7f972bfff640 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88580 0x55d9c4b952d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9714010010 tx=0x7f97140135f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=574221fb9082e54d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:25.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.759+0000 7f971b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:56:25.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.760+0000 7f971b7fe640 1 -- 172.21.15.49:0/463424517 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971402d070 con 0x55d9c4b88580 2024-03-18T00:56:25.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.760+0000 7f971b7fe640 1 -- 172.21.15.49:0/463424517 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f971401c4c0 con 0x55d9c4b88580 2024-03-18T00:56:25.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.763+0000 7f973522e740 1 -- 172.21.15.49:0/463424517 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f972401d050 con 0x55d9c4b88580 2024-03-18T00:56:25.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.769+0000 7f971b7fe640 1 -- 172.21.15.49:0/463424517 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9714018040 con 0x55d9c4b88580 2024-03-18T00:56:25.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.769+0000 7f973522e740 10 client.26231 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:25.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.769+0000 7f973522e740 1 -- 172.21.15.49:0/463424517 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d9c4b99fd0 con 0x55d9c4b88580 2024-03-18T00:56:25.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.770+0000 7f971b7fe640 1 -- 172.21.15.49:0/463424517 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f972401d050 con 0x55d9c4b88580 2024-03-18T00:56:25.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.771+0000 7f971b7fe640 1 -- 172.21.15.49:0/463424517 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9714030020 con 0x55d9c4b88580 2024-03-18T00:56:25.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.771+0000 7f973522e740 10 client.26231 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:25.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.771+0000 7f973522e740 10 client.26231 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:25.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.771+0000 7f973522e740 20 client.26231 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:25.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.771+0000 7f973522e740 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9c4c1a4c0 0x55d9c4c1c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:25.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.771+0000 7f973522e740 4 client.26231 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:56:25.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.771+0000 7f973522e740 1 -- 172.21.15.49:0/463424517 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d9c4af0d90 con 0x55d9c4c1a4c0 2024-03-18T00:56:25.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.777+0000 7f97237fe640 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9c4c1a4c0 0x55d9c4c1c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:25.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.779+0000 7f971b7fe640 10 client.26231 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:25.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.779+0000 7f97237fe640 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9c4c1a4c0 0x55d9c4c1c8b0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c03e344766e1a77 server_cookie=4d1cebdc170f174a in_seq=0 out_seq=0 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f971b7fe640 1 -- 172.21.15.49:0/463424517 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55d9c4af0d90 con 0x55d9c4c1a4c0 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f971b7fe640 10 client.26231 handle_command_reply: tid=0 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f973522e740 1 client.26231 shutdown 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f973522e740 1 -- 172.21.15.49:0/463424517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88580 msgr2=0x55d9c4b952d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f973522e740 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88580 0x55d9c4b952d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9714010010 tx=0x7f97140135f0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f973522e740 1 -- 172.21.15.49:0/463424517 shutdown_connections 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f973522e740 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d9c4c1a4c0 0x55d9c4c1c8b0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f973522e740 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d9c4b8e0e0 0x55d9c4b98830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f971b7fe640 0 client.26231 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f973522e740 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d9c4b88f70 0x55d9c4b95810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f973522e740 1 --2- 172.21.15.49:0/463424517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d9c4b88580 0x55d9c4b952d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.782+0000 7f973522e740 1 -- 172.21.15.49:0/463424517 >> 172.21.15.49:0/463424517 conn(0x55d9c4b7f6e0 msgr2=0x55d9c4b7e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:25.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.783+0000 7f973522e740 1 -- 172.21.15.49:0/463424517 shutdown_connections 2024-03-18T00:56:25.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.783+0000 7f973522e740 1 -- 172.21.15.49:0/463424517 wait complete. 2024-03-18T00:56:25.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.783+0000 7f973522e740 20 client.26231 trim_cache size 0 max 16384 2024-03-18T00:56:25.784 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:56:25.784 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (1 inodes in the stack)", 2024-03-18T00:56:25.784 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:56:25.784 INFO:teuthology.orchestra.run.smithi049.stdout: "a6f56545-7b60-4703-be56-ded402dc5c70": { 2024-03-18T00:56:25.784 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:56:25.785 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "a6f56545-7b60-4703-be56-ded402dc5c70", 2024-03-18T00:56:25.785 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:56:25.785 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:56:25.785 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:56:25.785 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 -- 172.21.15.49:0/222571670 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f970c0a28d0 msgr2=0x7f970c0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f970c0a28d0 0x7f970c0a4d90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9724008fd0 tx=0x7f972400b6f0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 -- 172.21.15.49:0/222571670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f972c1029e0 msgr2=0x7f972c197f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f972c1029e0 0x7f972c197f80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f971c036040 tx=0x7f971c004270 comp rx=0 tx=0).stop 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 -- 172.21.15.49:0/222571670 shutdown_connections 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f970c0a28d0 0x7f970c0a4d90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f972c1033a0 0x7f972c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f972c1029e0 0x7f972c197f80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 --2- 172.21.15.49:0/222571670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f972c1017f0 0x7f972c197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 -- 172.21.15.49:0/222571670 >> 172.21.15.49:0/222571670 conn(0x7f972c0fcd40 msgr2=0x7f972c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 -- 172.21.15.49:0/222571670 shutdown_connections 2024-03-18T00:56:25.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:25.784+0000 7f9732644640 1 -- 172.21.15.49:0/222571670 wait complete. 2024-03-18T00:56:25.811 INFO:tasks.cephfs.filesystem:scrub status for tag:a6f56545-7b60-4703-be56-ded402dc5c70 - {'path': '/', 'tag': 'a6f56545-7b60-4703-be56-ded402dc5c70', 'options': 'recursive,force'} 2024-03-18T00:56:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:25 smithi107 ceph-mon[36270]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 869 MiB/s wr, 2.29k op/s 2024-03-18T00:56:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:25 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:56:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:25 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:56:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:25 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:56:25.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:25 smithi106 ceph-mon[35899]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 869 MiB/s wr, 2.29k op/s 2024-03-18T00:56:25.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:25 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:56:25.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:25 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:56:25.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:25 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:56:27.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:27 smithi049 ceph-mon[32130]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 862 MiB/s wr, 2.29k op/s 2024-03-18T00:56:27.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:27 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:56:27.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:27 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:56:27.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:27 smithi107 ceph-mon[36270]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 862 MiB/s wr, 2.29k op/s 2024-03-18T00:56:27.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:27 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:56:27.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:27 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:56:27.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:27 smithi106 ceph-mon[35899]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 862 MiB/s wr, 2.29k op/s 2024-03-18T00:56:27.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:27 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:56:27.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:27 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:56:28.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:28 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:56:28.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:28 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:56:29.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:28 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:56:29.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:29 smithi107 ceph-mon[36270]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 884 MiB/s wr, 2.29k op/s 2024-03-18T00:56:29.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:29 smithi106 ceph-mon[35899]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 884 MiB/s wr, 2.29k op/s 2024-03-18T00:56:30.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:29 smithi049 ceph-mon[32130]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 884 MiB/s wr, 2.29k op/s 2024-03-18T00:56:30.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.685+0000 7f7479d37640 1 -- 172.21.15.49:0/1273113981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474078f30 msgr2=0x7f747407b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:30.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.685+0000 7f7479d37640 1 --2- 172.21.15.49:0/1273113981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474078f30 0x7f747407b3c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f746c009ca0 tx=0x7f746c02c140 comp rx=0 tx=0).stop 2024-03-18T00:56:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.687+0000 7f7479d37640 1 -- 172.21.15.49:0/1273113981 shutdown_connections 2024-03-18T00:56:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.687+0000 7f7479d37640 1 --2- 172.21.15.49:0/1273113981 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7474076580 0x7f74740789f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.687+0000 7f7479d37640 1 --2- 172.21.15.49:0/1273113981 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474078f30 0x7f747407b3c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.687+0000 7f7479d37640 1 --2- 172.21.15.49:0/1273113981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f74740736a0 0x7f7474073ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.687+0000 7f7479d37640 1 -- 172.21.15.49:0/1273113981 >> 172.21.15.49:0/1273113981 conn(0x7f747406e820 msgr2=0x7f7474070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.687+0000 7f7479d37640 1 -- 172.21.15.49:0/1273113981 shutdown_connections 2024-03-18T00:56:30.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.687+0000 7f7479d37640 1 -- 172.21.15.49:0/1273113981 wait complete. 2024-03-18T00:56:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.688+0000 7f7479d37640 1 Processor -- start 2024-03-18T00:56:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7479d37640 1 -- start start 2024-03-18T00:56:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7479d37640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74740736a0 0x7f74741b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7479d37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474076580 0x7f74741b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.691 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7479d37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7474078f30 0x7f74741b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7479d37640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f747407fd80 con 0x7f7474076580 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7479d37640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f747407fc00 con 0x7f74740736a0 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7479d37640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f747407ff00 con 0x7f7474078f30 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7472ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474076580 0x7f74741b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7472ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474076580 0x7f74741b0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:49724/0 (socket says 172.21.15.49:49724) 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.689+0000 7f7472ffd640 1 -- 172.21.15.49:0/2729182167 learned_addr learned my addr 172.21.15.49:0/2729182167 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.690+0000 7f7472ffd640 1 -- 172.21.15.49:0/2729182167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7474078f30 msgr2=0x7f74741b76f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.690+0000 7f7472ffd640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7474078f30 0x7f74741b76f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.690+0000 7f7472ffd640 1 -- 172.21.15.49:0/2729182167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74740736a0 msgr2=0x7f74741b0140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.690+0000 7f7472ffd640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74740736a0 0x7f74741b0140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.690+0000 7f7472ffd640 1 -- 172.21.15.49:0/2729182167 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747407c240 con 0x7f7474076580 2024-03-18T00:56:30.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.690+0000 7f7472ffd640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474076580 0x7f74741b0680 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f746400c890 tx=0x7f746400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.692+0000 7f7470ff9640 1 -- 172.21.15.49:0/2729182167 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464004220 con 0x7f7474076580 2024-03-18T00:56:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.692+0000 7f7470ff9640 1 -- 172.21.15.49:0/2729182167 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7464007e10 con 0x7f7474076580 2024-03-18T00:56:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.692+0000 7f7470ff9640 1 -- 172.21.15.49:0/2729182167 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464004a00 con 0x7f7474076580 2024-03-18T00:56:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.694+0000 7f7479d37640 1 -- 172.21.15.49:0/2729182167 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f747407c530 con 0x7f7474076580 2024-03-18T00:56:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.694+0000 7f7479d37640 1 -- 172.21.15.49:0/2729182167 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f747407ca00 con 0x7f7474076580 2024-03-18T00:56:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.701+0000 7f747c9f7740 1 Processor -- start 2024-03-18T00:56:30.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.703+0000 7f7470ff9640 1 -- 172.21.15.49:0/2729182167 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7464007930 con 0x7f7474076580 2024-03-18T00:56:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.704+0000 7f747c9f7740 1 -- start start 2024-03-18T00:56:30.706 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.705+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31932ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.705+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b319330540 0x55b319335040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.705+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b3193356b0 0x55b319337b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.705+0000 7f747c9f7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b31928ca40 con 0x55b3193356b0 2024-03-18T00:56:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.705+0000 7f747c9f7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b319235920 con 0x55b31932fb50 2024-03-18T00:56:30.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.705+0000 7f747c9f7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b319288ae0 con 0x55b319330540 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.705+0000 7f7470ff9640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f745c0a2b10 0x7f745c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.705+0000 7f7473fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31932ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.706+0000 7f7473fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31932ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:53160/0 (socket says 172.21.15.49:53160) 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.706+0000 7f7473fff640 1 -- 172.21.15.49:0/994801112 learned_addr learned my addr 172.21.15.49:0/994801112 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.706+0000 7f74737fe640 1 --2- 172.21.15.49:0/994801112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b319330540 0x55b319335040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.706+0000 7f7473fff640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f745c0a2b10 0x7f745c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.706+0000 7f7470ff9640 1 -- 172.21.15.49:0/2729182167 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f74640cea30 con 0x7f7474076580 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.707+0000 7f7473fff640 1 -- 172.21.15.49:0/994801112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b319330540 msgr2=0x55b319335040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.707+0000 7f7473fff640 1 --2- 172.21.15.49:0/994801112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b319330540 0x55b319335040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.707+0000 7f7473fff640 1 -- 172.21.15.49:0/994801112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b3193356b0 msgr2=0x55b319337b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.707+0000 7f7473fff640 1 --2- 172.21.15.49:0/994801112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b3193356b0 0x55b319337b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.707+0000 7f7473fff640 1 -- 172.21.15.49:0/994801112 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b31924d460 con 0x55b31932fb50 2024-03-18T00:56:30.710 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.708+0000 7f7473fff640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f745c0a2b10 0x7f745c0a4fd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f746c009a60 tx=0x7f746c009910 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.709+0000 7f7473fff640 1 --2- 172.21.15.49:0/994801112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31932ff70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f746c009dc0 tx=0x7f746c044000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c67ad431ba1a06e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.711+0000 7f7459ffb640 1 -- 172.21.15.49:0/994801112 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c050070 con 0x55b31932fb50 2024-03-18T00:56:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.712+0000 7f7459ffb640 1 -- 172.21.15.49:0/994801112 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746c034be0 con 0x55b31932fb50 2024-03-18T00:56:30.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.712+0000 7f7459ffb640 1 -- 172.21.15.49:0/994801112 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c039070 con 0x55b31932fb50 2024-03-18T00:56:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.713+0000 7f747c9f7740 1 -- 172.21.15.49:0/994801112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 msgr2=0x55b31932ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.713+0000 7f747c9f7740 1 --2- 172.21.15.49:0/994801112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31932ff70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f746c009dc0 tx=0x7f746c044000 comp rx=0 tx=0).stop 2024-03-18T00:56:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.713+0000 7f747c9f7740 1 -- 172.21.15.49:0/994801112 shutdown_connections 2024-03-18T00:56:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.713+0000 7f747c9f7740 1 --2- 172.21.15.49:0/994801112 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b3193356b0 0x55b319337b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.714 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.713+0000 7f747c9f7740 1 --2- 172.21.15.49:0/994801112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b319330540 0x55b319335040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.713+0000 7f747c9f7740 1 --2- 172.21.15.49:0/994801112 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31932ff70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.713+0000 7f747c9f7740 1 -- 172.21.15.49:0/994801112 >> 172.21.15.49:0/994801112 conn(0x55b3193228b0 msgr2=0x55b319322cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.713+0000 7f747c9f7740 1 -- 172.21.15.49:0/994801112 shutdown_connections 2024-03-18T00:56:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.713+0000 7f747c9f7740 1 -- 172.21.15.49:0/994801112 wait complete. 2024-03-18T00:56:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.714+0000 7f747c9f7740 1 -- start start 2024-03-18T00:56:30.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.714+0000 7f747c9f7740 1 Processor -- start 2024-03-18T00:56:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.715+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.715+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b31933a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.715+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b319341480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.715+0000 7f747c9f7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b319338600 con 0x55b319330540 2024-03-18T00:56:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.715+0000 7f747c9f7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b319248d70 con 0x55b31932fb50 2024-03-18T00:56:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.715+0000 7f747c9f7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b319338780 con 0x55b3193356b0 2024-03-18T00:56:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.715+0000 7f747c9f7740 10 client.0 _ll_register_callbacks cb 0x55b318fa0a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:30.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.716+0000 7f74737fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.716+0000 7f74737fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:53172/0 (socket says 172.21.15.49:53172) 2024-03-18T00:56:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.716+0000 7f74737fe640 1 -- 172.21.15.49:0/2666158272 learned_addr learned my addr 172.21.15.49:0/2666158272 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:30.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.716+0000 7f7473fff640 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b319341480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.717+0000 7f7472ffd640 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b31933a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.718+0000 7f74737fe640 1 -- 172.21.15.49:0/2666158272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 msgr2=0x55b319341480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.718+0000 7f74737fe640 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b319341480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.718+0000 7f74737fe640 1 -- 172.21.15.49:0/2666158272 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 msgr2=0x55b31933a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.718+0000 7f74737fe640 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b31933a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.718+0000 7f74737fe640 1 -- 172.21.15.49:0/2666158272 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b318fa1210 con 0x55b31932fb50 2024-03-18T00:56:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.722+0000 7f74737fe640 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339f70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f746800f670 tx=0x7f746800fb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4b89c13553e30c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:30.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.723+0000 7f7452ffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:56:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.725+0000 7f7452ffd640 1 -- 172.21.15.49:0/2666158272 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746800daa0 con 0x55b31932fb50 2024-03-18T00:56:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.725+0000 7f7452ffd640 1 -- 172.21.15.49:0/2666158272 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f746800dc40 con 0x55b31932fb50 2024-03-18T00:56:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.726+0000 7f747c9f7740 1 -- 172.21.15.49:0/2666158272 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f746c034be0 con 0x55b31932fb50 2024-03-18T00:56:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.727+0000 7f7452ffd640 1 -- 172.21.15.49:0/2666158272 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f746800de40 con 0x55b31932fb50 2024-03-18T00:56:30.734 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.727+0000 7f747c9f7740 10 client.26237 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.727+0000 7f747c9f7740 1 -- 172.21.15.49:0/2666158272 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b31906d620 con 0x55b31932fb50 2024-03-18T00:56:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.729+0000 7f7452ffd640 1 -- 172.21.15.49:0/2666158272 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746c034be0 con 0x55b31932fb50 2024-03-18T00:56:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.729+0000 7f7452ffd640 1 -- 172.21.15.49:0/2666158272 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f74680105b0 con 0x55b31932fb50 2024-03-18T00:56:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.729+0000 7f747c9f7740 10 client.26237 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.729+0000 7f747c9f7740 10 client.26237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.729+0000 7f747c9f7740 20 client.26237 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.729+0000 7f747c9f7740 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b319345d00 0x55b319348170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.729+0000 7f747c9f7740 4 client.26237 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:30.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.729+0000 7f747c9f7740 1 -- 172.21.15.49:0/2666158272 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b319299f80 con 0x55b319345d00 2024-03-18T00:56:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.740+0000 7f7472ffd640 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b319345d00 0x55b319348170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.750+0000 7f7472ffd640 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b319345d00 0x55b319348170 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ee4f0d5556ca89f server_cookie=69fb6cf14b6e859e in_seq=0 out_seq=0 2024-03-18T00:56:30.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.752+0000 7f7452ffd640 10 client.26237 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.757+0000 7f7452ffd640 1 -- 172.21.15.49:0/2666158272 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b319299f80 con 0x55b319345d00 2024-03-18T00:56:30.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.757+0000 7f7452ffd640 10 client.26237 handle_command_reply: tid=0 2024-03-18T00:56:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.757+0000 7f747c9f7740 1 client.26237 shutdown 2024-03-18T00:56:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.758+0000 7f747c9f7740 1 -- 172.21.15.49:0/2666158272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 msgr2=0x55b319339f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.758+0000 7f747c9f7740 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339f70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f746800f670 tx=0x7f746800fb30 comp rx=0 tx=0).stop 2024-03-18T00:56:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.758+0000 7f747c9f7740 1 -- 172.21.15.49:0/2666158272 shutdown_connections 2024-03-18T00:56:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.758+0000 7f747c9f7740 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b319345d00 0x55b319348170 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.761 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.758+0000 7f747c9f7740 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b319341480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.758+0000 7f747c9f7740 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b31933a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.758+0000 7f747c9f7740 1 --2- 172.21.15.49:0/2666158272 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339f70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.758+0000 7f747c9f7740 1 -- 172.21.15.49:0/2666158272 >> 172.21.15.49:0/2666158272 conn(0x55b3193228b0 msgr2=0x55b319321bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.759+0000 7f7452ffd640 0 client.26237 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.760+0000 7f747c9f7740 1 -- 172.21.15.49:0/2666158272 shutdown_connections 2024-03-18T00:56:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.760+0000 7f747c9f7740 1 -- 172.21.15.49:0/2666158272 wait complete. 2024-03-18T00:56:30.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.760+0000 7f747c9f7740 20 client.26237 trim_cache size 0 max 16384 2024-03-18T00:56:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.795+0000 7f747c9f7740 1 Processor -- start 2024-03-18T00:56:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.802+0000 7f747c9f7740 1 -- start start 2024-03-18T00:56:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.803+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.803+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b319341440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.803+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b319339ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.803+0000 7f747c9f7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f74741c03b0 con 0x55b319330540 2024-03-18T00:56:30.804 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.803+0000 7f747c9f7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f74741c1ef0 con 0x55b31932fb50 2024-03-18T00:56:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.803+0000 7f74737fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.804+0000 7f74737fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:53184/0 (socket says 172.21.15.49:53184) 2024-03-18T00:56:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.804+0000 7f74737fe640 1 -- 172.21.15.49:0/3444308447 learned_addr learned my addr 172.21.15.49:0/3444308447 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.804+0000 7f747c9f7740 1 -- 172.21.15.49:0/3444308447 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f74741c0a80 con 0x55b3193356b0 2024-03-18T00:56:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.806+0000 7f7472ffd640 1 --2- 172.21.15.49:0/3444308447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b319341440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.806+0000 7f7473fff640 1 --2- 172.21.15.49:0/3444308447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b319339ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.807+0000 7f74737fe640 1 -- 172.21.15.49:0/3444308447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 msgr2=0x55b319339ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.807+0000 7f74737fe640 1 --2- 172.21.15.49:0/3444308447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b319339ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.807+0000 7f74737fe640 1 -- 172.21.15.49:0/3444308447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 msgr2=0x55b319341440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.807+0000 7f74737fe640 1 --2- 172.21.15.49:0/3444308447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b319341440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.807+0000 7f74737fe640 1 -- 172.21.15.49:0/3444308447 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3193302b0 con 0x55b31932fb50 2024-03-18T00:56:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.812+0000 7f74737fe640 1 --2- 172.21.15.49:0/3444308447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f74680109f0 tx=0x7f746800af50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c94dc9acff3578 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.816+0000 7f74597fa640 1 -- 172.21.15.49:0/3444308447 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468027040 con 0x55b31932fb50 2024-03-18T00:56:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.816+0000 7f74597fa640 1 -- 172.21.15.49:0/3444308447 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746800d8a0 con 0x55b31932fb50 2024-03-18T00:56:30.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.816+0000 7f74597fa640 1 -- 172.21.15.49:0/3444308447 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468004460 con 0x55b31932fb50 2024-03-18T00:56:30.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.818+0000 7f747c9f7740 1 -- 172.21.15.49:0/3444308447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 msgr2=0x55b319339590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:30.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.818+0000 7f747c9f7740 1 --2- 172.21.15.49:0/3444308447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f74680109f0 tx=0x7f746800af50 comp rx=0 tx=0).stop 2024-03-18T00:56:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.818+0000 7f747c9f7740 1 -- 172.21.15.49:0/3444308447 shutdown_connections 2024-03-18T00:56:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.819+0000 7f747c9f7740 1 --2- 172.21.15.49:0/3444308447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b319339ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.819+0000 7f747c9f7740 1 --2- 172.21.15.49:0/3444308447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b319341440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.819+0000 7f747c9f7740 1 --2- 172.21.15.49:0/3444308447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b319339590 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.819+0000 7f747c9f7740 1 -- 172.21.15.49:0/3444308447 >> 172.21.15.49:0/3444308447 conn(0x55b3193228b0 msgr2=0x55b3193263a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.824+0000 7f747c9f7740 1 -- 172.21.15.49:0/3444308447 shutdown_connections 2024-03-18T00:56:30.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.824+0000 7f747c9f7740 1 -- 172.21.15.49:0/3444308447 wait complete. 2024-03-18T00:56:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.825+0000 7f747c9f7740 1 -- start start 2024-03-18T00:56:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.825+0000 7f747c9f7740 1 Processor -- start 2024-03-18T00:56:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.828+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31933e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.828+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b31933ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.829+0000 7f747c9f7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b31933fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.829+0000 7f747c9f7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f74741c1820 con 0x55b319330540 2024-03-18T00:56:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.829+0000 7f747c9f7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f74741c1150 con 0x55b31932fb50 2024-03-18T00:56:30.831 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.829+0000 7f747c9f7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f74741be390 con 0x55b3193356b0 2024-03-18T00:56:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.829+0000 7f747c9f7740 10 client.0 _ll_register_callbacks cb 0x7f746800a440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.830+0000 7f74737fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31933e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.831+0000 7f7473fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b31933fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.831+0000 7f74737fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31933e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:53190/0 (socket says 172.21.15.49:53190) 2024-03-18T00:56:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.831+0000 7f74737fe640 1 -- 172.21.15.49:0/1170963286 learned_addr learned my addr 172.21.15.49:0/1170963286 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.834+0000 7f7473fff640 1 -- 172.21.15.49:0/1170963286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 msgr2=0x55b31933e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.834+0000 7f7473fff640 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31933e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.834+0000 7f7473fff640 1 -- 172.21.15.49:0/1170963286 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 msgr2=0x55b31933ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.834+0000 7f7473fff640 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b31933ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:30.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.835+0000 7f7473fff640 1 -- 172.21.15.49:0/1170963286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b319340410 con 0x55b3193356b0 2024-03-18T00:56:30.837 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.836+0000 7f7473fff640 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b31933fe20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f746c032500 tx=0x7f746c034730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8368b2fd0fdd8678 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.839+0000 7f7452ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:56:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.839+0000 7f7452ffd640 1 -- 172.21.15.49:0/1170963286 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746c058020 con 0x55b3193356b0 2024-03-18T00:56:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.839+0000 7f7452ffd640 1 -- 172.21.15.49:0/1170963286 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f746c004b40 con 0x55b3193356b0 2024-03-18T00:56:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.841+0000 7f747c9f7740 1 -- 172.21.15.49:0/1170963286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f746800d8a0 con 0x55b3193356b0 2024-03-18T00:56:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.848+0000 7f7452ffd640 1 -- 172.21.15.49:0/1170963286 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f746c050040 con 0x55b3193356b0 2024-03-18T00:56:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.849+0000 7f747c9f7740 10 client.26329 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.849+0000 7f747c9f7740 1 -- 172.21.15.49:0/1170963286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b319341720 con 0x55b3193356b0 2024-03-18T00:56:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.854+0000 7f7452ffd640 1 -- 172.21.15.49:0/1170963286 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f746c009540 con 0x55b3193356b0 2024-03-18T00:56:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.854+0000 7f7452ffd640 1 -- 172.21.15.49:0/1170963286 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f746c053070 con 0x55b3193356b0 2024-03-18T00:56:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.855+0000 7f747c9f7740 10 client.26329 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.855+0000 7f747c9f7740 10 client.26329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.855+0000 7f747c9f7740 20 client.26329 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.855+0000 7f747c9f7740 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b3193ac6f0 0x55b3193aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.856+0000 7f747c9f7740 4 client.26329 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:56:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.856+0000 7f747c9f7740 1 -- 172.21.15.49:0/1170963286 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f746800d8a0 con 0x55b3193ac6f0 2024-03-18T00:56:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.858+0000 7f74737fe640 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b3193ac6f0 0x55b3193aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.862+0000 7f74737fe640 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b3193ac6f0 0x55b3193aeb10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=596f67c0dc4d2320 server_cookie=45781aaf8a60d316 in_seq=0 out_seq=0 2024-03-18T00:56:30.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:30.863+0000 7f7452ffd640 10 client.26329 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:31.928 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:31 smithi049 ceph-mon[32130]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 934 MiB/s wr, 2.31k op/s 2024-03-18T00:56:31.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:31 smithi107 ceph-mon[36270]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 934 MiB/s wr, 2.31k op/s 2024-03-18T00:56:31.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:31 smithi106 ceph-mon[35899]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 934 MiB/s wr, 2.31k op/s 2024-03-18T00:56:33.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:33 smithi106 ceph-mon[35899]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 917 MiB/s wr, 2.33k op/s 2024-03-18T00:56:34.128 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:33 smithi049 ceph-mon[32130]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 917 MiB/s wr, 2.33k op/s 2024-03-18T00:56:34.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:33 smithi107 ceph-mon[36270]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 917 MiB/s wr, 2.33k op/s 2024-03-18T00:56:35.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:34 smithi049 ceph-mon[32130]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 882 MiB/s wr, 2.25k op/s 2024-03-18T00:56:35.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:34 smithi107 ceph-mon[36270]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 882 MiB/s wr, 2.25k op/s 2024-03-18T00:56:35.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:34 smithi106 ceph-mon[35899]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 882 MiB/s wr, 2.25k op/s 2024-03-18T00:56:36.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:35 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:56:36.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:35 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:56:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:35 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:56:36.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:35 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:56:36.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:35 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:56:36.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:35 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:56:37.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:36 smithi049 ceph-mon[32130]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 873 MiB/s wr, 2.12k op/s 2024-03-18T00:56:37.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:36 smithi107 ceph-mon[36270]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 873 MiB/s wr, 2.12k op/s 2024-03-18T00:56:37.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:36 smithi106 ceph-mon[35899]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 873 MiB/s wr, 2.12k op/s 2024-03-18T00:56:39.127 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:38 smithi049 ceph-mon[32130]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 900 MiB/s wr, 2.01k op/s 2024-03-18T00:56:39.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:38 smithi107 ceph-mon[36270]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 900 MiB/s wr, 2.01k op/s 2024-03-18T00:56:39.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:38 smithi106 ceph-mon[35899]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 900 MiB/s wr, 2.01k op/s 2024-03-18T00:56:39.652 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.651+0000 7f7452ffd640 1 -- 172.21.15.49:0/1170963286 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f746800d8a0 con 0x55b3193ac6f0 2024-03-18T00:56:39.653 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.651+0000 7f7452ffd640 10 client.26329 handle_command_reply: tid=0 2024-03-18T00:56:39.653 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.652+0000 7f747c9f7740 1 client.26329 shutdown 2024-03-18T00:56:39.653 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.652+0000 7f747c9f7740 1 -- 172.21.15.49:0/1170963286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 msgr2=0x55b31933fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:39.653 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.652+0000 7f747c9f7740 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b31933fe20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f746c032500 tx=0x7f746c034730 comp rx=0 tx=0).stop 2024-03-18T00:56:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.656+0000 7f747c9f7740 1 -- 172.21.15.49:0/1170963286 shutdown_connections 2024-03-18T00:56:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.656+0000 7f747c9f7740 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b3193ac6f0 0x55b3193aeb10 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.656+0000 7f747c9f7740 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b3193356b0 0x55b31933fe20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.656+0000 7f747c9f7740 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b319330540 0x55b31933ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.656+0000 7f747c9f7740 1 --2- 172.21.15.49:0/1170963286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b31932fb50 0x55b31933e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.656+0000 7f747c9f7740 1 -- 172.21.15.49:0/1170963286 >> 172.21.15.49:0/1170963286 conn(0x55b3193228b0 msgr2=0x55b319321320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:39.656 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.656+0000 7f7452ffd640 0 client.26329 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:39.662 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.661+0000 7f747c9f7740 1 -- 172.21.15.49:0/1170963286 shutdown_connections 2024-03-18T00:56:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.662+0000 7f747c9f7740 1 -- 172.21.15.49:0/1170963286 wait complete. 2024-03-18T00:56:39.663 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.662+0000 7f747c9f7740 20 client.26329 trim_cache size 0 max 16384 2024-03-18T00:56:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:56:39.664 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:56:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:56:39.665 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:56:39.669 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.668+0000 7f7479d37640 1 -- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f745c0a2b10 msgr2=0x7f745c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:39.669 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.668+0000 7f7479d37640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f745c0a2b10 0x7f745c0a4fd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f746c009a60 tx=0x7f746c009910 comp rx=0 tx=0).stop 2024-03-18T00:56:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.668+0000 7f7479d37640 1 -- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474076580 msgr2=0x7f74741b0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:39.670 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.668+0000 7f7479d37640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474076580 0x7f74741b0680 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f746400c890 tx=0x7f746400cd50 comp rx=0 tx=0).stop 2024-03-18T00:56:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.669+0000 7f7479d37640 1 -- 172.21.15.49:0/2729182167 shutdown_connections 2024-03-18T00:56:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.669+0000 7f7479d37640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f745c0a2b10 0x7f745c0a4fd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.669+0000 7f7479d37640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7474078f30 0x7f74741b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.669+0000 7f7479d37640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7474076580 0x7f74741b0680 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.669+0000 7f7479d37640 1 --2- 172.21.15.49:0/2729182167 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f74740736a0 0x7f74741b0140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:39.671 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.669+0000 7f7479d37640 1 -- 172.21.15.49:0/2729182167 >> 172.21.15.49:0/2729182167 conn(0x7f747406e820 msgr2=0x7f7474078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:39.672 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.671+0000 7f7479d37640 1 -- 172.21.15.49:0/2729182167 shutdown_connections 2024-03-18T00:56:39.673 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:39.672+0000 7f7479d37640 1 -- 172.21.15.49:0/2729182167 wait complete. 2024-03-18T00:56:41.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:40 smithi049 ceph-mon[32130]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 911 MiB/s wr, 1.88k op/s 2024-03-18T00:56:41.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:40 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:56:41.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:40 smithi107 ceph-mon[36270]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 911 MiB/s wr, 1.88k op/s 2024-03-18T00:56:41.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:40 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:56:41.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:40 smithi106 ceph-mon[35899]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 911 MiB/s wr, 1.88k op/s 2024-03-18T00:56:41.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:40 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:56:43.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:42 smithi049 ceph-mon[32130]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 928 MiB/s wr, 1.70k op/s 2024-03-18T00:56:43.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:42 smithi107 ceph-mon[36270]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 928 MiB/s wr, 1.70k op/s 2024-03-18T00:56:43.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:42 smithi106 ceph-mon[35899]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 928 MiB/s wr, 1.70k op/s 2024-03-18T00:56:45.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:44 smithi049 ceph-mon[32130]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 896 MiB/s wr, 1.50k op/s 2024-03-18T00:56:45.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:44 smithi107 ceph-mon[36270]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 896 MiB/s wr, 1.50k op/s 2024-03-18T00:56:45.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:44 smithi106 ceph-mon[35899]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 896 MiB/s wr, 1.50k op/s 2024-03-18T00:56:46.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:45 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi049 to 176.2M 2024-03-18T00:56:46.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:45 smithi049 ceph-mon[32130]: Unable to set osd_memory_target on smithi049 to 184763084: error parsing value: Value '184763084' is below minimum 939524096 2024-03-18T00:56:46.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:56:46.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:56:46.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:45 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:56:46.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:45 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi049 to 176.2M 2024-03-18T00:56:46.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:45 smithi107 ceph-mon[36270]: Unable to set osd_memory_target on smithi049 to 184763084: error parsing value: Value '184763084' is below minimum 939524096 2024-03-18T00:56:46.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:56:46.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:56:46.181 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:45 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:56:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:56:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:45 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi049 to 176.2M 2024-03-18T00:56:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:45 smithi106 ceph-mon[35899]: Unable to set osd_memory_target on smithi049 to 184763084: error parsing value: Value '184763084' is below minimum 939524096 2024-03-18T00:56:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:56:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:56:46.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:45 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:56:47.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:47 smithi107 ceph-mon[36270]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 883 MiB/s wr, 1.48k op/s 2024-03-18T00:56:47.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:47 smithi106 ceph-mon[35899]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 883 MiB/s wr, 1.48k op/s 2024-03-18T00:56:47.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:47 smithi049 ceph-mon[32130]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 883 MiB/s wr, 1.48k op/s 2024-03-18T00:56:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:49 smithi107 ceph-mon[36270]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 909 MiB/s wr, 1.63k op/s 2024-03-18T00:56:49.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:49 smithi106 ceph-mon[35899]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 909 MiB/s wr, 1.63k op/s 2024-03-18T00:56:49.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:49 smithi049 ceph-mon[32130]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 909 MiB/s wr, 1.63k op/s 2024-03-18T00:56:51.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:51 smithi107 ceph-mon[36270]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 898 MiB/s wr, 1.67k op/s 2024-03-18T00:56:51.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:51 smithi106 ceph-mon[35899]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 898 MiB/s wr, 1.67k op/s 2024-03-18T00:56:51.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:51 smithi049 ceph-mon[32130]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 898 MiB/s wr, 1.67k op/s 2024-03-18T00:56:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.058+0000 7fe2b0627640 1 -- 172.21.15.49:0/3871495308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a8103a30 msgr2=0x7fe2a8103e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:53.058 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.058+0000 7fe2b0627640 1 --2- 172.21.15.49:0/3871495308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a8103a30 0x7fe2a8103e50 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe298009900 tx=0x7fe29802bc80 comp rx=0 tx=0).stop 2024-03-18T00:56:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.062+0000 7fe2b0627640 1 -- 172.21.15.49:0/3871495308 shutdown_connections 2024-03-18T00:56:53.065 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.062+0000 7fe2b0627640 1 --2- 172.21.15.49:0/3871495308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a81055e0 0x7fe2a810be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.062+0000 7fe2b0627640 1 --2- 172.21.15.49:0/3871495308 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2a8104c20 0x7fe2a81050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.062+0000 7fe2b0627640 1 --2- 172.21.15.49:0/3871495308 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a8103a30 0x7fe2a8103e50 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.062+0000 7fe2b0627640 1 -- 172.21.15.49:0/3871495308 >> 172.21.15.49:0/3871495308 conn(0x7fe2a80fefa0 msgr2=0x7fe2a81013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:53.067 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.068+0000 7fe2b0627640 1 -- 172.21.15.49:0/3871495308 shutdown_connections 2024-03-18T00:56:53.072 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.071+0000 7fe2b0627640 1 -- 172.21.15.49:0/3871495308 wait complete. 2024-03-18T00:56:53.073 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.073+0000 7fe2b0627640 1 Processor -- start 2024-03-18T00:56:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.074+0000 7fe2b0627640 1 -- start start 2024-03-18T00:56:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.074+0000 7fe2b0627640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2a8103a30 0x7fe2a8070040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.074+0000 7fe2b0627640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a8104c20 0x7fe2a8070580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.074+0000 7fe2b0627640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a81055e0 0x7fe2a81ae650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.074+0000 7fe2b0627640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a810f750 con 0x7fe2a8103a30 2024-03-18T00:56:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.074+0000 7fe2b0627640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a810f5d0 con 0x7fe2a8104c20 2024-03-18T00:56:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.074+0000 7fe2b0627640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a810f8d0 con 0x7fe2a81055e0 2024-03-18T00:56:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.074+0000 7fe2ae3c3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2a8103a30 0x7fe2a8070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.075+0000 7fe2ae3c3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2a8103a30 0x7fe2a8070040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43844/0 (socket says 172.21.15.49:43844) 2024-03-18T00:56:53.075 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.075+0000 7fe2ae3c3640 1 -- 172.21.15.49:0/2767380885 learned_addr learned my addr 172.21.15.49:0/2767380885 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.076+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a81055e0 0x7fe2a81ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.076 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.076+0000 7fe2ae3c3640 1 -- 172.21.15.49:0/2767380885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a81055e0 msgr2=0x7fe2a81ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.076+0000 7fe2ae3c3640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a81055e0 0x7fe2a81ae650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.076+0000 7fe2ae3c3640 1 -- 172.21.15.49:0/2767380885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a8104c20 msgr2=0x7fe2a8070580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:56:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.076+0000 7fe2ae3c3640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a8104c20 0x7fe2a8070580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.077 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.076+0000 7fe2ae3c3640 1 -- 172.21.15.49:0/2767380885 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a8079d70 con 0x7fe2a8103a30 2024-03-18T00:56:53.078 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.078+0000 7fe2ae3c3640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2a8103a30 0x7fe2a8070040 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe298004bb0 tx=0x7fe298004fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.084+0000 7fe29f7fe640 1 -- 172.21.15.49:0/2767380885 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298039040 con 0x7fe2a8103a30 2024-03-18T00:56:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.084+0000 7fe29f7fe640 1 -- 172.21.15.49:0/2767380885 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe298043070 con 0x7fe2a8103a30 2024-03-18T00:56:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.084+0000 7fe29f7fe640 1 -- 172.21.15.49:0/2767380885 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29803e670 con 0x7fe2a8103a30 2024-03-18T00:56:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.084+0000 7fe2b0627640 1 -- 172.21.15.49:0/2767380885 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a80735d0 con 0x7fe2a8103a30 2024-03-18T00:56:53.087 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.085+0000 7fe2b0627640 1 -- 172.21.15.49:0/2767380885 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a81a6020 con 0x7fe2a8103a30 2024-03-18T00:56:53.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.100+0000 7fe29f7fe640 1 -- 172.21.15.49:0/2767380885 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe29803e810 con 0x7fe2a8103a30 2024-03-18T00:56:53.102 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.102+0000 7fe2b3213740 1 Processor -- start 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.102+0000 7fe2b3213740 1 -- start start 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.102+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57ba340 0x5649c57ba760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 0x5649c57bf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bfea0 0x5649c57c2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2b3213740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5649c571fa40 con 0x5649c57ba340 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2b3213740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5649c56c8920 con 0x5649c57bad30 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2b3213740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5649c571bae0 con 0x5649c57bfea0 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2aebc4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57ba340 0x5649c57ba760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2aebc4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57ba340 0x5649c57ba760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43852/0 (socket says 172.21.15.49:43852) 2024-03-18T00:56:53.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2aebc4640 1 -- 172.21.15.49:0/1463802939 learned_addr learned my addr 172.21.15.49:0/1463802939 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2aebc4640 1 -- 172.21.15.49:0/1463802939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bfea0 msgr2=0x5649c57c2330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:56:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/1463802939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bfea0 0x5649c57c2330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2aebc4640 1 -- 172.21.15.49:0/1463802939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 msgr2=0x5649c57bf830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/1463802939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 0x5649c57bf830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.103+0000 7fe2aebc4640 1 -- 172.21.15.49:0/1463802939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c56e0460 con 0x5649c57ba340 2024-03-18T00:56:53.104 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.104+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/1463802939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57ba340 0x5649c57ba760 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009ee0 tx=0x7fe2a0010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4561e52ba692ced7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:53.105 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.106+0000 7fe29f7fe640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2940a2ad0 0x7fe2940a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.106 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.107+0000 7fe29f7fe640 1 -- 172.21.15.49:0/2767380885 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe2980f1520 con 0x7fe2a8103a30 2024-03-18T00:56:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.108+0000 7fe29cbfe640 1 -- 172.21.15.49:0/1463802939 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0016070 con 0x5649c57ba340 2024-03-18T00:56:53.107 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.108+0000 7fe29cbfe640 1 -- 172.21.15.49:0/1463802939 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a0004d20 con 0x5649c57ba340 2024-03-18T00:56:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.109+0000 7fe2b3213740 1 -- 172.21.15.49:0/1463802939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57ba340 msgr2=0x5649c57ba760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.109+0000 7fe2b3213740 1 --2- 172.21.15.49:0/1463802939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57ba340 0x5649c57ba760 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0009ee0 tx=0x7fe2a0010740 comp rx=0 tx=0).stop 2024-03-18T00:56:53.110 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.108+0000 7fe29cbfe640 1 -- 172.21.15.49:0/1463802939 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0012810 con 0x5649c57ba340 2024-03-18T00:56:53.112 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.113+0000 7fe2ae3c3640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2940a2ad0 0x7fe2940a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.114+0000 7fe2b3213740 1 -- 172.21.15.49:0/1463802939 shutdown_connections 2024-03-18T00:56:53.114 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.114+0000 7fe2b3213740 1 --2- 172.21.15.49:0/1463802939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 0x5649c57bf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.115+0000 7fe2b3213740 1 --2- 172.21.15.49:0/1463802939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bfea0 0x5649c57c2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.115+0000 7fe2b3213740 1 --2- 172.21.15.49:0/1463802939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57ba340 0x5649c57ba760 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.115 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.115+0000 7fe2b3213740 1 -- 172.21.15.49:0/1463802939 >> 172.21.15.49:0/1463802939 conn(0x5649c57b58b0 msgr2=0x5649c57b5cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:53.116 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.116+0000 7fe2ae3c3640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2940a2ad0 0x7fe2940a4f90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe2980c60b0 tx=0x7fe298005be0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:53.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.118+0000 7fe2b3213740 1 -- 172.21.15.49:0/1463802939 shutdown_connections 2024-03-18T00:56:53.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.118+0000 7fe2b3213740 1 -- 172.21.15.49:0/1463802939 wait complete. 2024-03-18T00:56:53.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.119+0000 7fe2b3213740 1 -- start start 2024-03-18T00:56:53.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.119+0000 7fe2b3213740 1 Processor -- start 2024-03-18T00:56:53.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.121+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57ba340 0x5649c57c4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.121+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 0x5649c57c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.121+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 0x5649c57cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.122+0000 7fe2b3213740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5649c57c2ea0 con 0x5649c57bfea0 2024-03-18T00:56:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.123+0000 7fe2b3213740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5649c56dbd70 con 0x5649c57bad30 2024-03-18T00:56:53.123 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.123+0000 7fe2b3213740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5649c57c3020 con 0x5649c57ba340 2024-03-18T00:56:53.124 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.123+0000 7fe2b3213740 10 client.0 _ll_register_callbacks cb 0x5649c5433a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.124+0000 7fe2adbc2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 0x5649c57c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.125+0000 7fe2adbc2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 0x5649c57c4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:43270/0 (socket says 172.21.15.49:43270) 2024-03-18T00:56:53.125 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.125+0000 7fe2adbc2640 1 -- 172.21.15.49:0/2188396950 learned_addr learned my addr 172.21.15.49:0/2188396950 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:53.126 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.125+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57ba340 0x5649c57c4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.127 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.126+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 0x5649c57cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.126+0000 7fe2adbc2640 1 -- 172.21.15.49:0/2188396950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57ba340 msgr2=0x5649c57c4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.126+0000 7fe2adbc2640 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57ba340 0x5649c57c4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.127+0000 7fe2adbc2640 1 -- 172.21.15.49:0/2188396950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 msgr2=0x5649c57cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.127+0000 7fe2adbc2640 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 0x5649c57cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.127+0000 7fe2adbc2640 1 -- 172.21.15.49:0/2188396950 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c5434210 con 0x5649c57bad30 2024-03-18T00:56:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.128+0000 7fe2adbc2640 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 0x5649c57c4c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009dd0 tx=0x7fe2a400b4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8de16e20de87159e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.132+0000 7fe291ffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:56:53.134 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.134+0000 7fe291ffb640 1 -- 172.21.15.49:0/2188396950 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4015020 con 0x5649c57bad30 2024-03-18T00:56:53.135 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.134+0000 7fe2b3213740 1 -- 172.21.15.49:0/2188396950 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2a0004d20 con 0x5649c57bad30 2024-03-18T00:56:53.136 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.134+0000 7fe291ffb640 1 -- 172.21.15.49:0/2188396950 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe2a4011420 con 0x5649c57bad30 2024-03-18T00:56:53.137 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.136+0000 7fe291ffb640 1 -- 172.21.15.49:0/2188396950 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe2a4020050 con 0x5649c57bad30 2024-03-18T00:56:53.137 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.136+0000 7fe2b3213740 10 client.26249 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:53.138 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.136+0000 7fe2b3213740 1 -- 172.21.15.49:0/2188396950 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649c5500620 con 0x5649c57bad30 2024-03-18T00:56:53.139 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.140+0000 7fe291ffb640 1 -- 172.21.15.49:0/2188396950 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a4011b20 con 0x5649c57bad30 2024-03-18T00:56:53.139 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.140+0000 7fe291ffb640 1 -- 172.21.15.49:0/2188396950 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe2a4019b50 con 0x5649c57bad30 2024-03-18T00:56:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.140+0000 7fe2b3213740 10 client.26249 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.140+0000 7fe2b3213740 10 client.26249 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.140+0000 7fe2b3213740 20 client.26249 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.140+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649c57d08b0 0x5649c57d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.140+0000 7fe2b3213740 4 client.26249 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:53.140 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.140+0000 7fe2b3213740 1 -- 172.21.15.49:0/2188396950 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe2a0004d20 con 0x5649c57d08b0 2024-03-18T00:56:53.145 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.146+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649c57d08b0 0x5649c57d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.168+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649c57d08b0 0x5649c57d2cd0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf365c4736f2668f server_cookie=6e1e25fd42292b54 in_seq=0 out_seq=0 2024-03-18T00:56:53.169 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.168+0000 7fe291ffb640 10 client.26249 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:53.178 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.175+0000 7fe291ffb640 1 -- 172.21.15.49:0/2188396950 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fe2a0004d20 con 0x5649c57d08b0 2024-03-18T00:56:53.178 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.175+0000 7fe291ffb640 10 client.26249 handle_command_reply: tid=0 2024-03-18T00:56:53.178 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.175+0000 7fe2b3213740 1 client.26249 shutdown 2024-03-18T00:56:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe2b3213740 1 -- 172.21.15.49:0/2188396950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 msgr2=0x5649c57c4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 0x5649c57c4c50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009dd0 tx=0x7fe2a400b4b0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe2b3213740 1 -- 172.21.15.49:0/2188396950 shutdown_connections 2024-03-18T00:56:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649c57d08b0 0x5649c57d2cd0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57bad30 0x5649c57c4c50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 0x5649c57cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2188396950 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57ba340 0x5649c57c4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe291ffb640 0 client.26249 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:53.179 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe2b3213740 1 -- 172.21.15.49:0/2188396950 >> 172.21.15.49:0/2188396950 conn(0x5649c57b58b0 msgr2=0x5649c57b4bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.179+0000 7fe2b3213740 1 -- 172.21.15.49:0/2188396950 shutdown_connections 2024-03-18T00:56:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.180+0000 7fe2b3213740 1 -- 172.21.15.49:0/2188396950 wait complete. 2024-03-18T00:56:53.180 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.180+0000 7fe2b3213740 20 client.26249 trim_cache size 0 max 16384 2024-03-18T00:56:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.237+0000 7fe2b3213740 1 Processor -- start 2024-03-18T00:56:53.236 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.237+0000 7fe2b3213740 1 -- start start 2024-03-18T00:56:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.238+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 0x5649c57b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.237 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.238+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bad30 0x5649c57cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.238 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.238+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 0x5649c57b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.238+0000 7fe2aebc4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 0x5649c57b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.238+0000 7fe2aebc4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 0x5649c57b0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:43286/0 (socket says 172.21.15.49:43286) 2024-03-18T00:56:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.239+0000 7fe2adbc2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bad30 0x5649c57cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.239+0000 7fe2adbc2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bad30 0x5649c57cbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46198/0 (socket says 172.21.15.49:46198) 2024-03-18T00:56:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.239+0000 7fe2aebc4640 1 -- 172.21.15.49:0/2542018231 learned_addr learned my addr 172.21.15.49:0/2542018231 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.239+0000 7fe2b3213740 1 -- 172.21.15.49:0/2542018231 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c00e690 con 0x5649c57bfea0 2024-03-18T00:56:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.239+0000 7fe2b3213740 1 -- 172.21.15.49:0/2542018231 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c0101d0 con 0x5649c57ba340 2024-03-18T00:56:53.239 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.239+0000 7fe2b3213740 1 -- 172.21.15.49:0/2542018231 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c00ed60 con 0x5649c57bad30 2024-03-18T00:56:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.239+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2542018231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 0x5649c57b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.240+0000 7fe2aebc4640 1 -- 172.21.15.49:0/2542018231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bad30 msgr2=0x5649c57cbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.240+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2542018231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bad30 0x5649c57cbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.240+0000 7fe2aebc4640 1 -- 172.21.15.49:0/2542018231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 msgr2=0x5649c57b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.240+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2542018231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 0x5649c57b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.240 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.240+0000 7fe2aebc4640 1 -- 172.21.15.49:0/2542018231 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649c57b6620 con 0x5649c57bfea0 2024-03-18T00:56:53.241 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.242+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2542018231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 0x5649c57b8b10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0010fc0 tx=0x7fe2a0005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7124f98aa08c425 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.251+0000 7fe293fff640 1 -- 172.21.15.49:0/2542018231 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0016040 con 0x5649c57bfea0 2024-03-18T00:56:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.251+0000 7fe293fff640 1 -- 172.21.15.49:0/2542018231 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a002b500 con 0x5649c57bfea0 2024-03-18T00:56:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.251+0000 7fe293fff640 1 -- 172.21.15.49:0/2542018231 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0012d00 con 0x5649c57bfea0 2024-03-18T00:56:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.253+0000 7fe2b3213740 1 -- 172.21.15.49:0/2542018231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 msgr2=0x5649c57b8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:53.252 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.253+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2542018231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 0x5649c57b8b10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0010fc0 tx=0x7fe2a0005a30 comp rx=0 tx=0).stop 2024-03-18T00:56:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.254+0000 7fe2b3213740 1 -- 172.21.15.49:0/2542018231 shutdown_connections 2024-03-18T00:56:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.254+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2542018231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bfea0 0x5649c57b8b10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.254+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2542018231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bad30 0x5649c57cbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.254+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2542018231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 0x5649c57b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.254+0000 7fe2b3213740 1 -- 172.21.15.49:0/2542018231 >> 172.21.15.49:0/2542018231 conn(0x5649c57b58b0 msgr2=0x5649c57b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:53.254 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.254+0000 7fe2b3213740 1 -- 172.21.15.49:0/2542018231 shutdown_connections 2024-03-18T00:56:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 1 -- 172.21.15.49:0/2542018231 wait complete. 2024-03-18T00:56:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 1 -- start start 2024-03-18T00:56:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 1 Processor -- start 2024-03-18T00:56:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 0x5649c57c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bad30 0x5649c57c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bfea0 0x5649c57ca800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c00fb00 con 0x5649c57bad30 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c00f430 con 0x5649c57ba340 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c00c670 con 0x5649c57bfea0 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.255+0000 7fe2b3213740 10 client.0 _ll_register_callbacks cb 0x7fe2a4006ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.256+0000 7fe2adbc2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bad30 0x5649c57c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.256+0000 7fe2adbc2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bad30 0x5649c57c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43870/0 (socket says 172.21.15.49:43870) 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.256+0000 7fe2adbc2640 1 -- 172.21.15.49:0/2418761447 learned_addr learned my addr 172.21.15.49:0/2418761447 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.256+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bfea0 0x5649c57ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.256+0000 7fe2adbc2640 1 -- 172.21.15.49:0/2418761447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bfea0 msgr2=0x5649c57ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.256+0000 7fe2adbc2640 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bfea0 0x5649c57ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.256 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.256+0000 7fe2adbc2640 1 -- 172.21.15.49:0/2418761447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 msgr2=0x5649c57c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.256+0000 7fe2adbc2640 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 0x5649c57c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:53.257 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.256+0000 7fe2adbc2640 1 -- 172.21.15.49:0/2418761447 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649c57cadf0 con 0x5649c57bad30 2024-03-18T00:56:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.261+0000 7fe2adbc2640 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bad30 0x5649c57c97f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009da0 tx=0x7fe2a4006bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53f0b63a70ffc5d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:53.260 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.261+0000 7fe291ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.262+0000 7fe291ffb640 1 -- 172.21.15.49:0/2418761447 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a4015040 con 0x5649c57bad30 2024-03-18T00:56:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.262+0000 7fe291ffb640 1 -- 172.21.15.49:0/2418761447 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe2a4010950 con 0x5649c57bad30 2024-03-18T00:56:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.262+0000 7fe2b3213740 1 -- 172.21.15.49:0/2418761447 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2a002b500 con 0x5649c57bad30 2024-03-18T00:56:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.263+0000 7fe291ffb640 1 -- 172.21.15.49:0/2418761447 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe2a4010b50 con 0x5649c57bad30 2024-03-18T00:56:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.263+0000 7fe2b3213740 10 client.16989 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.263+0000 7fe2b3213740 1 -- 172.21.15.49:0/2418761447 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649c5836bd0 con 0x5649c57bad30 2024-03-18T00:56:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.265+0000 7fe291ffb640 1 -- 172.21.15.49:0/2418761447 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2a002b500 con 0x5649c57bad30 2024-03-18T00:56:53.265 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.266+0000 7fe291ffb640 1 -- 172.21.15.49:0/2418761447 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fe2a402c030 con 0x5649c57bad30 2024-03-18T00:56:53.266 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.266+0000 7fe2b3213740 10 client.16989 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:53.266 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.266+0000 7fe2b3213740 10 client.16989 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:53.266 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.266+0000 7fe2b3213740 20 client.16989 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:53.266 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.266+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649c58375a0 0x5649c58399c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:53.266 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.266+0000 7fe2b3213740 4 client.16989 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:56:53.267 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.266+0000 7fe2b3213740 1 -- 172.21.15.49:0/2418761447 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649c572cf80 con 0x5649c58375a0 2024-03-18T00:56:53.270 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.270+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649c58375a0 0x5649c58399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.272+0000 7fe2aebc4640 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649c58375a0 0x5649c58399c0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=431f9b261388966f server_cookie=25eeffe0440640f5 in_seq=0 out_seq=0 2024-03-18T00:56:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:53.272+0000 7fe291ffb640 10 client.16989 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:53.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:53 smithi107 ceph-mon[36270]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 944 MiB/s wr, 1.82k op/s 2024-03-18T00:56:53.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:53 smithi106 ceph-mon[35899]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 944 MiB/s wr, 1.82k op/s 2024-03-18T00:56:53.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:53 smithi049 ceph-mon[32130]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 944 MiB/s wr, 1.82k op/s 2024-03-18T00:56:55.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:55 smithi106 ceph-mon[35899]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 865 MiB/s wr, 1.82k op/s 2024-03-18T00:56:55.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:55 smithi049 ceph-mon[32130]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 865 MiB/s wr, 1.82k op/s 2024-03-18T00:56:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:55 smithi107 ceph-mon[36270]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 865 MiB/s wr, 1.82k op/s 2024-03-18T00:56:55.812 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:56:55.812 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:56:56.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.389+0000 7f5420e85640 1 -- 172.21.15.49:0/2601232199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f541c1029e0 msgr2=0x7f541c102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:56.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.389+0000 7f5420e85640 1 --2- 172.21.15.49:0/2601232199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f541c1029e0 0x7f541c102e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5410009860 tx=0x7f541002bbb0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.396+0000 7f5420e85640 1 -- 172.21.15.49:0/2601232199 shutdown_connections 2024-03-18T00:56:56.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.396+0000 7f5420e85640 1 --2- 172.21.15.49:0/2601232199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f541c1033a0 0x7f541c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.396+0000 7f5420e85640 1 --2- 172.21.15.49:0/2601232199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f541c1029e0 0x7f541c102e60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.396+0000 7f5420e85640 1 --2- 172.21.15.49:0/2601232199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f541c1017f0 0x7f541c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.397 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.396+0000 7f5420e85640 1 -- 172.21.15.49:0/2601232199 >> 172.21.15.49:0/2601232199 conn(0x7f541c0fcd80 msgr2=0x7f541c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:56.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.402+0000 7f5420e85640 1 -- 172.21.15.49:0/2601232199 shutdown_connections 2024-03-18T00:56:56.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.405+0000 7f5420e85640 1 -- 172.21.15.49:0/2601232199 wait complete. 2024-03-18T00:56:56.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.406+0000 7f5420e85640 1 Processor -- start 2024-03-18T00:56:56.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.414+0000 7f5420e85640 1 -- start start 2024-03-18T00:56:56.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.414+0000 7f5420e85640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f541c1017f0 0x7f541c195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.414+0000 7f5420e85640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f541c1029e0 0x7f541c195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.414+0000 7f5420e85640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f541c1033a0 0x7f541c1ae450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.414+0000 7f5420e85640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f541c10d540 con 0x7f541c1033a0 2024-03-18T00:56:56.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.414+0000 7f5420e85640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f541c10d3c0 con 0x7f541c1017f0 2024-03-18T00:56:56.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.414+0000 7f5420e85640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f541c10d6c0 con 0x7f541c1029e0 2024-03-18T00:56:56.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.420+0000 7f541a59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f541c1017f0 0x7f541c195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:56.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.423+0000 7f541a59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f541c1017f0 0x7f541c195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:43318/0 (socket says 172.21.15.49:43318) 2024-03-18T00:56:56.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.423+0000 7f541a59c640 1 -- 172.21.15.49:0/3211368440 learned_addr learned my addr 172.21.15.49:0/3211368440 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:56.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.424+0000 7f541ad9d640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f541c1033a0 0x7f541c1ae450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:56.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.425+0000 7f541a59c640 1 -- 172.21.15.49:0/3211368440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f541c1029e0 msgr2=0x7f541c195f60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:56:56.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.425+0000 7f541a59c640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f541c1029e0 0x7f541c195f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.425+0000 7f541a59c640 1 -- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f541c1033a0 msgr2=0x7f541c1ae450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:56.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.425+0000 7f541a59c640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f541c1033a0 0x7f541c1ae450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.425+0000 7f541a59c640 1 -- 172.21.15.49:0/3211368440 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c19b0f0 con 0x7f541c1017f0 2024-03-18T00:56:56.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.425+0000 7f541a59c640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f541c1017f0 0x7f541c195a20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f540c00e9f0 tx=0x7f540c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:56.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.427+0000 7f540b7fe640 1 -- 172.21.15.49:0/3211368440 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c00ccc0 con 0x7f541c1017f0 2024-03-18T00:56:56.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.427+0000 7f5420e85640 1 -- 172.21.15.49:0/3211368440 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c06b200 con 0x7f541c1017f0 2024-03-18T00:56:56.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.427+0000 7f5420e85640 1 -- 172.21.15.49:0/3211368440 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c06b7d0 con 0x7f541c1017f0 2024-03-18T00:56:56.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.427+0000 7f540b7fe640 1 -- 172.21.15.49:0/3211368440 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f540c0035f0 con 0x7f541c1017f0 2024-03-18T00:56:56.428 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.427+0000 7f540b7fe640 1 -- 172.21.15.49:0/3211368440 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c00ce60 con 0x7f541c1017f0 2024-03-18T00:56:56.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.432+0000 7f5423a7c740 1 Processor -- start 2024-03-18T00:56:56.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.432+0000 7f5423a7c740 1 -- start start 2024-03-18T00:56:56.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.433+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236170 0x5570a8236590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.433+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a823b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.433+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a823e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.438+0000 7f541ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236170 0x5570a8236590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:56.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.438+0000 7f541ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236170 0x5570a8236590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43912/0 (socket says 172.21.15.49:43912) 2024-03-18T00:56:56.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.438+0000 7f540b7fe640 1 -- 172.21.15.49:0/3211368440 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f540c004020 con 0x7f541c1017f0 2024-03-18T00:56:56.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.438+0000 7f541ad9d640 1 -- 172.21.15.49:0/13484881 learned_addr learned my addr 172.21.15.49:0/13484881 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:56.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.438+0000 7f5423a7c740 1 -- 172.21.15.49:0/13484881 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5570a819b500 con 0x5570a8236170 2024-03-18T00:56:56.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.438+0000 7f5423a7c740 1 -- 172.21.15.49:0/13484881 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5570a8143be0 con 0x5570a8236b60 2024-03-18T00:56:56.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.438+0000 7f5423a7c740 1 -- 172.21.15.49:0/13484881 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5570a81975a0 con 0x5570a823bcd0 2024-03-18T00:56:56.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.440+0000 7f540b7fe640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f54040a2950 0x7f54040a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.440+0000 7f540b7fe640 1 -- 172.21.15.49:0/3211368440 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f540c0ced20 con 0x7f541c1017f0 2024-03-18T00:56:56.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.442+0000 7f5419d9b640 1 --2- 172.21.15.49:0/13484881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a823e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:56.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.442+0000 7f541a59c640 1 --2- 172.21.15.49:0/13484881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a823b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:56.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.446+0000 7f541ad9d640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f54040a2950 0x7f54040a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:56.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.447+0000 7f541ad9d640 1 -- 172.21.15.49:0/13484881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 msgr2=0x5570a823e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:56.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.447+0000 7f541ad9d640 1 --2- 172.21.15.49:0/13484881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a823e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.447+0000 7f541ad9d640 1 -- 172.21.15.49:0/13484881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 msgr2=0x5570a823b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:56.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.447+0000 7f541ad9d640 1 --2- 172.21.15.49:0/13484881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a823b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.447+0000 7f541ad9d640 1 -- 172.21.15.49:0/13484881 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570a81574b0 con 0x5570a8236170 2024-03-18T00:56:56.451 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.451+0000 7f541ad9d640 1 --2- 172.21.15.49:0/13484881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236170 0x5570a8236590 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f541400ab70 tx=0x7f54140155c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f53b0b741dc554 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:56.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.453+0000 7f541ad9d640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f54040a2950 0x7f54040a4e10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f541401b000 tx=0x7f541401d000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:56.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.460+0000 7f5408ff9640 1 -- 172.21.15.49:0/13484881 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414026070 con 0x5570a8236170 2024-03-18T00:56:56.459 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.460+0000 7f5408ff9640 1 -- 172.21.15.49:0/13484881 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5414019e50 con 0x5570a8236170 2024-03-18T00:56:56.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.460+0000 7f5408ff9640 1 -- 172.21.15.49:0/13484881 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541401d070 con 0x5570a8236170 2024-03-18T00:56:56.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.461+0000 7f5423a7c740 1 -- 172.21.15.49:0/13484881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236170 msgr2=0x5570a8236590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:56.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.461+0000 7f5423a7c740 1 --2- 172.21.15.49:0/13484881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236170 0x5570a8236590 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f541400ab70 tx=0x7f54140155c0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.464+0000 7f5423a7c740 1 -- 172.21.15.49:0/13484881 shutdown_connections 2024-03-18T00:56:56.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.464+0000 7f5423a7c740 1 --2- 172.21.15.49:0/13484881 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a823b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.464+0000 7f5423a7c740 1 --2- 172.21.15.49:0/13484881 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a823e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.464+0000 7f5423a7c740 1 --2- 172.21.15.49:0/13484881 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236170 0x5570a8236590 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.464+0000 7f5423a7c740 1 -- 172.21.15.49:0/13484881 >> 172.21.15.49:0/13484881 conn(0x5570a82316e0 msgr2=0x5570a8231ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:56.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.466+0000 7f5423a7c740 1 -- 172.21.15.49:0/13484881 shutdown_connections 2024-03-18T00:56:56.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.466+0000 7f5423a7c740 1 -- 172.21.15.49:0/13484881 wait complete. 2024-03-18T00:56:56.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.467+0000 7f5423a7c740 1 -- start start 2024-03-18T00:56:56.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.467+0000 7f5423a7c740 1 Processor -- start 2024-03-18T00:56:56.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.469+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236170 0x5570a82406b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.469+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236b60 0x5570a8240bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.469+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a8247c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.469+0000 7f5423a7c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5570a823ec20 con 0x5570a8236b60 2024-03-18T00:56:56.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.469+0000 7f541ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a8247c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:56.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.469+0000 7f5423a7c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5570a8157160 con 0x5570a8236170 2024-03-18T00:56:56.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.469+0000 7f541ad9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a8247c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46240/0 (socket says 172.21.15.49:46240) 2024-03-18T00:56:56.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.469+0000 7f541ad9d640 1 -- 172.21.15.49:0/64204388 learned_addr learned my addr 172.21.15.49:0/64204388 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:56.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.470+0000 7f5423a7c740 1 -- 172.21.15.49:0/64204388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5570a823eda0 con 0x5570a823bcd0 2024-03-18T00:56:56.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.470+0000 7f5419d9b640 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236b60 0x5570a8240bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:56.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.470+0000 7f5423a7c740 10 client.0 _ll_register_callbacks cb 0x5570a813b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:56.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.470+0000 7f541ad9d640 1 -- 172.21.15.49:0/64204388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236170 msgr2=0x5570a82406b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:56.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.471+0000 7f541ad9d640 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236170 0x5570a82406b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.471+0000 7f541ad9d640 1 -- 172.21.15.49:0/64204388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236b60 msgr2=0x5570a8240bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:56.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.471+0000 7f541ad9d640 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236b60 0x5570a8240bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:56.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.471+0000 7f541ad9d640 1 -- 172.21.15.49:0/64204388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570a7f797d0 con 0x5570a823bcd0 2024-03-18T00:56:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.476+0000 7f541ad9d640 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a8247c30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5414021ba0 tx=0x7f5414021bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d79b71b2de20e464 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.476+0000 7f5401ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:56:56.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.477+0000 7f5401ffb640 1 -- 172.21.15.49:0/64204388 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414030030 con 0x5570a823bcd0 2024-03-18T00:56:56.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.477+0000 7f5401ffb640 1 -- 172.21.15.49:0/64204388 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5414019c50 con 0x5570a823bcd0 2024-03-18T00:56:56.478 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.478+0000 7f5423a7c740 1 -- 172.21.15.49:0/64204388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5414019e50 con 0x5570a823bcd0 2024-03-18T00:56:56.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.483+0000 7f5401ffb640 1 -- 172.21.15.49:0/64204388 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5414026040 con 0x5570a823bcd0 2024-03-18T00:56:56.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.483+0000 7f5423a7c740 10 client.26353 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:56.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.483+0000 7f5423a7c740 1 -- 172.21.15.49:0/64204388 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5570a7eaf3f0 con 0x5570a823bcd0 2024-03-18T00:56:56.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.486+0000 7f5401ffb640 1 -- 172.21.15.49:0/64204388 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5414019e50 con 0x5570a823bcd0 2024-03-18T00:56:56.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.486+0000 7f5401ffb640 1 -- 172.21.15.49:0/64204388 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5414004430 con 0x5570a823bcd0 2024-03-18T00:56:56.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.487+0000 7f5423a7c740 10 client.26353 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:56.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.487+0000 7f5423a7c740 10 client.26353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:56.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.487+0000 7f5423a7c740 20 client.26353 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:56.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.487+0000 7f5423a7c740 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5570a824c620 0x5570a824eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:56.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.487+0000 7f5423a7c740 4 client.26353 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:56.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.487+0000 7f5423a7c740 1 -- 172.21.15.49:0/64204388 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570a81a8a40 con 0x5570a824c620 2024-03-18T00:56:56.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.489+0000 7f541a59c640 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5570a824c620 0x5570a824eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:56.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.490+0000 7f541a59c640 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5570a824c620 0x5570a824eac0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21f6cf839d88eca6 server_cookie=a60ac30a80ed780b in_seq=0 out_seq=0 2024-03-18T00:56:56.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:56.491+0000 7f5401ffb640 10 client.26353 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.331+0000 7fe291ffb640 1 -- 172.21.15.49:0/2418761447 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649c572cf80 con 0x5649c58375a0 2024-03-18T00:56:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.331+0000 7fe291ffb640 10 client.16989 handle_command_reply: tid=0 2024-03-18T00:56:57.335 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.331+0000 7fe2b3213740 1 client.16989 shutdown 2024-03-18T00:56:57.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.336+0000 7f5401ffb640 1 -- 172.21.15.49:0/64204388 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5570a81a8a40 con 0x5570a824c620 2024-03-18T00:56:57.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.336+0000 7f5401ffb640 10 client.26353 handle_command_reply: tid=0 2024-03-18T00:56:57.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.337+0000 7f5423a7c740 1 client.26353 shutdown 2024-03-18T00:56:57.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.337+0000 7f5423a7c740 1 -- 172.21.15.49:0/64204388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 msgr2=0x5570a8247c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.337+0000 7f5423a7c740 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a8247c30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5414021ba0 tx=0x7f5414021bd0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.337+0000 7f5423a7c740 1 -- 172.21.15.49:0/64204388 shutdown_connections 2024-03-18T00:56:57.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.337+0000 7f5423a7c740 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5570a824c620 0x5570a824eac0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.337+0000 7f5423a7c740 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a823bcd0 0x5570a8247c30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.337+0000 7f5423a7c740 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a8236b60 0x5570a8240bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.337+0000 7f5423a7c740 1 --2- 172.21.15.49:0/64204388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236170 0x5570a82406b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.337+0000 7f5423a7c740 1 -- 172.21.15.49:0/64204388 >> 172.21.15.49:0/64204388 conn(0x5570a82316e0 msgr2=0x5570a8230900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:57.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.338+0000 7f5401ffb640 0 client.26353 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:57.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.340+0000 7f5423a7c740 1 -- 172.21.15.49:0/64204388 shutdown_connections 2024-03-18T00:56:57.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.341+0000 7f5423a7c740 1 -- 172.21.15.49:0/64204388 wait complete. 2024-03-18T00:56:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.341+0000 7fe2b3213740 1 -- 172.21.15.49:0/2418761447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bad30 msgr2=0x5649c57c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.343 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.341+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bad30 0x5649c57c97f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4009da0 tx=0x7fe2a4006bd0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.341+0000 7fe2b3213740 1 -- 172.21.15.49:0/2418761447 shutdown_connections 2024-03-18T00:56:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.341+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649c58375a0 0x5649c58399c0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.341+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649c57bfea0 0x5649c57ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.341+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649c57bad30 0x5649c57c97f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.341+0000 7fe2b3213740 1 --2- 172.21.15.49:0/2418761447 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649c57ba340 0x5649c57c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.341+0000 7fe2b3213740 1 -- 172.21.15.49:0/2418761447 >> 172.21.15.49:0/2418761447 conn(0x5649c57b58b0 msgr2=0x5649c57b5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:57.344 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.341+0000 7fe291ffb640 0 client.16989 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:57.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.341+0000 7f5423a7c740 20 client.26353 trim_cache size 0 max 16384 2024-03-18T00:56:57.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.342+0000 7fe2b3213740 1 -- 172.21.15.49:0/2418761447 shutdown_connections 2024-03-18T00:56:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.345+0000 7fe2b3213740 1 -- 172.21.15.49:0/2418761447 wait complete. 2024-03-18T00:56:57.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.347+0000 7fe2b3213740 20 client.16989 trim_cache size 0 max 16384 2024-03-18T00:56:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:56:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:56:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:56:57.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:56:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.357+0000 7fe2b0627640 1 -- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2940a2ad0 msgr2=0x7fe2940a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.357+0000 7fe2b0627640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2940a2ad0 0x7fe2940a4f90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe2980c60b0 tx=0x7fe298005be0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.357+0000 7fe2b0627640 1 -- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2a8103a30 msgr2=0x7fe2a8070040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.357+0000 7fe2b0627640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2a8103a30 0x7fe2a8070040 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe298004bb0 tx=0x7fe298004fb0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.365+0000 7fe2b0627640 1 -- 172.21.15.49:0/2767380885 shutdown_connections 2024-03-18T00:56:57.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.365+0000 7fe2b0627640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2940a2ad0 0x7fe2940a4f90 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.365+0000 7fe2b0627640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe2a8104c20 0x7fe2a8070580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.365+0000 7fe2b0627640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe2a81055e0 0x7fe2a81ae650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.365+0000 7fe2b0627640 1 --2- 172.21.15.49:0/2767380885 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe2a8103a30 0x7fe2a8070040 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.365+0000 7fe2b0627640 1 -- 172.21.15.49:0/2767380885 >> 172.21.15.49:0/2767380885 conn(0x7fe2a80fefa0 msgr2=0x7fe2a8107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:57.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.366+0000 7fe2b0627640 1 -- 172.21.15.49:0/2767380885 shutdown_connections 2024-03-18T00:56:57.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:56:57.370+0000 7fe2b0627640 1 -- 172.21.15.49:0/2767380885 wait complete. 2024-03-18T00:56:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.430+0000 7f5423a7c740 1 Processor -- start 2024-03-18T00:56:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.430+0000 7f5423a7c740 1 -- start start 2024-03-18T00:56:57.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.431+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a823f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.431+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a8247bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.432+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a823fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.432+0000 7f5423a7c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc00e610 con 0x5570a823bcd0 2024-03-18T00:56:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.432+0000 7f5423a7c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc010150 con 0x5570a8236b60 2024-03-18T00:56:57.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.432+0000 7f5423a7c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc00ece0 con 0x5570a8236170 2024-03-18T00:56:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.433+0000 7f541ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a823fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.433+0000 7f541ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a823fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:43926/0 (socket says 172.21.15.49:43926) 2024-03-18T00:56:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.433+0000 7f541ad9d640 1 -- 172.21.15.49:0/1556972254 learned_addr learned my addr 172.21.15.49:0/1556972254 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.433+0000 7f541a59c640 1 --2- 172.21.15.49:0/1556972254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a823f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.433+0000 7f5419d9b640 1 --2- 172.21.15.49:0/1556972254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a8247bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.434+0000 7f541ad9d640 1 -- 172.21.15.49:0/1556972254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 msgr2=0x5570a823f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.434+0000 7f541ad9d640 1 --2- 172.21.15.49:0/1556972254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a823f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.434+0000 7f541ad9d640 1 -- 172.21.15.49:0/1556972254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 msgr2=0x5570a8247bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.434+0000 7f541ad9d640 1 --2- 172.21.15.49:0/1556972254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a8247bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.434+0000 7f541ad9d640 1 -- 172.21.15.49:0/1556972254 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570a7e6f0f0 con 0x5570a823bcd0 2024-03-18T00:56:57.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.435+0000 7f541ad9d640 1 --2- 172.21.15.49:0/1556972254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a823fec0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5414015600 tx=0x7f5414015f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9cd146946bd00d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:57.437 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.435+0000 7f5403fff640 1 -- 172.21.15.49:0/1556972254 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414030040 con 0x5570a823bcd0 2024-03-18T00:56:57.438 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.436+0000 7f5403fff640 1 -- 172.21.15.49:0/1556972254 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5414043070 con 0x5570a823bcd0 2024-03-18T00:56:57.439 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.438+0000 7f5423a7c740 1 -- 172.21.15.49:0/1556972254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 msgr2=0x5570a823fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.438+0000 7f5423a7c740 1 --2- 172.21.15.49:0/1556972254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a823fec0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f5414015600 tx=0x7f5414015f90 comp rx=0 tx=0).stop 2024-03-18T00:56:57.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.438+0000 7f5423a7c740 1 -- 172.21.15.49:0/1556972254 shutdown_connections 2024-03-18T00:56:57.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.438+0000 7f5423a7c740 1 --2- 172.21.15.49:0/1556972254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a8247bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.438+0000 7f5423a7c740 1 --2- 172.21.15.49:0/1556972254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a823fec0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.438+0000 7f5423a7c740 1 --2- 172.21.15.49:0/1556972254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a823f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.438+0000 7f5423a7c740 1 -- 172.21.15.49:0/1556972254 >> 172.21.15.49:0/1556972254 conn(0x5570a82316e0 msgr2=0x5570a8237580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:57.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.442+0000 7f5423a7c740 1 -- 172.21.15.49:0/1556972254 shutdown_connections 2024-03-18T00:56:57.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.442+0000 7f5423a7c740 1 -- 172.21.15.49:0/1556972254 wait complete. 2024-03-18T00:56:57.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.443+0000 7f5423a7c740 1 -- start start 2024-03-18T00:56:57.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.444+0000 7f5423a7c740 1 Processor -- start 2024-03-18T00:56:57.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.444+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a82448a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.445+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a8244de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.445+0000 7f5423a7c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a82b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.445+0000 7f541a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a82448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.445+0000 7f541a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a82448a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:46252/0 (socket says 172.21.15.49:46252) 2024-03-18T00:56:57.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.446+0000 7f5419d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a8244de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.446+0000 7f541a59c640 1 -- 172.21.15.49:0/785079544 learned_addr learned my addr 172.21.15.49:0/785079544 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:57.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.446+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc00fa80 con 0x5570a823bcd0 2024-03-18T00:56:57.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.446+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc00f3b0 con 0x5570a8236b60 2024-03-18T00:56:57.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.446+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f53fc00c5f0 con 0x5570a8236170 2024-03-18T00:56:57.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.447+0000 7f5423a7c740 10 client.0 _ll_register_callbacks cb 0x7f5414024070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:57.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.450+0000 7f541ad9d640 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a82b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.451+0000 7f541a59c640 1 -- 172.21.15.49:0/785079544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 msgr2=0x5570a8244de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:57.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.451+0000 7f541a59c640 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a8244de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.451+0000 7f541a59c640 1 -- 172.21.15.49:0/785079544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 msgr2=0x5570a82b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:57.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.451+0000 7f541a59c640 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a82b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.452+0000 7f541a59c640 1 -- 172.21.15.49:0/785079544 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570a81629d0 con 0x5570a8236170 2024-03-18T00:56:57.454 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.455+0000 7f541a59c640 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a82448a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f540c000c00 tx=0x7f540c0045d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88897daed310f789 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:57.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.457+0000 7f5401ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:56:57.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.457+0000 7f5401ffb640 1 -- 172.21.15.49:0/785079544 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c0a3de0 con 0x5570a8236170 2024-03-18T00:56:57.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.457+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5414043070 con 0x5570a8236170 2024-03-18T00:56:57.457 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.457+0000 7f5401ffb640 1 -- 172.21.15.49:0/785079544 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f540c097cb0 con 0x5570a8236170 2024-03-18T00:56:57.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.465+0000 7f5401ffb640 1 -- 172.21.15.49:0/785079544 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f540c004310 con 0x5570a8236170 2024-03-18T00:56:57.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.467+0000 7f5423a7c740 10 client.26359 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:57.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.467+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5570a82b1f10 con 0x5570a8236170 2024-03-18T00:56:57.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.471+0000 7f5401ffb640 1 -- 172.21.15.49:0/785079544 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5414043070 con 0x5570a8236170 2024-03-18T00:56:57.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.471+0000 7f5401ffb640 1 -- 172.21.15.49:0/785079544 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f540c084bd0 con 0x5570a8236170 2024-03-18T00:56:57.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.471+0000 7f5423a7c740 10 client.26359 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:57.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.471+0000 7f5423a7c740 10 client.26359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:56:57.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.471+0000 7f5423a7c740 20 client.26359 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:57.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.471+0000 7f5423a7c740 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5570a82b2910 0x5570a82b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.472+0000 7f5423a7c740 4 client.26359 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:56:57.472 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.472+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5570a81a2d90 con 0x5570a82b2910 2024-03-18T00:56:57.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.475+0000 7f5419d9b640 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5570a82b2910 0x5570a82b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.479+0000 7f5419d9b640 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5570a82b2910 0x5570a82b4d30 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8085b66e930396d1 server_cookie=6202c08a3d53f2ab in_seq=0 out_seq=0 2024-03-18T00:56:57.479 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.479+0000 7f5401ffb640 10 client.26359 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:57.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.482+0000 7f5401ffb640 1 -- 172.21.15.49:0/785079544 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5570a81a2d90 con 0x5570a82b2910 2024-03-18T00:56:57.482 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.482+0000 7f5401ffb640 10 client.26359 handle_command_reply: tid=0 2024-03-18T00:56:57.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.482+0000 7f5423a7c740 1 client.26359 shutdown 2024-03-18T00:56:57.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 msgr2=0x5570a82448a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a82448a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f540c000c00 tx=0x7f540c0045d0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 shutdown_connections 2024-03-18T00:56:57.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5570a82b2910 0x5570a82b4d30 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5570a8236b60 0x5570a8244de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5570a823bcd0 0x5570a82b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 --2- 172.21.15.49:0/785079544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5570a8236170 0x5570a82448a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 >> 172.21.15.49:0/785079544 conn(0x5570a82316e0 msgr2=0x5570a823f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:57.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5401ffb640 0 client.26359 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:56:57.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 shutdown_connections 2024-03-18T00:56:57.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 1 -- 172.21.15.49:0/785079544 wait complete. 2024-03-18T00:56:57.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.483+0000 7f5423a7c740 20 client.26359 trim_cache size 0 max 16384 2024-03-18T00:56:57.486 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:56:57.486 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:56:57.486 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:56:57.486 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 -- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f54040a2950 msgr2=0x7f54040a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f54040a2950 0x7f54040a4e10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f541401b000 tx=0x7f541401d000 comp rx=0 tx=0).stop 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 -- 172.21.15.49:0/3211368440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f541c1017f0 msgr2=0x7f541c195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f541c1017f0 0x7f541c195a20 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f540c00e9f0 tx=0x7f540c00eeb0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 -- 172.21.15.49:0/3211368440 shutdown_connections 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f54040a2950 0x7f54040a4e10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f541c1033a0 0x7f541c1ae450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f541c1029e0 0x7f541c195f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 --2- 172.21.15.49:0/3211368440 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f541c1017f0 0x7f541c195a20 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.484+0000 7f5420e85640 1 -- 172.21.15.49:0/3211368440 >> 172.21.15.49:0/3211368440 conn(0x7f541c0fcd80 msgr2=0x7f541c103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:57.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.485+0000 7f5420e85640 1 -- 172.21.15.49:0/3211368440 shutdown_connections 2024-03-18T00:56:57.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.485+0000 7f5420e85640 1 -- 172.21.15.49:0/3211368440 wait complete. 2024-03-18T00:56:57.503 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:57 smithi049 ceph-mon[32130]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 855 MiB/s wr, 2.12k op/s 2024-03-18T00:56:57.527 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:56:57.528 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:56:57.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:57 smithi107 ceph-mon[36270]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 855 MiB/s wr, 2.12k op/s 2024-03-18T00:56:57.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:57 smithi106 ceph-mon[35899]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 855 MiB/s wr, 2.12k op/s 2024-03-18T00:56:57.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.847+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/3575536148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f010d8a0 msgr2=0x7fa0f0118720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.847+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/3575536148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f010d8a0 0x7fa0f0118720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009900 tx=0x7fa0e402bc40 comp rx=0 tx=0).stop 2024-03-18T00:56:57.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.849+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/3575536148 shutdown_connections 2024-03-18T00:56:57.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.849+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/3575536148 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0f010adc0 0x7fa0f010d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.849+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/3575536148 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f010d8a0 0x7fa0f0118720 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.849+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/3575536148 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f00735e0 0x7fa0f010a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.849+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/3575536148 >> 172.21.15.49:0/3575536148 conn(0x7fa0f006e510 msgr2=0x7fa0f0070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:57.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.852+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/3575536148 shutdown_connections 2024-03-18T00:56:57.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.857+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/3575536148 wait complete. 2024-03-18T00:56:57.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.859+0000 7fa0f4cd9640 1 Processor -- start 2024-03-18T00:56:57.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.860+0000 7fa0f4cd9640 1 -- start start 2024-03-18T00:56:57.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.861+0000 7fa0f4cd9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f00735e0 0x7fa0f01a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.861+0000 7fa0f4cd9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0f010adc0 0x7fa0f01a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.861+0000 7fa0f4cd9640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f010d8a0 0x7fa0f01adc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.862+0000 7fa0f4cd9640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f011c0a0 con 0x7fa0f010d8a0 2024-03-18T00:56:57.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.862+0000 7fa0f4cd9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f011bf20 con 0x7fa0f010adc0 2024-03-18T00:56:57.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.862+0000 7fa0f4cd9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f011c220 con 0x7fa0f00735e0 2024-03-18T00:56:57.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.862+0000 7fa0effff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f010d8a0 0x7fa0f01adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.862+0000 7fa0effff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f010d8a0 0x7fa0f01adc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54442/0 (socket says 172.21.15.49:54442) 2024-03-18T00:56:57.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.862+0000 7fa0effff640 1 -- 172.21.15.49:0/1279677087 learned_addr learned my addr 172.21.15.49:0/1279677087 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:57.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.867+0000 7fa0ef7fe640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f00735e0 0x7fa0f01a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.867+0000 7fa0eeffd640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0f010adc0 0x7fa0f01a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.869+0000 7fa0effff640 1 -- 172.21.15.49:0/1279677087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f00735e0 msgr2=0x7fa0f01a66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.869+0000 7fa0effff640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f00735e0 0x7fa0f01a66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.869+0000 7fa0effff640 1 -- 172.21.15.49:0/1279677087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0f010adc0 msgr2=0x7fa0f01a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:57.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.869+0000 7fa0effff640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0f010adc0 0x7fa0f01a6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:57.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.869+0000 7fa0effff640 1 -- 172.21.15.49:0/1279677087 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f01ae2c0 con 0x7fa0f010d8a0 2024-03-18T00:56:57.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.871+0000 7fa0effff640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f010d8a0 0x7fa0f01adc50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa0e402efd0 tx=0x7fa0e402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:57.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.873+0000 7fa0ecff9640 1 -- 172.21.15.49:0/1279677087 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4041070 con 0x7fa0f010d8a0 2024-03-18T00:56:57.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.873+0000 7fa0ecff9640 1 -- 172.21.15.49:0/1279677087 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0e40044d0 con 0x7fa0f010d8a0 2024-03-18T00:56:57.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.873+0000 7fa0ecff9640 1 -- 172.21.15.49:0/1279677087 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4034440 con 0x7fa0f010d8a0 2024-03-18T00:56:57.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.874+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f01ae550 con 0x7fa0f010d8a0 2024-03-18T00:56:57.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.874+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f01ae9f0 con 0x7fa0f010d8a0 2024-03-18T00:56:57.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.881+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f0071010 con 0x7fa0f010d8a0 2024-03-18T00:56:57.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.881+0000 7fa0ecff9640 1 -- 172.21.15.49:0/1279677087 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa0e4044070 con 0x7fa0f010d8a0 2024-03-18T00:56:57.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.882+0000 7fa0ecff9640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0d80a2a00 0x7fa0d80a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:57.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.883+0000 7fa0ecff9640 1 -- 172.21.15.49:0/1279677087 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa0e40f01c0 con 0x7fa0f010d8a0 2024-03-18T00:56:57.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.890+0000 7fa0ef7fe640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0d80a2a00 0x7fa0d80a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:57.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.891+0000 7fa0ef7fe640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0d80a2a00 0x7fa0d80a4ec0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00739e0 tx=0x7fa0e8009870 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:57.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:57.897+0000 7fa0ecff9640 1 -- 172.21.15.49:0/1279677087 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa0e40bd1e0 con 0x7fa0f010d8a0 2024-03-18T00:56:58.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.686+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0f00623e0 con 0x7fa0f010d8a0 2024-03-18T00:56:58.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.691+0000 7fa0ecff9640 1 -- 172.21.15.49:0/1279677087 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7fa0e40bcae0 con 0x7fa0f010d8a0 2024-03-18T00:56:58.694 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:56:58.695 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:53:25.016081+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24470,24463,24481,24506,24499]},"id":1}]} 2024-03-18T00:56:58.695 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 27 2024-03-18T00:56:58.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.705+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0d80a2a00 msgr2=0x7fa0d80a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:58.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.705+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0d80a2a00 0x7fa0d80a4ec0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00739e0 tx=0x7fa0e8009870 comp rx=0 tx=0).stop 2024-03-18T00:56:58.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.705+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f010d8a0 msgr2=0x7fa0f01adc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:58.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.705+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f010d8a0 0x7fa0f01adc50 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa0e402efd0 tx=0x7fa0e402c650 comp rx=0 tx=0).stop 2024-03-18T00:56:58.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.713+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 shutdown_connections 2024-03-18T00:56:58.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.713+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa0d80a2a00 0x7fa0d80a4ec0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:58.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.713+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa0f010adc0 0x7fa0f01a6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:58.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.713+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa0f010d8a0 0x7fa0f01adc50 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:58.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.713+0000 7fa0f4cd9640 1 --2- 172.21.15.49:0/1279677087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa0f00735e0 0x7fa0f01a66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:58.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.713+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 >> 172.21.15.49:0/1279677087 conn(0x7fa0f006e510 msgr2=0x7fa0f010a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:58.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.717+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 shutdown_connections 2024-03-18T00:56:58.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:58.719+0000 7fa0f4cd9640 1 -- 172.21.15.49:0/1279677087 wait complete. 2024-03-18T00:56:58.789 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:56:59.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.148+0000 7f0cd7f77640 1 -- 172.21.15.49:0/3748292681 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cd0110060 msgr2=0x7f0cd0112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.148+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/3748292681 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cd0110060 0x7f0cd0112450 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009ca0 tx=0x7f0cc802c120 comp rx=0 tx=0).stop 2024-03-18T00:56:59.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.149+0000 7f0cd7f77640 1 -- 172.21.15.49:0/3748292681 shutdown_connections 2024-03-18T00:56:59.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.149+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/3748292681 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cd0110060 0x7f0cd0112450 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.149+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/3748292681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cd0109140 0x7f0cd010fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.149+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/3748292681 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0cd01087e0 0x7f0cd0108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.149+0000 7f0cd7f77640 1 -- 172.21.15.49:0/3748292681 >> 172.21.15.49:0/3748292681 conn(0x7f0cd006e640 msgr2=0x7f0cd0070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.150+0000 7f0cd7f77640 1 -- 172.21.15.49:0/3748292681 shutdown_connections 2024-03-18T00:56:59.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.150+0000 7f0cd7f77640 1 -- 172.21.15.49:0/3748292681 wait complete. 2024-03-18T00:56:59.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.152+0000 7f0cd7f77640 1 Processor -- start 2024-03-18T00:56:59.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.152+0000 7f0cd7f77640 1 -- start start 2024-03-18T00:56:59.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd7f77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cd01087e0 0x7f0cd0119500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd7f77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cd0109140 0x7f0cd0119a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd7f77640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0cd0110060 0x7f0cd011eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd5512640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cd0109140 0x7f0cd0119a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd5512640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cd0109140 0x7f0cd0119a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54454/0 (socket says 172.21.15.49:54454) 2024-03-18T00:56:59.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd5d13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cd01087e0 0x7f0cd0119500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd5d13640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cd01087e0 0x7f0cd0119500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:37722/0 (socket says 172.21.15.49:37722) 2024-03-18T00:56:59.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd5512640 1 -- 172.21.15.49:0/1377554791 learned_addr learned my addr 172.21.15.49:0/1377554791 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd010b860 con 0x7f0cd0109140 2024-03-18T00:56:59.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd010b6e0 con 0x7f0cd0110060 2024-03-18T00:56:59.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.153+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd0113df0 con 0x7f0cd01087e0 2024-03-18T00:56:59.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.154+0000 7f0cd6514640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0cd0110060 0x7f0cd011eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.155+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1377554791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0cd0110060 msgr2=0x7f0cd011eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.155+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0cd0110060 0x7f0cd011eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.155+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1377554791 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cd0109140 msgr2=0x7f0cd0119a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.155+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cd0109140 0x7f0cd0119a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.155+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1377554791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd011f110 con 0x7f0cd01087e0 2024-03-18T00:56:59.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.156+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cd01087e0 0x7f0cd0119500 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00b3f0 tx=0x7f0ccc00b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.159+0000 7f0cc6ffd640 1 -- 172.21.15.49:0/1377554791 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc015020 con 0x7f0cd01087e0 2024-03-18T00:56:59.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.159+0000 7f0cc6ffd640 1 -- 172.21.15.49:0/1377554791 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ccc011d10 con 0x7f0cd01087e0 2024-03-18T00:56:59.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.159+0000 7f0cc6ffd640 1 -- 172.21.15.49:0/1377554791 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc00f2e0 con 0x7f0cd01087e0 2024-03-18T00:56:59.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.159+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cd011f400 con 0x7f0cd01087e0 2024-03-18T00:56:59.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.159+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cd011f900 con 0x7f0cd01087e0 2024-03-18T00:56:59.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.163+0000 7f0cda7fd740 1 Processor -- start 2024-03-18T00:56:59.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.163+0000 7f0cda7fd740 1 -- start start 2024-03-18T00:56:59.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.163+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d5d850 0x55b9a9d5dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.163+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 0x55b9a9d62d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.163+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d633b0 0x55b9a9d65840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.163+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b9a9cc2490 con 0x55b9a9d633b0 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.163+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b9a9c6ac90 con 0x55b9a9d5e240 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.163+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b9a9cbe530 con 0x55b9a9d5d850 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cd5512640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d633b0 0x55b9a9d65840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cd5512640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d633b0 0x55b9a9d65840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54470/0 (socket says 172.21.15.49:54470) 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cd5512640 1 -- 172.21.15.49:0/2314153696 learned_addr learned my addr 172.21.15.49:0/2314153696 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cd5512640 1 -- 172.21.15.49:0/2314153696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d5d850 msgr2=0x55b9a9d5dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cd5512640 1 --2- 172.21.15.49:0/2314153696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d5d850 0x55b9a9d5dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cd5512640 1 -- 172.21.15.49:0/2314153696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 msgr2=0x55b9a9d62d40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cd5512640 1 --2- 172.21.15.49:0/2314153696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 0x55b9a9d62d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cd5512640 1 -- 172.21.15.49:0/2314153696 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9a9d25500 con 0x55b9a9d633b0 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cd5512640 1 --2- 172.21.15.49:0/2314153696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d633b0 0x55b9a9d65840 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0007810 tx=0x7f0cc0007cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3dd657248f29cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cbbfff640 1 -- 172.21.15.49:0/2314153696 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000d950 con 0x55b9a9d633b0 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.164+0000 7f0cbbfff640 1 -- 172.21.15.49:0/2314153696 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cc0018070 con 0x55b9a9d633b0 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.165+0000 7f0cbbfff640 1 -- 172.21.15.49:0/2314153696 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc00123e0 con 0x55b9a9d633b0 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.165+0000 7f0cda7fd740 1 -- 172.21.15.49:0/2314153696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d633b0 msgr2=0x55b9a9d65840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.165+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/2314153696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d633b0 0x55b9a9d65840 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0007810 tx=0x7f0cc0007cd0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.165+0000 7f0cda7fd740 1 -- 172.21.15.49:0/2314153696 shutdown_connections 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.165+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/2314153696 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 0x55b9a9d62d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.165+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/2314153696 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d633b0 0x55b9a9d65840 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.165+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/2314153696 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d5d850 0x55b9a9d5dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.165+0000 7f0cda7fd740 1 -- 172.21.15.49:0/2314153696 >> 172.21.15.49:0/2314153696 conn(0x55b9a9d58dc0 msgr2=0x55b9a9d591c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.166+0000 7f0cc6ffd640 1 -- 172.21.15.49:0/1377554791 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0ccc018020 con 0x7f0cd01087e0 2024-03-18T00:56:59.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.167+0000 7f0cc6ffd640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0cbc0a2880 0x7f0cbc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.167+0000 7f0cc6ffd640 1 -- 172.21.15.49:0/1377554791 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0ccc0ce1c0 con 0x7f0cd01087e0 2024-03-18T00:56:59.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.168+0000 7f0cda7fd740 1 -- 172.21.15.49:0/2314153696 shutdown_connections 2024-03-18T00:56:59.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.168+0000 7f0cd5512640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0cbc0a2880 0x7f0cbc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.168+0000 7f0cda7fd740 1 -- 172.21.15.49:0/2314153696 wait complete. 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.168+0000 7f0cda7fd740 1 -- start start 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.168+0000 7f0cd5512640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0cbc0a2880 0x7f0cbc0a4d40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55b9a9d59d30 tx=0x7f0cc0006ff0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.168+0000 7f0cda7fd740 1 Processor -- start 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5d850 0x55b9a9d67d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 0x55b9a9d682b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d633b0 0x55b9a9d6f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b9a9d66360 con 0x55b9a9d5d850 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b9a9c7e230 con 0x55b9a9d5e240 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b9a9d664e0 con 0x55b9a9d633b0 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cda7fd740 10 client.0 _ll_register_callbacks cb 0x55b9a9cba010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cd5d13640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5d850 0x55b9a9d67d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cd5d13640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5d850 0x55b9a9d67d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54482/0 (socket says 172.21.15.49:54482) 2024-03-18T00:56:59.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.169+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1735362237 learned_addr learned my addr 172.21.15.49:0/1735362237 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1735362237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d633b0 msgr2=0x55b9a9d6f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d633b0 0x55b9a9d6f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1735362237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 msgr2=0x55b9a9d682b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 0x55b9a9d682b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1735362237 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9a9c8d100 con 0x55b9a9d5d850 2024-03-18T00:56:59.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5d850 0x55b9a9d67d70 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc097510 tx=0x7f0ccc009200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62697954eb94fd1d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cb97fa640 1 -- 172.21.15.49:0/1735362237 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc0b2020 con 0x55b9a9d5d850 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1735362237 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cc0018070 con 0x55b9a9d5d850 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cb97fa640 1 -- 172.21.15.49:0/1735362237 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0ccc0971b0 con 0x55b9a9d5d850 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cb97fa640 1 -- 172.21.15.49:0/1735362237 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0ccc09ab40 con 0x55b9a9d5d850 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cda7fd740 10 client.17025 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.170+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1735362237 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b9a99d63f0 con 0x55b9a9d5d850 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.171+0000 7f0cb97fa640 1 -- 172.21.15.49:0/1735362237 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cc0018070 con 0x55b9a9d5d850 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.171+0000 7f0cb97fa640 1 -- 172.21.15.49:0/1735362237 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0ccc0b9200 con 0x55b9a9d5d850 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.171+0000 7f0cda7fd740 10 client.17025 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.171+0000 7f0cda7fd740 10 client.17025 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.171+0000 7f0cda7fd740 20 client.17025 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:59.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.171+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b9a9d73f50 0x55b9a9d763f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.171+0000 7f0cda7fd740 4 client.17025 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:59.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.171+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1735362237 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9a9ccf9d0 con 0x55b9a9d73f50 2024-03-18T00:56:59.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.174+0000 7f0cd5512640 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b9a9d73f50 0x55b9a9d763f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.179+0000 7f0cd5512640 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b9a9d73f50 0x55b9a9d763f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cd023e206873f9ac server_cookie=c81b540753e93cc5 in_seq=0 out_seq=0 2024-03-18T00:56:59.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.179+0000 7f0cb97fa640 10 client.17025 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:56:59.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cb97fa640 1 -- 172.21.15.49:0/1735362237 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b9a9ccf9d0 con 0x55b9a9d73f50 2024-03-18T00:56:59.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cb97fa640 10 client.17025 handle_command_reply: tid=0 2024-03-18T00:56:59.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 client.17025 shutdown 2024-03-18T00:56:59.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1735362237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5d850 msgr2=0x55b9a9d67d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5d850 0x55b9a9d67d70 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc097510 tx=0x7f0ccc009200 comp rx=0 tx=0).stop 2024-03-18T00:56:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1735362237 shutdown_connections 2024-03-18T00:56:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b9a9d73f50 0x55b9a9d763f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 0x55b9a9d682b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d633b0 0x55b9a9d6f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1735362237 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5d850 0x55b9a9d67d70 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1735362237 >> 172.21.15.49:0/1735362237 conn(0x55b9a9d58dc0 msgr2=0x55b9a9d57fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cb97fa640 0 client.17025 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:56:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1735362237 shutdown_connections 2024-03-18T00:56:59.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.182+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1735362237 wait complete. 2024-03-18T00:56:59.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.183+0000 7f0cda7fd740 20 client.17025 trim_cache size 0 max 16384 2024-03-18T00:56:59.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cda7fd740 1 Processor -- start 2024-03-18T00:56:59.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cda7fd740 1 -- start start 2024-03-18T00:56:59.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d58dc0 0x55b9a9d5f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d5d850 0x55b9a9d5fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 0x55b9a9d6f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01c0500 con 0x55b9a9d58dc0 2024-03-18T00:56:59.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01c2040 con 0x55b9a9d5e240 2024-03-18T00:56:59.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01c0bd0 con 0x55b9a9d5d850 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cd5d13640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d58dc0 0x55b9a9d5f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cd5d13640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d58dc0 0x55b9a9d5f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54498/0 (socket says 172.21.15.49:54498) 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.211+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1411810275 learned_addr learned my addr 172.21.15.49:0/1411810275 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.212+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1411810275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d5d850 msgr2=0x55b9a9d5fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.212+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/1411810275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d5d850 0x55b9a9d5fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.212+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1411810275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 msgr2=0x55b9a9d6f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.212+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/1411810275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 0x55b9a9d6f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.212+0000 7f0cd5d13640 1 -- 172.21.15.49:0/1411810275 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9a9aa07d0 con 0x55b9a9d58dc0 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.212+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/1411810275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d58dc0 0x55b9a9d5f7a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00bdb0 tx=0x7f0ccc099050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857046bf06ad56d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.212+0000 7f0cbb7fe640 1 -- 172.21.15.49:0/1411810275 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc0b2070 con 0x55b9a9d58dc0 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.212+0000 7f0cbb7fe640 1 -- 172.21.15.49:0/1411810275 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ccc09a280 con 0x55b9a9d58dc0 2024-03-18T00:56:59.213 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.212+0000 7f0cbb7fe640 1 -- 172.21.15.49:0/1411810275 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc0b9200 con 0x55b9a9d58dc0 2024-03-18T00:56:59.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.213+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1411810275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d58dc0 msgr2=0x55b9a9d5f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.213+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1411810275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d58dc0 0x55b9a9d5f7a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00bdb0 tx=0x7f0ccc099050 comp rx=0 tx=0).stop 2024-03-18T00:56:59.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.213+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1411810275 shutdown_connections 2024-03-18T00:56:59.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.213+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1411810275 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5e240 0x55b9a9d6f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.213+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1411810275 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d5d850 0x55b9a9d5fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.213+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/1411810275 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d58dc0 0x55b9a9d5f7a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.213+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1411810275 >> 172.21.15.49:0/1411810275 conn(0x55b9a9d67700 msgr2=0x55b9a9d591a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.213+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1411810275 shutdown_connections 2024-03-18T00:56:59.216 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.213+0000 7f0cda7fd740 1 -- 172.21.15.49:0/1411810275 wait complete. 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.214+0000 7f0cda7fd740 1 -- start start 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.214+0000 7f0cda7fd740 1 Processor -- start 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d58dc0 0x55b9a9d6f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5d850 0x55b9a9d6bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cda7fd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5e240 0x55b9a9d6c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01c1970 con 0x55b9a9d5e240 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01c12a0 con 0x55b9a9d5d850 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cda7fd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd01be4e0 con 0x55b9a9d58dc0 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cda7fd740 10 client.0 _ll_register_callbacks cb 0x7f0ccc010380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cd6514640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5e240 0x55b9a9d6c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cd6514640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5e240 0x55b9a9d6c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54502/0 (socket says 172.21.15.49:54502) 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cd6514640 1 -- 172.21.15.49:0/3137374582 learned_addr learned my addr 172.21.15.49:0/3137374582 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.215+0000 7f0cd5512640 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5d850 0x55b9a9d6bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cd6514640 1 -- 172.21.15.49:0/3137374582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d58dc0 msgr2=0x55b9a9d6f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.217 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cd6514640 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d58dc0 0x55b9a9d6f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cd6514640 1 -- 172.21.15.49:0/3137374582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5d850 msgr2=0x55b9a9d6bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cd6514640 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5d850 0x55b9a9d6bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cd6514640 1 -- 172.21.15.49:0/3137374582 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9a9dd83d0 con 0x55b9a9d5e240 2024-03-18T00:56:59.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cd6514640 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5e240 0x55b9a9d6c340 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0cc802c730 tx=0x7f0cc8003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de201daf247cd791 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cb97fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:59.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cb97fa640 1 -- 172.21.15.49:0/3137374582 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8043070 con 0x55b9a9d5e240 2024-03-18T00:56:59.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cb97fa640 1 -- 172.21.15.49:0/3137374582 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0cc80328f0 con 0x55b9a9d5e240 2024-03-18T00:56:59.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cda7fd740 1 -- 172.21.15.49:0/3137374582 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b9a9cc9d20 con 0x55b9a9d5e240 2024-03-18T00:56:59.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.216+0000 7f0cb97fa640 1 -- 172.21.15.49:0/3137374582 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0cc804d050 con 0x55b9a9d5e240 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.217+0000 7f0cda7fd740 10 client.17031 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.217+0000 7f0cda7fd740 1 -- 172.21.15.49:0/3137374582 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b9a9dd9740 con 0x55b9a9d5e240 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.217+0000 7f0cb97fa640 1 -- 172.21.15.49:0/3137374582 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b9a9cc9d20 con 0x55b9a9d5e240 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.218+0000 7f0cb97fa640 1 -- 172.21.15.49:0/3137374582 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0cc8004650 con 0x55b9a9d5e240 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.218+0000 7f0cda7fd740 10 client.17031 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.218+0000 7f0cda7fd740 10 client.17031 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.218+0000 7f0cda7fd740 20 client.17031 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.218+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b9a9d633b0 0x55b9a9ddc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.218+0000 7f0cda7fd740 4 client.17031 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.218+0000 7f0cda7fd740 1 -- 172.21.15.49:0/3137374582 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b9a9c476a0 con 0x55b9a9d633b0 2024-03-18T00:56:59.220 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.219+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b9a9d633b0 0x55b9a9ddc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.225+0000 7f0cd5d13640 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b9a9d633b0 0x55b9a9ddc200 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=289860f435a98f78 server_cookie=e769f2791bb7c17c in_seq=0 out_seq=0 2024-03-18T00:56:59.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.225+0000 7f0cb97fa640 10 client.17031 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:56:59.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.227+0000 7f0cb97fa640 1 -- 172.21.15.49:0/3137374582 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b9a9c476a0 con 0x55b9a9d633b0 2024-03-18T00:56:59.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.227+0000 7f0cb97fa640 10 client.17031 handle_command_reply: tid=0 2024-03-18T00:56:59.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.227+0000 7f0cda7fd740 1 client.17031 shutdown 2024-03-18T00:56:59.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.227+0000 7f0cda7fd740 1 -- 172.21.15.49:0/3137374582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5e240 msgr2=0x55b9a9d6c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.227+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5e240 0x55b9a9d6c340 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f0cc802c730 tx=0x7f0cc8003f00 comp rx=0 tx=0).stop 2024-03-18T00:56:59.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.227+0000 7f0cda7fd740 1 -- 172.21.15.49:0/3137374582 shutdown_connections 2024-03-18T00:56:59.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.227+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55b9a9d633b0 0x55b9a9ddc200 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.228+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b9a9d5d850 0x55b9a9d6bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.228+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b9a9d5e240 0x55b9a9d6c340 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.228+0000 7f0cda7fd740 1 --2- 172.21.15.49:0/3137374582 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b9a9d58dc0 0x55b9a9d6f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.228+0000 7f0cda7fd740 1 -- 172.21.15.49:0/3137374582 >> 172.21.15.49:0/3137374582 conn(0x55b9a9d67700 msgr2=0x55b9a9d5f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.228+0000 7f0cb97fa640 0 client.17031 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:56:59.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.228+0000 7f0cda7fd740 1 -- 172.21.15.49:0/3137374582 shutdown_connections 2024-03-18T00:56:59.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.229+0000 7f0cda7fd740 1 -- 172.21.15.49:0/3137374582 wait complete. 2024-03-18T00:56:59.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.229+0000 7f0cda7fd740 20 client.17031 trim_cache size 0 max 16384 2024-03-18T00:56:59.229 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.230+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0cbc0a2880 msgr2=0x7f0cbc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.230+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0cbc0a2880 0x7f0cbc0a4d40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x55b9a9d59d30 tx=0x7f0cc0006ff0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.230+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cd01087e0 msgr2=0x7f0cd0119500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.230+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cd01087e0 0x7f0cd0119500 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00b3f0 tx=0x7f0ccc00b8b0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.231+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 shutdown_connections 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.231+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0cbc0a2880 0x7f0cbc0a4d40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.231+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0cd0110060 0x7f0cd011eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.231+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0cd0109140 0x7f0cd0119a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.231+0000 7f0cd7f77640 1 --2- 172.21.15.49:0/1377554791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cd01087e0 0x7f0cd0119500 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.231+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 >> 172.21.15.49:0/1377554791 conn(0x7f0cd006e640 msgr2=0x7f0cd0070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.231+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 shutdown_connections 2024-03-18T00:56:59.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.231+0000 7f0cd7f77640 1 -- 172.21.15.49:0/1377554791 wait complete. 2024-03-18T00:56:59.266 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:56:59.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.605+0000 7f5764fdd640 1 -- 172.21.15.49:0/1134405608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5760109870 msgr2=0x7f576010e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.605+0000 7f5764fdd640 1 --2- 172.21.15.49:0/1134405608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5760109870 0x7f576010e110 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5750009900 tx=0x7f575002bc40 comp rx=0 tx=0).stop 2024-03-18T00:56:59.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.605+0000 7f5764fdd640 1 -- 172.21.15.49:0/1134405608 shutdown_connections 2024-03-18T00:56:59.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.605+0000 7f5764fdd640 1 --2- 172.21.15.49:0/1134405608 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5760109870 0x7f576010e110 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.605+0000 7f5764fdd640 1 --2- 172.21.15.49:0/1134405608 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5760108eb0 0x7f5760109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.605+0000 7f5764fdd640 1 --2- 172.21.15.49:0/1134405608 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5760072bf0 0x7f5760073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.605+0000 7f5764fdd640 1 -- 172.21.15.49:0/1134405608 >> 172.21.15.49:0/1134405608 conn(0x7f576006e6c0 msgr2=0x7f5760070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.608+0000 7f5764fdd640 1 -- 172.21.15.49:0/1134405608 shutdown_connections 2024-03-18T00:56:59.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.608+0000 7f5764fdd640 1 -- 172.21.15.49:0/1134405608 wait complete. 2024-03-18T00:56:59.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.609+0000 7f5764fdd640 1 Processor -- start 2024-03-18T00:56:59.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.610+0000 7f5764fdd640 1 -- start start 2024-03-18T00:56:59.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.610+0000 7f5764fdd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5760072bf0 0x7f5760119e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.610+0000 7f5764fdd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5760108eb0 0x7f57601184a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.610+0000 7f5764fdd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5760109870 0x7f57601189e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.610+0000 7f5764fdd640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5760110dd0 con 0x7f5760109870 2024-03-18T00:56:59.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.610+0000 7f5764fdd640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5760110c50 con 0x7f5760072bf0 2024-03-18T00:56:59.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.610+0000 7f5764fdd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5760110f50 con 0x7f5760108eb0 2024-03-18T00:56:59.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.612+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5760072bf0 0x7f5760119e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.612+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5760072bf0 0x7f5760119e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:44696/0 (socket says 172.21.15.49:44696) 2024-03-18T00:56:59.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.612+0000 7f575f7fe640 1 -- 172.21.15.49:0/2908161132 learned_addr learned my addr 172.21.15.49:0/2908161132 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.614+0000 7f575effd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5760108eb0 0x7f57601184a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.617+0000 7f575ffff640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5760109870 0x7f57601189e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.619+0000 7f575effd640 1 -- 172.21.15.49:0/2908161132 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5760072bf0 msgr2=0x7f5760119e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.619+0000 7f575effd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5760072bf0 0x7f5760119e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.619+0000 7f575effd640 1 -- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5760109870 msgr2=0x7f57601189e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.619+0000 7f575effd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5760109870 0x7f57601189e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.619+0000 7f575effd640 1 -- 172.21.15.49:0/2908161132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5760119050 con 0x7f5760108eb0 2024-03-18T00:56:59.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.621+0000 7f575f7fe640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5760072bf0 0x7f5760119e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:56:59.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.622+0000 7f575effd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5760108eb0 0x7f57601184a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f575400d6d0 tx=0x7f575400db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:59 smithi049 ceph-mon[32130]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 877 MiB/s wr, 2.42k op/s 2024-03-18T00:56:59.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:56:59 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1279677087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:56:59.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.626+0000 7f575cff9640 1 -- 172.21.15.49:0/2908161132 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754017070 con 0x7f5760108eb0 2024-03-18T00:56:59.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.626+0000 7f5764fdd640 1 -- 172.21.15.49:0/2908161132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57601b6fc0 con 0x7f5760108eb0 2024-03-18T00:56:59.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.626+0000 7f5764fdd640 1 -- 172.21.15.49:0/2908161132 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57601b73c0 con 0x7f5760108eb0 2024-03-18T00:56:59.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.627+0000 7f575cff9640 1 -- 172.21.15.49:0/2908161132 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5754009820 con 0x7f5760108eb0 2024-03-18T00:56:59.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.627+0000 7f575cff9640 1 -- 172.21.15.49:0/2908161132 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754004ca0 con 0x7f5760108eb0 2024-03-18T00:56:59.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.636+0000 7f5768e20740 1 Processor -- start 2024-03-18T00:56:59.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.637+0000 7f5768e20740 1 -- start start 2024-03-18T00:56:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.637+0000 7f5768e20740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd3a40 0x55a475cd3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.638+0000 7f5768e20740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd4430 0x55a475cd8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.638+0000 7f5768e20740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 0x55a475cdba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.638+0000 7f5768e20740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a475c38490 con 0x55a475cd95a0 2024-03-18T00:56:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.638+0000 7f5768e20740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a475be0c90 con 0x55a475cd4430 2024-03-18T00:56:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.638+0000 7f5768e20740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a475c34530 con 0x55a475cd3a40 2024-03-18T00:56:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.638+0000 7f575ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd3a40 0x55a475cd3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.638+0000 7f575ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd3a40 0x55a475cd3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:37796/0 (socket says 172.21.15.49:37796) 2024-03-18T00:56:59.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.638+0000 7f575ffff640 1 -- 172.21.15.49:0/3275135009 learned_addr learned my addr 172.21.15.49:0/3275135009 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.640+0000 7f575cff9640 1 -- 172.21.15.49:0/2908161132 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5754005220 con 0x7f5760108eb0 2024-03-18T00:56:59.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.642+0000 7f575cff9640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57480a28d0 0x7f57480a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.643+0000 7f575cff9640 1 -- 172.21.15.49:0/2908161132 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f57540ce290 con 0x7f5760108eb0 2024-03-18T00:56:59.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.643+0000 7f575f7fe640 1 --2- 172.21.15.49:0/3275135009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd4430 0x55a475cd8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.645+0000 7f575effd640 1 --2- 172.21.15.49:0/3275135009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 0x55a475cdba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.645+0000 7f575ffff640 1 -- 172.21.15.49:0/3275135009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd4430 msgr2=0x55a475cd8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.645+0000 7f575ffff640 1 --2- 172.21.15.49:0/3275135009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd4430 0x55a475cd8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.645+0000 7f575ffff640 1 -- 172.21.15.49:0/3275135009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 msgr2=0x55a475cdba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.645+0000 7f575ffff640 1 --2- 172.21.15.49:0/3275135009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 0x55a475cdba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.645+0000 7f575ffff640 1 -- 172.21.15.49:0/3275135009 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a475c9b500 con 0x55a475cd3a40 2024-03-18T00:56:59.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.646+0000 7f575ffff640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57480a28d0 0x7f57480a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.646+0000 7f575ffff640 1 --2- 172.21.15.49:0/3275135009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd3a40 0x55a475cd3e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f575002c250 tx=0x7f5750009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d3a75fe7d490f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.654+0000 7f5745ffb640 1 -- 172.21.15.49:0/3275135009 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575002cc70 con 0x55a475cd3a40 2024-03-18T00:56:59.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.654+0000 7f5745ffb640 1 -- 172.21.15.49:0/3275135009 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5750044040 con 0x55a475cd3a40 2024-03-18T00:56:59.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.655+0000 7f5745ffb640 1 -- 172.21.15.49:0/3275135009 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57500049b0 con 0x55a475cd3a40 2024-03-18T00:56:59.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.657+0000 7f5768e20740 1 -- 172.21.15.49:0/3275135009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd3a40 msgr2=0x55a475cd3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.657+0000 7f5768e20740 1 --2- 172.21.15.49:0/3275135009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd3a40 0x55a475cd3e60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f575002c250 tx=0x7f5750009580 comp rx=0 tx=0).stop 2024-03-18T00:56:59.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.657+0000 7f575ffff640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57480a28d0 0x7f57480a4d90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5750045000 tx=0x7f5750004c30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.657+0000 7f5768e20740 1 -- 172.21.15.49:0/3275135009 shutdown_connections 2024-03-18T00:56:59.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.657+0000 7f5768e20740 1 --2- 172.21.15.49:0/3275135009 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd4430 0x55a475cd8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.657+0000 7f5768e20740 1 --2- 172.21.15.49:0/3275135009 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 0x55a475cdba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.657+0000 7f5768e20740 1 --2- 172.21.15.49:0/3275135009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd3a40 0x55a475cd3e60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.657+0000 7f5768e20740 1 -- 172.21.15.49:0/3275135009 >> 172.21.15.49:0/3275135009 conn(0x55a475ccefb0 msgr2=0x55a475ccf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.660+0000 7f5768e20740 1 -- 172.21.15.49:0/3275135009 shutdown_connections 2024-03-18T00:56:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.663+0000 7f5768e20740 1 -- 172.21.15.49:0/3275135009 wait complete. 2024-03-18T00:56:59.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.663+0000 7f5768e20740 1 -- start start 2024-03-18T00:56:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.664+0000 7f5768e20740 1 Processor -- start 2024-03-18T00:56:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.665+0000 7f5768e20740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd3a40 0x55a475cddf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.665+0000 7f5768e20740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd4430 0x55a475cde4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.665+0000 7f5768e20740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 0x55a475ce54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.665+0000 7f5768e20740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a475cdc550 con 0x55a475cd95a0 2024-03-18T00:56:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.665+0000 7f5768e20740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a475bf4230 con 0x55a475cd3a40 2024-03-18T00:56:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.665+0000 7f5768e20740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a475cdc6d0 con 0x55a475cd4430 2024-03-18T00:56:59.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.665+0000 7f5768e20740 10 client.0 _ll_register_callbacks cb 0x55a475c30010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:59.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.670+0000 7f575effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd4430 0x55a475cde4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.670+0000 7f575effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd4430 0x55a475cde4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:37808/0 (socket says 172.21.15.49:37808) 2024-03-18T00:56:59.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.670+0000 7f575effd640 1 -- 172.21.15.49:0/1423428348 learned_addr learned my addr 172.21.15.49:0/1423428348 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.671+0000 7f575ffff640 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 0x55a475ce54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.672+0000 7f575effd640 1 -- 172.21.15.49:0/1423428348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd3a40 msgr2=0x55a475cddf60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.672+0000 7f575effd640 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd3a40 0x55a475cddf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.672+0000 7f575effd640 1 -- 172.21.15.49:0/1423428348 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 msgr2=0x55a475ce54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.672+0000 7f575effd640 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 0x55a475ce54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.672+0000 7f575effd640 1 -- 172.21.15.49:0/1423428348 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a475c03100 con 0x55a475cd4430 2024-03-18T00:56:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.676+0000 7f575effd640 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd4430 0x55a475cde4a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5754000c00 tx=0x7f5754004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de9db01f0ab87c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.676+0000 7f573effd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:56:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.676+0000 7f573effd640 1 -- 172.21.15.49:0/1423428348 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754020040 con 0x55a475cd4430 2024-03-18T00:56:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.676+0000 7f5768e20740 1 -- 172.21.15.49:0/1423428348 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5750044040 con 0x55a475cd4430 2024-03-18T00:56:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.677+0000 7f573effd640 1 -- 172.21.15.49:0/1423428348 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5754005510 con 0x55a475cd4430 2024-03-18T00:56:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.677+0000 7f573effd640 1 -- 172.21.15.49:0/1423428348 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f57540a3350 con 0x55a475cd4430 2024-03-18T00:56:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.677+0000 7f5768e20740 10 client.26380 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:59.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.677+0000 7f5768e20740 1 -- 172.21.15.49:0/1423428348 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a47594c3f0 con 0x55a475cd4430 2024-03-18T00:56:59.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.679+0000 7f573effd640 1 -- 172.21.15.49:0/1423428348 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5750044040 con 0x55a475cd4430 2024-03-18T00:56:59.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.679+0000 7f573effd640 1 -- 172.21.15.49:0/1423428348 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f575409a090 con 0x55a475cd4430 2024-03-18T00:56:59.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.679+0000 7f5768e20740 10 client.26380 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:59.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.679+0000 7f5768e20740 10 client.26380 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:56:59.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.679+0000 7f5768e20740 20 client.26380 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:59.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.679+0000 7f5768e20740 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55a475d0bd60 0x55a475d0e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.679+0000 7f5768e20740 4 client.26380 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:56:59.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.679+0000 7f5768e20740 1 -- 172.21.15.49:0/1423428348 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a475c459d0 con 0x55a475d0bd60 2024-03-18T00:56:59.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:59 smithi107 ceph-mon[36270]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 877 MiB/s wr, 2.42k op/s 2024-03-18T00:56:59.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:56:59 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1279677087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:56:59.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.681+0000 7f575f7fe640 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55a475d0bd60 0x55a475d0e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.685+0000 7f575f7fe640 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55a475d0bd60 0x55a475d0e1f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=43f5a6cbd95da5a1 server_cookie=1ba03152f58c18fb in_seq=0 out_seq=0 2024-03-18T00:56:59.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.685+0000 7f573effd640 10 client.26380 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:56:59.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.688+0000 7f573effd640 1 -- 172.21.15.49:0/1423428348 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a475c459d0 con 0x55a475d0bd60 2024-03-18T00:56:59.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.688+0000 7f573effd640 10 client.26380 handle_command_reply: tid=0 2024-03-18T00:56:59.689 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.688+0000 7f5768e20740 1 client.26380 shutdown 2024-03-18T00:56:59.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:59 smithi106 ceph-mon[35899]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 877 MiB/s wr, 2.42k op/s 2024-03-18T00:56:59.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:56:59 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1279677087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:56:59.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.704+0000 7f5768e20740 1 -- 172.21.15.49:0/1423428348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd4430 msgr2=0x55a475cde4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.704+0000 7f5768e20740 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd4430 0x55a475cde4a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5754000c00 tx=0x7f5754004f80 comp rx=0 tx=0).stop 2024-03-18T00:56:59.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.706+0000 7f5768e20740 1 -- 172.21.15.49:0/1423428348 shutdown_connections 2024-03-18T00:56:59.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.706+0000 7f5768e20740 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55a475d0bd60 0x55a475d0e1f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.706+0000 7f5768e20740 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd95a0 0x55a475ce54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.706+0000 7f5768e20740 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd4430 0x55a475cde4a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.706+0000 7f5768e20740 1 --2- 172.21.15.49:0/1423428348 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd3a40 0x55a475cddf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.706+0000 7f573effd640 0 client.26380 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:56:59.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.706+0000 7f5768e20740 1 -- 172.21.15.49:0/1423428348 >> 172.21.15.49:0/1423428348 conn(0x55a475ccefb0 msgr2=0x55a475cce140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.709+0000 7f5768e20740 1 -- 172.21.15.49:0/1423428348 shutdown_connections 2024-03-18T00:56:59.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.709+0000 7f5768e20740 1 -- 172.21.15.49:0/1423428348 wait complete. 2024-03-18T00:56:59.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.710+0000 7f5768e20740 20 client.26380 trim_cache size 0 max 16384 2024-03-18T00:56:59.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.757+0000 7f5768e20740 1 Processor -- start 2024-03-18T00:56:59.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.757+0000 7f5768e20740 1 -- start start 2024-03-18T00:56:59.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.758+0000 7f5768e20740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd3a40 0x55a475cdd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.758+0000 7f5768e20740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd4430 0x55a475ce54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.758+0000 7f5768e20740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd95a0 0x55a475ccd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.758+0000 7f5768e20740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f574c00e550 con 0x55a475cd4430 2024-03-18T00:56:59.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.758+0000 7f5768e20740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f574c010090 con 0x55a475cd3a40 2024-03-18T00:56:59.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.758+0000 7f5768e20740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f574c00ec20 con 0x55a475cd95a0 2024-03-18T00:56:59.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.759+0000 7f575effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd4430 0x55a475ce54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.759+0000 7f575effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd4430 0x55a475ce54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54542/0 (socket says 172.21.15.49:54542) 2024-03-18T00:56:59.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.759+0000 7f575effd640 1 -- 172.21.15.49:0/2004910869 learned_addr learned my addr 172.21.15.49:0/2004910869 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.763+0000 7f575effd640 1 -- 172.21.15.49:0/2004910869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd95a0 msgr2=0x55a475ccd8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:59.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.763+0000 7f575effd640 1 --2- 172.21.15.49:0/2004910869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd95a0 0x55a475ccd8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.763+0000 7f575effd640 1 -- 172.21.15.49:0/2004910869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd3a40 msgr2=0x55a475cdd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.763+0000 7f575effd640 1 --2- 172.21.15.49:0/2004910869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd3a40 0x55a475cdd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.763+0000 7f575effd640 1 -- 172.21.15.49:0/2004910869 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a475a167d0 con 0x55a475cd4430 2024-03-18T00:56:59.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.767+0000 7f575effd640 1 --2- 172.21.15.49:0/2004910869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd4430 0x55a475ce54a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5754009610 tx=0x7f575409a510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ca1becbab86990 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.771+0000 7f57457fa640 1 -- 172.21.15.49:0/2004910869 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5754005510 con 0x55a475cd4430 2024-03-18T00:56:59.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.771+0000 7f57457fa640 1 -- 172.21.15.49:0/2004910869 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57540b92a0 con 0x55a475cd4430 2024-03-18T00:56:59.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.771+0000 7f57457fa640 1 -- 172.21.15.49:0/2004910869 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575409fc20 con 0x55a475cd4430 2024-03-18T00:56:59.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.773+0000 7f5768e20740 1 -- 172.21.15.49:0/2004910869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd4430 msgr2=0x55a475ce54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.773+0000 7f5768e20740 1 --2- 172.21.15.49:0/2004910869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd4430 0x55a475ce54a0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f5754009610 tx=0x7f575409a510 comp rx=0 tx=0).stop 2024-03-18T00:56:59.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.781+0000 7f5768e20740 1 -- 172.21.15.49:0/2004910869 shutdown_connections 2024-03-18T00:56:59.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.781+0000 7f5768e20740 1 --2- 172.21.15.49:0/2004910869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd95a0 0x55a475ccd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.781+0000 7f5768e20740 1 --2- 172.21.15.49:0/2004910869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd4430 0x55a475ce54a0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.781+0000 7f5768e20740 1 --2- 172.21.15.49:0/2004910869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd3a40 0x55a475cdd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.782 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.781+0000 7f5768e20740 1 -- 172.21.15.49:0/2004910869 >> 172.21.15.49:0/2004910869 conn(0x55a475ccefb0 msgr2=0x55a475cdd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.787+0000 7f5768e20740 1 -- 172.21.15.49:0/2004910869 shutdown_connections 2024-03-18T00:56:59.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.787+0000 7f5768e20740 1 -- 172.21.15.49:0/2004910869 wait complete. 2024-03-18T00:56:59.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.788+0000 7f5768e20740 1 -- start start 2024-03-18T00:56:59.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.789+0000 7f5768e20740 1 Processor -- start 2024-03-18T00:56:59.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.791+0000 7f5768e20740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd3a40 0x55a475ce2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.791+0000 7f5768e20740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd4430 0x55a475ce2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.791+0000 7f5768e20740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd95a0 0x55a475ce4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.791+0000 7f5768e20740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f574c00f9c0 con 0x55a475cd3a40 2024-03-18T00:56:59.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.791+0000 7f5768e20740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f574c00f2f0 con 0x55a475cd4430 2024-03-18T00:56:59.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.791+0000 7f5768e20740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f574c00c530 con 0x55a475cd95a0 2024-03-18T00:56:59.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.791+0000 7f5768e20740 10 client.0 _ll_register_callbacks cb 0x7f5754004720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:56:59.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.792+0000 7f575ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd95a0 0x55a475ce4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.793+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd3a40 0x55a475ce2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.793+0000 7f575f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd3a40 0x55a475ce2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54556/0 (socket says 172.21.15.49:54556) 2024-03-18T00:56:59.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.793+0000 7f575f7fe640 1 -- 172.21.15.49:0/1737291139 learned_addr learned my addr 172.21.15.49:0/1737291139 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:56:59.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.794+0000 7f575f7fe640 1 -- 172.21.15.49:0/1737291139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd95a0 msgr2=0x55a475ce4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:56:59.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.794+0000 7f575f7fe640 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd95a0 0x55a475ce4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.794+0000 7f575f7fe640 1 -- 172.21.15.49:0/1737291139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd4430 msgr2=0x55a475ce2ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:56:59.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.794+0000 7f575f7fe640 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd4430 0x55a475ce2ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.794+0000 7f575f7fe640 1 -- 172.21.15.49:0/1737291139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a475ce45f0 con 0x55a475cd3a40 2024-03-18T00:56:59.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.794+0000 7f575f7fe640 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd3a40 0x55a475ce2ab0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5758011710 tx=0x7f5758004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a7afb3f962a478 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:56:59.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.794+0000 7f573effd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:56:59.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.795+0000 7f573effd640 1 -- 172.21.15.49:0/1737291139 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575801e640 con 0x55a475cd3a40 2024-03-18T00:56:59.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.795+0000 7f5768e20740 1 -- 172.21.15.49:0/1737291139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57540b92a0 con 0x55a475cd3a40 2024-03-18T00:56:59.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.795+0000 7f573effd640 1 -- 172.21.15.49:0/1737291139 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5758005040 con 0x55a475cd3a40 2024-03-18T00:56:59.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.795+0000 7f573effd640 1 -- 172.21.15.49:0/1737291139 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5758005280 con 0x55a475cd3a40 2024-03-18T00:56:59.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.796+0000 7f5768e20740 10 client.17049 fetch_fsmap learned FSMap version 27 2024-03-18T00:56:59.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.796+0000 7f5768e20740 1 -- 172.21.15.49:0/1737291139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a475d72150 con 0x55a475cd3a40 2024-03-18T00:56:59.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.798+0000 7f573effd640 1 -- 172.21.15.49:0/1737291139 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5758022030 con 0x55a475cd3a40 2024-03-18T00:56:59.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.799+0000 7f573effd640 1 -- 172.21.15.49:0/1737291139 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f57580317f0 con 0x55a475cd3a40 2024-03-18T00:56:59.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.799+0000 7f5768e20740 10 client.17049 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:56:59.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.799+0000 7f5768e20740 10 client.17049 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:56:59.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.799+0000 7f5768e20740 20 client.17049 populate_metadata read hostname 'smithi049' 2024-03-18T00:56:59.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.799+0000 7f5768e20740 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55a475d72bd0 0x55a475d74ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:56:59.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.799+0000 7f5768e20740 4 client.17049 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:56:59.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.799+0000 7f5768e20740 1 -- 172.21.15.49:0/1737291139 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f57540b92a0 con 0x55a475d72bd0 2024-03-18T00:56:59.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.802+0000 7f575effd640 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55a475d72bd0 0x55a475d74ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:56:59.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.805+0000 7f575effd640 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55a475d72bd0 0x55a475d74ff0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=437a30f388600cb3 server_cookie=337e331976a189fd in_seq=0 out_seq=0 2024-03-18T00:56:59.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.805+0000 7f573effd640 10 client.17049 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:56:59.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f573effd640 1 -- 172.21.15.49:0/1737291139 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f57540b92a0 con 0x55a475d72bd0 2024-03-18T00:56:59.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f573effd640 10 client.17049 handle_command_reply: tid=0 2024-03-18T00:56:59.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 client.17049 shutdown 2024-03-18T00:56:59.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 -- 172.21.15.49:0/1737291139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd3a40 msgr2=0x55a475ce2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd3a40 0x55a475ce2ab0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f5758011710 tx=0x7f5758004e80 comp rx=0 tx=0).stop 2024-03-18T00:56:59.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 -- 172.21.15.49:0/1737291139 shutdown_connections 2024-03-18T00:56:59.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55a475d72bd0 0x55a475d74ff0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a475cd4430 0x55a475ce2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a475cd95a0 0x55a475ce4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 --2- 172.21.15.49:0/1737291139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a475cd3a40 0x55a475ce2ab0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 -- 172.21.15.49:0/1737291139 >> 172.21.15.49:0/1737291139 conn(0x55a475ccefb0 msgr2=0x55a475ccd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f573effd640 0 client.17049 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:56:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 -- 172.21.15.49:0/1737291139 shutdown_connections 2024-03-18T00:56:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.806+0000 7f5768e20740 1 -- 172.21.15.49:0/1737291139 wait complete. 2024-03-18T00:56:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.807+0000 7f5768e20740 20 client.17049 trim_cache size 0 max 16384 2024-03-18T00:56:59.808 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:56:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 -- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57480a28d0 msgr2=0x7f57480a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57480a28d0 0x7f57480a4d90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5750045000 tx=0x7f5750004c30 comp rx=0 tx=0).stop 2024-03-18T00:56:59.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 -- 172.21.15.49:0/2908161132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5760108eb0 msgr2=0x7f57601184a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:56:59.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5760108eb0 0x7f57601184a0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f575400d6d0 tx=0x7f575400db90 comp rx=0 tx=0).stop 2024-03-18T00:56:59.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 -- 172.21.15.49:0/2908161132 shutdown_connections 2024-03-18T00:56:59.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f57480a28d0 0x7f57480a4d90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5760109870 0x7f57601189e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5760108eb0 0x7f57601184a0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 --2- 172.21.15.49:0/2908161132 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5760072bf0 0x7f5760119e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:56:59.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 -- 172.21.15.49:0/2908161132 >> 172.21.15.49:0/2908161132 conn(0x7f576006e6c0 msgr2=0x7f576010c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:56:59.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 -- 172.21.15.49:0/2908161132 shutdown_connections 2024-03-18T00:56:59.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:56:59.808+0000 7f5764fdd640 1 -- 172.21.15.49:0/2908161132 wait complete. 2024-03-18T00:56:59.833 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:57:00.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.222+0000 7fcf6359e640 1 -- 172.21.15.49:0/2860938319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf641005c0 msgr2=0x7fcf641009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.224 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.222+0000 7fcf6359e640 1 --2- 172.21.15.49:0/2860938319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf641005c0 0x7fcf641009e0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009900 tx=0x7fcf5402bc80 comp rx=0 tx=0).stop 2024-03-18T00:57:00.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.223+0000 7fcf6359e640 1 -- 172.21.15.49:0/2860938319 shutdown_connections 2024-03-18T00:57:00.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.223+0000 7fcf6359e640 1 --2- 172.21.15.49:0/2860938319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf641017b0 0x7fcf64101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.223+0000 7fcf6359e640 1 --2- 172.21.15.49:0/2860938319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf64102170 0x7fcf64068940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.223+0000 7fcf6359e640 1 --2- 172.21.15.49:0/2860938319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf641005c0 0x7fcf641009e0 secure :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009900 tx=0x7fcf5402bc80 comp rx=0 tx=0).stop 2024-03-18T00:57:00.225 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.223+0000 7fcf6359e640 1 -- 172.21.15.49:0/2860938319 >> 172.21.15.49:0/2860938319 conn(0x7fcf640fbb50 msgr2=0x7fcf640fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.226 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.226+0000 7fcf6359e640 1 -- 172.21.15.49:0/2860938319 shutdown_connections 2024-03-18T00:57:00.227 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.227+0000 7fcf6359e640 1 -- 172.21.15.49:0/2860938319 wait complete. 2024-03-18T00:57:00.229 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.229+0000 7fcf6359e640 1 Processor -- start 2024-03-18T00:57:00.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.230+0000 7fcf6359e640 1 -- start start 2024-03-18T00:57:00.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.230+0000 7fcf6359e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf64102170 0x7fcf6419e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.230+0000 7fcf6359e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf6419ea10 0x7fcf641a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.230+0000 7fcf6359e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf6419ee90 0x7fcf641a6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.230+0000 7fcf6359e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6410b2a0 con 0x7fcf6419ea10 2024-03-18T00:57:00.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.230+0000 7fcf6359e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6410b120 con 0x7fcf6419ee90 2024-03-18T00:57:00.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.230+0000 7fcf6359e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6410b420 con 0x7fcf64102170 2024-03-18T00:57:00.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.231+0000 7fcf62d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf6419ee90 0x7fcf641a6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.231+0000 7fcf62d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf6419ee90 0x7fcf641a6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:44738/0 (socket says 172.21.15.49:44738) 2024-03-18T00:57:00.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.231+0000 7fcf62d9d640 1 -- 172.21.15.49:0/3779153992 learned_addr learned my addr 172.21.15.49:0/3779153992 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.231+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf6419ea10 0x7fcf641a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.234+0000 7fcf6259c640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf64102170 0x7fcf6419e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.236+0000 7fcf62d9d640 1 -- 172.21.15.49:0/3779153992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf64102170 msgr2=0x7fcf6419e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.236+0000 7fcf62d9d640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf64102170 0x7fcf6419e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.236+0000 7fcf62d9d640 1 -- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf6419ea10 msgr2=0x7fcf641a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.236+0000 7fcf62d9d640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf6419ea10 0x7fcf641a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.236+0000 7fcf62d9d640 1 -- 172.21.15.49:0/3779153992 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf641a68c0 con 0x7fcf6419ee90 2024-03-18T00:57:00.238 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.237+0000 7fcf62d9d640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf6419ee90 0x7fcf641a6000 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00b990 tx=0x7fcf5c00be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.239+0000 7fcf5b7fe640 1 -- 172.21.15.49:0/3779153992 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c019070 con 0x7fcf6419ee90 2024-03-18T00:57:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.239+0000 7fcf5b7fe640 1 -- 172.21.15.49:0/3779153992 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf5c00cba0 con 0x7fcf6419ee90 2024-03-18T00:57:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.239+0000 7fcf5b7fe640 1 -- 172.21.15.49:0/3779153992 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c0076e0 con 0x7fcf6419ee90 2024-03-18T00:57:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.239+0000 7fcf6359e640 1 -- 172.21.15.49:0/3779153992 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf641a6bb0 con 0x7fcf6419ee90 2024-03-18T00:57:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.243+0000 7fcf6359e640 1 -- 172.21.15.49:0/3779153992 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf641a70b0 con 0x7fcf6419ee90 2024-03-18T00:57:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.247+0000 7fcf5b7fe640 1 -- 172.21.15.49:0/3779153992 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcf5c014440 con 0x7fcf6419ee90 2024-03-18T00:57:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.249+0000 7fcf5b7fe640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcf440a29d0 0x7fcf440a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.250+0000 7fcf6259c640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcf440a29d0 0x7fcf440a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.250+0000 7fcf5b7fe640 1 -- 172.21.15.49:0/3779153992 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcf5c0ce2d0 con 0x7fcf6419ee90 2024-03-18T00:57:00.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.250+0000 7fcf6259c640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcf440a29d0 0x7fcf440a4e90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009230 tx=0x7fcf5402c8d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.251+0000 7fcf6bbc7740 1 Processor -- start 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.252+0000 7fcf6bbc7740 1 -- start start 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.252+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 0x562e0a426e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.252+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a42bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.252+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a42c5a0 0x562e0a42ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.252+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x562e0a38b490 con 0x562e0a427430 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.252+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x562e0a333c90 con 0x562e0a426a40 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.252+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562e0a387530 con 0x562e0a42c5a0 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.253+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a42bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.253+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a42bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54578/0 (socket says 172.21.15.49:54578) 2024-03-18T00:57:00.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.253+0000 7fcf5bfff640 1 -- 172.21.15.49:0/4074033888 learned_addr learned my addr 172.21.15.49:0/4074033888 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.254+0000 7fcf5bfff640 1 -- 172.21.15.49:0/4074033888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a42c5a0 msgr2=0x562e0a42ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.254+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/4074033888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a42c5a0 0x562e0a42ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.254+0000 7fcf5bfff640 1 -- 172.21.15.49:0/4074033888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 msgr2=0x562e0a426e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:00.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.254+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/4074033888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 0x562e0a426e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.254+0000 7fcf5bfff640 1 -- 172.21.15.49:0/4074033888 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e0a3ee500 con 0x562e0a427430 2024-03-18T00:57:00.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.255+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/4074033888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a42bf30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009960 tx=0x7fcf4c009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2409ee3b7ffb001 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.257+0000 7fcf58ff9640 1 -- 172.21.15.49:0/4074033888 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c017070 con 0x562e0a427430 2024-03-18T00:57:00.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.257+0000 7fcf58ff9640 1 -- 172.21.15.49:0/4074033888 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf4c013470 con 0x562e0a427430 2024-03-18T00:57:00.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.257+0000 7fcf58ff9640 1 -- 172.21.15.49:0/4074033888 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c012610 con 0x562e0a427430 2024-03-18T00:57:00.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.259+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/4074033888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 msgr2=0x562e0a42bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.259+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/4074033888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a42bf30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c009960 tx=0x7fcf4c009e20 comp rx=0 tx=0).stop 2024-03-18T00:57:00.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.259+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/4074033888 shutdown_connections 2024-03-18T00:57:00.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.259+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/4074033888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a42c5a0 0x562e0a42ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.259+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/4074033888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a42bf30 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.259+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/4074033888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 0x562e0a426e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.259+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/4074033888 >> 172.21.15.49:0/4074033888 conn(0x562e0a421fb0 msgr2=0x562e0a4223b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.259+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/4074033888 shutdown_connections 2024-03-18T00:57:00.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.259+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/4074033888 wait complete. 2024-03-18T00:57:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.260+0000 7fcf6bbc7740 1 -- start start 2024-03-18T00:57:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.260+0000 7fcf6bbc7740 1 Processor -- start 2024-03-18T00:57:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a426a40 0x562e0a442090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a4425d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a42c5a0 0x562e0a449610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x562e0a440560 con 0x562e0a427430 2024-03-18T00:57:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x562e0a347230 con 0x562e0a42c5a0 2024-03-18T00:57:00.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562e0a4406e0 con 0x562e0a426a40 2024-03-18T00:57:00.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf6bbc7740 10 client.0 _ll_register_callbacks cb 0x562e0a383010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:00.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a4425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a4425d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54594/0 (socket says 172.21.15.49:54594) 2024-03-18T00:57:00.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.261+0000 7fcf5bfff640 1 -- 172.21.15.49:0/2810858417 learned_addr learned my addr 172.21.15.49:0/2810858417 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.262+0000 7fcf5bfff640 1 -- 172.21.15.49:0/2810858417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a426a40 msgr2=0x562e0a442090 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:00.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.262+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a426a40 0x562e0a442090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.262+0000 7fcf5bfff640 1 -- 172.21.15.49:0/2810858417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a42c5a0 msgr2=0x562e0a449610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.262+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a42c5a0 0x562e0a449610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.262+0000 7fcf5bfff640 1 -- 172.21.15.49:0/2810858417 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e0a356100 con 0x562e0a427430 2024-03-18T00:57:00.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.263+0000 7fcf51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:00.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.263+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a4425d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c013850 tx=0x7fcf4c013880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468d58b839431c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.264+0000 7fcf51ffb640 1 -- 172.21.15.49:0/2810858417 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c017050 con 0x562e0a427430 2024-03-18T00:57:00.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.264+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/2810858417 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf4c013470 con 0x562e0a427430 2024-03-18T00:57:00.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.264+0000 7fcf51ffb640 1 -- 172.21.15.49:0/2810858417 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcf4c01c650 con 0x562e0a427430 2024-03-18T00:57:00.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.272+0000 7fcf51ffb640 1 -- 172.21.15.49:0/2810858417 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcf4c01c850 con 0x562e0a427430 2024-03-18T00:57:00.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.273+0000 7fcf6bbc7740 10 client.17061 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:00.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.273+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/2810858417 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562e0a09f3f0 con 0x562e0a427430 2024-03-18T00:57:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.275+0000 7fcf51ffb640 1 -- 172.21.15.49:0/2810858417 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf4c013470 con 0x562e0a427430 2024-03-18T00:57:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.275+0000 7fcf51ffb640 1 -- 172.21.15.49:0/2810858417 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcf4c031a00 con 0x562e0a427430 2024-03-18T00:57:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.275+0000 7fcf6bbc7740 10 client.17061 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.275+0000 7fcf6bbc7740 10 client.17061 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:57:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.275+0000 7fcf6bbc7740 20 client.17061 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.275+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562e0a44e1c0 0x562e0a450640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.275+0000 7fcf6bbc7740 4 client.17061 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:00.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.275+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/2810858417 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e0a3989d0 con 0x562e0a44e1c0 2024-03-18T00:57:00.277 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.277+0000 7fcf6259c640 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562e0a44e1c0 0x562e0a450640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.281+0000 7fcf6259c640 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562e0a44e1c0 0x562e0a450640 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=faf9fbeef831f3a9 server_cookie=fd636ede207dbdde in_seq=0 out_seq=0 2024-03-18T00:57:00.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.281+0000 7fcf51ffb640 10 client.17061 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.282+0000 7fcf51ffb640 1 -- 172.21.15.49:0/2810858417 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562e0a3989d0 con 0x562e0a44e1c0 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.282+0000 7fcf51ffb640 10 client.17061 handle_command_reply: tid=0 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 client.17061 shutdown 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/2810858417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 msgr2=0x562e0a4425d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a4425d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c013850 tx=0x7fcf4c013880 comp rx=0 tx=0).stop 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/2810858417 shutdown_connections 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562e0a44e1c0 0x562e0a450640 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a42c5a0 0x562e0a449610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a427430 0x562e0a4425d0 secure :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c013850 tx=0x7fcf4c013880 comp rx=0 tx=0).stop 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/2810858417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a426a40 0x562e0a442090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/2810858417 >> 172.21.15.49:0/2810858417 conn(0x562e0a421fb0 msgr2=0x562e0a421140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf51ffb640 0 client.17061 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:57:00.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.283+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/2810858417 shutdown_connections 2024-03-18T00:57:00.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.284+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/2810858417 wait complete. 2024-03-18T00:57:00.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.284+0000 7fcf6bbc7740 20 client.17061 trim_cache size 0 max 16384 2024-03-18T00:57:00.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.310+0000 7fcf6bbc7740 1 Processor -- start 2024-03-18T00:57:00.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.310+0000 7fcf6bbc7740 1 -- start start 2024-03-18T00:57:00.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.311+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a426a40 0x562e0a440ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.311+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a42c5a0 0x562e0a449530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.311+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a428a70 0x562e0a428ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.311+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcf641a75b0 con 0x562e0a426a40 2024-03-18T00:57:00.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.311+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcf641a90f0 con 0x562e0a42c5a0 2024-03-18T00:57:00.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.311+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcf641a7c80 con 0x562e0a428a70 2024-03-18T00:57:00.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.311+0000 7fcf6259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a426a40 0x562e0a440ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.311+0000 7fcf6259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a426a40 0x562e0a440ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54598/0 (socket says 172.21.15.49:54598) 2024-03-18T00:57:00.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.311+0000 7fcf6259c640 1 -- 172.21.15.49:0/3291381211 learned_addr learned my addr 172.21.15.49:0/3291381211 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.312+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/3291381211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a42c5a0 0x562e0a449530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.312+0000 7fcf62d9d640 1 --2- 172.21.15.49:0/3291381211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a428a70 0x562e0a428ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.312+0000 7fcf6259c640 1 -- 172.21.15.49:0/3291381211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a428a70 msgr2=0x562e0a428ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.312+0000 7fcf6259c640 1 --2- 172.21.15.49:0/3291381211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a428a70 0x562e0a428ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.312+0000 7fcf6259c640 1 -- 172.21.15.49:0/3291381211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a42c5a0 msgr2=0x562e0a449530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.312+0000 7fcf6259c640 1 --2- 172.21.15.49:0/3291381211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a42c5a0 0x562e0a449530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.312+0000 7fcf6259c640 1 -- 172.21.15.49:0/3291381211 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e0a1697d0 con 0x562e0a426a40 2024-03-18T00:57:00.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.313+0000 7fcf6259c640 1 --2- 172.21.15.49:0/3291381211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a426a40 0x562e0a440ab0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcf5402c800 tx=0x7fcf5403f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508a2bd0fc5df6a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.313+0000 7fcf53fff640 1 -- 172.21.15.49:0/3291381211 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54052bb0 con 0x562e0a426a40 2024-03-18T00:57:00.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.314+0000 7fcf53fff640 1 -- 172.21.15.49:0/3291381211 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf54052d50 con 0x562e0a426a40 2024-03-18T00:57:00.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.314+0000 7fcf53fff640 1 -- 172.21.15.49:0/3291381211 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54052250 con 0x562e0a426a40 2024-03-18T00:57:00.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.316+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/3291381211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a426a40 msgr2=0x562e0a440ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.316+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/3291381211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a426a40 0x562e0a440ab0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fcf5402c800 tx=0x7fcf5403f000 comp rx=0 tx=0).stop 2024-03-18T00:57:00.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.317+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/3291381211 shutdown_connections 2024-03-18T00:57:00.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.317+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/3291381211 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a42c5a0 0x562e0a449530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.317+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/3291381211 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a428a70 0x562e0a428ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.317+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/3291381211 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a426a40 0x562e0a440ab0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.317+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/3291381211 >> 172.21.15.49:0/3291381211 conn(0x562e0a421fb0 msgr2=0x562e0a442090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.317+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/3291381211 shutdown_connections 2024-03-18T00:57:00.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.318+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/3291381211 wait complete. 2024-03-18T00:57:00.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.318+0000 7fcf6bbc7740 1 -- start start 2024-03-18T00:57:00.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.319+0000 7fcf6bbc7740 1 Processor -- start 2024-03-18T00:57:00.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.320+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 0x562e0a446460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.320+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a428a70 0x562e0a4469a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.321+0000 7fcf6bbc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a42c5a0 0x562e0a4b2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.321+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a428a70 0x562e0a4469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.321+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcf641a8a20 con 0x562e0a428a70 2024-03-18T00:57:00.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.321+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcf641a8350 con 0x562e0a426a40 2024-03-18T00:57:00.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.321+0000 7fcf6bbc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcf64071c70 con 0x562e0a42c5a0 2024-03-18T00:57:00.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.321+0000 7fcf5bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a428a70 0x562e0a4469a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54604/0 (socket says 172.21.15.49:54604) 2024-03-18T00:57:00.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.321+0000 7fcf5bfff640 1 -- 172.21.15.49:0/723851843 learned_addr learned my addr 172.21.15.49:0/723851843 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.321+0000 7fcf6bbc7740 10 client.0 _ll_register_callbacks cb 0x7fcf4c0097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:00.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.322+0000 7fcf6259c640 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 0x562e0a446460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.324+0000 7fcf62d9d640 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a42c5a0 0x562e0a4b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.325+0000 7fcf6259c640 1 -- 172.21.15.49:0/723851843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a42c5a0 msgr2=0x562e0a4b2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.325+0000 7fcf6259c640 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a42c5a0 0x562e0a4b2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.325+0000 7fcf6259c640 1 -- 172.21.15.49:0/723851843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a428a70 msgr2=0x562e0a4469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.325+0000 7fcf6259c640 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a428a70 0x562e0a4469a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.325+0000 7fcf6259c640 1 -- 172.21.15.49:0/723851843 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e0a447040 con 0x562e0a426a40 2024-03-18T00:57:00.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.329+0000 7fcf6259c640 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 0x562e0a446460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009900 tx=0x7fcf54004a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d10dd9418430b85c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.329+0000 7fcf51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:57:00.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.330+0000 7fcf51ffb640 1 -- 172.21.15.49:0/723851843 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54062060 con 0x562e0a426a40 2024-03-18T00:57:00.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.331+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/723851843 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf54052d50 con 0x562e0a426a40 2024-03-18T00:57:00.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.331+0000 7fcf51ffb640 1 -- 172.21.15.49:0/723851843 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcf54004040 con 0x562e0a426a40 2024-03-18T00:57:00.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.334+0000 7fcf51ffb640 1 -- 172.21.15.49:0/723851843 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fcf54004240 con 0x562e0a426a40 2024-03-18T00:57:00.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.338+0000 7fcf6bbc7740 10 client.26294 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:00.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.338+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/723851843 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562e0a4b3be0 con 0x562e0a426a40 2024-03-18T00:57:00.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.340+0000 7fcf51ffb640 1 -- 172.21.15.49:0/723851843 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf54052d50 con 0x562e0a426a40 2024-03-18T00:57:00.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.340+0000 7fcf51ffb640 1 -- 172.21.15.49:0/723851843 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fcf540035f0 con 0x562e0a426a40 2024-03-18T00:57:00.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.341+0000 7fcf6bbc7740 10 client.26294 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:00.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.341+0000 7fcf6bbc7740 10 client.26294 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:57:00.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.341+0000 7fcf6bbc7740 20 client.26294 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:00.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.341+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562e0a4b44d0 0x562e0a4b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.341+0000 7fcf6bbc7740 4 client.26294 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:57:00.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.341+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/723851843 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562e0a441190 con 0x562e0a4b44d0 2024-03-18T00:57:00.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.344+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562e0a4b44d0 0x562e0a4b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.348+0000 7fcf5bfff640 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562e0a4b44d0 0x562e0a4b68f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27b26d8f47a4845a server_cookie=9c3c5cbf8ceb39a8 in_seq=0 out_seq=0 2024-03-18T00:57:00.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.348+0000 7fcf51ffb640 10 client.26294 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:57:00.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.350+0000 7fcf51ffb640 1 -- 172.21.15.49:0/723851843 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562e0a441190 con 0x562e0a4b44d0 2024-03-18T00:57:00.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.350+0000 7fcf51ffb640 10 client.26294 handle_command_reply: tid=0 2024-03-18T00:57:00.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.350+0000 7fcf6bbc7740 1 client.26294 shutdown 2024-03-18T00:57:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.351+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/723851843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 msgr2=0x562e0a446460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.351+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 0x562e0a446460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009900 tx=0x7fcf54004a30 comp rx=0 tx=0).stop 2024-03-18T00:57:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.351+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/723851843 shutdown_connections 2024-03-18T00:57:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.351+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x562e0a4b44d0 0x562e0a4b68f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.351+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562e0a42c5a0 0x562e0a4b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.351+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562e0a428a70 0x562e0a4469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.351+0000 7fcf6bbc7740 1 --2- 172.21.15.49:0/723851843 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562e0a426a40 0x562e0a446460 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.351+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/723851843 >> 172.21.15.49:0/723851843 conn(0x562e0a421fb0 msgr2=0x562e0a427b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.352+0000 7fcf51ffb640 0 client.26294 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:57:00.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.352+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/723851843 shutdown_connections 2024-03-18T00:57:00.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.352+0000 7fcf6bbc7740 1 -- 172.21.15.49:0/723851843 wait complete. 2024-03-18T00:57:00.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.352+0000 7fcf6bbc7740 20 client.26294 trim_cache size 0 max 16384 2024-03-18T00:57:00.354 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:57:00.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.358+0000 7fcf6359e640 1 -- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcf440a29d0 msgr2=0x7fcf440a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.358+0000 7fcf6359e640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcf440a29d0 0x7fcf440a4e90 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009230 tx=0x7fcf5402c8d0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.358+0000 7fcf6359e640 1 -- 172.21.15.49:0/3779153992 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf6419ee90 msgr2=0x7fcf641a6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.358+0000 7fcf6359e640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf6419ee90 0x7fcf641a6000 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00b990 tx=0x7fcf5c00be50 comp rx=0 tx=0).stop 2024-03-18T00:57:00.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.359+0000 7fcf6359e640 1 -- 172.21.15.49:0/3779153992 shutdown_connections 2024-03-18T00:57:00.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.359+0000 7fcf6359e640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcf440a29d0 0x7fcf440a4e90 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.359+0000 7fcf6359e640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcf6419ee90 0x7fcf641a6000 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.359+0000 7fcf6359e640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcf6419ea10 0x7fcf641a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.359+0000 7fcf6359e640 1 --2- 172.21.15.49:0/3779153992 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf64102170 0x7fcf6419e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.359+0000 7fcf6359e640 1 -- 172.21.15.49:0/3779153992 >> 172.21.15.49:0/3779153992 conn(0x7fcf640fbb50 msgr2=0x7fcf6410bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.360+0000 7fcf6359e640 1 -- 172.21.15.49:0/3779153992 shutdown_connections 2024-03-18T00:57:00.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.363+0000 7fcf6359e640 1 -- 172.21.15.49:0/3779153992 wait complete. 2024-03-18T00:57:00.421 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:57:00.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.734+0000 7fae42310640 1 -- 172.21.15.49:0/4148007647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c102340 msgr2=0x7fae3c104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.734+0000 7fae42310640 1 --2- 172.21.15.49:0/4148007647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c102340 0x7fae3c104730 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009900 tx=0x7fae2c02bc80 comp rx=0 tx=0).stop 2024-03-18T00:57:00.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.735+0000 7fae42310640 1 -- 172.21.15.49:0/4148007647 shutdown_connections 2024-03-18T00:57:00.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.735+0000 7fae42310640 1 --2- 172.21.15.49:0/4148007647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae3c1075a0 0x7fae3c109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.735+0000 7fae42310640 1 --2- 172.21.15.49:0/4148007647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae3c104c70 0x7fae3c107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.735+0000 7fae42310640 1 --2- 172.21.15.49:0/4148007647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c102340 0x7fae3c104730 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.735+0000 7fae42310640 1 -- 172.21.15.49:0/4148007647 >> 172.21.15.49:0/4148007647 conn(0x7fae3c0fbc90 msgr2=0x7fae3c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.735+0000 7fae42310640 1 -- 172.21.15.49:0/4148007647 shutdown_connections 2024-03-18T00:57:00.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.735+0000 7fae42310640 1 -- 172.21.15.49:0/4148007647 wait complete. 2024-03-18T00:57:00.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.736+0000 7fae42310640 1 Processor -- start 2024-03-18T00:57:00.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.736+0000 7fae42310640 1 -- start start 2024-03-18T00:57:00.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.736+0000 7fae42310640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae3c102340 0x7fae3c197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.736+0000 7fae42310640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae3c104c70 0x7fae3c1981f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.736+0000 7fae42310640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c1075a0 0x7fae3c19f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.736+0000 7fae42310640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c10d5b0 con 0x7fae3c1075a0 2024-03-18T00:57:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.736+0000 7fae42310640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c10d430 con 0x7fae3c102340 2024-03-18T00:57:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.736+0000 7fae42310640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c10d730 con 0x7fae3c104c70 2024-03-18T00:57:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae408ad640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c1075a0 0x7fae3c19f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae408ad640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c1075a0 0x7fae3c19f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54626/0 (socket says 172.21.15.49:54626) 2024-03-18T00:57:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae408ad640 1 -- 172.21.15.49:0/554997203 learned_addr learned my addr 172.21.15.49:0/554997203 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae408ad640 1 -- 172.21.15.49:0/554997203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae3c104c70 msgr2=0x7fae3c1981f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:57:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae408ad640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae3c104c70 0x7fae3c1981f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae3bfff640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae3c102340 0x7fae3c197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae408ad640 1 -- 172.21.15.49:0/554997203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae3c102340 msgr2=0x7fae3c197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae408ad640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae3c102340 0x7fae3c197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae408ad640 1 -- 172.21.15.49:0/554997203 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae3c19f8f0 con 0x7fae3c1075a0 2024-03-18T00:57:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae3bfff640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae3c102340 0x7fae3c197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:57:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.737+0000 7fae408ad640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c1075a0 0x7fae3c19f250 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fae3400b990 tx=0x7fae3400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.739+0000 7fae397fa640 1 -- 172.21.15.49:0/554997203 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae34019070 con 0x7fae3c1075a0 2024-03-18T00:57:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.739+0000 7fae397fa640 1 -- 172.21.15.49:0/554997203 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae3400cba0 con 0x7fae3c1075a0 2024-03-18T00:57:00.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.739+0000 7fae42310640 1 -- 172.21.15.49:0/554997203 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae3c19fbe0 con 0x7fae3c1075a0 2024-03-18T00:57:00.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.739+0000 7fae397fa640 1 -- 172.21.15.49:0/554997203 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae34007650 con 0x7fae3c1075a0 2024-03-18T00:57:00.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.739+0000 7fae42310640 1 -- 172.21.15.49:0/554997203 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae3c1a0130 con 0x7fae3c1075a0 2024-03-18T00:57:00.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.743+0000 7fae44fd9740 1 Processor -- start 2024-03-18T00:57:00.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.744+0000 7fae397fa640 1 -- 172.21.15.49:0/554997203 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fae3401c070 con 0x7fae3c1075a0 2024-03-18T00:57:00.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.744+0000 7fae44fd9740 1 -- start start 2024-03-18T00:57:00.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.744+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8e8850 0x563a1b8e8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.744 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.744+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b8edd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.744+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8ee3b0 0x563a1b8f0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.744+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b8edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.744+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b8edd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54642/0 (socket says 172.21.15.49:54642) 2024-03-18T00:57:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.744+0000 7fae3bfff640 1 -- 172.21.15.49:0/3862280642 learned_addr learned my addr 172.21.15.49:0/3862280642 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae44fd9740 1 -- 172.21.15.49:0/3862280642 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563a1b84d490 con 0x563a1b8e9240 2024-03-18T00:57:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae44fd9740 1 -- 172.21.15.49:0/3862280642 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563a1b7f5c90 con 0x563a1b8e8850 2024-03-18T00:57:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae44fd9740 1 -- 172.21.15.49:0/3862280642 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563a1b849530 con 0x563a1b8ee3b0 2024-03-18T00:57:00.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/3862280642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8ee3b0 0x563a1b8f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae397fa640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fae240a2880 0x7fae240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae3bfff640 1 -- 172.21.15.49:0/3862280642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8ee3b0 msgr2=0x563a1b8f0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae3bfff640 1 --2- 172.21.15.49:0/3862280642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8ee3b0 0x563a1b8f0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae3bfff640 1 -- 172.21.15.49:0/3862280642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8e8850 msgr2=0x563a1b8e8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae3bfff640 1 --2- 172.21.15.49:0/3862280642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8e8850 0x563a1b8e8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.745+0000 7fae3bfff640 1 -- 172.21.15.49:0/3862280642 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a1b8b0500 con 0x563a1b8e9240 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.746+0000 7fae408ad640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fae240a2880 0x7fae240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.746+0000 7fae397fa640 1 -- 172.21.15.49:0/554997203 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fae340cdfd0 con 0x7fae3c1075a0 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.746+0000 7fae408ad640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fae240a2880 0x7fae240a4d40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fae340a2c60 tx=0x7fae34014c20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.747+0000 7fae3bfff640 1 --2- 172.21.15.49:0/3862280642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b8edd40 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fae2c000c00 tx=0x7fae2c02c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e7c3316de118ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.747+0000 7fae227fc640 1 -- 172.21.15.49:0/3862280642 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c042070 con 0x563a1b8e9240 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.747+0000 7fae227fc640 1 -- 172.21.15.49:0/3862280642 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae2c02cc50 con 0x563a1b8e9240 2024-03-18T00:57:00.747 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.747+0000 7fae227fc640 1 -- 172.21.15.49:0/3862280642 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c004900 con 0x563a1b8e9240 2024-03-18T00:57:00.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.748+0000 7fae44fd9740 1 -- 172.21.15.49:0/3862280642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 msgr2=0x563a1b8edd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.748+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3862280642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b8edd40 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fae2c000c00 tx=0x7fae2c02c6a0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.748+0000 7fae44fd9740 1 -- 172.21.15.49:0/3862280642 shutdown_connections 2024-03-18T00:57:00.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.748+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3862280642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8ee3b0 0x563a1b8f0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.748+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3862280642 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b8edd40 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.748+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3862280642 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8e8850 0x563a1b8e8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.748+0000 7fae44fd9740 1 -- 172.21.15.49:0/3862280642 >> 172.21.15.49:0/3862280642 conn(0x563a1b8e3dc0 msgr2=0x563a1b8e41c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.748+0000 7fae44fd9740 1 -- 172.21.15.49:0/3862280642 shutdown_connections 2024-03-18T00:57:00.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.748+0000 7fae44fd9740 1 -- 172.21.15.49:0/3862280642 wait complete. 2024-03-18T00:57:00.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.749+0000 7fae44fd9740 1 -- start start 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.749+0000 7fae44fd9740 1 Processor -- start 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8e8850 0x563a1b8fb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e9240 0x563a1b8fba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8ee3b0 0x563a1b902a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae44fd9740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563a1b8f1300 con 0x563a1b8ee3b0 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae44fd9740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563a1b809230 con 0x563a1b8e8850 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae44fd9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563a1b8f1480 con 0x563a1b8e9240 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae44fd9740 10 client.0 _ll_register_callbacks cb 0x563a1b845010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae3b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8ee3b0 0x563a1b902a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae3b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8ee3b0 0x563a1b902a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54658/0 (socket says 172.21.15.49:54658) 2024-03-18T00:57:00.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.750+0000 7fae3b7fe640 1 -- 172.21.15.49:0/3996912037 learned_addr learned my addr 172.21.15.49:0/3996912037 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.751+0000 7fae3b7fe640 1 -- 172.21.15.49:0/3996912037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e9240 msgr2=0x563a1b8fba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.751+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e9240 0x563a1b8fba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.751+0000 7fae3b7fe640 1 -- 172.21.15.49:0/3996912037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8e8850 msgr2=0x563a1b8fb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.751 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.751+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8e8850 0x563a1b8fb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.751+0000 7fae3b7fe640 1 -- 172.21.15.49:0/3996912037 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a1b818100 con 0x563a1b8ee3b0 2024-03-18T00:57:00.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.751+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8ee3b0 0x563a1b902a90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fae3000d110 tx=0x7fae3000d5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eaa0bcd01330a04 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.751+0000 7fae1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:00.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.752+0000 7fae1b7fe640 1 -- 172.21.15.49:0/3996912037 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30004020 con 0x563a1b8ee3b0 2024-03-18T00:57:00.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.752+0000 7fae1b7fe640 1 -- 172.21.15.49:0/3996912037 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fae300041c0 con 0x563a1b8ee3b0 2024-03-18T00:57:00.752 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.752+0000 7fae44fd9740 1 -- 172.21.15.49:0/3996912037 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae2c02cc50 con 0x563a1b8ee3b0 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.752+0000 7fae1b7fe640 1 -- 172.21.15.49:0/3996912037 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fae300043c0 con 0x563a1b8ee3b0 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.752+0000 7fae44fd9740 10 client.17079 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.752+0000 7fae44fd9740 1 -- 172.21.15.49:0/3996912037 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a1b5613f0 con 0x563a1b8ee3b0 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.753+0000 7fae1b7fe640 1 -- 172.21.15.49:0/3996912037 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae2c02cc50 con 0x563a1b8ee3b0 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.753+0000 7fae1b7fe640 1 -- 172.21.15.49:0/3996912037 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fae30013760 con 0x563a1b8ee3b0 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.753+0000 7fae44fd9740 10 client.17079 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.753+0000 7fae44fd9740 10 client.17079 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.753+0000 7fae44fd9740 20 client.17079 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.753+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x563a1b907440 0x563a1b909900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.753+0000 7fae44fd9740 4 client.17079 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:00.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.753+0000 7fae44fd9740 1 -- 172.21.15.49:0/3996912037 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a1b85a9d0 con 0x563a1b907440 2024-03-18T00:57:00.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.755+0000 7fae3bfff640 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x563a1b907440 0x563a1b909900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.757+0000 7fae3bfff640 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x563a1b907440 0x563a1b909900 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e1e54eae238f8225 server_cookie=183413f425443ca8 in_seq=0 out_seq=0 2024-03-18T00:57:00.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.757+0000 7fae1b7fe640 10 client.17079 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:57:00.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.758+0000 7fae1b7fe640 1 -- 172.21.15.49:0/3996912037 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a1b85a9d0 con 0x563a1b907440 2024-03-18T00:57:00.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.758+0000 7fae1b7fe640 10 client.17079 handle_command_reply: tid=0 2024-03-18T00:57:00.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.758+0000 7fae44fd9740 1 client.17079 shutdown 2024-03-18T00:57:00.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.759+0000 7fae44fd9740 1 -- 172.21.15.49:0/3996912037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8ee3b0 msgr2=0x563a1b902a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.759+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8ee3b0 0x563a1b902a90 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fae3000d110 tx=0x7fae3000d5d0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae44fd9740 1 -- 172.21.15.49:0/3996912037 shutdown_connections 2024-03-18T00:57:00.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x563a1b907440 0x563a1b909900 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8ee3b0 0x563a1b902a90 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e9240 0x563a1b8fba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae44fd9740 1 --2- 172.21.15.49:0/3996912037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8e8850 0x563a1b8fb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae44fd9740 1 -- 172.21.15.49:0/3996912037 >> 172.21.15.49:0/3996912037 conn(0x563a1b8e3dc0 msgr2=0x563a1b8e2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae1b7fe640 0 client.17079 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:57:00.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae44fd9740 1 -- 172.21.15.49:0/3996912037 shutdown_connections 2024-03-18T00:57:00.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae44fd9740 1 -- 172.21.15.49:0/3996912037 wait complete. 2024-03-18T00:57:00.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.760+0000 7fae44fd9740 20 client.17079 trim_cache size 0 max 16384 2024-03-18T00:57:00.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.783+0000 7fae44fd9740 1 Processor -- start 2024-03-18T00:57:00.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.783+0000 7fae44fd9740 1 -- start start 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.783+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 0x563a1b8e2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.783+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b902a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.783+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 0x563a1b8e2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.783+0000 7fae44fd9740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c1a6cd0 con 0x563a1b8e9240 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.783+0000 7fae44fd9740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c1a8810 con 0x563a1b8ee3b0 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.783+0000 7fae44fd9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c1a73a0 con 0x563a1b8e8850 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b902a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b902a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54668/0 (socket says 172.21.15.49:54668) 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3bfff640 1 -- 172.21.15.49:0/2128974327 learned_addr learned my addr 172.21.15.49:0/2128974327 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/2128974327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 0x563a1b8e2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/2128974327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 0x563a1b8e2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3b7fe640 1 -- 172.21.15.49:0/2128974327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 msgr2=0x563a1b8e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/2128974327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 0x563a1b8e2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3b7fe640 1 -- 172.21.15.49:0/2128974327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 msgr2=0x563a1b902a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/2128974327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b902a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.784+0000 7fae3b7fe640 1 -- 172.21.15.49:0/2128974327 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a1b62b7d0 con 0x563a1b8e8850 2024-03-18T00:57:00.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.785+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/2128974327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 0x563a1b8e2750 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fae30004930 tx=0x7fae3000d050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64cf2f6a2928a24a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.785+0000 7fae21ffb640 1 -- 172.21.15.49:0/2128974327 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30013640 con 0x563a1b8e8850 2024-03-18T00:57:00.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.785+0000 7fae21ffb640 1 -- 172.21.15.49:0/2128974327 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae300137e0 con 0x563a1b8e8850 2024-03-18T00:57:00.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.785+0000 7fae21ffb640 1 -- 172.21.15.49:0/2128974327 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3001c400 con 0x563a1b8e8850 2024-03-18T00:57:00.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.786+0000 7fae44fd9740 1 -- 172.21.15.49:0/2128974327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 msgr2=0x563a1b8e2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.786+0000 7fae44fd9740 1 --2- 172.21.15.49:0/2128974327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 0x563a1b8e2750 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fae30004930 tx=0x7fae3000d050 comp rx=0 tx=0).stop 2024-03-18T00:57:00.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.786+0000 7fae44fd9740 1 -- 172.21.15.49:0/2128974327 shutdown_connections 2024-03-18T00:57:00.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.786+0000 7fae44fd9740 1 --2- 172.21.15.49:0/2128974327 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 0x563a1b8e2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.786+0000 7fae44fd9740 1 --2- 172.21.15.49:0/2128974327 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b902a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.786+0000 7fae44fd9740 1 --2- 172.21.15.49:0/2128974327 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 0x563a1b8e2750 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.786+0000 7fae44fd9740 1 -- 172.21.15.49:0/2128974327 >> 172.21.15.49:0/2128974327 conn(0x563a1b8e3dc0 msgr2=0x563a1b8e41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.786+0000 7fae44fd9740 1 -- 172.21.15.49:0/2128974327 shutdown_connections 2024-03-18T00:57:00.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.786+0000 7fae44fd9740 1 -- 172.21.15.49:0/2128974327 wait complete. 2024-03-18T00:57:00.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.787+0000 7fae44fd9740 1 -- start start 2024-03-18T00:57:00.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.787+0000 7fae44fd9740 1 Processor -- start 2024-03-18T00:57:00.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 0x563a1b900130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b900670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae44fd9740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 0x563a1b901630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae44fd9740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c1a8140 con 0x563a1b8e9240 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae44fd9740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c1a7a70 con 0x563a1b8ee3b0 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae44fd9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fae3c1a4cb0 con 0x563a1b8e8850 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae44fd9740 10 client.0 _ll_register_callbacks cb 0x7fae300060d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b900670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae3bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b900670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54678/0 (socket says 172.21.15.49:54678) 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae3bfff640 1 -- 172.21.15.49:0/18364805 learned_addr learned my addr 172.21.15.49:0/18364805 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.788+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 0x563a1b900130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 0x563a1b901630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae3bfff640 1 -- 172.21.15.49:0/18364805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 msgr2=0x563a1b900130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae3bfff640 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 0x563a1b900130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae3bfff640 1 -- 172.21.15.49:0/18364805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 msgr2=0x563a1b901630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae3bfff640 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 0x563a1b901630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae3bfff640 1 -- 172.21.15.49:0/18364805 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a1b901c20 con 0x563a1b8e9240 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae3bfff640 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b900670 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x563a1b8e9cb0 tx=0x7fae2c02c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edb4faa052b6338 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae1b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae1b7fe640 1 -- 172.21.15.49:0/18364805 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c042040 con 0x563a1b8e9240 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae1b7fe640 1 -- 172.21.15.49:0/18364805 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fae2c02ca50 con 0x563a1b8e9240 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.789+0000 7fae44fd9740 1 -- 172.21.15.49:0/18364805 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae300137e0 con 0x563a1b8e9240 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.790+0000 7fae1b7fe640 1 -- 172.21.15.49:0/18364805 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fae2c05c070 con 0x563a1b8e9240 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.790+0000 7fae44fd9740 10 client.17085 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:00.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.790+0000 7fae44fd9740 1 -- 172.21.15.49:0/18364805 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563a1b96d800 con 0x563a1b8e9240 2024-03-18T00:57:00.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.791+0000 7fae1b7fe640 1 -- 172.21.15.49:0/18364805 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae300137e0 con 0x563a1b8e9240 2024-03-18T00:57:00.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.791+0000 7fae1b7fe640 1 -- 172.21.15.49:0/18364805 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fae2c0344e0 con 0x563a1b8e9240 2024-03-18T00:57:00.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.791+0000 7fae44fd9740 10 client.17085 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:00.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.791+0000 7fae44fd9740 10 client.17085 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:57:00.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.791+0000 7fae44fd9740 20 client.17085 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:00.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.791+0000 7fae44fd9740 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x563a1b96dc70 0x563a1b970060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:00.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.791+0000 7fae44fd9740 4 client.17085 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:57:00.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.791+0000 7fae44fd9740 1 -- 172.21.15.49:0/18364805 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a1b854d20 con 0x563a1b96dc70 2024-03-18T00:57:00.793 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.792+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x563a1b96dc70 0x563a1b970060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:00.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.795+0000 7fae3b7fe640 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x563a1b96dc70 0x563a1b970060 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c0dd2a226c80ea25 server_cookie=77fa50c4c5b1eb42 in_seq=0 out_seq=0 2024-03-18T00:57:00.795 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.795+0000 7fae1b7fe640 10 client.17085 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:57:00.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.796+0000 7fae1b7fe640 1 -- 172.21.15.49:0/18364805 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a1b854d20 con 0x563a1b96dc70 2024-03-18T00:57:00.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.796+0000 7fae1b7fe640 10 client.17085 handle_command_reply: tid=0 2024-03-18T00:57:00.796 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.796+0000 7fae44fd9740 1 client.17085 shutdown 2024-03-18T00:57:00.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.798+0000 7fae44fd9740 1 -- 172.21.15.49:0/18364805 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 msgr2=0x563a1b900670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.801+0000 7fae44fd9740 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b900670 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x563a1b8e9cb0 tx=0x7fae2c02c500 comp rx=0 tx=0).stop 2024-03-18T00:57:00.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.811+0000 7fae44fd9740 1 -- 172.21.15.49:0/18364805 shutdown_connections 2024-03-18T00:57:00.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.811+0000 7fae44fd9740 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x563a1b96dc70 0x563a1b970060 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.811+0000 7fae44fd9740 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563a1b8ee3b0 0x563a1b901630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.811+0000 7fae44fd9740 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563a1b8e9240 0x563a1b900670 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.811+0000 7fae44fd9740 1 --2- 172.21.15.49:0/18364805 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563a1b8e8850 0x563a1b900130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.811+0000 7fae44fd9740 1 -- 172.21.15.49:0/18364805 >> 172.21.15.49:0/18364805 conn(0x563a1b8e3dc0 msgr2=0x563a1b8e7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.815+0000 7fae1b7fe640 0 client.17085 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:57:00.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.834+0000 7fae44fd9740 1 -- 172.21.15.49:0/18364805 shutdown_connections 2024-03-18T00:57:00.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.836+0000 7fae44fd9740 1 -- 172.21.15.49:0/18364805 wait complete. 2024-03-18T00:57:00.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.837+0000 7fae44fd9740 20 client.17085 trim_cache size 0 max 16384 2024-03-18T00:57:00.841 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:57:00.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.855+0000 7fae42310640 1 -- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fae240a2880 msgr2=0x7fae240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.855+0000 7fae42310640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fae240a2880 0x7fae240a4d40 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fae340a2c60 tx=0x7fae34014c20 comp rx=0 tx=0).stop 2024-03-18T00:57:00.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.855+0000 7fae42310640 1 -- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c1075a0 msgr2=0x7fae3c19f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:00.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.855+0000 7fae42310640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c1075a0 0x7fae3c19f250 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fae3400b990 tx=0x7fae3400be50 comp rx=0 tx=0).stop 2024-03-18T00:57:00.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.856+0000 7fae42310640 1 -- 172.21.15.49:0/554997203 shutdown_connections 2024-03-18T00:57:00.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.856+0000 7fae42310640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fae240a2880 0x7fae240a4d40 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.856+0000 7fae42310640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fae3c1075a0 0x7fae3c19f250 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.856+0000 7fae42310640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fae3c104c70 0x7fae3c1981f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.856+0000 7fae42310640 1 --2- 172.21.15.49:0/554997203 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fae3c102340 0x7fae3c197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:00.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.857+0000 7fae42310640 1 -- 172.21.15.49:0/554997203 >> 172.21.15.49:0/554997203 conn(0x7fae3c0fbc90 msgr2=0x7fae3c104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:00.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.859+0000 7fae42310640 1 -- 172.21.15.49:0/554997203 shutdown_connections 2024-03-18T00:57:00.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:00.862+0000 7fae42310640 1 -- 172.21.15.49:0/554997203 wait complete. 2024-03-18T00:57:00.894 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:57:01.287 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.285+0000 7ff507fff640 1 -- 172.21.15.49:0/4068539300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff508072bf0 msgr2=0x7ff508073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:01.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.285+0000 7ff507fff640 1 --2- 172.21.15.49:0/4068539300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff508072bf0 0x7ff508073010 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff500009ca0 tx=0x7ff50002c140 comp rx=0 tx=0).stop 2024-03-18T00:57:01.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.286+0000 7ff507fff640 1 -- 172.21.15.49:0/4068539300 shutdown_connections 2024-03-18T00:57:01.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.286+0000 7ff507fff640 1 --2- 172.21.15.49:0/4068539300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff508108eb0 0x7ff508109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.286+0000 7ff507fff640 1 --2- 172.21.15.49:0/4068539300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff508109870 0x7ff50810e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.286+0000 7ff507fff640 1 --2- 172.21.15.49:0/4068539300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff508072bf0 0x7ff508073010 secure :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff500009ca0 tx=0x7ff50002c140 comp rx=0 tx=0).stop 2024-03-18T00:57:01.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.286+0000 7ff507fff640 1 -- 172.21.15.49:0/4068539300 >> 172.21.15.49:0/4068539300 conn(0x7ff50806e6c0 msgr2=0x7ff508070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:01.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.288+0000 7ff507fff640 1 -- 172.21.15.49:0/4068539300 shutdown_connections 2024-03-18T00:57:01.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.290+0000 7ff507fff640 1 -- 172.21.15.49:0/4068539300 wait complete. 2024-03-18T00:57:01.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.293+0000 7ff507fff640 1 Processor -- start 2024-03-18T00:57:01.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.295+0000 7ff507fff640 1 -- start start 2024-03-18T00:57:01.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.297+0000 7ff507fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff508108eb0 0x7ff508117230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.297+0000 7ff507fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff508109870 0x7ff508117770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.297+0000 7ff507fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5081187b0 0x7ff508118c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.297+0000 7ff507fff640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff50811c2a0 con 0x7ff508108eb0 2024-03-18T00:57:01.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.297+0000 7ff507fff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff50811c120 con 0x7ff5081187b0 2024-03-18T00:57:01.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.297+0000 7ff507fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff50811c420 con 0x7ff508109870 2024-03-18T00:57:01.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.299+0000 7ff5067fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff508109870 0x7ff508117770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.299+0000 7ff5067fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff508109870 0x7ff508117770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:37950/0 (socket says 172.21.15.49:37950) 2024-03-18T00:57:01.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.299+0000 7ff5067fc640 1 -- 172.21.15.49:0/2948149269 learned_addr learned my addr 172.21.15.49:0/2948149269 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:01.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.299+0000 7ff5077fe640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5081187b0 0x7ff508118c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.302+0000 7ff5067fc640 1 -- 172.21.15.49:0/2948149269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5081187b0 msgr2=0x7ff508118c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:01.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.302+0000 7ff5067fc640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5081187b0 0x7ff508118c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.302+0000 7ff5067fc640 1 -- 172.21.15.49:0/2948149269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff508108eb0 msgr2=0x7ff508117230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:01.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.302+0000 7ff5067fc640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff508108eb0 0x7ff508117230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.302+0000 7ff5067fc640 1 -- 172.21.15.49:0/2948149269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff508119220 con 0x7ff508109870 2024-03-18T00:57:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.303+0000 7ff5067fc640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff508109870 0x7ff508117770 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00b450 tx=0x7ff4fc00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.304+0000 7ff4effff640 1 -- 172.21.15.49:0/2948149269 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc015020 con 0x7ff508109870 2024-03-18T00:57:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.304+0000 7ff4effff640 1 -- 172.21.15.49:0/2948149269 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4fc011d70 con 0x7ff508109870 2024-03-18T00:57:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.304+0000 7ff4effff640 1 -- 172.21.15.49:0/2948149269 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc00f340 con 0x7ff508109870 2024-03-18T00:57:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.304+0000 7ff507fff640 1 -- 172.21.15.49:0/2948149269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff508119510 con 0x7ff508109870 2024-03-18T00:57:01.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.304+0000 7ff507fff640 1 -- 172.21.15.49:0/2948149269 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff508119a10 con 0x7ff508109870 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.313+0000 7ff4effff640 1 -- 172.21.15.49:0/2948149269 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff4fc018020 con 0x7ff508109870 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.313+0000 7ff50ffb1740 1 Processor -- start 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.315+0000 7ff4effff640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4f00a2980 0x7ff4f00a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.315+0000 7ff4effff640 1 -- 172.21.15.49:0/2948149269 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff4fc0ce3b0 con 0x7ff508109870 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.317+0000 7ff50ffb1740 1 -- start start 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.317+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b9a40 0x561ae85b9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.320+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85bef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.320+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85bf5a0 0x561ae85c1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.320+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b9a40 0x561ae85b9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.320+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b9a40 0x561ae85b9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54694/0 (socket says 172.21.15.49:54694) 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.320+0000 7ff506ffd640 1 -- 172.21.15.49:0/492009421 learned_addr learned my addr 172.21.15.49:0/492009421 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.320+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x561ae851e490 con 0x561ae85b9a40 2024-03-18T00:57:01.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.320+0000 7ff50ffb1740 1 -- 172.21.15.49:0/492009421 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x561ae84c6c90 con 0x561ae85bf5a0 2024-03-18T00:57:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.320+0000 7ff50ffb1740 1 -- 172.21.15.49:0/492009421 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561ae851a530 con 0x561ae85ba430 2024-03-18T00:57:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.323+0000 7ff5067fc640 1 --2- 172.21.15.49:0/492009421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85bef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.323+0000 7ff506ffd640 1 -- 172.21.15.49:0/492009421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 msgr2=0x561ae85bef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:01.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.323+0000 7ff506ffd640 1 --2- 172.21.15.49:0/492009421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85bef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.323+0000 7ff506ffd640 1 -- 172.21.15.49:0/492009421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85bf5a0 msgr2=0x561ae85c1a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:57:01.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.323+0000 7ff506ffd640 1 --2- 172.21.15.49:0/492009421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85bf5a0 0x561ae85c1a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.323+0000 7ff506ffd640 1 -- 172.21.15.49:0/492009421 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ae8581500 con 0x561ae85b9a40 2024-03-18T00:57:01.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.326+0000 7ff5077fe640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4f00a2980 0x7ff4f00a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.329+0000 7ff506ffd640 1 --2- 172.21.15.49:0/492009421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b9a40 0x561ae85b9e60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff50002c6d0 tx=0x7ff500005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff5f483d93c12d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:01.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.331+0000 7ff5077fe640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4f00a2980 0x7ff4f00a4e40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8006fd0 tx=0x7ff4f8009060 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:01.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.331+0000 7ff4ed7fa640 1 -- 172.21.15.49:0/492009421 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500039040 con 0x561ae85b9a40 2024-03-18T00:57:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.331+0000 7ff4ed7fa640 1 -- 172.21.15.49:0/492009421 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff500042070 con 0x561ae85b9a40 2024-03-18T00:57:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.331+0000 7ff4ed7fa640 1 -- 172.21.15.49:0/492009421 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5000042a0 con 0x561ae85b9a40 2024-03-18T00:57:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.333+0000 7ff50ffb1740 1 -- 172.21.15.49:0/492009421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b9a40 msgr2=0x561ae85b9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.333+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/492009421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b9a40 0x561ae85b9e60 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff50002c6d0 tx=0x7ff500005fa0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.333+0000 7ff50ffb1740 1 -- 172.21.15.49:0/492009421 shutdown_connections 2024-03-18T00:57:01.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.333+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/492009421 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85bf5a0 0x561ae85c1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.333+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/492009421 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85bef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.333+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/492009421 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b9a40 0x561ae85b9e60 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.333+0000 7ff50ffb1740 1 -- 172.21.15.49:0/492009421 >> 172.21.15.49:0/492009421 conn(0x561ae85b4fb0 msgr2=0x561ae85b53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:01.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.334+0000 7ff50ffb1740 1 -- 172.21.15.49:0/492009421 shutdown_connections 2024-03-18T00:57:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.334+0000 7ff50ffb1740 1 -- 172.21.15.49:0/492009421 wait complete. 2024-03-18T00:57:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.334+0000 7ff50ffb1740 1 -- start start 2024-03-18T00:57:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.335+0000 7ff50ffb1740 1 Processor -- start 2024-03-18T00:57:01.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.336+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.337+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85ba430 0x561ae85e62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.338+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.338+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85bf5a0 0x561ae85ed2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.338+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x561ae85e4500 con 0x561ae85ba430 2024-03-18T00:57:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.338+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x561ae84da230 con 0x561ae85b9a40 2024-03-18T00:57:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.338+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561ae85e4680 con 0x561ae85bf5a0 2024-03-18T00:57:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.339+0000 7ff50ffb1740 10 client.0 _ll_register_callbacks cb 0x561ae8516010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.339+0000 7ff5067fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85ba430 0x561ae85e62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.339+0000 7ff5067fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85ba430 0x561ae85e62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54700/0 (socket says 172.21.15.49:54700) 2024-03-18T00:57:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.339+0000 7ff5067fc640 1 -- 172.21.15.49:0/3267704601 learned_addr learned my addr 172.21.15.49:0/3267704601 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:01.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.339+0000 7ff5077fe640 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85bf5a0 0x561ae85ed2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.341+0000 7ff5067fc640 1 -- 172.21.15.49:0/3267704601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85bf5a0 msgr2=0x561ae85ed2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:01.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.341+0000 7ff5067fc640 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85bf5a0 0x561ae85ed2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.341+0000 7ff5067fc640 1 -- 172.21.15.49:0/3267704601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 msgr2=0x561ae85e5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:01.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.341+0000 7ff5067fc640 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85e5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.341+0000 7ff5067fc640 1 -- 172.21.15.49:0/3267704601 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ae84e9100 con 0x561ae85ba430 2024-03-18T00:57:01.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.345+0000 7ff5067fc640 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85ba430 0x561ae85e62a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc097700 tx=0x7ff4fc09ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be69c5672a99d6ed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:01.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.345+0000 7ff4eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:01.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.346+0000 7ff4eaff5640 1 -- 172.21.15.49:0/3267704601 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0b8060 con 0x561ae85ba430 2024-03-18T00:57:01.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.346+0000 7ff4eaff5640 1 -- 172.21.15.49:0/3267704601 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff4fc0103e0 con 0x561ae85ba430 2024-03-18T00:57:01.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.346+0000 7ff50ffb1740 1 -- 172.21.15.49:0/3267704601 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff500042070 con 0x561ae85ba430 2024-03-18T00:57:01.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.347+0000 7ff4eaff5640 1 -- 172.21.15.49:0/3267704601 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff4fc0105e0 con 0x561ae85ba430 2024-03-18T00:57:01.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.347+0000 7ff50ffb1740 10 client.17094 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:01.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.347+0000 7ff50ffb1740 1 -- 172.21.15.49:0/3267704601 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561ae82323f0 con 0x561ae85ba430 2024-03-18T00:57:01.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.348+0000 7ff4eaff5640 1 -- 172.21.15.49:0/3267704601 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff500042070 con 0x561ae85ba430 2024-03-18T00:57:01.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.350+0000 7ff4eaff5640 1 -- 172.21.15.49:0/3267704601 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff4fc099bf0 con 0x561ae85ba430 2024-03-18T00:57:01.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.350+0000 7ff50ffb1740 10 client.17094 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:01.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.350+0000 7ff50ffb1740 10 client.17094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:01.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.350+0000 7ff50ffb1740 20 client.17094 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:01.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.350+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x561ae85f2070 0x561ae85f44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.350+0000 7ff50ffb1740 4 client.17094 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:01.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.350+0000 7ff50ffb1740 1 -- 172.21.15.49:0/3267704601 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ae852b9d0 con 0x561ae85f2070 2024-03-18T00:57:01.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.352+0000 7ff506ffd640 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x561ae85f2070 0x561ae85f44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.356+0000 7ff4eaff5640 10 client.17094 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:01.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.356+0000 7ff506ffd640 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x561ae85f2070 0x561ae85f44b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7a8a4e9c326e7c1 server_cookie=cc31d1c504c1d3e3 in_seq=0 out_seq=0 2024-03-18T00:57:01.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.360+0000 7ff4eaff5640 1 -- 172.21.15.49:0/3267704601 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561ae852b9d0 con 0x561ae85f2070 2024-03-18T00:57:01.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.360+0000 7ff4eaff5640 10 client.17094 handle_command_reply: tid=0 2024-03-18T00:57:01.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.360+0000 7ff50ffb1740 1 client.17094 shutdown 2024-03-18T00:57:01.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.362+0000 7ff50ffb1740 1 -- 172.21.15.49:0/3267704601 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85ba430 msgr2=0x561ae85e62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:01.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.362+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85ba430 0x561ae85e62a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc097700 tx=0x7ff4fc09ee80 comp rx=0 tx=0).stop 2024-03-18T00:57:01.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.364+0000 7ff50ffb1740 1 -- 172.21.15.49:0/3267704601 shutdown_connections 2024-03-18T00:57:01.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.364+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x561ae85f2070 0x561ae85f44b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.364+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85bf5a0 0x561ae85ed2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.364+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85ba430 0x561ae85e62a0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.364+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/3267704601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85e5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.364+0000 7ff50ffb1740 1 -- 172.21.15.49:0/3267704601 >> 172.21.15.49:0/3267704601 conn(0x561ae85b4fb0 msgr2=0x561ae85b4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:01.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.364+0000 7ff4eaff5640 0 client.17094 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:01.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.364+0000 7ff50ffb1740 1 -- 172.21.15.49:0/3267704601 shutdown_connections 2024-03-18T00:57:01.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.365+0000 7ff50ffb1740 1 -- 172.21.15.49:0/3267704601 wait complete. 2024-03-18T00:57:01.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.365+0000 7ff50ffb1740 20 client.17094 trim_cache size 0 max 16384 2024-03-18T00:57:01.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.389+0000 7ff50ffb1740 1 Processor -- start 2024-03-18T00:57:01.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.389+0000 7ff50ffb1740 1 -- start start 2024-03-18T00:57:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.390+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85f9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.390+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.390+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85b8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.390+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f400e4c0 con 0x561ae85b4fb0 2024-03-18T00:57:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.390+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f4010000 con 0x561ae85b9a40 2024-03-18T00:57:01.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.390+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f400eb90 con 0x561ae85ba430 2024-03-18T00:57:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85f9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85f9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54702/0 (socket says 172.21.15.49:54702) 2024-03-18T00:57:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff506ffd640 1 -- 172.21.15.49:0/2788127390 learned_addr learned my addr 172.21.15.49:0/2788127390 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff5067fc640 1 --2- 172.21.15.49:0/2788127390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff506ffd640 1 -- 172.21.15.49:0/2788127390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 msgr2=0x561ae85b8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff506ffd640 1 --2- 172.21.15.49:0/2788127390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85b8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff506ffd640 1 -- 172.21.15.49:0/2788127390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 msgr2=0x561ae85ed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff506ffd640 1 --2- 172.21.15.49:0/2788127390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85ed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff506ffd640 1 -- 172.21.15.49:0/2788127390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ae82fc7d0 con 0x561ae85b4fb0 2024-03-18T00:57:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.391+0000 7ff5067fc640 1 --2- 172.21.15.49:0/2788127390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:57:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.392+0000 7ff506ffd640 1 --2- 172.21.15.49:0/2788127390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85f9dc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff50003f040 tx=0x7ff50002e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82ee0baba8f712c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.392+0000 7ff4ecff9640 1 -- 172.21.15.49:0/2788127390 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500039040 con 0x561ae85b4fb0 2024-03-18T00:57:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.392+0000 7ff4ecff9640 1 -- 172.21.15.49:0/2788127390 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff500042040 con 0x561ae85b4fb0 2024-03-18T00:57:01.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.392+0000 7ff4ecff9640 1 -- 172.21.15.49:0/2788127390 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500003bd0 con 0x561ae85b4fb0 2024-03-18T00:57:01.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.393+0000 7ff50ffb1740 1 -- 172.21.15.49:0/2788127390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 msgr2=0x561ae85f9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:01.393 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.393+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/2788127390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85f9dc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7ff50003f040 tx=0x7ff50002e040 comp rx=0 tx=0).stop 2024-03-18T00:57:01.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.394+0000 7ff50ffb1740 1 -- 172.21.15.49:0/2788127390 shutdown_connections 2024-03-18T00:57:01.394 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.394+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/2788127390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.394+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/2788127390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85b8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.394+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/2788127390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85f9dc0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.394+0000 7ff50ffb1740 1 -- 172.21.15.49:0/2788127390 >> 172.21.15.49:0/2788127390 conn(0x561ae85bb030 msgr2=0x561ae85bb750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.394+0000 7ff50ffb1740 1 -- 172.21.15.49:0/2788127390 shutdown_connections 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.394+0000 7ff50ffb1740 1 -- 172.21.15.49:0/2788127390 wait complete. 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.395+0000 7ff50ffb1740 1 -- start start 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.395+0000 7ff50ffb1740 1 Processor -- start 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.395+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85ea4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.395+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85ea9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.396+0000 7ff50ffb1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85eb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.396+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f400f930 con 0x561ae85b4fb0 2024-03-18T00:57:01.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.396+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f400f260 con 0x561ae85b9a40 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.396+0000 7ff50ffb1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f400c550 con 0x561ae85ba430 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.396+0000 7ff50ffb1740 10 client.0 _ll_register_callbacks cb 0x7ff4fc096fd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.396+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85ea4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.396+0000 7ff506ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85ea4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54708/0 (socket says 172.21.15.49:54708) 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.396+0000 7ff506ffd640 1 -- 172.21.15.49:0/1138356431 learned_addr learned my addr 172.21.15.49:0/1138356431 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.396+0000 7ff5067fc640 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85ea9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.397+0000 7ff5077fe640 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85eb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.397+0000 7ff506ffd640 1 -- 172.21.15.49:0/1138356431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 msgr2=0x561ae85eb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.397+0000 7ff506ffd640 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85eb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.397+0000 7ff506ffd640 1 -- 172.21.15.49:0/1138356431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 msgr2=0x561ae85ea9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.397+0000 7ff506ffd640 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85ea9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.397+0000 7ff506ffd640 1 -- 172.21.15.49:0/1138356431 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ae85ebfe0 con 0x561ae85b4fb0 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.397+0000 7ff5077fe640 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85eb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:57:01.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.397+0000 7ff506ffd640 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85ea4a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff500009bf0 tx=0x7ff500032e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d0595d19f8249f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:01.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.397+0000 7ff4eaff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:01.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.398+0000 7ff4eaff5640 1 -- 172.21.15.49:0/1138356431 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff500055070 con 0x561ae85b4fb0 2024-03-18T00:57:01.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.398+0000 7ff4eaff5640 1 -- 172.21.15.49:0/1138356431 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff500003970 con 0x561ae85b4fb0 2024-03-18T00:57:01.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.398+0000 7ff50ffb1740 1 -- 172.21.15.49:0/1138356431 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff500042040 con 0x561ae85b4fb0 2024-03-18T00:57:01.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.398+0000 7ff4eaff5640 1 -- 172.21.15.49:0/1138356431 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff500039050 con 0x561ae85b4fb0 2024-03-18T00:57:01.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.398+0000 7ff50ffb1740 10 client.17100 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:01.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.398+0000 7ff50ffb1740 1 -- 172.21.15.49:0/1138356431 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561ae85ed2e0 con 0x561ae85b4fb0 2024-03-18T00:57:01.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.399+0000 7ff4eaff5640 1 -- 172.21.15.49:0/1138356431 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff500003bf0 con 0x561ae85b4fb0 2024-03-18T00:57:01.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.399+0000 7ff4eaff5640 1 -- 172.21.15.49:0/1138356431 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff50003c070 con 0x561ae85b4fb0 2024-03-18T00:57:01.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.399+0000 7ff50ffb1740 10 client.17100 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:01.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.399+0000 7ff50ffb1740 10 client.17100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:01.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.399+0000 7ff50ffb1740 20 client.17100 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:01.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.399+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x561ae85bf5a0 0x561ae865a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:01.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.400+0000 7ff50ffb1740 4 client.17100 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:57:01.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.400+0000 7ff50ffb1740 1 -- 172.21.15.49:0/1138356431 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff500042040 con 0x561ae85bf5a0 2024-03-18T00:57:01.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.400+0000 7ff5067fc640 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x561ae85bf5a0 0x561ae865a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:01.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.408+0000 7ff4eaff5640 10 client.17100 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:01.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.408+0000 7ff5067fc640 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x561ae85bf5a0 0x561ae865a400 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7805a28713cdd361 server_cookie=a0deec84937ade01 in_seq=0 out_seq=0 2024-03-18T00:57:01.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.413+0000 7ff4eaff5640 1 -- 172.21.15.49:0/1138356431 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff500042040 con 0x561ae85bf5a0 2024-03-18T00:57:01.413 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.413+0000 7ff4eaff5640 10 client.17100 handle_command_reply: tid=0 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.413+0000 7ff50ffb1740 1 client.17100 shutdown 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 -- 172.21.15.49:0/1138356431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 msgr2=0x561ae85ea4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85ea4a0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ff500009bf0 tx=0x7ff500032e70 comp rx=0 tx=0).stop 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 -- 172.21.15.49:0/1138356431 shutdown_connections 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x561ae85bf5a0 0x561ae865a400 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561ae85b9a40 0x561ae85ea9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561ae85ba430 0x561ae85eb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 --2- 172.21.15.49:0/1138356431 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561ae85b4fb0 0x561ae85ea4a0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 -- 172.21.15.49:0/1138356431 >> 172.21.15.49:0/1138356431 conn(0x561ae85bb030 msgr2=0x561ae85b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:01.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff4eaff5640 0 client.17100 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:01.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 -- 172.21.15.49:0/1138356431 shutdown_connections 2024-03-18T00:57:01.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.414+0000 7ff50ffb1740 1 -- 172.21.15.49:0/1138356431 wait complete. 2024-03-18T00:57:01.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.415+0000 7ff50ffb1740 20 client.17100 trim_cache size 0 max 16384 2024-03-18T00:57:01.415 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:57:01.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 -- 172.21.15.49:0/2948149269 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4f00a2980 msgr2=0x7ff4f00a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:01.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4f00a2980 0x7ff4f00a4e40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff4f8006fd0 tx=0x7ff4f8009060 comp rx=0 tx=0).stop 2024-03-18T00:57:01.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 -- 172.21.15.49:0/2948149269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff508109870 msgr2=0x7ff508117770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:01.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff508109870 0x7ff508117770 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc00b450 tx=0x7ff4fc00b910 comp rx=0 tx=0).stop 2024-03-18T00:57:01.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 -- 172.21.15.49:0/2948149269 shutdown_connections 2024-03-18T00:57:01.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff4f00a2980 0x7ff4f00a4e40 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff5081187b0 0x7ff508118c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff508109870 0x7ff508117770 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 --2- 172.21.15.49:0/2948149269 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff508108eb0 0x7ff508117230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:01.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.416+0000 7ff507fff640 1 -- 172.21.15.49:0/2948149269 >> 172.21.15.49:0/2948149269 conn(0x7ff50806e6c0 msgr2=0x7ff50810c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:01.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.417+0000 7ff507fff640 1 -- 172.21.15.49:0/2948149269 shutdown_connections 2024-03-18T00:57:01.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:01.417+0000 7ff507fff640 1 -- 172.21.15.49:0/2948149269 wait complete. 2024-03-18T00:57:01.629 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:01 smithi049 ceph-mon[32130]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 872 MiB/s wr, 2.48k op/s 2024-03-18T00:57:01.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:01 smithi107 ceph-mon[36270]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 872 MiB/s wr, 2.48k op/s 2024-03-18T00:57:01.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:01 smithi106 ceph-mon[35899]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 872 MiB/s wr, 2.48k op/s 2024-03-18T00:57:02.442 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:57:02.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:57:02.442 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:57:02.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.962+0000 7f47114f0640 1 -- 172.21.15.49:0/1143630574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f470c109870 msgr2=0x7f470c10e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:02.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.962+0000 7f47114f0640 1 --2- 172.21.15.49:0/1143630574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f470c109870 0x7f470c10e110 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f46fc009900 tx=0x7f46fc02bc50 comp rx=0 tx=0).stop 2024-03-18T00:57:02.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.963+0000 7f47114f0640 1 -- 172.21.15.49:0/1143630574 shutdown_connections 2024-03-18T00:57:02.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.963+0000 7f47114f0640 1 --2- 172.21.15.49:0/1143630574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f470c108eb0 0x7f470c109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:02.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.963+0000 7f47114f0640 1 --2- 172.21.15.49:0/1143630574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f470c109870 0x7f470c10e110 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:02.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.963+0000 7f47114f0640 1 --2- 172.21.15.49:0/1143630574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f470c072bf0 0x7f470c073010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:02.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.963+0000 7f47114f0640 1 -- 172.21.15.49:0/1143630574 >> 172.21.15.49:0/1143630574 conn(0x7f470c06e6c0 msgr2=0x7f470c070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:02.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.963+0000 7f47114f0640 1 -- 172.21.15.49:0/1143630574 shutdown_connections 2024-03-18T00:57:02.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.963+0000 7f47114f0640 1 -- 172.21.15.49:0/1143630574 wait complete. 2024-03-18T00:57:02.963 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.964+0000 7f47114f0640 1 Processor -- start 2024-03-18T00:57:02.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.964+0000 7f47114f0640 1 -- start start 2024-03-18T00:57:02.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.964+0000 7f47114f0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f470c072bf0 0x7f470c1a6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:02.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.964+0000 7f47114f0640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f470c108eb0 0x7f470c1a6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:02.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.964+0000 7f47114f0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f470c109870 0x7f470c1add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:02.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.965+0000 7f47114f0640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f470c11c170 con 0x7f470c109870 2024-03-18T00:57:02.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.965+0000 7f47114f0640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f470c11bff0 con 0x7f470c108eb0 2024-03-18T00:57:02.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.965+0000 7f47114f0640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f470c11c2f0 con 0x7f470c072bf0 2024-03-18T00:57:02.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.965+0000 7f4710cef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f470c109870 0x7f470c1add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:02.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.965+0000 7f470b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f470c108eb0 0x7f470c1a6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:02.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.965+0000 7f470bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f470c072bf0 0x7f470c1a6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:02.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.965+0000 7f470b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f470c108eb0 0x7f470c1a6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:44910/0 (socket says 172.21.15.49:44910) 2024-03-18T00:57:02.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.965+0000 7f470b7fe640 1 -- 172.21.15.49:0/716121235 learned_addr learned my addr 172.21.15.49:0/716121235 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:02.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.965+0000 7f470bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f470c072bf0 0x7f470c1a6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38010/0 (socket says 172.21.15.49:38010) 2024-03-18T00:57:02.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.966+0000 7f470bfff640 1 -- 172.21.15.49:0/716121235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f470c108eb0 msgr2=0x7f470c1a6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:02.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.966+0000 7f470bfff640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f470c108eb0 0x7f470c1a6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:02.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.966+0000 7f470bfff640 1 -- 172.21.15.49:0/716121235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f470c109870 msgr2=0x7f470c1add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:02.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.966+0000 7f470bfff640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f470c109870 0x7f470c1add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:02.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.966+0000 7f470bfff640 1 -- 172.21.15.49:0/716121235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c1ae370 con 0x7f470c072bf0 2024-03-18T00:57:02.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.967+0000 7f470bfff640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f470c072bf0 0x7f470c1a6750 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f470c073940 tx=0x7f470000dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:02.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.981+0000 7f47097fa640 1 -- 172.21.15.49:0/716121235 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700017070 con 0x7f470c072bf0 2024-03-18T00:57:02.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.981+0000 7f47097fa640 1 -- 172.21.15.49:0/716121235 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4700010460 con 0x7f470c072bf0 2024-03-18T00:57:02.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.982+0000 7f47097fa640 1 -- 172.21.15.49:0/716121235 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470000f620 con 0x7f470c072bf0 2024-03-18T00:57:02.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.983+0000 7f47114f0640 1 -- 172.21.15.49:0/716121235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c1ae660 con 0x7f470c072bf0 2024-03-18T00:57:02.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.983+0000 7f47114f0640 1 -- 172.21.15.49:0/716121235 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c1aeb30 con 0x7f470c072bf0 2024-03-18T00:57:02.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.996+0000 7f47097fa640 1 -- 172.21.15.49:0/716121235 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f470001a070 con 0x7f470c072bf0 2024-03-18T00:57:02.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.998+0000 7f47097fa640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46f40a2870 0x7f46f40a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:02.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:02.999+0000 7f47097fa640 1 -- 172.21.15.49:0/716121235 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f47000cdf90 con 0x7f470c072bf0 2024-03-18T00:57:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.000+0000 7f471540e740 1 Processor -- start 2024-03-18T00:57:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.000+0000 7f471540e740 1 -- start start 2024-03-18T00:57:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.001+0000 7f471540e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9535b80 0x563eb9535fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.001+0000 7f471540e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 0x563eb953b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.001+0000 7f471540e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb953b6e0 0x563eb953db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.001+0000 7f471540e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563eb949a0d0 con 0x563eb953b6e0 2024-03-18T00:57:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.001+0000 7f471540e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563eb9442c70 con 0x563eb9536570 2024-03-18T00:57:03.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.001+0000 7f471540e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563eb9496170 con 0x563eb9535b80 2024-03-18T00:57:03.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.004+0000 7f470b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 0x563eb953b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.006 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.006+0000 7f4710cef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb953b6e0 0x563eb953db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.007+0000 7f4710cef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb953b6e0 0x563eb953db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54726/0 (socket says 172.21.15.49:54726) 2024-03-18T00:57:03.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.007+0000 7f4710cef640 1 -- 172.21.15.49:0/1047225442 learned_addr learned my addr 172.21.15.49:0/1047225442 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:03.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.008+0000 7f470b7fe640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46f40a2870 0x7f46f40a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.008+0000 7f470bfff640 1 --2- 172.21.15.49:0/1047225442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9535b80 0x563eb9535fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.009+0000 7f4710cef640 1 -- 172.21.15.49:0/1047225442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9535b80 msgr2=0x563eb9535fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.009+0000 7f4710cef640 1 --2- 172.21.15.49:0/1047225442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9535b80 0x563eb9535fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.009+0000 7f4710cef640 1 -- 172.21.15.49:0/1047225442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 msgr2=0x563eb953b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.009+0000 7f4710cef640 1 --2- 172.21.15.49:0/1047225442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 0x563eb953b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.009+0000 7f4710cef640 1 -- 172.21.15.49:0/1047225442 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eb92767e0 con 0x563eb953b6e0 2024-03-18T00:57:03.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.010+0000 7f4710cef640 1 --2- 172.21.15.49:0/1047225442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb953b6e0 0x563eb953db70 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f46fc02c260 tx=0x7f46fc02c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa53d03abfcc34f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.011+0000 7f46f27fc640 1 -- 172.21.15.49:0/1047225442 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc045020 con 0x563eb953b6e0 2024-03-18T00:57:03.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.011+0000 7f46f27fc640 1 -- 172.21.15.49:0/1047225442 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46fc004040 con 0x563eb953b6e0 2024-03-18T00:57:03.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.012+0000 7f471540e740 1 -- 172.21.15.49:0/1047225442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb953b6e0 msgr2=0x563eb953db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.012+0000 7f471540e740 1 --2- 172.21.15.49:0/1047225442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb953b6e0 0x563eb953db70 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f46fc02c260 tx=0x7f46fc02c6a0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.012+0000 7f471540e740 1 -- 172.21.15.49:0/1047225442 shutdown_connections 2024-03-18T00:57:03.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.012+0000 7f471540e740 1 --2- 172.21.15.49:0/1047225442 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 0x563eb953b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.012+0000 7f471540e740 1 --2- 172.21.15.49:0/1047225442 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb953b6e0 0x563eb953db70 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.012+0000 7f471540e740 1 --2- 172.21.15.49:0/1047225442 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9535b80 0x563eb9535fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.012+0000 7f471540e740 1 -- 172.21.15.49:0/1047225442 >> 172.21.15.49:0/1047225442 conn(0x563eb94fdc50 msgr2=0x563eb95314a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.013+0000 7f471540e740 1 -- 172.21.15.49:0/1047225442 shutdown_connections 2024-03-18T00:57:03.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.014+0000 7f470b7fe640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46f40a2870 0x7f46f40a4d30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x563eb9536970 tx=0x7f4704009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.017+0000 7f471540e740 1 -- 172.21.15.49:0/1047225442 wait complete. 2024-03-18T00:57:03.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.017+0000 7f471540e740 1 -- start start 2024-03-18T00:57:03.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.018+0000 7f471540e740 1 Processor -- start 2024-03-18T00:57:03.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.019+0000 7f471540e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb953fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.019+0000 7f471540e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9536570 0x563eb9540430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.019+0000 7f471540e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9545460 0x563eb9547850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.019+0000 7f471540e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563eb953e680 con 0x563eb9535b80 2024-03-18T00:57:03.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.020+0000 7f471540e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563eb94560d0 con 0x563eb9545460 2024-03-18T00:57:03.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.020+0000 7f471540e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563eb953e800 con 0x563eb9536570 2024-03-18T00:57:03.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.020+0000 7f471540e740 10 client.0 _ll_register_callbacks cb 0x563eb941f760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:03.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.020+0000 7f470bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb953fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.020+0000 7f470bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb953fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54728/0 (socket says 172.21.15.49:54728) 2024-03-18T00:57:03.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.020+0000 7f470bfff640 1 -- 172.21.15.49:0/2676688303 learned_addr learned my addr 172.21.15.49:0/2676688303 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:03.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.021+0000 7f4710cef640 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9545460 0x563eb9547850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.021+0000 7f470bfff640 1 -- 172.21.15.49:0/2676688303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9536570 msgr2=0x563eb9540430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.021+0000 7f470bfff640 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9536570 0x563eb9540430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.021+0000 7f470bfff640 1 -- 172.21.15.49:0/2676688303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9545460 msgr2=0x563eb9547850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.021+0000 7f470bfff640 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9545460 0x563eb9547850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.021+0000 7f470bfff640 1 -- 172.21.15.49:0/2676688303 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eb92d6f70 con 0x563eb9535b80 2024-03-18T00:57:03.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.026+0000 7f470bfff640 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb953fef0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f470009a920 tx=0x7f470009a990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd81b4c9c704c208 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.026+0000 7f46eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.030+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2676688303 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47000b2070 con 0x563eb9535b80 2024-03-18T00:57:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.030+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2676688303 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4700097000 con 0x563eb9535b80 2024-03-18T00:57:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.030+0000 7f471540e740 1 -- 172.21.15.49:0/2676688303 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563eb94a7610 con 0x563eb9535b80 2024-03-18T00:57:03.030 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.031+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2676688303 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f47000a3130 con 0x563eb9535b80 2024-03-18T00:57:03.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.032+0000 7f471540e740 10 client.17112 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:03.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.032+0000 7f471540e740 1 -- 172.21.15.49:0/2676688303 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563eb916e170 con 0x563eb9535b80 2024-03-18T00:57:03.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.041+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2676688303 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x563eb94a7610 con 0x563eb9535b80 2024-03-18T00:57:03.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.041+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2676688303 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4700099de0 con 0x563eb9535b80 2024-03-18T00:57:03.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.041+0000 7f471540e740 10 client.17112 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:03.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.041+0000 7f471540e740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:03.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.042+0000 7f471540e740 20 client.17112 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.042+0000 7f471540e740 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563eb954bed0 0x563eb954e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.042+0000 7f471540e740 4 client.17112 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:03.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.042+0000 7f471540e740 1 -- 172.21.15.49:0/2676688303 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563eb94a1960 con 0x563eb954bed0 2024-03-18T00:57:03.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.045+0000 7f470b7fe640 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563eb954bed0 0x563eb954e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.067+0000 7f470b7fe640 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563eb954bed0 0x563eb954e340 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b22aab65d60d816e server_cookie=2303bf0d09d987f9 in_seq=0 out_seq=0 2024-03-18T00:57:03.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.067+0000 7f46eb7fe640 10 client.17112 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:03.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.076+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2676688303 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563eb94a1960 con 0x563eb954bed0 2024-03-18T00:57:03.076 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.076+0000 7f46eb7fe640 10 client.17112 handle_command_reply: tid=0 2024-03-18T00:57:03.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.080+0000 7f471540e740 1 client.17112 shutdown 2024-03-18T00:57:03.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f471540e740 1 -- 172.21.15.49:0/2676688303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 msgr2=0x563eb953fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.080 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f471540e740 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb953fef0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f470009a920 tx=0x7f470009a990 comp rx=0 tx=0).stop 2024-03-18T00:57:03.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f471540e740 1 -- 172.21.15.49:0/2676688303 shutdown_connections 2024-03-18T00:57:03.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f471540e740 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563eb954bed0 0x563eb954e340 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f471540e740 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9545460 0x563eb9547850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f471540e740 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9536570 0x563eb9540430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f471540e740 1 --2- 172.21.15.49:0/2676688303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb953fef0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f471540e740 1 -- 172.21.15.49:0/2676688303 >> 172.21.15.49:0/2676688303 conn(0x563eb94fdc50 msgr2=0x563eb953d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f46eb7fe640 0 client.17112 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:03.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.081+0000 7f471540e740 1 -- 172.21.15.49:0/2676688303 shutdown_connections 2024-03-18T00:57:03.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.082+0000 7f471540e740 1 -- 172.21.15.49:0/2676688303 wait complete. 2024-03-18T00:57:03.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.082+0000 7f471540e740 20 client.17112 trim_cache size 0 max 16384 2024-03-18T00:57:03.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.159+0000 7f471540e740 1 Processor -- start 2024-03-18T00:57:03.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.159+0000 7f471540e740 1 -- start start 2024-03-18T00:57:03.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.160+0000 7f471540e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9535b80 0x563eb9534e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.160+0000 7f471540e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9536570 0x563eb953fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.161+0000 7f471540e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9545460 0x563eb9540040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.161+0000 7f471540e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f470c1b77e0 con 0x563eb9545460 2024-03-18T00:57:03.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.161+0000 7f471540e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f470c1b9320 con 0x563eb9535b80 2024-03-18T00:57:03.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.162+0000 7f4710cef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9545460 0x563eb9540040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.162+0000 7f4710cef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9545460 0x563eb9540040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54744/0 (socket says 172.21.15.49:54744) 2024-03-18T00:57:03.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.162+0000 7f4710cef640 1 -- 172.21.15.49:0/2752675734 learned_addr learned my addr 172.21.15.49:0/2752675734 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:03.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.163+0000 7f471540e740 1 -- 172.21.15.49:0/2752675734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f470c1b7eb0 con 0x563eb9536570 2024-03-18T00:57:03.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.164+0000 7f4710cef640 1 -- 172.21.15.49:0/2752675734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9536570 msgr2=0x563eb953fb00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:03.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.164+0000 7f4710cef640 1 --2- 172.21.15.49:0/2752675734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9536570 0x563eb953fb00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.164+0000 7f4710cef640 1 -- 172.21.15.49:0/2752675734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9535b80 msgr2=0x563eb9534e40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:03.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.164+0000 7f4710cef640 1 --2- 172.21.15.49:0/2752675734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9535b80 0x563eb9534e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.164+0000 7f4710cef640 1 -- 172.21.15.49:0/2752675734 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563eb9536d60 con 0x563eb9545460 2024-03-18T00:57:03.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.164+0000 7f4710cef640 1 --2- 172.21.15.49:0/2752675734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9545460 0x563eb9540040 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0059d0 tx=0x7f46fc02c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e970bfd22cc3aa4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.166+0000 7f46f1ffb640 1 -- 172.21.15.49:0/2752675734 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc045040 con 0x563eb9545460 2024-03-18T00:57:03.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.166+0000 7f46f1ffb640 1 -- 172.21.15.49:0/2752675734 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46fc02cad0 con 0x563eb9545460 2024-03-18T00:57:03.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.166+0000 7f46f1ffb640 1 -- 172.21.15.49:0/2752675734 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc003dd0 con 0x563eb9545460 2024-03-18T00:57:03.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.169+0000 7f471540e740 1 -- 172.21.15.49:0/2752675734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9545460 msgr2=0x563eb9540040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.169+0000 7f471540e740 1 --2- 172.21.15.49:0/2752675734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9545460 0x563eb9540040 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0059d0 tx=0x7f46fc02c3f0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.171+0000 7f471540e740 1 -- 172.21.15.49:0/2752675734 shutdown_connections 2024-03-18T00:57:03.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.171+0000 7f471540e740 1 --2- 172.21.15.49:0/2752675734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9545460 0x563eb9540040 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.171+0000 7f471540e740 1 --2- 172.21.15.49:0/2752675734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9536570 0x563eb953fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.171+0000 7f471540e740 1 --2- 172.21.15.49:0/2752675734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9535b80 0x563eb9534e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.171+0000 7f471540e740 1 -- 172.21.15.49:0/2752675734 >> 172.21.15.49:0/2752675734 conn(0x563eb94fdc50 msgr2=0x563eb9534730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.177+0000 7f471540e740 1 -- 172.21.15.49:0/2752675734 shutdown_connections 2024-03-18T00:57:03.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.178+0000 7f471540e740 1 -- 172.21.15.49:0/2752675734 wait complete. 2024-03-18T00:57:03.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.178+0000 7f471540e740 1 -- start start 2024-03-18T00:57:03.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.179+0000 7f471540e740 1 Processor -- start 2024-03-18T00:57:03.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.185+0000 7f471540e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb955dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.185+0000 7f471540e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 0x563eb955e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.185+0000 7f471540e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9545460 0x563eb955e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.185+0000 7f471540e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f470c1b8c50 con 0x563eb9535b80 2024-03-18T00:57:03.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.185+0000 7f471540e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f470c1b8580 con 0x563eb9536570 2024-03-18T00:57:03.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.185+0000 7f471540e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f470c1b57c0 con 0x563eb9545460 2024-03-18T00:57:03.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.185+0000 7f471540e740 10 client.0 _ll_register_callbacks cb 0x7f470009a4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:03.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.187+0000 7f470bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb955dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.187+0000 7f470bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb955dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54756/0 (socket says 172.21.15.49:54756) 2024-03-18T00:57:03.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.187+0000 7f470bfff640 1 -- 172.21.15.49:0/2955600939 learned_addr learned my addr 172.21.15.49:0/2955600939 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:03.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.188+0000 7f4710cef640 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9545460 0x563eb955e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.189+0000 7f470b7fe640 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 0x563eb955e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.189+0000 7f470bfff640 1 -- 172.21.15.49:0/2955600939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9545460 msgr2=0x563eb955e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.189+0000 7f470bfff640 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9545460 0x563eb955e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.189+0000 7f470bfff640 1 -- 172.21.15.49:0/2955600939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 msgr2=0x563eb955e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.189+0000 7f470bfff640 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 0x563eb955e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.189+0000 7f470bfff640 1 -- 172.21.15.49:0/2955600939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eb95c2c10 con 0x563eb9535b80 2024-03-18T00:57:03.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.192+0000 7f470bfff640 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb955dbe0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4700010e50 tx=0x7f4700099140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4806f652628671 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.193+0000 7f46eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:03.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.194+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2955600939 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47000b2070 con 0x563eb9535b80 2024-03-18T00:57:03.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.194+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2955600939 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4700010940 con 0x563eb9535b80 2024-03-18T00:57:03.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.195+0000 7f471540e740 1 -- 172.21.15.49:0/2955600939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46fc02cad0 con 0x563eb9535b80 2024-03-18T00:57:03.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.198+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2955600939 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4700099a80 con 0x563eb9535b80 2024-03-18T00:57:03.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.198+0000 7f471540e740 10 client.17118 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:03.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.198+0000 7f471540e740 1 -- 172.21.15.49:0/2955600939 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563eb95c3ef0 con 0x563eb9535b80 2024-03-18T00:57:03.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.199+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2955600939 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46fc02cad0 con 0x563eb9535b80 2024-03-18T00:57:03.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.199+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2955600939 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f470009fa00 con 0x563eb9535b80 2024-03-18T00:57:03.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.205+0000 7f471540e740 10 client.17118 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:03.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.205+0000 7f471540e740 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:03.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.205+0000 7f471540e740 20 client.17118 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:03.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.205+0000 7f471540e740 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563eb95c4870 0x563eb95c6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.205+0000 7f471540e740 4 client.17118 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:57:03.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.205+0000 7f471540e740 1 -- 172.21.15.49:0/2955600939 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563eb941fbe0 con 0x563eb95c4870 2024-03-18T00:57:03.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.211+0000 7f470b7fe640 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563eb95c4870 0x563eb95c6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.219 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.220+0000 7f470b7fe640 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563eb95c4870 0x563eb95c6c90 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0bc2efc827420ad server_cookie=e00168c57908cfda in_seq=0 out_seq=0 2024-03-18T00:57:03.221 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.220+0000 7f46eb7fe640 10 client.17118 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:03.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.229+0000 7f46eb7fe640 1 -- 172.21.15.49:0/2955600939 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563eb941fbe0 con 0x563eb95c4870 2024-03-18T00:57:03.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.229+0000 7f46eb7fe640 10 client.17118 handle_command_reply: tid=0 2024-03-18T00:57:03.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.231+0000 7f471540e740 1 client.17118 shutdown 2024-03-18T00:57:03.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.242+0000 7f471540e740 1 -- 172.21.15.49:0/2955600939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 msgr2=0x563eb955dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.242+0000 7f471540e740 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb955dbe0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f4700010e50 tx=0x7f4700099140 comp rx=0 tx=0).stop 2024-03-18T00:57:03.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.247+0000 7f471540e740 1 -- 172.21.15.49:0/2955600939 shutdown_connections 2024-03-18T00:57:03.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.247+0000 7f471540e740 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563eb95c4870 0x563eb95c6c90 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.247+0000 7f471540e740 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563eb9536570 0x563eb955e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.247+0000 7f471540e740 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563eb9545460 0x563eb955e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.247+0000 7f471540e740 1 --2- 172.21.15.49:0/2955600939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563eb9535b80 0x563eb955dbe0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.247+0000 7f46eb7fe640 0 client.17118 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:03.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.247+0000 7f471540e740 1 -- 172.21.15.49:0/2955600939 >> 172.21.15.49:0/2955600939 conn(0x563eb94fdc50 msgr2=0x563eb953fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.250+0000 7f471540e740 1 -- 172.21.15.49:0/2955600939 shutdown_connections 2024-03-18T00:57:03.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.251+0000 7f471540e740 1 -- 172.21.15.49:0/2955600939 wait complete. 2024-03-18T00:57:03.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.252+0000 7f471540e740 20 client.17118 trim_cache size 0 max 16384 2024-03-18T00:57:03.252 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:57:03.252 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:57:03.253 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "20f34e44-3d92-4000-bce5-380e5de28e1e", 2024-03-18T00:57:03.253 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:57:03.253 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:57:03.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.259+0000 7f47114f0640 1 -- 172.21.15.49:0/716121235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46f40a2870 msgr2=0x7f46f40a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.259+0000 7f47114f0640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46f40a2870 0x7f46f40a4d30 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x563eb9536970 tx=0x7f4704009500 comp rx=0 tx=0).stop 2024-03-18T00:57:03.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.259+0000 7f47114f0640 1 -- 172.21.15.49:0/716121235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f470c072bf0 msgr2=0x7f470c1a6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.259+0000 7f47114f0640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f470c072bf0 0x7f470c1a6750 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f470c073940 tx=0x7f470000dbb0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.264+0000 7f47114f0640 1 -- 172.21.15.49:0/716121235 shutdown_connections 2024-03-18T00:57:03.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.264+0000 7f47114f0640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f46f40a2870 0x7f46f40a4d30 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.264+0000 7f47114f0640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f470c108eb0 0x7f470c1a6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.264+0000 7f47114f0640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f470c109870 0x7f470c1add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.264+0000 7f47114f0640 1 --2- 172.21.15.49:0/716121235 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f470c072bf0 0x7f470c1a6750 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.264+0000 7f47114f0640 1 -- 172.21.15.49:0/716121235 >> 172.21.15.49:0/716121235 conn(0x7f470c06e6c0 msgr2=0x7f470c0702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.270+0000 7f47114f0640 1 -- 172.21.15.49:0/716121235 shutdown_connections 2024-03-18T00:57:03.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.270+0000 7f47114f0640 1 -- 172.21.15.49:0/716121235 wait complete. 2024-03-18T00:57:03.317 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:57:03.317 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:57:03.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:03 smithi049 ceph-mon[32130]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 939 MiB/s wr, 2.56k op/s 2024-03-18T00:57:03.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:03 smithi107 ceph-mon[36270]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 939 MiB/s wr, 2.56k op/s 2024-03-18T00:57:03.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:03 smithi106 ceph-mon[35899]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 939 MiB/s wr, 2.56k op/s 2024-03-18T00:57:03.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.719+0000 7f8f2eb22640 1 -- 172.21.15.49:0/1864075741 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 msgr2=0x7f8f280ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.719+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/1864075741 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 0x7f8f280ffac0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009900 tx=0x7f8f1c02bc60 comp rx=0 tx=0).stop 2024-03-18T00:57:03.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.720+0000 7f8f2eb22640 1 -- 172.21.15.49:0/1864075741 shutdown_connections 2024-03-18T00:57:03.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.720+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/1864075741 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f28101170 0x7f8f28105a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.720+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/1864075741 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f281007d0 0x7f8f28100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.720+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/1864075741 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 0x7f8f280ffac0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.720+0000 7f8f2eb22640 1 -- 172.21.15.49:0/1864075741 >> 172.21.15.49:0/1864075741 conn(0x7f8f280fafa0 msgr2=0x7f8f280fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.723+0000 7f8f2eb22640 1 -- 172.21.15.49:0/1864075741 shutdown_connections 2024-03-18T00:57:03.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.726+0000 7f8f2eb22640 1 -- 172.21.15.49:0/1864075741 wait complete. 2024-03-18T00:57:03.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.727+0000 7f8f2eb22640 1 Processor -- start 2024-03-18T00:57:03.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.732+0000 7f8f2eb22640 1 -- start start 2024-03-18T00:57:03.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.732+0000 7f8f2eb22640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 0x7f8f2819e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.732+0000 7f8f2eb22640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f281007d0 0x7f8f2819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.732+0000 7f8f2eb22640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f28101170 0x7f8f281a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.732+0000 7f8f2eb22640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8f28113ae0 con 0x7f8f280ff6a0 2024-03-18T00:57:03.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.732+0000 7f8f2eb22640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8f28113960 con 0x7f8f281007d0 2024-03-18T00:57:03.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.732+0000 7f8f2eb22640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8f28113c60 con 0x7f8f28101170 2024-03-18T00:57:03.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.734+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f28101170 0x7f8f281a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.735+0000 7f8f2c8be640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 0x7f8f2819e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.735+0000 7f8f2c8be640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 0x7f8f2819e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54774/0 (socket says 172.21.15.49:54774) 2024-03-18T00:57:03.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.735+0000 7f8f2c8be640 1 -- 172.21.15.49:0/2830806695 learned_addr learned my addr 172.21.15.49:0/2830806695 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:03.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.736+0000 7f8f27fff640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f281007d0 0x7f8f2819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.737+0000 7f8f2c8be640 1 -- 172.21.15.49:0/2830806695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f28101170 msgr2=0x7f8f281a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.737+0000 7f8f2c8be640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f28101170 0x7f8f281a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.737+0000 7f8f2c8be640 1 -- 172.21.15.49:0/2830806695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f281007d0 msgr2=0x7f8f2819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.737+0000 7f8f2c8be640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f281007d0 0x7f8f2819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.737+0000 7f8f2c8be640 1 -- 172.21.15.49:0/2830806695 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f281a5ce0 con 0x7f8f280ff6a0 2024-03-18T00:57:03.739 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.739+0000 7f8f27fff640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f281007d0 0x7f8f2819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:57:03.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.745+0000 7f8f2c8be640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 0x7f8f2819e0c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c02c810 tx=0x7f8f1c0045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.755+0000 7f8f25ffb640 1 -- 172.21.15.49:0/2830806695 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c042020 con 0x7f8f280ff6a0 2024-03-18T00:57:03.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.755+0000 7f8f25ffb640 1 -- 172.21.15.49:0/2830806695 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f1c005410 con 0x7f8f280ff6a0 2024-03-18T00:57:03.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.755+0000 7f8f2eb22640 1 -- 172.21.15.49:0/2830806695 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f281a5f70 con 0x7f8f280ff6a0 2024-03-18T00:57:03.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.755+0000 7f8f25ffb640 1 -- 172.21.15.49:0/2830806695 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c03e860 con 0x7f8f280ff6a0 2024-03-18T00:57:03.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.756+0000 7f8f2eb22640 1 -- 172.21.15.49:0/2830806695 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f281a6370 con 0x7f8f280ff6a0 2024-03-18T00:57:03.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.760+0000 7f8f317ef740 1 Processor -- start 2024-03-18T00:57:03.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.761+0000 7f8f317ef740 1 -- start start 2024-03-18T00:57:03.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.761+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027e170 0x564b2027e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.761+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b2027eb60 0x564b20283660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.761+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b20283cd0 0x564b20286160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.761+0000 7f8f317ef740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564b201e3500 con 0x564b2027eb60 2024-03-18T00:57:03.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.761+0000 7f8f317ef740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x564b2018bbe0 con 0x564b2027e170 2024-03-18T00:57:03.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.761+0000 7f8f317ef740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564b201df5a0 con 0x564b20283cd0 2024-03-18T00:57:03.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.762+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027e170 0x564b2027e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.762+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027e170 0x564b2027e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:44980/0 (socket says 172.21.15.49:44980) 2024-03-18T00:57:03.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.762+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/2659961802 learned_addr learned my addr 172.21.15.49:0/2659961802 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:03.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.764+0000 7f8f27fff640 1 --2- 172.21.15.49:0/2659961802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b20283cd0 0x564b20286160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.766+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/2659961802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b20283cd0 msgr2=0x564b20286160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.766+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/2659961802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b20283cd0 0x564b20286160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.766+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/2659961802 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b2027eb60 msgr2=0x564b20283660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.766+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/2659961802 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b2027eb60 0x564b20283660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.766 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.766+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/2659961802 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b2019f4b0 con 0x564b2027e170 2024-03-18T00:57:03.769 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.768+0000 7f8f25ffb640 1 -- 172.21.15.49:0/2830806695 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8f1c045020 con 0x7f8f280ff6a0 2024-03-18T00:57:03.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.770+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/2659961802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027e170 0x564b2027e590 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800b700 tx=0x7f8f1800bbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b709f80202cd7f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.772+0000 7f8f25ffb640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f100a2a10 0x7f8f100a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.773+0000 7f8f25ffb640 1 -- 172.21.15.49:0/2830806695 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8f1c0f2a60 con 0x7f8f280ff6a0 2024-03-18T00:57:03.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.773+0000 7f8f0effd640 1 -- 172.21.15.49:0/2659961802 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18003dd0 con 0x564b2027e170 2024-03-18T00:57:03.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.774+0000 7f8f0effd640 1 -- 172.21.15.49:0/2659961802 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f18015070 con 0x564b2027e170 2024-03-18T00:57:03.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.774+0000 7f8f2c8be640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f100a2a10 0x7f8f100a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.775 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.775+0000 7f8f0effd640 1 -- 172.21.15.49:0/2659961802 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f180106a0 con 0x564b2027e170 2024-03-18T00:57:03.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.775+0000 7f8f317ef740 1 -- 172.21.15.49:0/2659961802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027e170 msgr2=0x564b2027e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.775+0000 7f8f317ef740 1 --2- 172.21.15.49:0/2659961802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027e170 0x564b2027e590 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800b700 tx=0x7f8f1800bbc0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.776+0000 7f8f317ef740 1 -- 172.21.15.49:0/2659961802 shutdown_connections 2024-03-18T00:57:03.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.776+0000 7f8f317ef740 1 --2- 172.21.15.49:0/2659961802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b20283cd0 0x564b20286160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.776+0000 7f8f317ef740 1 --2- 172.21.15.49:0/2659961802 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b2027eb60 0x564b20283660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.776+0000 7f8f317ef740 1 --2- 172.21.15.49:0/2659961802 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027e170 0x564b2027e590 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.776+0000 7f8f317ef740 1 -- 172.21.15.49:0/2659961802 >> 172.21.15.49:0/2659961802 conn(0x564b202796e0 msgr2=0x564b20279ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.779+0000 7f8f2c8be640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f100a2a10 0x7f8f100a4ed0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0bb7e0 tx=0x7f8f1c005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.779+0000 7f8f317ef740 1 -- 172.21.15.49:0/2659961802 shutdown_connections 2024-03-18T00:57:03.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.781+0000 7f8f317ef740 1 -- 172.21.15.49:0/2659961802 wait complete. 2024-03-18T00:57:03.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.782+0000 7f8f317ef740 1 -- start start 2024-03-18T00:57:03.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.782+0000 7f8f317ef740 1 Processor -- start 2024-03-18T00:57:03.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.783+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b20288830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.783+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b20288d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.783+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.783+0000 7f8f317ef740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564b20286cd0 con 0x564b20283cd0 2024-03-18T00:57:03.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.783+0000 7f8f317ef740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x564b2019f160 con 0x564b2027eb60 2024-03-18T00:57:03.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.783+0000 7f8f317ef740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564b20286e50 con 0x564b2027e170 2024-03-18T00:57:03.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.783+0000 7f8f317ef740 10 client.0 _ll_register_callbacks cb 0x564b20183150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:03.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.787+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b20288830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.787+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.787+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54780/0 (socket says 172.21.15.49:54780) 2024-03-18T00:57:03.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.787+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/343329405 learned_addr learned my addr 172.21.15.49:0/343329405 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:03.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.787+0000 7f8f27fff640 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b20288d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.788+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/343329405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 msgr2=0x564b20288830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.788+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b20288830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.788+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/343329405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 msgr2=0x564b20288d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.788+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b20288d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.788+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/343329405 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b1ffc17d0 con 0x564b20283cd0 2024-03-18T00:57:03.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.792+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028fd40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800b6d0 tx=0x7f8f18000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d36dc43849295a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.792+0000 7f8f07fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:03.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.795+0000 7f8f07fff640 1 -- 172.21.15.49:0/343329405 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1802b060 con 0x564b20283cd0 2024-03-18T00:57:03.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.795+0000 7f8f07fff640 1 -- 172.21.15.49:0/343329405 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8f18003d80 con 0x564b20283cd0 2024-03-18T00:57:03.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.795+0000 7f8f317ef740 1 -- 172.21.15.49:0/343329405 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f18015070 con 0x564b20283cd0 2024-03-18T00:57:03.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.798+0000 7f8f07fff640 1 -- 172.21.15.49:0/343329405 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8f18011440 con 0x564b20283cd0 2024-03-18T00:57:03.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.798+0000 7f8f317ef740 10 client.17127 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:03.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.798+0000 7f8f317ef740 1 -- 172.21.15.49:0/343329405 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564b1fef73f0 con 0x564b20283cd0 2024-03-18T00:57:03.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.799+0000 7f8f07fff640 1 -- 172.21.15.49:0/343329405 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f18015070 con 0x564b20283cd0 2024-03-18T00:57:03.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.799+0000 7f8f07fff640 1 -- 172.21.15.49:0/343329405 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8f1801a540 con 0x564b20283cd0 2024-03-18T00:57:03.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.801+0000 7f8f317ef740 10 client.17127 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:03.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.801+0000 7f8f317ef740 10 client.17127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:03.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.801+0000 7f8f317ef740 20 client.17127 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:03.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.801+0000 7f8f317ef740 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564b20294030 0x564b20296420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.801+0000 7f8f317ef740 4 client.17127 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:03.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.801+0000 7f8f317ef740 1 -- 172.21.15.49:0/343329405 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b201f0a40 con 0x564b20294030 2024-03-18T00:57:03.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.802+0000 7f8f27fff640 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564b20294030 0x564b20296420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.807+0000 7f8f27fff640 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564b20294030 0x564b20296420 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96881c0aa9b42222 server_cookie=559de0be89a5d0fc in_seq=0 out_seq=0 2024-03-18T00:57:03.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.807+0000 7f8f07fff640 10 client.17127 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.814+0000 7f8f07fff640 1 -- 172.21.15.49:0/343329405 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564b201f0a40 con 0x564b20294030 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.814+0000 7f8f07fff640 10 client.17127 handle_command_reply: tid=0 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.814+0000 7f8f317ef740 1 client.17127 shutdown 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 -- 172.21.15.49:0/343329405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 msgr2=0x564b2028fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028fd40 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f8f1800b6d0 tx=0x7f8f18000bc0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 -- 172.21.15.49:0/343329405 shutdown_connections 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564b20294030 0x564b20296420 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b20288d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028fd40 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f07fff640 0 client.17127 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 --2- 172.21.15.49:0/343329405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b20288830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 -- 172.21.15.49:0/343329405 >> 172.21.15.49:0/343329405 conn(0x564b202796e0 msgr2=0x564b20278900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 -- 172.21.15.49:0/343329405 shutdown_connections 2024-03-18T00:57:03.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 1 -- 172.21.15.49:0/343329405 wait complete. 2024-03-18T00:57:03.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.815+0000 7f8f317ef740 20 client.17127 trim_cache size 0 max 16384 2024-03-18T00:57:03.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.852+0000 7f8f317ef740 1 Processor -- start 2024-03-18T00:57:03.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.853+0000 7f8f317ef740 1 -- start start 2024-03-18T00:57:03.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.853+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b20278070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.854+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b2028fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.854+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b202785b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.854+0000 7f8f317ef740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1400e650 con 0x564b20283cd0 2024-03-18T00:57:03.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.854+0000 7f8f317ef740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8f14010190 con 0x564b2027eb60 2024-03-18T00:57:03.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.854+0000 7f8f317ef740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1400ed20 con 0x564b2027e170 2024-03-18T00:57:03.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.854+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b202785b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.854+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b202785b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54796/0 (socket says 172.21.15.49:54796) 2024-03-18T00:57:03.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.854+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/371178405 learned_addr learned my addr 172.21.15.49:0/371178405 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:03.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.854+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/371178405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b20278070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.855+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/371178405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 msgr2=0x564b20278070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.855+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/371178405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b20278070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.855+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/371178405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 msgr2=0x564b2028fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.855+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/371178405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b2028fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.855+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/371178405 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b1feb70f0 con 0x564b20283cd0 2024-03-18T00:57:03.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.855+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/371178405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b202785b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x564b20288760 tx=0x7f8f1800b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8307eb40edd8627 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.856+0000 7f8f0e7fc640 1 -- 172.21.15.49:0/371178405 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1802b070 con 0x564b20283cd0 2024-03-18T00:57:03.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.856+0000 7f8f0e7fc640 1 -- 172.21.15.49:0/371178405 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f18015040 con 0x564b20283cd0 2024-03-18T00:57:03.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.856+0000 7f8f0e7fc640 1 -- 172.21.15.49:0/371178405 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f18035080 con 0x564b20283cd0 2024-03-18T00:57:03.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.858+0000 7f8f317ef740 1 -- 172.21.15.49:0/371178405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 msgr2=0x564b202785b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.858+0000 7f8f317ef740 1 --2- 172.21.15.49:0/371178405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b202785b0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x564b20288760 tx=0x7f8f1800b2e0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.858+0000 7f8f317ef740 1 -- 172.21.15.49:0/371178405 shutdown_connections 2024-03-18T00:57:03.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.858+0000 7f8f317ef740 1 --2- 172.21.15.49:0/371178405 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b2028fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.858+0000 7f8f317ef740 1 --2- 172.21.15.49:0/371178405 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b202785b0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.858+0000 7f8f317ef740 1 --2- 172.21.15.49:0/371178405 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b20278070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.858+0000 7f8f317ef740 1 -- 172.21.15.49:0/371178405 >> 172.21.15.49:0/371178405 conn(0x564b202796e0 msgr2=0x564b20288830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.858+0000 7f8f317ef740 1 -- 172.21.15.49:0/371178405 shutdown_connections 2024-03-18T00:57:03.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.858+0000 7f8f317ef740 1 -- 172.21.15.49:0/371178405 wait complete. 2024-03-18T00:57:03.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.859+0000 7f8f317ef740 1 -- start start 2024-03-18T00:57:03.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.859+0000 7f8f317ef740 1 Processor -- start 2024-03-18T00:57:03.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.859+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b2028d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.859+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b2028dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.859+0000 7f8f317ef740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.859+0000 7f8f317ef740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1400fac0 con 0x564b20283cd0 2024-03-18T00:57:03.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.859+0000 7f8f317ef740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1400f3f0 con 0x564b2027eb60 2024-03-18T00:57:03.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.859+0000 7f8f317ef740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1400c630 con 0x564b2027e170 2024-03-18T00:57:03.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.859+0000 7f8f317ef740 10 client.0 _ll_register_callbacks cb 0x7f8f18004590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:03.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.860+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.860+0000 7f8f27fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b2028dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.860+0000 7f8f2d0bf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54802/0 (socket says 172.21.15.49:54802) 2024-03-18T00:57:03.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.860+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/3700949747 learned_addr learned my addr 172.21.15.49:0/3700949747 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:03.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.861+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b2028d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.861+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/3700949747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 msgr2=0x564b2028d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.861+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b2028d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.861+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/3700949747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 msgr2=0x564b2028dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:03.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.861+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b2028dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.861+0000 7f8f2d0bf640 1 -- 172.21.15.49:0/3700949747 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b201aa9d0 con 0x564b20283cd0 2024-03-18T00:57:03.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.861+0000 7f8f27fff640 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b2028dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:57:03.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.862+0000 7f8f2d0bf640 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028eae0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8f180039c0 tx=0x7f8f1800bbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c345cd372a57b68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:03.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.862+0000 7f8f07fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:03.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.862+0000 7f8f07fff640 1 -- 172.21.15.49:0/3700949747 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1802b070 con 0x564b20283cd0 2024-03-18T00:57:03.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.862+0000 7f8f07fff640 1 -- 172.21.15.49:0/3700949747 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8f1801a470 con 0x564b20283cd0 2024-03-18T00:57:03.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.862+0000 7f8f317ef740 1 -- 172.21.15.49:0/3700949747 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f18015040 con 0x564b20283cd0 2024-03-18T00:57:03.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.862+0000 7f8f07fff640 1 -- 172.21.15.49:0/3700949747 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8f1801a690 con 0x564b20283cd0 2024-03-18T00:57:03.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.862+0000 7f8f317ef740 10 client.17133 fetch_fsmap learned FSMap version 27 2024-03-18T00:57:03.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.862+0000 7f8f317ef740 1 -- 172.21.15.49:0/3700949747 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564b202fa300 con 0x564b20283cd0 2024-03-18T00:57:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.863+0000 7f8f07fff640 1 -- 172.21.15.49:0/3700949747 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f18015040 con 0x564b20283cd0 2024-03-18T00:57:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.863+0000 7f8f07fff640 1 -- 172.21.15.49:0/3700949747 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8f18004ac0 con 0x564b20283cd0 2024-03-18T00:57:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.863+0000 7f8f317ef740 10 client.17133 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T00:57:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.863+0000 7f8f317ef740 10 client.17133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.863+0000 7f8f317ef740 20 client.17133 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.863+0000 7f8f317ef740 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564b202fac50 0x564b202fd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.863+0000 7f8f317ef740 4 client.17133 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:57:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.863+0000 7f8f317ef740 1 -- 172.21.15.49:0/3700949747 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564b201ead90 con 0x564b202fac50 2024-03-18T00:57:03.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.864+0000 7f8f27fff640 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564b202fac50 0x564b202fd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:03.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.866+0000 7f8f27fff640 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564b202fac50 0x564b202fd070 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acc521d5514ec6a0 server_cookie=40219d0fbd179d5c in_seq=0 out_seq=0 2024-03-18T00:57:03.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.866+0000 7f8f07fff640 10 client.17133 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:03.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.870+0000 7f8f07fff640 1 -- 172.21.15.49:0/3700949747 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x564b201ead90 con 0x564b202fac50 2024-03-18T00:57:03.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.870+0000 7f8f07fff640 10 client.17133 handle_command_reply: tid=0 2024-03-18T00:57:03.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.870+0000 7f8f317ef740 1 client.17133 shutdown 2024-03-18T00:57:03.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.871+0000 7f8f317ef740 1 -- 172.21.15.49:0/3700949747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 msgr2=0x564b2028eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.871+0000 7f8f317ef740 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028eae0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8f180039c0 tx=0x7f8f1800bbc0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.871+0000 7f8f317ef740 1 -- 172.21.15.49:0/3700949747 shutdown_connections 2024-03-18T00:57:03.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.872+0000 7f8f317ef740 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564b202fac50 0x564b202fd070 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.872+0000 7f8f07fff640 0 client.17133 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:03.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.872+0000 7f8f317ef740 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564b2027eb60 0x564b2028dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.872+0000 7f8f317ef740 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564b20283cd0 0x564b2028eae0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.872+0000 7f8f317ef740 1 --2- 172.21.15.49:0/3700949747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564b2027e170 0x564b2028d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.873+0000 7f8f317ef740 1 -- 172.21.15.49:0/3700949747 >> 172.21.15.49:0/3700949747 conn(0x564b202796e0 msgr2=0x564b2027c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.873+0000 7f8f317ef740 1 -- 172.21.15.49:0/3700949747 shutdown_connections 2024-03-18T00:57:03.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.873+0000 7f8f317ef740 1 -- 172.21.15.49:0/3700949747 wait complete. 2024-03-18T00:57:03.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.873+0000 7f8f317ef740 20 client.17133 trim_cache size 0 max 16384 2024-03-18T00:57:03.874 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:57:03.874 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-18T00:57:03.874 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:57:03.874 INFO:teuthology.orchestra.run.smithi049.stdout: "20f34e44-3d92-4000-bce5-380e5de28e1e": { 2024-03-18T00:57:03.874 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:57:03.874 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "20f34e44-3d92-4000-bce5-380e5de28e1e", 2024-03-18T00:57:03.874 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:57:03.874 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:57:03.874 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:57:03.875 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:57:03.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 -- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f100a2a10 msgr2=0x7f8f100a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f100a2a10 0x7f8f100a4ed0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0bb7e0 tx=0x7f8f1c005b50 comp rx=0 tx=0).stop 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 -- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 msgr2=0x7f8f2819e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 0x7f8f2819e0c0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c02c810 tx=0x7f8f1c0045c0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 -- 172.21.15.49:0/2830806695 shutdown_connections 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8f100a2a10 0x7f8f100a4ed0 secure :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0bb7e0 tx=0x7f8f1c005b50 comp rx=0 tx=0).stop 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8f281007d0 0x7f8f2819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f28101170 0x7f8f281a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 --2- 172.21.15.49:0/2830806695 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8f280ff6a0 0x7f8f2819e0c0 secure :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c02c810 tx=0x7f8f1c0045c0 comp rx=0 tx=0).stop 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.874+0000 7f8f2eb22640 1 -- 172.21.15.49:0/2830806695 >> 172.21.15.49:0/2830806695 conn(0x7f8f280fafa0 msgr2=0x7f8f28101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.875+0000 7f8f2eb22640 1 -- 172.21.15.49:0/2830806695 shutdown_connections 2024-03-18T00:57:03.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:03.875+0000 7f8f2eb22640 1 -- 172.21.15.49:0/2830806695 wait complete. 2024-03-18T00:57:03.909 INFO:tasks.cephfs.filesystem:scrub status for tag:20f34e44-3d92-4000-bce5-380e5de28e1e - {'path': '/', 'tag': '20f34e44-3d92-4000-bce5-380e5de28e1e', 'options': 'recursive,force'} 2024-03-18T00:57:04.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:04 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:57:04.631 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:04 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:04.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:04 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:57:04.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:04 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:57:04.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:04 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:04.632 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:04 smithi049 ceph-mon[32130]: client.25201 does not advance its oldest_flush_tid (206491), 100000 completed flushes recorded in session 2024-03-18T00:57:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:04 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:57:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:04 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:04 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:57:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:04 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:57:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:04 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:04.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:04 smithi107 ceph-mon[36270]: client.25201 does not advance its oldest_flush_tid (206491), 100000 completed flushes recorded in session 2024-03-18T00:57:04.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:04 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:57:04.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:04 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:04.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:04 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:57:04.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:04 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:57:04.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:04 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:04.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:04 smithi106 ceph-mon[35899]: client.25201 does not advance its oldest_flush_tid (206491), 100000 completed flushes recorded in session 2024-03-18T00:57:05.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:05 smithi049 ceph-mon[32130]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 870 MiB/s wr, 2.33k op/s 2024-03-18T00:57:05.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:05 smithi107 ceph-mon[36270]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 870 MiB/s wr, 2.33k op/s 2024-03-18T00:57:05.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:05 smithi106 ceph-mon[35899]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 870 MiB/s wr, 2.33k op/s 2024-03-18T00:57:07.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:07 smithi049 ceph-mon[32130]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 869 MiB/s wr, 2.30k op/s 2024-03-18T00:57:07.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:07 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:07.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:07 smithi107 ceph-mon[36270]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 869 MiB/s wr, 2.30k op/s 2024-03-18T00:57:07.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:07 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:07.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:07 smithi106 ceph-mon[35899]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 869 MiB/s wr, 2.30k op/s 2024-03-18T00:57:07.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:07 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:08.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:08 smithi049 ceph-mon[32130]: Health check failed: 1 clients failing to advance oldest client/flush tid (MDS_CLIENT_OLDEST_TID) 2024-03-18T00:57:08.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:08 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:active 2024-03-18T00:57:08.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:08 smithi049 ceph-mon[32130]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:57:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:08 smithi107 ceph-mon[36270]: Health check failed: 1 clients failing to advance oldest client/flush tid (MDS_CLIENT_OLDEST_TID) 2024-03-18T00:57:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:08 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:active 2024-03-18T00:57:08.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:08 smithi107 ceph-mon[36270]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:57:08.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:08 smithi106 ceph-mon[35899]: Health check failed: 1 clients failing to advance oldest client/flush tid (MDS_CLIENT_OLDEST_TID) 2024-03-18T00:57:08.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:08 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:active 2024-03-18T00:57:08.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:08 smithi106 ceph-mon[35899]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:57:09.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:09 smithi049 ceph-mon[32130]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 898 MiB/s wr, 2.13k op/s 2024-03-18T00:57:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:09 smithi107 ceph-mon[36270]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 898 MiB/s wr, 2.13k op/s 2024-03-18T00:57:09.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:09 smithi106 ceph-mon[35899]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 898 MiB/s wr, 2.13k op/s 2024-03-18T00:57:10.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:10 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:57:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.668+0000 7f0a7e947640 1 -- 172.21.15.49:0/3004816369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a78103200 msgr2=0x7f0a78109a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:10.669 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.668+0000 7f0a7e947640 1 --2- 172.21.15.49:0/3004816369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a78103200 0x7f0a78109a90 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009900 tx=0x7f0a7002bc30 comp rx=0 tx=0).stop 2024-03-18T00:57:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.668+0000 7f0a7e947640 1 -- 172.21.15.49:0/3004816369 shutdown_connections 2024-03-18T00:57:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.668+0000 7f0a7e947640 1 --2- 172.21.15.49:0/3004816369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a78102840 0x7f0a78102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.668+0000 7f0a7e947640 1 --2- 172.21.15.49:0/3004816369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a78103200 0x7f0a78109a90 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.668+0000 7f0a7e947640 1 --2- 172.21.15.49:0/3004816369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a78101650 0x7f0a78101a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.668+0000 7f0a7e947640 1 -- 172.21.15.49:0/3004816369 >> 172.21.15.49:0/3004816369 conn(0x7f0a780fcbc0 msgr2=0x7f0a780ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.669+0000 7f0a7e947640 1 -- 172.21.15.49:0/3004816369 shutdown_connections 2024-03-18T00:57:10.670 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.669+0000 7f0a7e947640 1 -- 172.21.15.49:0/3004816369 wait complete. 2024-03-18T00:57:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.670+0000 7f0a7e947640 1 Processor -- start 2024-03-18T00:57:10.671 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.671+0000 7f0a7e947640 1 -- start start 2024-03-18T00:57:10.672 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.671+0000 7f0a7e947640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a78101650 0x7f0a78197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.671+0000 7f0a7e947640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a78102840 0x7f0a78197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.672+0000 7f0a7d945640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a78101650 0x7f0a78197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.672+0000 7f0a7d945640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a78101650 0x7f0a78197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:48282/0 (socket says 172.21.15.49:48282) 2024-03-18T00:57:10.673 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.673+0000 7f0a7d945640 1 -- 172.21.15.49:0/646553680 learned_addr learned my addr 172.21.15.49:0/646553680 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.673+0000 7f0a7e947640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a78103200 0x7f0a7819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.673+0000 7f0a7d144640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a78102840 0x7f0a78197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.673+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7810d320 con 0x7f0a78102840 2024-03-18T00:57:10.674 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.673+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7810d1a0 con 0x7f0a78103200 2024-03-18T00:57:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.673+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7810d4a0 con 0x7f0a78101650 2024-03-18T00:57:10.675 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.674+0000 7f0a7e146640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a78103200 0x7f0a7819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.675+0000 7f0a7d144640 1 -- 172.21.15.49:0/646553680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a78101650 msgr2=0x7f0a78197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.675+0000 7f0a7d144640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a78101650 0x7f0a78197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.676 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.676+0000 7f0a7d144640 1 -- 172.21.15.49:0/646553680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a78103200 msgr2=0x7f0a7819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.676+0000 7f0a7d144640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a78103200 0x7f0a7819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.677 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.676+0000 7f0a7d144640 1 -- 172.21.15.49:0/646553680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7819f580 con 0x7f0a78102840 2024-03-18T00:57:10.678 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.677+0000 7f0a7d144640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a78102840 0x7f0a78197ea0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f0a64005fd0 tx=0x7f0a6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.678+0000 7f0a76ffd640 1 -- 172.21.15.49:0/646553680 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6400cd80 con 0x7f0a78102840 2024-03-18T00:57:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.678+0000 7f0a76ffd640 1 -- 172.21.15.49:0/646553680 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a64005020 con 0x7f0a78102840 2024-03-18T00:57:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.678+0000 7f0a76ffd640 1 -- 172.21.15.49:0/646553680 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a640116d0 con 0x7f0a78102840 2024-03-18T00:57:10.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:10 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:57:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.678+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a7819f870 con 0x7f0a78102840 2024-03-18T00:57:10.681 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.679+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7819fd70 con 0x7f0a78102840 2024-03-18T00:57:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.683+0000 7f0a82791740 1 Processor -- start 2024-03-18T00:57:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.683+0000 7f0a82791740 1 -- start start 2024-03-18T00:57:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.684+0000 7f0a82791740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bff36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.684+0000 7f0a82791740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0bff8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.684+0000 7f0a82791740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff88c0 0x557d0bffad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.684+0000 7f0a82791740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557d0bf4fa40 con 0x557d0bff88c0 2024-03-18T00:57:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.684+0000 7f0a82791740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557d0bef8920 con 0x557d0bff32d0 2024-03-18T00:57:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.684+0000 7f0a82791740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557d0bf4bae0 con 0x557d0bff3cc0 2024-03-18T00:57:10.685 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.684+0000 7f0a76ffd640 1 -- 172.21.15.49:0/646553680 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0a64005520 con 0x7f0a78102840 2024-03-18T00:57:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.685+0000 7f0a7d144640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff88c0 0x557d0bffad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a76ffd640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a680a29c0 0x7f0a680a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a7d144640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff88c0 0x557d0bffad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54874/0 (socket says 172.21.15.49:54874) 2024-03-18T00:57:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a7d144640 1 -- 172.21.15.49:0/4004198034 learned_addr learned my addr 172.21.15.49:0/4004198034 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a7e146640 1 --2- 172.21.15.49:0/4004198034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bff36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a76ffd640 1 -- 172.21.15.49:0/646553680 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0a640ceaa0 con 0x7f0a78102840 2024-03-18T00:57:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a7e146640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a680a29c0 0x7f0a680a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a7d144640 1 -- 172.21.15.49:0/4004198034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 msgr2=0x557d0bff8250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a7d144640 1 --2- 172.21.15.49:0/4004198034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0bff8250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a7d144640 1 -- 172.21.15.49:0/4004198034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 msgr2=0x557d0bff36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.686+0000 7f0a7d144640 1 --2- 172.21.15.49:0/4004198034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bff36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.687+0000 7f0a7d144640 1 -- 172.21.15.49:0/4004198034 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d0bf10460 con 0x557d0bff88c0 2024-03-18T00:57:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.687+0000 7f0a7d144640 1 --2- 172.21.15.49:0/4004198034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff88c0 0x557d0bffad30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0a64095f10 tx=0x7f0a64005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4464430d934c9727 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.688+0000 7f0a63fff640 1 -- 172.21.15.49:0/4004198034 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6401f050 con 0x557d0bff88c0 2024-03-18T00:57:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.688+0000 7f0a63fff640 1 -- 172.21.15.49:0/4004198034 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a6409b470 con 0x557d0bff88c0 2024-03-18T00:57:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.688+0000 7f0a63fff640 1 -- 172.21.15.49:0/4004198034 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a64097ae0 con 0x557d0bff88c0 2024-03-18T00:57:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a7e146640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a680a29c0 0x7f0a680a4e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a70036040 tx=0x7f0a7002bc30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a82791740 1 -- 172.21.15.49:0/4004198034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff88c0 msgr2=0x557d0bffad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a82791740 1 --2- 172.21.15.49:0/4004198034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff88c0 0x557d0bffad30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f0a64095f10 tx=0x7f0a64005490 comp rx=0 tx=0).stop 2024-03-18T00:57:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a82791740 1 -- 172.21.15.49:0/4004198034 shutdown_connections 2024-03-18T00:57:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a82791740 1 --2- 172.21.15.49:0/4004198034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff88c0 0x557d0bffad30 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a82791740 1 --2- 172.21.15.49:0/4004198034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0bff8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a82791740 1 --2- 172.21.15.49:0/4004198034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bff36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.690 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a82791740 1 -- 172.21.15.49:0/4004198034 >> 172.21.15.49:0/4004198034 conn(0x557d0bfe59c0 msgr2=0x557d0bfe5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a82791740 1 -- 172.21.15.49:0/4004198034 shutdown_connections 2024-03-18T00:57:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.689+0000 7f0a82791740 1 -- 172.21.15.49:0/4004198034 wait complete. 2024-03-18T00:57:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.690+0000 7f0a82791740 1 -- start start 2024-03-18T00:57:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.690+0000 7f0a82791740 1 Processor -- start 2024-03-18T00:57:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.691+0000 7f0a82791740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bffd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.691+0000 7f0a82791740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff3cc0 0x557d0bffd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.691+0000 7f0a82791740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0c0026f0 0x557d0c004ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.691+0000 7f0a82791740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557d0bffb790 con 0x557d0bff3cc0 2024-03-18T00:57:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.692+0000 7f0a82791740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557d0bf0bd70 con 0x557d0bff32d0 2024-03-18T00:57:10.692 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.692+0000 7f0a82791740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557d0bffb910 con 0x557d0c0026f0 2024-03-18T00:57:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.692+0000 7f0a82791740 10 client.0 _ll_register_callbacks cb 0x557d0bc63a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.693+0000 7f0a7d144640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff3cc0 0x557d0bffd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.693+0000 7f0a7d144640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff3cc0 0x557d0bffd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54878/0 (socket says 172.21.15.49:54878) 2024-03-18T00:57:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.693+0000 7f0a7d144640 1 -- 172.21.15.49:0/2366694423 learned_addr learned my addr 172.21.15.49:0/2366694423 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.693+0000 7f0a7d945640 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bffd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.693+0000 7f0a7d144640 1 -- 172.21.15.49:0/2366694423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0c0026f0 msgr2=0x557d0c004ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.693+0000 7f0a7d144640 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0c0026f0 0x557d0c004ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.693+0000 7f0a7d144640 1 -- 172.21.15.49:0/2366694423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 msgr2=0x557d0bffd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.693+0000 7f0a7d144640 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bffd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.693+0000 7f0a7d144640 1 -- 172.21.15.49:0/2366694423 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d0bc64210 con 0x557d0bff3cc0 2024-03-18T00:57:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.694+0000 7f0a7d144640 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff3cc0 0x557d0bffd6c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0a64005c40 tx=0x7f0a640a3b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22152a62aed19d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.694+0000 7f0a617fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:10.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:10 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:57:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.702+0000 7f0a617fa640 1 -- 172.21.15.49:0/2366694423 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6401f070 con 0x557d0bff3cc0 2024-03-18T00:57:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.702+0000 7f0a617fa640 1 -- 172.21.15.49:0/2366694423 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0a6409f280 con 0x557d0bff3cc0 2024-03-18T00:57:10.704 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.703+0000 7f0a82791740 1 -- 172.21.15.49:0/2366694423 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a6409b470 con 0x557d0bff3cc0 2024-03-18T00:57:10.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.708+0000 7f0a617fa640 1 -- 172.21.15.49:0/2366694423 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0a640bb0b0 con 0x557d0bff3cc0 2024-03-18T00:57:10.710 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.708+0000 7f0a82791740 10 client.17145 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:10.710 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.708+0000 7f0a82791740 1 -- 172.21.15.49:0/2366694423 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557d0bd30620 con 0x557d0bff3cc0 2024-03-18T00:57:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.713+0000 7f0a617fa640 1 -- 172.21.15.49:0/2366694423 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a6409a8c0 con 0x557d0bff3cc0 2024-03-18T00:57:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.713+0000 7f0a617fa640 1 -- 172.21.15.49:0/2366694423 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0a6400f040 con 0x557d0bff3cc0 2024-03-18T00:57:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.713+0000 7f0a82791740 10 client.17145 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.713+0000 7f0a82791740 10 client.17145 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.713+0000 7f0a82791740 20 client.17145 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.713+0000 7f0a82791740 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557d0c008fd0 0x557d0c00b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.714+0000 7f0a82791740 4 client.17145 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:10.716 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.714+0000 7f0a82791740 1 -- 172.21.15.49:0/2366694423 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0a6409b470 con 0x557d0c008fd0 2024-03-18T00:57:10.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.717+0000 7f0a7d945640 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557d0c008fd0 0x557d0c00b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.745+0000 7f0a7d945640 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557d0c008fd0 0x557d0c00b3c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff28495d5f33e564 server_cookie=f9fda2440d5d8638 in_seq=0 out_seq=0 2024-03-18T00:57:10.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.745+0000 7f0a617fa640 10 client.17145 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.754+0000 7f0a617fa640 1 -- 172.21.15.49:0/2366694423 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f0a6409b470 con 0x557d0c008fd0 2024-03-18T00:57:10.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.755+0000 7f0a617fa640 10 client.17145 handle_command_reply: tid=0 2024-03-18T00:57:10.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.755+0000 7f0a82791740 1 client.17145 shutdown 2024-03-18T00:57:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.758+0000 7f0a82791740 1 -- 172.21.15.49:0/2366694423 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff3cc0 msgr2=0x557d0bffd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:10.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.758+0000 7f0a82791740 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff3cc0 0x557d0bffd6c0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0a64005c40 tx=0x7f0a640a3b60 comp rx=0 tx=0).stop 2024-03-18T00:57:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.763+0000 7f0a82791740 1 -- 172.21.15.49:0/2366694423 shutdown_connections 2024-03-18T00:57:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.763+0000 7f0a82791740 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557d0c008fd0 0x557d0c00b3c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.763+0000 7f0a617fa640 0 client.17145 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.763+0000 7f0a82791740 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0c0026f0 0x557d0c004ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.763+0000 7f0a82791740 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0bff3cc0 0x557d0bffd6c0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.763+0000 7f0a82791740 1 --2- 172.21.15.49:0/2366694423 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bffd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.764+0000 7f0a82791740 1 -- 172.21.15.49:0/2366694423 >> 172.21.15.49:0/2366694423 conn(0x557d0bfe59c0 msgr2=0x557d0bff2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.764+0000 7f0a82791740 1 -- 172.21.15.49:0/2366694423 shutdown_connections 2024-03-18T00:57:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.764+0000 7f0a82791740 1 -- 172.21.15.49:0/2366694423 wait complete. 2024-03-18T00:57:10.765 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.764+0000 7f0a82791740 20 client.17145 trim_cache size 0 max 16384 2024-03-18T00:57:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.812+0000 7f0a82791740 1 Processor -- start 2024-03-18T00:57:10.813 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.812+0000 7f0a82791740 1 -- start start 2024-03-18T00:57:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.813+0000 7f0a82791740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bfe43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.813+0000 7f0a82791740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0bfe4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.814 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.813+0000 7f0a82791740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0bfe4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.814+0000 7f0a82791740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781a6ab0 con 0x557d0c0026f0 2024-03-18T00:57:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.814+0000 7f0a82791740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781a85f0 con 0x557d0bff32d0 2024-03-18T00:57:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.814+0000 7f0a82791740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781a7180 con 0x557d0bff3cc0 2024-03-18T00:57:10.815 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.814+0000 7f0a7d144640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0bfe4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.816 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.815+0000 7f0a7e146640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0bfe4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.816 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.815+0000 7f0a7e146640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0bfe4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54882/0 (socket says 172.21.15.49:54882) 2024-03-18T00:57:10.816 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.815+0000 7f0a7e146640 1 -- 172.21.15.49:0/597773445 learned_addr learned my addr 172.21.15.49:0/597773445 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.816+0000 7f0a7d945640 1 --2- 172.21.15.49:0/597773445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bfe43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.816+0000 7f0a7e146640 1 -- 172.21.15.49:0/597773445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 msgr2=0x557d0bfe4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.816+0000 7f0a7e146640 1 --2- 172.21.15.49:0/597773445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0bfe4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.816+0000 7f0a7e146640 1 -- 172.21.15.49:0/597773445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 msgr2=0x557d0bfe43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:10.817 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.816+0000 7f0a7e146640 1 --2- 172.21.15.49:0/597773445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bfe43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.816+0000 7f0a7e146640 1 -- 172.21.15.49:0/597773445 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d0bfe9000 con 0x557d0c0026f0 2024-03-18T00:57:10.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.817+0000 7f0a7e146640 1 --2- 172.21.15.49:0/597773445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0bfe4e50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009900 tx=0x7f0a70005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcef7cd42a1da1eb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:10.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.818+0000 7f0a637fe640 1 -- 172.21.15.49:0/597773445 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70045040 con 0x557d0c0026f0 2024-03-18T00:57:10.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.818+0000 7f0a637fe640 1 -- 172.21.15.49:0/597773445 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a70050070 con 0x557d0c0026f0 2024-03-18T00:57:10.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.818+0000 7f0a637fe640 1 -- 172.21.15.49:0/597773445 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70004dd0 con 0x557d0c0026f0 2024-03-18T00:57:10.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.823+0000 7f0a82791740 1 -- 172.21.15.49:0/597773445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 msgr2=0x557d0bfe4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:10.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.823+0000 7f0a82791740 1 --2- 172.21.15.49:0/597773445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0bfe4e50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0a70009900 tx=0x7f0a70005fd0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.825+0000 7f0a82791740 1 -- 172.21.15.49:0/597773445 shutdown_connections 2024-03-18T00:57:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.825+0000 7f0a82791740 1 --2- 172.21.15.49:0/597773445 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0bfe4e50 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.825+0000 7f0a82791740 1 --2- 172.21.15.49:0/597773445 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0bfe4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.825+0000 7f0a82791740 1 --2- 172.21.15.49:0/597773445 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0bfe43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.826+0000 7f0a82791740 1 -- 172.21.15.49:0/597773445 >> 172.21.15.49:0/597773445 conn(0x557d0bfe59c0 msgr2=0x557d0bff4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:10.833 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.832+0000 7f0a82791740 1 -- 172.21.15.49:0/597773445 shutdown_connections 2024-03-18T00:57:10.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.833+0000 7f0a82791740 1 -- 172.21.15.49:0/597773445 wait complete. 2024-03-18T00:57:10.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.834+0000 7f0a82791740 1 -- start start 2024-03-18T00:57:10.834 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.834+0000 7f0a82791740 1 Processor -- start 2024-03-18T00:57:10.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.834+0000 7f0a82791740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0c01a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.835+0000 7f0a82791740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0c01a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.835+0000 7f0a82791740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0c01b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.835+0000 7f0a82791740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781a7f20 con 0x557d0c0026f0 2024-03-18T00:57:10.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.835+0000 7f0a82791740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781a7850 con 0x557d0bff32d0 2024-03-18T00:57:10.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.835+0000 7f0a82791740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781a4a70 con 0x557d0bff3cc0 2024-03-18T00:57:10.836 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.835+0000 7f0a82791740 10 client.0 _ll_register_callbacks cb 0x7f0a64005810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:10.838 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.837+0000 7f0a7d144640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0c01a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.838 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.837+0000 7f0a7e146640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0c01b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.838 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.837+0000 7f0a7d945640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0c01a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.838 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.837+0000 7f0a7e146640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0c01b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:54890/0 (socket says 172.21.15.49:54890) 2024-03-18T00:57:10.838 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.837+0000 7f0a7e146640 1 -- 172.21.15.49:0/3357231857 learned_addr learned my addr 172.21.15.49:0/3357231857 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.842+0000 7f0a7d945640 1 -- 172.21.15.49:0/3357231857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 msgr2=0x557d0c01a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.842+0000 7f0a7d945640 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0c01a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.842+0000 7f0a7d945640 1 -- 172.21.15.49:0/3357231857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 msgr2=0x557d0c01b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.842+0000 7f0a7d945640 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0c01b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:10.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.842+0000 7f0a7d945640 1 -- 172.21.15.49:0/3357231857 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d0c01bbe0 con 0x557d0bff32d0 2024-03-18T00:57:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.844+0000 7f0a7d945640 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0c01a0a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x557d0c00bac0 tx=0x7f0a6c01bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e953ee80e552b5c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:10.847 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.846+0000 7f0a617fa640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:57:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.847+0000 7f0a617fa640 1 -- 172.21.15.49:0/3357231857 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c01b6c0 con 0x557d0bff32d0 2024-03-18T00:57:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.847+0000 7f0a82791740 1 -- 172.21.15.49:0/3357231857 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a70050070 con 0x557d0bff32d0 2024-03-18T00:57:10.848 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.847+0000 7f0a617fa640 1 -- 172.21.15.49:0/3357231857 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0a6c012920 con 0x557d0bff32d0 2024-03-18T00:57:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.862+0000 7f0a617fa640 1 -- 172.21.15.49:0/3357231857 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f0a6c02a050 con 0x557d0bff32d0 2024-03-18T00:57:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.863+0000 7f0a82791740 10 client.26348 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:10.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.863+0000 7f0a82791740 1 -- 172.21.15.49:0/3357231857 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557d0c080200 con 0x557d0bff32d0 2024-03-18T00:57:10.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.866+0000 7f0a617fa640 1 -- 172.21.15.49:0/3357231857 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a70050070 con 0x557d0bff32d0 2024-03-18T00:57:10.870 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.868+0000 7f0a617fa640 1 -- 172.21.15.49:0/3357231857 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f0a6c0047a0 con 0x557d0bff32d0 2024-03-18T00:57:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.868+0000 7f0a82791740 10 client.26348 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.868+0000 7f0a82791740 10 client.26348 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.868+0000 7f0a82791740 20 client.26348 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.868+0000 7f0a82791740 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557d0c008fd0 0x557d0c082be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.868+0000 7f0a82791740 4 client.26348 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:57:10.871 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.868+0000 7f0a82791740 1 -- 172.21.15.49:0/3357231857 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557d0bf5cf80 con 0x557d0c008fd0 2024-03-18T00:57:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.878+0000 7f0a7d144640 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557d0c008fd0 0x557d0c082be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:10.881 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.881+0000 7f0a7d144640 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557d0c008fd0 0x557d0c082be0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=285a8f02015c0945 server_cookie=e0776631fbd29730 in_seq=0 out_seq=0 2024-03-18T00:57:10.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:10.881+0000 7f0a617fa640 10 client.26348 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:11.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:11 smithi049 ceph-mon[32130]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 900 MiB/s wr, 1.99k op/s 2024-03-18T00:57:11.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:11 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:57:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:11 smithi107 ceph-mon[36270]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 900 MiB/s wr, 1.99k op/s 2024-03-18T00:57:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:11 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:57:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:11 smithi106 ceph-mon[35899]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 900 MiB/s wr, 1.99k op/s 2024-03-18T00:57:11.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:11 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:57:13.626 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:13 smithi049 ceph-mon[32130]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 976 MiB/s wr, 2.12k op/s 2024-03-18T00:57:13.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:13 smithi107 ceph-mon[36270]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 976 MiB/s wr, 2.12k op/s 2024-03-18T00:57:13.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:13 smithi106 ceph-mon[35899]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 976 MiB/s wr, 2.12k op/s 2024-03-18T00:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a617fa640 1 -- 172.21.15.49:0/3357231857 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557d0bf5cf80 con 0x557d0c008fd0 2024-03-18T00:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a617fa640 10 client.26348 handle_command_reply: tid=0 2024-03-18T00:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 client.26348 shutdown 2024-03-18T00:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 -- 172.21.15.49:0/3357231857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 msgr2=0x557d0c01a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0c01a0a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x557d0c00bac0 tx=0x7f0a6c01bee0 comp rx=0 tx=0).stop 2024-03-18T00:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 -- 172.21.15.49:0/3357231857 shutdown_connections 2024-03-18T00:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557d0c008fd0 0x557d0c082be0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557d0c0026f0 0x557d0c01b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:14.367 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557d0bff3cc0 0x557d0c01a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a617fa640 0 client.26348 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 --2- 172.21.15.49:0/3357231857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557d0bff32d0 0x557d0c01a0a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 -- 172.21.15.49:0/3357231857 >> 172.21.15.49:0/3357231857 conn(0x557d0bfe59c0 msgr2=0x557d0bfe43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.367+0000 7f0a82791740 1 -- 172.21.15.49:0/3357231857 shutdown_connections 2024-03-18T00:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.368+0000 7f0a82791740 1 -- 172.21.15.49:0/3357231857 wait complete. 2024-03-18T00:57:14.368 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.368+0000 7f0a82791740 20 client.26348 trim_cache size 0 max 16384 2024-03-18T00:57:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:57:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:57:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:57:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:57:14.369 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a680a29c0 msgr2=0x7f0a680a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a680a29c0 0x7f0a680a4e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0a70036040 tx=0x7f0a7002bc30 comp rx=0 tx=0).stop 2024-03-18T00:57:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a78102840 msgr2=0x7f0a78197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a78102840 0x7f0a78197ea0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f0a64005fd0 tx=0x7f0a6400ee80 comp rx=0 tx=0).stop 2024-03-18T00:57:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 shutdown_connections 2024-03-18T00:57:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0a680a29c0 0x7f0a680a4e80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0a78103200 0x7f0a7819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0a78102840 0x7f0a78197ea0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 --2- 172.21.15.49:0/646553680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0a78101650 0x7f0a78197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:14.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.369+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 >> 172.21.15.49:0/646553680 conn(0x7f0a780fcbc0 msgr2=0x7f0a78105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.372+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 shutdown_connections 2024-03-18T00:57:14.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:14.372+0000 7f0a7e947640 1 -- 172.21.15.49:0/646553680 wait complete. 2024-03-18T00:57:15.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:15 smithi049 ceph-mon[32130]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 894 MiB/s wr, 2.04k op/s 2024-03-18T00:57:15.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:15 smithi107 ceph-mon[36270]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 894 MiB/s wr, 2.04k op/s 2024-03-18T00:57:15.698 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:15 smithi106 ceph-mon[35899]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 894 MiB/s wr, 2.04k op/s 2024-03-18T00:57:17.628 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:17 smithi049 ceph-mon[32130]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 888 MiB/s wr, 1.98k op/s 2024-03-18T00:57:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:17 smithi107 ceph-mon[36270]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 888 MiB/s wr, 1.98k op/s 2024-03-18T00:57:17.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:17 smithi106 ceph-mon[35899]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 888 MiB/s wr, 1.98k op/s 2024-03-18T00:57:19.630 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:19 smithi049 ceph-mon[32130]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 900 MiB/s wr, 1.88k op/s 2024-03-18T00:57:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:19 smithi107 ceph-mon[36270]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 900 MiB/s wr, 1.88k op/s 2024-03-18T00:57:19.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:19 smithi106 ceph-mon[35899]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 900 MiB/s wr, 1.88k op/s 2024-03-18T00:57:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:21 smithi107 ceph-mon[36270]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 900 MiB/s wr, 1.80k op/s 2024-03-18T00:57:21.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:21 smithi106 ceph-mon[35899]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 900 MiB/s wr, 1.80k op/s 2024-03-18T00:57:21.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:21 smithi049 ceph-mon[32130]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 900 MiB/s wr, 1.80k op/s 2024-03-18T00:57:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:23 smithi107 ceph-mon[36270]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 982 MiB/s wr, 1.91k op/s 2024-03-18T00:57:23.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:23 smithi106 ceph-mon[35899]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 982 MiB/s wr, 1.91k op/s 2024-03-18T00:57:23.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:23 smithi049 ceph-mon[32130]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 982 MiB/s wr, 1.91k op/s 2024-03-18T00:57:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:25 smithi107 ceph-mon[36270]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 901 MiB/s wr, 1.56k op/s 2024-03-18T00:57:25.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:25 smithi106 ceph-mon[35899]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 901 MiB/s wr, 1.56k op/s 2024-03-18T00:57:25.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:25 smithi049 ceph-mon[32130]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 901 MiB/s wr, 1.56k op/s 2024-03-18T00:57:27.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:27 smithi107 ceph-mon[36270]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 890 MiB/s wr, 1.39k op/s 2024-03-18T00:57:27.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:27 smithi106 ceph-mon[35899]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 890 MiB/s wr, 1.39k op/s 2024-03-18T00:57:27.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.709+0000 7f2ad5de2640 1 -- 172.21.15.49:0/3859448367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ad0072a90 msgr2=0x7f2ad0072eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:27.709 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.709+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/3859448367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ad0072a90 0x7f2ad0072eb0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009860 tx=0x7f2ac402bb90 comp rx=0 tx=0).stop 2024-03-18T00:57:27.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.710+0000 7f2ad5de2640 1 -- 172.21.15.49:0/3859448367 shutdown_connections 2024-03-18T00:57:27.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.710+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/3859448367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ad00733f0 0x7f2ad0108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.712 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.710+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/3859448367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ad01094d0 0x7f2ad010e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.711+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/3859448367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ad0072a90 0x7f2ad0072eb0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.713 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.711+0000 7f2ad5de2640 1 -- 172.21.15.49:0/3859448367 >> 172.21.15.49:0/3859448367 conn(0x7f2ad006e650 msgr2=0x7f2ad0070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:27.715 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.715+0000 7f2ad5de2640 1 -- 172.21.15.49:0/3859448367 shutdown_connections 2024-03-18T00:57:27.717 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.717+0000 7f2ad5de2640 1 -- 172.21.15.49:0/3859448367 wait complete. 2024-03-18T00:57:27.718 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.719+0000 7f2ad5de2640 1 Processor -- start 2024-03-18T00:57:27.719 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.719+0000 7f2ad5de2640 1 -- start start 2024-03-18T00:57:27.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.720+0000 7f2ad5de2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ad0072a90 0x7f2ad01a67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.720 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.720+0000 7f2ad5de2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ad00733f0 0x7f2ad01a6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.721+0000 7f2ad5de2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ad01094d0 0x7f2ad01add70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.721+0000 7f2ad5de2640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad011c110 con 0x7f2ad0072a90 2024-03-18T00:57:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.721+0000 7f2ad5de2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad011bf90 con 0x7f2ad00733f0 2024-03-18T00:57:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.721+0000 7f2ad5de2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad011c290 con 0x7f2ad01094d0 2024-03-18T00:57:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.723+0000 7f2ad55e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ad01094d0 0x7f2ad01add70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.723+0000 7f2acffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ad00733f0 0x7f2ad01a6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.724+0000 7f2acffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ad00733f0 0x7f2ad01a6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:44378/0 (socket says 172.21.15.49:44378) 2024-03-18T00:57:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.724+0000 7f2ad55e1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ad01094d0 0x7f2ad01add70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:37946/0 (socket says 172.21.15.49:37946) 2024-03-18T00:57:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.724+0000 7f2ad55e1640 1 -- 172.21.15.49:0/1712161380 learned_addr learned my addr 172.21.15.49:0/1712161380 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:27.724 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.724+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ad0072a90 0x7f2ad01a67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.726+0000 7f2ad55e1640 1 -- 172.21.15.49:0/1712161380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ad00733f0 msgr2=0x7f2ad01a6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.726+0000 7f2ad55e1640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ad00733f0 0x7f2ad01a6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.726+0000 7f2ad55e1640 1 -- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ad0072a90 msgr2=0x7f2ad01a67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.726+0000 7f2ad55e1640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ad0072a90 0x7f2ad01a67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.726 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.726+0000 7f2ad55e1640 1 -- 172.21.15.49:0/1712161380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad01ae3e0 con 0x7f2ad01094d0 2024-03-18T00:57:27.728 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.727+0000 7f2ad55e1640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ad01094d0 0x7f2ad01add70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009fd0 tx=0x7f2ac800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.729+0000 7f2acdffb640 1 -- 172.21.15.49:0/1712161380 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8004b20 con 0x7f2ad01094d0 2024-03-18T00:57:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.729+0000 7f2acdffb640 1 -- 172.21.15.49:0/1712161380 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac8003e50 con 0x7f2ad01094d0 2024-03-18T00:57:27.729 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.729+0000 7f2ad5de2640 1 -- 172.21.15.49:0/1712161380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad01ae6d0 con 0x7f2ad01094d0 2024-03-18T00:57:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.730+0000 7f2ad5de2640 1 -- 172.21.15.49:0/1712161380 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad01aec20 con 0x7f2ad01094d0 2024-03-18T00:57:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.730+0000 7f2acdffb640 1 -- 172.21.15.49:0/1712161380 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac800b630 con 0x7f2ad01094d0 2024-03-18T00:57:27.738 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.737+0000 7f2acdffb640 1 -- 172.21.15.49:0/1712161380 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2ac801d020 con 0x7f2ad01094d0 2024-03-18T00:57:27.742 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.741+0000 7f2ad9c2f740 1 Processor -- start 2024-03-18T00:57:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.742+0000 7f2acdffb640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ab80a2930 0x7f2ab80a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.742+0000 7f2ad9c2f740 1 -- start start 2024-03-18T00:57:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.742+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc2529500 0x55adc2529920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.743+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc252e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.743+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 0x55adc25314f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.743+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55adc248ea40 con 0x55adc252f060 2024-03-18T00:57:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.743+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55adc2437920 con 0x55adc2529500 2024-03-18T00:57:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.743+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55adc248aae0 con 0x55adc2529ef0 2024-03-18T00:57:27.743 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.743+0000 7f2acdffb640 1 -- 172.21.15.49:0/1712161380 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2ac80cf9f0 con 0x7f2ad01094d0 2024-03-18T00:57:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.743+0000 7f2ad55e1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 0x55adc25314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.743+0000 7f2ad55e1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 0x55adc25314f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38432/0 (socket says 172.21.15.49:38432) 2024-03-18T00:57:27.744 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.743+0000 7f2ad55e1640 1 -- 172.21.15.49:0/1753157169 learned_addr learned my addr 172.21.15.49:0/1753157169 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.744+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/1753157169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc2529500 0x55adc2529920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.745+0000 7f2ad55e1640 1 -- 172.21.15.49:0/1753157169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 msgr2=0x55adc252e9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.745+0000 7f2ad55e1640 1 --2- 172.21.15.49:0/1753157169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc252e9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.745+0000 7f2ad55e1640 1 -- 172.21.15.49:0/1753157169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc2529500 msgr2=0x55adc2529920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.745+0000 7f2ad55e1640 1 --2- 172.21.15.49:0/1753157169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc2529500 0x55adc2529920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.745+0000 7f2ad55e1640 1 -- 172.21.15.49:0/1753157169 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adc244f460 con 0x55adc252f060 2024-03-18T00:57:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.746+0000 7f2acffff640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ab80a2930 0x7f2ab80a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.746+0000 7f2ad55e1640 1 --2- 172.21.15.49:0/1753157169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 0x55adc25314f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800cd00 tx=0x7f2ac809c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c6bae845197a66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.747+0000 7f2acffff640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ab80a2930 0x7f2ab80a4df0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55adc252a2f0 tx=0x7f2ac000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.751+0000 7f2ab6ffd640 1 -- 172.21.15.49:0/1753157169 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac801a040 con 0x55adc252f060 2024-03-18T00:57:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.751+0000 7f2ab6ffd640 1 -- 172.21.15.49:0/1753157169 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac80a4ab0 con 0x55adc252f060 2024-03-18T00:57:27.751 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.752+0000 7f2ab6ffd640 1 -- 172.21.15.49:0/1753157169 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8098980 con 0x55adc252f060 2024-03-18T00:57:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.753+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1753157169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 msgr2=0x55adc25314f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.753+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1753157169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 0x55adc25314f0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f2ac800cd00 tx=0x7f2ac809c210 comp rx=0 tx=0).stop 2024-03-18T00:57:27.753 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.754+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1753157169 shutdown_connections 2024-03-18T00:57:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.754+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1753157169 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 0x55adc25314f0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.754+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1753157169 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc252e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.754+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1753157169 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc2529500 0x55adc2529920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.754+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1753157169 >> 172.21.15.49:0/1753157169 conn(0x55adc2524a70 msgr2=0x55adc2524e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:27.754 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.754+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1753157169 shutdown_connections 2024-03-18T00:57:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.755+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1753157169 wait complete. 2024-03-18T00:57:27.755 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.756+0000 7f2ad9c2f740 1 -- start start 2024-03-18T00:57:27.756 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.756+0000 7f2ad9c2f740 1 Processor -- start 2024-03-18T00:57:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.757+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc25446e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.757+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc2544c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.757+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc252f060 0x55adc254bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.757+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55adc2542ea0 con 0x55adc2529500 2024-03-18T00:57:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.757+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55adc244ad70 con 0x55adc252f060 2024-03-18T00:57:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.757+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55adc2543020 con 0x55adc2529ef0 2024-03-18T00:57:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.757+0000 7f2ad9c2f740 10 client.0 _ll_register_callbacks cb 0x55adc21a2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:27.757 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.757+0000 7f2ad4de0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc25446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.758+0000 7f2ad4de0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc25446e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38434/0 (socket says 172.21.15.49:38434) 2024-03-18T00:57:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.758+0000 7f2ad4de0640 1 -- 172.21.15.49:0/1589845330 learned_addr learned my addr 172.21.15.49:0/1589845330 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.758+0000 7f2ad4de0640 1 -- 172.21.15.49:0/1589845330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 msgr2=0x55adc2544c20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.758+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc2544c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.758+0000 7f2ad4de0640 1 -- 172.21.15.49:0/1589845330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc252f060 msgr2=0x55adc254bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.758+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc252f060 0x55adc254bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.758 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.758+0000 7f2ad4de0640 1 -- 172.21.15.49:0/1589845330 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adc21a3210 con 0x55adc2529500 2024-03-18T00:57:27.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.759+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc25446e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4034b10 tx=0x7f2ac40094f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1811dc4c98e52df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:27.759 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.759+0000 7f2aaffff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:27.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.760+0000 7f2aaffff640 1 -- 172.21.15.49:0/1589845330 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4043070 con 0x55adc2529500 2024-03-18T00:57:27.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.760+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1589845330 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac80a4ab0 con 0x55adc2529500 2024-03-18T00:57:27.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.760+0000 7f2aaffff640 1 -- 172.21.15.49:0/1589845330 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2ac4004240 con 0x55adc2529500 2024-03-18T00:57:27.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.761+0000 7f2aaffff640 1 -- 172.21.15.49:0/1589845330 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2ac404f050 con 0x55adc2529500 2024-03-18T00:57:27.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.761+0000 7f2ad9c2f740 10 client.17163 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:27.760 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.761+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1589845330 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55adc226f620 con 0x55adc2529500 2024-03-18T00:57:27.762 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.762+0000 7f2aaffff640 1 -- 172.21.15.49:0/1589845330 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac80a4ab0 con 0x55adc2529500 2024-03-18T00:57:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.762+0000 7f2aaffff640 1 -- 172.21.15.49:0/1589845330 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2ac4003950 con 0x55adc2529500 2024-03-18T00:57:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.763+0000 7f2ad9c2f740 10 client.17163 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.763+0000 7f2ad9c2f740 10 client.17163 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.763+0000 7f2ad9c2f740 20 client.17163 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.763+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55adc2550a30 0x55adc2552ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.763 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.763+0000 7f2ad9c2f740 4 client.17163 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:27.764 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.763+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1589845330 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adc249bf80 con 0x55adc2550a30 2024-03-18T00:57:27.772 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.771+0000 7f2acffff640 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55adc2550a30 0x55adc2552ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.779+0000 7f2aaffff640 10 client.17163 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.779+0000 7f2acffff640 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55adc2550a30 0x55adc2552ef0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72141e84c5709eb5 server_cookie=98cf3f15b932e6a6 in_seq=0 out_seq=0 2024-03-18T00:57:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.787+0000 7f2aaffff640 1 -- 172.21.15.49:0/1589845330 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55adc249bf80 con 0x55adc2550a30 2024-03-18T00:57:27.788 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.787+0000 7f2aaffff640 10 client.17163 handle_command_reply: tid=0 2024-03-18T00:57:27.791 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.787+0000 7f2ad9c2f740 1 client.17163 shutdown 2024-03-18T00:57:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.788+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1589845330 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 msgr2=0x55adc25446e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.788+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc25446e0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4034b10 tx=0x7f2ac40094f0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.792+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1589845330 shutdown_connections 2024-03-18T00:57:27.792 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.792+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55adc2550a30 0x55adc2552ef0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.792+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc252f060 0x55adc254bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.792+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc2544c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.792+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1589845330 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc25446e0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.792+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1589845330 >> 172.21.15.49:0/1589845330 conn(0x55adc2524a70 msgr2=0x55adc2523c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.792+0000 7f2aaffff640 0 client.17163 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.792+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1589845330 shutdown_connections 2024-03-18T00:57:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.793+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1589845330 wait complete. 2024-03-18T00:57:27.793 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.793+0000 7f2ad9c2f740 20 client.17163 trim_cache size 0 max 16384 2024-03-18T00:57:27.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:27 smithi049 ceph-mon[32130]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 890 MiB/s wr, 1.39k op/s 2024-03-18T00:57:27.898 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.898+0000 7f2ad9c2f740 1 Processor -- start 2024-03-18T00:57:27.898 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.899+0000 7f2ad9c2f740 1 -- start start 2024-03-18T00:57:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.899+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529500 0x55adc2524e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.899+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc2529ef0 0x55adc2543d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.899+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 0x55adc2544240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.899+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc00e690 con 0x55adc252f060 2024-03-18T00:57:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.899+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc0101d0 con 0x55adc2529ef0 2024-03-18T00:57:27.899 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.899+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc00ed60 con 0x55adc2529500 2024-03-18T00:57:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.900+0000 7f2ad4de0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529500 0x55adc2524e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.900+0000 7f2ad4de0640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529500 0x55adc2524e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:37966/0 (socket says 172.21.15.49:37966) 2024-03-18T00:57:27.900 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.900+0000 7f2ad4de0640 1 -- 172.21.15.49:0/1975248772 learned_addr learned my addr 172.21.15.49:0/1975248772 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.902+0000 7f2ad4de0640 1 -- 172.21.15.49:0/1975248772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc2529ef0 msgr2=0x55adc2543d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.902+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/1975248772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc2529ef0 0x55adc2543d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.902+0000 7f2ad4de0640 1 -- 172.21.15.49:0/1975248772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 msgr2=0x55adc2544240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.902+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/1975248772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 0x55adc2544240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.902 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.902+0000 7f2ad4de0640 1 -- 172.21.15.49:0/1975248772 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adc252e430 con 0x55adc2529500 2024-03-18T00:57:27.904 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.904+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/1975248772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529500 0x55adc2524e50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4001e50 tx=0x7f2ac4034e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dea0e04c2d4b7077 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:27.911 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.910+0000 7f2ab67fc640 1 -- 172.21.15.49:0/1975248772 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4043040 con 0x55adc2529500 2024-03-18T00:57:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.910+0000 7f2ab67fc640 1 -- 172.21.15.49:0/1975248772 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac4058070 con 0x55adc2529500 2024-03-18T00:57:27.912 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.912+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1975248772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529500 msgr2=0x55adc2524e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:27.913 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.912+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1975248772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529500 0x55adc2524e50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4001e50 tx=0x7f2ac4034e40 comp rx=0 tx=0).stop 2024-03-18T00:57:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.913+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1975248772 shutdown_connections 2024-03-18T00:57:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.913+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1975248772 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc2529ef0 0x55adc2543d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.913+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1975248772 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc252f060 0x55adc2544240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.913+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/1975248772 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529500 0x55adc2524e50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.913+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1975248772 >> 172.21.15.49:0/1975248772 conn(0x55adc2524a70 msgr2=0x55adc252e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:27.914 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.913+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1975248772 shutdown_connections 2024-03-18T00:57:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.913+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/1975248772 wait complete. 2024-03-18T00:57:27.915 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.914+0000 7f2ad9c2f740 1 -- start start 2024-03-18T00:57:27.916 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.915+0000 7f2ad9c2f740 1 Processor -- start 2024-03-18T00:57:27.917 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.916+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc2549080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.917+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc25495c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.918 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.917+0000 7f2ad9c2f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc252f060 0x55adc254a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.917+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc00fb00 con 0x55adc2529500 2024-03-18T00:57:27.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.917+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc00f430 con 0x55adc252f060 2024-03-18T00:57:27.919 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.917+0000 7f2ad9c2f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2abc00c670 con 0x55adc2529ef0 2024-03-18T00:57:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.918+0000 7f2ad4de0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc2549080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.918+0000 7f2ad9c2f740 10 client.0 _ll_register_callbacks cb 0x7f2ac4034f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.919+0000 7f2acffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc25495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.919+0000 7f2ad4de0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc2549080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38442/0 (socket says 172.21.15.49:38442) 2024-03-18T00:57:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.919+0000 7f2ad4de0640 1 -- 172.21.15.49:0/534233368 learned_addr learned my addr 172.21.15.49:0/534233368 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.920+0000 7f2acffff640 1 -- 172.21.15.49:0/534233368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc252f060 msgr2=0x55adc254a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.920+0000 7f2acffff640 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc252f060 0x55adc254a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.920+0000 7f2acffff640 1 -- 172.21.15.49:0/534233368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 msgr2=0x55adc2549080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:27.920 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.920+0000 7f2acffff640 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc2549080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.920+0000 7f2acffff640 1 -- 172.21.15.49:0/534233368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adc254ab20 con 0x55adc2529ef0 2024-03-18T00:57:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.921+0000 7f2acffff640 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc25495c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x55adc2545bc0 tx=0x7f2ac00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3debf72da5da3e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:27.921 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.921+0000 7f2aaffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:57:27.922 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.922+0000 7f2aaffff640 1 -- 172.21.15.49:0/534233368 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac001e420 con 0x55adc2529ef0 2024-03-18T00:57:27.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.923+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/534233368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2ac4058070 con 0x55adc2529ef0 2024-03-18T00:57:27.923 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.923+0000 7f2aaffff640 1 -- 172.21.15.49:0/534233368 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2ac001e5c0 con 0x55adc2529ef0 2024-03-18T00:57:27.927 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.928+0000 7f2aaffff640 1 -- 172.21.15.49:0/534233368 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2ac0035050 con 0x55adc2529ef0 2024-03-18T00:57:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.928+0000 7f2ad9c2f740 10 client.26449 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:27.928 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.928+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/534233368 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55adc25b6e60 con 0x55adc2529ef0 2024-03-18T00:57:27.932 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.932+0000 7f2aaffff640 1 -- 172.21.15.49:0/534233368 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ac4058070 con 0x55adc2529ef0 2024-03-18T00:57:27.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.934+0000 7f2aaffff640 1 -- 172.21.15.49:0/534233368 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2ac002e8f0 con 0x55adc2529ef0 2024-03-18T00:57:27.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.934+0000 7f2ad9c2f740 10 client.26449 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:27.934 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.935+0000 7f2ad9c2f740 10 client.26449 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.935+0000 7f2ad9c2f740 20 client.26449 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.935+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55adc25b77e0 0x55adc25b9c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.935+0000 7f2ad9c2f740 4 client.26449 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:57:27.935 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.935+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/534233368 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55adc24962d0 con 0x55adc25b77e0 2024-03-18T00:57:27.943 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.942+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55adc25b77e0 0x55adc25b9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.952+0000 7f2ad4de0640 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55adc25b77e0 0x55adc25b9c00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=895cc3ca2d034ed5 server_cookie=eaa4395b5cd629c5 in_seq=0 out_seq=0 2024-03-18T00:57:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:27.952+0000 7f2aaffff640 10 client.26449 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:29 smithi107 ceph-mon[36270]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 907 MiB/s wr, 1.46k op/s 2024-03-18T00:57:29.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:29 smithi106 ceph-mon[35899]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 907 MiB/s wr, 1.46k op/s 2024-03-18T00:57:29.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:29 smithi049 ceph-mon[32130]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 907 MiB/s wr, 1.46k op/s 2024-03-18T00:57:31.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:31 smithi106 ceph-mon[35899]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 877 MiB/s wr, 1.46k op/s 2024-03-18T00:57:31.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:31 smithi049 ceph-mon[32130]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 877 MiB/s wr, 1.46k op/s 2024-03-18T00:57:31.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:31 smithi107 ceph-mon[36270]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 877 MiB/s wr, 1.46k op/s 2024-03-18T00:57:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.185+0000 7f2aaffff640 1 -- 172.21.15.49:0/534233368 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55adc24962d0 con 0x55adc25b77e0 2024-03-18T00:57:33.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.185+0000 7f2aaffff640 10 client.26449 handle_command_reply: tid=0 2024-03-18T00:57:33.186 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.185+0000 7f2ad9c2f740 1 client.26449 shutdown 2024-03-18T00:57:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.189+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/534233368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 msgr2=0x55adc25495c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:33.189 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.189+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc25495c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x55adc2545bc0 tx=0x7f2ac00046c0 comp rx=0 tx=0).stop 2024-03-18T00:57:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.190+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/534233368 shutdown_connections 2024-03-18T00:57:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.190+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55adc25b77e0 0x55adc25b9c00 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.190+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55adc252f060 0x55adc254a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.190+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55adc2529ef0 0x55adc25495c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.190+0000 7f2ad9c2f740 1 --2- 172.21.15.49:0/534233368 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55adc2529500 0x55adc2549080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.190+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/534233368 >> 172.21.15.49:0/534233368 conn(0x55adc2524a70 msgr2=0x55adc2527cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.190+0000 7f2aaffff640 0 client.26449 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.190+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/534233368 shutdown_connections 2024-03-18T00:57:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.191+0000 7f2ad9c2f740 1 -- 172.21.15.49:0/534233368 wait complete. 2024-03-18T00:57:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.191+0000 7f2ad9c2f740 20 client.26449 trim_cache size 0 max 16384 2024-03-18T00:57:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:57:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:57:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:57:33.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:57:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.198+0000 7f2ad5de2640 1 -- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ab80a2930 msgr2=0x7f2ab80a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.198+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ab80a2930 0x7f2ab80a4df0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55adc252a2f0 tx=0x7f2ac000b390 comp rx=0 tx=0).stop 2024-03-18T00:57:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.198+0000 7f2ad5de2640 1 -- 172.21.15.49:0/1712161380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ad01094d0 msgr2=0x7f2ad01add70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.198+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ad01094d0 0x7f2ad01add70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009fd0 tx=0x7f2ac800c800 comp rx=0 tx=0).stop 2024-03-18T00:57:33.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.200+0000 7f2ad5de2640 1 -- 172.21.15.49:0/1712161380 shutdown_connections 2024-03-18T00:57:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.200+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2ab80a2930 0x7f2ab80a4df0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.200+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2ad00733f0 0x7f2ad01a6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.201+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2ad01094d0 0x7f2ad01add70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.201+0000 7f2ad5de2640 1 --2- 172.21.15.49:0/1712161380 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2ad0072a90 0x7f2ad01a67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.201+0000 7f2ad5de2640 1 -- 172.21.15.49:0/1712161380 >> 172.21.15.49:0/1712161380 conn(0x7f2ad006e650 msgr2=0x7f2ad0070340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:33.203 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.203+0000 7f2ad5de2640 1 -- 172.21.15.49:0/1712161380 shutdown_connections 2024-03-18T00:57:33.203 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:33.203+0000 7f2ad5de2640 1 -- 172.21.15.49:0/1712161380 wait complete. 2024-03-18T00:57:33.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:33 smithi106 ceph-mon[35899]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 989 MiB/s wr, 1.68k op/s 2024-03-18T00:57:33.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:33 smithi049 ceph-mon[32130]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 989 MiB/s wr, 1.68k op/s 2024-03-18T00:57:33.910 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:57:33.910 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:57:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:33 smithi107 ceph-mon[36270]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 989 MiB/s wr, 1.68k op/s 2024-03-18T00:57:34.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.368+0000 7febab812640 1 -- 172.21.15.49:0/755737942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba4078f20 msgr2=0x7feba407b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:34.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.368+0000 7febab812640 1 --2- 172.21.15.49:0/755737942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba4078f20 0x7feba407b3b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009900 tx=0x7feb9c02bc60 comp rx=0 tx=0).stop 2024-03-18T00:57:34.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.372+0000 7febab812640 1 -- 172.21.15.49:0/755737942 shutdown_connections 2024-03-18T00:57:34.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.372+0000 7febab812640 1 --2- 172.21.15.49:0/755737942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba4078f20 0x7feba407b3b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.372+0000 7febab812640 1 --2- 172.21.15.49:0/755737942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4076570 0x7feba40789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.372+0000 7febab812640 1 --2- 172.21.15.49:0/755737942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba4073690 0x7feba4073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.372+0000 7febab812640 1 -- 172.21.15.49:0/755737942 >> 172.21.15.49:0/755737942 conn(0x7feba406e640 msgr2=0x7feba4070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:34.376 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.377+0000 7febab812640 1 -- 172.21.15.49:0/755737942 shutdown_connections 2024-03-18T00:57:34.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.380+0000 7febab812640 1 -- 172.21.15.49:0/755737942 wait complete. 2024-03-18T00:57:34.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.381+0000 7febab812640 1 Processor -- start 2024-03-18T00:57:34.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.381+0000 7febab812640 1 -- start start 2024-03-18T00:57:34.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.382+0000 7febab812640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba4073690 0x7feba4080240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.382+0000 7febab812640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba4076570 0x7feba4080780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.382+0000 7febab812640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4078f20 0x7feba40837d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.382+0000 7febab812640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7feba412b750 con 0x7feba4078f20 2024-03-18T00:57:34.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.382+0000 7febab812640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7feba412b5d0 con 0x7feba4076570 2024-03-18T00:57:34.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.382+0000 7febab812640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feba412b8d0 con 0x7feba4073690 2024-03-18T00:57:34.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.382+0000 7feba8dad640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba4076570 0x7feba4080780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.383+0000 7feba8dad640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba4076570 0x7feba4080780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:44440/0 (socket says 172.21.15.49:44440) 2024-03-18T00:57:34.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.383+0000 7feba8dad640 1 -- 172.21.15.49:0/1821630483 learned_addr learned my addr 172.21.15.49:0/1821630483 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:34.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.384+0000 7feba9daf640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4078f20 0x7feba40837d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.385+0000 7feba95ae640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba4073690 0x7feba4080240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.385+0000 7feba8dad640 1 -- 172.21.15.49:0/1821630483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba4073690 msgr2=0x7feba4080240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:34.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.385+0000 7feba8dad640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba4073690 0x7feba4080240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.385+0000 7feba8dad640 1 -- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4078f20 msgr2=0x7feba40837d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:34.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.385+0000 7feba8dad640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4078f20 0x7feba40837d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.385 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.385+0000 7feba8dad640 1 -- 172.21.15.49:0/1821630483 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba4083e40 con 0x7feba4076570 2024-03-18T00:57:34.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.393+0000 7feba8dad640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba4076570 0x7feba4080780 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feba000ba50 tx=0x7feba000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:34.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.398+0000 7feb9a7fc640 1 -- 172.21.15.49:0/1821630483 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0019070 con 0x7feba4076570 2024-03-18T00:57:34.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.398+0000 7feb9a7fc640 1 -- 172.21.15.49:0/1821630483 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feba000cbe0 con 0x7feba4076570 2024-03-18T00:57:34.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.398+0000 7feb9a7fc640 1 -- 172.21.15.49:0/1821630483 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba00076e0 con 0x7feba4076570 2024-03-18T00:57:34.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.400+0000 7febab812640 1 -- 172.21.15.49:0/1821630483 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba4084130 con 0x7feba4076570 2024-03-18T00:57:34.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.400+0000 7febab812640 1 -- 172.21.15.49:0/1821630483 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba4084630 con 0x7feba4076570 2024-03-18T00:57:34.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.405+0000 7febae3fe740 1 Processor -- start 2024-03-18T00:57:34.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.406+0000 7febae3fe740 1 -- start start 2024-03-18T00:57:34.406 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.406+0000 7febae3fe740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 0x55b43e256590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.407+0000 7febae3fe740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256b60 0x55b43e25b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.407+0000 7febae3fe740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 0x55b43e25e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.407+0000 7febae3fe740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b43e1bb500 con 0x55b43e256b60 2024-03-18T00:57:34.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.407+0000 7febae3fe740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b43e163be0 con 0x55b43e25bcd0 2024-03-18T00:57:34.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.407+0000 7feba95ae640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256b60 0x55b43e25b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.407+0000 7feba95ae640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256b60 0x55b43e25b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38448/0 (socket says 172.21.15.49:38448) 2024-03-18T00:57:34.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.407+0000 7feba95ae640 1 -- 172.21.15.49:0/1400525409 learned_addr learned my addr 172.21.15.49:0/1400525409 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:34.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.407+0000 7febae3fe740 1 -- 172.21.15.49:0/1400525409 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b43e1b75a0 con 0x55b43e256170 2024-03-18T00:57:34.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.409+0000 7feba8dad640 1 --2- 172.21.15.49:0/1400525409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 0x55b43e25e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.410+0000 7feba9daf640 1 --2- 172.21.15.49:0/1400525409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 0x55b43e256590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.412+0000 7feba95ae640 1 -- 172.21.15.49:0/1400525409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 msgr2=0x55b43e256590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:34.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.412+0000 7feba95ae640 1 --2- 172.21.15.49:0/1400525409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 0x55b43e256590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.412+0000 7feba95ae640 1 -- 172.21.15.49:0/1400525409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 msgr2=0x55b43e25e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:34.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.412+0000 7feba95ae640 1 --2- 172.21.15.49:0/1400525409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 0x55b43e25e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.412 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.412+0000 7feba95ae640 1 -- 172.21.15.49:0/1400525409 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b43e1774b0 con 0x55b43e256b60 2024-03-18T00:57:34.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.415+0000 7feb9a7fc640 1 -- 172.21.15.49:0/1821630483 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7feba001c070 con 0x7feba4076570 2024-03-18T00:57:34.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.415+0000 7feba95ae640 1 --2- 172.21.15.49:0/1400525409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256b60 0x55b43e25b660 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7feb94009bc0 tx=0x7feb9400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cacbda72efcee94 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:34.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.416+0000 7feb9a7fc640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a2a00 0x7feb900a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.417 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.417+0000 7feb9a7fc640 1 -- 172.21.15.49:0/1821630483 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7feba00d14c0 con 0x7feba4076570 2024-03-18T00:57:34.418 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.419+0000 7feba9daf640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a2a00 0x7feb900a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.420+0000 7feb8f7fe640 1 -- 172.21.15.49:0/1400525409 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94016070 con 0x55b43e256b60 2024-03-18T00:57:34.420 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.420+0000 7feb8f7fe640 1 -- 172.21.15.49:0/1400525409 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb94005500 con 0x55b43e256b60 2024-03-18T00:57:34.423 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.422+0000 7febae3fe740 1 -- 172.21.15.49:0/1400525409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256b60 msgr2=0x55b43e25b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:34.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.422+0000 7febae3fe740 1 --2- 172.21.15.49:0/1400525409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256b60 0x55b43e25b660 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7feb94009bc0 tx=0x7feb9400e6c0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.424 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.424+0000 7feba9daf640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a2a00 0x7feb900a4ec0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009230 tx=0x7feb9c02c8b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:34.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.424+0000 7febae3fe740 1 -- 172.21.15.49:0/1400525409 shutdown_connections 2024-03-18T00:57:34.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.424+0000 7febae3fe740 1 --2- 172.21.15.49:0/1400525409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 0x55b43e25e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.424+0000 7febae3fe740 1 --2- 172.21.15.49:0/1400525409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256b60 0x55b43e25b660 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.424+0000 7febae3fe740 1 --2- 172.21.15.49:0/1400525409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 0x55b43e256590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.425 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.424+0000 7febae3fe740 1 -- 172.21.15.49:0/1400525409 >> 172.21.15.49:0/1400525409 conn(0x55b43e2516e0 msgr2=0x55b43e251ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:34.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.433+0000 7febae3fe740 1 -- 172.21.15.49:0/1400525409 shutdown_connections 2024-03-18T00:57:34.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.436+0000 7febae3fe740 1 -- 172.21.15.49:0/1400525409 wait complete. 2024-03-18T00:57:34.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.436+0000 7febae3fe740 1 -- start start 2024-03-18T00:57:34.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.436+0000 7febae3fe740 1 Processor -- start 2024-03-18T00:57:34.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7febae3fe740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e2606d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7febae3fe740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256b60 0x55b43e260c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7febae3fe740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 0x55b43e267be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7febae3fe740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b43e25ecd0 con 0x55b43e256170 2024-03-18T00:57:34.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7febae3fe740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b43e177160 con 0x55b43e25bcd0 2024-03-18T00:57:34.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7febae3fe740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b43e25ee50 con 0x55b43e256b60 2024-03-18T00:57:34.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7febae3fe740 10 client.0 _ll_register_callbacks cb 0x55b43e15b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:34.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7feba95ae640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e2606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7feba95ae640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e2606d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38450/0 (socket says 172.21.15.49:38450) 2024-03-18T00:57:34.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.440+0000 7feba95ae640 1 -- 172.21.15.49:0/3431930436 learned_addr learned my addr 172.21.15.49:0/3431930436 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:34.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.441+0000 7feba95ae640 1 -- 172.21.15.49:0/3431930436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256b60 msgr2=0x55b43e260c10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:34.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.441+0000 7feba95ae640 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256b60 0x55b43e260c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.441+0000 7feba9daf640 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 0x55b43e267be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.441+0000 7feba95ae640 1 -- 172.21.15.49:0/3431930436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 msgr2=0x55b43e267be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:34.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.441+0000 7feba95ae640 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 0x55b43e267be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.441+0000 7feba95ae640 1 -- 172.21.15.49:0/3431930436 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b43df997d0 con 0x55b43e256170 2024-03-18T00:57:34.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.441+0000 7feba9daf640 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 0x55b43e267be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:57:34.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.442+0000 7feba95ae640 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e2606d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7feb940058c0 tx=0x7feb94004850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ba4366a6564888 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:34.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.452+0000 7feb8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:34.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.452+0000 7feb8cff9640 1 -- 172.21.15.49:0/3431930436 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94028060 con 0x55b43e256170 2024-03-18T00:57:34.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.452+0000 7febae3fe740 1 -- 172.21.15.49:0/3431930436 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb94005500 con 0x55b43e256170 2024-03-18T00:57:34.452 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.452+0000 7feb8cff9640 1 -- 172.21.15.49:0/3431930436 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7feb94004eb0 con 0x55b43e256170 2024-03-18T00:57:34.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.458+0000 7feb8cff9640 1 -- 172.21.15.49:0/3431930436 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feb94016040 con 0x55b43e256170 2024-03-18T00:57:34.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.458+0000 7febae3fe740 10 client.17175 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:34.458 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.458+0000 7febae3fe740 1 -- 172.21.15.49:0/3431930436 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b43decf3f0 con 0x55b43e256170 2024-03-18T00:57:34.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.463+0000 7feb8cff9640 1 -- 172.21.15.49:0/3431930436 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb94005500 con 0x55b43e256170 2024-03-18T00:57:34.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.463+0000 7feb8cff9640 1 -- 172.21.15.49:0/3431930436 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7feb9400f040 con 0x55b43e256170 2024-03-18T00:57:34.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.463+0000 7febae3fe740 10 client.17175 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:34.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.463+0000 7febae3fe740 10 client.17175 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:34.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.463+0000 7febae3fe740 20 client.17175 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:34.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.463+0000 7febae3fe740 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b43e26c410 0x55b43e26e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.463+0000 7febae3fe740 4 client.17175 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:34.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.463+0000 7febae3fe740 1 -- 172.21.15.49:0/3431930436 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b43e1c8a40 con 0x55b43e26c410 2024-03-18T00:57:34.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.465+0000 7feba8dad640 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b43e26c410 0x55b43e26e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.471+0000 7feba8dad640 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b43e26c410 0x55b43e26e8b0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=311dd9d17917e0bd server_cookie=c8fcdbfafdac6635 in_seq=0 out_seq=0 2024-03-18T00:57:34.471 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.471+0000 7feb8cff9640 10 client.17175 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:34.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.475+0000 7feb8cff9640 1 -- 172.21.15.49:0/3431930436 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b43e1c8a40 con 0x55b43e26c410 2024-03-18T00:57:34.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.475+0000 7feb8cff9640 10 client.17175 handle_command_reply: tid=0 2024-03-18T00:57:34.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.475+0000 7febae3fe740 1 client.17175 shutdown 2024-03-18T00:57:34.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.476+0000 7febae3fe740 1 -- 172.21.15.49:0/3431930436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 msgr2=0x55b43e2606d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:34.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.476+0000 7febae3fe740 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e2606d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7feb940058c0 tx=0x7feb94004850 comp rx=0 tx=0).stop 2024-03-18T00:57:34.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.476+0000 7febae3fe740 1 -- 172.21.15.49:0/3431930436 shutdown_connections 2024-03-18T00:57:34.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.476+0000 7febae3fe740 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b43e26c410 0x55b43e26e8b0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.480 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.476+0000 7febae3fe740 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e25bcd0 0x55b43e267be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.476+0000 7febae3fe740 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256b60 0x55b43e260c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.476+0000 7febae3fe740 1 --2- 172.21.15.49:0/3431930436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e2606d0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.476+0000 7febae3fe740 1 -- 172.21.15.49:0/3431930436 >> 172.21.15.49:0/3431930436 conn(0x55b43e2516e0 msgr2=0x55b43e250900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:34.481 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.479+0000 7feb8cff9640 0 client.17175 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:34.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.481+0000 7febae3fe740 1 -- 172.21.15.49:0/3431930436 shutdown_connections 2024-03-18T00:57:34.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.481+0000 7febae3fe740 1 -- 172.21.15.49:0/3431930436 wait complete. 2024-03-18T00:57:34.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.483+0000 7febae3fe740 20 client.17175 trim_cache size 0 max 16384 2024-03-18T00:57:34.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.541+0000 7febae3fe740 1 Processor -- start 2024-03-18T00:57:34.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.541+0000 7febae3fe740 1 -- start start 2024-03-18T00:57:34.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.541+0000 7febae3fe740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e2516e0 0x55b43e250070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.541+0000 7febae3fe740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 0x55b43e267b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.541+0000 7febae3fe740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 0x55b43e2505b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.541+0000 7feba95ae640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e2516e0 0x55b43e250070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.542+0000 7feba95ae640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e2516e0 0x55b43e250070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38454/0 (socket says 172.21.15.49:38454) 2024-03-18T00:57:34.542 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.542+0000 7feba95ae640 1 -- 172.21.15.49:0/3388524959 learned_addr learned my addr 172.21.15.49:0/3388524959 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:34.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.542+0000 7febae3fe740 1 -- 172.21.15.49:0/3388524959 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7feba41c8d00 con 0x55b43e2516e0 2024-03-18T00:57:34.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.542+0000 7febae3fe740 1 -- 172.21.15.49:0/3388524959 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7feba41ca840 con 0x55b43e256b60 2024-03-18T00:57:34.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.542+0000 7febae3fe740 1 -- 172.21.15.49:0/3388524959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feba41c93d0 con 0x55b43e256170 2024-03-18T00:57:34.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.542+0000 7feba8dad640 1 --2- 172.21.15.49:0/3388524959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 0x55b43e267b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.542+0000 7feba9daf640 1 --2- 172.21.15.49:0/3388524959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 0x55b43e2505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.543+0000 7feba8dad640 1 -- 172.21.15.49:0/3388524959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 msgr2=0x55b43e2505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:34.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.543+0000 7feba8dad640 1 --2- 172.21.15.49:0/3388524959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 0x55b43e2505b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.543+0000 7feba8dad640 1 -- 172.21.15.49:0/3388524959 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e2516e0 msgr2=0x55b43e250070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:34.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.543+0000 7feba8dad640 1 --2- 172.21.15.49:0/3388524959 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e2516e0 0x55b43e250070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.543+0000 7feba8dad640 1 -- 172.21.15.49:0/3388524959 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b43de8f0f0 con 0x55b43e256170 2024-03-18T00:57:34.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.544+0000 7feba8dad640 1 --2- 172.21.15.49:0/3388524959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 0x55b43e267b00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feba0004a80 tx=0x7feba001c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9eaf6ea7a4931c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:34.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.544+0000 7feb8effd640 1 -- 172.21.15.49:0/3388524959 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba00ba070 con 0x55b43e256170 2024-03-18T00:57:34.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.545+0000 7feb8effd640 1 -- 172.21.15.49:0/3388524959 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feba00a1f10 con 0x55b43e256170 2024-03-18T00:57:34.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.546+0000 7febae3fe740 1 -- 172.21.15.49:0/3388524959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 msgr2=0x55b43e267b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:34.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.546+0000 7febae3fe740 1 --2- 172.21.15.49:0/3388524959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 0x55b43e267b00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7feba0004a80 tx=0x7feba001c000 comp rx=0 tx=0).stop 2024-03-18T00:57:34.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.553+0000 7febae3fe740 1 -- 172.21.15.49:0/3388524959 shutdown_connections 2024-03-18T00:57:34.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.553+0000 7febae3fe740 1 --2- 172.21.15.49:0/3388524959 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 0x55b43e2505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.553+0000 7febae3fe740 1 --2- 172.21.15.49:0/3388524959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e256170 0x55b43e267b00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.552 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.553+0000 7febae3fe740 1 --2- 172.21.15.49:0/3388524959 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e2516e0 0x55b43e250070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.553+0000 7febae3fe740 1 -- 172.21.15.49:0/3388524959 >> 172.21.15.49:0/3388524959 conn(0x55b43e257940 msgr2=0x55b43e258060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:34.555 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.555+0000 7febae3fe740 1 -- 172.21.15.49:0/3388524959 shutdown_connections 2024-03-18T00:57:34.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.556+0000 7febae3fe740 1 -- 172.21.15.49:0/3388524959 wait complete. 2024-03-18T00:57:34.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.556+0000 7febae3fe740 1 -- start start 2024-03-18T00:57:34.557 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.557+0000 7febae3fe740 1 Processor -- start 2024-03-18T00:57:34.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.559+0000 7febae3fe740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e2516e0 0x55b43e267960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.559+0000 7febae3fe740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e264450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.559+0000 7febae3fe740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 0x55b43e264990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.559+0000 7febae3fe740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7feba41ca170 con 0x55b43e256170 2024-03-18T00:57:34.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.559+0000 7febae3fe740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7feba41c9aa0 con 0x55b43e256b60 2024-03-18T00:57:34.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.559+0000 7febae3fe740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feba41c6ce0 con 0x55b43e2516e0 2024-03-18T00:57:34.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.559+0000 7febae3fe740 10 client.0 _ll_register_callbacks cb 0x7feb9c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:34.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.560+0000 7feba95ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e2516e0 0x55b43e267960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.561 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.560+0000 7feba95ae640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e2516e0 0x55b43e267960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38024/0 (socket says 172.21.15.49:38024) 2024-03-18T00:57:34.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.560+0000 7feba95ae640 1 -- 172.21.15.49:0/309862121 learned_addr learned my addr 172.21.15.49:0/309862121 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:34.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.560+0000 7feba9daf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 0x55b43e264990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.562+0000 7feba8dad640 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e264450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.562+0000 7feba95ae640 1 -- 172.21.15.49:0/309862121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 msgr2=0x55b43e264990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:34.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.562+0000 7feba95ae640 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 0x55b43e264990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.562+0000 7feba95ae640 1 -- 172.21.15.49:0/309862121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 msgr2=0x55b43e264450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:34.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.562+0000 7feba95ae640 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e264450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.562+0000 7feba95ae640 1 -- 172.21.15.49:0/309862121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b43e1829d0 con 0x55b43e2516e0 2024-03-18T00:57:34.573 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.573+0000 7feba95ae640 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e2516e0 0x55b43e267960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7feb94006010 tx=0x7feb94009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=befa847166332e65 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:34.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.577+0000 7feb8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:57:34.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.577+0000 7feb8cff9640 1 -- 172.21.15.49:0/309862121 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94028030 con 0x55b43e2516e0 2024-03-18T00:57:34.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.578+0000 7feb8cff9640 1 -- 172.21.15.49:0/309862121 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7feb94020040 con 0x55b43e2516e0 2024-03-18T00:57:34.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.578+0000 7febae3fe740 1 -- 172.21.15.49:0/309862121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feba00a1f10 con 0x55b43e2516e0 2024-03-18T00:57:34.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.589+0000 7feb8cff9640 1 -- 172.21.15.49:0/309862121 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7feb9403d050 con 0x55b43e2516e0 2024-03-18T00:57:34.590 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.589+0000 7febae3fe740 10 client.26461 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:34.591 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.589+0000 7febae3fe740 1 -- 172.21.15.49:0/309862121 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b43e2e28a0 con 0x55b43e2516e0 2024-03-18T00:57:34.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.591+0000 7feb8cff9640 1 -- 172.21.15.49:0/309862121 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feba00a1f10 con 0x55b43e2516e0 2024-03-18T00:57:34.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.591+0000 7feb8cff9640 1 -- 172.21.15.49:0/309862121 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7feb94023410 con 0x55b43e2516e0 2024-03-18T00:57:34.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.592+0000 7febae3fe740 10 client.26461 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:34.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.592+0000 7febae3fe740 10 client.26461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:34.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.592+0000 7febae3fe740 20 client.26461 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:34.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.592+0000 7febae3fe740 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b43e25bcd0 0x55b43e2e5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:34.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.592+0000 7febae3fe740 4 client.26461 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:57:34.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.592+0000 7febae3fe740 1 -- 172.21.15.49:0/309862121 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b43e1c2d90 con 0x55b43e25bcd0 2024-03-18T00:57:34.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.599+0000 7feba8dad640 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b43e25bcd0 0x55b43e2e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:34.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.612+0000 7feba8dad640 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b43e25bcd0 0x55b43e2e5290 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=787b6a0a7af1ca5d server_cookie=1e0489eb7fe3c104 in_seq=0 out_seq=0 2024-03-18T00:57:34.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.612+0000 7feb8cff9640 10 client.26461 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:34.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.615+0000 7feb8cff9640 1 -- 172.21.15.49:0/309862121 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b43e1c2d90 con 0x55b43e25bcd0 2024-03-18T00:57:34.620 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.615+0000 7feb8cff9640 10 client.26461 handle_command_reply: tid=0 2024-03-18T00:57:34.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.616+0000 7febae3fe740 1 client.26461 shutdown 2024-03-18T00:57:34.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.619+0000 7febae3fe740 1 -- 172.21.15.49:0/309862121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e2516e0 msgr2=0x55b43e267960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:34.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.619+0000 7febae3fe740 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e2516e0 0x55b43e267960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7feb94006010 tx=0x7feb94009310 comp rx=0 tx=0).stop 2024-03-18T00:57:34.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.623+0000 7febae3fe740 1 -- 172.21.15.49:0/309862121 shutdown_connections 2024-03-18T00:57:34.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.623+0000 7febae3fe740 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b43e25bcd0 0x55b43e2e5290 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.623+0000 7febae3fe740 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b43e256b60 0x55b43e264990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.623+0000 7febae3fe740 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b43e256170 0x55b43e264450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.623+0000 7febae3fe740 1 --2- 172.21.15.49:0/309862121 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b43e2516e0 0x55b43e267960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.623+0000 7febae3fe740 1 -- 172.21.15.49:0/309862121 >> 172.21.15.49:0/309862121 conn(0x55b43e257940 msgr2=0x55b43e254940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:34.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.623+0000 7feb8cff9640 0 client.26461 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:34.631 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.630+0000 7febae3fe740 1 -- 172.21.15.49:0/309862121 shutdown_connections 2024-03-18T00:57:34.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.634+0000 7febae3fe740 1 -- 172.21.15.49:0/309862121 wait complete. 2024-03-18T00:57:34.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.635+0000 7febae3fe740 20 client.26461 trim_cache size 0 max 16384 2024-03-18T00:57:34.637 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:57:34.637 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:57:34.638 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:57:34.638 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:57:34.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.637+0000 7febab812640 1 -- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a2a00 msgr2=0x7feb900a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:34.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.637+0000 7febab812640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a2a00 0x7feb900a4ec0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009230 tx=0x7feb9c02c8b0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.637+0000 7febab812640 1 -- 172.21.15.49:0/1821630483 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba4076570 msgr2=0x7feba4080780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:34.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.637+0000 7febab812640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba4076570 0x7feba4080780 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feba000ba50 tx=0x7feba000bf10 comp rx=0 tx=0).stop 2024-03-18T00:57:34.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.639+0000 7febab812640 1 -- 172.21.15.49:0/1821630483 shutdown_connections 2024-03-18T00:57:34.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.640+0000 7febab812640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a2a00 0x7feb900a4ec0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.640+0000 7febab812640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba4076570 0x7feba4080780 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.640+0000 7febab812640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4078f20 0x7feba40837d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.640+0000 7febab812640 1 --2- 172.21.15.49:0/1821630483 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba4073690 0x7feba4080240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:34.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.640+0000 7febab812640 1 -- 172.21.15.49:0/1821630483 >> 172.21.15.49:0/1821630483 conn(0x7feba406e640 msgr2=0x7feba40769b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:34.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.642+0000 7febab812640 1 -- 172.21.15.49:0/1821630483 shutdown_connections 2024-03-18T00:57:34.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:34.645+0000 7febab812640 1 -- 172.21.15.49:0/1821630483 wait complete. 2024-03-18T00:57:34.736 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:57:34.736 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:57:35.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.048+0000 7fe17e79f640 1 -- 172.21.15.49:0/2673615377 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe178101740 msgr2=0x7fe178101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:35.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.048+0000 7fe17e79f640 1 --2- 172.21.15.49:0/2673615377 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe178101740 0x7fe178101bc0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe168009900 tx=0x7fe16802bc60 comp rx=0 tx=0).stop 2024-03-18T00:57:35.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.049+0000 7fe17e79f640 1 -- 172.21.15.49:0/2673615377 shutdown_connections 2024-03-18T00:57:35.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.049+0000 7fe17e79f640 1 --2- 172.21.15.49:0/2673615377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe178102100 0x7fe178108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.049+0000 7fe17e79f640 1 --2- 172.21.15.49:0/2673615377 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe178101740 0x7fe178101bc0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.049+0000 7fe17e79f640 1 --2- 172.21.15.49:0/2673615377 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe178100550 0x7fe178100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.049 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.049+0000 7fe17e79f640 1 -- 172.21.15.49:0/2673615377 >> 172.21.15.49:0/2673615377 conn(0x7fe1780fbac0 msgr2=0x7fe1780fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:35.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.055+0000 7fe17e79f640 1 -- 172.21.15.49:0/2673615377 shutdown_connections 2024-03-18T00:57:35.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.056+0000 7fe17e79f640 1 -- 172.21.15.49:0/2673615377 wait complete. 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.057+0000 7fe17e79f640 1 Processor -- start 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.057+0000 7fe17e79f640 1 -- start start 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.058+0000 7fe17e79f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe178100550 0x7fe17806ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.058+0000 7fe17e79f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe178101740 0x7fe178070450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.058+0000 7fe17e79f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe178102100 0x7fe178196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.058+0000 7fe17e79f640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe17810d410 con 0x7fe178100550 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.058+0000 7fe17e79f640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe17810d290 con 0x7fe178101740 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.058+0000 7fe17e79f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe17810d590 con 0x7fe178102100 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.060+0000 7fe17cf9c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe178101740 0x7fe178070450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.060+0000 7fe17cf9c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe178101740 0x7fe178070450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:44494/0 (socket says 172.21.15.49:44494) 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.060+0000 7fe17cf9c640 1 -- 172.21.15.49:0/1830664499 learned_addr learned my addr 172.21.15.49:0/1830664499 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:35.063 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.060+0000 7fe17df9e640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe178102100 0x7fe178196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.067+0000 7fe17cf9c640 1 -- 172.21.15.49:0/1830664499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe178102100 msgr2=0x7fe178196b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.067+0000 7fe17cf9c640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe178102100 0x7fe178196b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.067+0000 7fe17cf9c640 1 -- 172.21.15.49:0/1830664499 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe178100550 msgr2=0x7fe17806ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.067+0000 7fe17cf9c640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe178100550 0x7fe17806ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.067+0000 7fe17cf9c640 1 -- 172.21.15.49:0/1830664499 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe178073640 con 0x7fe178101740 2024-03-18T00:57:35.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.077+0000 7fe17cf9c640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe178101740 0x7fe178070450 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe168004f50 tx=0x7fe16802cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:35.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.080+0000 7fe16e7fc640 1 -- 172.21.15.49:0/1830664499 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168042cb0 con 0x7fe178101740 2024-03-18T00:57:35.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.080+0000 7fe16e7fc640 1 -- 172.21.15.49:0/1830664499 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe168042e50 con 0x7fe178101740 2024-03-18T00:57:35.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.080+0000 7fe16e7fc640 1 -- 172.21.15.49:0/1830664499 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168003e10 con 0x7fe178101740 2024-03-18T00:57:35.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.080+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17810a570 con 0x7fe178101740 2024-03-18T00:57:35.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.080+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17810a950 con 0x7fe178101740 2024-03-18T00:57:35.087 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.087+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe178067c60 con 0x7fe178101740 2024-03-18T00:57:35.089 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.089+0000 7fe16e7fc640 1 -- 172.21.15.49:0/1830664499 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe168079a60 con 0x7fe178101740 2024-03-18T00:57:35.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.091+0000 7fe16e7fc640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1640a2950 0x7fe1640a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:35.095 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.092+0000 7fe16e7fc640 1 -- 172.21.15.49:0/1830664499 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe1680f8ea0 con 0x7fe178101740 2024-03-18T00:57:35.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.103+0000 7fe17d79d640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1640a2950 0x7fe1640a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:35.110 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.110+0000 7fe17d79d640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1640a2950 0x7fe1640a4e10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe178071460 tx=0x7fe174006c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:35.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.112+0000 7fe16e7fc640 1 -- 172.21.15.49:0/1830664499 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1680c5f40 con 0x7fe178101740 2024-03-18T00:57:35.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.758+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1781077a0 con 0x7fe178101740 2024-03-18T00:57:35.763 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.762+0000 7fe16e7fc640 1 -- 172.21.15.49:0/1830664499 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11934 (secure 0 0 0) 0x7fe1680c5840 con 0x7fe178101740 2024-03-18T00:57:35.763 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:57:35.763 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:57:06.597245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":138,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24499,24506,24481,24470,24463]},"id":1}]} 2024-03-18T00:57:35.764 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 28 2024-03-18T00:57:35.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.779+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1640a2950 msgr2=0x7fe1640a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:35.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.779+0000 7fe17e79f640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1640a2950 0x7fe1640a4e10 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe178071460 tx=0x7fe174006c40 comp rx=0 tx=0).stop 2024-03-18T00:57:35.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.779+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe178101740 msgr2=0x7fe178070450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:35.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.779+0000 7fe17e79f640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe178101740 0x7fe178070450 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe168004f50 tx=0x7fe16802cdb0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.781+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 shutdown_connections 2024-03-18T00:57:35.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.781+0000 7fe17e79f640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe1640a2950 0x7fe1640a4e10 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.781+0000 7fe17e79f640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe178101740 0x7fe178070450 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe168004f50 tx=0x7fe16802cdb0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.781+0000 7fe17e79f640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe178102100 0x7fe178196b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.781+0000 7fe17e79f640 1 --2- 172.21.15.49:0/1830664499 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe178100550 0x7fe17806ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:35.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.781+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 >> 172.21.15.49:0/1830664499 conn(0x7fe1780fbac0 msgr2=0x7fe178104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:35.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.783+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 shutdown_connections 2024-03-18T00:57:35.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:35.787+0000 7fe17e79f640 1 -- 172.21.15.49:0/1830664499 wait complete. 2024-03-18T00:57:35.835 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:35 smithi049 ceph-mon[32130]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 893 MiB/s wr, 1.55k op/s 2024-03-18T00:57:35.837 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:57:35.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:35 smithi107 ceph-mon[36270]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 893 MiB/s wr, 1.55k op/s 2024-03-18T00:57:35.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:35 smithi106 ceph-mon[35899]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 893 MiB/s wr, 1.55k op/s 2024-03-18T00:57:36.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.199+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/856141926 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac10db00 msgr2=0x7f7bac10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.199+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/856141926 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac10db00 0x7f7bac10df80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4009ca0 tx=0x7f7ba402c140 comp rx=0 tx=0).stop 2024-03-18T00:57:36.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.202+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/856141926 shutdown_connections 2024-03-18T00:57:36.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.202+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/856141926 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac10db00 0x7f7bac10df80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.202+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/856141926 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bac10b150 0x7f7bac10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.202+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/856141926 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bac1087e0 0x7f7bac10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.202+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/856141926 >> 172.21.15.49:0/856141926 conn(0x7f7bac06e640 msgr2=0x7f7bac070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:36.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.206+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/856141926 shutdown_connections 2024-03-18T00:57:36.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.210+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/856141926 wait complete. 2024-03-18T00:57:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.221+0000 7f7bb0b3b640 1 Processor -- start 2024-03-18T00:57:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.222+0000 7f7bb0b3b640 1 -- start start 2024-03-18T00:57:36.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.222+0000 7f7bb0b3b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1087e0 0x7f7bac1a6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.222+0000 7f7bb0b3b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bac10b150 0x7f7bac1a6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.222+0000 7f7bb0b3b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bac10db00 0x7f7bac1adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.222+0000 7f7bb0b3b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac11c2d0 con 0x7f7bac10db00 2024-03-18T00:57:36.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.222+0000 7f7bb0b3b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac11c150 con 0x7f7bac1087e0 2024-03-18T00:57:36.230 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.222+0000 7f7bb0b3b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7bac11c450 con 0x7f7bac10b150 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.222+0000 7f7baad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bac10db00 0x7f7bac1adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.223+0000 7f7baad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bac10db00 0x7f7bac1adf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38514/0 (socket says 172.21.15.49:38514) 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.223+0000 7f7baad9d640 1 -- 172.21.15.49:0/1086254057 learned_addr learned my addr 172.21.15.49:0/1086254057 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.223+0000 7f7baad9d640 1 -- 172.21.15.49:0/1086254057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bac10b150 msgr2=0x7f7bac1a6ec0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.223+0000 7f7baad9d640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bac10b150 0x7f7bac1a6ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.223+0000 7f7baad9d640 1 -- 172.21.15.49:0/1086254057 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1087e0 msgr2=0x7f7bac1a6980 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.223+0000 7f7baad9d640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1087e0 0x7f7bac1a6980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.223+0000 7f7baad9d640 1 -- 172.21.15.49:0/1086254057 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bac110040 con 0x7f7bac10db00 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.224+0000 7f7baad9d640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bac10db00 0x7f7bac1adf30 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7ba402c750 tx=0x7f7ba4032420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.228+0000 7f7b9b7fe640 1 -- 172.21.15.49:0/1086254057 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba4039040 con 0x7f7bac10db00 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.228+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/1086254057 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bac1102d0 con 0x7f7bac10db00 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.228+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/1086254057 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bac110840 con 0x7f7bac10db00 2024-03-18T00:57:36.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.228+0000 7f7b9b7fe640 1 -- 172.21.15.49:0/1086254057 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ba4044070 con 0x7f7bac10db00 2024-03-18T00:57:36.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.230+0000 7f7b9b7fe640 1 -- 172.21.15.49:0/1086254057 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba4004540 con 0x7f7bac10db00 2024-03-18T00:57:36.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.239+0000 7f7bb3808740 1 Processor -- start 2024-03-18T00:57:36.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.242+0000 7f7b9b7fe640 1 -- 172.21.15.49:0/1086254057 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7ba40046e0 con 0x7f7bac10db00 2024-03-18T00:57:36.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.242+0000 7f7bb3808740 1 -- start start 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.243+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c49c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.243+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 0x55c205c4ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.243+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c51840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.243+0000 7f7bb3808740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c205bae490 con 0x55c205c4f3b0 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.243+0000 7f7bb3808740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c205b56c90 con 0x55c205c4a240 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.243+0000 7f7bb3808740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c205baa530 con 0x55c205c49850 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.244+0000 7f7b9b7fe640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7b940a2830 0x7f7b940a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.244+0000 7f7b9b7fe640 1 -- 172.21.15.49:0/1086254057 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7ba40f0990 con 0x7f7bac10db00 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.245+0000 7f7ba9d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c51840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.245+0000 7f7ba9d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c51840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38518/0 (socket says 172.21.15.49:38518) 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.245+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/2482422302 learned_addr learned my addr 172.21.15.49:0/2482422302 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:36.248 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.245+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/2482422302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 msgr2=0x55c205c49c70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:36.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.245+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/2482422302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c49c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.245+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/2482422302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 msgr2=0x55c205c4ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:36.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.245+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/2482422302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 0x55c205c4ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.245+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/2482422302 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c205c11500 con 0x55c205c4f3b0 2024-03-18T00:57:36.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.245+0000 7f7baad9d640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7b940a2830 0x7f7b940a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.248+0000 7f7baad9d640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7b940a2830 0x7f7b940a4cf0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4037040 tx=0x7f7ba402c140 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:36.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.248+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/2482422302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c51840 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0005d00 tx=0x7f7ba000beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04fc28aa79ed3c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:36.254 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.254+0000 7f7b98ff9640 1 -- 172.21.15.49:0/2482422302 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0016070 con 0x55c205c4f3b0 2024-03-18T00:57:36.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.254+0000 7f7b98ff9640 1 -- 172.21.15.49:0/2482422302 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ba000dbe0 con 0x55c205c4f3b0 2024-03-18T00:57:36.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.255+0000 7f7bb3808740 1 -- 172.21.15.49:0/2482422302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 msgr2=0x55c205c51840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.255+0000 7f7bb3808740 1 --2- 172.21.15.49:0/2482422302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c51840 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0005d00 tx=0x7f7ba000beb0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.260+0000 7f7bb3808740 1 -- 172.21.15.49:0/2482422302 shutdown_connections 2024-03-18T00:57:36.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.260+0000 7f7bb3808740 1 --2- 172.21.15.49:0/2482422302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 0x55c205c4ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.260+0000 7f7bb3808740 1 --2- 172.21.15.49:0/2482422302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c51840 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.260+0000 7f7bb3808740 1 --2- 172.21.15.49:0/2482422302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c49c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.260+0000 7f7bb3808740 1 -- 172.21.15.49:0/2482422302 >> 172.21.15.49:0/2482422302 conn(0x55c205c44dc0 msgr2=0x55c205c451c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:36.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.261+0000 7f7bb3808740 1 -- 172.21.15.49:0/2482422302 shutdown_connections 2024-03-18T00:57:36.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.262+0000 7f7bb3808740 1 -- 172.21.15.49:0/2482422302 wait complete. 2024-03-18T00:57:36.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.263+0000 7f7bb3808740 1 -- start start 2024-03-18T00:57:36.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.263+0000 7f7bb3808740 1 Processor -- start 2024-03-18T00:57:36.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.263+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c53df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.264+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 0x55c205c54330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.264+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.264+0000 7f7bb3808740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c205c523b0 con 0x55c205c4f3b0 2024-03-18T00:57:36.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.264+0000 7f7bb3808740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c205b6a230 con 0x55c205c4a240 2024-03-18T00:57:36.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.264+0000 7f7bb3808740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c205c52530 con 0x55c205c49850 2024-03-18T00:57:36.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.264+0000 7f7bb3808740 10 client.0 _ll_register_callbacks cb 0x55c205ba6010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.264+0000 7f7ba9d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.265+0000 7f7ba9d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38524/0 (socket says 172.21.15.49:38524) 2024-03-18T00:57:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.265+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/1029417369 learned_addr learned my addr 172.21.15.49:0/1029417369 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.265+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/1029417369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 msgr2=0x55c205c53df0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.265+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c53df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.265+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/1029417369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 msgr2=0x55c205c54330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.265+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 0x55c205c54330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.265+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/1029417369 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c205b79100 con 0x55c205c4f3b0 2024-03-18T00:57:36.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.266+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b330 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000dfc0 tx=0x7f7ba000b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf159ed5ec8e67e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:36.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.272+0000 7f7b91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:36.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.272+0000 7f7b91ffb640 1 -- 172.21.15.49:0/1029417369 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0016070 con 0x55c205c4f3b0 2024-03-18T00:57:36.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.272+0000 7f7bb3808740 1 -- 172.21.15.49:0/1029417369 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ba000dbe0 con 0x55c205c4f3b0 2024-03-18T00:57:36.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.272+0000 7f7b91ffb640 1 -- 172.21.15.49:0/1029417369 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7ba00109b0 con 0x55c205c4f3b0 2024-03-18T00:57:36.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.275+0000 7f7b91ffb640 1 -- 172.21.15.49:0/1029417369 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7ba0029050 con 0x55c205c4f3b0 2024-03-18T00:57:36.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.277+0000 7f7bb3808740 10 client.17196 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:36.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.277+0000 7f7bb3808740 1 -- 172.21.15.49:0/1029417369 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c2058c23f0 con 0x55c205c4f3b0 2024-03-18T00:57:36.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.280+0000 7f7b91ffb640 1 -- 172.21.15.49:0/1029417369 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ba000dbe0 con 0x55c205c4f3b0 2024-03-18T00:57:36.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.281+0000 7f7b91ffb640 1 -- 172.21.15.49:0/1029417369 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7ba0004620 con 0x55c205c4f3b0 2024-03-18T00:57:36.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.281+0000 7f7bb3808740 10 client.17196 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:36.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.281+0000 7f7bb3808740 10 client.17196 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:57:36.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.281+0000 7f7bb3808740 20 client.17196 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:36.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.281+0000 7f7bb3808740 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c205c81f40 0x55c205c843c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.282+0000 7f7bb3808740 4 client.17196 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:36.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.282+0000 7f7bb3808740 1 -- 172.21.15.49:0/1029417369 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c205bbb9d0 con 0x55c205c81f40 2024-03-18T00:57:36.288 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.288+0000 7f7baa59c640 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c205c81f40 0x55c205c843c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.304+0000 7f7baa59c640 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c205c81f40 0x55c205c843c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c019e594c7f9eb1 server_cookie=4f04930fae94c8a1 in_seq=0 out_seq=0 2024-03-18T00:57:36.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.304+0000 7f7b91ffb640 10 client.17196 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:57:36.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.307+0000 7f7b91ffb640 1 -- 172.21.15.49:0/1029417369 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c205bbb9d0 con 0x55c205c81f40 2024-03-18T00:57:36.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.307+0000 7f7b91ffb640 10 client.17196 handle_command_reply: tid=0 2024-03-18T00:57:36.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.310+0000 7f7bb3808740 1 client.17196 shutdown 2024-03-18T00:57:36.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.314+0000 7f7bb3808740 1 -- 172.21.15.49:0/1029417369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 msgr2=0x55c205c5b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.314+0000 7f7bb3808740 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b330 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f7ba000dfc0 tx=0x7f7ba000b800 comp rx=0 tx=0).stop 2024-03-18T00:57:36.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.316+0000 7f7bb3808740 1 -- 172.21.15.49:0/1029417369 shutdown_connections 2024-03-18T00:57:36.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.316+0000 7f7bb3808740 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c205c81f40 0x55c205c843c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.316+0000 7f7bb3808740 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 0x55c205c54330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.316+0000 7f7bb3808740 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b330 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.316+0000 7f7bb3808740 1 --2- 172.21.15.49:0/1029417369 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c53df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.316+0000 7f7bb3808740 1 -- 172.21.15.49:0/1029417369 >> 172.21.15.49:0/1029417369 conn(0x55c205c44dc0 msgr2=0x55c205c43fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:36.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.316+0000 7f7b91ffb640 0 client.17196 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:57:36.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.323+0000 7f7bb3808740 1 -- 172.21.15.49:0/1029417369 shutdown_connections 2024-03-18T00:57:36.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.323+0000 7f7bb3808740 1 -- 172.21.15.49:0/1029417369 wait complete. 2024-03-18T00:57:36.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.324+0000 7f7bb3808740 20 client.17196 trim_cache size 0 max 16384 2024-03-18T00:57:36.379 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.378+0000 7f7bb3808740 1 Processor -- start 2024-03-18T00:57:36.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.378+0000 7f7bb3808740 1 -- start start 2024-03-18T00:57:36.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.379+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c4b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.379+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 0x55c205c4bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.380 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.379+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.379+0000 7f7ba9d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.379+0000 7f7bb3808740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c00e500 con 0x55c205c4f3b0 2024-03-18T00:57:36.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.379+0000 7f7bb3808740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c010040 con 0x55c205c4a240 2024-03-18T00:57:36.381 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.379+0000 7f7bb3808740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c00ebd0 con 0x55c205c49850 2024-03-18T00:57:36.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.380+0000 7f7ba9d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38532/0 (socket says 172.21.15.49:38532) 2024-03-18T00:57:36.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.380+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/2355495253 learned_addr learned my addr 172.21.15.49:0/2355495253 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:36.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.380+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/2355495253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c4b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.381+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/2355495253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 msgr2=0x55c205c4b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:36.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.381+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/2355495253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c4b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.382 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.381+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/2355495253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 msgr2=0x55c205c4bbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:36.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.381+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/2355495253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 0x55c205c4bbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.381+0000 7f7ba9d9b640 1 -- 172.21.15.49:0/2355495253 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c20598c7d0 con 0x55c205c4f3b0 2024-03-18T00:57:36.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.382+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/2355495253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b200 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x55c205bf6180 tx=0x7f7ba000beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2c8a97d9ce96c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:36.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.383+0000 7f7b93fff640 1 -- 172.21.15.49:0/2355495253 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0016070 con 0x55c205c4f3b0 2024-03-18T00:57:36.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.383+0000 7f7b93fff640 1 -- 172.21.15.49:0/2355495253 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ba000b5a0 con 0x55c205c4f3b0 2024-03-18T00:57:36.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.385+0000 7f7bb3808740 1 -- 172.21.15.49:0/2355495253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 msgr2=0x55c205c5b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.385+0000 7f7bb3808740 1 --2- 172.21.15.49:0/2355495253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b200 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x55c205bf6180 tx=0x7f7ba000beb0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.386+0000 7f7bb3808740 1 -- 172.21.15.49:0/2355495253 shutdown_connections 2024-03-18T00:57:36.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.386+0000 7f7bb3808740 1 --2- 172.21.15.49:0/2355495253 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4a240 0x55c205c4bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.386+0000 7f7bb3808740 1 --2- 172.21.15.49:0/2355495253 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4f3b0 0x55c205c5b200 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.386+0000 7f7bb3808740 1 --2- 172.21.15.49:0/2355495253 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c4b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.386+0000 7f7bb3808740 1 -- 172.21.15.49:0/2355495253 >> 172.21.15.49:0/2355495253 conn(0x55c205c44dc0 msgr2=0x55c205c451a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:36.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.386+0000 7f7bb3808740 1 -- 172.21.15.49:0/2355495253 shutdown_connections 2024-03-18T00:57:36.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.386+0000 7f7bb3808740 1 -- 172.21.15.49:0/2355495253 wait complete. 2024-03-18T00:57:36.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.387+0000 7f7bb3808740 1 -- start start 2024-03-18T00:57:36.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.387+0000 7f7bb3808740 1 Processor -- start 2024-03-18T00:57:36.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.387+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c5ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.387+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4a240 0x55c205c5b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.387+0000 7f7bb3808740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4f3b0 0x55c205c576b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.387+0000 7f7bb3808740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c00f970 con 0x55c205c4a240 2024-03-18T00:57:36.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.387+0000 7f7bb3808740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c00f2a0 con 0x55c205c4f3b0 2024-03-18T00:57:36.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.387+0000 7f7bb3808740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7b8c00c590 con 0x55c205c49850 2024-03-18T00:57:36.388 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.387+0000 7f7bb3808740 10 client.0 _ll_register_callbacks cb 0x7f7ba000b920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:36.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.389+0000 7f7baa59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4a240 0x55c205c5b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.389+0000 7f7baa59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4a240 0x55c205c5b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38546/0 (socket says 172.21.15.49:38546) 2024-03-18T00:57:36.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.389+0000 7f7baa59c640 1 -- 172.21.15.49:0/471474199 learned_addr learned my addr 172.21.15.49:0/471474199 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:36.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.390+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c5ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.390+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4f3b0 0x55c205c576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.390+0000 7f7baa59c640 1 -- 172.21.15.49:0/471474199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 msgr2=0x55c205c5ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:36.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.390+0000 7f7baa59c640 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c5ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.390+0000 7f7baa59c640 1 -- 172.21.15.49:0/471474199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4f3b0 msgr2=0x55c205c576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:36.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.390+0000 7f7baa59c640 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4f3b0 0x55c205c576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.390+0000 7f7baa59c640 1 -- 172.21.15.49:0/471474199 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c205c57ee0 con 0x55c205c4a240 2024-03-18T00:57:36.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.390+0000 7f7baa59c640 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4a240 0x55c205c5b150 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c000c00 tx=0x7f7b9c009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d5ec245daf0234 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:36.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.390+0000 7f7b91ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:36.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.391+0000 7f7b91ffb640 1 -- 172.21.15.49:0/471474199 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c016260 con 0x55c205c4a240 2024-03-18T00:57:36.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.391+0000 7f7b91ffb640 1 -- 172.21.15.49:0/471474199 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7b9c016400 con 0x55c205c4a240 2024-03-18T00:57:36.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.391+0000 7f7bb3808740 1 -- 172.21.15.49:0/471474199 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ba000b5a0 con 0x55c205c4a240 2024-03-18T00:57:36.392 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.393+0000 7f7b91ffb640 1 -- 172.21.15.49:0/471474199 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7b9c016600 con 0x55c205c4a240 2024-03-18T00:57:36.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.393+0000 7f7bb3808740 10 client.17202 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:36.398 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.393+0000 7f7bb3808740 1 -- 172.21.15.49:0/471474199 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c205ce7160 con 0x55c205c4a240 2024-03-18T00:57:36.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.398+0000 7f7b91ffb640 1 -- 172.21.15.49:0/471474199 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ba000b5a0 con 0x55c205c4a240 2024-03-18T00:57:36.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.398+0000 7f7b91ffb640 1 -- 172.21.15.49:0/471474199 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7b9c014680 con 0x55c205c4a240 2024-03-18T00:57:36.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.398+0000 7f7bb3808740 10 client.17202 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:36.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.398+0000 7f7bb3808740 10 client.17202 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:57:36.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.398+0000 7f7bb3808740 20 client.17202 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:36.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.398+0000 7f7bb3808740 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c205ce7b40 0x55c205ce9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.398+0000 7f7bb3808740 4 client.17202 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:57:36.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.398+0000 7f7bb3808740 1 -- 172.21.15.49:0/471474199 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c205bb5d20 con 0x55c205ce7b40 2024-03-18T00:57:36.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.401+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c205ce7b40 0x55c205ce9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.405+0000 7f7ba9d9b640 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c205ce7b40 0x55c205ce9f60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=504035b70e0d84d0 server_cookie=1be6d08cad70ce08 in_seq=0 out_seq=0 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.405+0000 7f7b91ffb640 10 client.17202 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.406+0000 7f7b91ffb640 1 -- 172.21.15.49:0/471474199 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c205bb5d20 con 0x55c205ce7b40 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.406+0000 7f7b91ffb640 10 client.17202 handle_command_reply: tid=0 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.406+0000 7f7bb3808740 1 client.17202 shutdown 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.407+0000 7f7bb3808740 1 -- 172.21.15.49:0/471474199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4a240 msgr2=0x55c205c5b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.407+0000 7f7bb3808740 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4a240 0x55c205c5b150 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c000c00 tx=0x7f7b9c009aa0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.407+0000 7f7bb3808740 1 -- 172.21.15.49:0/471474199 shutdown_connections 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.407+0000 7f7bb3808740 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c205ce7b40 0x55c205ce9f60 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.407+0000 7f7bb3808740 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c205c4f3b0 0x55c205c576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.407+0000 7f7bb3808740 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c205c4a240 0x55c205c5b150 secure :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c000c00 tx=0x7f7b9c009aa0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.407+0000 7f7bb3808740 1 --2- 172.21.15.49:0/471474199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c205c49850 0x55c205c5ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.407+0000 7f7bb3808740 1 -- 172.21.15.49:0/471474199 >> 172.21.15.49:0/471474199 conn(0x55c205c44dc0 msgr2=0x55c205c48020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:36.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.407+0000 7f7b91ffb640 0 client.17202 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:57:36.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.409+0000 7f7bb3808740 1 -- 172.21.15.49:0/471474199 shutdown_connections 2024-03-18T00:57:36.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.409+0000 7f7bb3808740 1 -- 172.21.15.49:0/471474199 wait complete. 2024-03-18T00:57:36.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.409+0000 7f7bb3808740 20 client.17202 trim_cache size 0 max 16384 2024-03-18T00:57:36.410 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:57:36.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7b940a2830 msgr2=0x7f7b940a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7b940a2830 0x7f7b940a4cf0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7ba4037040 tx=0x7f7ba402c140 comp rx=0 tx=0).stop 2024-03-18T00:57:36.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bac10db00 msgr2=0x7f7bac1adf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.414 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bac10db00 0x7f7bac1adf30 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7ba402c750 tx=0x7f7ba4032420 comp rx=0 tx=0).stop 2024-03-18T00:57:36.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/1086254057 shutdown_connections 2024-03-18T00:57:36.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7b940a2830 0x7f7b940a4cf0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bac10db00 0x7f7bac1adf30 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bac10b150 0x7f7bac1a6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.415 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 --2- 172.21.15.49:0/1086254057 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bac1087e0 0x7f7bac1a6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/1086254057 >> 172.21.15.49:0/1086254057 conn(0x7f7bac06e640 msgr2=0x7f7bac109df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:36.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/1086254057 shutdown_connections 2024-03-18T00:57:36.416 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.414+0000 7f7bb0b3b640 1 -- 172.21.15.49:0/1086254057 wait complete. 2024-03-18T00:57:36.466 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:57:36.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:36 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1830664499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:57:36.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:36 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1830664499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:57:36.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:36 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1830664499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:57:36.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.967+0000 7f3440f5e640 1 -- 172.21.15.49:0/4244380499 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3438032990 con 0x7f343c1098f0 2024-03-18T00:57:36.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.969+0000 7f3442f62640 1 -- 172.21.15.49:0/4244380499 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c1098f0 msgr2=0x7f343c10bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.969+0000 7f3442f62640 1 --2- 172.21.15.49:0/4244380499 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c1098f0 0x7f343c10bce0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f3438009ca0 tx=0x7f343802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.969+0000 7f3442f62640 1 -- 172.21.15.49:0/4244380499 shutdown_connections 2024-03-18T00:57:36.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.969+0000 7f3442f62640 1 --2- 172.21.15.49:0/4244380499 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f343c10eb50 0x7f343c110f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.969+0000 7f3442f62640 1 --2- 172.21.15.49:0/4244380499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c10c220 0x7f343c10e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.969+0000 7f3442f62640 1 --2- 172.21.15.49:0/4244380499 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c1098f0 0x7f343c10bce0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.969+0000 7f3442f62640 1 -- 172.21.15.49:0/4244380499 >> 172.21.15.49:0/4244380499 conn(0x7f343c06e6c0 msgr2=0x7f343c070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:36.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.970+0000 7f3442f62640 1 -- 172.21.15.49:0/4244380499 shutdown_connections 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.970+0000 7f3442f62640 1 -- 172.21.15.49:0/4244380499 wait complete. 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.971+0000 7f3442f62640 1 Processor -- start 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.971+0000 7f3442f62640 1 -- start start 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.971+0000 7f3442f62640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f343c1098f0 0x7f343c1a2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.971+0000 7f3442f62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c10c220 0x7f343c1a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.972+0000 7f3442f62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c10eb50 0x7f343c1a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.972+0000 7f344175f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c10c220 0x7f343c1a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.972+0000 7f344175f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c10c220 0x7f343c1a29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38564/0 (socket says 172.21.15.49:38564) 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.972+0000 7f344175f640 1 -- 172.21.15.49:0/2477234832 learned_addr learned my addr 172.21.15.49:0/2477234832 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.972+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1123a0 con 0x7f343c10c220 2024-03-18T00:57:36.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.972+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f343c112220 con 0x7f343c1098f0 2024-03-18T00:57:36.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.972+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f343c112520 con 0x7f343c10eb50 2024-03-18T00:57:36.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.973+0000 7f344175f640 1 -- 172.21.15.49:0/2477234832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c10eb50 msgr2=0x7f343c1a9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.973+0000 7f344175f640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c10eb50 0x7f343c1a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.973+0000 7f344175f640 1 -- 172.21.15.49:0/2477234832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f343c1098f0 msgr2=0x7f343c1a2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.973+0000 7f344175f640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f343c1098f0 0x7f343c1a2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.973+0000 7f344175f640 1 -- 172.21.15.49:0/2477234832 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c1aa090 con 0x7f343c10c220 2024-03-18T00:57:36.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.973+0000 7f344175f640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c10c220 0x7f343c1a29b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f342c00e9c0 tx=0x7f342c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:36.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.973+0000 7f3432ffd640 1 -- 172.21.15.49:0/2477234832 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c00cd80 con 0x7f343c10c220 2024-03-18T00:57:36.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.974+0000 7f3432ffd640 1 -- 172.21.15.49:0/2477234832 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f342c005500 con 0x7f343c10c220 2024-03-18T00:57:36.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.974+0000 7f3432ffd640 1 -- 172.21.15.49:0/2477234832 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c010690 con 0x7f343c10c220 2024-03-18T00:57:36.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.974+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f343c1151a0 con 0x7f343c10c220 2024-03-18T00:57:36.975 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.974+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f343c1156e0 con 0x7f343c10c220 2024-03-18T00:57:36.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.978+0000 7f3432ffd640 1 -- 172.21.15.49:0/2477234832 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f342c0046c0 con 0x7f343c10c220 2024-03-18T00:57:36.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.981+0000 7f3446dac740 1 Processor -- start 2024-03-18T00:57:36.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.982+0000 7f3446dac740 1 -- start start 2024-03-18T00:57:36.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.982+0000 7f3446dac740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf525fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.982+0000 7f3446dac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf5260430 0x556cf5264f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.982+0000 7f3446dac740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf52655a0 0x556cf5267a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.982+0000 7f3446dac740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x556cf51c4490 con 0x556cf52655a0 2024-03-18T00:57:36.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.982+0000 7f3446dac740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x556cf516cc90 con 0x556cf525fa40 2024-03-18T00:57:36.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.982+0000 7f3446dac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556cf51c0530 con 0x556cf5260430 2024-03-18T00:57:36.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.983+0000 7f3441f60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf5260430 0x556cf5264f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.984+0000 7f344175f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf52655a0 0x556cf5267a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.984+0000 7f344175f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf52655a0 0x556cf5267a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38574/0 (socket says 172.21.15.49:38574) 2024-03-18T00:57:36.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.984+0000 7f344175f640 1 -- 172.21.15.49:0/2099521760 learned_addr learned my addr 172.21.15.49:0/2099521760 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:36.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.985+0000 7f3442761640 1 --2- 172.21.15.49:0/2099521760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf525fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.985+0000 7f3442761640 1 -- 172.21.15.49:0/2099521760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf5260430 msgr2=0x556cf5264f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:36.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.985+0000 7f3442761640 1 --2- 172.21.15.49:0/2099521760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf5260430 0x556cf5264f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.986+0000 7f3442761640 1 -- 172.21.15.49:0/2099521760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf52655a0 msgr2=0x556cf5267a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:36.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.986+0000 7f3442761640 1 --2- 172.21.15.49:0/2099521760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf52655a0 0x556cf5267a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:36.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.986+0000 7f3442761640 1 -- 172.21.15.49:0/2099521760 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf5227500 con 0x556cf525fa40 2024-03-18T00:57:36.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.986+0000 7f3432ffd640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34280ab090 0x7f34280ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:36.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.987+0000 7f3432ffd640 1 -- 172.21.15.49:0/2477234832 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f342c0cef30 con 0x7f343c10c220 2024-03-18T00:57:36.987 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.988+0000 7f3442761640 1 --2- 172.21.15.49:0/2099521760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf525fe60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f343400b9a0 tx=0x7f343400be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c0916bba143d94a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:36.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.988+0000 7f3427fff640 1 -- 172.21.15.49:0/2099521760 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434016070 con 0x556cf525fa40 2024-03-18T00:57:36.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.989+0000 7f3427fff640 1 -- 172.21.15.49:0/2099521760 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3434004220 con 0x556cf525fa40 2024-03-18T00:57:36.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.990+0000 7f3446dac740 1 -- 172.21.15.49:0/2099521760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 msgr2=0x556cf525fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:36.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.990+0000 7f3446dac740 1 --2- 172.21.15.49:0/2099521760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf525fe60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f343400b9a0 tx=0x7f343400be60 comp rx=0 tx=0).stop 2024-03-18T00:57:36.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.992+0000 7f3441f60640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34280ab090 0x7f34280ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:36.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:36.992+0000 7f3441f60640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34280ab090 0x7f34280ad550 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3438006010 tx=0x7f3438032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.005+0000 7f3446dac740 1 -- 172.21.15.49:0/2099521760 shutdown_connections 2024-03-18T00:57:37.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.005+0000 7f3446dac740 1 --2- 172.21.15.49:0/2099521760 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf52655a0 0x556cf5267a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.005+0000 7f3446dac740 1 --2- 172.21.15.49:0/2099521760 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf5260430 0x556cf5264f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.005+0000 7f3446dac740 1 --2- 172.21.15.49:0/2099521760 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf525fe60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.005+0000 7f3446dac740 1 -- 172.21.15.49:0/2099521760 >> 172.21.15.49:0/2099521760 conn(0x556cf525afb0 msgr2=0x556cf525b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.006+0000 7f3446dac740 1 -- 172.21.15.49:0/2099521760 shutdown_connections 2024-03-18T00:57:37.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.007+0000 7f3446dac740 1 -- 172.21.15.49:0/2099521760 wait complete. 2024-03-18T00:57:37.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.007+0000 7f3446dac740 1 -- start start 2024-03-18T00:57:37.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.008+0000 7f3446dac740 1 Processor -- start 2024-03-18T00:57:37.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.009+0000 7f3446dac740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf525fa40 0x556cf5269fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.009+0000 7f3446dac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf5260430 0x556cf526a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.009+0000 7f3446dac740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf52655a0 0x556cf5271520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.009+0000 7f3446dac740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x556cf52685a0 con 0x556cf525fa40 2024-03-18T00:57:37.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.009+0000 7f3446dac740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x556cf5180230 con 0x556cf52655a0 2024-03-18T00:57:37.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.009+0000 7f3446dac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556cf5268720 con 0x556cf5260430 2024-03-18T00:57:37.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.009+0000 7f3446dac740 10 client.0 _ll_register_callbacks cb 0x556cf51bc010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:37.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.010+0000 7f3441f60640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf525fa40 0x556cf5269fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.010+0000 7f3441f60640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf525fa40 0x556cf5269fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38582/0 (socket says 172.21.15.49:38582) 2024-03-18T00:57:37.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.010+0000 7f3441f60640 1 -- 172.21.15.49:0/1258862204 learned_addr learned my addr 172.21.15.49:0/1258862204 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.012+0000 7f3441f60640 1 -- 172.21.15.49:0/1258862204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf5260430 msgr2=0x556cf526a4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.012+0000 7f3441f60640 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf5260430 0x556cf526a4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.012+0000 7f3441f60640 1 -- 172.21.15.49:0/1258862204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf52655a0 msgr2=0x556cf5271520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.012+0000 7f3441f60640 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf52655a0 0x556cf5271520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.012+0000 7f3441f60640 1 -- 172.21.15.49:0/1258862204 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf518f100 con 0x556cf525fa40 2024-03-18T00:57:37.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.013+0000 7f3441f60640 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf525fa40 0x556cf5269fa0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f343802c630 tx=0x7f3438000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43b5d1279cc24e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.016+0000 7f34257fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:37.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.020+0000 7f34257fa640 1 -- 172.21.15.49:0/1258862204 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343804e020 con 0x556cf525fa40 2024-03-18T00:57:37.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.021+0000 7f3446dac740 1 -- 172.21.15.49:0/1258862204 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3434004220 con 0x556cf525fa40 2024-03-18T00:57:37.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.022+0000 7f34257fa640 1 -- 172.21.15.49:0/1258862204 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3438003fc0 con 0x556cf525fa40 2024-03-18T00:57:37.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.027+0000 7f34257fa640 1 -- 172.21.15.49:0/1258862204 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3438059050 con 0x556cf525fa40 2024-03-18T00:57:37.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.027+0000 7f3446dac740 10 client.17214 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:37.028 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.027+0000 7f3446dac740 1 -- 172.21.15.49:0/1258862204 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556cf4ed83f0 con 0x556cf525fa40 2024-03-18T00:57:37.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.030+0000 7f34257fa640 1 -- 172.21.15.49:0/1258862204 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3434004220 con 0x556cf525fa40 2024-03-18T00:57:37.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.030+0000 7f34257fa640 1 -- 172.21.15.49:0/1258862204 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3438049030 con 0x556cf525fa40 2024-03-18T00:57:37.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.037+0000 7f3446dac740 10 client.17214 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:37.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.037+0000 7f3446dac740 10 client.17214 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:57:37.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.038+0000 7f3446dac740 20 client.17214 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:37.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.038+0000 7f3446dac740 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x556cf5275ef0 0x556cf5278360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.038+0000 7f3446dac740 4 client.17214 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:37.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.038+0000 7f3446dac740 1 -- 172.21.15.49:0/1258862204 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cf51d19d0 con 0x556cf5275ef0 2024-03-18T00:57:37.053 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.053+0000 7f344175f640 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x556cf5275ef0 0x556cf5278360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.059+0000 7f344175f640 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x556cf5275ef0 0x556cf5278360 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5e98e81496f73cb4 server_cookie=4a1f703b0385f4af in_seq=0 out_seq=0 2024-03-18T00:57:37.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.061+0000 7f34257fa640 10 client.17214 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:57:37.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.067+0000 7f34257fa640 1 -- 172.21.15.49:0/1258862204 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556cf51d19d0 con 0x556cf5275ef0 2024-03-18T00:57:37.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.067+0000 7f34257fa640 10 client.17214 handle_command_reply: tid=0 2024-03-18T00:57:37.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.067+0000 7f3446dac740 1 client.17214 shutdown 2024-03-18T00:57:37.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.068+0000 7f3446dac740 1 -- 172.21.15.49:0/1258862204 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf525fa40 msgr2=0x556cf5269fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.069 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.068+0000 7f3446dac740 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf525fa40 0x556cf5269fa0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f343802c630 tx=0x7f3438000f40 comp rx=0 tx=0).stop 2024-03-18T00:57:37.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.069+0000 7f3446dac740 1 -- 172.21.15.49:0/1258862204 shutdown_connections 2024-03-18T00:57:37.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.069+0000 7f3446dac740 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x556cf5275ef0 0x556cf5278360 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.069+0000 7f3446dac740 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf52655a0 0x556cf5271520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.069+0000 7f3446dac740 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf5260430 0x556cf526a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.069+0000 7f3446dac740 1 --2- 172.21.15.49:0/1258862204 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf525fa40 0x556cf5269fa0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.069+0000 7f3446dac740 1 -- 172.21.15.49:0/1258862204 >> 172.21.15.49:0/1258862204 conn(0x556cf525afb0 msgr2=0x556cf525a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.069+0000 7f34257fa640 0 client.17214 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:57:37.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.070+0000 7f3446dac740 1 -- 172.21.15.49:0/1258862204 shutdown_connections 2024-03-18T00:57:37.070 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.070+0000 7f3446dac740 1 -- 172.21.15.49:0/1258862204 wait complete. 2024-03-18T00:57:37.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.072+0000 7f3446dac740 20 client.17214 trim_cache size 0 max 16384 2024-03-18T00:57:37.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.122+0000 7f3446dac740 1 Processor -- start 2024-03-18T00:57:37.122 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.122+0000 7f3446dac740 1 -- start start 2024-03-18T00:57:37.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.123+0000 7f3446dac740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.123+0000 7f3446dac740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf52714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.123+0000 7f3446dac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf52655a0 0x556cf52598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.123+0000 7f3446dac740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1af3b0 con 0x556cf5260430 2024-03-18T00:57:37.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.123+0000 7f3446dac740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1b0ef0 con 0x556cf525fa40 2024-03-18T00:57:37.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.123+0000 7f3446dac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1afa80 con 0x556cf52655a0 2024-03-18T00:57:37.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.123+0000 7f344175f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf52714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.123+0000 7f344175f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf52714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38592/0 (socket says 172.21.15.49:38592) 2024-03-18T00:57:37.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.123+0000 7f344175f640 1 -- 172.21.15.49:0/4216224215 learned_addr learned my addr 172.21.15.49:0/4216224215 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:37.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.124+0000 7f3441f60640 1 --2- 172.21.15.49:0/4216224215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.124+0000 7f3441f60640 1 -- 172.21.15.49:0/4216224215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf52655a0 msgr2=0x556cf52598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.124+0000 7f3441f60640 1 --2- 172.21.15.49:0/4216224215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf52655a0 0x556cf52598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.124+0000 7f3441f60640 1 -- 172.21.15.49:0/4216224215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 msgr2=0x556cf52714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.124+0000 7f3441f60640 1 --2- 172.21.15.49:0/4216224215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf52714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.125+0000 7f3441f60640 1 -- 172.21.15.49:0/4216224215 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cf4fa27d0 con 0x556cf525fa40 2024-03-18T00:57:37.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.125+0000 7f3441f60640 1 --2- 172.21.15.49:0/4216224215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526a2c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f343802c630 tx=0x7f34380093f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7a385c322ba2c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.131+0000 7f34277fe640 1 -- 172.21.15.49:0/4216224215 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3438061070 con 0x556cf525fa40 2024-03-18T00:57:37.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.131+0000 7f34277fe640 1 -- 172.21.15.49:0/4216224215 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f343803a040 con 0x556cf525fa40 2024-03-18T00:57:37.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.133+0000 7f3446dac740 1 -- 172.21.15.49:0/4216224215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 msgr2=0x556cf526a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.133+0000 7f3446dac740 1 --2- 172.21.15.49:0/4216224215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526a2c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f343802c630 tx=0x7f34380093f0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.134+0000 7f3446dac740 1 -- 172.21.15.49:0/4216224215 shutdown_connections 2024-03-18T00:57:37.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.134+0000 7f3446dac740 1 --2- 172.21.15.49:0/4216224215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf52655a0 0x556cf52598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.134+0000 7f3446dac740 1 --2- 172.21.15.49:0/4216224215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf52714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.134+0000 7f3446dac740 1 --2- 172.21.15.49:0/4216224215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526a2c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.134+0000 7f3446dac740 1 -- 172.21.15.49:0/4216224215 >> 172.21.15.49:0/4216224215 conn(0x556cf525afb0 msgr2=0x556cf5269bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.140+0000 7f3446dac740 1 -- 172.21.15.49:0/4216224215 shutdown_connections 2024-03-18T00:57:37.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.143+0000 7f3446dac740 1 -- 172.21.15.49:0/4216224215 wait complete. 2024-03-18T00:57:37.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.144+0000 7f3446dac740 1 -- start start 2024-03-18T00:57:37.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.144+0000 7f3446dac740 1 Processor -- start 2024-03-18T00:57:37.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.145+0000 7f3446dac740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.145+0000 7f3446dac740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf526f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.145+0000 7f3446dac740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf52655a0 0x556cf52703d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.145+0000 7f3446dac740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1b0820 con 0x556cf5260430 2024-03-18T00:57:37.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.145+0000 7f3446dac740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1b0150 con 0x556cf525fa40 2024-03-18T00:57:37.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.145+0000 7f3446dac740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f343c1ad390 con 0x556cf52655a0 2024-03-18T00:57:37.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.145+0000 7f3446dac740 10 client.0 _ll_register_callbacks cb 0x7f3438037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:37.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.146+0000 7f344175f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf526f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.146+0000 7f344175f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf526f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38608/0 (socket says 172.21.15.49:38608) 2024-03-18T00:57:37.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.146+0000 7f344175f640 1 -- 172.21.15.49:0/2024293164 learned_addr learned my addr 172.21.15.49:0/2024293164 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:37.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.147+0000 7f3441f60640 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.148+0000 7f3441f60640 1 -- 172.21.15.49:0/2024293164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf52655a0 msgr2=0x556cf52703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.148+0000 7f3441f60640 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf52655a0 0x556cf52703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.148+0000 7f3441f60640 1 -- 172.21.15.49:0/2024293164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 msgr2=0x556cf526f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.148+0000 7f3441f60640 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf526f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.148+0000 7f3441f60640 1 -- 172.21.15.49:0/2024293164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cf52709c0 con 0x556cf525fa40 2024-03-18T00:57:37.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.149+0000 7f3441f60640 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526ee80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3438034c20 tx=0x7f3438009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb732e5a4006d531 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.158+0000 7f34257fa640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:57:37.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.159+0000 7f34257fa640 1 -- 172.21.15.49:0/2024293164 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343806b020 con 0x556cf525fa40 2024-03-18T00:57:37.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.159+0000 7f34257fa640 1 -- 172.21.15.49:0/2024293164 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3438034440 con 0x556cf525fa40 2024-03-18T00:57:37.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.161+0000 7f3446dac740 1 -- 172.21.15.49:0/2024293164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556cf51cbd20 con 0x556cf525fa40 2024-03-18T00:57:37.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.165+0000 7f34257fa640 1 -- 172.21.15.49:0/2024293164 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3438061040 con 0x556cf525fa40 2024-03-18T00:57:37.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.170+0000 7f3446dac740 10 client.26387 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:37.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.170+0000 7f3446dac740 1 -- 172.21.15.49:0/2024293164 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556cf52dc4f0 con 0x556cf525fa40 2024-03-18T00:57:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.176+0000 7f34257fa640 1 -- 172.21.15.49:0/2024293164 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556cf51cbd20 con 0x556cf525fa40 2024-03-18T00:57:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.176+0000 7f34257fa640 1 -- 172.21.15.49:0/2024293164 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3438049040 con 0x556cf525fa40 2024-03-18T00:57:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.176+0000 7f3446dac740 10 client.26387 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:37.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.176+0000 7f3446dac740 10 client.26387 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:57:37.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.176+0000 7f3446dac740 20 client.26387 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:37.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.176+0000 7f3446dac740 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x556cf52dcd30 0x556cf52df150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.176+0000 7f3446dac740 4 client.26387 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:57:37.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.176+0000 7f3446dac740 1 -- 172.21.15.49:0/2024293164 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556cf51496a0 con 0x556cf52dcd30 2024-03-18T00:57:37.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.178+0000 7f344175f640 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x556cf52dcd30 0x556cf52df150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.188+0000 7f34257fa640 10 client.26387 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:57:37.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.188+0000 7f344175f640 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x556cf52dcd30 0x556cf52df150 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=66596b9557e8094c server_cookie=47ac10eb4466750b in_seq=0 out_seq=0 2024-03-18T00:57:37.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.189+0000 7f34257fa640 1 -- 172.21.15.49:0/2024293164 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556cf51496a0 con 0x556cf52dcd30 2024-03-18T00:57:37.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.189+0000 7f34257fa640 10 client.26387 handle_command_reply: tid=0 2024-03-18T00:57:37.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.189+0000 7f3446dac740 1 client.26387 shutdown 2024-03-18T00:57:37.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.192+0000 7f3446dac740 1 -- 172.21.15.49:0/2024293164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 msgr2=0x556cf526ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.192+0000 7f3446dac740 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526ee80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3438034c20 tx=0x7f3438009800 comp rx=0 tx=0).stop 2024-03-18T00:57:37.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.196+0000 7f3446dac740 1 -- 172.21.15.49:0/2024293164 shutdown_connections 2024-03-18T00:57:37.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.196+0000 7f3446dac740 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x556cf52dcd30 0x556cf52df150 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.196+0000 7f3446dac740 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556cf52655a0 0x556cf52703d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.196+0000 7f3446dac740 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556cf5260430 0x556cf526f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.196+0000 7f3446dac740 1 --2- 172.21.15.49:0/2024293164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556cf525fa40 0x556cf526ee80 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3438034c20 tx=0x7f3438009800 comp rx=0 tx=0).stop 2024-03-18T00:57:37.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.196+0000 7f3446dac740 1 -- 172.21.15.49:0/2024293164 >> 172.21.15.49:0/2024293164 conn(0x556cf525afb0 msgr2=0x556cf52598b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.197+0000 7f34257fa640 0 client.26387 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:57:37.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.205+0000 7f3446dac740 1 -- 172.21.15.49:0/2024293164 shutdown_connections 2024-03-18T00:57:37.210 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.210+0000 7f3446dac740 1 -- 172.21.15.49:0/2024293164 wait complete. 2024-03-18T00:57:37.212 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.210+0000 7f3446dac740 20 client.26387 trim_cache size 0 max 16384 2024-03-18T00:57:37.215 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:57:37.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.225+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34280ab090 msgr2=0x7f34280ad550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.225+0000 7f3442f62640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34280ab090 0x7f34280ad550 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3438006010 tx=0x7f3438032590 comp rx=0 tx=0).stop 2024-03-18T00:57:37.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.225+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c10c220 msgr2=0x7f343c1a29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.225+0000 7f3442f62640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c10c220 0x7f343c1a29b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f342c00e9c0 tx=0x7f342c00ee80 comp rx=0 tx=0).stop 2024-03-18T00:57:37.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.226+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 shutdown_connections 2024-03-18T00:57:37.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.226+0000 7f3442f62640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34280ab090 0x7f34280ad550 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.231 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.226+0000 7f3442f62640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f343c10eb50 0x7f343c1a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.226+0000 7f3442f62640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f343c10c220 0x7f343c1a29b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.226+0000 7f3442f62640 1 --2- 172.21.15.49:0/2477234832 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f343c1098f0 0x7f343c1a2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.226+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 >> 172.21.15.49:0/2477234832 conn(0x7f343c06e6c0 msgr2=0x7f343c10a3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.228+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 shutdown_connections 2024-03-18T00:57:37.232 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.228+0000 7f3442f62640 1 -- 172.21.15.49:0/2477234832 wait complete. 2024-03-18T00:57:37.273 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:57:37.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:37 smithi049 ceph-mon[32130]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 874 MiB/s wr, 1.65k op/s 2024-03-18T00:57:37.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.635+0000 7fc711b87640 1 -- 172.21.15.49:0/2594115955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70c111f20 msgr2=0x7fc70c114330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.635+0000 7fc711b87640 1 --2- 172.21.15.49:0/2594115955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70c111f20 0x7fc70c114330 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc009900 tx=0x7fc6fc02bc80 comp rx=0 tx=0).stop 2024-03-18T00:57:37.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.635+0000 7fc711b87640 1 -- 172.21.15.49:0/2594115955 shutdown_connections 2024-03-18T00:57:37.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.635+0000 7fc711b87640 1 --2- 172.21.15.49:0/2594115955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc70c10b150 0x7fc70c10b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.635+0000 7fc711b87640 1 --2- 172.21.15.49:0/2594115955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70c111f20 0x7fc70c114330 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.635+0000 7fc711b87640 1 --2- 172.21.15.49:0/2594115955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc70c1087e0 0x7fc70c10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.635+0000 7fc711b87640 1 -- 172.21.15.49:0/2594115955 >> 172.21.15.49:0/2594115955 conn(0x7fc70c06e640 msgr2=0x7fc70c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.639+0000 7fc711b87640 1 -- 172.21.15.49:0/2594115955 shutdown_connections 2024-03-18T00:57:37.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.640+0000 7fc711b87640 1 -- 172.21.15.49:0/2594115955 wait complete. 2024-03-18T00:57:37.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.641+0000 7fc711b87640 1 Processor -- start 2024-03-18T00:57:37.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.641+0000 7fc711b87640 1 -- start start 2024-03-18T00:57:37.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.642+0000 7fc711b87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70c1087e0 0x7fc70c1a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.642+0000 7fc711b87640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc70c10b150 0x7fc70c1a6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.642+0000 7fc711b87640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc70c111f20 0x7fc70c1adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.642+0000 7fc711b87640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c117f70 con 0x7fc70c10b150 2024-03-18T00:57:37.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.642+0000 7fc711b87640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c117df0 con 0x7fc70c111f20 2024-03-18T00:57:37.642 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.642+0000 7fc711b87640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc70c1180f0 con 0x7fc70c1087e0 2024-03-18T00:57:37.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.643+0000 7fc70affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc70c10b150 0x7fc70c1a6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.643+0000 7fc70affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc70c10b150 0x7fc70c1a6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47880/0 (socket says 172.21.15.49:47880) 2024-03-18T00:57:37.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.643+0000 7fc70affd640 1 -- 172.21.15.49:0/3698472254 learned_addr learned my addr 172.21.15.49:0/3698472254 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:37.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.643+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70c1087e0 0x7fc70c1a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.643+0000 7fc70bfff640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc70c111f20 0x7fc70c1adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.644+0000 7fc70affd640 1 -- 172.21.15.49:0/3698472254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70c1087e0 msgr2=0x7fc70c1a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.644+0000 7fc70affd640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70c1087e0 0x7fc70c1a6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.644+0000 7fc70affd640 1 -- 172.21.15.49:0/3698472254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc70c111f20 msgr2=0x7fc70c1adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.644+0000 7fc70affd640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc70c111f20 0x7fc70c1adfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.644 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.644+0000 7fc70affd640 1 -- 172.21.15.49:0/3698472254 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc70c1ae630 con 0x7fc70c10b150 2024-03-18T00:57:37.654 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.653+0000 7fc70affd640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc70c10b150 0x7fc70c1a6fa0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc70400ba50 tx=0x7fc70400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.657+0000 7fc708ff9640 1 -- 172.21.15.49:0/3698472254 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704019070 con 0x7fc70c10b150 2024-03-18T00:57:37.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.658+0000 7fc711b87640 1 -- 172.21.15.49:0/3698472254 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc70c1ae920 con 0x7fc70c10b150 2024-03-18T00:57:37.659 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.658+0000 7fc711b87640 1 -- 172.21.15.49:0/3698472254 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc70c1aee70 con 0x7fc70c10b150 2024-03-18T00:57:37.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.658+0000 7fc708ff9640 1 -- 172.21.15.49:0/3698472254 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc70400cc10 con 0x7fc70c10b150 2024-03-18T00:57:37.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.658+0000 7fc708ff9640 1 -- 172.21.15.49:0/3698472254 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7040076a0 con 0x7fc70c10b150 2024-03-18T00:57:37.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.670+0000 7fc708ff9640 1 -- 172.21.15.49:0/3698472254 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc70401c070 con 0x7fc70c10b150 2024-03-18T00:57:37.671 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.672+0000 7fc708ff9640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6f40a2a00 0x7fc6f40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.673 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.673+0000 7fc708ff9640 1 -- 172.21.15.49:0/3698472254 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc7040cdff0 con 0x7fc70c10b150 2024-03-18T00:57:37.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.676+0000 7fc714774740 1 Processor -- start 2024-03-18T00:57:37.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.678+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6f40a2a00 0x7fc6f40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.683+0000 7fc714774740 1 -- start start 2024-03-18T00:57:37.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.683+0000 7fc714774740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b11c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.684+0000 7fc714774740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b12240 0x555cd6b16d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.684+0000 7fc714774740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b173b0 0x555cd6b19840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.684+0000 7fc714774740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555cd6a76490 con 0x555cd6b173b0 2024-03-18T00:57:37.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.684+0000 7fc714774740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555cd6a1ec90 con 0x555cd6b12240 2024-03-18T00:57:37.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.684+0000 7fc714774740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555cd6a72530 con 0x555cd6b11850 2024-03-18T00:57:37.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.685+0000 7fc70b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b11c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.685+0000 7fc70b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b11c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:41958/0 (socket says 172.21.15.49:41958) 2024-03-18T00:57:37.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.685+0000 7fc70b7fe640 1 -- 172.21.15.49:0/1258121939 learned_addr learned my addr 172.21.15.49:0/1258121939 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:37.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.685+0000 7fc70affd640 1 --2- 172.21.15.49:0/1258121939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b12240 0x555cd6b16d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.686+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6f40a2a00 0x7fc6f40a4ec0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc700009890 tx=0x7fc700008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.686+0000 7fc70b7fe640 1 -- 172.21.15.49:0/1258121939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b12240 msgr2=0x555cd6b16d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.686+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/1258121939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b12240 0x555cd6b16d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.686+0000 7fc70b7fe640 1 -- 172.21.15.49:0/1258121939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b173b0 msgr2=0x555cd6b19840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.686+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/1258121939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b173b0 0x555cd6b19840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.686+0000 7fc70b7fe640 1 -- 172.21.15.49:0/1258121939 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cd6ad9500 con 0x555cd6b11850 2024-03-18T00:57:37.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.687+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/1258121939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b11c70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc70000d860 tx=0x7fc700010000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=340724af31d721e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.721+0000 7fc6f1ffb640 1 -- 172.21.15.49:0/1258121939 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc700024070 con 0x555cd6b11850 2024-03-18T00:57:37.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.722+0000 7fc6f1ffb640 1 -- 172.21.15.49:0/1258121939 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc700014bc0 con 0x555cd6b11850 2024-03-18T00:57:37.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.722+0000 7fc6f1ffb640 1 -- 172.21.15.49:0/1258121939 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc700012640 con 0x555cd6b11850 2024-03-18T00:57:37.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.727+0000 7fc714774740 1 -- 172.21.15.49:0/1258121939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 msgr2=0x555cd6b11c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.727+0000 7fc714774740 1 --2- 172.21.15.49:0/1258121939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b11c70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc70000d860 tx=0x7fc700010000 comp rx=0 tx=0).stop 2024-03-18T00:57:37.734 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.734+0000 7fc714774740 1 -- 172.21.15.49:0/1258121939 shutdown_connections 2024-03-18T00:57:37.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.734+0000 7fc714774740 1 --2- 172.21.15.49:0/1258121939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b12240 0x555cd6b16d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.734+0000 7fc714774740 1 --2- 172.21.15.49:0/1258121939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b173b0 0x555cd6b19840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.734+0000 7fc714774740 1 --2- 172.21.15.49:0/1258121939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b11c70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.736 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.734+0000 7fc714774740 1 -- 172.21.15.49:0/1258121939 >> 172.21.15.49:0/1258121939 conn(0x555cd6b0cdc0 msgr2=0x555cd6b0d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.736+0000 7fc714774740 1 -- 172.21.15.49:0/1258121939 shutdown_connections 2024-03-18T00:57:37.737 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.736+0000 7fc714774740 1 -- 172.21.15.49:0/1258121939 wait complete. 2024-03-18T00:57:37.738 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.737+0000 7fc714774740 1 -- start start 2024-03-18T00:57:37.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.738+0000 7fc714774740 1 Processor -- start 2024-03-18T00:57:37.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.739+0000 7fc714774740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b11850 0x555cd6b1bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.740+0000 7fc714774740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.740+0000 7fc714774740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b173b0 0x555cd6b23320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.740+0000 7fc714774740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555cd6b1a360 con 0x555cd6b12240 2024-03-18T00:57:37.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.740+0000 7fc714774740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555cd6a32230 con 0x555cd6b11850 2024-03-18T00:57:37.740 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.740+0000 7fc714774740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555cd6b1a4e0 con 0x555cd6b173b0 2024-03-18T00:57:37.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.745+0000 7fc70affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.745+0000 7fc70affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47912/0 (socket says 172.21.15.49:47912) 2024-03-18T00:57:37.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.745+0000 7fc70affd640 1 -- 172.21.15.49:0/645766221 learned_addr learned my addr 172.21.15.49:0/645766221 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:37.745 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.745+0000 7fc70bfff640 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b173b0 0x555cd6b23320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.745+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b11850 0x555cd6b1bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.746 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.745+0000 7fc714774740 10 client.0 _ll_register_callbacks cb 0x555cd6a6e010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:37.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.754+0000 7fc70affd640 1 -- 172.21.15.49:0/645766221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b173b0 msgr2=0x555cd6b23320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.754+0000 7fc70affd640 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b173b0 0x555cd6b23320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.754+0000 7fc70affd640 1 -- 172.21.15.49:0/645766221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b11850 msgr2=0x555cd6b1bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.754+0000 7fc70affd640 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b11850 0x555cd6b1bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.754+0000 7fc70affd640 1 -- 172.21.15.49:0/645766221 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cd6a41100 con 0x555cd6b12240 2024-03-18T00:57:37.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.754+0000 7fc70bfff640 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b173b0 0x555cd6b23320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:57:37.754 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.754+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b11850 0x555cd6b1bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:57:37.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.757+0000 7fc70affd640 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc70400c7d0 tx=0x7fc70409a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910ca5f4e696b826 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.757+0000 7fc6eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:37.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.760+0000 7fc6eaffd640 1 -- 172.21.15.49:0/645766221 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7040a30b0 con 0x555cd6b12240 2024-03-18T00:57:37.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.760+0000 7fc6eaffd640 1 -- 172.21.15.49:0/645766221 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc70400cdc0 con 0x555cd6b12240 2024-03-18T00:57:37.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.760+0000 7fc714774740 1 -- 172.21.15.49:0/645766221 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc700014bc0 con 0x555cd6b12240 2024-03-18T00:57:37.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.761+0000 7fc6eaffd640 1 -- 172.21.15.49:0/645766221 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc704096f80 con 0x555cd6b12240 2024-03-18T00:57:37.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.761+0000 7fc714774740 10 client.17226 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:37.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.761+0000 7fc714774740 1 -- 172.21.15.49:0/645766221 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555cd678a3f0 con 0x555cd6b12240 2024-03-18T00:57:37.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.764+0000 7fc6eaffd640 1 -- 172.21.15.49:0/645766221 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc700014bc0 con 0x555cd6b12240 2024-03-18T00:57:37.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.764+0000 7fc6eaffd640 1 -- 172.21.15.49:0/645766221 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc7040997d0 con 0x555cd6b12240 2024-03-18T00:57:37.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.764+0000 7fc714774740 10 client.17226 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:37.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.764+0000 7fc714774740 10 client.17226 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:57:37.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.764+0000 7fc714774740 20 client.17226 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:37.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.764+0000 7fc714774740 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555cd6b27ab0 0x555cd6b29f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.764+0000 7fc714774740 4 client.17226 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:37.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.764+0000 7fc714774740 1 -- 172.21.15.49:0/645766221 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555cd6a839d0 con 0x555cd6b27ab0 2024-03-18T00:57:37.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.765+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555cd6b27ab0 0x555cd6b29f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.768+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555cd6b27ab0 0x555cd6b29f50 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=305a502cb122a742 server_cookie=62f01c2177cef3ea in_seq=0 out_seq=0 2024-03-18T00:57:37.771 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.768+0000 7fc6eaffd640 10 client.17226 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:57:37.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.771+0000 7fc6eaffd640 1 -- 172.21.15.49:0/645766221 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555cd6a839d0 con 0x555cd6b27ab0 2024-03-18T00:57:37.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.771+0000 7fc6eaffd640 10 client.17226 handle_command_reply: tid=0 2024-03-18T00:57:37.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.771+0000 7fc714774740 1 client.17226 shutdown 2024-03-18T00:57:37.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.773+0000 7fc714774740 1 -- 172.21.15.49:0/645766221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 msgr2=0x555cd6b1c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.773 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.773+0000 7fc714774740 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc70400c7d0 tx=0x7fc70409a910 comp rx=0 tx=0).stop 2024-03-18T00:57:37.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.778+0000 7fc714774740 1 -- 172.21.15.49:0/645766221 shutdown_connections 2024-03-18T00:57:37.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.778+0000 7fc714774740 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555cd6b27ab0 0x555cd6b29f50 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.778+0000 7fc714774740 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b173b0 0x555cd6b23320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.778+0000 7fc714774740 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.778+0000 7fc714774740 1 --2- 172.21.15.49:0/645766221 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b11850 0x555cd6b1bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.778+0000 7fc714774740 1 -- 172.21.15.49:0/645766221 >> 172.21.15.49:0/645766221 conn(0x555cd6b0cdc0 msgr2=0x555cd6b0bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.778+0000 7fc6eaffd640 0 client.17226 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:57:37.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.778+0000 7fc714774740 1 -- 172.21.15.49:0/645766221 shutdown_connections 2024-03-18T00:57:37.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.780+0000 7fc714774740 1 -- 172.21.15.49:0/645766221 wait complete. 2024-03-18T00:57:37.781 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.781+0000 7fc714774740 20 client.17226 trim_cache size 0 max 16384 2024-03-18T00:57:37.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.829+0000 7fc714774740 1 Processor -- start 2024-03-18T00:57:37.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.829+0000 7fc714774740 1 -- start start 2024-03-18T00:57:37.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.830+0000 7fc714774740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b1bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.830+0000 7fc714774740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.830+0000 7fc714774740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 0x555cd6b232e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.831+0000 7fc714774740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f800e650 con 0x555cd6b12240 2024-03-18T00:57:37.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.831+0000 7fc714774740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f8010190 con 0x555cd6b173b0 2024-03-18T00:57:37.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.831+0000 7fc714774740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f800ed20 con 0x555cd6b11850 2024-03-18T00:57:37.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.831+0000 7fc70affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.831+0000 7fc70affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47924/0 (socket says 172.21.15.49:47924) 2024-03-18T00:57:37.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.831+0000 7fc70affd640 1 -- 172.21.15.49:0/2083410024 learned_addr learned my addr 172.21.15.49:0/2083410024 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:37.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.832+0000 7fc70affd640 1 -- 172.21.15.49:0/2083410024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 msgr2=0x555cd6b1bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.832+0000 7fc70affd640 1 --2- 172.21.15.49:0/2083410024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b1bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.832+0000 7fc70affd640 1 -- 172.21.15.49:0/2083410024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 msgr2=0x555cd6b232e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.832+0000 7fc70affd640 1 --2- 172.21.15.49:0/2083410024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 0x555cd6b232e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.832+0000 7fc70affd640 1 -- 172.21.15.49:0/2083410024 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555cd68547d0 con 0x555cd6b12240 2024-03-18T00:57:37.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.832+0000 7fc70affd640 1 --2- 172.21.15.49:0/2083410024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc70409a610 tx=0x7fc70409a450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cccff6599f49c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.833+0000 7fc6f17fa640 1 -- 172.21.15.49:0/2083410024 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704097120 con 0x555cd6b12240 2024-03-18T00:57:37.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.833+0000 7fc6f17fa640 1 -- 172.21.15.49:0/2083410024 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7040ab070 con 0x555cd6b12240 2024-03-18T00:57:37.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.833+0000 7fc6f17fa640 1 -- 172.21.15.49:0/2083410024 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704099d60 con 0x555cd6b12240 2024-03-18T00:57:37.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.835+0000 7fc714774740 1 -- 172.21.15.49:0/2083410024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 msgr2=0x555cd6b1c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.835+0000 7fc714774740 1 --2- 172.21.15.49:0/2083410024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fc70409a610 tx=0x7fc70409a450 comp rx=0 tx=0).stop 2024-03-18T00:57:37.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.835+0000 7fc714774740 1 -- 172.21.15.49:0/2083410024 shutdown_connections 2024-03-18T00:57:37.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.835+0000 7fc714774740 1 --2- 172.21.15.49:0/2083410024 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 0x555cd6b232e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.835+0000 7fc714774740 1 --2- 172.21.15.49:0/2083410024 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1c2e0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.835+0000 7fc714774740 1 --2- 172.21.15.49:0/2083410024 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b1bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.835+0000 7fc714774740 1 -- 172.21.15.49:0/2083410024 >> 172.21.15.49:0/2083410024 conn(0x555cd6b0cdc0 msgr2=0x555cd6b137d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.835+0000 7fc714774740 1 -- 172.21.15.49:0/2083410024 shutdown_connections 2024-03-18T00:57:37.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.836+0000 7fc714774740 1 -- 172.21.15.49:0/2083410024 wait complete. 2024-03-18T00:57:37.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.836+0000 7fc714774740 1 -- start start 2024-03-18T00:57:37.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.838+0000 7fc714774740 1 Processor -- start 2024-03-18T00:57:37.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.838+0000 7fc714774740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b1f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.839+0000 7fc714774740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.839+0000 7fc714774740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 0x555cd6b22b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.839+0000 7fc714774740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f800fac0 con 0x555cd6b12240 2024-03-18T00:57:37.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.839+0000 7fc714774740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f800f3f0 con 0x555cd6b173b0 2024-03-18T00:57:37.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.839+0000 7fc714774740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f800c630 con 0x555cd6b11850 2024-03-18T00:57:37.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.840+0000 7fc714774740 10 client.0 _ll_register_callbacks cb 0x7fc704009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:37.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.840+0000 7fc70bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 0x555cd6b22b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.840+0000 7fc70affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.840+0000 7fc70affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47934/0 (socket says 172.21.15.49:47934) 2024-03-18T00:57:37.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.840+0000 7fc70affd640 1 -- 172.21.15.49:0/3059218997 learned_addr learned my addr 172.21.15.49:0/3059218997 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:37.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.841+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b1f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.842+0000 7fc70affd640 1 -- 172.21.15.49:0/3059218997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 msgr2=0x555cd6b1f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.842+0000 7fc70affd640 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b1f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.842+0000 7fc70affd640 1 -- 172.21.15.49:0/3059218997 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 msgr2=0x555cd6b22b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:37.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.842+0000 7fc70affd640 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 0x555cd6b22b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.842+0000 7fc70affd640 1 -- 172.21.15.49:0/3059218997 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555cd6b230f0 con 0x555cd6b12240 2024-03-18T00:57:37.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.842+0000 7fc70bfff640 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 0x555cd6b22b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:57:37.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.842+0000 7fc70affd640 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1fd60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc70409ec20 tx=0x7fc7040092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0cea7af3b8ebe51 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:37.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.842+0000 7fc6eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:37.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.843+0000 7fc6eaffd640 1 -- 172.21.15.49:0/3059218997 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc704097120 con 0x555cd6b12240 2024-03-18T00:57:37.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.843+0000 7fc6eaffd640 1 -- 172.21.15.49:0/3059218997 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc7040a2c20 con 0x555cd6b12240 2024-03-18T00:57:37.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.843+0000 7fc714774740 1 -- 172.21.15.49:0/3059218997 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7040ab070 con 0x555cd6b12240 2024-03-18T00:57:37.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.844+0000 7fc6eaffd640 1 -- 172.21.15.49:0/3059218997 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc70407f050 con 0x555cd6b12240 2024-03-18T00:57:37.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.844+0000 7fc714774740 10 client.17232 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:37.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.844+0000 7fc714774740 1 -- 172.21.15.49:0/3059218997 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555cd6b8c960 con 0x555cd6b12240 2024-03-18T00:57:37.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.845+0000 7fc6eaffd640 1 -- 172.21.15.49:0/3059218997 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc7040ab070 con 0x555cd6b12240 2024-03-18T00:57:37.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.846+0000 7fc6eaffd640 1 -- 172.21.15.49:0/3059218997 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc70409fbc0 con 0x555cd6b12240 2024-03-18T00:57:37.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.846+0000 7fc714774740 10 client.17232 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:37.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.846+0000 7fc714774740 10 client.17232 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:57:37.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.846+0000 7fc714774740 20 client.17232 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:37.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.846+0000 7fc714774740 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555cd6b8d140 0x555cd6b8f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:37.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.846+0000 7fc714774740 4 client.17232 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:57:37.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.846+0000 7fc714774740 1 -- 172.21.15.49:0/3059218997 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555cd6a7dd20 con 0x555cd6b8d140 2024-03-18T00:57:37.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.847+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555cd6b8d140 0x555cd6b8f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:37.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.848+0000 7fc70b7fe640 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555cd6b8d140 0x555cd6b8f560 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=af7c2718c1e797c6 server_cookie=98ee187d8b5db530 in_seq=0 out_seq=0 2024-03-18T00:57:37.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.848+0000 7fc6eaffd640 10 client.17232 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:57:37.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.849+0000 7fc6eaffd640 1 -- 172.21.15.49:0/3059218997 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555cd6a7dd20 con 0x555cd6b8d140 2024-03-18T00:57:37.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.849+0000 7fc6eaffd640 10 client.17232 handle_command_reply: tid=0 2024-03-18T00:57:37.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.849+0000 7fc714774740 1 client.17232 shutdown 2024-03-18T00:57:37.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.850+0000 7fc714774740 1 -- 172.21.15.49:0/3059218997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 msgr2=0x555cd6b1fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.850+0000 7fc714774740 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1fd60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fc70409ec20 tx=0x7fc7040092d0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.850+0000 7fc714774740 1 -- 172.21.15.49:0/3059218997 shutdown_connections 2024-03-18T00:57:37.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.850+0000 7fc714774740 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555cd6b8d140 0x555cd6b8f560 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.851+0000 7fc714774740 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555cd6b173b0 0x555cd6b22b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.851+0000 7fc714774740 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555cd6b12240 0x555cd6b1fd60 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.851+0000 7fc714774740 1 --2- 172.21.15.49:0/3059218997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555cd6b11850 0x555cd6b1f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.851+0000 7fc714774740 1 -- 172.21.15.49:0/3059218997 >> 172.21.15.49:0/3059218997 conn(0x555cd6b0cdc0 msgr2=0x555cd6b137d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.851+0000 7fc6eaffd640 0 client.17232 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:57:37.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.851+0000 7fc714774740 1 -- 172.21.15.49:0/3059218997 shutdown_connections 2024-03-18T00:57:37.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.851+0000 7fc714774740 1 -- 172.21.15.49:0/3059218997 wait complete. 2024-03-18T00:57:37.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.851+0000 7fc714774740 20 client.17232 trim_cache size 0 max 16384 2024-03-18T00:57:37.852 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:57:37.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.853+0000 7fc711b87640 1 -- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6f40a2a00 msgr2=0x7fc6f40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.853+0000 7fc711b87640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6f40a2a00 0x7fc6f40a4ec0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc700009890 tx=0x7fc700008040 comp rx=0 tx=0).stop 2024-03-18T00:57:37.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.853+0000 7fc711b87640 1 -- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc70c10b150 msgr2=0x7fc70c1a6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:37.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.853+0000 7fc711b87640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc70c10b150 0x7fc70c1a6fa0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc70400ba50 tx=0x7fc70400bf10 comp rx=0 tx=0).stop 2024-03-18T00:57:37.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.854+0000 7fc711b87640 1 -- 172.21.15.49:0/3698472254 shutdown_connections 2024-03-18T00:57:37.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.854+0000 7fc711b87640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc6f40a2a00 0x7fc6f40a4ec0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.854 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.854+0000 7fc711b87640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc70c111f20 0x7fc70c1adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.854+0000 7fc711b87640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc70c10b150 0x7fc70c1a6fa0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.854+0000 7fc711b87640 1 --2- 172.21.15.49:0/3698472254 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc70c1087e0 0x7fc70c1a6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:37.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.854+0000 7fc711b87640 1 -- 172.21.15.49:0/3698472254 >> 172.21.15.49:0/3698472254 conn(0x7fc70c06e640 msgr2=0x7fc70c109e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:37.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.854+0000 7fc711b87640 1 -- 172.21.15.49:0/3698472254 shutdown_connections 2024-03-18T00:57:37.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:37.854+0000 7fc711b87640 1 -- 172.21.15.49:0/3698472254 wait complete. 2024-03-18T00:57:37.890 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:57:37.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:37 smithi107 ceph-mon[36270]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 874 MiB/s wr, 1.65k op/s 2024-03-18T00:57:37.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:37 smithi106 ceph-mon[35899]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 874 MiB/s wr, 1.65k op/s 2024-03-18T00:57:38.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.158+0000 7f3d60b1d640 1 -- 172.21.15.49:0/2128974367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0741e0 msgr2=0x7f3d5c074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.158+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/2128974367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0741e0 0x7f3d5c074640 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009900 tx=0x7f3d5002bcd0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.159+0000 7f3d60b1d640 1 -- 172.21.15.49:0/2128974367 shutdown_connections 2024-03-18T00:57:38.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.159+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/2128974367 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d5c109680 0x7f3d5c10bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.159+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/2128974367 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0741e0 0x7f3d5c074640 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.159+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/2128974367 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d5c0757c0 0x7f3d5c073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.159+0000 7f3d60b1d640 1 -- 172.21.15.49:0/2128974367 >> 172.21.15.49:0/2128974367 conn(0x7f3d5c0fcd40 msgr2=0x7f3d5c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.159+0000 7f3d60b1d640 1 -- 172.21.15.49:0/2128974367 shutdown_connections 2024-03-18T00:57:38.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.159+0000 7f3d60b1d640 1 -- 172.21.15.49:0/2128974367 wait complete. 2024-03-18T00:57:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.159+0000 7f3d60b1d640 1 Processor -- start 2024-03-18T00:57:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.159+0000 7f3d60b1d640 1 -- start start 2024-03-18T00:57:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d60b1d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d5c0741e0 0x7f3d5c199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d60b1d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0757c0 0x7f3d5c19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d60b1d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d5c109680 0x7f3d5c1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d60b1d640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c10f6e0 con 0x7f3d5c0757c0 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d60b1d640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c10f560 con 0x7f3d5c109680 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d60b1d640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c10f860 con 0x7f3d5c0741e0 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d59d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0757c0 0x7f3d5c19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d59d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0757c0 0x7f3d5c19a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47962/0 (socket says 172.21.15.49:47962) 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d59d9b640 1 -- 172.21.15.49:0/1222284045 learned_addr learned my addr 172.21.15.49:0/1222284045 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d59d9b640 1 -- 172.21.15.49:0/1222284045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d5c0741e0 msgr2=0x7f3d5c199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d59d9b640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d5c0741e0 0x7f3d5c199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d59d9b640 1 -- 172.21.15.49:0/1222284045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d5c109680 msgr2=0x7f3d5c1a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d59d9b640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d5c109680 0x7f3d5c1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.160+0000 7f3d59d9b640 1 -- 172.21.15.49:0/1222284045 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c1a19b0 con 0x7f3d5c0757c0 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.161+0000 7f3d59d9b640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0757c0 0x7f3d5c19a330 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3d50005ab0 tx=0x7f3d50005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.161+0000 7f3d4b7fe640 1 -- 172.21.15.49:0/1222284045 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50041070 con 0x7f3d5c0757c0 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.161+0000 7f3d4b7fe640 1 -- 172.21.15.49:0/1222284045 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d500054e0 con 0x7f3d5c0757c0 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.161+0000 7f3d60b1d640 1 -- 172.21.15.49:0/1222284045 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5c1a1c40 con 0x7f3d5c0757c0 2024-03-18T00:57:38.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.161+0000 7f3d4b7fe640 1 -- 172.21.15.49:0/1222284045 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50034440 con 0x7f3d5c0757c0 2024-03-18T00:57:38.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.161+0000 7f3d60b1d640 1 -- 172.21.15.49:0/1222284045 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5c1a2180 con 0x7f3d5c0757c0 2024-03-18T00:57:38.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.164+0000 7f3d637df740 1 Processor -- start 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.164+0000 7f3d637df740 1 -- start start 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d637df740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235d7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d637df740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235dcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d637df740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235dd3b0 0x555b235df840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d637df740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555b2353c490 con 0x555b235d8240 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d637df740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555b234e4c90 con 0x555b235dd3b0 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d637df740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555b23538530 con 0x555b235d7850 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d5a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235dcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d5a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235dcd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47974/0 (socket says 172.21.15.49:47974) 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d5a59c640 1 -- 172.21.15.49:0/40228614 learned_addr learned my addr 172.21.15.49:0/40228614 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d5ad9d640 1 --2- 172.21.15.49:0/40228614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235d7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.165+0000 7f3d4b7fe640 1 -- 172.21.15.49:0/1222284045 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3d50044070 con 0x7f3d5c0757c0 2024-03-18T00:57:38.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.166+0000 7f3d5a59c640 1 -- 172.21.15.49:0/40228614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 msgr2=0x555b235d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.166+0000 7f3d5a59c640 1 --2- 172.21.15.49:0/40228614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235d7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.166+0000 7f3d5a59c640 1 -- 172.21.15.49:0/40228614 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235dd3b0 msgr2=0x555b235df840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.166+0000 7f3d5a59c640 1 --2- 172.21.15.49:0/40228614 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235dd3b0 0x555b235df840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.166+0000 7f3d5a59c640 1 -- 172.21.15.49:0/40228614 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b2359f500 con 0x555b235d8240 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.166+0000 7f3d5a59c640 1 --2- 172.21.15.49:0/40228614 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235dcd40 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00bae0 tx=0x7f3d4c00d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2493604f7ce32e4c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.166+0000 7f3d48ff9640 1 -- 172.21.15.49:0/40228614 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c016070 con 0x555b235d8240 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.166+0000 7f3d48ff9640 1 -- 172.21.15.49:0/40228614 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d4c005500 con 0x555b235d8240 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.166+0000 7f3d48ff9640 1 -- 172.21.15.49:0/40228614 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c010460 con 0x555b235d8240 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.167+0000 7f3d4b7fe640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d440a2880 0x7f3d440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.167+0000 7f3d4b7fe640 1 -- 172.21.15.49:0/1222284045 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3d500f0b70 con 0x7f3d5c0757c0 2024-03-18T00:57:38.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.167+0000 7f3d5ad9d640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d440a2880 0x7f3d440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.167+0000 7f3d5ad9d640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d440a2880 0x7f3d440a4d40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x555b235d8aa0 tx=0x7f3d5400d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.167+0000 7f3d637df740 1 -- 172.21.15.49:0/40228614 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 msgr2=0x555b235dcd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.167+0000 7f3d637df740 1 --2- 172.21.15.49:0/40228614 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235dcd40 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00bae0 tx=0x7f3d4c00d2e0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.168+0000 7f3d637df740 1 -- 172.21.15.49:0/40228614 shutdown_connections 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.168+0000 7f3d637df740 1 --2- 172.21.15.49:0/40228614 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235dd3b0 0x555b235df840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.168+0000 7f3d637df740 1 --2- 172.21.15.49:0/40228614 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235dcd40 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.168+0000 7f3d637df740 1 --2- 172.21.15.49:0/40228614 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235d7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.168+0000 7f3d637df740 1 -- 172.21.15.49:0/40228614 >> 172.21.15.49:0/40228614 conn(0x555b235d2dc0 msgr2=0x555b235d31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.168+0000 7f3d637df740 1 -- 172.21.15.49:0/40228614 shutdown_connections 2024-03-18T00:57:38.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.168+0000 7f3d637df740 1 -- 172.21.15.49:0/40228614 wait complete. 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.168+0000 7f3d637df740 1 -- start start 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.169+0000 7f3d637df740 1 Processor -- start 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.169+0000 7f3d637df740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d7850 0x555b235e1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d637df740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 0x555b235e2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d5a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d7850 0x555b235e1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d5a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d7850 0x555b235e1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47988/0 (socket says 172.21.15.49:47988) 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d5a59c640 1 -- 172.21.15.49:0/616062386 learned_addr learned my addr 172.21.15.49:0/616062386 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d637df740 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235dd3b0 0x555b235e9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555b235e0360 con 0x555b235d7850 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555b234f8230 con 0x555b235d8240 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555b235e04e0 con 0x555b235dd3b0 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d637df740 10 client.0 _ll_register_callbacks cb 0x555b23534010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d5a59c640 1 -- 172.21.15.49:0/616062386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235dd3b0 msgr2=0x555b235e9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d5a59c640 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235dd3b0 0x555b235e9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d5a59c640 1 -- 172.21.15.49:0/616062386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 msgr2=0x555b235e2330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:57:38.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d5a59c640 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 0x555b235e2330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.170+0000 7f3d5a59c640 1 -- 172.21.15.49:0/616062386 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b23507100 con 0x555b235d7850 2024-03-18T00:57:38.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.171+0000 7f3d5a59c640 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d7850 0x555b235e1df0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c004700 tx=0x7f3d4c0049d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0cfe5f17edda83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.171+0000 7f3d41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:38.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.171+0000 7f3d41ffb640 1 -- 172.21.15.49:0/616062386 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c016030 con 0x555b235d7850 2024-03-18T00:57:38.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.171+0000 7f3d41ffb640 1 -- 172.21.15.49:0/616062386 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3d4c005180 con 0x555b235d7850 2024-03-18T00:57:38.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.171+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d4c005500 con 0x555b235d7850 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.171+0000 7f3d41ffb640 1 -- 172.21.15.49:0/616062386 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3d4c00dab0 con 0x555b235d7850 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.171+0000 7f3d637df740 10 client.17244 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.171+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555b232503f0 con 0x555b235d7850 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.172+0000 7f3d41ffb640 1 -- 172.21.15.49:0/616062386 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d4c005500 con 0x555b235d7850 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.172+0000 7f3d41ffb640 1 -- 172.21.15.49:0/616062386 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3d4c022070 con 0x555b235d7850 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.172+0000 7f3d637df740 10 client.17244 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.172+0000 7f3d637df740 10 client.17244 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.172+0000 7f3d637df740 20 client.17244 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.172+0000 7f3d637df740 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x555b235eda40 0x555b235eff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.172+0000 7f3d637df740 4 client.17244 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.172+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b235499d0 con 0x555b235eda40 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.172+0000 7f3d59d9b640 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x555b235eda40 0x555b235eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.173+0000 7f3d59d9b640 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x555b235eda40 0x555b235eff00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9dabae626ddf3d3f server_cookie=d0f6b687a6bb8080 in_seq=0 out_seq=0 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.173+0000 7f3d41ffb640 10 client.17244 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:57:38.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d41ffb640 1 -- 172.21.15.49:0/616062386 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555b235499d0 con 0x555b235eda40 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d41ffb640 10 client.17244 handle_command_reply: tid=0 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 client.17244 shutdown 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d7850 msgr2=0x555b235e1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d7850 0x555b235e1df0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c004700 tx=0x7f3d4c0049d0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 shutdown_connections 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x555b235eda40 0x555b235eff00 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 0x555b235e2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235dd3b0 0x555b235e9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 --2- 172.21.15.49:0/616062386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d7850 0x555b235e1df0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 >> 172.21.15.49:0/616062386 conn(0x555b235d2dc0 msgr2=0x555b235d1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d41ffb640 0 client.17244 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:57:38.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.174+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 shutdown_connections 2024-03-18T00:57:38.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.175+0000 7f3d637df740 1 -- 172.21.15.49:0/616062386 wait complete. 2024-03-18T00:57:38.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.175+0000 7f3d637df740 20 client.17244 trim_cache size 0 max 16384 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d637df740 1 Processor -- start 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d637df740 1 -- start start 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d637df740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235d6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d637df740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235e9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d637df740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235dd3b0 0x555b235e1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d637df740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1ab020 con 0x555b235d8240 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d637df740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1acb60 con 0x555b235dd3b0 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d637df740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1ab6f0 con 0x555b235d7850 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d59d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d59d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235e9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48002/0 (socket says 172.21.15.49:48002) 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d59d9b640 1 -- 172.21.15.49:0/3004232916 learned_addr learned my addr 172.21.15.49:0/3004232916 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.192+0000 7f3d5ad9d640 1 --2- 172.21.15.49:0/3004232916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235dd3b0 0x555b235e1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d5a59c640 1 --2- 172.21.15.49:0/3004232916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235d6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d59d9b640 1 -- 172.21.15.49:0/3004232916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 msgr2=0x555b235d6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d59d9b640 1 --2- 172.21.15.49:0/3004232916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235d6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d59d9b640 1 -- 172.21.15.49:0/3004232916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235dd3b0 msgr2=0x555b235e1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d59d9b640 1 --2- 172.21.15.49:0/3004232916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235dd3b0 0x555b235e1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d59d9b640 1 -- 172.21.15.49:0/3004232916 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b2331a7d0 con 0x555b235d8240 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d59d9b640 1 --2- 172.21.15.49:0/3004232916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235e9330 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f3d500b98c0 tx=0x7f3d50009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418d1b5f27ccd04c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d43fff640 1 -- 172.21.15.49:0/3004232916 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5004b040 con 0x555b235d8240 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d43fff640 1 -- 172.21.15.49:0/3004232916 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d50005690 con 0x555b235d8240 2024-03-18T00:57:38.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.193+0000 7f3d43fff640 1 -- 172.21.15.49:0/3004232916 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d500c5c90 con 0x555b235d8240 2024-03-18T00:57:38.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.194+0000 7f3d637df740 1 -- 172.21.15.49:0/3004232916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 msgr2=0x555b235e9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.194+0000 7f3d637df740 1 --2- 172.21.15.49:0/3004232916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235e9330 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f3d500b98c0 tx=0x7f3d50009290 comp rx=0 tx=0).stop 2024-03-18T00:57:38.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.194+0000 7f3d637df740 1 -- 172.21.15.49:0/3004232916 shutdown_connections 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.194+0000 7f3d637df740 1 --2- 172.21.15.49:0/3004232916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235dd3b0 0x555b235e1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.194+0000 7f3d637df740 1 --2- 172.21.15.49:0/3004232916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235d8240 0x555b235e9330 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.194+0000 7f3d637df740 1 --2- 172.21.15.49:0/3004232916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235d6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.194+0000 7f3d637df740 1 -- 172.21.15.49:0/3004232916 >> 172.21.15.49:0/3004232916 conn(0x555b235d2dc0 msgr2=0x555b235d6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.194+0000 7f3d637df740 1 -- 172.21.15.49:0/3004232916 shutdown_connections 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.194+0000 7f3d637df740 1 -- 172.21.15.49:0/3004232916 wait complete. 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d637df740 1 -- start start 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d637df740 1 Processor -- start 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d637df740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235e62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d637df740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 0x555b235e6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d637df740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235dd3b0 0x555b23651df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d637df740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1ac490 con 0x555b235dd3b0 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d637df740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1abdc0 con 0x555b235d8240 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d637df740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c1a9000 con 0x555b235d7850 2024-03-18T00:57:38.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d637df740 10 client.0 _ll_register_callbacks cb 0x7f3d54009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d5ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235dd3b0 0x555b23651df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d5ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235dd3b0 0x555b23651df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48008/0 (socket says 172.21.15.49:48008) 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d59d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 0x555b235e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d5a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235e62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.195+0000 7f3d5ad9d640 1 -- 172.21.15.49:0/2381611680 learned_addr learned my addr 172.21.15.49:0/2381611680 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d5ad9d640 1 -- 172.21.15.49:0/2381611680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 msgr2=0x555b235e62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d5ad9d640 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235e62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d5ad9d640 1 -- 172.21.15.49:0/2381611680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 msgr2=0x555b235e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d5ad9d640 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 0x555b235e6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d5ad9d640 1 -- 172.21.15.49:0/2381611680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b23652330 con 0x555b235dd3b0 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d59d9b640 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 0x555b235e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d5ad9d640 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235dd3b0 0x555b23651df0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f3d54016dc0 tx=0x7f3d540095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88266f2585fea88a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:38.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d41ffb640 1 -- 172.21.15.49:0/2381611680 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54026020 con 0x555b235dd3b0 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d41ffb640 1 -- 172.21.15.49:0/2381611680 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3d54022640 con 0x555b235dd3b0 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d637df740 1 -- 172.21.15.49:0/2381611680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b23543d20 con 0x555b235dd3b0 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d41ffb640 1 -- 172.21.15.49:0/2381611680 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3d54031050 con 0x555b235dd3b0 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d637df740 10 client.17250 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d637df740 1 -- 172.21.15.49:0/2381611680 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555b236536a0 con 0x555b235dd3b0 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.196+0000 7f3d41ffb640 1 -- 172.21.15.49:0/2381611680 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x555b23543d20 con 0x555b235dd3b0 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d41ffb640 1 -- 172.21.15.49:0/2381611680 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3d54021800 con 0x555b235dd3b0 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d637df740 10 client.17250 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d637df740 10 client.17250 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d637df740 20 client.17250 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d637df740 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x555b23654100 0x555b23656520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d637df740 4 client.17250 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:57:38.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d637df740 1 -- 172.21.15.49:0/2381611680 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555b234c16a0 con 0x555b23654100 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d5a59c640 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x555b23654100 0x555b23656520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d5a59c640 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x555b23654100 0x555b23656520 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75afcd1c7ac8a0b3 server_cookie=b01b86e47dfead62 in_seq=0 out_seq=0 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.197+0000 7f3d41ffb640 10 client.17250 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d41ffb640 1 -- 172.21.15.49:0/2381611680 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555b234c16a0 con 0x555b23654100 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d41ffb640 10 client.17250 handle_command_reply: tid=0 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 client.17250 shutdown 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 -- 172.21.15.49:0/2381611680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235dd3b0 msgr2=0x555b23651df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235dd3b0 0x555b23651df0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f3d54016dc0 tx=0x7f3d540095d0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 -- 172.21.15.49:0/2381611680 shutdown_connections 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x555b23654100 0x555b23656520 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555b235d8240 0x555b235e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555b235dd3b0 0x555b23651df0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 --2- 172.21.15.49:0/2381611680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555b235d7850 0x555b235e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 -- 172.21.15.49:0/2381611680 >> 172.21.15.49:0/2381611680 conn(0x555b235d2dc0 msgr2=0x555b235d68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d41ffb640 0 client.17250 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:57:38.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 -- 172.21.15.49:0/2381611680 shutdown_connections 2024-03-18T00:57:38.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 1 -- 172.21.15.49:0/2381611680 wait complete. 2024-03-18T00:57:38.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.198+0000 7f3d637df740 20 client.17250 trim_cache size 0 max 16384 2024-03-18T00:57:38.203 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:57:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 -- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d440a2880 msgr2=0x7f3d440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d440a2880 0x7f3d440a4d40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x555b235d8aa0 tx=0x7f3d5400d040 comp rx=0 tx=0).stop 2024-03-18T00:57:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 -- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0757c0 msgr2=0x7f3d5c19a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0757c0 0x7f3d5c19a330 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f3d50005ab0 tx=0x7f3d50005de0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 -- 172.21.15.49:0/1222284045 shutdown_connections 2024-03-18T00:57:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3d440a2880 0x7f3d440a4d40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3d5c109680 0x7f3d5c1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3d5c0757c0 0x7f3d5c19a330 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 --2- 172.21.15.49:0/1222284045 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3d5c0741e0 0x7f3d5c199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 -- 172.21.15.49:0/1222284045 >> 172.21.15.49:0/1222284045 conn(0x7f3d5c0fcd40 msgr2=0x7f3d5c105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 -- 172.21.15.49:0/1222284045 shutdown_connections 2024-03-18T00:57:38.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.199+0000 7f3d60b1d640 1 -- 172.21.15.49:0/1222284045 wait complete. 2024-03-18T00:57:38.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:57:38.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 -- 172.21.15.49:0/688737411 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf01006e0 msgr2=0x7fecf0100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/688737411 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf01006e0 0x7fecf0100b00 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009900 tx=0x7fecd802bc30 comp rx=0 tx=0).stop 2024-03-18T00:57:38.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 -- 172.21.15.49:0/688737411 shutdown_connections 2024-03-18T00:57:38.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/688737411 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecf01018d0 0x7fecf0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/688737411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecf0102290 0x7fecf0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/688737411 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf01006e0 0x7fecf0100b00 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 -- 172.21.15.49:0/688737411 >> 172.21.15.49:0/688737411 conn(0x7fecf00fbc70 msgr2=0x7fecf00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 -- 172.21.15.49:0/688737411 shutdown_connections 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 -- 172.21.15.49:0/688737411 wait complete. 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.322+0000 7fecf4c0b640 1 Processor -- start 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecf4c0b640 1 -- start start 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecf4c0b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecf01006e0 0x7fecf0197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecf4c0b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecf01018d0 0x7fecf0198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecf4c0b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf0102290 0x7fecf019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecf4c0b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fecf010c330 con 0x7fecf0102290 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecf4c0b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fecf010c1b0 con 0x7fecf01006e0 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecf4c0b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fecf010c4b0 con 0x7fecf01018d0 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7feceed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf0102290 0x7fecf019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7feceed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf0102290 0x7fecf019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48022/0 (socket says 172.21.15.49:48022) 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7feceed9d640 1 -- 172.21.15.49:0/3796651261 learned_addr learned my addr 172.21.15.49:0/3796651261 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecedd9b640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecf01018d0 0x7fecf0198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecee59c640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecf01006e0 0x7fecf0197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7feceed9d640 1 -- 172.21.15.49:0/3796651261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecf01018d0 msgr2=0x7fecf0198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7feceed9d640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecf01018d0 0x7fecf0198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7feceed9d640 1 -- 172.21.15.49:0/3796651261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecf01006e0 msgr2=0x7fecf0197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7feceed9d640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecf01006e0 0x7fecf0197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7feceed9d640 1 -- 172.21.15.49:0/3796651261 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf019f720 con 0x7fecf0102290 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecee59c640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecf01006e0 0x7fecf0197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7feceed9d640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf0102290 0x7fecf019f0b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fece800eb10 tx=0x7fece800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fece77fe640 1 -- 172.21.15.49:0/3796651261 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8004b20 con 0x7fecf0102290 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fece77fe640 1 -- 172.21.15.49:0/3796651261 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece8004cc0 con 0x7fecf0102290 2024-03-18T00:57:38.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.323+0000 7fecf4c0b640 1 -- 172.21.15.49:0/3796651261 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf019fa10 con 0x7fecf0102290 2024-03-18T00:57:38.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.324+0000 7fece77fe640 1 -- 172.21.15.49:0/3796651261 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece8010760 con 0x7fecf0102290 2024-03-18T00:57:38.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.324+0000 7fecf4c0b640 1 -- 172.21.15.49:0/3796651261 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf019ff10 con 0x7fecf0102290 2024-03-18T00:57:38.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.326+0000 7fece77fe640 1 -- 172.21.15.49:0/3796651261 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fece8005020 con 0x7fecf0102290 2024-03-18T00:57:38.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.326+0000 7fecf77fc740 1 Processor -- start 2024-03-18T00:57:38.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecf77fc740 1 -- start start 2024-03-18T00:57:38.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c86839a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839a650 0x55c86839f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 0x55c8683a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecf77fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c8682fa490 con 0x55c868399c60 2024-03-18T00:57:38.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecf77fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c8682a2c90 con 0x55c86839a650 2024-03-18T00:57:38.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecf77fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c8682f6530 con 0x55c86839f7c0 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecedd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c86839a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecedd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c86839a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48030/0 (socket says 172.21.15.49:48030) 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecee59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839a650 0x55c86839f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecedd9b640 1 -- 172.21.15.49:0/146474417 learned_addr learned my addr 172.21.15.49:0/146474417 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecedd9b640 1 --2- 172.21.15.49:0/146474417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 0x55c8683a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fece77fe640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecedd9b640 1 -- 172.21.15.49:0/146474417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 msgr2=0x55c8683a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecedd9b640 1 --2- 172.21.15.49:0/146474417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 0x55c8683a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecedd9b640 1 -- 172.21.15.49:0/146474417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839a650 msgr2=0x55c86839f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecedd9b640 1 --2- 172.21.15.49:0/146474417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839a650 0x55c86839f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecedd9b640 1 -- 172.21.15.49:0/146474417 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c86835d500 con 0x55c868399c60 2024-03-18T00:57:38.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7fecee59c640 1 --2- 172.21.15.49:0/146474417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839a650 0x55c86839f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.327+0000 7feceed9d640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.328+0000 7fece77fe640 1 -- 172.21.15.49:0/3796651261 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fece80cf000 con 0x7fecf0102290 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.328+0000 7fecedd9b640 1 --2- 172.21.15.49:0/146474417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c86839a080 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fece0013630 tx=0x7fece0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef62d04cc32b64f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.328+0000 7fece4ff9640 1 -- 172.21.15.49:0/146474417 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece000f500 con 0x55c868399c60 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.328+0000 7fece4ff9640 1 -- 172.21.15.49:0/146474417 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece000fb80 con 0x55c868399c60 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.328+0000 7fece4ff9640 1 -- 172.21.15.49:0/146474417 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0010af0 con 0x55c868399c60 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 -- 172.21.15.49:0/146474417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 msgr2=0x55c86839a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7feceed9d640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fece80983f0 tx=0x7fece8066730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 --2- 172.21.15.49:0/146474417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c86839a080 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fece0013630 tx=0x7fece0005f90 comp rx=0 tx=0).stop 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 -- 172.21.15.49:0/146474417 shutdown_connections 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 --2- 172.21.15.49:0/146474417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839a650 0x55c86839f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 --2- 172.21.15.49:0/146474417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 0x55c8683a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 --2- 172.21.15.49:0/146474417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c86839a080 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 -- 172.21.15.49:0/146474417 >> 172.21.15.49:0/146474417 conn(0x55c868390dc0 msgr2=0x55c8683911c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 -- 172.21.15.49:0/146474417 shutdown_connections 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 -- 172.21.15.49:0/146474417 wait complete. 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 -- start start 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 Processor -- start 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 0x55c8683a4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 0x55c8683ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c8683a2770 con 0x55c868399c60 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c8682b6230 con 0x55c86839f7c0 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c8683a28f0 con 0x55c86839a650 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecf77fc740 10 client.0 _ll_register_callbacks cb 0x55c8682f2010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:38.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecedd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecedd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48032/0 (socket says 172.21.15.49:48032) 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.329+0000 7fecedd9b640 1 -- 172.21.15.49:0/489079660 learned_addr learned my addr 172.21.15.49:0/489079660 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecee59c640 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 0x55c8683a4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecedd9b640 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 0x55c8683ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecedd9b640 1 -- 172.21.15.49:0/489079660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 msgr2=0x55c8683a4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecedd9b640 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 0x55c8683a4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecedd9b640 1 -- 172.21.15.49:0/489079660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 msgr2=0x55c8683ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecedd9b640 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 0x55c8683ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecedd9b640 1 -- 172.21.15.49:0/489079660 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8682c5100 con 0x55c868399c60 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecedd9b640 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a40f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fece000a010 tx=0x7fece0009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70acd5c9035d62ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecd5ffb640 1 -- 172.21.15.49:0/489079660 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece002c060 con 0x55c868399c60 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecd5ffb640 1 -- 172.21.15.49:0/489079660 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fece0004050 con 0x55c868399c60 2024-03-18T00:57:38.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecf77fc740 1 -- 172.21.15.49:0/489079660 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fece000fb80 con 0x55c868399c60 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecd5ffb640 1 -- 172.21.15.49:0/489079660 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fece0004250 con 0x55c868399c60 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecf77fc740 10 client.17262 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecf77fc740 1 -- 172.21.15.49:0/489079660 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c86800e3f0 con 0x55c868399c60 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.330+0000 7fecd5ffb640 1 -- 172.21.15.49:0/489079660 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece000fb80 con 0x55c868399c60 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecd5ffb640 1 -- 172.21.15.49:0/489079660 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fece0010b40 con 0x55c868399c60 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecf77fc740 10 client.17262 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecf77fc740 10 client.17262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecf77fc740 20 client.17262 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecf77fc740 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c8683afd10 0x55c8683b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecf77fc740 4 client.17262 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecf77fc740 1 -- 172.21.15.49:0/489079660 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c8683079d0 con 0x55c8683afd10 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecee59c640 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c8683afd10 0x55c8683b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecd5ffb640 10 client.17262 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:38.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.331+0000 7fecee59c640 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c8683afd10 0x55c8683b2120 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f6d3433374835bf server_cookie=6ce5f38026149af2 in_seq=0 out_seq=0 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecd5ffb640 1 -- 172.21.15.49:0/489079660 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c8683079d0 con 0x55c8683afd10 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecd5ffb640 10 client.17262 handle_command_reply: tid=0 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 client.17262 shutdown 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 -- 172.21.15.49:0/489079660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 msgr2=0x55c8683a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a40f0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fece000a010 tx=0x7fece0009660 comp rx=0 tx=0).stop 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 -- 172.21.15.49:0/489079660 shutdown_connections 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c8683afd10 0x55c8683b2120 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 0x55c8683ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 0x55c8683a4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecd5ffb640 0 client.17262 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:38.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 --2- 172.21.15.49:0/489079660 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a40f0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 -- 172.21.15.49:0/489079660 >> 172.21.15.49:0/489079660 conn(0x55c868390dc0 msgr2=0x55c86838ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 -- 172.21.15.49:0/489079660 shutdown_connections 2024-03-18T00:57:38.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 1 -- 172.21.15.49:0/489079660 wait complete. 2024-03-18T00:57:38.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.334+0000 7fecf77fc740 20 client.17262 trim_cache size 0 max 16384 2024-03-18T00:57:38.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecf77fc740 1 Processor -- start 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecf77fc740 1 -- start start 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c868399c60 0x55c8683a4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c86839a650 0x55c8683ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 0x55c86838fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecf77fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01a6c50 con 0x55c86839a650 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecf77fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01a8790 con 0x55c868399c60 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecf77fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01a7320 con 0x55c86839f7c0 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecee59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c86839a650 0x55c8683ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecee59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c86839a650 0x55c8683ab630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48040/0 (socket says 172.21.15.49:48040) 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecedd9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c868399c60 0x55c8683a4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecee59c640 1 -- 172.21.15.49:0/682324239 learned_addr learned my addr 172.21.15.49:0/682324239 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecedd9b640 1 --2- 172.21.15.49:0/682324239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 0x55c86838fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecee59c640 1 -- 172.21.15.49:0/682324239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 msgr2=0x55c86838fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecee59c640 1 --2- 172.21.15.49:0/682324239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 0x55c86838fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecee59c640 1 -- 172.21.15.49:0/682324239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c868399c60 msgr2=0x55c8683a4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecee59c640 1 --2- 172.21.15.49:0/682324239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c868399c60 0x55c8683a4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecee59c640 1 -- 172.21.15.49:0/682324239 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c8680d87d0 con 0x55c86839a650 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.348+0000 7fecee59c640 1 --2- 172.21.15.49:0/682324239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c86839a650 0x55c8683ab630 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fecd802c7b0 tx=0x7fecd802e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e3c6f470a7f9bf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecd7fff640 1 -- 172.21.15.49:0/682324239 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd802cb40 con 0x55c86839a650 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecd7fff640 1 -- 172.21.15.49:0/682324239 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecd804a070 con 0x55c86839a650 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecd7fff640 1 -- 172.21.15.49:0/682324239 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd8045530 con 0x55c86839a650 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecf77fc740 1 -- 172.21.15.49:0/682324239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c86839a650 msgr2=0x55c8683ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecf77fc740 1 --2- 172.21.15.49:0/682324239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c86839a650 0x55c8683ab630 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fecd802c7b0 tx=0x7fecd802e040 comp rx=0 tx=0).stop 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecf77fc740 1 -- 172.21.15.49:0/682324239 shutdown_connections 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecf77fc740 1 --2- 172.21.15.49:0/682324239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839f7c0 0x55c86838fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecf77fc740 1 --2- 172.21.15.49:0/682324239 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c86839a650 0x55c8683ab630 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecf77fc740 1 --2- 172.21.15.49:0/682324239 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c868399c60 0x55c8683a4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecf77fc740 1 -- 172.21.15.49:0/682324239 >> 172.21.15.49:0/682324239 conn(0x55c868390dc0 msgr2=0x55c86838f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecf77fc740 1 -- 172.21.15.49:0/682324239 shutdown_connections 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.349+0000 7fecf77fc740 1 -- 172.21.15.49:0/682324239 wait complete. 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecf77fc740 1 -- start start 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecf77fc740 1 Processor -- start 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 0x55c8683a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecf77fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 0x55c8683a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecf77fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01a80c0 con 0x55c868399c60 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecf77fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01a79f0 con 0x55c86839f7c0 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecf77fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fecf01a4c30 con 0x55c86839a650 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecf77fc740 10 client.0 _ll_register_callbacks cb 0x7fece0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:38.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48044/0 (socket says 172.21.15.49:48044) 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 -- 172.21.15.49:0/2543178758 learned_addr learned my addr 172.21.15.49:0/2543178758 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecee59c640 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 0x55c8683a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 0x55c8683a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 -- 172.21.15.49:0/2543178758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 msgr2=0x55c8683a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 0x55c8683a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 -- 172.21.15.49:0/2543178758 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 msgr2=0x55c8683a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 0x55c8683a9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 -- 172.21.15.49:0/2543178758 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c8683aa460 con 0x55c868399c60 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecedd9b640 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a8990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fece0010eb0 tx=0x7fece00104d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78acb6434332fe0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.350+0000 7fecd5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:38.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecd5ffb640 1 -- 172.21.15.49:0/2543178758 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece002c070 con 0x55c868399c60 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecd5ffb640 1 -- 172.21.15.49:0/2543178758 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fece0010540 con 0x55c868399c60 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecf77fc740 1 -- 172.21.15.49:0/2543178758 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecd804a070 con 0x55c868399c60 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecd5ffb640 1 -- 172.21.15.49:0/2543178758 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fece0010740 con 0x55c868399c60 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecf77fc740 10 client.17268 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecf77fc740 1 -- 172.21.15.49:0/2543178758 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c8683ab760 con 0x55c868399c60 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecd5ffb640 1 -- 172.21.15.49:0/2543178758 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecd804a070 con 0x55c868399c60 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecd5ffb640 1 -- 172.21.15.49:0/2543178758 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fece003b040 con 0x55c868399c60 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecf77fc740 10 client.17268 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecf77fc740 10 client.17268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecf77fc740 20 client.17268 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecf77fc740 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c8684164e0 0x55c868418900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecf77fc740 4 client.17268 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecf77fc740 1 -- 172.21.15.49:0/2543178758 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c868301d20 con 0x55c8684164e0 2024-03-18T00:57:38.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.351+0000 7fecee59c640 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c8684164e0 0x55c868418900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:38.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.352+0000 7fecd5ffb640 10 client.17268 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:38.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.352+0000 7fecee59c640 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c8684164e0 0x55c868418900 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ade8204a0dc8ed9 server_cookie=a319520b9a504e50 in_seq=0 out_seq=0 2024-03-18T00:57:38.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecd5ffb640 1 -- 172.21.15.49:0/2543178758 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c868301d20 con 0x55c8684164e0 2024-03-18T00:57:38.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecd5ffb640 10 client.17268 handle_command_reply: tid=0 2024-03-18T00:57:38.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 client.17268 shutdown 2024-03-18T00:57:38.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 -- 172.21.15.49:0/2543178758 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 msgr2=0x55c8683a8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a8990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fece0010eb0 tx=0x7fece00104d0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 -- 172.21.15.49:0/2543178758 shutdown_connections 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c8684164e0 0x55c868418900 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c86839f7c0 0x55c8683a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecd5ffb640 0 client.17268 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c86839a650 0x55c8683a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 --2- 172.21.15.49:0/2543178758 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c868399c60 0x55c8683a8990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 -- 172.21.15.49:0/2543178758 >> 172.21.15.49:0/2543178758 conn(0x55c868390dc0 msgr2=0x55c868394020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 -- 172.21.15.49:0/2543178758 shutdown_connections 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 1 -- 172.21.15.49:0/2543178758 wait complete. 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.354+0000 7fecf77fc740 20 client.17268 trim_cache size 0 max 16384 2024-03-18T00:57:38.359 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:57:38.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 -- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fecdc0a2800 msgr2=0x7fecdc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fece80983f0 tx=0x7fece8066730 comp rx=0 tx=0).stop 2024-03-18T00:57:38.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 -- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf0102290 msgr2=0x7fecf019f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf0102290 0x7fecf019f0b0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fece800eb10 tx=0x7fece800c2d0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 -- 172.21.15.49:0/3796651261 shutdown_connections 2024-03-18T00:57:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fecdc0a2800 0x7fecdc0a4cc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fecf0102290 0x7fecf019f0b0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fecf01018d0 0x7fecf0198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 --2- 172.21.15.49:0/3796651261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fecf01006e0 0x7fecf0197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 -- 172.21.15.49:0/3796651261 >> 172.21.15.49:0/3796651261 conn(0x7fecf00fbc70 msgr2=0x7fecf00fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 -- 172.21.15.49:0/3796651261 shutdown_connections 2024-03-18T00:57:38.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:38.355+0000 7fecf4c0b640 1 -- 172.21.15.49:0/3796651261 wait complete. 2024-03-18T00:57:39.366 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:57:39.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:57:39.366 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:57:39.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.483+0000 7f75f8d87640 1 -- 172.21.15.49:0/832828309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4101cb0 msgr2=0x7f75f41020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:39.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.483+0000 7f75f8d87640 1 --2- 172.21.15.49:0/832828309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4101cb0 0x7f75f41020d0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f75e4009900 tx=0x7f75e402bc30 comp rx=0 tx=0).stop 2024-03-18T00:57:39.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.483+0000 7f75f8d87640 1 -- 172.21.15.49:0/832828309 shutdown_connections 2024-03-18T00:57:39.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.483+0000 7f75f8d87640 1 --2- 172.21.15.49:0/832828309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f41009e0 0x7f75f4100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.483+0000 7f75f8d87640 1 --2- 172.21.15.49:0/832828309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f41013a0 0x7f75f4108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.483 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.483+0000 7f75f8d87640 1 --2- 172.21.15.49:0/832828309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f4101cb0 0x7f75f41020d0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.483+0000 7f75f8d87640 1 -- 172.21.15.49:0/832828309 >> 172.21.15.49:0/832828309 conn(0x7f75f40fb410 msgr2=0x7f75f40fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:39.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.483+0000 7f75f8d87640 1 -- 172.21.15.49:0/832828309 shutdown_connections 2024-03-18T00:57:39.484 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.483+0000 7f75f8d87640 1 -- 172.21.15.49:0/832828309 wait complete. 2024-03-18T00:57:39.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.484+0000 7f75f8d87640 1 Processor -- start 2024-03-18T00:57:39.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.484+0000 7f75f8d87640 1 -- start start 2024-03-18T00:57:39.485 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.484+0000 7f75f8d87640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f41009e0 0x7f75f4197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.484+0000 7f75f8d87640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f41013a0 0x7f75f41980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.484+0000 7f75f8d87640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f4101cb0 0x7f75f419f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.484+0000 7f75f8d87640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f75f410d490 con 0x7f75f41009e0 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.484+0000 7f75f8d87640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f75f410d310 con 0x7f75f4101cb0 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.484+0000 7f75f8d87640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f75f410d610 con 0x7f75f41013a0 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f41009e0 0x7f75f4197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f41013a0 0x7f75f41980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f2d9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f4101cb0 0x7f75f419f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f41009e0 0x7f75f4197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48058/0 (socket says 172.21.15.49:48058) 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f41013a0 0x7f75f41980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:42108/0 (socket says 172.21.15.49:42108) 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f259c640 1 -- 172.21.15.49:0/2526761869 learned_addr learned my addr 172.21.15.49:0/2526761869 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f2d9d640 1 -- 172.21.15.49:0/2526761869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f41013a0 msgr2=0x7f75f41980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:39.486 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f41013a0 0x7f75f41980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f2d9d640 1 -- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f41009e0 msgr2=0x7f75f4197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:39.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f41009e0 0x7f75f4197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f2d9d640 1 -- 172.21.15.49:0/2526761869 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f419f7d0 con 0x7f75f4101cb0 2024-03-18T00:57:39.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f259c640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f41009e0 0x7f75f4197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:57:39.487 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.485+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f4101cb0 0x7f75f419f160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009fd0 tx=0x7f75ec00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:39.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.487+0000 7f75e37fe640 1 -- 172.21.15.49:0/2526761869 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec00ce20 con 0x7f75f4101cb0 2024-03-18T00:57:39.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.487+0000 7f75e37fe640 1 -- 172.21.15.49:0/2526761869 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75ec005020 con 0x7f75f4101cb0 2024-03-18T00:57:39.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.487+0000 7f75f8d87640 1 -- 172.21.15.49:0/2526761869 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f419fac0 con 0x7f75f4101cb0 2024-03-18T00:57:39.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.487+0000 7f75e37fe640 1 -- 172.21.15.49:0/2526761869 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec010720 con 0x7f75f4101cb0 2024-03-18T00:57:39.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.487+0000 7f75f8d87640 1 -- 172.21.15.49:0/2526761869 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41a0010 con 0x7f75f4101cb0 2024-03-18T00:57:39.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.489+0000 7f75e37fe640 1 -- 172.21.15.49:0/2526761869 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f75ec004b20 con 0x7f75f4101cb0 2024-03-18T00:57:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75fb970740 1 Processor -- start 2024-03-18T00:57:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75fb970740 1 -- start start 2024-03-18T00:57:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75e37fe640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f75dc0a2880 0x7f75dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75fb970740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801c01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75fb970740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801c5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75fb970740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c5900 0x560c801c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75fb970740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x560c801200d0 con 0x560c801c0790 2024-03-18T00:57:39.494 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75fb970740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560c800c8c70 con 0x560c801c5900 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75fb970740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560c8011c170 con 0x560c801bfda0 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f75dc0a2880 0x7f75dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75f259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801c01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75f259c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801c01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:42110/0 (socket says 172.21.15.49:42110) 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75f259c640 1 -- 172.21.15.49:0/1740382913 learned_addr learned my addr 172.21.15.49:0/1740382913 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/1740382913 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/1740382913 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c5900 0x560c801c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75e37fe640 1 -- 172.21.15.49:0/2526761869 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f75ec0cf140 con 0x7f75f4101cb0 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.490+0000 7f75f259c640 1 -- 172.21.15.49:0/1740382913 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c5900 msgr2=0x560c801c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.491+0000 7f75f259c640 1 --2- 172.21.15.49:0/1740382913 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c5900 0x560c801c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.491+0000 7f75f259c640 1 -- 172.21.15.49:0/1740382913 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 msgr2=0x560c801c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.491+0000 7f75f259c640 1 --2- 172.21.15.49:0/1740382913 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.495 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.491+0000 7f75f259c640 1 -- 172.21.15.49:0/1740382913 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c7fefc7e0 con 0x560c801bfda0 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.491+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f75dc0a2880 0x7f75dc0a4d40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f75f41990c0 tx=0x7f75e8009b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.491+0000 7f75f259c640 1 --2- 172.21.15.49:0/1740382913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801c01c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f75e4036040 tx=0x7f75e402c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94e72ee957d77047 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.491+0000 7f75e0ff9640 1 -- 172.21.15.49:0/1740382913 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4005d50 con 0x560c801bfda0 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.491+0000 7f75e0ff9640 1 -- 172.21.15.49:0/1740382913 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75e4005520 con 0x560c801bfda0 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.491+0000 7f75e0ff9640 1 -- 172.21.15.49:0/1740382913 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4004cc0 con 0x560c801bfda0 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 -- 172.21.15.49:0/1740382913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 msgr2=0x560c801c01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 --2- 172.21.15.49:0/1740382913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801c01c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f75e4036040 tx=0x7f75e402c5d0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 -- 172.21.15.49:0/1740382913 shutdown_connections 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 --2- 172.21.15.49:0/1740382913 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c5900 0x560c801c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 --2- 172.21.15.49:0/1740382913 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 --2- 172.21.15.49:0/1740382913 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801c01c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 -- 172.21.15.49:0/1740382913 >> 172.21.15.49:0/1740382913 conn(0x560c80183c50 msgr2=0x560c801b7300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 -- 172.21.15.49:0/1740382913 shutdown_connections 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 -- 172.21.15.49:0/1740382913 wait complete. 2024-03-18T00:57:39.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 -- start start 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 Processor -- start 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 0x560c801ca290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 0x560c801d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x560c801c8850 con 0x560c801c0790 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560c800dc0d0 con 0x560c801bfda0 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.492+0000 7f75fb970740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560c801c89d0 con 0x560c801c5900 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75fb970740 10 client.0 _ll_register_callbacks cb 0x560c800a5760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 0x560c801ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48080/0 (socket says 172.21.15.49:48080) 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f1d9b640 1 -- 172.21.15.49:0/828962295 learned_addr learned my addr 172.21.15.49:0/828962295 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 0x560c801d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f1d9b640 1 -- 172.21.15.49:0/828962295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 msgr2=0x560c801d1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 0x560c801d1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f1d9b640 1 -- 172.21.15.49:0/828962295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 msgr2=0x560c801ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 0x560c801ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f1d9b640 1 -- 172.21.15.49:0/828962295 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c7ff5cf70 con 0x560c801c0790 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801ca7d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009da0 tx=0x7f75e801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb22d8742d0f390 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75d9ffb640 1 -- 172.21.15.49:0/828962295 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8005020 con 0x560c801c0790 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75d9ffb640 1 -- 172.21.15.49:0/828962295 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f75e80051c0 con 0x560c801c0790 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75fb970740 1 -- 172.21.15.49:0/828962295 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75e4005520 con 0x560c801c0790 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75d9ffb640 1 -- 172.21.15.49:0/828962295 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f75e80053c0 con 0x560c801c0790 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75fb970740 10 client.17280 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75fb970740 1 -- 172.21.15.49:0/828962295 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c7fdf4170 con 0x560c801c0790 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.493+0000 7f75d9ffb640 1 -- 172.21.15.49:0/828962295 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75e4005520 con 0x560c801c0790 2024-03-18T00:57:39.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75d9ffb640 1 -- 172.21.15.49:0/828962295 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f75e80049b0 con 0x560c801c0790 2024-03-18T00:57:39.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75fb970740 10 client.17280 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:39.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75fb970740 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:39.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75fb970740 20 client.17280 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:39.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75fb970740 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560c801d5e30 0x560c801d82f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75fb970740 4 client.17280 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:39.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75fb970740 1 -- 172.21.15.49:0/828962295 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c8012d610 con 0x560c801d5e30 2024-03-18T00:57:39.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75f259c640 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560c801d5e30 0x560c801d82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75d9ffb640 10 client.17280 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:39.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.494+0000 7f75f259c640 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560c801d5e30 0x560c801d82f0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98bb06bfda869bb5 server_cookie=f7ea811a29ddb199 in_seq=0 out_seq=0 2024-03-18T00:57:39.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75d9ffb640 1 -- 172.21.15.49:0/828962295 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560c8012d610 con 0x560c801d5e30 2024-03-18T00:57:39.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75d9ffb640 10 client.17280 handle_command_reply: tid=0 2024-03-18T00:57:39.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 client.17280 shutdown 2024-03-18T00:57:39.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 -- 172.21.15.49:0/828962295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 msgr2=0x560c801ca7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:39.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801ca7d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f75e8009da0 tx=0x7f75e801d000 comp rx=0 tx=0).stop 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 -- 172.21.15.49:0/828962295 shutdown_connections 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560c801d5e30 0x560c801d82f0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 0x560c801d1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75d9ffb640 0 client.17280 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801ca7d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 --2- 172.21.15.49:0/828962295 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 0x560c801ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 -- 172.21.15.49:0/828962295 >> 172.21.15.49:0/828962295 conn(0x560c80183c50 msgr2=0x560c801b64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 -- 172.21.15.49:0/828962295 shutdown_connections 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 1 -- 172.21.15.49:0/828962295 wait complete. 2024-03-18T00:57:39.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.497+0000 7f75fb970740 20 client.17280 trim_cache size 0 max 16384 2024-03-18T00:57:39.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75fb970740 1 Processor -- start 2024-03-18T00:57:39.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75fb970740 1 -- start start 2024-03-18T00:57:39.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75fb970740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801badd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75fb970740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c0790 0x560c801d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75fb970740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c5900 0x560c801c9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75fb970740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41a6ac0 con 0x560c801c5900 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75fb970740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41a8600 con 0x560c801c0790 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75fb970740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41a7190 con 0x560c801bfda0 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f2d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c5900 0x560c801c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f2d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c5900 0x560c801c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48092/0 (socket says 172.21.15.49:48092) 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f2d9d640 1 -- 172.21.15.49:0/2674484436 learned_addr learned my addr 172.21.15.49:0/2674484436 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f259c640 1 --2- 172.21.15.49:0/2674484436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801badd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/2674484436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c0790 0x560c801d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f2d9d640 1 -- 172.21.15.49:0/2674484436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 msgr2=0x560c801badd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/2674484436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801badd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f2d9d640 1 -- 172.21.15.49:0/2674484436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c0790 msgr2=0x560c801d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/2674484436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c0790 0x560c801d17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.515+0000 7f75f2d9d640 1 -- 172.21.15.49:0/2674484436 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c801c0b70 con 0x560c801c5900 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/2674484436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c5900 0x560c801c9510 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0ccbc0 tx=0x7f75ec05bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f120246d25fb11b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75dbfff640 1 -- 172.21.15.49:0/2674484436 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec099d60 con 0x560c801c5900 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75dbfff640 1 -- 172.21.15.49:0/2674484436 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75ec0cccd0 con 0x560c801c5900 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75dbfff640 1 -- 172.21.15.49:0/2674484436 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec09bcc0 con 0x560c801c5900 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75fb970740 1 -- 172.21.15.49:0/2674484436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c5900 msgr2=0x560c801c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75fb970740 1 --2- 172.21.15.49:0/2674484436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c5900 0x560c801c9510 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0ccbc0 tx=0x7f75ec05bd40 comp rx=0 tx=0).stop 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75fb970740 1 -- 172.21.15.49:0/2674484436 shutdown_connections 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75fb970740 1 --2- 172.21.15.49:0/2674484436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801c0790 0x560c801d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75fb970740 1 --2- 172.21.15.49:0/2674484436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c5900 0x560c801c9510 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75fb970740 1 --2- 172.21.15.49:0/2674484436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801bfda0 0x560c801badd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75fb970740 1 -- 172.21.15.49:0/2674484436 >> 172.21.15.49:0/2674484436 conn(0x560c80183c50 msgr2=0x560c801c8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75fb970740 1 -- 172.21.15.49:0/2674484436 shutdown_connections 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.516+0000 7f75fb970740 1 -- 172.21.15.49:0/2674484436 wait complete. 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75fb970740 1 -- start start 2024-03-18T00:57:39.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75fb970740 1 Processor -- start 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75fb970740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 0x560c801cf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75fb970740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75fb970740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 0x560c801d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75fb970740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41a7f30 con 0x560c801c0790 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75fb970740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41a7860 con 0x560c801bfda0 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75fb970740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f75f41a4aa0 con 0x560c801c5900 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75fb970740 10 client.0 _ll_register_callbacks cb 0x7f75e8011ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 0x560c801cf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f1d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801cfa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48094/0 (socket says 172.21.15.49:48094) 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f1d9b640 1 -- 172.21.15.49:0/2986950502 learned_addr learned my addr 172.21.15.49:0/2986950502 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 0x560c801d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f1d9b640 1 -- 172.21.15.49:0/2986950502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 msgr2=0x560c801d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 0x560c801d09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f1d9b640 1 -- 172.21.15.49:0/2986950502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 msgr2=0x560c801cf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 0x560c801cf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f1d9b640 1 -- 172.21.15.49:0/2986950502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c801d0f90 con 0x560c801c0790 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.517+0000 7f75f2d9d640 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 0x560c801d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75f1d9b640 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801cfa30 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f75e8011fd0 tx=0x7f75e8011350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f51ffe803abeba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75d9ffb640 1 -- 172.21.15.49:0/2986950502 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e80051d0 con 0x560c801c0790 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75d9ffb640 1 -- 172.21.15.49:0/2986950502 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f75e80047a0 con 0x560c801c0790 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75fb970740 1 -- 172.21.15.49:0/2986950502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75ec0cccd0 con 0x560c801c0790 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75d9ffb640 1 -- 172.21.15.49:0/2986950502 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f75e80049a0 con 0x560c801c0790 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75fb970740 10 client.17286 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75fb970740 1 -- 172.21.15.49:0/2986950502 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c8023c830 con 0x560c801c0790 2024-03-18T00:57:39.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75d9ffb640 1 -- 172.21.15.49:0/2986950502 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75ec0cccd0 con 0x560c801c0790 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.518+0000 7f75d9ffb640 1 -- 172.21.15.49:0/2986950502 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f75e801f860 con 0x560c801c0790 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.519+0000 7f75fb970740 10 client.17286 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.519+0000 7f75fb970740 10 client.17286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.519+0000 7f75fb970740 20 client.17286 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.519+0000 7f75fb970740 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560c8023d1d0 0x560c8023f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.519+0000 7f75fb970740 4 client.17286 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.519+0000 7f75fb970740 1 -- 172.21.15.49:0/2986950502 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560c80127960 con 0x560c8023d1d0 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.519+0000 7f75f259c640 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560c8023d1d0 0x560c8023f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.519+0000 7f75f259c640 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560c8023d1d0 0x560c8023f5f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5efa6de39acdfa1 server_cookie=7dbc9501d8c4e1cc in_seq=0 out_seq=0 2024-03-18T00:57:39.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:39.519+0000 7f75d9ffb640 10 client.17286 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:39.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:39 smithi049 ceph-mon[32130]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 894 MiB/s wr, 2.00k op/s 2024-03-18T00:57:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:39 smithi107 ceph-mon[36270]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 894 MiB/s wr, 2.00k op/s 2024-03-18T00:57:39.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:39 smithi106 ceph-mon[35899]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 894 MiB/s wr, 2.00k op/s 2024-03-18T00:57:40.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75d9ffb640 1 -- 172.21.15.49:0/2986950502 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560c80127960 con 0x560c8023d1d0 2024-03-18T00:57:40.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75d9ffb640 10 client.17286 handle_command_reply: tid=0 2024-03-18T00:57:40.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 client.17286 shutdown 2024-03-18T00:57:40.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 -- 172.21.15.49:0/2986950502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 msgr2=0x560c801cfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.498 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801cfa30 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f75e8011fd0 tx=0x7f75e8011350 comp rx=0 tx=0).stop 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 -- 172.21.15.49:0/2986950502 shutdown_connections 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x560c8023d1d0 0x560c8023f5f0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560c801c5900 0x560c801d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560c801c0790 0x560c801cfa30 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75d9ffb640 0 client.17286 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 --2- 172.21.15.49:0/2986950502 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560c801bfda0 0x560c801cf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 -- 172.21.15.49:0/2986950502 >> 172.21.15.49:0/2986950502 conn(0x560c80183c50 msgr2=0x560c801c9690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 -- 172.21.15.49:0/2986950502 shutdown_connections 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 1 -- 172.21.15.49:0/2986950502 wait complete. 2024-03-18T00:57:40.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.498+0000 7f75fb970740 20 client.17286 trim_cache size 0 max 16384 2024-03-18T00:57:40.500 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:57:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:57:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "4a77e765-b973-4cc7-b42b-359f648720b2", 2024-03-18T00:57:40.500 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:57:40.500 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 -- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f75dc0a2880 msgr2=0x7f75dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f75dc0a2880 0x7f75dc0a4d40 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f75f41990c0 tx=0x7f75e8009b60 comp rx=0 tx=0).stop 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 -- 172.21.15.49:0/2526761869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f4101cb0 msgr2=0x7f75f419f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f4101cb0 0x7f75f419f160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009fd0 tx=0x7f75ec00ef10 comp rx=0 tx=0).stop 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 -- 172.21.15.49:0/2526761869 shutdown_connections 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f75dc0a2880 0x7f75dc0a4d40 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f75f4101cb0 0x7f75f419f160 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f75f41013a0 0x7f75f41980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 --2- 172.21.15.49:0/2526761869 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f75f41009e0 0x7f75f4197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 -- 172.21.15.49:0/2526761869 >> 172.21.15.49:0/2526761869 conn(0x7f75f40fb410 msgr2=0x7f75f4104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 -- 172.21.15.49:0/2526761869 shutdown_connections 2024-03-18T00:57:40.501 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.499+0000 7f75f8d87640 1 -- 172.21.15.49:0/2526761869 wait complete. 2024-03-18T00:57:40.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:57:40.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.624+0000 7f507da36640 1 -- 172.21.15.49:0/546698032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50781074e0 msgr2=0x7f5078109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.624+0000 7f507da36640 1 --2- 172.21.15.49:0/546698032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50781074e0 0x7f5078109930 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f5070009ca0 tx=0x7f507002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.624+0000 7f507da36640 1 -- 172.21.15.49:0/546698032 shutdown_connections 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.624+0000 7f507da36640 1 --2- 172.21.15.49:0/546698032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f50781074e0 0x7f5078109930 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.624+0000 7f507da36640 1 --2- 172.21.15.49:0/546698032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5078104bb0 0x7f5078106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.624+0000 7f507da36640 1 --2- 172.21.15.49:0/546698032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5078102280 0x7f5078104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.624+0000 7f507da36640 1 -- 172.21.15.49:0/546698032 >> 172.21.15.49:0/546698032 conn(0x7f50780fbc90 msgr2=0x7f50780fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.624+0000 7f507da36640 1 -- 172.21.15.49:0/546698032 shutdown_connections 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.624+0000 7f507da36640 1 -- 172.21.15.49:0/546698032 wait complete. 2024-03-18T00:57:40.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f507da36640 1 Processor -- start 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f507da36640 1 -- start start 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f507da36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5078102280 0x7f5078199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f507da36640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5078104bb0 0x7f507819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f507da36640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50781074e0 0x7f50781a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f507da36640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f507810d480 con 0x7f5078104bb0 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f507da36640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f507810d300 con 0x7f50781074e0 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f507da36640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f507810d600 con 0x7f5078102280 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f506ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5078104bb0 0x7f507819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f506ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5078104bb0 0x7f507819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48124/0 (socket says 172.21.15.49:48124) 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f506ffff640 1 -- 172.21.15.49:0/3940039747 learned_addr learned my addr 172.21.15.49:0/3940039747 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.625+0000 7f5076ffd640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5078102280 0x7f5078199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f50777fe640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50781074e0 0x7f50781a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f506ffff640 1 -- 172.21.15.49:0/3940039747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5078102280 msgr2=0x7f5078199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f506ffff640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5078102280 0x7f5078199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f506ffff640 1 -- 172.21.15.49:0/3940039747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50781074e0 msgr2=0x7f50781a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f506ffff640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50781074e0 0x7f50781a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f506ffff640 1 -- 172.21.15.49:0/3940039747 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50781a1960 con 0x7f5078104bb0 2024-03-18T00:57:40.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f506ffff640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5078104bb0 0x7f507819a290 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f506000e9a0 tx=0x7f506000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:40.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f507ca34640 1 -- 172.21.15.49:0/3940039747 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506000cd80 con 0x7f5078104bb0 2024-03-18T00:57:40.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f507ca34640 1 -- 172.21.15.49:0/3940039747 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50600035f0 con 0x7f5078104bb0 2024-03-18T00:57:40.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f507da36640 1 -- 172.21.15.49:0/3940039747 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50781a1c50 con 0x7f5078104bb0 2024-03-18T00:57:40.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f507ca34640 1 -- 172.21.15.49:0/3940039747 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5060010670 con 0x7f5078104bb0 2024-03-18T00:57:40.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.626+0000 7f507da36640 1 -- 172.21.15.49:0/3940039747 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50781a21a0 con 0x7f5078104bb0 2024-03-18T00:57:40.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.628+0000 7f507ca34640 1 -- 172.21.15.49:0/3940039747 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f50600037a0 con 0x7f5078104bb0 2024-03-18T00:57:40.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.628+0000 7f5080629740 1 Processor -- start 2024-03-18T00:57:40.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5080629740 1 -- start start 2024-03-18T00:57:40.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5080629740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 0x56212a431590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5080629740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a436660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5080629740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a439160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5080629740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56212a396500 con 0x56212a431b60 2024-03-18T00:57:40.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5080629740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56212a33ebe0 con 0x56212a431170 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5080629740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56212a3925a0 con 0x56212a436cd0 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5076ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a436660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5076ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a436660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48130/0 (socket says 172.21.15.49:48130) 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5076ffd640 1 -- 172.21.15.49:0/345270408 learned_addr learned my addr 172.21.15.49:0/345270408 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f50777fe640 1 --2- 172.21.15.49:0/345270408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 0x56212a431590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f506ffff640 1 --2- 172.21.15.49:0/345270408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a439160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5076ffd640 1 -- 172.21.15.49:0/345270408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 msgr2=0x56212a439160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5076ffd640 1 --2- 172.21.15.49:0/345270408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a439160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5076ffd640 1 -- 172.21.15.49:0/345270408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 msgr2=0x56212a431590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5076ffd640 1 --2- 172.21.15.49:0/345270408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 0x56212a431590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5076ffd640 1 -- 172.21.15.49:0/345270408 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56212a3524b0 con 0x56212a431b60 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f507ca34640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50640a2880 0x7f50640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f5076ffd640 1 --2- 172.21.15.49:0/345270408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a436660 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5068009c90 tx=0x7f5068011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17631470f61a154e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:40.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f50777fe640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50640a2880 0x7f50640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.629+0000 7f507ca34640 1 -- 172.21.15.49:0/3940039747 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f50600ced00 con 0x7f5078104bb0 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f506dffb640 1 -- 172.21.15.49:0/345270408 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068005220 con 0x56212a431b60 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f506dffb640 1 -- 172.21.15.49:0/345270408 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5068004c80 con 0x56212a431b60 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f506dffb640 1 -- 172.21.15.49:0/345270408 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50680198d0 con 0x56212a431b60 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f5080629740 1 -- 172.21.15.49:0/345270408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 msgr2=0x56212a436660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f5080629740 1 --2- 172.21.15.49:0/345270408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a436660 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f5068009c90 tx=0x7f5068011810 comp rx=0 tx=0).stop 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f50777fe640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50640a2880 0x7f50640a4d40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5070005e30 tx=0x7f5070032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f5080629740 1 -- 172.21.15.49:0/345270408 shutdown_connections 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f5080629740 1 --2- 172.21.15.49:0/345270408 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a439160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f5080629740 1 --2- 172.21.15.49:0/345270408 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a436660 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f5080629740 1 --2- 172.21.15.49:0/345270408 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 0x56212a431590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f5080629740 1 -- 172.21.15.49:0/345270408 >> 172.21.15.49:0/345270408 conn(0x56212a42c6e0 msgr2=0x56212a42cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f5080629740 1 -- 172.21.15.49:0/345270408 shutdown_connections 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.630+0000 7f5080629740 1 -- 172.21.15.49:0/345270408 wait complete. 2024-03-18T00:57:40.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5080629740 1 -- start start 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5080629740 1 Processor -- start 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5080629740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 0x56212a43b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5080629740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a43bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5080629740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a442bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5080629740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56212a439c80 con 0x56212a431b60 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5080629740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56212a352160 con 0x56212a431170 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5080629740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56212a439e00 con 0x56212a436cd0 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f506ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a43bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5076ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 0x56212a43b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f5080629740 10 client.0 _ll_register_callbacks cb 0x56212a336150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f506ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a43bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48140/0 (socket says 172.21.15.49:48140) 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f506ffff640 1 -- 172.21.15.49:0/2797690302 learned_addr learned my addr 172.21.15.49:0/2797690302 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f50777fe640 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a442bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f506ffff640 1 -- 172.21.15.49:0/2797690302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 msgr2=0x56212a442bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f506ffff640 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a442bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f506ffff640 1 -- 172.21.15.49:0/2797690302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 msgr2=0x56212a43b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.631+0000 7f506ffff640 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 0x56212a43b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f506ffff640 1 -- 172.21.15.49:0/2797690302 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56212a1747d0 con 0x56212a431b60 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f506ffff640 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a43bb90 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f50600a38f0 tx=0x7f5060099690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5ae88ecd497895 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5056ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5056ffd640 1 -- 172.21.15.49:0/2797690302 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506009b450 con 0x56212a431b60 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5056ffd640 1 -- 172.21.15.49:0/2797690302 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5060084d80 con 0x56212a431b60 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5080629740 1 -- 172.21.15.49:0/2797690302 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5068004c80 con 0x56212a431b60 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5056ffd640 1 -- 172.21.15.49:0/2797690302 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f50600997c0 con 0x56212a431b60 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5080629740 10 client.17298 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5080629740 1 -- 172.21.15.49:0/2797690302 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56212a0aa3f0 con 0x56212a431b60 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5056ffd640 1 -- 172.21.15.49:0/2797690302 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5068004c80 con 0x56212a431b60 2024-03-18T00:57:40.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5056ffd640 1 -- 172.21.15.49:0/2797690302 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f50600d3040 con 0x56212a431b60 2024-03-18T00:57:40.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5080629740 10 client.17298 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:40.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5080629740 10 client.17298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:40.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5080629740 20 client.17298 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:40.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5080629740 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56212a447070 0x56212a449530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5080629740 4 client.17298 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:40.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.632+0000 7f5080629740 1 -- 172.21.15.49:0/2797690302 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56212a3a3a40 con 0x56212a447070 2024-03-18T00:57:40.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.633+0000 7f5076ffd640 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56212a447070 0x56212a449530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.633+0000 7f5076ffd640 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56212a447070 0x56212a449530 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e988cc04820885d server_cookie=1e32b3fb2cc219d4 in_seq=0 out_seq=0 2024-03-18T00:57:40.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.633+0000 7f5056ffd640 10 client.17298 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:40.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.635+0000 7f5056ffd640 1 -- 172.21.15.49:0/2797690302 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56212a3a3a40 con 0x56212a447070 2024-03-18T00:57:40.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.635+0000 7f5056ffd640 10 client.17298 handle_command_reply: tid=0 2024-03-18T00:57:40.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.635+0000 7f5080629740 1 client.17298 shutdown 2024-03-18T00:57:40.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.635+0000 7f5080629740 1 -- 172.21.15.49:0/2797690302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 msgr2=0x56212a43bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.635+0000 7f5080629740 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a43bb90 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f50600a38f0 tx=0x7f5060099690 comp rx=0 tx=0).stop 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5080629740 1 -- 172.21.15.49:0/2797690302 shutdown_connections 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5080629740 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56212a447070 0x56212a449530 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5080629740 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a442bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5080629740 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a43bb90 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5080629740 1 --2- 172.21.15.49:0/2797690302 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431170 0x56212a43b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5056ffd640 0 client.17298 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5080629740 1 -- 172.21.15.49:0/2797690302 >> 172.21.15.49:0/2797690302 conn(0x56212a42c6e0 msgr2=0x56212a42b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5080629740 1 -- 172.21.15.49:0/2797690302 shutdown_connections 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5080629740 1 -- 172.21.15.49:0/2797690302 wait complete. 2024-03-18T00:57:40.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.636+0000 7f5080629740 20 client.17298 trim_cache size 0 max 16384 2024-03-18T00:57:40.655 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.654+0000 7f5080629740 1 Processor -- start 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.654+0000 7f5080629740 1 -- start start 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.654+0000 7f5080629740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a431170 0x56212a433060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.654+0000 7f5080629740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a442b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.654+0000 7f5080629740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a436cd0 0x56212a44ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.654+0000 7f5080629740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f50781ab090 con 0x56212a431b60 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.654+0000 7f5080629740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f50781acbd0 con 0x56212a436cd0 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.654+0000 7f5080629740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f50781ab760 con 0x56212a431170 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.654+0000 7f506ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a442b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a442b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48152/0 (socket says 172.21.15.49:48152) 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f5076ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a431170 0x56212a433060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506ffff640 1 -- 172.21.15.49:0/3079391208 learned_addr learned my addr 172.21.15.49:0/3079391208 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f50777fe640 1 --2- 172.21.15.49:0/3079391208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a436cd0 0x56212a44ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.656 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506ffff640 1 -- 172.21.15.49:0/3079391208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a431170 msgr2=0x56212a433060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506ffff640 1 --2- 172.21.15.49:0/3079391208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a431170 0x56212a433060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506ffff640 1 -- 172.21.15.49:0/3079391208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a436cd0 msgr2=0x56212a44ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506ffff640 1 --2- 172.21.15.49:0/3079391208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a436cd0 0x56212a44ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506ffff640 1 -- 172.21.15.49:0/3079391208 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56212a06a0f0 con 0x56212a431b60 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f5076ffd640 1 --2- 172.21.15.49:0/3079391208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a431170 0x56212a433060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506ffff640 1 --2- 172.21.15.49:0/3079391208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a442b90 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f506000e970 tx=0x7f5060003d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af3a9d07a562ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506d7fa640 1 -- 172.21.15.49:0/3079391208 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506009afe0 con 0x56212a431b60 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506d7fa640 1 -- 172.21.15.49:0/3079391208 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50600bb040 con 0x56212a431b60 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.655+0000 7f506d7fa640 1 -- 172.21.15.49:0/3079391208 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506009b1c0 con 0x56212a431b60 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 -- 172.21.15.49:0/3079391208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 msgr2=0x56212a442b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 --2- 172.21.15.49:0/3079391208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a442b90 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f506000e970 tx=0x7f5060003d20 comp rx=0 tx=0).stop 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 -- 172.21.15.49:0/3079391208 shutdown_connections 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 --2- 172.21.15.49:0/3079391208 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a436cd0 0x56212a44ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.657 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 --2- 172.21.15.49:0/3079391208 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431b60 0x56212a442b90 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 --2- 172.21.15.49:0/3079391208 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a431170 0x56212a433060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 -- 172.21.15.49:0/3079391208 >> 172.21.15.49:0/3079391208 conn(0x56212a42c6e0 msgr2=0x56212a42b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:40.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 -- 172.21.15.49:0/3079391208 shutdown_connections 2024-03-18T00:57:40.658 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 -- 172.21.15.49:0/3079391208 wait complete. 2024-03-18T00:57:40.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.656+0000 7f5080629740 1 -- start start 2024-03-18T00:57:40.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5080629740 1 Processor -- start 2024-03-18T00:57:40.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5080629740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431170 0x56212a442590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5080629740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431b60 0x56212a442ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.660 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5080629740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a440cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5080629740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f50781ac500 con 0x56212a431170 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5080629740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f50781abe30 con 0x56212a431b60 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5080629740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f50781a9070 con 0x56212a436cd0 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5080629740 10 client.0 _ll_register_callbacks cb 0x7f506000ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5076ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431170 0x56212a442590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f506ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431b60 0x56212a442ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5076ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431170 0x56212a442590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48166/0 (socket says 172.21.15.49:48166) 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f506ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431b60 0x56212a442ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:53194/0 (socket says 172.21.15.49:53194) 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f5076ffd640 1 -- 172.21.15.49:0/1708768687 learned_addr learned my addr 172.21.15.49:0/1708768687 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.657+0000 7f50777fe640 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a440cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f506ffff640 1 -- 172.21.15.49:0/1708768687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 msgr2=0x56212a440cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f506ffff640 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a440cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f506ffff640 1 -- 172.21.15.49:0/1708768687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431170 msgr2=0x56212a442590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:40.661 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f506ffff640 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431170 0x56212a442590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f506ffff640 1 -- 172.21.15.49:0/1708768687 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56212a35d9d0 con 0x56212a431b60 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f5076ffd640 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431170 0x56212a442590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f506ffff640 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431b60 0x56212a442ad0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5060009230 tx=0x7f50600975f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecc5da752ffec9d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f5056ffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f5056ffd640 1 -- 172.21.15.49:0/1708768687 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50600b9070 con 0x56212a431b60 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f5080629740 1 -- 172.21.15.49:0/1708768687 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50600bb040 con 0x56212a431b60 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f5056ffd640 1 -- 172.21.15.49:0/1708768687 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f50600b43c0 con 0x56212a431b60 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f5056ffd640 1 -- 172.21.15.49:0/1708768687 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5060080070 con 0x56212a431b60 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f5080629740 10 client.26456 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f5080629740 1 -- 172.21.15.49:0/1708768687 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56212a4addb0 con 0x56212a431b60 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.658+0000 7f5056ffd640 1 -- 172.21.15.49:0/1708768687 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50600bb040 con 0x56212a431b60 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.659+0000 7f5056ffd640 1 -- 172.21.15.49:0/1708768687 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f506009a540 con 0x56212a431b60 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.659+0000 7f5080629740 10 client.26456 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:40.662 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.659+0000 7f5080629740 10 client.26456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:40.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.659+0000 7f5080629740 20 client.26456 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:40.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.659+0000 7f5080629740 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56212a4ae5d0 0x56212a4b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:40.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.659+0000 7f5080629740 4 client.26456 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:57:40.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.659+0000 7f5080629740 1 -- 172.21.15.49:0/1708768687 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56212a39dd90 con 0x56212a4ae5d0 2024-03-18T00:57:40.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.659+0000 7f5076ffd640 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56212a4ae5d0 0x56212a4b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:40.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.660+0000 7f5076ffd640 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56212a4ae5d0 0x56212a4b09f0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af825543ed5132e0 server_cookie=a9dd3cd8d0191b82 in_seq=0 out_seq=0 2024-03-18T00:57:40.663 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.660+0000 7f5056ffd640 10 client.26456 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:40.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5056ffd640 1 -- 172.21.15.49:0/1708768687 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x56212a39dd90 con 0x56212a4ae5d0 2024-03-18T00:57:40.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5056ffd640 10 client.26456 handle_command_reply: tid=0 2024-03-18T00:57:40.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 client.26456 shutdown 2024-03-18T00:57:40.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 -- 172.21.15.49:0/1708768687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431b60 msgr2=0x56212a442ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431b60 0x56212a442ad0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5060009230 tx=0x7f50600975f0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 -- 172.21.15.49:0/1708768687 shutdown_connections 2024-03-18T00:57:40.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56212a4ae5d0 0x56212a4b09f0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.664 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56212a431b60 0x56212a442ad0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56212a436cd0 0x56212a440cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 --2- 172.21.15.49:0/1708768687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56212a431170 0x56212a442590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5056ffd640 0 client.26456 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 -- 172.21.15.49:0/1708768687 >> 172.21.15.49:0/1708768687 conn(0x56212a42c6e0 msgr2=0x56212a42b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 -- 172.21.15.49:0/1708768687 shutdown_connections 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 1 -- 172.21.15.49:0/1708768687 wait complete. 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.662+0000 7f5080629740 20 client.26456 trim_cache size 0 max 16384 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:57:40.665 INFO:teuthology.orchestra.run.smithi049.stdout: "4a77e765-b973-4cc7-b42b-359f648720b2": { 2024-03-18T00:57:40.666 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:57:40.666 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "4a77e765-b973-4cc7-b42b-359f648720b2", 2024-03-18T00:57:40.666 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:57:40.666 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:57:40.666 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:57:40.666 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 -- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50640a2880 msgr2=0x7f50640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50640a2880 0x7f50640a4d40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f5070005e30 tx=0x7f5070032590 comp rx=0 tx=0).stop 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 -- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5078104bb0 msgr2=0x7f507819a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5078104bb0 0x7f507819a290 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f506000e9a0 tx=0x7f506000ee60 comp rx=0 tx=0).stop 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 -- 172.21.15.49:0/3940039747 shutdown_connections 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f50640a2880 0x7f50640a4d40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f50781074e0 0x7f50781a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5078104bb0 0x7f507819a290 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 --2- 172.21.15.49:0/3940039747 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5078102280 0x7f5078199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 -- 172.21.15.49:0/3940039747 >> 172.21.15.49:0/3940039747 conn(0x7f50780fbc90 msgr2=0x7f50781019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 -- 172.21.15.49:0/3940039747 shutdown_connections 2024-03-18T00:57:40.667 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:57:40.663+0000 7f507da36640 1 -- 172.21.15.49:0/3940039747 wait complete. 2024-03-18T00:57:40.674 INFO:tasks.cephfs.filesystem:scrub status for tag:4a77e765-b973-4cc7-b42b-359f648720b2 - {'path': '/', 'tag': '4a77e765-b973-4cc7-b42b-359f648720b2', 'options': 'recursive,force'} 2024-03-18T00:57:41.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:41 smithi049 ceph-mon[32130]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 855 MiB/s wr, 2.05k op/s 2024-03-18T00:57:41.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:41 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:57:41.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:41 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:41.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:41 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:57:41.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:41 smithi107 ceph-mon[36270]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 855 MiB/s wr, 2.05k op/s 2024-03-18T00:57:41.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:41 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:57:41.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:41 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:41.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:41 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:57:41.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:41 smithi106 ceph-mon[35899]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 855 MiB/s wr, 2.05k op/s 2024-03-18T00:57:41.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:41 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:57:41.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:41 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:41.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:41 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:57:43.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:43 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:57:43.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:43 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:43.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:43 smithi049 ceph-mon[32130]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 856 MiB/s wr, 2.15k op/s 2024-03-18T00:57:43.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:43 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:57:43.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:43 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:43.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:43 smithi107 ceph-mon[36270]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 856 MiB/s wr, 2.15k op/s 2024-03-18T00:57:43.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:43 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:57:43.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:43 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:43.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:43 smithi106 ceph-mon[35899]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 856 MiB/s wr, 2.15k op/s 2024-03-18T00:57:44.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:44 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:44.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:44 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:44.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:44 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:57:45.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:45 smithi049 ceph-mon[32130]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 613 MiB/s wr, 1.70k op/s 2024-03-18T00:57:45.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:45 smithi107 ceph-mon[36270]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 613 MiB/s wr, 1.70k op/s 2024-03-18T00:57:45.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:45 smithi106 ceph-mon[35899]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 613 MiB/s wr, 1.70k op/s 2024-03-18T00:57:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.345+0000 7f2ba3868640 1 -- 172.21.15.49:0/3399409281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c107520 msgr2=0x7f2b9c109930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.345+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3399409281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c107520 0x7f2b9c109930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2b94009ca0 tx=0x7f2b9402c0f0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.345+0000 7f2ba3868640 1 -- 172.21.15.49:0/3399409281 shutdown_connections 2024-03-18T00:57:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.345+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3399409281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c107520 0x7f2b9c109930 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.345+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3399409281 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c104bf0 0x7f2b9c106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.345+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3399409281 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b9c1022c0 0x7f2b9c1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.345+0000 7f2ba3868640 1 -- 172.21.15.49:0/3399409281 >> 172.21.15.49:0/3399409281 conn(0x7f2b9c0fbc70 msgr2=0x7f2b9c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.345+0000 7f2ba3868640 1 -- 172.21.15.49:0/3399409281 shutdown_connections 2024-03-18T00:57:46.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.345+0000 7f2ba3868640 1 -- 172.21.15.49:0/3399409281 wait complete. 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba3868640 1 Processor -- start 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba3868640 1 -- start start 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba3868640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c1022c0 0x7f2b9c101c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba3868640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c104bf0 0x7f2b9c100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba3868640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b9c107520 0x7f2b9c1007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba3868640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c10d4a0 con 0x7f2b9c1022c0 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba3868640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c10d320 con 0x7f2b9c107520 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba3868640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c10d620 con 0x7f2b9c104bf0 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba0e03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c104bf0 0x7f2b9c100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba1604640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c1022c0 0x7f2b9c101c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba0e03640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c104bf0 0x7f2b9c100270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:42216/0 (socket says 172.21.15.49:42216) 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba0e03640 1 -- 172.21.15.49:0/3178597734 learned_addr learned my addr 172.21.15.49:0/3178597734 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.346+0000 7f2ba1e05640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b9c107520 0x7f2b9c1007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2ba0e03640 1 -- 172.21.15.49:0/3178597734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b9c107520 msgr2=0x7f2b9c1007b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b9c107520 0x7f2b9c1007b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2ba0e03640 1 -- 172.21.15.49:0/3178597734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c1022c0 msgr2=0x7f2b9c101c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c1022c0 0x7f2b9c101c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2ba0e03640 1 -- 172.21.15.49:0/3178597734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b9c101040 con 0x7f2b9c104bf0 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c104bf0 0x7f2b9c100270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00e9a0 tx=0x7f2b8c00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2b927fc640 1 -- 172.21.15.49:0/3178597734 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c00cd80 con 0x7f2b9c104bf0 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2ba3868640 1 -- 172.21.15.49:0/3178597734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b9c1a1b80 con 0x7f2b9c104bf0 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2b927fc640 1 -- 172.21.15.49:0/3178597734 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b8c0035f0 con 0x7f2b9c104bf0 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2b927fc640 1 -- 172.21.15.49:0/3178597734 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c010670 con 0x7f2b9c104bf0 2024-03-18T00:57:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.347+0000 7f2ba3868640 1 -- 172.21.15.49:0/3178597734 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b9c1a2120 con 0x7f2b9c104bf0 2024-03-18T00:57:46.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.350+0000 7f2b927fc640 1 -- 172.21.15.49:0/3178597734 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2b8c0037a0 con 0x7f2b9c104bf0 2024-03-18T00:57:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.351+0000 7f2ba644f740 1 Processor -- start 2024-03-18T00:57:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.351+0000 7f2b927fc640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b880a28a0 0x7f2b880a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.351+0000 7f2ba644f740 1 -- start start 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.351+0000 7f2b927fc640 1 -- 172.21.15.49:0/3178597734 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2b8c0cece0 con 0x7f2b9c104bf0 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.351+0000 7f2ba1e05640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b880a28a0 0x7f2b880a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffb9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 0x56174ffbc330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba644f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56174ff19a40 con 0x56174ffb4d30 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba644f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56174fec2920 con 0x56174ffb4340 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba644f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56174ff15ae0 con 0x56174ffb9ea0 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1604640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1604640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:53232/0 (socket says 172.21.15.49:53232) 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1604640 1 -- 172.21.15.49:0/619748977 learned_addr learned my addr 172.21.15.49:0/619748977 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffb9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1e05640 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 0x56174ffbc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.354 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1e05640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b880a28a0 0x7f2b880a4d60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2b94005e30 tx=0x7f2b94032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1604640 1 -- 172.21.15.49:0/619748977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 msgr2=0x56174ffbc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1604640 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 0x56174ffbc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1604640 1 -- 172.21.15.49:0/619748977 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 msgr2=0x56174ffb9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1604640 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffb9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1604640 1 -- 172.21.15.49:0/619748977 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56174feda460 con 0x56174ffb4340 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffb9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.352+0000 7f2ba1604640 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb4760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009d20 tx=0x7f2b98011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37bbf9e54d37694f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.353+0000 7f2b877fe640 1 -- 172.21.15.49:0/619748977 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9801e070 con 0x56174ffb4340 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.353+0000 7f2b877fe640 1 -- 172.21.15.49:0/619748977 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b98019b10 con 0x56174ffb4340 2024-03-18T00:57:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.353+0000 7f2b877fe640 1 -- 172.21.15.49:0/619748977 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9800ebb0 con 0x56174ffb4340 2024-03-18T00:57:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 -- 172.21.15.49:0/619748977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 msgr2=0x56174ffb4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb4760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2b98009d20 tx=0x7f2b98011810 comp rx=0 tx=0).stop 2024-03-18T00:57:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 -- 172.21.15.49:0/619748977 shutdown_connections 2024-03-18T00:57:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 0x56174ffbc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffb9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 --2- 172.21.15.49:0/619748977 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb4760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 -- 172.21.15.49:0/619748977 >> 172.21.15.49:0/619748977 conn(0x56174ffaf8b0 msgr2=0x56174ffafcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 -- 172.21.15.49:0/619748977 shutdown_connections 2024-03-18T00:57:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 -- 172.21.15.49:0/619748977 wait complete. 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 -- start start 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.354+0000 7f2ba644f740 1 Processor -- start 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4340 0x56174ffbe610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4d30 0x56174ffbeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 0x56174ffc5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba644f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56174ffbce50 con 0x56174ffb4340 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba644f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56174fed5d70 con 0x56174ffb4d30 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba644f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56174ffbcfd0 con 0x56174ffb9ea0 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba1604640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4340 0x56174ffbe610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba644f740 10 client.0 _ll_register_callbacks cb 0x56174fc2da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba1604640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4340 0x56174ffbe610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48200/0 (socket says 172.21.15.49:48200) 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba1604640 1 -- 172.21.15.49:0/4058903042 learned_addr learned my addr 172.21.15.49:0/4058903042 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4d30 0x56174ffbeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba1e05640 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 0x56174ffc5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba0e03640 1 -- 172.21.15.49:0/4058903042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 msgr2=0x56174ffc5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 0x56174ffc5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba0e03640 1 -- 172.21.15.49:0/4058903042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4340 msgr2=0x56174ffbe610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.355+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4340 0x56174ffbe610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2ba0e03640 1 -- 172.21.15.49:0/4058903042 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56174fc2e210 con 0x56174ffb4d30 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2ba1604640 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4340 0x56174ffbe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4d30 0x56174ffbeb50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0cc810 tx=0x7f2b8c09bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6be02a68e481b440 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2b84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2b84ff9640 1 -- 172.21.15.49:0/4058903042 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c09f7b0 con 0x56174ffb4d30 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2b84ff9640 1 -- 172.21.15.49:0/4058903042 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2b8c097040 con 0x56174ffb4d30 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2ba644f740 1 -- 172.21.15.49:0/4058903042 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b98019b10 con 0x56174ffb4d30 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2b84ff9640 1 -- 172.21.15.49:0/4058903042 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2b8c0a3e00 con 0x56174ffb4d30 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2ba644f740 10 client.26468 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2ba644f740 1 -- 172.21.15.49:0/4058903042 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56174fcfa620 con 0x56174ffb4d30 2024-03-18T00:57:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.356+0000 7f2b84ff9640 1 -- 172.21.15.49:0/4058903042 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b98019b10 con 0x56174ffb4d30 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2b84ff9640 1 -- 172.21.15.49:0/4058903042 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2b8c0b4360 con 0x56174ffb4d30 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2ba644f740 10 client.26468 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2ba644f740 10 client.26468 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2ba644f740 20 client.26468 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2ba644f740 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56174ffca3c0 0x56174ffcc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2ba644f740 4 client.26468 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2ba644f740 1 -- 172.21.15.49:0/4058903042 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56174ff26f80 con 0x56174ffca3c0 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2ba1604640 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56174ffca3c0 0x56174ffcc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2b84ff9640 10 client.26468 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:46.362 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.357+0000 7f2ba1604640 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56174ffca3c0 0x56174ffcc7f0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb449fcbbf9e6240 server_cookie=1365ea3c4253008a in_seq=0 out_seq=0 2024-03-18T00:57:46.363 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.359+0000 7f2b84ff9640 1 -- 172.21.15.49:0/4058903042 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56174ff26f80 con 0x56174ffca3c0 2024-03-18T00:57:46.363 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.359+0000 7f2b84ff9640 10 client.26468 handle_command_reply: tid=0 2024-03-18T00:57:46.363 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.359+0000 7f2ba644f740 1 client.26468 shutdown 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 -- 172.21.15.49:0/4058903042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4d30 msgr2=0x56174ffbeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4d30 0x56174ffbeb50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0cc810 tx=0x7f2b8c09bd60 comp rx=0 tx=0).stop 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 -- 172.21.15.49:0/4058903042 shutdown_connections 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56174ffca3c0 0x56174ffcc7f0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4d30 0x56174ffbeb50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb9ea0 0x56174ffc5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 --2- 172.21.15.49:0/4058903042 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4340 0x56174ffbe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2b84ff9640 0 client.26468 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 -- 172.21.15.49:0/4058903042 >> 172.21.15.49:0/4058903042 conn(0x56174ffaf8b0 msgr2=0x56174ffaebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 -- 172.21.15.49:0/4058903042 shutdown_connections 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 1 -- 172.21.15.49:0/4058903042 wait complete. 2024-03-18T00:57:46.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.360+0000 7f2ba644f740 20 client.26468 trim_cache size 0 max 16384 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba644f740 1 Processor -- start 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba644f740 1 -- start start 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4d30 0x56174ffc5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb9ea0 0x56174ffd1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba644f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1aae90 con 0x56174ffb9ea0 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba644f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1ac9d0 con 0x56174ffb4340 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba644f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1ab560 con 0x56174ffb4d30 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba1604640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba1e05640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb9ea0 0x56174ffd1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba1604640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:53252/0 (socket says 172.21.15.49:53252) 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba1604640 1 -- 172.21.15.49:0/1163924372 learned_addr learned my addr 172.21.15.49:0/1163924372 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.375+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/1163924372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4d30 0x56174ffc5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.376+0000 7f2ba1604640 1 -- 172.21.15.49:0/1163924372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4d30 msgr2=0x56174ffc5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.376+0000 7f2ba1604640 1 --2- 172.21.15.49:0/1163924372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4d30 0x56174ffc5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.376+0000 7f2ba1604640 1 -- 172.21.15.49:0/1163924372 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb9ea0 msgr2=0x56174ffd1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.376+0000 7f2ba1604640 1 --2- 172.21.15.49:0/1163924372 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb9ea0 0x56174ffd1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.376+0000 7f2ba1604640 1 -- 172.21.15.49:0/1163924372 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56174ffb5110 con 0x56174ffb4340 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.376+0000 7f2ba1604640 1 --2- 172.21.15.49:0/1163924372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb3220 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2b9801c010 tx=0x7f2b980051d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a59504bd9e830e50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.376+0000 7f2b86ffd640 1 -- 172.21.15.49:0/1163924372 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9802b030 con 0x56174ffb4340 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.376+0000 7f2b86ffd640 1 -- 172.21.15.49:0/1163924372 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b9801f040 con 0x56174ffb4340 2024-03-18T00:57:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.376+0000 7f2b86ffd640 1 -- 172.21.15.49:0/1163924372 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b9801e040 con 0x56174ffb4340 2024-03-18T00:57:46.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 -- 172.21.15.49:0/1163924372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 msgr2=0x56174ffb3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 --2- 172.21.15.49:0/1163924372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb3220 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2b9801c010 tx=0x7f2b980051d0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 -- 172.21.15.49:0/1163924372 shutdown_connections 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 --2- 172.21.15.49:0/1163924372 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb9ea0 0x56174ffd1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 --2- 172.21.15.49:0/1163924372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4d30 0x56174ffc5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 --2- 172.21.15.49:0/1163924372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb4340 0x56174ffb3220 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 -- 172.21.15.49:0/1163924372 >> 172.21.15.49:0/1163924372 conn(0x56174ffaf8b0 msgr2=0x56174ffb2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 -- 172.21.15.49:0/1163924372 shutdown_connections 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 -- 172.21.15.49:0/1163924372 wait complete. 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 -- start start 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.377+0000 7f2ba644f740 1 Processor -- start 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4340 0x56174ffc3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffc3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba644f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb9ea0 0x56174ffc4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba644f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1ac300 con 0x56174ffb4d30 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba644f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1abc30 con 0x56174ffb9ea0 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba644f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2b9c1a8e70 con 0x56174ffb4340 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba644f740 10 client.0 _ll_register_callbacks cb 0x7f2b8c00ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba0e03640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffc3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba1604640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4340 0x56174ffc3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba1e05640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb9ea0 0x56174ffc4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba0e03640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffc3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48222/0 (socket says 172.21.15.49:48222) 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba0e03640 1 -- 172.21.15.49:0/787470131 learned_addr learned my addr 172.21.15.49:0/787470131 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba0e03640 1 -- 172.21.15.49:0/787470131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4340 msgr2=0x56174ffc3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4340 0x56174ffc3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba0e03640 1 -- 172.21.15.49:0/787470131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb9ea0 msgr2=0x56174ffc4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:57:46.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb9ea0 0x56174ffc4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba0e03640 1 -- 172.21.15.49:0/787470131 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56174ffc54d0 con 0x56174ffb4d30 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2ba0e03640 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffc3ed0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0a3fc0 tx=0x7f2b8c09afc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6e68843768b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2b84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.378+0000 7f2b84ff9640 1 -- 172.21.15.49:0/787470131 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8c09b1a0 con 0x56174ffb4d30 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2b84ff9640 1 -- 172.21.15.49:0/787470131 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2b8c0b4240 con 0x56174ffb4d30 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba644f740 1 -- 172.21.15.49:0/787470131 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b9801f040 con 0x56174ffb4d30 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2b84ff9640 1 -- 172.21.15.49:0/787470131 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2b8c0860b0 con 0x56174ffb4d30 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba644f740 10 client.17322 fetch_fsmap learned FSMap version 28 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba644f740 1 -- 172.21.15.49:0/787470131 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561750030f40 con 0x56174ffb4d30 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2b84ff9640 1 -- 172.21.15.49:0/787470131 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b9801f040 con 0x56174ffb4d30 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2b84ff9640 1 -- 172.21.15.49:0/787470131 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2b8c004190 con 0x56174ffb4d30 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba644f740 10 client.17322 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba644f740 10 client.17322 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:57:46.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba644f740 20 client.17322 populate_metadata read hostname 'smithi049' 2024-03-18T00:57:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba644f740 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5617500318e0 0x561750033d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:57:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba644f740 4 client.17322 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:57:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba644f740 1 -- 172.21.15.49:0/787470131 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56174ff212d0 con 0x5617500318e0 2024-03-18T00:57:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.379+0000 7f2ba1604640 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5617500318e0 0x561750033d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:57:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.380+0000 7f2ba1604640 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5617500318e0 0x561750033d00 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d42f535d8d63556 server_cookie=c8353b371cb3a0c0 in_seq=0 out_seq=0 2024-03-18T00:57:46.385 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:46.380+0000 7f2b84ff9640 10 client.17322 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.776 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:57:46.777 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:46 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:57:46.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:57:46.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:57:46.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:46 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:57:46.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:57:46.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:46 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:57:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:47 smithi049 ceph-mon[32130]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 495 MiB/s wr, 1.40k op/s 2024-03-18T00:57:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:47 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-18T00:57:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:47 smithi049 ceph-mon[32130]: Unable to set osd_memory_target on smithi106 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-18T00:57:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:47 smithi049 ceph-mon[32130]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-18T00:57:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:47 smithi049 ceph-mon[32130]: Unable to set osd_memory_target on smithi107 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-18T00:57:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:57:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:47 smithi107 ceph-mon[36270]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 495 MiB/s wr, 1.40k op/s 2024-03-18T00:57:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:47 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-18T00:57:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:47 smithi107 ceph-mon[36270]: Unable to set osd_memory_target on smithi106 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-18T00:57:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:47 smithi107 ceph-mon[36270]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-18T00:57:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:47 smithi107 ceph-mon[36270]: Unable to set osd_memory_target on smithi107 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-18T00:57:47.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:57:47.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:47 smithi106 ceph-mon[35899]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 495 MiB/s wr, 1.40k op/s 2024-03-18T00:57:47.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:47 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi106 to 174.1M 2024-03-18T00:57:47.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:47 smithi106 ceph-mon[35899]: Unable to set osd_memory_target on smithi106 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-18T00:57:47.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:47 smithi106 ceph-mon[35899]: Adjusting osd_memory_target on smithi107 to 174.1M 2024-03-18T00:57:47.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:47 smithi106 ceph-mon[35899]: Unable to set osd_memory_target on smithi107 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-18T00:57:47.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:57:49.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:49 smithi049 ceph-mon[32130]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 397 MiB/s wr, 1.11k op/s 2024-03-18T00:57:49.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:49 smithi107 ceph-mon[36270]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 397 MiB/s wr, 1.11k op/s 2024-03-18T00:57:49.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:49 smithi106 ceph-mon[35899]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 397 MiB/s wr, 1.11k op/s 2024-03-18T00:57:50.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:50 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:57:50.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:50 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:57:50.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:50 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:57:51.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:51 smithi107 ceph-mon[36270]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 267 MiB/s wr, 611 op/s 2024-03-18T00:57:51.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:51 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:57:51.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:51 smithi049 ceph-mon[32130]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 267 MiB/s wr, 611 op/s 2024-03-18T00:57:51.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:51 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:57:51.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:51 smithi106 ceph-mon[35899]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 267 MiB/s wr, 611 op/s 2024-03-18T00:57:51.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:51 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:57:52.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2b84ff9640 1 -- 172.21.15.49:0/787470131 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56174ff212d0 con 0x5617500318e0 2024-03-18T00:57:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2b84ff9640 10 client.17322 handle_command_reply: tid=0 2024-03-18T00:57:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2ba644f740 1 client.17322 shutdown 2024-03-18T00:57:52.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2ba644f740 1 -- 172.21.15.49:0/787470131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 msgr2=0x56174ffc3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2ba644f740 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffc3ed0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c0a3fc0 tx=0x7f2b8c09afc0 comp rx=0 tx=0).stop 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2ba644f740 1 -- 172.21.15.49:0/787470131 shutdown_connections 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2ba644f740 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5617500318e0 0x561750033d00 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2ba644f740 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56174ffb9ea0 0x56174ffc4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2ba644f740 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56174ffb4d30 0x56174ffc3ed0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2ba644f740 1 --2- 172.21.15.49:0/787470131 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56174ffb4340 0x56174ffc3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2ba644f740 1 -- 172.21.15.49:0/787470131 >> 172.21.15.49:0/787470131 conn(0x56174ffaf8b0 msgr2=0x56174ffb33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.327+0000 7f2b84ff9640 0 client.17322 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.328+0000 7f2ba644f740 1 -- 172.21.15.49:0/787470131 shutdown_connections 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.328+0000 7f2ba644f740 1 -- 172.21.15.49:0/787470131 wait complete. 2024-03-18T00:57:52.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.328+0000 7f2ba644f740 20 client.17322 trim_cache size 0 max 16384 2024-03-18T00:57:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:57:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:57:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:57:52.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 -- 172.21.15.49:0/3178597734 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b880a28a0 msgr2=0x7f2b880a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b880a28a0 0x7f2b880a4d60 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2b94005e30 tx=0x7f2b94032590 comp rx=0 tx=0).stop 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 -- 172.21.15.49:0/3178597734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c104bf0 msgr2=0x7f2b9c100270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c104bf0 0x7f2b9c100270 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2b8c00e9a0 tx=0x7f2b8c00ee60 comp rx=0 tx=0).stop 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 -- 172.21.15.49:0/3178597734 shutdown_connections 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2b880a28a0 0x7f2b880a4d60 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2b9c107520 0x7f2b9c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2b9c104bf0 0x7f2b9c100270 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 --2- 172.21.15.49:0/3178597734 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2b9c1022c0 0x7f2b9c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.329+0000 7f2ba3868640 1 -- 172.21.15.49:0/3178597734 >> 172.21.15.49:0/3178597734 conn(0x7f2b9c0fbc70 msgr2=0x7f2b9c102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.330+0000 7f2ba3868640 1 -- 172.21.15.49:0/3178597734 shutdown_connections 2024-03-18T00:57:52.330 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:57:52.330+0000 7f2ba3868640 1 -- 172.21.15.49:0/3178597734 wait complete. 2024-03-18T00:57:53.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:53 smithi049 ceph-mon[32130]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 190 MiB/s wr, 376 op/s 2024-03-18T00:57:53.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:53 smithi107 ceph-mon[36270]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 190 MiB/s wr, 376 op/s 2024-03-18T00:57:53.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:53 smithi106 ceph-mon[35899]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 190 MiB/s wr, 376 op/s 2024-03-18T00:57:55.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:55 smithi049 ceph-mon[32130]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 89 MiB/s wr, 113 op/s 2024-03-18T00:57:55.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:55 smithi107 ceph-mon[36270]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 89 MiB/s wr, 113 op/s 2024-03-18T00:57:55.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:55 smithi106 ceph-mon[35899]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 89 MiB/s wr, 113 op/s 2024-03-18T00:57:57.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:57 smithi049 ceph-mon[32130]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 87 MiB/s wr, 110 op/s 2024-03-18T00:57:57.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:57 smithi107 ceph-mon[36270]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 87 MiB/s wr, 110 op/s 2024-03-18T00:57:57.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:57 smithi106 ceph-mon[35899]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 87 MiB/s wr, 110 op/s 2024-03-18T00:57:59.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:57:59 smithi049 ceph-mon[32130]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 87 MiB/s wr, 111 op/s 2024-03-18T00:57:59.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:57:59 smithi107 ceph-mon[36270]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 87 MiB/s wr, 111 op/s 2024-03-18T00:57:59.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:57:59 smithi106 ceph-mon[35899]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 87 MiB/s wr, 111 op/s 2024-03-18T00:58:01.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:01 smithi049 ceph-mon[32130]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 84 MiB/s wr, 108 op/s 2024-03-18T00:58:01.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:01 smithi107 ceph-mon[36270]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 84 MiB/s wr, 108 op/s 2024-03-18T00:58:01.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:01 smithi106 ceph-mon[35899]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 84 MiB/s wr, 108 op/s 2024-03-18T00:58:03.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:03 smithi049 ceph-mon[32130]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 100 MiB/s wr, 130 op/s 2024-03-18T00:58:03.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:03 smithi107 ceph-mon[36270]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 100 MiB/s wr, 130 op/s 2024-03-18T00:58:03.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:03 smithi106 ceph-mon[35899]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 100 MiB/s wr, 130 op/s 2024-03-18T00:58:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.433+0000 7fdba7fff640 1 -- 172.21.15.49:0/3494950566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8102120 msgr2=0x7fdba81089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.433+0000 7fdba7fff640 1 --2- 172.21.15.49:0/3494950566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8102120 0x7fdba81089b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009be0 tx=0x7fdba002bef0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.433+0000 7fdba7fff640 1 -- 172.21.15.49:0/3494950566 shutdown_connections 2024-03-18T00:58:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.433+0000 7fdba7fff640 1 --2- 172.21.15.49:0/3494950566 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8102120 0x7fdba81089b0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.433+0000 7fdba7fff640 1 --2- 172.21.15.49:0/3494950566 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdba8101760 0x7fdba8101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.433+0000 7fdba7fff640 1 --2- 172.21.15.49:0/3494950566 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdba8100570 0x7fdba8100990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.435 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.433+0000 7fdba7fff640 1 -- 172.21.15.49:0/3494950566 >> 172.21.15.49:0/3494950566 conn(0x7fdba80fbae0 msgr2=0x7fdba80fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.433+0000 7fdba7fff640 1 -- 172.21.15.49:0/3494950566 shutdown_connections 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.433+0000 7fdba7fff640 1 -- 172.21.15.49:0/3494950566 wait complete. 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba7fff640 1 Processor -- start 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba7fff640 1 -- start start 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdba8100570 0x7fdba8196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8101760 0x7fdba8196d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdba8102120 0x7fdba819ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba7fff640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdba810c240 con 0x7fdba8102120 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba7fff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdba810c0c0 con 0x7fdba8100570 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba7fff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdba810c3c0 con 0x7fdba8101760 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdba8100570 0x7fdba8196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdb9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8101760 0x7fdba8196d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdba8102120 0x7fdba819ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdba8100570 0x7fdba8196820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48172/0 (socket says 172.21.15.49:48172) 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdb9ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8101760 0x7fdba8196d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56090/0 (socket says 172.21.15.49:56090) 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdba8102120 0x7fdba819ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47466/0 (socket says 172.21.15.49:47466) 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdba6ffd640 1 -- 172.21.15.49:0/978656526 learned_addr learned my addr 172.21.15.49:0/978656526 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdb9ffff640 1 -- 172.21.15.49:0/978656526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdba8100570 msgr2=0x7fdba8196820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdba8100570 0x7fdba8196820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdb9ffff640 1 -- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdba8102120 msgr2=0x7fdba819ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdba8102120 0x7fdba819ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.434+0000 7fdb9ffff640 1 -- 172.21.15.49:0/978656526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba819e440 con 0x7fdba8101760 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.435+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdba8100570 0x7fdba8196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.435+0000 7fdba77fe640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdba8102120 0x7fdba819ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.435+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8101760 0x7fdba8196d60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdb90005fd0 tx=0x7fdb9000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.435+0000 7fdba4ff9640 1 -- 172.21.15.49:0/978656526 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9000cd00 con 0x7fdba8101760 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.435+0000 7fdba7fff640 1 -- 172.21.15.49:0/978656526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba8069cb0 con 0x7fdba8101760 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.435+0000 7fdba4ff9640 1 -- 172.21.15.49:0/978656526 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb900035f0 con 0x7fdba8101760 2024-03-18T00:58:05.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.435+0000 7fdba4ff9640 1 -- 172.21.15.49:0/978656526 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb90010670 con 0x7fdba8101760 2024-03-18T00:58:05.438 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.435+0000 7fdba7fff640 1 -- 172.21.15.49:0/978656526 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba806a200 con 0x7fdba8101760 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.437+0000 7fdba4ff9640 1 -- 172.21.15.49:0/978656526 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdb900037a0 con 0x7fdba8101760 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.437+0000 7fdbb01e2740 1 Processor -- start 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.437+0000 7fdbb01e2740 1 -- start start 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b7058f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b70d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d47b66aa40 con 0x55d47b70b030 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d47b613920 con 0x55d47b7054d0 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d47b666ae0 con 0x55d47b705ec0 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba77fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b7058f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdb9ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b70d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.443 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba77fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b7058f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48174/0 (socket says 172.21.15.49:48174) 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba77fe640 1 -- 172.21.15.49:0/37146152 learned_addr learned my addr 172.21.15.49:0/37146152 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56100/0 (socket says 172.21.15.49:56100) 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba4ff9640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdb940a27f0 0x7fdb940a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba77fe640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdb940a27f0 0x7fdb940a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba4ff9640 1 -- 172.21.15.49:0/978656526 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fdb900cec70 con 0x7fdba8101760 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba6ffd640 1 -- 172.21.15.49:0/37146152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 msgr2=0x55d47b7058f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/37146152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b7058f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba6ffd640 1 -- 172.21.15.49:0/37146152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 msgr2=0x55d47b70d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/37146152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b70d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.438+0000 7fdba6ffd640 1 -- 172.21.15.49:0/37146152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d47b62b460 con 0x55d47b705ec0 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.439+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/37146152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70a9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdb980058c0 tx=0x7fdb98005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7839a4583fc48ea6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.439+0000 7fdba77fe640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdb940a27f0 0x7fdb940a4cb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55d47b706720 tx=0x7fdba002bef0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.439+0000 7fdb9d7fa640 1 -- 172.21.15.49:0/37146152 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9801c070 con 0x55d47b705ec0 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.439+0000 7fdb9d7fa640 1 -- 172.21.15.49:0/37146152 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb98004230 con 0x55d47b705ec0 2024-03-18T00:58:05.444 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.439+0000 7fdb9d7fa640 1 -- 172.21.15.49:0/37146152 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9800b940 con 0x55d47b705ec0 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 -- 172.21.15.49:0/37146152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 msgr2=0x55d47b70a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/37146152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70a9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdb980058c0 tx=0x7fdb98005f90 comp rx=0 tx=0).stop 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 -- 172.21.15.49:0/37146152 shutdown_connections 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/37146152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b70d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/37146152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70a9c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/37146152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b7058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 -- 172.21.15.49:0/37146152 >> 172.21.15.49:0/37146152 conn(0x55d47b700a40 msgr2=0x55d47b700e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 -- 172.21.15.49:0/37146152 shutdown_connections 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 -- 172.21.15.49:0/37146152 wait complete. 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 -- start start 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 Processor -- start 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b7054d0 0x55d47b70f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b70b030 0x55d47b716d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d47b70dfe0 con 0x55d47b7054d0 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d47b626d70 con 0x55d47b70b030 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.440+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d47b70e160 con 0x55d47b705ec0 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdbb01e2740 10 client.0 _ll_register_callbacks cb 0x55d47b37ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b7054d0 0x55d47b70f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b7054d0 0x55d47b70f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47478/0 (socket says 172.21.15.49:47478) 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba6ffd640 1 -- 172.21.15.49:0/1372360236 learned_addr learned my addr 172.21.15.49:0/1372360236 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba77fe640 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b70b030 0x55d47b716d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba6ffd640 1 -- 172.21.15.49:0/1372360236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 msgr2=0x55d47b70fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba6ffd640 1 -- 172.21.15.49:0/1372360236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b70b030 msgr2=0x55d47b716d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b70b030 0x55d47b716d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba6ffd640 1 -- 172.21.15.49:0/1372360236 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d47b37f210 con 0x55d47b7054d0 2024-03-18T00:58:05.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b7054d0 0x55d47b70f7a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fdb980097a0 tx=0x7fdb98003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab72f13860ac60e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdb8e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/1372360236 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9801c070 con 0x55d47b7054d0 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/1372360236 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fdb980043e0 con 0x55d47b7054d0 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.441+0000 7fdbb01e2740 1 -- 172.21.15.49:0/1372360236 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb98004230 con 0x55d47b7054d0 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/1372360236 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdb9800b680 con 0x55d47b7054d0 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdbb01e2740 10 client.17334 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdbb01e2740 1 -- 172.21.15.49:0/1372360236 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d47b44b620 con 0x55d47b7054d0 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/1372360236 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb98004230 con 0x55d47b7054d0 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/1372360236 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fdb9802a810 con 0x55d47b7054d0 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdbb01e2740 10 client.17334 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdbb01e2740 10 client.17334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdbb01e2740 20 client.17334 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d47b71b330 0x55d47b71d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdbb01e2740 4 client.17334 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:05.447 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdbb01e2740 1 -- 172.21.15.49:0/1372360236 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d47b677f80 con 0x55d47b71b330 2024-03-18T00:58:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.442+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d47b71b330 0x55d47b71d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.443+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d47b71b330 0x55d47b71d720 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68f0615f57517658 server_cookie=e816bcb8b928b708 in_seq=0 out_seq=0 2024-03-18T00:58:05.448 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.443+0000 7fdb8e7fc640 10 client.17334 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/1372360236 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d47b677f80 con 0x55d47b71b330 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdb8e7fc640 10 client.17334 handle_command_reply: tid=0 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 client.17334 shutdown 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 -- 172.21.15.49:0/1372360236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b7054d0 msgr2=0x55d47b70f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b7054d0 0x55d47b70f7a0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fdb980097a0 tx=0x7fdb98003cd0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 -- 172.21.15.49:0/1372360236 shutdown_connections 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d47b71b330 0x55d47b71d720 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b70b030 0x55d47b716d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b705ec0 0x55d47b70fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdb8e7fc640 0 client.17334 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/1372360236 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b7054d0 0x55d47b70f7a0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 -- 172.21.15.49:0/1372360236 >> 172.21.15.49:0/1372360236 conn(0x55d47b700a40 msgr2=0x55d47b6ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 -- 172.21.15.49:0/1372360236 shutdown_connections 2024-03-18T00:58:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 1 -- 172.21.15.49:0/1372360236 wait complete. 2024-03-18T00:58:05.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.446+0000 7fdbb01e2740 20 client.17334 trim_cache size 0 max 16384 2024-03-18T00:58:05.461 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdbb01e2740 1 Processor -- start 2024-03-18T00:58:05.461 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdbb01e2740 1 -- start start 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b7054d0 0x55d47b723030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b705ec0 0x55d47b716ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b6ffae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdba81a6a00 con 0x55d47b70b030 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdba81a8540 con 0x55d47b705ec0 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdba81a70d0 con 0x55d47b7054d0 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdba77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b6ffae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdba77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b6ffae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47480/0 (socket says 172.21.15.49:47480) 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdba77fe640 1 -- 172.21.15.49:0/3820848429 learned_addr learned my addr 172.21.15.49:0/3820848429 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b7054d0 0x55d47b723030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.463 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.462+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b705ec0 0x55d47b716ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdba77fe640 1 -- 172.21.15.49:0/3820848429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b7054d0 msgr2=0x55d47b723030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdba77fe640 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b7054d0 0x55d47b723030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdba77fe640 1 -- 172.21.15.49:0/3820848429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b705ec0 msgr2=0x55d47b716ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdba77fe640 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b705ec0 0x55d47b716ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdba77fe640 1 -- 172.21.15.49:0/3820848429 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d47b70d980 con 0x55d47b70b030 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b705ec0 0x55d47b716ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdba77fe640 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b6ffae0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009be0 tx=0x7fdba0005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f9a3344eaf6f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdb9cff9640 1 -- 172.21.15.49:0/3820848429 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba0049040 con 0x55d47b70b030 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdb9cff9640 1 -- 172.21.15.49:0/3820848429 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdba0053070 con 0x55d47b70b030 2024-03-18T00:58:05.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.463+0000 7fdb9cff9640 1 -- 172.21.15.49:0/3820848429 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdba00349a0 con 0x55d47b70b030 2024-03-18T00:58:05.467 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 -- 172.21.15.49:0/3820848429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 msgr2=0x55d47b6ffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:05.467 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b6ffae0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fdba0009be0 tx=0x7fdba0005e50 comp rx=0 tx=0).stop 2024-03-18T00:58:05.467 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 -- 172.21.15.49:0/3820848429 shutdown_connections 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b705ec0 0x55d47b716ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b70b030 0x55d47b6ffae0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/3820848429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b7054d0 0x55d47b723030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 -- 172.21.15.49:0/3820848429 >> 172.21.15.49:0/3820848429 conn(0x55d47b700a40 msgr2=0x55d47b6ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 -- 172.21.15.49:0/3820848429 shutdown_connections 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 -- 172.21.15.49:0/3820848429 wait complete. 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 -- start start 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 Processor -- start 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.464+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b714810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b705ec0 0x55d47b714d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdbb01e2740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b70b030 0x55d47b715cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdba81a7e70 con 0x55d47b705ec0 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdba81a77a0 con 0x55d47b7054d0 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdbb01e2740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdba81a49e0 con 0x55d47b70b030 2024-03-18T00:58:05.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdbb01e2740 10 client.0 _ll_register_callbacks cb 0x7fdb900976c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb9ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b705ec0 0x55d47b714d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdba6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b714810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdba6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b714810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48194/0 (socket says 172.21.15.49:48194) 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdba6ffd640 1 -- 172.21.15.49:0/398718293 learned_addr learned my addr 172.21.15.49:0/398718293 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb9ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b705ec0 0x55d47b714d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47496/0 (socket says 172.21.15.49:47496) 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdba77fe640 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b70b030 0x55d47b715cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb9ffff640 1 -- 172.21.15.49:0/398718293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b70b030 msgr2=0x55d47b715cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b70b030 0x55d47b715cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb9ffff640 1 -- 172.21.15.49:0/398718293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 msgr2=0x55d47b714810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b714810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb9ffff640 1 -- 172.21.15.49:0/398718293 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d47b7162e0 con 0x55d47b705ec0 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b714810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb9ffff640 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b705ec0 0x55d47b714d50 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdb900cc7a0 tx=0x7fdb90097770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ab418f0410f18b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb8e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:05.469 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/398718293 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9009af50 con 0x55d47b705ec0 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/398718293 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fdb90003af0 con 0x55d47b705ec0 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdbb01e2740 1 -- 172.21.15.49:0/398718293 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdba0053070 con 0x55d47b705ec0 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/398718293 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdb90009530 con 0x55d47b705ec0 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.465+0000 7fdbb01e2740 10 client.17340 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdbb01e2740 1 -- 172.21.15.49:0/398718293 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d47b781a40 con 0x55d47b705ec0 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/398718293 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdba0053070 con 0x55d47b705ec0 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/398718293 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fdb900d6040 con 0x55d47b705ec0 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdbb01e2740 10 client.17340 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdbb01e2740 10 client.17340 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdbb01e2740 20 client.17340 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d47b7823f0 0x55d47b784810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdbb01e2740 4 client.17340 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdbb01e2740 1 -- 172.21.15.49:0/398718293 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d47b70ad60 con 0x55d47b7823f0 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d47b7823f0 0x55d47b784810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:05.470 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdb8e7fc640 10 client.17340 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:05.471 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:05.466+0000 7fdba6ffd640 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d47b7823f0 0x55d47b784810 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcee6c645e6635a9 server_cookie=dd45bc80a5a1f3c0 in_seq=0 out_seq=0 2024-03-18T00:58:05.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:05 smithi049 ceph-mon[32130]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 85 MiB/s wr, 105 op/s 2024-03-18T00:58:05.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:05 smithi107 ceph-mon[36270]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 85 MiB/s wr, 105 op/s 2024-03-18T00:58:05.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:05 smithi106 ceph-mon[35899]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 85 MiB/s wr, 105 op/s 2024-03-18T00:58:07.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:07 smithi049 ceph-mon[32130]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 74 MiB/s wr, 99 op/s 2024-03-18T00:58:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:07 smithi107 ceph-mon[36270]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 74 MiB/s wr, 99 op/s 2024-03-18T00:58:07.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:07 smithi106 ceph-mon[35899]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 74 MiB/s wr, 99 op/s 2024-03-18T00:58:09.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:09 smithi049 ceph-mon[32130]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 62 MiB/s wr, 86 op/s 2024-03-18T00:58:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:09 smithi107 ceph-mon[36270]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 62 MiB/s wr, 86 op/s 2024-03-18T00:58:09.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:09 smithi106 ceph-mon[35899]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 62 MiB/s wr, 86 op/s 2024-03-18T00:58:10.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:58:10.675 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:58:10.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.800+0000 7fc03dc34640 1 -- 172.21.15.49:0/3740494503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc038102290 msgr2=0x7fc038108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:10.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.800+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3740494503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc038102290 0x7fc038108b20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc030009ca0 tx=0x7fc03002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.800+0000 7fc03dc34640 1 -- 172.21.15.49:0/3740494503 shutdown_connections 2024-03-18T00:58:10.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.800+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3740494503 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc038102290 0x7fc038108b20 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.800+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3740494503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0381018d0 0x7fc038101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.800+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3740494503 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc0381006e0 0x7fc038100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.800+0000 7fc03dc34640 1 -- 172.21.15.49:0/3740494503 >> 172.21.15.49:0/3740494503 conn(0x7fc0380fbc70 msgr2=0x7fc0380fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.800+0000 7fc03dc34640 1 -- 172.21.15.49:0/3740494503 shutdown_connections 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.800+0000 7fc03dc34640 1 -- 172.21.15.49:0/3740494503 wait complete. 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc03dc34640 1 Processor -- start 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc03dc34640 1 -- start start 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc03dc34640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0381006e0 0x7fc038104620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc03dc34640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0381018d0 0x7fc038102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc03dc34640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc038102290 0x7fc0381031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc03dc34640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc03810c360 con 0x7fc0381018d0 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc03dc34640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc03810c1e0 con 0x7fc038102290 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc03dc34640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc03810c4e0 con 0x7fc0381006e0 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc036ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0381018d0 0x7fc038102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0381006e0 0x7fc038104620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0381006e0 0x7fc038104620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:47280/0 (socket says 172.21.15.49:47280) 2024-03-18T00:58:10.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc036ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0381018d0 0x7fc038102cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47278/0 (socket says 172.21.15.49:47278) 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc0377fe640 1 -- 172.21.15.49:0/3809605600 learned_addr learned my addr 172.21.15.49:0/3809605600 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc037fff640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc038102290 0x7fc0381031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc036ffd640 1 -- 172.21.15.49:0/3809605600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0381006e0 msgr2=0x7fc038104620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.801+0000 7fc036ffd640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0381006e0 0x7fc038104620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc036ffd640 1 -- 172.21.15.49:0/3809605600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc038102290 msgr2=0x7fc0381031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc036ffd640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc038102290 0x7fc0381031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc036ffd640 1 -- 172.21.15.49:0/3809605600 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc038103ab0 con 0x7fc0381018d0 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc0377fe640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0381006e0 0x7fc038104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc036ffd640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0381018d0 0x7fc038102cb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00e9f0 tx=0x7fc02c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc034ff9640 1 -- 172.21.15.49:0/3809605600 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c00ccc0 con 0x7fc0381018d0 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc034ff9640 1 -- 172.21.15.49:0/3809605600 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc02c005020 con 0x7fc0381018d0 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc034ff9640 1 -- 172.21.15.49:0/3809605600 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c0049e0 con 0x7fc0381018d0 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc03dc34640 1 -- 172.21.15.49:0/3809605600 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0381a1c80 con 0x7fc0381018d0 2024-03-18T00:58:10.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.802+0000 7fc03dc34640 1 -- 172.21.15.49:0/3809605600 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0381a2150 con 0x7fc0381018d0 2024-03-18T00:58:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.805+0000 7fc034ff9640 1 -- 172.21.15.49:0/3809605600 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc02c005560 con 0x7fc0381018d0 2024-03-18T00:58:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.805+0000 7fc04081b740 1 Processor -- start 2024-03-18T00:58:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.805+0000 7fc04081b740 1 -- start start 2024-03-18T00:58:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.805+0000 7fc04081b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e90d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.805+0000 7fc04081b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e912660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.805+0000 7fc04081b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e915160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.805+0000 7fc04081b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56235e872500 con 0x56235e912cd0 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.805+0000 7fc04081b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56235e81abe0 con 0x56235e90db60 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.805+0000 7fc04081b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56235e86e5a0 con 0x56235e90d170 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc036ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e915160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e912660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc036ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e915160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47290/0 (socket says 172.21.15.49:47290) 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc036ffd640 1 -- 172.21.15.49:0/230981019 learned_addr learned my addr 172.21.15.49:0/230981019 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc037fff640 1 --2- 172.21.15.49:0/230981019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e90d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc034ff9640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0200a2800 0x7fc0200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc036ffd640 1 -- 172.21.15.49:0/230981019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 msgr2=0x56235e90d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc036ffd640 1 --2- 172.21.15.49:0/230981019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e90d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc036ffd640 1 -- 172.21.15.49:0/230981019 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 msgr2=0x56235e912660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc036ffd640 1 --2- 172.21.15.49:0/230981019 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e912660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc036ffd640 1 -- 172.21.15.49:0/230981019 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56235e82e4b0 con 0x56235e912cd0 2024-03-18T00:58:10.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc0377fe640 1 --2- 172.21.15.49:0/230981019 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e912660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc037fff640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0200a2800 0x7fc0200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc034ff9640 1 -- 172.21.15.49:0/3809605600 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc02c0cea40 con 0x7fc0381018d0 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc036ffd640 1 --2- 172.21.15.49:0/230981019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e915160 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc02c079310 tx=0x7fc02c066180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ac31e98d52a8e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc01dffb640 1 -- 172.21.15.49:0/230981019 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c09ad20 con 0x56235e912cd0 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc01dffb640 1 -- 172.21.15.49:0/230981019 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc02c07ab30 con 0x56235e912cd0 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc01dffb640 1 -- 172.21.15.49:0/230981019 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02c07a230 con 0x56235e912cd0 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.806+0000 7fc037fff640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0200a2800 0x7fc0200a4cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc030005e30 tx=0x7fc03002e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.807+0000 7fc04081b740 1 -- 172.21.15.49:0/230981019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 msgr2=0x56235e915160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.807+0000 7fc04081b740 1 --2- 172.21.15.49:0/230981019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e915160 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fc02c079310 tx=0x7fc02c066180 comp rx=0 tx=0).stop 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.807+0000 7fc04081b740 1 -- 172.21.15.49:0/230981019 shutdown_connections 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.807+0000 7fc04081b740 1 --2- 172.21.15.49:0/230981019 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e912660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.807+0000 7fc04081b740 1 --2- 172.21.15.49:0/230981019 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e915160 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.807+0000 7fc04081b740 1 --2- 172.21.15.49:0/230981019 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e90d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.807+0000 7fc04081b740 1 -- 172.21.15.49:0/230981019 >> 172.21.15.49:0/230981019 conn(0x56235e9086e0 msgr2=0x56235e908ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.807+0000 7fc04081b740 1 -- 172.21.15.49:0/230981019 shutdown_connections 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.807+0000 7fc04081b740 1 -- 172.21.15.49:0/230981019 wait complete. 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc04081b740 1 -- start start 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc04081b740 1 Processor -- start 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc04081b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e9175e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc04081b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e90db60 0x56235e917b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc04081b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e912cd0 0x56235e91eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc04081b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56235e915c20 con 0x56235e90db60 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc04081b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56235e82e160 con 0x56235e912cd0 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc04081b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56235e915da0 con 0x56235e90d170 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc04081b740 10 client.0 _ll_register_callbacks cb 0x56235e812150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc036ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e90db60 0x56235e917b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e9175e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc036ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e90db60 0x56235e917b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47298/0 (socket says 172.21.15.49:47298) 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc036ffd640 1 -- 172.21.15.49:0/3755519756 learned_addr learned my addr 172.21.15.49:0/3755519756 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.808+0000 7fc037fff640 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e912cd0 0x56235e91eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc0377fe640 1 -- 172.21.15.49:0/3755519756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e912cd0 msgr2=0x56235e91eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc0377fe640 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e912cd0 0x56235e91eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc0377fe640 1 -- 172.21.15.49:0/3755519756 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e90db60 msgr2=0x56235e917b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc0377fe640 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e90db60 0x56235e917b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc0377fe640 1 -- 172.21.15.49:0/3755519756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56235e6507d0 con 0x56235e90d170 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc036ffd640 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e90db60 0x56235e917b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc0377fe640 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e9175e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x56235e90df60 tx=0x7fc02800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0b4c9691756bc64 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc016ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc016ffd640 1 -- 172.21.15.49:0/3755519756 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028012110 con 0x56235e90d170 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc016ffd640 1 -- 172.21.15.49:0/3755519756 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc028004050 con 0x56235e90d170 2024-03-18T00:58:10.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc04081b740 1 -- 172.21.15.49:0/3755519756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56235e87fa40 con 0x56235e90d170 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc016ffd640 1 -- 172.21.15.49:0/3755519756 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc0280122b0 con 0x56235e90d170 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc04081b740 10 client.26596 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc04081b740 1 -- 172.21.15.49:0/3755519756 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56235e5863f0 con 0x56235e90d170 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.809+0000 7fc016ffd640 1 -- 172.21.15.49:0/3755519756 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56235e87fa40 con 0x56235e90d170 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc016ffd640 1 -- 172.21.15.49:0/3755519756 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc028010430 con 0x56235e90d170 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc04081b740 10 client.26596 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc04081b740 10 client.26596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc04081b740 20 client.26596 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc04081b740 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56235e923140 0x56235e9255e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc04081b740 4 client.26596 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc04081b740 1 -- 172.21.15.49:0/3755519756 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56235e879d90 con 0x56235e923140 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc036ffd640 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56235e923140 0x56235e9255e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc016ffd640 10 client.26596 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:10.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:10.810+0000 7fc036ffd640 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56235e923140 0x56235e9255e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55e5d5117078ea44 server_cookie=1700eb309e9e26c4 in_seq=0 out_seq=0 2024-03-18T00:58:11.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:11 smithi049 ceph-mon[32130]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 51 MiB/s wr, 78 op/s 2024-03-18T00:58:11.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:11 smithi107 ceph-mon[36270]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 51 MiB/s wr, 78 op/s 2024-03-18T00:58:11.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:11 smithi106 ceph-mon[35899]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 51 MiB/s wr, 78 op/s 2024-03-18T00:58:11.988 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.988+0000 7fdb8e7fc640 1 -- 172.21.15.49:0/398718293 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d47b70ad60 con 0x55d47b7823f0 2024-03-18T00:58:11.988 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.988+0000 7fdb8e7fc640 10 client.17340 handle_command_reply: tid=0 2024-03-18T00:58:11.988 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.988+0000 7fdbb01e2740 1 client.17340 shutdown 2024-03-18T00:58:11.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.988+0000 7fc016ffd640 1 -- 172.21.15.49:0/3755519756 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56235e879d90 con 0x56235e923140 2024-03-18T00:58:11.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.988+0000 7fc016ffd640 10 client.26596 handle_command_reply: tid=0 2024-03-18T00:58:11.989 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc04081b740 1 client.26596 shutdown 2024-03-18T00:58:11.989 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 -- 172.21.15.49:0/398718293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b705ec0 msgr2=0x55d47b714d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:11.989 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b705ec0 0x55d47b714d50 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fdb900cc7a0 tx=0x7fdb90097770 comp rx=0 tx=0).stop 2024-03-18T00:58:11.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc04081b740 1 -- 172.21.15.49:0/3755519756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 msgr2=0x56235e9175e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:11.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc04081b740 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e9175e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x56235e90df60 tx=0x7fc02800f790 comp rx=0 tx=0).stop 2024-03-18T00:58:11.990 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 -- 172.21.15.49:0/398718293 shutdown_connections 2024-03-18T00:58:11.990 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55d47b7823f0 0x55d47b784810 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.990 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d47b70b030 0x55d47b715cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.991 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d47b705ec0 0x55d47b714d50 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.991 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 --2- 172.21.15.49:0/398718293 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d47b7054d0 0x55d47b714810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.991 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 -- 172.21.15.49:0/398718293 >> 172.21.15.49:0/398718293 conn(0x55d47b700a40 msgr2=0x55d47b6ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:11.991 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdb8e7fc640 0 client.17340 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:11.991 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 -- 172.21.15.49:0/398718293 shutdown_connections 2024-03-18T00:58:11.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc04081b740 1 -- 172.21.15.49:0/3755519756 shutdown_connections 2024-03-18T00:58:11.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc04081b740 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56235e923140 0x56235e9255e0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc04081b740 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e912cd0 0x56235e91eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc04081b740 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e90db60 0x56235e917b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc04081b740 1 --2- 172.21.15.49:0/3755519756 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e9175e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc04081b740 1 -- 172.21.15.49:0/3755519756 >> 172.21.15.49:0/3755519756 conn(0x56235e9086e0 msgr2=0x56235e907900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:11.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fc016ffd640 0 client.26596 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:11.992 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.989+0000 7fdbb01e2740 1 -- 172.21.15.49:0/398718293 wait complete. 2024-03-18T00:58:11.992 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.990+0000 7fdbb01e2740 20 client.17340 trim_cache size 0 max 16384 2024-03-18T00:58:11.993 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:58:11.993 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:58:11.993 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:58:11.993 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:58:11.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.990+0000 7fc04081b740 1 -- 172.21.15.49:0/3755519756 shutdown_connections 2024-03-18T00:58:11.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.990+0000 7fc04081b740 1 -- 172.21.15.49:0/3755519756 wait complete. 2024-03-18T00:58:11.994 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 -- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdb940a27f0 msgr2=0x7fdb940a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:11.994 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdb940a27f0 0x7fdb940a4cb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x55d47b706720 tx=0x7fdba002bef0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fc04081b740 20 client.26596 trim_cache size 0 max 16384 2024-03-18T00:58:11.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 -- 172.21.15.49:0/978656526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8101760 msgr2=0x7fdba8196d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:11.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8101760 0x7fdba8196d60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fdb90005fd0 tx=0x7fdb9000ee50 comp rx=0 tx=0).stop 2024-03-18T00:58:11.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 -- 172.21.15.49:0/978656526 shutdown_connections 2024-03-18T00:58:11.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fdb940a27f0 0x7fdb940a4cb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdba8102120 0x7fdba819ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdba8101760 0x7fdba8196d60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 --2- 172.21.15.49:0/978656526 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdba8100570 0x7fdba8196820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:11.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 -- 172.21.15.49:0/978656526 >> 172.21.15.49:0/978656526 conn(0x7fdba80fbae0 msgr2=0x7fdba8104a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:11.995 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 -- 172.21.15.49:0/978656526 shutdown_connections 2024-03-18T00:58:11.996 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:11.991+0000 7fdba7fff640 1 -- 172.21.15.49:0/978656526 wait complete. 2024-03-18T00:58:12.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc04081b740 1 Processor -- start 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc04081b740 1 -- start start 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc04081b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90d170 0x56235e90e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc04081b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90db60 0x56235e90c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc04081b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e9168f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc04081b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc0381aaff0 con 0x56235e912cd0 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc04081b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc0381acb30 con 0x56235e90d170 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc04081b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc0381ab6c0 con 0x56235e90db60 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc037fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e9168f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc037fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e9168f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47308/0 (socket says 172.21.15.49:47308) 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc037fff640 1 -- 172.21.15.49:0/2130738655 learned_addr learned my addr 172.21.15.49:0/2130738655 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc036ffd640 1 --2- 172.21.15.49:0/2130738655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90db60 0x56235e90c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc0377fe640 1 --2- 172.21.15.49:0/2130738655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90d170 0x56235e90e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc037fff640 1 -- 172.21.15.49:0/2130738655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90db60 msgr2=0x56235e90c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc037fff640 1 --2- 172.21.15.49:0/2130738655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90db60 0x56235e90c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc037fff640 1 -- 172.21.15.49:0/2130738655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90d170 msgr2=0x56235e90e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc037fff640 1 --2- 172.21.15.49:0/2130738655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90d170 0x56235e90e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.012+0000 7fc037fff640 1 -- 172.21.15.49:0/2130738655 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56235e5460f0 con 0x56235e912cd0 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.013+0000 7fc037fff640 1 --2- 172.21.15.49:0/2130738655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e9168f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc030009ca0 tx=0x7fc030041000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fb2d59b0d09de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.013+0000 7fc01d7fa640 1 -- 172.21.15.49:0/2130738655 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030039040 con 0x56235e912cd0 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.013+0000 7fc01d7fa640 1 -- 172.21.15.49:0/2130738655 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc030053070 con 0x56235e912cd0 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.013+0000 7fc01d7fa640 1 -- 172.21.15.49:0/2130738655 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0300416d0 con 0x56235e912cd0 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 -- 172.21.15.49:0/2130738655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 msgr2=0x56235e9168f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 --2- 172.21.15.49:0/2130738655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e9168f0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fc030009ca0 tx=0x7fc030041000 comp rx=0 tx=0).stop 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 -- 172.21.15.49:0/2130738655 shutdown_connections 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 --2- 172.21.15.49:0/2130738655 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e9168f0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 --2- 172.21.15.49:0/2130738655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90db60 0x56235e90c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 --2- 172.21.15.49:0/2130738655 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90d170 0x56235e90e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 -- 172.21.15.49:0/2130738655 >> 172.21.15.49:0/2130738655 conn(0x56235e9086e0 msgr2=0x56235e90b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 -- 172.21.15.49:0/2130738655 shutdown_connections 2024-03-18T00:58:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 -- 172.21.15.49:0/2130738655 wait complete. 2024-03-18T00:58:12.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.014+0000 7fc04081b740 1 -- start start 2024-03-18T00:58:12.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc04081b740 1 Processor -- start 2024-03-18T00:58:12.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc04081b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e91e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc04081b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e91b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc04081b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e91b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc04081b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc0381ac460 con 0x56235e912cd0 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc04081b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc0381abd90 con 0x56235e90db60 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc04081b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc0381a8fd0 con 0x56235e90d170 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc04081b740 10 client.0 _ll_register_callbacks cb 0x7fc028009cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc037fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e91b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc0377fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e91e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc036ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e91b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc037fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e91b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47322/0 (socket says 172.21.15.49:47322) 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc037fff640 1 -- 172.21.15.49:0/886760008 learned_addr learned my addr 172.21.15.49:0/886760008 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc0377fe640 1 -- 172.21.15.49:0/886760008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 msgr2=0x56235e91b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc0377fe640 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e91b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc0377fe640 1 -- 172.21.15.49:0/886760008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 msgr2=0x56235e91b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc0377fe640 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e91b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc0377fe640 1 -- 172.21.15.49:0/886760008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56235e8399d0 con 0x56235e90d170 2024-03-18T00:58:12.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc036ffd640 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e91b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.015+0000 7fc037fff640 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e91b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.016+0000 7fc0377fe640 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e91e730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc028010a60 tx=0x7fc028009c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=153fe8bf58939d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.016+0000 7fc016ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.016+0000 7fc016ffd640 1 -- 172.21.15.49:0/886760008 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc028035060 con 0x56235e90d170 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.016+0000 7fc016ffd640 1 -- 172.21.15.49:0/886760008 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc0280050a0 con 0x56235e90d170 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.016+0000 7fc04081b740 1 -- 172.21.15.49:0/886760008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc030053070 con 0x56235e90d170 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.016+0000 7fc016ffd640 1 -- 172.21.15.49:0/886760008 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc02802c040 con 0x56235e90d170 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.016+0000 7fc04081b740 10 client.26602 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.016+0000 7fc04081b740 1 -- 172.21.15.49:0/886760008 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56235e9885a0 con 0x56235e90d170 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.016+0000 7fc016ffd640 1 -- 172.21.15.49:0/886760008 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc030053070 con 0x56235e90d170 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc016ffd640 1 -- 172.21.15.49:0/886760008 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc028003ad0 con 0x56235e90d170 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc04081b740 10 client.26602 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc04081b740 10 client.26602 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc04081b740 20 client.26602 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:12.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc04081b740 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56235e988f80 0x56235e98b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc04081b740 4 client.26602 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc04081b740 1 -- 172.21.15.49:0/886760008 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56235e7f7540 con 0x56235e988f80 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc036ffd640 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56235e988f80 0x56235e98b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc036ffd640 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56235e988f80 0x56235e98b3a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f62175c644f46d7c server_cookie=d5c2a77565f8f438 in_seq=0 out_seq=0 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.017+0000 7fc016ffd640 10 client.26602 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc016ffd640 1 -- 172.21.15.49:0/886760008 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56235e7f7540 con 0x56235e988f80 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc016ffd640 10 client.26602 handle_command_reply: tid=0 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 client.26602 shutdown 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 -- 172.21.15.49:0/886760008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 msgr2=0x56235e91e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e91e730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc028010a60 tx=0x7fc028009c80 comp rx=0 tx=0).stop 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 -- 172.21.15.49:0/886760008 shutdown_connections 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x56235e988f80 0x56235e98b3a0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56235e90db60 0x56235e91b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56235e912cd0 0x56235e91b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc016ffd640 0 client.26602 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 --2- 172.21.15.49:0/886760008 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56235e90d170 0x56235e91e730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 -- 172.21.15.49:0/886760008 >> 172.21.15.49:0/886760008 conn(0x56235e9086e0 msgr2=0x56235e90b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 -- 172.21.15.49:0/886760008 shutdown_connections 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 1 -- 172.21.15.49:0/886760008 wait complete. 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.019+0000 7fc04081b740 20 client.26602 trim_cache size 0 max 16384 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:58:12.023 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:58:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 -- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0200a2800 msgr2=0x7fc0200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0200a2800 0x7fc0200a4cc0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc030005e30 tx=0x7fc03002e040 comp rx=0 tx=0).stop 2024-03-18T00:58:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 -- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0381018d0 msgr2=0x7fc038102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0381018d0 0x7fc038102cb0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc02c00e9f0 tx=0x7fc02c00eeb0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 -- 172.21.15.49:0/3809605600 shutdown_connections 2024-03-18T00:58:12.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc0200a2800 0x7fc0200a4cc0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc038102290 0x7fc0381031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc0381018d0 0x7fc038102cb0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 --2- 172.21.15.49:0/3809605600 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0381006e0 0x7fc038104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 -- 172.21.15.49:0/3809605600 >> 172.21.15.49:0/3809605600 conn(0x7fc0380fbc70 msgr2=0x7fc0380fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 -- 172.21.15.49:0/3809605600 shutdown_connections 2024-03-18T00:58:12.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.020+0000 7fc03dc34640 1 -- 172.21.15.49:0/3809605600 wait complete. 2024-03-18T00:58:12.034 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:58:12.034 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:58:12.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.151+0000 7fa59fd53640 1 -- 172.21.15.49:0/1075982722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598107560 msgr2=0x7fa5981099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.151+0000 7fa59fd53640 1 --2- 172.21.15.49:0/1075982722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598107560 0x7fa5981099b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa590009ca0 tx=0x7fa59002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 -- 172.21.15.49:0/1075982722 shutdown_connections 2024-03-18T00:58:12.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 --2- 172.21.15.49:0/1075982722 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598107560 0x7fa5981099b0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 --2- 172.21.15.49:0/1075982722 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa598104c30 0x7fa598107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 --2- 172.21.15.49:0/1075982722 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa598102300 0x7fa5981046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 -- 172.21.15.49:0/1075982722 >> 172.21.15.49:0/1075982722 conn(0x7fa5980fbd10 msgr2=0x7fa5980fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 -- 172.21.15.49:0/1075982722 shutdown_connections 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 -- 172.21.15.49:0/1075982722 wait complete. 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 Processor -- start 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 -- start start 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598102300 0x7fa598197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa598104c30 0x7fa598198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa598107560 0x7fa59819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa59810d4b0 con 0x7fa598102300 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa59810d330 con 0x7fa598107560 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.152+0000 7fa59fd53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa59810d630 con 0x7fa598104c30 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59daef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598102300 0x7fa598197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59daef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598102300 0x7fa598197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47338/0 (socket says 172.21.15.49:47338) 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59daef640 1 -- 172.21.15.49:0/669279417 learned_addr learned my addr 172.21.15.49:0/669279417 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59e2f0640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa598107560 0x7fa59819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59d2ee640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa598104c30 0x7fa598198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59daef640 1 -- 172.21.15.49:0/669279417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa598104c30 msgr2=0x7fa598198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59daef640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa598104c30 0x7fa598198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59daef640 1 -- 172.21.15.49:0/669279417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa598107560 msgr2=0x7fa59819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59daef640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa598107560 0x7fa59819f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59daef640 1 -- 172.21.15.49:0/669279417 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59819f700 con 0x7fa598102300 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59daef640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598102300 0x7fa598197b50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa59400e9f0 tx=0x7fa59400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa58effd640 1 -- 172.21.15.49:0/669279417 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59400cd80 con 0x7fa598102300 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa59819f9f0 con 0x7fa598102300 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa58effd640 1 -- 172.21.15.49:0/669279417 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa594010910 con 0x7fa598102300 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa58effd640 1 -- 172.21.15.49:0/669279417 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5940188f0 con 0x7fa598102300 2024-03-18T00:58:12.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.153+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa59819ff90 con 0x7fa598102300 2024-03-18T00:58:12.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.154+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5980fd800 con 0x7fa598102300 2024-03-18T00:58:12.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.155+0000 7fa58effd640 1 -- 172.21.15.49:0/669279417 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa594010ac0 con 0x7fa598102300 2024-03-18T00:58:12.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.156+0000 7fa58effd640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa5840a2800 0x7fa5840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.156+0000 7fa59d2ee640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa5840a2800 0x7fa5840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.156+0000 7fa58effd640 1 -- 172.21.15.49:0/669279417 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa5940cf1f0 con 0x7fa598102300 2024-03-18T00:58:12.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.157+0000 7fa59d2ee640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa5840a2800 0x7fa5840a4cc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa598198ff0 tx=0x7fa588005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.160+0000 7fa58effd640 1 -- 172.21.15.49:0/669279417 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa59409bc20 con 0x7fa598102300 2024-03-18T00:58:12.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.383+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa59802d420 con 0x7fa598102300 2024-03-18T00:58:12.384 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.384+0000 7fa58effd640 1 -- 172.21.15.49:0/669279417 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11934 (secure 0 0 0) 0x7fa594086090 con 0x7fa598102300 2024-03-18T00:58:12.384 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:58:12.384 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:57:06.597245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":138,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24499,24506,24481,24470,24463]},"id":1}]} 2024-03-18T00:58:12.384 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 28 2024-03-18T00:58:12.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa5840a2800 msgr2=0x7fa5840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa5840a2800 0x7fa5840a4cc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa598198ff0 tx=0x7fa588005f90 comp rx=0 tx=0).stop 2024-03-18T00:58:12.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598102300 msgr2=0x7fa598197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598102300 0x7fa598197b50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa59400e9f0 tx=0x7fa59400eeb0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 shutdown_connections 2024-03-18T00:58:12.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa5840a2800 0x7fa5840a4cc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa598107560 0x7fa59819f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa598104c30 0x7fa598198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.386 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 --2- 172.21.15.49:0/669279417 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa598102300 0x7fa598197b50 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 >> 172.21.15.49:0/669279417 conn(0x7fa5980fbd10 msgr2=0x7fa598102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 shutdown_connections 2024-03-18T00:58:12.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.386+0000 7fa59fd53640 1 -- 172.21.15.49:0/669279417 wait complete. 2024-03-18T00:58:12.397 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:58:12.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 -- 172.21.15.49:0/2626084942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 msgr2=0x7fb830102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 --2- 172.21.15.49:0/2626084942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 0x7fb830102e70 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb824009900 tx=0x7fb82402bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:12.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 -- 172.21.15.49:0/2626084942 shutdown_connections 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 --2- 172.21.15.49:0/2626084942 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8301033b0 0x7fb830109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 --2- 172.21.15.49:0/2626084942 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 0x7fb830102e70 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 --2- 172.21.15.49:0/2626084942 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb830101800 0x7fb830101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 -- 172.21.15.49:0/2626084942 >> 172.21.15.49:0/2626084942 conn(0x7fb8300fcd70 msgr2=0x7fb8300ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 -- 172.21.15.49:0/2626084942 shutdown_connections 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 -- 172.21.15.49:0/2626084942 wait complete. 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 Processor -- start 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.511+0000 7fb836716640 1 -- start start 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb836716640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb830101800 0x7fb830197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb836716640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 0x7fb830198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb836716640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8301033b0 0x7fb83019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb836716640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb83010d480 con 0x7fb8301029f0 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb836716640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb83010d300 con 0x7fb8301033b0 2024-03-18T00:58:12.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb836716640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb83010d600 con 0x7fb830101800 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 0x7fb830198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 0x7fb830198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47368/0 (socket says 172.21.15.49:47368) 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82f7fe640 1 -- 172.21.15.49:0/810364375 learned_addr learned my addr 172.21.15.49:0/810364375 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82ffff640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb830101800 0x7fb830197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb834cb3640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8301033b0 0x7fb83019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82f7fe640 1 -- 172.21.15.49:0/810364375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb830101800 msgr2=0x7fb830197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb830101800 0x7fb830197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82f7fe640 1 -- 172.21.15.49:0/810364375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8301033b0 msgr2=0x7fb83019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8301033b0 0x7fb83019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82f7fe640 1 -- 172.21.15.49:0/810364375 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb83019f6f0 con 0x7fb8301029f0 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 0x7fb830198010 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb824009230 tx=0x7fb82402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82d7fa640 1 -- 172.21.15.49:0/810364375 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824041070 con 0x7fb8301029f0 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82d7fa640 1 -- 172.21.15.49:0/810364375 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb824004450 con 0x7fb8301029f0 2024-03-18T00:58:12.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb836716640 1 -- 172.21.15.49:0/810364375 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb83019f980 con 0x7fb8301029f0 2024-03-18T00:58:12.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb82d7fa640 1 -- 172.21.15.49:0/810364375 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb824003e20 con 0x7fb8301029f0 2024-03-18T00:58:12.515 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.512+0000 7fb836716640 1 -- 172.21.15.49:0/810364375 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb83019fe70 con 0x7fb8301029f0 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.515+0000 7fb82d7fa640 1 -- 172.21.15.49:0/810364375 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb824044070 con 0x7fb8301029f0 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.515+0000 7fb8393d6740 1 Processor -- start 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.515+0000 7fb8393d6740 1 -- start start 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.515+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df24850 0x56449df24c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 0x56449df29d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df2a3b0 0x56449df2c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb8393d6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56449de89490 con 0x56449df2a3b0 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb8393d6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56449de31c90 con 0x56449df25240 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb8393d6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56449de85530 con 0x56449df24850 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb82d7fa640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8180a2800 0x7fb8180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb82f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df2a3b0 0x56449df2c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df24850 0x56449df24c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df24850 0x56449df24c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:47362/0 (socket says 172.21.15.49:47362) 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 -- 172.21.15.49:0/3699205429 learned_addr learned my addr 172.21.15.49:0/3699205429 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb82ffff640 1 --2- 172.21.15.49:0/3699205429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 0x56449df29d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8180a2800 0x7fb8180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 -- 172.21.15.49:0/3699205429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 msgr2=0x56449df29d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 --2- 172.21.15.49:0/3699205429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 0x56449df29d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 -- 172.21.15.49:0/3699205429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df2a3b0 msgr2=0x56449df2c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 --2- 172.21.15.49:0/3699205429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df2a3b0 0x56449df2c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb82d7fa640 1 -- 172.21.15.49:0/810364375 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb8240f0620 con 0x7fb8301029f0 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 -- 172.21.15.49:0/3699205429 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56449deec500 con 0x56449df24850 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 --2- 172.21.15.49:0/3699205429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df24850 0x56449df24c70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb828005c40 tx=0x7fb82800f3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5571d35af210078c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.516+0000 7fb834cb3640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8180a2800 0x7fb8180a4cc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb828008000 tx=0x7fb82801f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8167fc640 1 -- 172.21.15.49:0/3699205429 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82800fd30 con 0x56449df24850 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8167fc640 1 -- 172.21.15.49:0/3699205429 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb828022030 con 0x56449df24850 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8167fc640 1 -- 172.21.15.49:0/3699205429 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb828010910 con 0x56449df24850 2024-03-18T00:58:12.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8393d6740 1 -- 172.21.15.49:0/3699205429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df24850 msgr2=0x56449df24c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3699205429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df24850 0x56449df24c70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb828005c40 tx=0x7fb82800f3c0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8393d6740 1 -- 172.21.15.49:0/3699205429 shutdown_connections 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3699205429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 0x56449df29d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3699205429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df2a3b0 0x56449df2c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3699205429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df24850 0x56449df24c70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8393d6740 1 -- 172.21.15.49:0/3699205429 >> 172.21.15.49:0/3699205429 conn(0x56449df1fdc0 msgr2=0x56449df201c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8393d6740 1 -- 172.21.15.49:0/3699205429 shutdown_connections 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.517+0000 7fb8393d6740 1 -- 172.21.15.49:0/3699205429 wait complete. 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb8393d6740 1 -- start start 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb8393d6740 1 Processor -- start 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df24850 0x56449df2edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df25240 0x56449df2f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 0x56449df36370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb8393d6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56449df2d300 con 0x56449df25240 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb8393d6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x56449de45230 con 0x56449df24850 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb8393d6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56449df2d480 con 0x56449df2a3b0 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb8393d6740 10 client.0 _ll_register_callbacks cb 0x56449de81010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb82f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df25240 0x56449df2f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb82f7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df25240 0x56449df2f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47382/0 (socket says 172.21.15.49:47382) 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb834cb3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 0x56449df36370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df24850 0x56449df2edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.518+0000 7fb82f7fe640 1 -- 172.21.15.49:0/3674519078 learned_addr learned my addr 172.21.15.49:0/3674519078 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb82f7fe640 1 -- 172.21.15.49:0/3674519078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 msgr2=0x56449df36370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 0x56449df36370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb82f7fe640 1 -- 172.21.15.49:0/3674519078 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df24850 msgr2=0x56449df2edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df24850 0x56449df2edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb82f7fe640 1 -- 172.21.15.49:0/3674519078 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56449de54100 con 0x56449df25240 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df25240 0x56449df2f330 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb82402c7b0 tx=0x7fb82402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936b15fde937b82e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb80f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb80f7fe640 1 -- 172.21.15.49:0/3674519078 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82404b040 con 0x56449df25240 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb8393d6740 1 -- 172.21.15.49:0/3674519078 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb828022030 con 0x56449df25240 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb80f7fe640 1 -- 172.21.15.49:0/3674519078 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb8240c5290 con 0x56449df25240 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb80f7fe640 1 -- 172.21.15.49:0/3674519078 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb8240c10d0 con 0x56449df25240 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb8393d6740 10 client.17376 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb8393d6740 1 -- 172.21.15.49:0/3674519078 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56449db9d3f0 con 0x56449df25240 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb80f7fe640 1 -- 172.21.15.49:0/3674519078 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb828022030 con 0x56449df25240 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.519+0000 7fb80f7fe640 1 -- 172.21.15.49:0/3674519078 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb8240bc610 con 0x56449df25240 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.520+0000 7fb8393d6740 10 client.17376 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.520+0000 7fb8393d6740 10 client.17376 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.520+0000 7fb8393d6740 20 client.17376 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.520+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56449df3ab30 0x56449df3cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.524 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.520+0000 7fb8393d6740 4 client.17376 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:12.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.520+0000 7fb8393d6740 1 -- 172.21.15.49:0/3674519078 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56449de969d0 con 0x56449df3ab30 2024-03-18T00:58:12.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.520+0000 7fb82ffff640 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56449df3ab30 0x56449df3cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.520+0000 7fb82ffff640 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56449df3ab30 0x56449df3cfb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=43bcc276ac1425d5 server_cookie=3501800616f2cd67 in_seq=0 out_seq=0 2024-03-18T00:58:12.525 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.520+0000 7fb80f7fe640 10 client.17376 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:58:12.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.521+0000 7fb80f7fe640 1 -- 172.21.15.49:0/3674519078 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56449de969d0 con 0x56449df3ab30 2024-03-18T00:58:12.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.521+0000 7fb80f7fe640 10 client.17376 handle_command_reply: tid=0 2024-03-18T00:58:12.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.521+0000 7fb8393d6740 1 client.17376 shutdown 2024-03-18T00:58:12.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.521+0000 7fb8393d6740 1 -- 172.21.15.49:0/3674519078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df25240 msgr2=0x56449df2f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df25240 0x56449df2f330 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb82402c7b0 tx=0x7fb82402bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:12.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 1 -- 172.21.15.49:0/3674519078 shutdown_connections 2024-03-18T00:58:12.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56449df3ab30 0x56449df3cfb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 0x56449df36370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df25240 0x56449df2f330 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 1 --2- 172.21.15.49:0/3674519078 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df24850 0x56449df2edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 1 -- 172.21.15.49:0/3674519078 >> 172.21.15.49:0/3674519078 conn(0x56449df1fdc0 msgr2=0x56449df1efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb80f7fe640 0 client.17376 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:58:12.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 1 -- 172.21.15.49:0/3674519078 shutdown_connections 2024-03-18T00:58:12.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 1 -- 172.21.15.49:0/3674519078 wait complete. 2024-03-18T00:58:12.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.522+0000 7fb8393d6740 20 client.17376 trim_cache size 0 max 16384 2024-03-18T00:58:12.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb8393d6740 1 Processor -- start 2024-03-18T00:58:12.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb8393d6740 1 -- start start 2024-03-18T00:58:12.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df2d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 0x56449df36330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 0x56449df267d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb8393d6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301a8d10 con 0x56449df24850 2024-03-18T00:58:12.537 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb8393d6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301aa850 con 0x56449df25240 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb8393d6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301a93e0 con 0x56449df2a3b0 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df2d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df2d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47394/0 (socket says 172.21.15.49:47394) 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82ffff640 1 -- 172.21.15.49:0/1095016093 learned_addr learned my addr 172.21.15.49:0/1095016093 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 0x56449df36330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb834cb3640 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 0x56449df267d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82ffff640 1 -- 172.21.15.49:0/1095016093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 msgr2=0x56449df267d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82ffff640 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 0x56449df267d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82ffff640 1 -- 172.21.15.49:0/1095016093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 msgr2=0x56449df36330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82ffff640 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 0x56449df36330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82ffff640 1 -- 172.21.15.49:0/1095016093 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56449dc677d0 con 0x56449df24850 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb834cb3640 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 0x56449df267d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:58:12.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb82ffff640 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df2d8b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb820009dd0 tx=0x7fb820005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad2a925ad75d1c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb815ffb640 1 -- 172.21.15.49:0/1095016093 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820024070 con 0x56449df24850 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.536+0000 7fb815ffb640 1 -- 172.21.15.49:0/1095016093 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8200047a0 con 0x56449df24850 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb815ffb640 1 -- 172.21.15.49:0/1095016093 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8200204f0 con 0x56449df24850 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 -- 172.21.15.49:0/1095016093 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 msgr2=0x56449df2d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df2d8b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb820009dd0 tx=0x7fb820005f90 comp rx=0 tx=0).stop 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 -- 172.21.15.49:0/1095016093 shutdown_connections 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df25240 0x56449df36330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df2a3b0 0x56449df267d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 --2- 172.21.15.49:0/1095016093 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df2d8b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 -- 172.21.15.49:0/1095016093 >> 172.21.15.49:0/1095016093 conn(0x56449df1fdc0 msgr2=0x56449df25ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 -- 172.21.15.49:0/1095016093 shutdown_connections 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 -- 172.21.15.49:0/1095016093 wait complete. 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.537+0000 7fb8393d6740 1 -- start start 2024-03-18T00:58:12.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb8393d6740 1 Processor -- start 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df330e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df25240 0x56449df33620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb8393d6740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df2a3b0 0x56449df9eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb8393d6740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301aa180 con 0x56449df24850 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb8393d6740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301a9ab0 con 0x56449df2a3b0 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb8393d6740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb8301a6cf0 con 0x56449df25240 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb8393d6740 10 client.0 _ll_register_callbacks cb 0x7fb824005770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb82ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df330e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47404/0 (socket says 172.21.15.49:47404) 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb82ffff640 1 -- 172.21.15.49:0/2537331310 learned_addr learned my addr 172.21.15.49:0/2537331310 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df25240 0x56449df33620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.538+0000 7fb834cb3640 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df2a3b0 0x56449df9eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb82ffff640 1 -- 172.21.15.49:0/2537331310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df25240 msgr2=0x56449df33620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb82ffff640 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df25240 0x56449df33620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb82ffff640 1 -- 172.21.15.49:0/2537331310 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df2a3b0 msgr2=0x56449df9eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb82ffff640 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df2a3b0 0x56449df9eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb82ffff640 1 -- 172.21.15.49:0/2537331310 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56449df33cc0 con 0x56449df24850 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df25240 0x56449df33620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb82ffff640 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df330e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb820009c60 tx=0x7fb820004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f01ba4318d22c89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb80f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb80f7fe640 1 -- 172.21.15.49:0/2537331310 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb82002d070 con 0x56449df24850 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb80f7fe640 1 -- 172.21.15.49:0/2537331310 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb820020470 con 0x56449df24850 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb8393d6740 1 -- 172.21.15.49:0/2537331310 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8200047a0 con 0x56449df24850 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb80f7fe640 1 -- 172.21.15.49:0/2537331310 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb820039050 con 0x56449df24850 2024-03-18T00:58:12.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb8393d6740 10 client.17382 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb8393d6740 1 -- 172.21.15.49:0/2537331310 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56449dfa05c0 con 0x56449df24850 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb80f7fe640 1 -- 172.21.15.49:0/2537331310 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8200047a0 con 0x56449df24850 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb80f7fe640 1 -- 172.21.15.49:0/2537331310 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fb82001e5f0 con 0x56449df24850 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb8393d6740 10 client.17382 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.539+0000 7fb8393d6740 10 client.17382 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 20 client.17382 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56449dfa0f70 0x56449dfa3390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 4 client.17382 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 -- 172.21.15.49:0/2537331310 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56449de90d20 con 0x56449dfa0f70 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56449dfa0f70 0x56449dfa3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb80f7fe640 10 client.17382 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb82f7fe640 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56449dfa0f70 0x56449dfa3390 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86a18f641715f48c server_cookie=ea8c813e32c3d8a5 in_seq=0 out_seq=0 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb80f7fe640 1 -- 172.21.15.49:0/2537331310 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56449de90d20 con 0x56449dfa0f70 2024-03-18T00:58:12.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb80f7fe640 10 client.17382 handle_command_reply: tid=0 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 client.17382 shutdown 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 -- 172.21.15.49:0/2537331310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 msgr2=0x56449df330e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df330e0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fb820009c60 tx=0x7fb820004950 comp rx=0 tx=0).stop 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 -- 172.21.15.49:0/2537331310 shutdown_connections 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x56449dfa0f70 0x56449dfa3390 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x56449df2a3b0 0x56449df9eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb80f7fe640 0 client.17382 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x56449df25240 0x56449df33620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 --2- 172.21.15.49:0/2537331310 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x56449df24850 0x56449df330e0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.540+0000 7fb8393d6740 1 -- 172.21.15.49:0/2537331310 >> 172.21.15.49:0/2537331310 conn(0x56449df1fdc0 msgr2=0x56449df25c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb8393d6740 1 -- 172.21.15.49:0/2537331310 shutdown_connections 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb8393d6740 1 -- 172.21.15.49:0/2537331310 wait complete. 2024-03-18T00:58:12.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb8393d6740 20 client.17382 trim_cache size 0 max 16384 2024-03-18T00:58:12.547 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:12.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 -- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8180a2800 msgr2=0x7fb8180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8180a2800 0x7fb8180a4cc0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb828008000 tx=0x7fb82801f000 comp rx=0 tx=0).stop 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 -- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 msgr2=0x7fb830198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 0x7fb830198010 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fb824009230 tx=0x7fb82402c650 comp rx=0 tx=0).stop 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 -- 172.21.15.49:0/810364375 shutdown_connections 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb8180a2800 0x7fb8180a4cc0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb8301033b0 0x7fb83019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb8301029f0 0x7fb830198010 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 --2- 172.21.15.49:0/810364375 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb830101800 0x7fb830197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 -- 172.21.15.49:0/810364375 >> 172.21.15.49:0/810364375 conn(0x7fb8300fcd70 msgr2=0x7fb830105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 -- 172.21.15.49:0/810364375 shutdown_connections 2024-03-18T00:58:12.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.541+0000 7fb836716640 1 -- 172.21.15.49:0/810364375 wait complete. 2024-03-18T00:58:12.552 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:58:12.665 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.665+0000 7f66a7d0a640 1 -- 172.21.15.49:0/343528355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 msgr2=0x7f66a0102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.665+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/343528355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 0x7f66a0102e80 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6694009900 tx=0x7f669402bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.665+0000 7f66a7d0a640 1 -- 172.21.15.49:0/343528355 shutdown_connections 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.665+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/343528355 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66a01033c0 0x7f66a0109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.665+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/343528355 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 0x7f66a0102e80 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.665+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/343528355 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66a0101810 0x7f66a0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.665+0000 7f66a7d0a640 1 -- 172.21.15.49:0/343528355 >> 172.21.15.49:0/343528355 conn(0x7f66a00fcd80 msgr2=0x7f66a00ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 -- 172.21.15.49:0/343528355 shutdown_connections 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 -- 172.21.15.49:0/343528355 wait complete. 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 Processor -- start 2024-03-18T00:58:12.666 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 -- start start 2024-03-18T00:58:12.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66a0101810 0x7f66a0075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 0x7f66a0073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66a01033c0 0x7f66a0074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f66a010d470 con 0x7f66a0102a00 2024-03-18T00:58:12.668 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f66a010d2f0 con 0x7f66a0101810 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.666+0000 7f66a7d0a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f66a010d5f0 con 0x7f66a01033c0 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a5aa6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66a0101810 0x7f66a0075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a52a5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 0x7f66a0073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a5aa6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66a0101810 0x7f66a0075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:39556/0 (socket says 172.21.15.49:39556) 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a5aa6640 1 -- 172.21.15.49:0/3451649257 learned_addr learned my addr 172.21.15.49:0/3451649257 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a52a5640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 0x7f66a0073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47424/0 (socket says 172.21.15.49:47424) 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a62a7640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66a01033c0 0x7f66a0074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a5aa6640 1 -- 172.21.15.49:0/3451649257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66a01033c0 msgr2=0x7f66a0074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66a01033c0 0x7f66a0074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a5aa6640 1 -- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 msgr2=0x7f66a0073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 0x7f66a0073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a5aa6640 1 -- 172.21.15.49:0/3451649257 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a00749a0 con 0x7f66a0101810 2024-03-18T00:58:12.669 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a62a7640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66a01033c0 0x7f66a0074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a52a5640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 0x7f66a0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66a0101810 0x7f66a0075590 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f669800e9c0 tx=0x7f669800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f669effd640 1 -- 172.21.15.49:0/3451649257 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669800cd80 con 0x7f66a0101810 2024-03-18T00:58:12.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f669effd640 1 -- 172.21.15.49:0/3451649257 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66980035f0 con 0x7f66a0101810 2024-03-18T00:58:12.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a7d0a640 1 -- 172.21.15.49:0/3451649257 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a01a1c00 con 0x7f66a0101810 2024-03-18T00:58:12.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f669effd640 1 -- 172.21.15.49:0/3451649257 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698010690 con 0x7f66a0101810 2024-03-18T00:58:12.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.667+0000 7f66a7d0a640 1 -- 172.21.15.49:0/3451649257 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a01a2120 con 0x7f66a0101810 2024-03-18T00:58:12.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66aa9cd740 1 Processor -- start 2024-03-18T00:58:12.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66aa9cd740 1 -- start start 2024-03-18T00:58:12.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f669effd640 1 -- 172.21.15.49:0/3451649257 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f66980037a0 con 0x7f66a0101810 2024-03-18T00:58:12.676 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b0999fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a0240 0x561b099a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a53b0 0x561b099a7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x561b09904490 con 0x561b099a53b0 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x561b098acc90 con 0x561b0999f850 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561b09900530 con 0x561b099a0240 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66a62a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b0999fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66a5aa6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a0240 0x561b099a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66a52a5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a53b0 0x561b099a7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66a62a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b0999fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:39564/0 (socket says 172.21.15.49:39564) 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.670+0000 7f66a62a7640 1 -- 172.21.15.49:0/188172158 learned_addr learned my addr 172.21.15.49:0/188172158 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f66a5aa6640 1 -- 172.21.15.49:0/188172158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 msgr2=0x561b0999fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/188172158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b0999fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f66a5aa6640 1 -- 172.21.15.49:0/188172158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a53b0 msgr2=0x561b099a7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/188172158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a53b0 0x561b099a7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.677 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f66a5aa6640 1 -- 172.21.15.49:0/188172158 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b09967500 con 0x561b099a0240 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f66a62a7640 1 --2- 172.21.15.49:0/188172158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b0999fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f66a52a5640 1 --2- 172.21.15.49:0/188172158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a53b0 0x561b099a7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/188172158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a0240 0x561b099a4d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f66980cc920 tx=0x7f66980d74b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ffc440f55bffd71 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f669effd640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f66880a2800 0x7f66880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f66a62a7640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f66880a2800 0x7f66880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f6687fff640 1 -- 172.21.15.49:0/188172158 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66980e2070 con 0x561b099a0240 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f669effd640 1 -- 172.21.15.49:0/3451649257 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f66980cee70 con 0x7f66a0101810 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f6687fff640 1 -- 172.21.15.49:0/188172158 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f669809bef0 con 0x561b099a0240 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.671+0000 7f6687fff640 1 -- 172.21.15.49:0/188172158 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669809b150 con 0x561b099a0240 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66a62a7640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f66880a2800 0x7f66880a4cc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x561b099a0aa0 tx=0x7f6690009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 -- 172.21.15.49:0/188172158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a0240 msgr2=0x561b099a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/188172158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a0240 0x561b099a4d40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f66980cc920 tx=0x7f66980d74b0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 -- 172.21.15.49:0/188172158 shutdown_connections 2024-03-18T00:58:12.678 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/188172158 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a53b0 0x561b099a7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/188172158 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a0240 0x561b099a4d40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/188172158 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b0999fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 -- 172.21.15.49:0/188172158 >> 172.21.15.49:0/188172158 conn(0x561b0999adc0 msgr2=0x561b0999b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 -- 172.21.15.49:0/188172158 shutdown_connections 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 -- 172.21.15.49:0/188172158 wait complete. 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.672+0000 7f66aa9cd740 1 -- start start 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66aa9cd740 1 Processor -- start 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b099a9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099aa2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 0x561b099b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x561b099a8360 con 0x561b099a0240 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x561b098c0230 con 0x561b0999f850 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x561b099a84e0 con 0x561b099a53b0 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66aa9cd740 10 client.0 _ll_register_callbacks cb 0x561b098fc010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:12.679 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a52a5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099aa2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a5aa6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b099a9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a52a5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099aa2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47442/0 (socket says 172.21.15.49:47442) 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a62a7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 0x561b099b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a5aa6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b099a9d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:39578/0 (socket says 172.21.15.49:39578) 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a52a5640 1 -- 172.21.15.49:0/1159840388 learned_addr learned my addr 172.21.15.49:0/1159840388 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a52a5640 1 -- 172.21.15.49:0/1159840388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 msgr2=0x561b099b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a52a5640 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 0x561b099b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a52a5640 1 -- 172.21.15.49:0/1159840388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 msgr2=0x561b099a9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a52a5640 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b099a9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a52a5640 1 -- 172.21.15.49:0/1159840388 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b098cf100 con 0x561b099a0240 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.673+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b099a9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66a52a5640 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099aa2c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6694009900 tx=0x7f6694005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794abaa6cd1f2def server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66857fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:12.680 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66857fa640 1 -- 172.21.15.49:0/1159840388 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6694005d50 con 0x561b099a0240 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66857fa640 1 -- 172.21.15.49:0/1159840388 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f66940046c0 con 0x561b099a0240 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1159840388 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f669809bef0 con 0x561b099a0240 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66857fa640 1 -- 172.21.15.49:0/1159840388 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f669404c070 con 0x561b099a0240 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66aa9cd740 10 client.17394 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1159840388 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561b096183f0 con 0x561b099a0240 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66857fa640 1 -- 172.21.15.49:0/1159840388 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f669809bef0 con 0x561b099a0240 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66857fa640 1 -- 172.21.15.49:0/1159840388 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f66940039f0 con 0x561b099a0240 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66aa9cd740 10 client.17394 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66aa9cd740 10 client.17394 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66aa9cd740 20 client.17394 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x561b099b5ba0 0x561b099b8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66aa9cd740 4 client.17394 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.674+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1159840388 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b099119d0 con 0x561b099b5ba0 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.675+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x561b099b5ba0 0x561b099b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.681 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.675+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x561b099b5ba0 0x561b099b8040 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ee6bb1db07284100 server_cookie=bd0642f1d813ff09 in_seq=0 out_seq=0 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.675+0000 7f66857fa640 10 client.17394 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66857fa640 1 -- 172.21.15.49:0/1159840388 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561b099119d0 con 0x561b099b5ba0 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66857fa640 10 client.17394 handle_command_reply: tid=0 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 client.17394 shutdown 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1159840388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 msgr2=0x561b099aa2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099aa2c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6694009900 tx=0x7f6694005250 comp rx=0 tx=0).stop 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1159840388 shutdown_connections 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x561b099b5ba0 0x561b099b8040 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 0x561b099b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099aa2c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1159840388 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b0999f850 0x561b099a9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66857fa640 0 client.17394 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1159840388 >> 172.21.15.49:0/1159840388 conn(0x561b0999adc0 msgr2=0x561b09999fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.682 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1159840388 shutdown_connections 2024-03-18T00:58:12.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1159840388 wait complete. 2024-03-18T00:58:12.683 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.676+0000 7f66aa9cd740 20 client.17394 trim_cache size 0 max 16384 2024-03-18T00:58:12.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.689+0000 7f66aa9cd740 1 Processor -- start 2024-03-18T00:58:12.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.689+0000 7f66aa9cd740 1 -- start start 2024-03-18T00:58:12.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.689+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b0999f850 0x561b0999e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.694 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.689+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099b1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.689+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a53b0 0x561b099bc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.689+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f66a01aaf90 con 0x561b099a0240 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.689+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f66a01acad0 con 0x561b099a53b0 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.689+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f66a01ab660 con 0x561b0999f850 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.689+0000 7f66a52a5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a52a5640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47454/0 (socket says 172.21.15.49:47454) 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a52a5640 1 -- 172.21.15.49:0/2773530410 learned_addr learned my addr 172.21.15.49:0/2773530410 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/2773530410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b0999f850 0x561b0999e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a62a7640 1 --2- 172.21.15.49:0/2773530410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a53b0 0x561b099bc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a52a5640 1 -- 172.21.15.49:0/2773530410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b0999f850 msgr2=0x561b0999e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a52a5640 1 --2- 172.21.15.49:0/2773530410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b0999f850 0x561b0999e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a52a5640 1 -- 172.21.15.49:0/2773530410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a53b0 msgr2=0x561b099bc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a52a5640 1 --2- 172.21.15.49:0/2773530410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a53b0 0x561b099bc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a52a5640 1 -- 172.21.15.49:0/2773530410 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b096e27d0 con 0x561b099a0240 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66a52a5640 1 --2- 172.21.15.49:0/2773530410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099b1280 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6694003dc0 tx=0x7f6694005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29de0311b5bf85cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66877fe640 1 -- 172.21.15.49:0/2773530410 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6694055020 con 0x561b099a0240 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66877fe640 1 -- 172.21.15.49:0/2773530410 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6694041070 con 0x561b099a0240 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.690+0000 7f66877fe640 1 -- 172.21.15.49:0/2773530410 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66940035f0 con 0x561b099a0240 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 -- 172.21.15.49:0/2773530410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 msgr2=0x561b099b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/2773530410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099b1280 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6694003dc0 tx=0x7f6694005c90 comp rx=0 tx=0).stop 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 -- 172.21.15.49:0/2773530410 shutdown_connections 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/2773530410 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a53b0 0x561b099bc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/2773530410 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b099a0240 0x561b099b1280 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/2773530410 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b0999f850 0x561b0999e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 -- 172.21.15.49:0/2773530410 >> 172.21.15.49:0/2773530410 conn(0x561b0999adc0 msgr2=0x561b0999e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 -- 172.21.15.49:0/2773530410 shutdown_connections 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 -- 172.21.15.49:0/2773530410 wait complete. 2024-03-18T00:58:12.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 -- start start 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 Processor -- start 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b0999f850 0x561b099aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a0240 0x561b099aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 0x561b099afff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f66a01ac400 con 0x561b0999f850 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f66a01abd30 con 0x561b099a0240 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f66a01a8f70 con 0x561b099a53b0 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.691+0000 7f66aa9cd740 10 client.0 _ll_register_callbacks cb 0x7f6694009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a5aa6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b0999f850 0x561b099aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a5aa6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b0999f850 0x561b099aeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47458/0 (socket says 172.21.15.49:47458) 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a5aa6640 1 -- 172.21.15.49:0/1142763399 learned_addr learned my addr 172.21.15.49:0/1142763399 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a52a5640 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a0240 0x561b099aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a62a7640 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 0x561b099afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a5aa6640 1 -- 172.21.15.49:0/1142763399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 msgr2=0x561b099afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 0x561b099afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a5aa6640 1 -- 172.21.15.49:0/1142763399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a0240 msgr2=0x561b099aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a0240 0x561b099aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a5aa6640 1 -- 172.21.15.49:0/1142763399 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b099b05e0 con 0x561b0999f850 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a62a7640 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 0x561b099afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66a5aa6640 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b0999f850 0x561b099aeaa0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f66980981c0 tx=0x7f6698000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec93a1f5eea536e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66857fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66857fa640 1 -- 172.21.15.49:0/1142763399 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669809b150 con 0x561b0999f850 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66857fa640 1 -- 172.21.15.49:0/1142763399 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f66980d2640 con 0x561b0999f850 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1142763399 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6694041070 con 0x561b0999f850 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66857fa640 1 -- 172.21.15.49:0/1142763399 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f669809b5c0 con 0x561b0999f850 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66aa9cd740 10 client.17400 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1142763399 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561b09a1c0a0 con 0x561b0999f850 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.692+0000 7f66857fa640 1 -- 172.21.15.49:0/1142763399 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6694041070 con 0x561b0999f850 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66857fa640 1 -- 172.21.15.49:0/1142763399 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f66980bb040 con 0x561b0999f850 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66aa9cd740 10 client.17400 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:12.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66aa9cd740 10 client.17400 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66aa9cd740 20 client.17400 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x561b09a1c620 0x561b09a1ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66aa9cd740 4 client.17400 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1142763399 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b0990bd20 con 0x561b09a1c620 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66a52a5640 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x561b09a1c620 0x561b09a1ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66a52a5640 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x561b09a1c620 0x561b09a1ea10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7247788b9b5afef server_cookie=5bf9faf181ad8340 in_seq=0 out_seq=0 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66857fa640 10 client.17400 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66857fa640 1 -- 172.21.15.49:0/1142763399 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b0990bd20 con 0x561b09a1c620 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66857fa640 10 client.17400 handle_command_reply: tid=0 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66aa9cd740 1 client.17400 shutdown 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1142763399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b0999f850 msgr2=0x561b099aeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.693+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b0999f850 0x561b099aeaa0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f66980981c0 tx=0x7f6698000bc0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1142763399 shutdown_connections 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x561b09a1c620 0x561b09a1ea10 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x561b099a0240 0x561b099aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x561b099a53b0 0x561b099afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66aa9cd740 1 --2- 172.21.15.49:0/1142763399 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x561b0999f850 0x561b099aeaa0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66857fa640 0 client.17400 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:58:12.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1142763399 >> 172.21.15.49:0/1142763399 conn(0x561b0999adc0 msgr2=0x561b0999e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1142763399 shutdown_connections 2024-03-18T00:58:12.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66aa9cd740 1 -- 172.21.15.49:0/1142763399 wait complete. 2024-03-18T00:58:12.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66aa9cd740 20 client.17400 trim_cache size 0 max 16384 2024-03-18T00:58:12.700 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:12.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 -- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f66880a2800 msgr2=0x7f66880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f66880a2800 0x7f66880a4cc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x561b099a0aa0 tx=0x7f6690009500 comp rx=0 tx=0).stop 2024-03-18T00:58:12.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 -- 172.21.15.49:0/3451649257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66a0101810 msgr2=0x7f66a0075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66a0101810 0x7f66a0075590 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f669800e9c0 tx=0x7f669800ee80 comp rx=0 tx=0).stop 2024-03-18T00:58:12.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 -- 172.21.15.49:0/3451649257 shutdown_connections 2024-03-18T00:58:12.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f66880a2800 0x7f66880a4cc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f66a01033c0 0x7f66a0074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f66a0102a00 0x7f66a0073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 --2- 172.21.15.49:0/3451649257 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f66a0101810 0x7f66a0075590 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 -- 172.21.15.49:0/3451649257 >> 172.21.15.49:0/3451649257 conn(0x7f66a00fcd80 msgr2=0x7f66a0105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 -- 172.21.15.49:0/3451649257 shutdown_connections 2024-03-18T00:58:12.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.694+0000 7f66a7d0a640 1 -- 172.21.15.49:0/3451649257 wait complete. 2024-03-18T00:58:12.704 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:12 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/669279417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:58:12.705 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:58:12.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.817+0000 7f8823b0c640 1 -- 172.21.15.49:0/2658935680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 msgr2=0x7f881c102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.817+0000 7f8823b0c640 1 --2- 172.21.15.49:0/2658935680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 0x7f881c102e60 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f880c009900 tx=0x7f880c02bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:12.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 -- 172.21.15.49:0/2658935680 shutdown_connections 2024-03-18T00:58:12.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 --2- 172.21.15.49:0/2658935680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c1033a0 0x7f881c109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 --2- 172.21.15.49:0/2658935680 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 0x7f881c102e60 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 --2- 172.21.15.49:0/2658935680 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c1017f0 0x7f881c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 -- 172.21.15.49:0/2658935680 >> 172.21.15.49:0/2658935680 conn(0x7f881c0fcd80 msgr2=0x7f881c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 -- 172.21.15.49:0/2658935680 shutdown_connections 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 -- 172.21.15.49:0/2658935680 wait complete. 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 Processor -- start 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 -- start start 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c1017f0 0x7f881c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 0x7f881c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c1033a0 0x7f881c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f881c10d470 con 0x7f881c1029e0 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f881c10d2f0 con 0x7f881c1033a0 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.818+0000 7f8823b0c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f881c10d5f0 con 0x7f881c1017f0 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88210a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 0x7f881c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88218a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c1017f0 0x7f881c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88218a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c1017f0 0x7f881c197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:47456/0 (socket says 172.21.15.49:47456) 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88210a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 0x7f881c198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47472/0 (socket says 172.21.15.49:47472) 2024-03-18T00:58:12.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88218a8640 1 -- 172.21.15.49:0/1917772716 learned_addr learned my addr 172.21.15.49:0/1917772716 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88220a9640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c1033a0 0x7f881c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88218a8640 1 -- 172.21.15.49:0/1917772716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c1033a0 msgr2=0x7f881c19f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88218a8640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c1033a0 0x7f881c19f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88218a8640 1 -- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 msgr2=0x7f881c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88218a8640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 0x7f881c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88218a8640 1 -- 172.21.15.49:0/1917772716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f881c19f740 con 0x7f881c1017f0 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88210a7640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 0x7f881c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f88218a8640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c1017f0 0x7f881c197b20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f881800e9c0 tx=0x7f881800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f8812ffd640 1 -- 172.21.15.49:0/1917772716 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881800cd80 con 0x7f881c1017f0 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f8823b0c640 1 -- 172.21.15.49:0/1917772716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f881c19fa30 con 0x7f881c1017f0 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f8812ffd640 1 -- 172.21.15.49:0/1917772716 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88180035f0 con 0x7f881c1017f0 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f8812ffd640 1 -- 172.21.15.49:0/1917772716 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8818010690 con 0x7f881c1017f0 2024-03-18T00:58:12.821 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.819+0000 7f8823b0c640 1 -- 172.21.15.49:0/1917772716 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f881c19ff50 con 0x7f881c1017f0 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.821+0000 7f8812ffd640 1 -- 172.21.15.49:0/1917772716 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f88180037a0 con 0x7f881c1017f0 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f88267cb740 1 Processor -- start 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f88267cb740 1 -- start start 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f8812ffd640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f88080a2800 0x7f88080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f88267cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671602c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f88267cb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671607d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f88267cb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 0x55e67160a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f88267cb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e671567490 con 0x55e671603240 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f88267cb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e67150fc90 con 0x55e6716083b0 2024-03-18T00:58:12.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f88267cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e671563530 con 0x55e671602850 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f8812ffd640 1 -- 172.21.15.49:0/1917772716 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f88180cee30 con 0x7f881c1017f0 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.822+0000 7f88220a9640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f88080a2800 0x7f88080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88218a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671602c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88210a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671607d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88218a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671602c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:47470/0 (socket says 172.21.15.49:47470) 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88218a8640 1 -- 172.21.15.49:0/3129798923 learned_addr learned my addr 172.21.15.49:0/3129798923 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88220a9640 1 --2- 172.21.15.49:0/3129798923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 0x55e67160a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88218a8640 1 -- 172.21.15.49:0/3129798923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 msgr2=0x55e67160a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88218a8640 1 --2- 172.21.15.49:0/3129798923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 0x55e67160a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88218a8640 1 -- 172.21.15.49:0/3129798923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 msgr2=0x55e671607d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88218a8640 1 --2- 172.21.15.49:0/3129798923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671607d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88218a8640 1 -- 172.21.15.49:0/3129798923 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6715ca500 con 0x55e671602850 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88220a9640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f88080a2800 0x7f88080a4cc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8814005fd0 tx=0x7f8814013b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f88218a8640 1 --2- 172.21.15.49:0/3129798923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671602c70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f88180a3a20 tx=0x7f881809b110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e839fd884d83a1a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f8807fff640 1 -- 172.21.15.49:0/3129798923 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881809beb0 con 0x55e671602850 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f8807fff640 1 -- 172.21.15.49:0/3129798923 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f881809b7b0 con 0x55e671602850 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.823+0000 7f8807fff640 1 -- 172.21.15.49:0/3129798923 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8818086070 con 0x55e671602850 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 -- 172.21.15.49:0/3129798923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 msgr2=0x55e671602c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 --2- 172.21.15.49:0/3129798923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671602c70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f88180a3a20 tx=0x7f881809b110 comp rx=0 tx=0).stop 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 -- 172.21.15.49:0/3129798923 shutdown_connections 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 --2- 172.21.15.49:0/3129798923 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 0x55e67160a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 --2- 172.21.15.49:0/3129798923 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671607d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 --2- 172.21.15.49:0/3129798923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671602c70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 -- 172.21.15.49:0/3129798923 >> 172.21.15.49:0/3129798923 conn(0x55e6715fddc0 msgr2=0x55e6715fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 -- 172.21.15.49:0/3129798923 shutdown_connections 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 -- 172.21.15.49:0/3129798923 wait complete. 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 -- start start 2024-03-18T00:58:12.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.824+0000 7f88267cb740 1 Processor -- start 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88267cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88267cb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 0x55e6716116b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88267cb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e6716186f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88267cb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55e67160f710 con 0x55e6716083b0 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88267cb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55e671523230 con 0x55e671603240 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88267cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55e67160f890 con 0x55e671602850 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88267cb740 10 client.0 _ll_register_callbacks cb 0x55e67155f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88220a9640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e6716186f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88218a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88220a9640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e6716186f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47484/0 (socket says 172.21.15.49:47484) 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88220a9640 1 -- 172.21.15.49:0/2855821980 learned_addr learned my addr 172.21.15.49:0/2855821980 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88210a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 0x55e6716116b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88218a8640 1 -- 172.21.15.49:0/2855821980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 msgr2=0x55e6716116b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88218a8640 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 0x55e6716116b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88218a8640 1 -- 172.21.15.49:0/2855821980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 msgr2=0x55e6716186f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88218a8640 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e6716186f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88218a8640 1 -- 172.21.15.49:0/2855821980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e671532100 con 0x55e671602850 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.825+0000 7f88220a9640 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e6716186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88218a8640 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88180a3a20 tx=0x7f8818097b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9ad8c1a5adfd0fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88057fa640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88057fa640 1 -- 172.21.15.49:0/2855821980 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f881809beb0 con 0x55e671602850 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88057fa640 1 -- 172.21.15.49:0/2855821980 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f881804e6b0 con 0x55e671602850 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88267cb740 1 -- 172.21.15.49:0/2855821980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f881809b7b0 con 0x55e671602850 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88057fa640 1 -- 172.21.15.49:0/2855821980 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f88180a3f50 con 0x55e671602850 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88267cb740 10 client.26656 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88267cb740 1 -- 172.21.15.49:0/2855821980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e67127b3f0 con 0x55e671602850 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88057fa640 1 -- 172.21.15.49:0/2855821980 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f881809b7b0 con 0x55e671602850 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88057fa640 1 -- 172.21.15.49:0/2855821980 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f881804e200 con 0x55e671602850 2024-03-18T00:58:12.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88267cb740 10 client.26656 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88267cb740 10 client.26656 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88267cb740 20 client.26656 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88267cb740 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e67161cbe0 0x55e67161efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88267cb740 4 client.26656 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88267cb740 1 -- 172.21.15.49:0/2855821980 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6715749d0 con 0x55e67161cbe0 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.826+0000 7f88210a7640 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e67161cbe0 0x55e67161efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.827+0000 7f88210a7640 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e67161cbe0 0x55e67161efd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f714c54d0cfb3ed6 server_cookie=320e76a35ee4ac68 in_seq=0 out_seq=0 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.827+0000 7f88057fa640 10 client.26656 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.827+0000 7f88057fa640 1 -- 172.21.15.49:0/2855821980 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e6715749d0 con 0x55e67161cbe0 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.827+0000 7f88057fa640 10 client.26656 handle_command_reply: tid=0 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.827+0000 7f88267cb740 1 client.26656 shutdown 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 -- 172.21.15.49:0/2855821980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 msgr2=0x55e671611170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611170 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88180a3a20 tx=0x7f8818097b50 comp rx=0 tx=0).stop 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 -- 172.21.15.49:0/2855821980 shutdown_connections 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e67161cbe0 0x55e67161efd0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 0x55e6716116b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e6716186f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 --2- 172.21.15.49:0/2855821980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611170 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88057fa640 0 client.26656 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:58:12.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 -- 172.21.15.49:0/2855821980 >> 172.21.15.49:0/2855821980 conn(0x55e6715fddc0 msgr2=0x55e6715fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 -- 172.21.15.49:0/2855821980 shutdown_connections 2024-03-18T00:58:12.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 1 -- 172.21.15.49:0/2855821980 wait complete. 2024-03-18T00:58:12.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.828+0000 7f88267cb740 20 client.26656 trim_cache size 0 max 16384 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88267cb740 1 Processor -- start 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88267cb740 1 -- start start 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88267cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88267cb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 0x55e67161efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88267cb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e671603cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88267cb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1a6ba0 con 0x55e6716083b0 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88267cb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1a86e0 con 0x55e671603240 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88267cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1a7270 con 0x55e671602850 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88218a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88220a9640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e671603cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88210a7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 0x55e67161efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88218a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:47486/0 (socket says 172.21.15.49:47486) 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.841+0000 7f88218a8640 1 -- 172.21.15.49:0/2072561324 learned_addr learned my addr 172.21.15.49:0/2072561324 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.842+0000 7f88210a7640 1 -- 172.21.15.49:0/2072561324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 msgr2=0x55e671611610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.842+0000 7f88210a7640 1 --2- 172.21.15.49:0/2072561324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.842+0000 7f88210a7640 1 -- 172.21.15.49:0/2072561324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 msgr2=0x55e671603cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.842+0000 7f88210a7640 1 --2- 172.21.15.49:0/2072561324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e671603cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.842+0000 7f88210a7640 1 -- 172.21.15.49:0/2072561324 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6713457d0 con 0x55e671603240 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.842+0000 7f88210a7640 1 --2- 172.21.15.49:0/2072561324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 0x55e67161efd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f880c009230 tx=0x7f880c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=144b2fe85740292d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.842+0000 7f88077fe640 1 -- 172.21.15.49:0/2072561324 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c04a070 con 0x55e671603240 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.842+0000 7f88077fe640 1 -- 172.21.15.49:0/2072561324 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f880c046dd0 con 0x55e671603240 2024-03-18T00:58:12.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.842+0000 7f88077fe640 1 -- 172.21.15.49:0/2072561324 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c005330 con 0x55e671603240 2024-03-18T00:58:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 -- 172.21.15.49:0/2072561324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 msgr2=0x55e67161efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 --2- 172.21.15.49:0/2072561324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 0x55e67161efd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f880c009230 tx=0x7f880c005fd0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 -- 172.21.15.49:0/2072561324 shutdown_connections 2024-03-18T00:58:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 --2- 172.21.15.49:0/2072561324 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e671603240 0x55e67161efd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 --2- 172.21.15.49:0/2072561324 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e6716083b0 0x55e671603cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 --2- 172.21.15.49:0/2072561324 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e671611610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 -- 172.21.15.49:0/2072561324 >> 172.21.15.49:0/2072561324 conn(0x55e6715fddc0 msgr2=0x55e67160fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 -- 172.21.15.49:0/2072561324 shutdown_connections 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 -- 172.21.15.49:0/2072561324 wait complete. 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 -- start start 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 Processor -- start 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e6716181f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671614c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 0x55e6716151d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1a8010 con 0x55e671603240 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1a7940 con 0x55e6716083b0 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.843+0000 7f88267cb740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f881c1a4b80 con 0x55e671602850 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88267cb740 10 client.0 _ll_register_callbacks cb 0x7f88140093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88210a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671614c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88210a7640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671614c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47502/0 (socket says 172.21.15.49:47502) 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88220a9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 0x55e6716151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88218a8640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e6716181f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88210a7640 1 -- 172.21.15.49:0/3031194956 learned_addr learned my addr 172.21.15.49:0/3031194956 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88210a7640 1 -- 172.21.15.49:0/3031194956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 msgr2=0x55e6716181f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88210a7640 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e6716181f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88210a7640 1 -- 172.21.15.49:0/3031194956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 msgr2=0x55e6716151d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88210a7640 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 0x55e6716151d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88210a7640 1 -- 172.21.15.49:0/3031194956 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e671680ca0 con 0x55e671603240 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88220a9640 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 0x55e6716151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88218a8640 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e6716181f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88210a7640 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671614c90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f880c02cfd0 tx=0x7f880c046c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17160b2ef44a202 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88057fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88057fa640 1 -- 172.21.15.49:0/3031194956 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880c053030 con 0x55e671603240 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88057fa640 1 -- 172.21.15.49:0/3031194956 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f880c0040c0 con 0x55e671603240 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.844+0000 7f88267cb740 1 -- 172.21.15.49:0/3031194956 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f880c046dd0 con 0x55e671603240 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88057fa640 1 -- 172.21.15.49:0/3031194956 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f880c04a040 con 0x55e671603240 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88267cb740 10 client.17418 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88267cb740 1 -- 172.21.15.49:0/3031194956 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e671681e90 con 0x55e671603240 2024-03-18T00:58:12.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88057fa640 1 -- 172.21.15.49:0/3031194956 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f880c046dd0 con 0x55e671603240 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88057fa640 1 -- 172.21.15.49:0/3031194956 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f880c0047c0 con 0x55e671603240 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88267cb740 10 client.17418 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88267cb740 10 client.17418 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88267cb740 20 client.17418 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88267cb740 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e671682870 0x55e671684c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88267cb740 4 client.17418 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88267cb740 1 -- 172.21.15.49:0/3031194956 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e67156ed20 con 0x55e671682870 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88218a8640 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e671682870 0x55e671684c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88057fa640 10 client.17418 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.845+0000 7f88218a8640 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e671682870 0x55e671684c90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e6421c3fe394672d server_cookie=8b6344ad96d8fd2c in_seq=0 out_seq=0 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88057fa640 1 -- 172.21.15.49:0/3031194956 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e67156ed20 con 0x55e671682870 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88057fa640 10 client.17418 handle_command_reply: tid=0 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 client.17418 shutdown 2024-03-18T00:58:12.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 -- 172.21.15.49:0/3031194956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 msgr2=0x55e671614c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671614c90 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f880c02cfd0 tx=0x7f880c046c60 comp rx=0 tx=0).stop 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 -- 172.21.15.49:0/3031194956 shutdown_connections 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55e671682870 0x55e671684c90 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55e6716083b0 0x55e6716151d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55e671603240 0x55e671614c90 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 --2- 172.21.15.49:0/3031194956 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55e671602850 0x55e6716181f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88057fa640 0 client.17418 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 -- 172.21.15.49:0/3031194956 >> 172.21.15.49:0/3031194956 conn(0x55e6715fddc0 msgr2=0x55e6715fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 -- 172.21.15.49:0/3031194956 shutdown_connections 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 1 -- 172.21.15.49:0/3031194956 wait complete. 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f88267cb740 20 client.17418 trim_cache size 0 max 16384 2024-03-18T00:58:12.851 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:12.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f8823b0c640 1 -- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f88080a2800 msgr2=0x7f88080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f8823b0c640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f88080a2800 0x7f88080a4cc0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8814005fd0 tx=0x7f8814013b60 comp rx=0 tx=0).stop 2024-03-18T00:58:12.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f8823b0c640 1 -- 172.21.15.49:0/1917772716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c1017f0 msgr2=0x7f881c197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.846+0000 7f8823b0c640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c1017f0 0x7f881c197b20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f881800e9c0 tx=0x7f881800ee80 comp rx=0 tx=0).stop 2024-03-18T00:58:12.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.847+0000 7f8823b0c640 1 -- 172.21.15.49:0/1917772716 shutdown_connections 2024-03-18T00:58:12.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.847+0000 7f8823b0c640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f88080a2800 0x7f88080a4cc0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.847+0000 7f8823b0c640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f881c1033a0 0x7f881c19f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.847+0000 7f8823b0c640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f881c1029e0 0x7f881c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.847+0000 7f8823b0c640 1 --2- 172.21.15.49:0/1917772716 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f881c1017f0 0x7f881c197b20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.847+0000 7f8823b0c640 1 -- 172.21.15.49:0/1917772716 >> 172.21.15.49:0/1917772716 conn(0x7f881c0fcd80 msgr2=0x7f881c105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.847+0000 7f8823b0c640 1 -- 172.21.15.49:0/1917772716 shutdown_connections 2024-03-18T00:58:12.853 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.847+0000 7f8823b0c640 1 -- 172.21.15.49:0/1917772716 wait complete. 2024-03-18T00:58:12.857 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:58:12.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:12 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/669279417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:58:12.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:12 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/669279417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:58:12.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.969+0000 7fd0029cd640 1 -- 172.21.15.49:0/46057402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc100700 msgr2=0x7fcffc100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.969+0000 7fd0029cd640 1 --2- 172.21.15.49:0/46057402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc100700 0x7fcffc100b20 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009900 tx=0x7fcfec02bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:12.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.969+0000 7fd0029cd640 1 -- 172.21.15.49:0/46057402 shutdown_connections 2024-03-18T00:58:12.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.969+0000 7fd0029cd640 1 --2- 172.21.15.49:0/46057402 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcffc1022b0 0x7fcffc108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.969+0000 7fd0029cd640 1 --2- 172.21.15.49:0/46057402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcffc1018f0 0x7fcffc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.969+0000 7fd0029cd640 1 --2- 172.21.15.49:0/46057402 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc100700 0x7fcffc100b20 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.969+0000 7fd0029cd640 1 -- 172.21.15.49:0/46057402 >> 172.21.15.49:0/46057402 conn(0x7fcffc0fbc90 msgr2=0x7fcffc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.969+0000 7fd0029cd640 1 -- 172.21.15.49:0/46057402 shutdown_connections 2024-03-18T00:58:12.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.969+0000 7fd0029cd640 1 -- 172.21.15.49:0/46057402 wait complete. 2024-03-18T00:58:12.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd0029cd640 1 Processor -- start 2024-03-18T00:58:12.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd0029cd640 1 -- start start 2024-03-18T00:58:12.971 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd0029cd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcffc100700 0x7fcffc104620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd0029cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcffc1018f0 0x7fcffc102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd0029cd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc1022b0 0x7fcffc103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd0029cd640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc10c3c0 con 0x7fcffc1022b0 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd0029cd640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc10c240 con 0x7fcffc100700 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd0029cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc10c540 con 0x7fcffc1018f0 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd000f6a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc1022b0 0x7fcffc103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fcffbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcffc100700 0x7fcffc104620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fcffb7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcffc1018f0 0x7fcffc102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd000f6a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc1022b0 0x7fcffc103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47522/0 (socket says 172.21.15.49:47522) 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fcffbfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcffc100700 0x7fcffc104620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:39654/0 (socket says 172.21.15.49:39654) 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.970+0000 7fd000f6a640 1 -- 172.21.15.49:0/3267660409 learned_addr learned my addr 172.21.15.49:0/3267660409 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fd000f6a640 1 -- 172.21.15.49:0/3267660409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcffc1018f0 msgr2=0x7fcffc102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fd000f6a640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcffc1018f0 0x7fcffc102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fd000f6a640 1 -- 172.21.15.49:0/3267660409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcffc100700 msgr2=0x7fcffc104620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.972 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fd000f6a640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcffc100700 0x7fcffc104620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fd000f6a640 1 -- 172.21.15.49:0/3267660409 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcffc103a30 con 0x7fcffc1022b0 2024-03-18T00:58:12.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fd000f6a640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc1022b0 0x7fcffc103210 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fcff400eb10 tx=0x7fcff400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fcff97fa640 1 -- 172.21.15.49:0/3267660409 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4004b20 con 0x7fcffc1022b0 2024-03-18T00:58:12.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fd0029cd640 1 -- 172.21.15.49:0/3267660409 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcffc1a1cd0 con 0x7fcffc1022b0 2024-03-18T00:58:12.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fcff97fa640 1 -- 172.21.15.49:0/3267660409 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcff4004cc0 con 0x7fcffc1022b0 2024-03-18T00:58:12.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fcff97fa640 1 -- 172.21.15.49:0/3267660409 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff4010760 con 0x7fcffc1022b0 2024-03-18T00:58:12.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.971+0000 7fd0029cd640 1 -- 172.21.15.49:0/3267660409 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcffc1a21d0 con 0x7fcffc1022b0 2024-03-18T00:58:12.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.973+0000 7fcff97fa640 1 -- 172.21.15.49:0/3267660409 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcff4005020 con 0x7fcffc1022b0 2024-03-18T00:58:12.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.973+0000 7fd0055b8740 1 Processor -- start 2024-03-18T00:58:12.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fd0055b8740 1 -- start start 2024-03-18T00:58:12.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d646898c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d64689dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 0x55d6468a0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.978 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fd0055b8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d6467fd490 con 0x55d646898850 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fd0055b8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d6467a5c90 con 0x55d64689e3b0 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fd0055b8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d6467f9530 con 0x55d646899240 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d646898c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d646898c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47534/0 (socket says 172.21.15.49:47534) 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d64689dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffb7fe640 1 -- 172.21.15.49:0/3198890701 learned_addr learned my addr 172.21.15.49:0/3198890701 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/3198890701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 0x55d6468a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcff97fa640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffbfff640 1 -- 172.21.15.49:0/3198890701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 msgr2=0x55d6468a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffbfff640 1 --2- 172.21.15.49:0/3198890701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 0x55d6468a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffbfff640 1 -- 172.21.15.49:0/3198890701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 msgr2=0x55d646898c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffbfff640 1 --2- 172.21.15.49:0/3198890701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d646898c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffbfff640 1 -- 172.21.15.49:0/3198890701 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d646860500 con 0x55d646899240 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcff97fa640 1 -- 172.21.15.49:0/3267660409 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcff40ceff0 con 0x7fcffc1022b0 2024-03-18T00:58:12.979 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fd000f6a640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.974+0000 7fcffbfff640 1 --2- 172.21.15.49:0/3198890701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d64689dd40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009230 tx=0x7fcfec02c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15bbb800d76c0b3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fcfe27fc640 1 -- 172.21.15.49:0/3198890701 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec02c980 con 0x55d646899240 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fcfe27fc640 1 -- 172.21.15.49:0/3198890701 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfec02cb20 con 0x55d646899240 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fcfe27fc640 1 -- 172.21.15.49:0/3198890701 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0046c0 con 0x55d646899240 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd0055b8740 1 -- 172.21.15.49:0/3198890701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 msgr2=0x55d64689dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3198890701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d64689dd40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcfec009230 tx=0x7fcfec02c670 comp rx=0 tx=0).stop 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd000f6a640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcff4098340 tx=0x7fcff400efd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd0055b8740 1 -- 172.21.15.49:0/3198890701 shutdown_connections 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3198890701 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 0x55d6468a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3198890701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d64689dd40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3198890701 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d646898c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd0055b8740 1 -- 172.21.15.49:0/3198890701 >> 172.21.15.49:0/3198890701 conn(0x55d646893dc0 msgr2=0x55d6468941c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd0055b8740 1 -- 172.21.15.49:0/3198890701 shutdown_connections 2024-03-18T00:58:12.980 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.975+0000 7fd0055b8740 1 -- 172.21.15.49:0/3198890701 wait complete. 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fd0055b8740 1 -- start start 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fd0055b8740 1 Processor -- start 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646898850 0x55d6468a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d6468a3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d64689e3b0 0x55d6468aa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fd0055b8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55d6468a1300 con 0x55d64689e3b0 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fd0055b8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55d6467b9230 con 0x55d646898850 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fd0055b8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55d6468a1480 con 0x55d646899240 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fd0055b8740 10 client.0 _ll_register_callbacks cb 0x55d6467f5010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fcffbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d6468a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fcffb7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646898850 0x55d6468a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fcffbfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d6468a3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:47510/0 (socket says 172.21.15.49:47510) 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fcffb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d64689e3b0 0x55d6468aa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.976+0000 7fcffbfff640 1 -- 172.21.15.49:0/563003150 learned_addr learned my addr 172.21.15.49:0/563003150 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcffbfff640 1 -- 172.21.15.49:0/563003150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646898850 msgr2=0x55d6468a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcffbfff640 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646898850 0x55d6468a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcffbfff640 1 -- 172.21.15.49:0/563003150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d64689e3b0 msgr2=0x55d6468aa220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcffbfff640 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d64689e3b0 0x55d6468aa220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcffbfff640 1 -- 172.21.15.49:0/563003150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6467c8100 con 0x55d646899240 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d64689e3b0 0x55d6468aa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcffbfff640 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d6468a3260 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcfec005c40 tx=0x7fcfec009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94244259e0719f84 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcfdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/563003150 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec02c980 con 0x55d646899240 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/563003150 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcfec004e80 con 0x55d646899240 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fd0055b8740 1 -- 172.21.15.49:0/563003150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfec02cb20 con 0x55d646899240 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/563003150 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcfec03f440 con 0x55d646899240 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fd0055b8740 10 client.26674 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fd0055b8740 1 -- 172.21.15.49:0/563003150 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d6465113f0 con 0x55d646899240 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.977+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/563003150 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfec02cb20 con 0x55d646899240 2024-03-18T00:58:12.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/563003150 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcfec053440 con 0x55d646899240 2024-03-18T00:58:12.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fd0055b8740 10 client.26674 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:12.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fd0055b8740 10 client.26674 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:58:12.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fd0055b8740 20 client.26674 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:12.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fd0055b8740 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55d6468ae710 0x55d6468b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fd0055b8740 4 client.26674 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:12.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fd0055b8740 1 -- 172.21.15.49:0/563003150 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d64680a9d0 con 0x55d6468ae710 2024-03-18T00:58:12.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55d6468ae710 0x55d6468b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55d6468ae710 0x55d6468b0b00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edb86e2a17b35dd server_cookie=eb3e6b66f1048436 in_seq=0 out_seq=0 2024-03-18T00:58:12.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.978+0000 7fcfdb7fe640 10 client.26674 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:58:12.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/563003150 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d64680a9d0 con 0x55d6468ae710 2024-03-18T00:58:12.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fcfdb7fe640 10 client.26674 handle_command_reply: tid=0 2024-03-18T00:58:12.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 client.26674 shutdown 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 -- 172.21.15.49:0/563003150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 msgr2=0x55d6468a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d6468a3260 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcfec005c40 tx=0x7fcfec009260 comp rx=0 tx=0).stop 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 -- 172.21.15.49:0/563003150 shutdown_connections 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55d6468ae710 0x55d6468b0b00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d64689e3b0 0x55d6468aa220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fcfdb7fe640 0 client.26674 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646899240 0x55d6468a3260 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 --2- 172.21.15.49:0/563003150 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646898850 0x55d6468a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 -- 172.21.15.49:0/563003150 >> 172.21.15.49:0/563003150 conn(0x55d646893dc0 msgr2=0x55d646892fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 -- 172.21.15.49:0/563003150 shutdown_connections 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.979+0000 7fd0055b8740 1 -- 172.21.15.49:0/563003150 wait complete. 2024-03-18T00:58:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.980+0000 7fd0055b8740 20 client.26674 trim_cache size 0 max 16384 2024-03-18T00:58:12.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fd0055b8740 1 Processor -- start 2024-03-18T00:58:12.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fd0055b8740 1 -- start start 2024-03-18T00:58:12.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646898850 0x55d6468941a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646899240 0x55d6468aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 0x55d6468a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:12.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fd0055b8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1aaf30 con 0x55d646899240 2024-03-18T00:58:12.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fd0055b8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1aca70 con 0x55d64689e3b0 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fd0055b8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1ab600 con 0x55d646898850 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646899240 0x55d6468aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffbfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646899240 0x55d6468aa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47542/0 (socket says 172.21.15.49:47542) 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffbfff640 1 -- 172.21.15.49:0/1567017601 learned_addr learned my addr 172.21.15.49:0/1567017601 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/1567017601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646898850 0x55d6468941a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/1567017601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 0x55d6468a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffbfff640 1 -- 172.21.15.49:0/1567017601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646898850 msgr2=0x55d6468941a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffbfff640 1 --2- 172.21.15.49:0/1567017601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646898850 0x55d6468941a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffbfff640 1 -- 172.21.15.49:0/1567017601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 msgr2=0x55d6468a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffbfff640 1 --2- 172.21.15.49:0/1567017601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 0x55d6468a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.993+0000 7fcffbfff640 1 -- 172.21.15.49:0/1567017601 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6465db7d0 con 0x55d646899240 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fcffbfff640 1 --2- 172.21.15.49:0/1567017601 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646899240 0x55d6468aa1e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0096c0 tx=0x7fcfec02c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad841346be0d650d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fcfe1ffb640 1 -- 172.21.15.49:0/1567017601 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec005850 con 0x55d646899240 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fcfe1ffb640 1 -- 172.21.15.49:0/1567017601 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfec053e50 con 0x55d646899240 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fcfe1ffb640 1 -- 172.21.15.49:0/1567017601 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec004f00 con 0x55d646899240 2024-03-18T00:58:12.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fd0055b8740 1 -- 172.21.15.49:0/1567017601 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646899240 msgr2=0x55d6468aa1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fd0055b8740 1 --2- 172.21.15.49:0/1567017601 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646899240 0x55d6468aa1e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fcfec0096c0 tx=0x7fcfec02c2b0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fd0055b8740 1 -- 172.21.15.49:0/1567017601 shutdown_connections 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fd0055b8740 1 --2- 172.21.15.49:0/1567017601 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d64689e3b0 0x55d6468a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fd0055b8740 1 --2- 172.21.15.49:0/1567017601 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646899240 0x55d6468aa1e0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fd0055b8740 1 --2- 172.21.15.49:0/1567017601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d646898850 0x55d6468941a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fd0055b8740 1 -- 172.21.15.49:0/1567017601 >> 172.21.15.49:0/1567017601 conn(0x55d646893dc0 msgr2=0x55d646899cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fd0055b8740 1 -- 172.21.15.49:0/1567017601 shutdown_connections 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.994+0000 7fd0055b8740 1 -- 172.21.15.49:0/1567017601 wait complete. 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fd0055b8740 1 -- start start 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fd0055b8740 1 Processor -- start 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d6468a9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646899240 0x55d6468a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fd0055b8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d64689e3b0 0x55d6468a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fd0055b8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1ac3a0 con 0x55d646898850 2024-03-18T00:58:13.000 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fd0055b8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1abcd0 con 0x55d646899240 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fd0055b8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc1a8f10 con 0x55d64689e3b0 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fd0055b8740 10 client.0 _ll_register_callbacks cb 0x7fcff0009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fcffb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d6468a9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fcffb7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d6468a9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47550/0 (socket says 172.21.15.49:47550) 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fcffb7fe640 1 -- 172.21.15.49:0/3817543850 learned_addr learned my addr 172.21.15.49:0/3817543850 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fcffbfff640 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646899240 0x55d6468a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.995+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d64689e3b0 0x55d6468a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcffbfff640 1 -- 172.21.15.49:0/3817543850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d64689e3b0 msgr2=0x55d6468a6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcffbfff640 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d64689e3b0 0x55d6468a6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcffbfff640 1 -- 172.21.15.49:0/3817543850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 msgr2=0x55d6468a9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcffbfff640 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d6468a9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcffbfff640 1 -- 172.21.15.49:0/3817543850 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6469129f0 con 0x55d646899240 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d6468a9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcffbfff640 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646899240 0x55d6468a6a80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcfec02c170 tx=0x7fcfec004990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8759b22d061cde63 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:13.001 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcfdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/3817543850 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec02cc50 con 0x55d646899240 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/3817543850 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fcfec004c80 con 0x55d646899240 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fd0055b8740 1 -- 172.21.15.49:0/3817543850 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcfec053e50 con 0x55d646899240 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/3817543850 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcfec004e80 con 0x55d646899240 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fd0055b8740 10 client.26588 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fd0055b8740 1 -- 172.21.15.49:0/3817543850 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d646913b60 con 0x55d646899240 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.996+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/3817543850 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfec053e50 con 0x55d646899240 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/3817543850 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcfec034440 con 0x55d646899240 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fd0055b8740 10 client.26588 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fd0055b8740 10 client.26588 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fd0055b8740 20 client.26588 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55d646914320 0x55d646916740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fd0055b8740 4 client.26588 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fd0055b8740 1 -- 172.21.15.49:0/3817543850 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d646804d20 con 0x55d646914320 2024-03-18T00:58:13.002 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55d646914320 0x55d646916740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fcfdb7fe640 10 client.26588 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fcffb7fe640 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55d646914320 0x55d646916740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9e78c4579b643789 server_cookie=4465d671622d1c6b in_seq=0 out_seq=0 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fcfdb7fe640 1 -- 172.21.15.49:0/3817543850 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d646804d20 con 0x55d646914320 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fcfdb7fe640 10 client.26588 handle_command_reply: tid=0 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.997+0000 7fd0055b8740 1 client.26588 shutdown 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 -- 172.21.15.49:0/3817543850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646899240 msgr2=0x55d6468a6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646899240 0x55d6468a6a80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcfec02c170 tx=0x7fcfec004990 comp rx=0 tx=0).stop 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 -- 172.21.15.49:0/3817543850 shutdown_connections 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55d646914320 0x55d646916740 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55d646899240 0x55d6468a6a80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d64689e3b0 0x55d6468a6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fcfdb7fe640 0 client.26588 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 --2- 172.21.15.49:0/3817543850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55d646898850 0x55d6468a9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 -- 172.21.15.49:0/3817543850 >> 172.21.15.49:0/3817543850 conn(0x55d646893dc0 msgr2=0x55d64689a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 -- 172.21.15.49:0/3817543850 shutdown_connections 2024-03-18T00:58:13.003 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 1 -- 172.21.15.49:0/3817543850 wait complete. 2024-03-18T00:58:13.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0055b8740 20 client.26588 trim_cache size 0 max 16384 2024-03-18T00:58:13.004 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 -- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfe40a2800 msgr2=0x7fcfe40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fcff4098340 tx=0x7fcff400efd0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 -- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc1022b0 msgr2=0x7fcffc103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc1022b0 0x7fcffc103210 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fcff400eb10 tx=0x7fcff400c2d0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 -- 172.21.15.49:0/3267660409 shutdown_connections 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcfe40a2800 0x7fcfe40a4cc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcffc1022b0 0x7fcffc103210 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcffc1018f0 0x7fcffc102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 --2- 172.21.15.49:0/3267660409 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcffc100700 0x7fcffc104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 -- 172.21.15.49:0/3267660409 >> 172.21.15.49:0/3267660409 conn(0x7fcffc0fbc90 msgr2=0x7fcffc0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 -- 172.21.15.49:0/3267660409 shutdown_connections 2024-03-18T00:58:13.005 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:12.998+0000 7fd0029cd640 1 -- 172.21.15.49:0/3267660409 wait complete. 2024-03-18T00:58:13.009 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:58:13.120 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 -- 172.21.15.49:0/3192961979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 msgr2=0x7fa148102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 --2- 172.21.15.49:0/3192961979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 0x7fa148102e70 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa130009900 tx=0x7fa13002bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:13.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 -- 172.21.15.49:0/3192961979 shutdown_connections 2024-03-18T00:58:13.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 --2- 172.21.15.49:0/3192961979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1481033b0 0x7fa148109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 --2- 172.21.15.49:0/3192961979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 0x7fa148102e70 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 --2- 172.21.15.49:0/3192961979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa148101800 0x7fa148101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 -- 172.21.15.49:0/3192961979 >> 172.21.15.49:0/3192961979 conn(0x7fa1480fcd70 msgr2=0x7fa1480ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:13.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 -- 172.21.15.49:0/3192961979 shutdown_connections 2024-03-18T00:58:13.121 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 -- 172.21.15.49:0/3192961979 wait complete. 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.121+0000 7fa14d66e640 1 Processor -- start 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa14d66e640 1 -- start start 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa14d66e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa148101800 0x7fa148197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa14d66e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 0x7fa148198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa14d66e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1481033b0 0x7fa14819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa14d66e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa14810d480 con 0x7fa1481029f0 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa14d66e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa14810d300 con 0x7fa1481033b0 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa14d66e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa14810d600 con 0x7fa148101800 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1467fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 0x7fa148198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1467fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 0x7fa148198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47568/0 (socket says 172.21.15.49:47568) 2024-03-18T00:58:13.123 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1467fc640 1 -- 172.21.15.49:0/1212697096 learned_addr learned my addr 172.21.15.49:0/1212697096 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa148101800 0x7fa148197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1477fe640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1481033b0 0x7fa14819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1467fc640 1 -- 172.21.15.49:0/1212697096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa148101800 msgr2=0x7fa148197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1467fc640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa148101800 0x7fa148197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1467fc640 1 -- 172.21.15.49:0/1212697096 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1481033b0 msgr2=0x7fa14819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1467fc640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1481033b0 0x7fa14819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1467fc640 1 -- 172.21.15.49:0/1212697096 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14819f6f0 con 0x7fa1481029f0 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.122+0000 7fa1467fc640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 0x7fa148198010 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa130009230 tx=0x7fa13002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.123+0000 7fa13ffff640 1 -- 172.21.15.49:0/1212697096 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa130041070 con 0x7fa1481029f0 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.123+0000 7fa14d66e640 1 -- 172.21.15.49:0/1212697096 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa14819f980 con 0x7fa1481029f0 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.123+0000 7fa13ffff640 1 -- 172.21.15.49:0/1212697096 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa130004450 con 0x7fa1481029f0 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.123+0000 7fa14d66e640 1 -- 172.21.15.49:0/1212697096 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa14819fec0 con 0x7fa1481029f0 2024-03-18T00:58:13.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.123+0000 7fa13ffff640 1 -- 172.21.15.49:0/1212697096 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa130003e20 con 0x7fa1481029f0 2024-03-18T00:58:13.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.125+0000 7fa13ffff640 1 -- 172.21.15.49:0/1212697096 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa130044070 con 0x7fa1481029f0 2024-03-18T00:58:13.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.125+0000 7fa150257740 1 Processor -- start 2024-03-18T00:58:13.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.125+0000 7fa150257740 1 -- start start 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa150257740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57b850 0x55eabb57bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa150257740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb580d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa150257740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb5813b0 0x55eabb583840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa150257740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55eabb4e0490 con 0x55eabb5813b0 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa150257740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55eabb488c90 con 0x55eabb57b850 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa150257740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eabb4dc530 con 0x55eabb57c240 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa1467fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb5813b0 0x55eabb583840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb580d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa1477fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57b850 0x55eabb57bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb580d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:47556/0 (socket says 172.21.15.49:47556) 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa1477fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57b850 0x55eabb57bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:39704/0 (socket says 172.21.15.49:39704) 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa1467fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb5813b0 0x55eabb583840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47582/0 (socket says 172.21.15.49:47582) 2024-03-18T00:58:13.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa146ffd640 1 -- 172.21.15.49:0/3266988141 learned_addr learned my addr 172.21.15.49:0/3266988141 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa146ffd640 1 -- 172.21.15.49:0/3266988141 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57b850 msgr2=0x55eabb57bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa146ffd640 1 --2- 172.21.15.49:0/3266988141 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57b850 0x55eabb57bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa146ffd640 1 -- 172.21.15.49:0/3266988141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb5813b0 msgr2=0x55eabb583840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa146ffd640 1 --2- 172.21.15.49:0/3266988141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb5813b0 0x55eabb583840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa146ffd640 1 -- 172.21.15.49:0/3266988141 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eabb543500 con 0x55eabb57c240 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa1467fc640 1 --2- 172.21.15.49:0/3266988141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb5813b0 0x55eabb583840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa13ffff640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1340a2880 0x7fa1340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa1477fe640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1340a2880 0x7fa1340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa146ffd640 1 --2- 172.21.15.49:0/3266988141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb580d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa138011470 tx=0x7fa138011930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90d499b4a81e2bc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.126+0000 7fa13ffff640 1 -- 172.21.15.49:0/1212697096 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa1300f05b0 con 0x7fa1481029f0 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa13d7fa640 1 -- 172.21.15.49:0/3266988141 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138003970 con 0x55eabb57c240 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa13d7fa640 1 -- 172.21.15.49:0/3266988141 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa138004020 con 0x55eabb57c240 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa13d7fa640 1 -- 172.21.15.49:0/3266988141 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13800f9c0 con 0x55eabb57c240 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa150257740 1 -- 172.21.15.49:0/3266988141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 msgr2=0x55eabb580d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa1477fe640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1340a2880 0x7fa1340a4d40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x55eabb57caa0 tx=0x7fa140005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa150257740 1 --2- 172.21.15.49:0/3266988141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb580d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa138011470 tx=0x7fa138011930 comp rx=0 tx=0).stop 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa150257740 1 -- 172.21.15.49:0/3266988141 shutdown_connections 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa150257740 1 --2- 172.21.15.49:0/3266988141 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb5813b0 0x55eabb583840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa150257740 1 --2- 172.21.15.49:0/3266988141 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb580d40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa150257740 1 --2- 172.21.15.49:0/3266988141 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57b850 0x55eabb57bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa150257740 1 -- 172.21.15.49:0/3266988141 >> 172.21.15.49:0/3266988141 conn(0x55eabb576dc0 msgr2=0x55eabb5771c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa150257740 1 -- 172.21.15.49:0/3266988141 shutdown_connections 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.127+0000 7fa150257740 1 -- 172.21.15.49:0/3266988141 wait complete. 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa150257740 1 -- start start 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa150257740 1 Processor -- start 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa150257740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb57b850 0x55eabb585da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa150257740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb5862e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa150257740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb5813b0 0x55eabb58d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa150257740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55eabb584360 con 0x55eabb57b850 2024-03-18T00:58:13.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa150257740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55eabb49c230 con 0x55eabb5813b0 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa150257740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55eabb5844e0 con 0x55eabb57c240 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb57b850 0x55eabb585da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa150257740 10 client.0 _ll_register_callbacks cb 0x55eabb4d8010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa1467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb5862e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa1477fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb5813b0 0x55eabb58d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb57b850 0x55eabb585da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47598/0 (socket says 172.21.15.49:47598) 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa146ffd640 1 -- 172.21.15.49:0/3432581652 learned_addr learned my addr 172.21.15.49:0/3432581652 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa146ffd640 1 -- 172.21.15.49:0/3432581652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 msgr2=0x55eabb5862e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa146ffd640 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb5862e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa146ffd640 1 -- 172.21.15.49:0/3432581652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb5813b0 msgr2=0x55eabb58d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa146ffd640 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb5813b0 0x55eabb58d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.128+0000 7fa146ffd640 1 -- 172.21.15.49:0/3432581652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eabb4ab100 con 0x55eabb57b850 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa1477fe640 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb5813b0 0x55eabb58d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa146ffd640 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb57b850 0x55eabb585da0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa138006010 tx=0x7fa138004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b050adfc407dd3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa12e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa12e7fc640 1 -- 172.21.15.49:0/3432581652 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138027020 con 0x55eabb57b850 2024-03-18T00:58:13.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa12e7fc640 1 -- 172.21.15.49:0/3432581652 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa138003b80 con 0x55eabb57b850 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa150257740 1 -- 172.21.15.49:0/3432581652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa138004020 con 0x55eabb57b850 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa12e7fc640 1 -- 172.21.15.49:0/3432581652 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa138032050 con 0x55eabb57b850 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa150257740 10 client.17448 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa150257740 1 -- 172.21.15.49:0/3432581652 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eabb1f43f0 con 0x55eabb57b850 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa12e7fc640 1 -- 172.21.15.49:0/3432581652 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa138004020 con 0x55eabb57b850 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa12e7fc640 1 -- 172.21.15.49:0/3432581652 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa13800fc70 con 0x55eabb57b850 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa150257740 10 client.17448 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.129+0000 7fa150257740 10 client.17448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.130+0000 7fa150257740 20 client.17448 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.130+0000 7fa150257740 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55eabb5919e0 0x55eabb593e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.130+0000 7fa150257740 4 client.17448 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.130+0000 7fa150257740 1 -- 172.21.15.49:0/3432581652 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eabb4ed9d0 con 0x55eabb5919e0 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.130+0000 7fa1467fc640 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55eabb5919e0 0x55eabb593e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.130+0000 7fa12e7fc640 10 client.17448 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:13.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.130+0000 7fa1467fc640 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55eabb5919e0 0x55eabb593e80 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=516f4ae27d7ee71a server_cookie=849c7f62ea6e8d8a in_seq=0 out_seq=0 2024-03-18T00:58:13.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa12e7fc640 1 -- 172.21.15.49:0/3432581652 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fa13009be50 con 0x55eabb5919e0 2024-03-18T00:58:13.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa12e7fc640 10 client.17448 handle_command_reply: tid=0 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 client.17448 shutdown 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 -- 172.21.15.49:0/3432581652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb57b850 msgr2=0x55eabb585da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb57b850 0x55eabb585da0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa138006010 tx=0x7fa138004c70 comp rx=0 tx=0).stop 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 -- 172.21.15.49:0/3432581652 shutdown_connections 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55eabb5919e0 0x55eabb593e80 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb5813b0 0x55eabb58d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa12e7fc640 0 client.17448 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57c240 0x55eabb5862e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 --2- 172.21.15.49:0/3432581652 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb57b850 0x55eabb585da0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 -- 172.21.15.49:0/3432581652 >> 172.21.15.49:0/3432581652 conn(0x55eabb576dc0 msgr2=0x55eabb575fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 -- 172.21.15.49:0/3432581652 shutdown_connections 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 1 -- 172.21.15.49:0/3432581652 wait complete. 2024-03-18T00:58:13.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.132+0000 7fa150257740 20 client.17448 trim_cache size 0 max 16384 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa150257740 1 Processor -- start 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa150257740 1 -- start start 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa150257740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5861d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa150257740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 0x55eabb57a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa150257740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 0x55eabb598740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa150257740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa1481a8db0 con 0x55eabb576dc0 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa150257740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa1481aa8f0 con 0x55eabb57c240 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa150257740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1481a9480 con 0x55eabb57b850 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5861d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.150 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5861d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47606/0 (socket says 172.21.15.49:47606) 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa146ffd640 1 -- 172.21.15.49:0/3359021341 learned_addr learned my addr 172.21.15.49:0/3359021341 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa1477fe640 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 0x55eabb598740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.146+0000 7fa1467fc640 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 0x55eabb57a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa146ffd640 1 -- 172.21.15.49:0/3359021341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 msgr2=0x55eabb57a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa146ffd640 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 0x55eabb57a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa146ffd640 1 -- 172.21.15.49:0/3359021341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 msgr2=0x55eabb598740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa146ffd640 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 0x55eabb598740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa146ffd640 1 -- 172.21.15.49:0/3359021341 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eabb2be7d0 con 0x55eabb576dc0 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa1477fe640 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 0x55eabb598740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa146ffd640 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5861d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa138005090 tx=0x7fa138005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af85b8f62a7507c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa13cff9640 1 -- 172.21.15.49:0/3359021341 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138027070 con 0x55eabb576dc0 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa13cff9640 1 -- 172.21.15.49:0/3359021341 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa138004ce0 con 0x55eabb576dc0 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa13cff9640 1 -- 172.21.15.49:0/3359021341 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138003b80 con 0x55eabb576dc0 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa150257740 1 -- 172.21.15.49:0/3359021341 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 msgr2=0x55eabb5861d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.147+0000 7fa150257740 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5861d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa138005090 tx=0x7fa138005d30 comp rx=0 tx=0).stop 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 -- 172.21.15.49:0/3359021341 shutdown_connections 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 0x55eabb598740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 0x55eabb57a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 --2- 172.21.15.49:0/3359021341 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5861d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 -- 172.21.15.49:0/3359021341 >> 172.21.15.49:0/3359021341 conn(0x55eabb57d020 msgr2=0x55eabb57a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 -- 172.21.15.49:0/3359021341 shutdown_connections 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 -- 172.21.15.49:0/3359021341 wait complete. 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 -- start start 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 Processor -- start 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5f3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 0x55eabb5f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 0x55eabb5fa800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa1481aa220 con 0x55eabb576dc0 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa1481a9b50 con 0x55eabb57c240 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa1481a6d90 con 0x55eabb57b850 2024-03-18T00:58:13.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa150257740 10 client.0 _ll_register_callbacks cb 0x7fa1300f6060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa146ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5f3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47620/0 (socket says 172.21.15.49:47620) 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa146ffd640 1 -- 172.21.15.49:0/2871315181 learned_addr learned my addr 172.21.15.49:0/2871315181 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa1467fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 0x55eabb5f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.148+0000 7fa1477fe640 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 0x55eabb5fa800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa146ffd640 1 -- 172.21.15.49:0/2871315181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 msgr2=0x55eabb5f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa146ffd640 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 0x55eabb5f37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa146ffd640 1 -- 172.21.15.49:0/2871315181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 msgr2=0x55eabb5fa800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa146ffd640 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 0x55eabb5fa800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa146ffd640 1 -- 172.21.15.49:0/2871315181 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eabb5fadf0 con 0x55eabb576dc0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa146ffd640 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5f3280 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa138006010 tx=0x7fa138009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335630ccf1e3ba37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa12e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa12e7fc640 1 -- 172.21.15.49:0/2871315181 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138027040 con 0x55eabb576dc0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa12e7fc640 1 -- 172.21.15.49:0/2871315181 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa138002030 con 0x55eabb576dc0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa150257740 1 -- 172.21.15.49:0/2871315181 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa138004ce0 con 0x55eabb576dc0 2024-03-18T00:58:13.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa12e7fc640 1 -- 172.21.15.49:0/2871315181 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa138046050 con 0x55eabb576dc0 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa150257740 10 client.17454 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa150257740 1 -- 172.21.15.49:0/2871315181 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eabb5fc0f0 con 0x55eabb576dc0 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa12e7fc640 1 -- 172.21.15.49:0/2871315181 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa138004ce0 con 0x55eabb576dc0 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa12e7fc640 1 -- 172.21.15.49:0/2871315181 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa138003b80 con 0x55eabb576dc0 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa150257740 10 client.17454 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa150257740 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.149+0000 7fa150257740 20 client.17454 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.150+0000 7fa150257740 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55eabb5813b0 0x55eabb5fe9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.150+0000 7fa150257740 4 client.17454 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.150+0000 7fa150257740 1 -- 172.21.15.49:0/2871315181 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55eabb4e7d20 con 0x55eabb5813b0 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.150+0000 7fa1467fc640 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55eabb5813b0 0x55eabb5fe9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.150+0000 7fa1467fc640 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55eabb5813b0 0x55eabb5fe9e0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0178af2b2eba789 server_cookie=37b74180617e6d18 in_seq=0 out_seq=0 2024-03-18T00:58:13.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.150+0000 7fa12e7fc640 10 client.17454 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:13.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa12e7fc640 1 -- 172.21.15.49:0/2871315181 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55eabb4e7d20 con 0x55eabb5813b0 2024-03-18T00:58:13.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa12e7fc640 10 client.17454 handle_command_reply: tid=0 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 client.17454 shutdown 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 -- 172.21.15.49:0/2871315181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 msgr2=0x55eabb5f3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5f3280 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa138006010 tx=0x7fa138009610 comp rx=0 tx=0).stop 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 -- 172.21.15.49:0/2871315181 shutdown_connections 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55eabb5813b0 0x55eabb5fe9e0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55eabb57c240 0x55eabb5fa800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55eabb57b850 0x55eabb5f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 --2- 172.21.15.49:0/2871315181 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55eabb576dc0 0x55eabb5f3280 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa12e7fc640 0 client.17454 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 -- 172.21.15.49:0/2871315181 >> 172.21.15.49:0/2871315181 conn(0x55eabb57d020 msgr2=0x55eabb57a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 -- 172.21.15.49:0/2871315181 shutdown_connections 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 1 -- 172.21.15.49:0/2871315181 wait complete. 2024-03-18T00:58:13.156 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.152+0000 7fa150257740 20 client.17454 trim_cache size 0 max 16384 2024-03-18T00:58:13.157 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:13.157 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 -- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1340a2880 msgr2=0x7fa1340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1340a2880 0x7fa1340a4d40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x55eabb57caa0 tx=0x7fa140005f90 comp rx=0 tx=0).stop 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 -- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 msgr2=0x7fa148198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 0x7fa148198010 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa130009230 tx=0x7fa13002c650 comp rx=0 tx=0).stop 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 -- 172.21.15.49:0/1212697096 shutdown_connections 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa1340a2880 0x7fa1340a4d40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa1481033b0 0x7fa14819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa1481029f0 0x7fa148198010 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 --2- 172.21.15.49:0/1212697096 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa148101800 0x7fa148197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 -- 172.21.15.49:0/1212697096 >> 172.21.15.49:0/1212697096 conn(0x7fa1480fcd70 msgr2=0x7fa148105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 -- 172.21.15.49:0/1212697096 shutdown_connections 2024-03-18T00:58:13.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:13.153+0000 7fa14d66e640 1 -- 172.21.15.49:0/1212697096 wait complete. 2024-03-18T00:58:13.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:13 smithi049 ceph-mon[32130]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 50 MiB/s wr, 83 op/s 2024-03-18T00:58:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:13 smithi107 ceph-mon[36270]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 50 MiB/s wr, 83 op/s 2024-03-18T00:58:13.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:13 smithi106 ceph-mon[35899]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 50 MiB/s wr, 83 op/s 2024-03-18T00:58:14.165 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:58:14.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:58:14.165 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.323+0000 7f3b68436640 1 -- 172.21.15.49:0/3965648745 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b60104bd0 msgr2=0x7f3b60106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.323+0000 7f3b68436640 1 --2- 172.21.15.49:0/3965648745 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b60104bd0 0x7f3b60106fc0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f3b50009900 tx=0x7f3b5002bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.323+0000 7f3b68436640 1 -- 172.21.15.49:0/3965648745 shutdown_connections 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.323+0000 7f3b68436640 1 --2- 172.21.15.49:0/3965648745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b60107500 0x7f3b60109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.323+0000 7f3b68436640 1 --2- 172.21.15.49:0/3965648745 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b60104bd0 0x7f3b60106fc0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.323+0000 7f3b68436640 1 --2- 172.21.15.49:0/3965648745 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b601022a0 0x7f3b60104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.323+0000 7f3b68436640 1 -- 172.21.15.49:0/3965648745 >> 172.21.15.49:0/3965648745 conn(0x7f3b600fbc90 msgr2=0x7f3b600fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 -- 172.21.15.49:0/3965648745 shutdown_connections 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 -- 172.21.15.49:0/3965648745 wait complete. 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 Processor -- start 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 -- start start 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b601022a0 0x7f3b60197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b60104bd0 0x7f3b601981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b60107500 0x7f3b6019f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6010d570 con 0x7f3b601022a0 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6010d3f0 con 0x7f3b60104bd0 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b68436640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6010d6f0 con 0x7f3b60107500 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b661d2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b601022a0 0x7f3b60197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b661d2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b601022a0 0x7f3b60197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47644/0 (socket says 172.21.15.49:47644) 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.324+0000 7f3b661d2640 1 -- 172.21.15.49:0/738679250 learned_addr learned my addr 172.21.15.49:0/738679250 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b661d2640 1 -- 172.21.15.49:0/738679250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b60107500 msgr2=0x7f3b6019f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b661d2640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b60107500 0x7f3b6019f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b661d2640 1 -- 172.21.15.49:0/738679250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b60104bd0 msgr2=0x7f3b601981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b661d2640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b60104bd0 0x7f3b601981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b661d2640 1 -- 172.21.15.49:0/738679250 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b6019f8e0 con 0x7f3b601022a0 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b661d2640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b601022a0 0x7f3b60197ca0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00b4a0 tx=0x7f3b5c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:14.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b577fe640 1 -- 172.21.15.49:0/738679250 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c014070 con 0x7f3b601022a0 2024-03-18T00:58:14.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b577fe640 1 -- 172.21.15.49:0/738679250 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b5c010af0 con 0x7f3b601022a0 2024-03-18T00:58:14.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b68436640 1 -- 172.21.15.49:0/738679250 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b6019fbd0 con 0x7f3b601022a0 2024-03-18T00:58:14.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b577fe640 1 -- 172.21.15.49:0/738679250 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c017d10 con 0x7f3b601022a0 2024-03-18T00:58:14.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.325+0000 7f3b68436640 1 -- 172.21.15.49:0/738679250 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b601a0120 con 0x7f3b601022a0 2024-03-18T00:58:14.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.327+0000 7f3b577fe640 1 -- 172.21.15.49:0/738679250 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3b5c017070 con 0x7f3b601022a0 2024-03-18T00:58:14.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.328+0000 7f3b577fe640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3b4c0a2800 0x7f3b4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.328+0000 7f3b577fe640 1 -- 172.21.15.49:0/738679250 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3b5c0cde60 con 0x7f3b601022a0 2024-03-18T00:58:14.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.328+0000 7f3b659d1640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3b4c0a2800 0x7f3b4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.328+0000 7f3b6b023740 1 Processor -- start 2024-03-18T00:58:14.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.328+0000 7f3b6b023740 1 -- start start 2024-03-18T00:58:14.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b659d1640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3b4c0a2800 0x7f3b4c0a4cc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3b50036040 tx=0x7f3b50000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:14.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc615bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615c2d0 0x559dc615e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 0x559dc6161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b6b023740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559dc60b50d0 con 0x559dc61599a0 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b6b023740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559dc605dc70 con 0x559dc615c2d0 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b6b023740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559dc60b1170 con 0x559dc615ed30 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b661d2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc615bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b661d2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc615bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47656/0 (socket says 172.21.15.49:47656) 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b661d2640 1 -- 172.21.15.49:0/3389415574 learned_addr learned my addr 172.21.15.49:0/3389415574 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b661d2640 1 -- 172.21.15.49:0/3389415574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 msgr2=0x559dc6161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b661d2640 1 --2- 172.21.15.49:0/3389415574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 0x559dc6161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b661d2640 1 -- 172.21.15.49:0/3389415574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615c2d0 msgr2=0x559dc615e6c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b661d2640 1 --2- 172.21.15.49:0/3389415574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615c2d0 0x559dc615e6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b661d2640 1 -- 172.21.15.49:0/3389415574 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dc5e917e0 con 0x559dc61599a0 2024-03-18T00:58:14.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.329+0000 7f3b661d2640 1 --2- 172.21.15.49:0/3389415574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc615bd90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0971b0 tx=0x7f3b5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6242614c5efc760 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:14.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.330+0000 7f3b54bfe640 1 -- 172.21.15.49:0/3389415574 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c01e050 con 0x559dc61599a0 2024-03-18T00:58:14.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.330+0000 7f3b54bfe640 1 -- 172.21.15.49:0/3389415574 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b5c0a2f80 con 0x559dc61599a0 2024-03-18T00:58:14.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.330+0000 7f3b54bfe640 1 -- 172.21.15.49:0/3389415574 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0851f0 con 0x559dc61599a0 2024-03-18T00:58:14.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.330+0000 7f3b6b023740 1 -- 172.21.15.49:0/3389415574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 msgr2=0x559dc615bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:14.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.330+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3389415574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc615bd90 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0971b0 tx=0x7f3b5c0046c0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.330+0000 7f3b6b023740 1 -- 172.21.15.49:0/3389415574 shutdown_connections 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.330+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3389415574 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615c2d0 0x559dc615e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.330+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3389415574 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 0x559dc6161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3389415574 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc615bd90 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 -- 172.21.15.49:0/3389415574 >> 172.21.15.49:0/3389415574 conn(0x559dc6118c50 msgr2=0x559dc614c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 -- 172.21.15.49:0/3389415574 shutdown_connections 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 -- 172.21.15.49:0/3389415574 wait complete. 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 -- start start 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 Processor -- start 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc61599a0 0x559dc61636e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6163c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 0x559dc616ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559dc6161c40 con 0x559dc615c2d0 2024-03-18T00:58:14.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559dc60710d0 con 0x559dc61599a0 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559dc6161dc0 con 0x559dc615ed30 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b6b023740 10 client.0 _ll_register_callbacks cb 0x559dc603a760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.331+0000 7f3b659d1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6163c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b659d1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6163c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47672/0 (socket says 172.21.15.49:47672) 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b659d1640 1 -- 172.21.15.49:0/3244963016 learned_addr learned my addr 172.21.15.49:0/3244963016 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b669d3640 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 0x559dc616ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b659d1640 1 -- 172.21.15.49:0/3244963016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 msgr2=0x559dc616ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b659d1640 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 0x559dc616ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b659d1640 1 -- 172.21.15.49:0/3244963016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc61599a0 msgr2=0x559dc61636e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b659d1640 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc61599a0 0x559dc61636e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b659d1640 1 -- 172.21.15.49:0/3244963016 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dc5ef1f70 con 0x559dc615c2d0 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b669d3640 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 0x559dc616ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b659d1640 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6163c20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006a680 tx=0x7f3b50009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddd922003d2fd89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b49ffb640 1 -- 172.21.15.49:0/3244963016 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5004d020 con 0x559dc615c2d0 2024-03-18T00:58:14.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b49ffb640 1 -- 172.21.15.49:0/3244963016 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3b500048c0 con 0x559dc615c2d0 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.332+0000 7f3b6b023740 1 -- 172.21.15.49:0/3244963016 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b5c0a2f80 con 0x559dc615c2d0 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b49ffb640 1 -- 172.21.15.49:0/3244963016 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b50058050 con 0x559dc615c2d0 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b6b023740 10 client.17466 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b6b023740 1 -- 172.21.15.49:0/3244963016 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559dc5d89170 con 0x559dc615c2d0 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b49ffb640 1 -- 172.21.15.49:0/3244963016 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b5c0a2f80 con 0x559dc615c2d0 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b49ffb640 1 -- 172.21.15.49:0/3244963016 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3b5003f440 con 0x559dc615c2d0 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b6b023740 10 client.17466 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b6b023740 10 client.17466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b6b023740 20 client.17466 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559dc616f200 0x559dc61716c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b6b023740 4 client.17466 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:14.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.333+0000 7f3b6b023740 1 -- 172.21.15.49:0/3244963016 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dc60c2610 con 0x559dc616f200 2024-03-18T00:58:14.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.334+0000 7f3b661d2640 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559dc616f200 0x559dc61716c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.337+0000 7f3b661d2640 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559dc616f200 0x559dc61716c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9be150515b3d577f server_cookie=a8056d1e586a9e35 in_seq=0 out_seq=0 2024-03-18T00:58:14.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.337+0000 7f3b49ffb640 10 client.17466 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:14.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.341+0000 7f3b49ffb640 1 -- 172.21.15.49:0/3244963016 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559dc60c2610 con 0x559dc616f200 2024-03-18T00:58:14.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.341+0000 7f3b49ffb640 10 client.17466 handle_command_reply: tid=0 2024-03-18T00:58:14.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.341+0000 7f3b6b023740 1 client.17466 shutdown 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 -- 172.21.15.49:0/3244963016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 msgr2=0x559dc6163c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6163c20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006a680 tx=0x7f3b50009260 comp rx=0 tx=0).stop 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 -- 172.21.15.49:0/3244963016 shutdown_connections 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559dc616f200 0x559dc61716c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615ed30 0x559dc616ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6163c20 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b49ffb640 0 client.17466 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 --2- 172.21.15.49:0/3244963016 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc61599a0 0x559dc61636e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 -- 172.21.15.49:0/3244963016 >> 172.21.15.49:0/3244963016 conn(0x559dc6118c50 msgr2=0x559dc6159220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 -- 172.21.15.49:0/3244963016 shutdown_connections 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 1 -- 172.21.15.49:0/3244963016 wait complete. 2024-03-18T00:58:14.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.342+0000 7f3b6b023740 20 client.17466 trim_cache size 0 max 16384 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.360+0000 7f3b6b023740 1 Processor -- start 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.360+0000 7f3b6b023740 1 -- start start 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc614ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615c2d0 0x559dc616ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 0x559dc614b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b6b023740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3b601a6e30 con 0x559dc61599a0 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b6b023740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3b601a8970 con 0x559dc615ed30 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b6b023740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3b601a7500 con 0x559dc615c2d0 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b661d2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc614ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b661d2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc614ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47684/0 (socket says 172.21.15.49:47684) 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b661d2640 1 -- 172.21.15.49:0/4136932919 learned_addr learned my addr 172.21.15.49:0/4136932919 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b669d3640 1 --2- 172.21.15.49:0/4136932919 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 0x559dc614b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b659d1640 1 --2- 172.21.15.49:0/4136932919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615c2d0 0x559dc616ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b661d2640 1 -- 172.21.15.49:0/4136932919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615c2d0 msgr2=0x559dc616ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b661d2640 1 --2- 172.21.15.49:0/4136932919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615c2d0 0x559dc616ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b661d2640 1 -- 172.21.15.49:0/4136932919 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 msgr2=0x559dc614b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b661d2640 1 --2- 172.21.15.49:0/4136932919 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 0x559dc614b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b661d2640 1 -- 172.21.15.49:0/4136932919 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dc614cff0 con 0x559dc61599a0 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b661d2640 1 --2- 172.21.15.49:0/4136932919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc614ac50 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00b470 tx=0x7f3b5c09a960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368ec6f855176898 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b4bfff640 1 -- 172.21.15.49:0/4136932919 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c0b7070 con 0x559dc61599a0 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b4bfff640 1 -- 172.21.15.49:0/4136932919 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b5c0a2f80 con 0x559dc61599a0 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.361+0000 7f3b4bfff640 1 -- 172.21.15.49:0/4136932919 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c01e040 con 0x559dc61599a0 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 -- 172.21.15.49:0/4136932919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 msgr2=0x559dc614ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 --2- 172.21.15.49:0/4136932919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc614ac50 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00b470 tx=0x7f3b5c09a960 comp rx=0 tx=0).stop 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 -- 172.21.15.49:0/4136932919 shutdown_connections 2024-03-18T00:58:14.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 --2- 172.21.15.49:0/4136932919 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 0x559dc614b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 --2- 172.21.15.49:0/4136932919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc615c2d0 0x559dc616ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 --2- 172.21.15.49:0/4136932919 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc61599a0 0x559dc614ac50 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 -- 172.21.15.49:0/4136932919 >> 172.21.15.49:0/4136932919 conn(0x559dc6118c50 msgr2=0x559dc615ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:14.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 -- 172.21.15.49:0/4136932919 shutdown_connections 2024-03-18T00:58:14.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 -- 172.21.15.49:0/4136932919 wait complete. 2024-03-18T00:58:14.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.362+0000 7f3b6b023740 1 -- start start 2024-03-18T00:58:14.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b6b023740 1 Processor -- start 2024-03-18T00:58:14.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc61599a0 0x559dc61689b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6168ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b6b023740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 0x559dc6169e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b6b023740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f3b601a82a0 con 0x559dc615c2d0 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b6b023740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f3b601a7bd0 con 0x559dc615ed30 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b6b023740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3b601a4e10 con 0x559dc61599a0 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b6b023740 10 client.0 _ll_register_callbacks cb 0x7f3b50009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b659d1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6168ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b661d2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc61599a0 0x559dc61689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b659d1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6168ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47692/0 (socket says 172.21.15.49:47692) 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b659d1640 1 -- 172.21.15.49:0/2395285481 learned_addr learned my addr 172.21.15.49:0/2395285481 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b669d3640 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 0x559dc6169e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b659d1640 1 -- 172.21.15.49:0/2395285481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc61599a0 msgr2=0x559dc61689b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b659d1640 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc61599a0 0x559dc61689b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b659d1640 1 -- 172.21.15.49:0/2395285481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 msgr2=0x559dc6169e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b659d1640 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 0x559dc6169e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:14.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b659d1640 1 -- 172.21.15.49:0/2395285481 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dc616a480 con 0x559dc615c2d0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b659d1640 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6168ef0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3b50006010 tx=0x7f3b5003f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d11e716706ea5a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.363+0000 7f3b49ffb640 1 -- 172.21.15.49:0/2395285481 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5004d040 con 0x559dc615c2d0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b49ffb640 1 -- 172.21.15.49:0/2395285481 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3b500046c0 con 0x559dc615c2d0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b6b023740 1 -- 172.21.15.49:0/2395285481 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b5c0a2f80 con 0x559dc615c2d0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b49ffb640 1 -- 172.21.15.49:0/2395285481 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3b500048c0 con 0x559dc615c2d0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b6b023740 10 client.17472 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b6b023740 1 -- 172.21.15.49:0/2395285481 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559dc61d5cc0 con 0x559dc615c2d0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b49ffb640 1 -- 172.21.15.49:0/2395285481 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b5c0a2f80 con 0x559dc615c2d0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b49ffb640 1 -- 172.21.15.49:0/2395285481 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3b50003d00 con 0x559dc615c2d0 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b6b023740 10 client.17472 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b6b023740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b6b023740 20 client.17472 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b6b023740 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559dc61d6660 0x559dc61d8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:14.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b6b023740 4 client.17472 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:58:14.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b6b023740 1 -- 172.21.15.49:0/2395285481 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559dc61634d0 con 0x559dc61d6660 2024-03-18T00:58:14.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.364+0000 7f3b661d2640 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559dc61d6660 0x559dc61d8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:14.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.365+0000 7f3b49ffb640 10 client.17472 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:14.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:14.365+0000 7f3b661d2640 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559dc61d6660 0x559dc61d8a80 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=496264c99005909e server_cookie=542f865027792013 in_seq=0 out_seq=0 2024-03-18T00:58:15.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b49ffb640 1 -- 172.21.15.49:0/2395285481 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559dc61634d0 con 0x559dc61d6660 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b49ffb640 10 client.17472 handle_command_reply: tid=0 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 client.17472 shutdown 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 -- 172.21.15.49:0/2395285481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 msgr2=0x559dc6168ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6168ef0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3b50006010 tx=0x7f3b5003f670 comp rx=0 tx=0).stop 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 -- 172.21.15.49:0/2395285481 shutdown_connections 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x559dc61d6660 0x559dc61d8a80 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559dc615ed30 0x559dc6169e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559dc615c2d0 0x559dc6168ef0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b49ffb640 0 client.17472 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 --2- 172.21.15.49:0/2395285481 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559dc61599a0 0x559dc61689b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 -- 172.21.15.49:0/2395285481 >> 172.21.15.49:0/2395285481 conn(0x559dc6118c50 msgr2=0x559dc6158990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:15.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.848+0000 7f3b6b023740 1 -- 172.21.15.49:0/2395285481 shutdown_connections 2024-03-18T00:58:15.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b6b023740 1 -- 172.21.15.49:0/2395285481 wait complete. 2024-03-18T00:58:15.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b6b023740 20 client.17472 trim_cache size 0 max 16384 2024-03-18T00:58:15.850 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:58:15.850 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:58:15.850 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "2940e63b-8acd-4f56-b531-7067c34f2518", 2024-03-18T00:58:15.850 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:58:15.850 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:58:15.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 -- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3b4c0a2800 msgr2=0x7f3b4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:15.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3b4c0a2800 0x7f3b4c0a4cc0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f3b50036040 tx=0x7f3b50000bc0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 -- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b601022a0 msgr2=0x7f3b60197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:15.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b601022a0 0x7f3b60197ca0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c00b4a0 tx=0x7f3b5c00b960 comp rx=0 tx=0).stop 2024-03-18T00:58:15.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 -- 172.21.15.49:0/738679250 shutdown_connections 2024-03-18T00:58:15.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f3b4c0a2800 0x7f3b4c0a4cc0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f3b60104bd0 0x7f3b601981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3b60107500 0x7f3b6019f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 --2- 172.21.15.49:0/738679250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f3b601022a0 0x7f3b60197ca0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 -- 172.21.15.49:0/738679250 >> 172.21.15.49:0/738679250 conn(0x7f3b600fbc90 msgr2=0x7f3b60102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:15.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 -- 172.21.15.49:0/738679250 shutdown_connections 2024-03-18T00:58:15.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.849+0000 7f3b68436640 1 -- 172.21.15.49:0/738679250 wait complete. 2024-03-18T00:58:15.861 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:15 smithi049 ceph-mon[32130]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 29 MiB/s wr, 57 op/s 2024-03-18T00:58:15.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:58:15.863 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:58:15.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:15 smithi107 ceph-mon[36270]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 29 MiB/s wr, 57 op/s 2024-03-18T00:58:15.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:15 smithi106 ceph-mon[35899]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 29 MiB/s wr, 57 op/s 2024-03-18T00:58:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.982+0000 7f441f297640 1 -- 172.21.15.49:0/1330479246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44181006e0 msgr2=0x7f4418100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.982+0000 7f441f297640 1 --2- 172.21.15.49:0/1330479246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44181006e0 0x7f4418100b00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4414009860 tx=0x7f441402bb70 comp rx=0 tx=0).stop 2024-03-18T00:58:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.982+0000 7f441f297640 1 -- 172.21.15.49:0/1330479246 shutdown_connections 2024-03-18T00:58:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.982+0000 7f441f297640 1 --2- 172.21.15.49:0/1330479246 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4418102290 0x7f4418108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.982+0000 7f441f297640 1 --2- 172.21.15.49:0/1330479246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44181018d0 0x7f4418101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.982+0000 7f441f297640 1 --2- 172.21.15.49:0/1330479246 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44181006e0 0x7f4418100b00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.982+0000 7f441f297640 1 -- 172.21.15.49:0/1330479246 >> 172.21.15.49:0/1330479246 conn(0x7f44180fbc70 msgr2=0x7f44180fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.982+0000 7f441f297640 1 -- 172.21.15.49:0/1330479246 shutdown_connections 2024-03-18T00:58:15.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.982+0000 7f441f297640 1 -- 172.21.15.49:0/1330479246 wait complete. 2024-03-18T00:58:15.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441f297640 1 Processor -- start 2024-03-18T00:58:15.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441f297640 1 -- start start 2024-03-18T00:58:15.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441f297640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44181006e0 0x7f4418195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441f297640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44181018d0 0x7f4418195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441f297640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4418102290 0x7f441819cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441f297640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f441810b260 con 0x7f4418102290 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441f297640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f441810b0e0 con 0x7f44181006e0 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441f297640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f441810b3e0 con 0x7f44181018d0 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441c832640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44181018d0 0x7f4418195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441d834640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4418102290 0x7f441819cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441d033640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44181006e0 0x7f4418195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441c832640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44181018d0 0x7f4418195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:47644/0 (socket says 172.21.15.49:47644) 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441d834640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4418102290 0x7f441819cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47702/0 (socket says 172.21.15.49:47702) 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441c832640 1 -- 172.21.15.49:0/2176592876 learned_addr learned my addr 172.21.15.49:0/2176592876 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441c832640 1 -- 172.21.15.49:0/2176592876 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44181006e0 msgr2=0x7f4418195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441c832640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44181006e0 0x7f4418195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.985 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441c832640 1 -- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4418102290 msgr2=0x7f441819cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441c832640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4418102290 0x7f441819cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441c832640 1 -- 172.21.15.49:0/2176592876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441819d5b0 con 0x7f44181018d0 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441d834640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4418102290 0x7f441819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.983+0000 7f441d033640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44181006e0 0x7f4418195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.984+0000 7f441c832640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44181018d0 0x7f4418195ed0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f440800e9f0 tx=0x7f440800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.984+0000 7f440e7fc640 1 -- 172.21.15.49:0/2176592876 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440800cd80 con 0x7f44181018d0 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.984+0000 7f441f297640 1 -- 172.21.15.49:0/2176592876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f441819d8a0 con 0x7f44181018d0 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.984+0000 7f440e7fc640 1 -- 172.21.15.49:0/2176592876 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4408005500 con 0x7f44181018d0 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.984+0000 7f440e7fc640 1 -- 172.21.15.49:0/2176592876 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408018630 con 0x7f44181018d0 2024-03-18T00:58:15.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.984+0000 7f441f297640 1 -- 172.21.15.49:0/2176592876 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f441819dda0 con 0x7f44181018d0 2024-03-18T00:58:15.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.986+0000 7f4421e83740 1 Processor -- start 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.986+0000 7f440e7fc640 1 -- 172.21.15.49:0/2176592876 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f44080046c0 con 0x7f44181018d0 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.986+0000 7f4421e83740 1 -- start start 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f4421e83740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9170 0x5578965a9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f4421e83740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 0x5578965ae660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f4421e83740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 0x5578965b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f4421e83740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55789650e500 con 0x5578965a9170 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f4421e83740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5578964b6be0 con 0x5578965a9b60 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f4421e83740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55789650a5a0 con 0x5578965aecd0 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d834640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9170 0x5578965a9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d834640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9170 0x5578965a9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47712/0 (socket says 172.21.15.49:47712) 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d834640 1 -- 172.21.15.49:0/719372186 learned_addr learned my addr 172.21.15.49:0/719372186 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d033640 1 --2- 172.21.15.49:0/719372186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 0x5578965ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441c832640 1 --2- 172.21.15.49:0/719372186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 0x5578965b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d834640 1 -- 172.21.15.49:0/719372186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 msgr2=0x5578965b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d834640 1 --2- 172.21.15.49:0/719372186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 0x5578965b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d834640 1 -- 172.21.15.49:0/719372186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 msgr2=0x5578965ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d834640 1 --2- 172.21.15.49:0/719372186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 0x5578965ae660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d834640 1 -- 172.21.15.49:0/719372186 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578964ca4b0 con 0x5578965a9170 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f440e7fc640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44040a4a10 0x7f44040a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d834640 1 --2- 172.21.15.49:0/719372186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9170 0x5578965a9590 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f4410009fd0 tx=0x7f441000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ec26558be0ff30f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f441d033640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44040a4a10 0x7f44040a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f440e7fc640 1 -- 172.21.15.49:0/2176592876 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f44080cef70 con 0x7f44181018d0 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.987+0000 7f44037fe640 1 -- 172.21.15.49:0/719372186 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441001d070 con 0x5578965a9170 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f44037fe640 1 -- 172.21.15.49:0/719372186 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4410003e50 con 0x5578965a9170 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f44037fe640 1 -- 172.21.15.49:0/719372186 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44100183f0 con 0x5578965a9170 2024-03-18T00:58:15.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f4421e83740 1 -- 172.21.15.49:0/719372186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9170 msgr2=0x5578965a9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f441d033640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44040a4a10 0x7f44040a6ed0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4414009830 tx=0x7f441402e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f4421e83740 1 --2- 172.21.15.49:0/719372186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9170 0x5578965a9590 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f4410009fd0 tx=0x7f441000ec80 comp rx=0 tx=0).stop 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f4421e83740 1 -- 172.21.15.49:0/719372186 shutdown_connections 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f4421e83740 1 --2- 172.21.15.49:0/719372186 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 0x5578965ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f4421e83740 1 --2- 172.21.15.49:0/719372186 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 0x5578965b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f4421e83740 1 --2- 172.21.15.49:0/719372186 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9170 0x5578965a9590 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f4421e83740 1 -- 172.21.15.49:0/719372186 >> 172.21.15.49:0/719372186 conn(0x5578965a46e0 msgr2=0x5578965a4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f4421e83740 1 -- 172.21.15.49:0/719372186 shutdown_connections 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.988+0000 7f4421e83740 1 -- 172.21.15.49:0/719372186 wait complete. 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f4421e83740 1 -- start start 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f4421e83740 1 Processor -- start 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f4421e83740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 0x5578965b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f4421e83740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9b60 0x5578965b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f4421e83740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965bacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f4421e83740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5578965b1c80 con 0x5578965aecd0 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f4421e83740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5578964ca160 con 0x5578965a9170 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f4421e83740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5578965b1e00 con 0x5578965a9b60 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f4421e83740 10 client.0 _ll_register_callbacks cb 0x5578964ae150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f441d834640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965bacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f441d033640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 0x5578965b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f441c832640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9b60 0x5578965b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f441d033640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 0x5578965b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:39838/0 (socket says 172.21.15.49:39838) 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.989+0000 7f441d033640 1 -- 172.21.15.49:0/148212517 learned_addr learned my addr 172.21.15.49:0/148212517 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f441c832640 1 -- 172.21.15.49:0/148212517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 msgr2=0x5578965b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f441c832640 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 0x5578965b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f441c832640 1 -- 172.21.15.49:0/148212517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 msgr2=0x5578965bacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f441c832640 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965bacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f441c832640 1 -- 172.21.15.49:0/148212517 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578962ec7d0 con 0x5578965a9b60 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f441c832640 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9b60 0x5578965b3c80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f44080ccaa0 tx=0x7f4408009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f42fcaff0acd53f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4400ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:58:15.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4400ff9640 1 -- 172.21.15.49:0/148212517 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440809b430 con 0x5578965a9b60 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4400ff9640 1 -- 172.21.15.49:0/148212517 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f44080860f0 con 0x5578965a9b60 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4421e83740 1 -- 172.21.15.49:0/148212517 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4410003e50 con 0x5578965a9b60 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4400ff9640 1 -- 172.21.15.49:0/148212517 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f44080b4050 con 0x5578965a9b60 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4421e83740 10 client.26716 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4421e83740 1 -- 172.21.15.49:0/148212517 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5578962223f0 con 0x5578965a9b60 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4400ff9640 1 -- 172.21.15.49:0/148212517 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4410003e50 con 0x5578965a9b60 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4400ff9640 1 -- 172.21.15.49:0/148212517 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f44080a3c00 con 0x5578965a9b60 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4421e83740 10 client.26716 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4421e83740 10 client.26716 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4421e83740 20 client.26716 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.990+0000 7f4421e83740 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5578965bf260 0x5578965c1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.991+0000 7f4421e83740 4 client.26716 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.991+0000 7f4421e83740 1 -- 172.21.15.49:0/148212517 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55789651ba40 con 0x5578965bf260 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.991+0000 7f441d033640 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5578965bf260 0x5578965c1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:15.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.991+0000 7f441d033640 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5578965bf260 0x5578965c1670 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82cdb7d6148bf793 server_cookie=f42ab3426e16a550 in_seq=0 out_seq=0 2024-03-18T00:58:15.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.991+0000 7f4400ff9640 10 client.26716 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4400ff9640 1 -- 172.21.15.49:0/148212517 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55789651ba40 con 0x5578965bf260 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4400ff9640 10 client.26716 handle_command_reply: tid=0 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 client.26716 shutdown 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 -- 172.21.15.49:0/148212517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9b60 msgr2=0x5578965b3c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9b60 0x5578965b3c80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f44080ccaa0 tx=0x7f4408009720 comp rx=0 tx=0).stop 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 -- 172.21.15.49:0/148212517 shutdown_connections 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5578965bf260 0x5578965c1670 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965bacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9b60 0x5578965b3c80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4400ff9640 0 client.26716 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 --2- 172.21.15.49:0/148212517 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 0x5578965b3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 -- 172.21.15.49:0/148212517 >> 172.21.15.49:0/148212517 conn(0x5578965a46e0 msgr2=0x5578965a3900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:15.998 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 -- 172.21.15.49:0/148212517 shutdown_connections 2024-03-18T00:58:15.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 1 -- 172.21.15.49:0/148212517 wait complete. 2024-03-18T00:58:15.999 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:15.994+0000 7f4421e83740 20 client.26716 trim_cache size 0 max 16384 2024-03-18T00:58:16.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f4421e83740 1 Processor -- start 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f4421e83740 1 -- start start 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f4421e83740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 0x5578965aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f4421e83740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9b60 0x5578965bac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f4421e83740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 0x5578965a3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f4421e83740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f44181a6cc0 con 0x5578965a9b60 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f4421e83740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f44181a8800 con 0x5578965a9170 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f4421e83740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f44181a7390 con 0x5578965aecd0 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f441c832640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9b60 0x5578965bac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f441c832640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9b60 0x5578965bac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47732/0 (socket says 172.21.15.49:47732) 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f441c832640 1 -- 172.21.15.49:0/225633147 learned_addr learned my addr 172.21.15.49:0/225633147 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f441d033640 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 0x5578965aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.013+0000 7f441d834640 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 0x5578965a3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f441c832640 1 -- 172.21.15.49:0/225633147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 msgr2=0x5578965a3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:16.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f441c832640 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 0x5578965a3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f441c832640 1 -- 172.21.15.49:0/225633147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 msgr2=0x5578965aa3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f441c832640 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 0x5578965aa3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f441c832640 1 -- 172.21.15.49:0/225633147 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578961e20f0 con 0x5578965a9b60 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f441d834640 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 0x5578965a3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f441c832640 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9b60 0x5578965bac80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f44080049d0 tx=0x7f4408099e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82110906d08d4056 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4402ffd640 1 -- 172.21.15.49:0/225633147 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440809b250 con 0x5578965a9b60 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4402ffd640 1 -- 172.21.15.49:0/225633147 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f440809bff0 con 0x5578965a9b60 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4402ffd640 1 -- 172.21.15.49:0/225633147 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440809fa40 con 0x5578965a9b60 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4421e83740 1 -- 172.21.15.49:0/225633147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9b60 msgr2=0x5578965bac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4421e83740 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9b60 0x5578965bac80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f44080049d0 tx=0x7f4408099e00 comp rx=0 tx=0).stop 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4421e83740 1 -- 172.21.15.49:0/225633147 shutdown_connections 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4421e83740 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965aecd0 0x5578965a3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4421e83740 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965a9b60 0x5578965bac80 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4421e83740 1 --2- 172.21.15.49:0/225633147 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9170 0x5578965aa3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.019 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.014+0000 7f4421e83740 1 -- 172.21.15.49:0/225633147 >> 172.21.15.49:0/225633147 conn(0x5578965a46e0 msgr2=0x5578965b21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 -- 172.21.15.49:0/225633147 shutdown_connections 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 -- 172.21.15.49:0/225633147 wait complete. 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 -- start start 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 Processor -- start 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9170 0x5578965b8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 0x5578965b8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965b99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f44181a8130 con 0x5578965aecd0 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f44181a7a60 con 0x5578965a9b60 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f44181a4ca0 con 0x5578965a9170 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f4421e83740 10 client.0 _ll_register_callbacks cb 0x7f440800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f441d834640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965b99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f441d834640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965b99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:47742/0 (socket says 172.21.15.49:47742) 2024-03-18T00:58:16.020 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f441c832640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 0x5578965b8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f441d834640 1 -- 172.21.15.49:0/4158315180 learned_addr learned my addr 172.21.15.49:0/4158315180 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.015+0000 7f441d033640 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9170 0x5578965b8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f441d834640 1 -- 172.21.15.49:0/4158315180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9170 msgr2=0x5578965b8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f441d834640 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9170 0x5578965b8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f441d834640 1 -- 172.21.15.49:0/4158315180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 msgr2=0x5578965b8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f441d834640 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 0x5578965b8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f441d834640 1 -- 172.21.15.49:0/4158315180 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578964d59d0 con 0x5578965aecd0 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f441d834640 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965b99a0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4410009fd0 tx=0x7f4410003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8426cffc0fe52ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f4400ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f4400ff9640 1 -- 172.21.15.49:0/4158315180 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441001d030 con 0x5578965aecd0 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f4400ff9640 1 -- 172.21.15.49:0/4158315180 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f44100052b0 con 0x5578965aecd0 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f4421e83740 1 -- 172.21.15.49:0/4158315180 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f440809bff0 con 0x5578965aecd0 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f4400ff9640 1 -- 172.21.15.49:0/4158315180 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4410005bd0 con 0x5578965aecd0 2024-03-18T00:58:16.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f4421e83740 10 client.17490 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f4421e83740 1 -- 172.21.15.49:0/4158315180 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557896625570 con 0x5578965aecd0 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.016+0000 7f4400ff9640 1 -- 172.21.15.49:0/4158315180 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f440809bff0 con 0x5578965aecd0 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f4400ff9640 1 -- 172.21.15.49:0/4158315180 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f441000b040 con 0x5578965aecd0 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f4421e83740 10 client.17490 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f4421e83740 10 client.17490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f4421e83740 20 client.17490 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f4421e83740 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557896625b20 0x557896627f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f4421e83740 4 client.17490 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f4421e83740 1 -- 172.21.15.49:0/4158315180 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557896515d90 con 0x557896625b20 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f441d033640 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557896625b20 0x557896627f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f4400ff9640 10 client.17490 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:16.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.017+0000 7f441d033640 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557896625b20 0x557896627f10 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efa924c954bb86b7 server_cookie=99cbd427c28083c in_seq=0 out_seq=0 2024-03-18T00:58:16.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.019+0000 7f4400ff9640 1 -- 172.21.15.49:0/4158315180 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x557896515d90 con 0x557896625b20 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.019+0000 7f4400ff9640 10 client.17490 handle_command_reply: tid=0 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.019+0000 7f4421e83740 1 client.17490 shutdown 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.019+0000 7f4421e83740 1 -- 172.21.15.49:0/4158315180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 msgr2=0x5578965b99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.019+0000 7f4421e83740 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965b99a0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4410009fd0 tx=0x7f4410003ba0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.019+0000 7f4421e83740 1 -- 172.21.15.49:0/4158315180 shutdown_connections 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.019+0000 7f4421e83740 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557896625b20 0x557896627f10 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.019+0000 7f4421e83740 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578965a9b60 0x5578965b8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.019+0000 7f4400ff9640 0 client.17490 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f4421e83740 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578965aecd0 0x5578965b99a0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f4421e83740 1 --2- 172.21.15.49:0/4158315180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578965a9170 0x5578965b8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f4421e83740 1 -- 172.21.15.49:0/4158315180 >> 172.21.15.49:0/4158315180 conn(0x5578965a46e0 msgr2=0x5578965a7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f4421e83740 1 -- 172.21.15.49:0/4158315180 shutdown_connections 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f4421e83740 1 -- 172.21.15.49:0/4158315180 wait complete. 2024-03-18T00:58:16.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f4421e83740 20 client.17490 trim_cache size 0 max 16384 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (3 inodes in the stack)", 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout: "2940e63b-8acd-4f56-b531-7067c34f2518": { 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "2940e63b-8acd-4f56-b531-7067c34f2518", 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:58:16.025 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:58:16.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 -- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44040a4a10 msgr2=0x7f44040a6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:16.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44040a4a10 0x7f44040a6ed0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4414009830 tx=0x7f441402e040 comp rx=0 tx=0).stop 2024-03-18T00:58:16.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 -- 172.21.15.49:0/2176592876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44181018d0 msgr2=0x7f4418195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:16.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44181018d0 0x7f4418195ed0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f440800e9f0 tx=0x7f440800edf0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 -- 172.21.15.49:0/2176592876 shutdown_connections 2024-03-18T00:58:16.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f44040a4a10 0x7f44040a6ed0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4418102290 0x7f441819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f44181018d0 0x7f4418195ed0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 --2- 172.21.15.49:0/2176592876 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f44181006e0 0x7f4418195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:16.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 -- 172.21.15.49:0/2176592876 >> 172.21.15.49:0/2176592876 conn(0x7f44180fbc70 msgr2=0x7f4418104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:16.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 -- 172.21.15.49:0/2176592876 shutdown_connections 2024-03-18T00:58:16.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:16.020+0000 7f441f297640 1 -- 172.21.15.49:0/2176592876 wait complete. 2024-03-18T00:58:16.031 INFO:tasks.cephfs.filesystem:scrub status for tag:2940e63b-8acd-4f56-b531-7067c34f2518 - {'path': '/', 'tag': '2940e63b-8acd-4f56-b531-7067c34f2518', 'options': 'recursive,force'} 2024-03-18T00:58:16.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:16 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:58:16.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:16 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:16.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:16 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:58:16.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:16 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:58:16.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:16 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:16.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:16 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:58:16.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:16 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:58:16.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:16 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:16.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:16 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:58:17.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:17 smithi049 ceph-mon[32130]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 27 MiB/s wr, 96 op/s 2024-03-18T00:58:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:17 smithi107 ceph-mon[36270]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 27 MiB/s wr, 96 op/s 2024-03-18T00:58:17.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:17 smithi106 ceph-mon[35899]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 27 MiB/s wr, 96 op/s 2024-03-18T00:58:18.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:18 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:58:18.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:18 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:18.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:18 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:58:18.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:18 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:18.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:18 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:58:18.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:18 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:19.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:19 smithi049 ceph-mon[32130]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 45 MiB/s wr, 185 op/s 2024-03-18T00:58:19.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:19 smithi107 ceph-mon[36270]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 45 MiB/s wr, 185 op/s 2024-03-18T00:58:19.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:19 smithi106 ceph-mon[35899]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 45 MiB/s wr, 185 op/s 2024-03-18T00:58:21.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:21 smithi049 ceph-mon[32130]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 52 MiB/s wr, 224 op/s 2024-03-18T00:58:21.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:21 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:21.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:21 smithi107 ceph-mon[36270]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 52 MiB/s wr, 224 op/s 2024-03-18T00:58:21.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:21 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:21.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:21 smithi106 ceph-mon[35899]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 52 MiB/s wr, 224 op/s 2024-03-18T00:58:21.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:21 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:23.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:23 smithi049 ceph-mon[32130]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 74 MiB/s wr, 330 op/s 2024-03-18T00:58:23.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:23 smithi107 ceph-mon[36270]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 74 MiB/s wr, 330 op/s 2024-03-18T00:58:23.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:23 smithi106 ceph-mon[35899]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 74 MiB/s wr, 330 op/s 2024-03-18T00:58:25.088 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.085+0000 7f3422668640 1 -- 172.21.15.49:0/1463058067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f341c102a20 msgr2=0x7f341c102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:25.088 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.085+0000 7f3422668640 1 --2- 172.21.15.49:0/1463058067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f341c102a20 0x7f341c102ea0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3410009900 tx=0x7f341002bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:25.088 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.085+0000 7f3422668640 1 -- 172.21.15.49:0/1463058067 shutdown_connections 2024-03-18T00:58:25.088 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.085+0000 7f3422668640 1 --2- 172.21.15.49:0/1463058067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f341c1033e0 0x7f341c109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.085+0000 7f3422668640 1 --2- 172.21.15.49:0/1463058067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f341c102a20 0x7f341c102ea0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.085+0000 7f3422668640 1 --2- 172.21.15.49:0/1463058067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f341c101830 0x7f341c101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.085+0000 7f3422668640 1 -- 172.21.15.49:0/1463058067 >> 172.21.15.49:0/1463058067 conn(0x7f341c0fcda0 msgr2=0x7f341c0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.085+0000 7f3422668640 1 -- 172.21.15.49:0/1463058067 shutdown_connections 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.085+0000 7f3422668640 1 -- 172.21.15.49:0/1463058067 wait complete. 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f3422668640 1 Processor -- start 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f3422668640 1 -- start start 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f3422668640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f341c101830 0x7f341c199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f3422668640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f341c102a20 0x7f341c19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f3422668640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f341c1033e0 0x7f341c1a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f3422668640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f341c10d4b0 con 0x7f341c101830 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f3422668640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f341c10d330 con 0x7f341c1033e0 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f3422668640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f341c10d630 con 0x7f341c102a20 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f341bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f341c101830 0x7f341c199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f341bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f341c101830 0x7f341c199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37588/0 (socket says 172.21.15.49:37588) 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f341bfff640 1 -- 172.21.15.49:0/3900560998 learned_addr learned my addr 172.21.15.49:0/3900560998 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f3420c05640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f341c1033e0 0x7f341c1a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.086+0000 7f341b7fe640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f341c102a20 0x7f341c19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f341bfff640 1 -- 172.21.15.49:0/3900560998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f341c102a20 msgr2=0x7f341c19a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f341bfff640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f341c102a20 0x7f341c19a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f341bfff640 1 -- 172.21.15.49:0/3900560998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f341c1033e0 msgr2=0x7f341c1a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f341bfff640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f341c1033e0 0x7f341c1a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f341bfff640 1 -- 172.21.15.49:0/3900560998 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c1a18d0 con 0x7f341c101830 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f3420c05640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f341c1033e0 0x7f341c1a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f341bfff640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f341c101830 0x7f341c199c90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f340c00e9c0 tx=0x7f340c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f34197fa640 1 -- 172.21.15.49:0/3900560998 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c00cd80 con 0x7f341c101830 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f3422668640 1 -- 172.21.15.49:0/3900560998 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341c1a1bc0 con 0x7f341c101830 2024-03-18T00:58:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f3422668640 1 -- 172.21.15.49:0/3900560998 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f341c1a2130 con 0x7f341c101830 2024-03-18T00:58:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f34197fa640 1 -- 172.21.15.49:0/3900560998 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f340c0035f0 con 0x7f341c101830 2024-03-18T00:58:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.087+0000 7f34197fa640 1 -- 172.21.15.49:0/3900560998 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c010690 con 0x7f341c101830 2024-03-18T00:58:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.089+0000 7f34197fa640 1 -- 172.21.15.49:0/3900560998 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f340c004020 con 0x7f341c101830 2024-03-18T00:58:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f342524f740 1 Processor -- start 2024-03-18T00:58:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f342524f740 1 -- start start 2024-03-18T00:58:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f34197fa640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34040a28a0 0x7f34040a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f342524f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.096 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f342524f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f342524f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32cdea0 0x563db32d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f342524f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563db322da40 con 0x563db32c8340 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f342524f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563db31d6920 con 0x563db32cdea0 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f342524f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563db3229ae0 con 0x563db32c8d30 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f3420c05640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34040a28a0 0x7f34040a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f341b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f34197fa640 1 -- 172.21.15.49:0/3900560998 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f340c0cedd0 con 0x7f341c101830 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f341b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:35736/0 (socket says 172.21.15.49:35736) 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f341bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f341b7fe640 1 -- 172.21.15.49:0/2665843731 learned_addr learned my addr 172.21.15.49:0/2665843731 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f3420c05640 1 --2- 172.21.15.49:0/2665843731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32cdea0 0x563db32d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.097 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f341b7fe640 1 -- 172.21.15.49:0/2665843731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32cdea0 msgr2=0x563db32d0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f341b7fe640 1 --2- 172.21.15.49:0/2665843731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32cdea0 0x563db32d0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f341b7fe640 1 -- 172.21.15.49:0/2665843731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 msgr2=0x563db32c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f341b7fe640 1 --2- 172.21.15.49:0/2665843731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.090+0000 7f341b7fe640 1 -- 172.21.15.49:0/2665843731 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563db31ee460 con 0x563db32c8d30 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.091+0000 7f3420c05640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34040a28a0 0x7f34040a4d60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f34140099e0 tx=0x7f3414013660 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.091+0000 7f341b7fe640 1 --2- 172.21.15.49:0/2665843731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32cd830 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3410009900 tx=0x7f3410009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5ec8c52a2318f25 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.091+0000 7f34027fc640 1 -- 172.21.15.49:0/2665843731 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341002cb80 con 0x563db32c8d30 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.091+0000 7f34027fc640 1 -- 172.21.15.49:0/2665843731 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3410005040 con 0x563db32c8d30 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.091+0000 7f34027fc640 1 -- 172.21.15.49:0/2665843731 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410004920 con 0x563db32c8d30 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.091+0000 7f342524f740 1 -- 172.21.15.49:0/2665843731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 msgr2=0x563db32cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.091+0000 7f342524f740 1 --2- 172.21.15.49:0/2665843731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32cd830 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3410009900 tx=0x7f3410009560 comp rx=0 tx=0).stop 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 -- 172.21.15.49:0/2665843731 shutdown_connections 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 --2- 172.21.15.49:0/2665843731 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32cdea0 0x563db32d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 --2- 172.21.15.49:0/2665843731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32cd830 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 --2- 172.21.15.49:0/2665843731 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 -- 172.21.15.49:0/2665843731 >> 172.21.15.49:0/2665843731 conn(0x563db32c38b0 msgr2=0x563db32c3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 -- 172.21.15.49:0/2665843731 shutdown_connections 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 -- 172.21.15.49:0/2665843731 wait complete. 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 -- start start 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 Processor -- start 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c8340 0x563db32d25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8d30 0x563db32d2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32cdea0 0x563db32d9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563db32d0df0 con 0x563db32c8d30 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563db31e9d70 con 0x563db32c8340 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563db32d0f70 con 0x563db32cdea0 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f342524f740 10 client.0 _ll_register_callbacks cb 0x563db2f41a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.092+0000 7f341b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8d30 0x563db32d2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c8340 0x563db32d25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8d30 0x563db32d2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37614/0 (socket says 172.21.15.49:37614) 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341b7fe640 1 -- 172.21.15.49:0/2875734587 learned_addr learned my addr 172.21.15.49:0/2875734587 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f3420c05640 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32cdea0 0x563db32d9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341bfff640 1 -- 172.21.15.49:0/2875734587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32cdea0 msgr2=0x563db32d9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341bfff640 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32cdea0 0x563db32d9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341bfff640 1 -- 172.21.15.49:0/2875734587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8d30 msgr2=0x563db32d2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341bfff640 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8d30 0x563db32d2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341bfff640 1 -- 172.21.15.49:0/2875734587 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563db2f42210 con 0x563db32c8340 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341b7fe640 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8d30 0x563db32d2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f341bfff640 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c8340 0x563db32d25e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f340c0cc900 tx=0x7f340c0b4ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb13e612c2800e2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f33fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2875734587 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c09b750 con 0x563db32c8340 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2875734587 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f340c0b4bd0 con 0x563db32c8340 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f342524f740 1 -- 172.21.15.49:0/2875734587 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3410005040 con 0x563db32c8340 2024-03-18T00:58:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2875734587 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f340c09f8a0 con 0x563db32c8340 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f342524f740 10 client.26642 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.093+0000 7f342524f740 1 -- 172.21.15.49:0/2875734587 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563db300e620 con 0x563db32c8340 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2875734587 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3410005040 con 0x563db32c8340 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2875734587 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f340c084ce0 con 0x563db32c8340 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f342524f740 10 client.26642 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f342524f740 10 client.26642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f342524f740 20 client.26642 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f342524f740 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563db32de460 0x563db32e0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f342524f740 4 client.26642 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f342524f740 1 -- 172.21.15.49:0/2875734587 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563db323af80 con 0x563db32de460 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f341b7fe640 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563db32de460 0x563db32e0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f33fb7fe640 10 client.26642 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.094+0000 7f341b7fe640 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563db32de460 0x563db32e0920 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f82bc94924ac9c72 server_cookie=f7d50152234369b3 in_seq=0 out_seq=0 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2875734587 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563db323af80 con 0x563db32de460 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f33fb7fe640 10 client.26642 handle_command_reply: tid=0 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 client.26642 shutdown 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 -- 172.21.15.49:0/2875734587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c8340 msgr2=0x563db32d25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c8340 0x563db32d25e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f340c0cc900 tx=0x7f340c0b4ab0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 -- 172.21.15.49:0/2875734587 shutdown_connections 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563db32de460 0x563db32e0920 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32cdea0 0x563db32d9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8d30 0x563db32d2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 --2- 172.21.15.49:0/2875734587 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c8340 0x563db32d25e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f33fb7fe640 0 client.26642 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 -- 172.21.15.49:0/2875734587 >> 172.21.15.49:0/2875734587 conn(0x563db32c38b0 msgr2=0x563db32c2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 -- 172.21.15.49:0/2875734587 shutdown_connections 2024-03-18T00:58:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 1 -- 172.21.15.49:0/2875734587 wait complete. 2024-03-18T00:58:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.097+0000 7f342524f740 20 client.26642 trim_cache size 0 max 16384 2024-03-18T00:58:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f342524f740 1 Processor -- start 2024-03-18T00:58:25.117 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f342524f740 1 -- start start 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f342524f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 0x563db32d21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f342524f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f342524f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f342524f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f340800e440 con 0x563db32c8340 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f342524f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f340800ff80 con 0x563db32c38b0 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f342524f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f340800eb10 con 0x563db32c8d30 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37616/0 (socket says 172.21.15.49:37616) 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341b7fe640 1 -- 172.21.15.49:0/3487307032 learned_addr learned my addr 172.21.15.49:0/3487307032 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f3420c05640 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341bfff640 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 0x563db32d21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341b7fe640 1 -- 172.21.15.49:0/3487307032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 msgr2=0x563db32d2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341b7fe640 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.118 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341b7fe640 1 -- 172.21.15.49:0/3487307032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 msgr2=0x563db32d21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341b7fe640 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 0x563db32d21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341b7fe640 1 -- 172.21.15.49:0/3487307032 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563db32c4300 con 0x563db32c8340 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f341bfff640 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 0x563db32d21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.112+0000 7f3420c05640 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f341b7fe640 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9af0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3410005940 tx=0x7f3410000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd369f78e595998 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f3401ffb640 1 -- 172.21.15.49:0/3487307032 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410005610 con 0x563db32c8340 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f3401ffb640 1 -- 172.21.15.49:0/3487307032 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f341002ccb0 con 0x563db32c8340 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f3401ffb640 1 -- 172.21.15.49:0/3487307032 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410003bc0 con 0x563db32c8340 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f342524f740 1 -- 172.21.15.49:0/3487307032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 msgr2=0x563db32d9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f342524f740 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9af0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f3410005940 tx=0x7f3410000bc0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f342524f740 1 -- 172.21.15.49:0/3487307032 shutdown_connections 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f342524f740 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f342524f740 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9af0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f342524f740 1 --2- 172.21.15.49:0/3487307032 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 0x563db32d21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.119 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f342524f740 1 -- 172.21.15.49:0/3487307032 >> 172.21.15.49:0/3487307032 conn(0x563db32cd970 msgr2=0x563db32c6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f342524f740 1 -- 172.21.15.49:0/3487307032 shutdown_connections 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.113+0000 7f342524f740 1 -- 172.21.15.49:0/3487307032 wait complete. 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f342524f740 1 -- start start 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f342524f740 1 Processor -- start 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f342524f740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 0x563db32d9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f342524f740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f342524f740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f342524f740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f340800f8b0 con 0x563db32c8340 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f342524f740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f340800f1e0 con 0x563db32c38b0 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f342524f740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f340800c420 con 0x563db32c8d30 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f342524f740 10 client.0 _ll_register_callbacks cb 0x7f340c09b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f341b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f341b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37626/0 (socket says 172.21.15.49:37626) 2024-03-18T00:58:25.120 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f341b7fe640 1 -- 172.21.15.49:0/2042926606 learned_addr learned my addr 172.21.15.49:0/2042926606 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f3420c05640 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.114+0000 7f341bfff640 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 0x563db32d9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f3420c05640 1 -- 172.21.15.49:0/2042926606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 msgr2=0x563db32d9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f3420c05640 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 0x563db32d9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f3420c05640 1 -- 172.21.15.49:0/2042926606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 msgr2=0x563db32d9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f3420c05640 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f3420c05640 1 -- 172.21.15.49:0/2042926606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563db32d8010 con 0x563db32c8d30 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f341b7fe640 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f3420c05640 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d79a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3414009fd0 tx=0x7f341401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=432afb01e8df9e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f33fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2042926606 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34140047a0 con 0x563db32c8d30 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f342524f740 1 -- 172.21.15.49:0/2042926606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f341002ccb0 con 0x563db32c8d30 2024-03-18T00:58:25.121 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2042926606 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f3414010430 con 0x563db32c8d30 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2042926606 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3414010670 con 0x563db32c8d30 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f342524f740 10 client.26740 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f342524f740 1 -- 172.21.15.49:0/2042926606 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563db33452c0 con 0x563db32c8d30 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.115+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2042926606 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f341002ccb0 con 0x563db32c8d30 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2042926606 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f3414020050 con 0x563db32c8d30 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f342524f740 10 client.26740 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f342524f740 10 client.26740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f342524f740 20 client.26740 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f342524f740 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563db32cdea0 0x563db3347d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f342524f740 4 client.26740 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f342524f740 1 -- 172.21.15.49:0/2042926606 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563db32352d0 con 0x563db32cdea0 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f341bfff640 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563db32cdea0 0x563db3347d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f341bfff640 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563db32cdea0 0x563db3347d10 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23096c8df34670a3 server_cookie=61a0a5caed48386b in_seq=0 out_seq=0 2024-03-18T00:58:25.122 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:25.116+0000 7f33fb7fe640 10 client.26740 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:25.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:25 smithi049 ceph-mon[32130]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 74 MiB/s wr, 378 op/s 2024-03-18T00:58:25.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:25 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:58:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:25 smithi107 ceph-mon[36270]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 74 MiB/s wr, 378 op/s 2024-03-18T00:58:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:25 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:58:25.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:25 smithi106 ceph-mon[35899]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 74 MiB/s wr, 378 op/s 2024-03-18T00:58:25.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:25 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:58:26.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:26 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:58:26.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:26 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:58:26.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:26 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:58:27.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:27 smithi049 ceph-mon[32130]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 80 MiB/s wr, 418 op/s 2024-03-18T00:58:27.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:27 smithi107 ceph-mon[36270]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 80 MiB/s wr, 418 op/s 2024-03-18T00:58:27.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:27 smithi106 ceph-mon[35899]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 80 MiB/s wr, 418 op/s 2024-03-18T00:58:29.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:29 smithi049 ceph-mon[32130]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 88 MiB/s wr, 459 op/s 2024-03-18T00:58:29.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:29 smithi107 ceph-mon[36270]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 88 MiB/s wr, 459 op/s 2024-03-18T00:58:29.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:29 smithi106 ceph-mon[35899]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 88 MiB/s wr, 459 op/s 2024-03-18T00:58:31.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:31 smithi049 ceph-mon[32130]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 77 MiB/s wr, 401 op/s 2024-03-18T00:58:31.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:31 smithi107 ceph-mon[36270]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 77 MiB/s wr, 401 op/s 2024-03-18T00:58:31.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:31 smithi106 ceph-mon[35899]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 77 MiB/s wr, 401 op/s 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.195+0000 7f33fb7fe640 1 -- 172.21.15.49:0/2042926606 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563db32352d0 con 0x563db32cdea0 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.195+0000 7f33fb7fe640 10 client.26740 handle_command_reply: tid=0 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.195+0000 7f342524f740 1 client.26740 shutdown 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.195+0000 7f342524f740 1 -- 172.21.15.49:0/2042926606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 msgr2=0x563db32d79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.195+0000 7f342524f740 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d79a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3414009fd0 tx=0x7f341401d000 comp rx=0 tx=0).stop 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f342524f740 1 -- 172.21.15.49:0/2042926606 shutdown_connections 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f342524f740 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563db32cdea0 0x563db3347d10 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f342524f740 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563db32c8d30 0x563db32d79a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f342524f740 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563db32c8340 0x563db32d9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f33fb7fe640 0 client.26740 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f342524f740 1 --2- 172.21.15.49:0/2042926606 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563db32c38b0 0x563db32d9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f342524f740 1 -- 172.21.15.49:0/2042926606 >> 172.21.15.49:0/2042926606 conn(0x563db32cd970 msgr2=0x563db32d2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:33.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f342524f740 1 -- 172.21.15.49:0/2042926606 shutdown_connections 2024-03-18T00:58:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f342524f740 1 -- 172.21.15.49:0/2042926606 wait complete. 2024-03-18T00:58:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.196+0000 7f342524f740 20 client.26740 trim_cache size 0 max 16384 2024-03-18T00:58:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:58:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:58:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:58:33.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 -- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34040a28a0 msgr2=0x7f34040a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34040a28a0 0x7f34040a4d60 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f34140099e0 tx=0x7f3414013660 comp rx=0 tx=0).stop 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 -- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f341c101830 msgr2=0x7f341c199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f341c101830 0x7f341c199c90 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f340c00e9c0 tx=0x7f340c00ee80 comp rx=0 tx=0).stop 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 -- 172.21.15.49:0/3900560998 shutdown_connections 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f34040a28a0 0x7f34040a4d60 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f341c1033e0 0x7f341c1a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f341c102a20 0x7f341c19a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 --2- 172.21.15.49:0/3900560998 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f341c101830 0x7f341c199c90 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:33.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 -- 172.21.15.49:0/3900560998 >> 172.21.15.49:0/3900560998 conn(0x7f341c0fcda0 msgr2=0x7f341c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:33.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 -- 172.21.15.49:0/3900560998 shutdown_connections 2024-03-18T00:58:33.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:33.197+0000 7f3422668640 1 -- 172.21.15.49:0/3900560998 wait complete. 2024-03-18T00:58:33.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:33 smithi049 ceph-mon[32130]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 81 MiB/s wr, 429 op/s 2024-03-18T00:58:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:33 smithi107 ceph-mon[36270]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 81 MiB/s wr, 429 op/s 2024-03-18T00:58:33.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:33 smithi106 ceph-mon[35899]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 81 MiB/s wr, 429 op/s 2024-03-18T00:58:35.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:35 smithi049 ceph-mon[32130]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 64 MiB/s wr, 344 op/s 2024-03-18T00:58:35.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:35 smithi107 ceph-mon[36270]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 64 MiB/s wr, 344 op/s 2024-03-18T00:58:35.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:35 smithi106 ceph-mon[35899]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 64 MiB/s wr, 344 op/s 2024-03-18T00:58:37.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:37 smithi049 ceph-mon[32130]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 64 MiB/s wr, 419 op/s 2024-03-18T00:58:37.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:37 smithi107 ceph-mon[36270]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 64 MiB/s wr, 419 op/s 2024-03-18T00:58:37.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:37 smithi106 ceph-mon[35899]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 64 MiB/s wr, 419 op/s 2024-03-18T00:58:39.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:39 smithi049 ceph-mon[32130]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 70 MiB/s wr, 553 op/s 2024-03-18T00:58:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:39 smithi107 ceph-mon[36270]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 70 MiB/s wr, 553 op/s 2024-03-18T00:58:39.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:39 smithi106 ceph-mon[35899]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 70 MiB/s wr, 553 op/s 2024-03-18T00:58:41.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:41 smithi049 ceph-mon[32130]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 57 MiB/s wr, 522 op/s 2024-03-18T00:58:41.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:41 smithi107 ceph-mon[36270]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 57 MiB/s wr, 522 op/s 2024-03-18T00:58:41.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:41 smithi106 ceph-mon[35899]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 57 MiB/s wr, 522 op/s 2024-03-18T00:58:43.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:43 smithi049 ceph-mon[32130]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 83 MiB/s wr, 870 op/s 2024-03-18T00:58:43.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:43 smithi107 ceph-mon[36270]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 83 MiB/s wr, 870 op/s 2024-03-18T00:58:43.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:43 smithi106 ceph-mon[35899]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 83 MiB/s wr, 870 op/s 2024-03-18T00:58:45.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:45 smithi049 ceph-mon[32130]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 83 MiB/s wr, 945 op/s 2024-03-18T00:58:45.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:45 smithi107 ceph-mon[36270]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 83 MiB/s wr, 945 op/s 2024-03-18T00:58:45.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:45 smithi106 ceph-mon[35899]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 83 MiB/s wr, 945 op/s 2024-03-18T00:58:46.032 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:58:46.032 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 -- 172.21.15.49:0/3742014126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41006e0 msgr2=0x7f5be4100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3742014126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41006e0 0x7f5be4100b00 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009900 tx=0x7f5bdc02bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 -- 172.21.15.49:0/3742014126 shutdown_connections 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3742014126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be41018d0 0x7f5be4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3742014126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be4102290 0x7f5be4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3742014126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41006e0 0x7f5be4100b00 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 -- 172.21.15.49:0/3742014126 >> 172.21.15.49:0/3742014126 conn(0x7f5be40fbc70 msgr2=0x7f5be40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 -- 172.21.15.49:0/3742014126 shutdown_connections 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 -- 172.21.15.49:0/3742014126 wait complete. 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.158+0000 7f5beaec6640 1 Processor -- start 2024-03-18T00:58:46.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5beaec6640 1 -- start start 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5beaec6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be41006e0 0x7f5be410b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5beaec6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41018d0 0x7f5be41096a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5beaec6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be4102290 0x7f5be4109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5beaec6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5be410d440 con 0x7f5be41018d0 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5beaec6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5be410d2c0 con 0x7f5be4102290 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5beaec6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5be410d5c0 con 0x7f5be41006e0 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41018d0 0x7f5be41096a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41018d0 0x7f5be41096a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38032/0 (socket says 172.21.15.49:38032) 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be3fff640 1 -- 172.21.15.49:0/3741039157 learned_addr learned my addr 172.21.15.49:0/3741039157 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be8c62640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be41006e0 0x7f5be410b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be9463640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be4102290 0x7f5be4109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be3fff640 1 -- 172.21.15.49:0/3741039157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be41006e0 msgr2=0x7f5be410b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be3fff640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be41006e0 0x7f5be410b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be3fff640 1 -- 172.21.15.49:0/3741039157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be4102290 msgr2=0x7f5be4109be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be3fff640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be4102290 0x7f5be4109be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be3fff640 1 -- 172.21.15.49:0/3741039157 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5be410a470 con 0x7f5be41018d0 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be8c62640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be41006e0 0x7f5be410b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.159+0000 7f5be3fff640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41018d0 0x7f5be41096a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400ebc0 tx=0x7f5bd400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.160+0000 7f5be1ffb640 1 -- 172.21.15.49:0/3741039157 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4010430 con 0x7f5be41018d0 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.160+0000 7f5be1ffb640 1 -- 172.21.15.49:0/3741039157 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd4010ab0 con 0x7f5be41018d0 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.160+0000 7f5beaec6640 1 -- 172.21.15.49:0/3741039157 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5be41a1ca0 con 0x7f5be41018d0 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.160+0000 7f5be1ffb640 1 -- 172.21.15.49:0/3741039157 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4018b30 con 0x7f5be41018d0 2024-03-18T00:58:46.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.160+0000 7f5beaec6640 1 -- 172.21.15.49:0/3741039157 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5be41a2190 con 0x7f5be41018d0 2024-03-18T00:58:46.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.162+0000 7f5be1ffb640 1 -- 172.21.15.49:0/3741039157 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5bd4018cd0 con 0x7f5be41018d0 2024-03-18T00:58:46.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.162+0000 7f5bedb8c740 1 Processor -- start 2024-03-18T00:58:46.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.162+0000 7f5bedb8c740 1 -- start start 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7170 0x556ffe5f7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe5fc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 0x556ffe5ff160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x556ffe55c500 con 0x556ffe5f7b60 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x556ffe504be0 con 0x556ffe5fccd0 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556ffe5585a0 con 0x556ffe5f7170 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be1ffb640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bd00a2880 0x7f5bd00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be8c62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe5fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be9463640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7170 0x556ffe5f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be8c62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe5fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38048/0 (socket says 172.21.15.49:38048) 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be8c62640 1 -- 172.21.15.49:0/1149077223 learned_addr learned my addr 172.21.15.49:0/1149077223 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 0x556ffe5ff160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be9463640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bd00a2880 0x7f5bd00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be1ffb640 1 -- 172.21.15.49:0/3741039157 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5bd40cf010 con 0x7f5be41018d0 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be8c62640 1 -- 172.21.15.49:0/1149077223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7170 msgr2=0x556ffe5f7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1149077223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7170 0x556ffe5f7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be8c62640 1 -- 172.21.15.49:0/1149077223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 msgr2=0x556ffe5ff160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1149077223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 0x556ffe5ff160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be8c62640 1 -- 172.21.15.49:0/1149077223 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ffe5184b0 con 0x556ffe5f7b60 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be9463640 1 --2- 172.21.15.49:0/1149077223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7170 0x556ffe5f7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be3fff640 1 --2- 172.21.15.49:0/1149077223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 0x556ffe5ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be9463640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bd00a2880 0x7f5bd00a4d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x556ffe5f83c0 tx=0x7f5bcc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.163+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1149077223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe5fc660 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009900 tx=0x7f5bdc02cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fd9236fb21d7f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bdaffd640 1 -- 172.21.15.49:0/1149077223 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc005d50 con 0x556ffe5f7b60 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bdaffd640 1 -- 172.21.15.49:0/1149077223 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bdc005520 con 0x556ffe5f7b60 2024-03-18T00:58:46.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bdaffd640 1 -- 172.21.15.49:0/1149077223 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc004c20 con 0x556ffe5f7b60 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1149077223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 msgr2=0x556ffe5fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1149077223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe5fc660 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc009900 tx=0x7f5bdc02cd30 comp rx=0 tx=0).stop 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1149077223 shutdown_connections 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1149077223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 0x556ffe5ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1149077223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe5fc660 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1149077223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7170 0x556ffe5f7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1149077223 >> 172.21.15.49:0/1149077223 conn(0x556ffe5f26e0 msgr2=0x556ffe5f2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1149077223 shutdown_connections 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.164+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1149077223 wait complete. 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5bedb8c740 1 -- start start 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5bedb8c740 1 Processor -- start 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7170 0x556ffe601670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe601bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 0x556ffe608bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x556ffe5ffc20 con 0x556ffe5f7b60 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x556ffe518160 con 0x556ffe5f7170 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x556ffe5ffda0 con 0x556ffe5fccd0 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5bedb8c740 10 client.0 _ll_register_callbacks cb 0x556ffe4fc150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5be3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe601bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5be3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe601bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38054/0 (socket says 172.21.15.49:38054) 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7170 0x556ffe601670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5be3fff640 1 -- 172.21.15.49:0/1623145079 learned_addr learned my addr 172.21.15.49:0/1623145079 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.165+0000 7f5be9463640 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 0x556ffe608bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5be3fff640 1 -- 172.21.15.49:0/1623145079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 msgr2=0x556ffe608bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5be3fff640 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 0x556ffe608bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5be3fff640 1 -- 172.21.15.49:0/1623145079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7170 msgr2=0x556ffe601670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5be3fff640 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7170 0x556ffe601670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5be3fff640 1 -- 172.21.15.49:0/1623145079 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ffe33a7d0 con 0x556ffe5f7b60 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7170 0x556ffe601670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5be3fff640 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe601bb0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400c7d0 tx=0x7f5bd4066c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96639d587254382 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5bc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1623145079 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4026040 con 0x556ffe5f7b60 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1623145079 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5bd4066860 con 0x556ffe5f7b60 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1623145079 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bdc005520 con 0x556ffe5f7b60 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1623145079 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5bd409fb80 con 0x556ffe5f7b60 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5bedb8c740 10 client.17520 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1623145079 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556ffe2703f0 con 0x556ffe5f7b60 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.166+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1623145079 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bdc005520 con 0x556ffe5f7b60 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1623145079 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5bd409ae80 con 0x556ffe5f7b60 2024-03-18T00:58:46.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5bedb8c740 10 client.17520 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:46.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5bedb8c740 10 client.17520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:46.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5bedb8c740 20 client.17520 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:46.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556ffe60d070 0x556ffe60f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5bedb8c740 4 client.17520 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:46.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1623145079 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ffe569a40 con 0x556ffe60d070 2024-03-18T00:58:46.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556ffe60d070 0x556ffe60f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5bc3fff640 10 client.17520 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:46.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.167+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556ffe60d070 0x556ffe60f4e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=618c6da00eb23c8e server_cookie=1fe1daa0ca29f7e5 in_seq=0 out_seq=0 2024-03-18T00:58:46.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1623145079 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556ffe569a40 con 0x556ffe60d070 2024-03-18T00:58:46.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bc3fff640 10 client.17520 handle_command_reply: tid=0 2024-03-18T00:58:46.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bedb8c740 1 client.17520 shutdown 2024-03-18T00:58:46.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1623145079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 msgr2=0x556ffe601bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe601bb0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400c7d0 tx=0x7f5bd4066c90 comp rx=0 tx=0).stop 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1623145079 shutdown_connections 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556ffe60d070 0x556ffe60f4e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 0x556ffe608bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7b60 0x556ffe601bb0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1623145079 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7170 0x556ffe601670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bc3fff640 0 client.17520 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.170+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1623145079 >> 172.21.15.49:0/1623145079 conn(0x556ffe5f26e0 msgr2=0x556ffe5f1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.171+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1623145079 shutdown_connections 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.171+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1623145079 wait complete. 2024-03-18T00:58:46.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.171+0000 7f5bedb8c740 20 client.17520 trim_cache size 0 max 16384 2024-03-18T00:58:46.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5bedb8c740 1 Processor -- start 2024-03-18T00:58:46.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5bedb8c740 1 -- start start 2024-03-18T00:58:46.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe5f1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7b60 0x556ffe608bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 0x556ffe5f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5be41aaf50 con 0x556ffe5f7170 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5be41aca90 con 0x556ffe5fccd0 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5be41ab620 con 0x556ffe5f7b60 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be8c62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe5f1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be8c62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe5f1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38070/0 (socket says 172.21.15.49:38070) 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be8c62640 1 -- 172.21.15.49:0/3407261474 learned_addr learned my addr 172.21.15.49:0/3407261474 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be3fff640 1 --2- 172.21.15.49:0/3407261474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7b60 0x556ffe608bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be9463640 1 --2- 172.21.15.49:0/3407261474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 0x556ffe5f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be8c62640 1 -- 172.21.15.49:0/3407261474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7b60 msgr2=0x556ffe608bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be8c62640 1 --2- 172.21.15.49:0/3407261474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7b60 0x556ffe608bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be8c62640 1 -- 172.21.15.49:0/3407261474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 msgr2=0x556ffe5f15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be8c62640 1 --2- 172.21.15.49:0/3407261474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 0x556ffe5f15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be8c62640 1 -- 172.21.15.49:0/3407261474 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ffe2300f0 con 0x556ffe5f7170 2024-03-18T00:58:46.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.189+0000 7f5be8c62640 1 --2- 172.21.15.49:0/3407261474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe5f1070 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc02c7b0 tx=0x7f5bdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383cf93ef52c0632 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bda7fc640 1 -- 172.21.15.49:0/3407261474 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc009520 con 0x556ffe5f7170 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bda7fc640 1 -- 172.21.15.49:0/3407261474 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bdc005510 con 0x556ffe5f7170 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bda7fc640 1 -- 172.21.15.49:0/3407261474 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc0500d0 con 0x556ffe5f7170 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bedb8c740 1 -- 172.21.15.49:0/3407261474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 msgr2=0x556ffe5f1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/3407261474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe5f1070 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc02c7b0 tx=0x7f5bdc0046c0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bedb8c740 1 -- 172.21.15.49:0/3407261474 shutdown_connections 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/3407261474 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5fccd0 0x556ffe5f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/3407261474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5f7b60 0x556ffe608bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/3407261474 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe5f1070 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bedb8c740 1 -- 172.21.15.49:0/3407261474 >> 172.21.15.49:0/3407261474 conn(0x556ffe5f26e0 msgr2=0x556ffe5f90f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bedb8c740 1 -- 172.21.15.49:0/3407261474 shutdown_connections 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.190+0000 7f5bedb8c740 1 -- 172.21.15.49:0/3407261474 wait complete. 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5bedb8c740 1 -- start start 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5bedb8c740 1 Processor -- start 2024-03-18T00:58:46.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe605b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7b60 0x556ffe6060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5bedb8c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 0x556ffe6713a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5be41ac3c0 con 0x556ffe5f7170 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5be41abcf0 con 0x556ffe5f7b60 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5bedb8c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5be41a8f30 con 0x556ffe5fccd0 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5bedb8c740 10 client.0 _ll_register_callbacks cb 0x7f5bd4097d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5be8c62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe605b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5be8c62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe605b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38078/0 (socket says 172.21.15.49:38078) 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5be8c62640 1 -- 172.21.15.49:0/1760187670 learned_addr learned my addr 172.21.15.49:0/1760187670 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5be3fff640 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7b60 0x556ffe6060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.191+0000 7f5be9463640 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 0x556ffe6713a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5be8c62640 1 -- 172.21.15.49:0/1760187670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 msgr2=0x556ffe6713a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 0x556ffe6713a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5be8c62640 1 -- 172.21.15.49:0/1760187670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7b60 msgr2=0x556ffe6060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7b60 0x556ffe6060b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5be8c62640 1 -- 172.21.15.49:0/1760187670 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ffe5239d0 con 0x556ffe5f7170 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5be8c62640 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe605b70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc02c170 tx=0x7f5bdc009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb0266a9896fdc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bc3fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1760187670 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc055060 con 0x556ffe5f7170 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1760187670 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5bdc004930 con 0x556ffe5f7170 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1760187670 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bdc005510 con 0x556ffe5f7170 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1760187670 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f5bdc004d20 con 0x556ffe5f7170 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bedb8c740 10 client.17526 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1760187670 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556ffe672aa0 con 0x556ffe5f7170 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1760187670 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bdc005510 con 0x556ffe5f7170 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1760187670 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f5bdc0035f0 con 0x556ffe5f7170 2024-03-18T00:58:46.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bedb8c740 10 client.17526 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:46.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bedb8c740 10 client.17526 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:46.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.192+0000 7f5bedb8c740 20 client.17526 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:46.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.193+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556ffe672fa0 0x556ffe675390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.193+0000 7f5bedb8c740 4 client.17526 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:58:46.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.193+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1760187670 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ffe563d90 con 0x556ffe672fa0 2024-03-18T00:58:46.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.193+0000 7f5be3fff640 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556ffe672fa0 0x556ffe675390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.193+0000 7f5be3fff640 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556ffe672fa0 0x556ffe675390 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a862a8e928ff805 server_cookie=76e078cf3ab6dcaa in_seq=0 out_seq=0 2024-03-18T00:58:46.200 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.193+0000 7f5bc3fff640 10 client.17526 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:46.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bc3fff640 1 -- 172.21.15.49:0/1760187670 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556ffe563d90 con 0x556ffe672fa0 2024-03-18T00:58:46.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bc3fff640 10 client.17526 handle_command_reply: tid=0 2024-03-18T00:58:46.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 client.17526 shutdown 2024-03-18T00:58:46.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1760187670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 msgr2=0x556ffe605b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe605b70 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc02c170 tx=0x7f5bdc009520 comp rx=0 tx=0).stop 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1760187670 shutdown_connections 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x556ffe672fa0 0x556ffe675390 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x556ffe5f7b60 0x556ffe6060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556ffe5fccd0 0x556ffe6713a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bc3fff640 0 client.17526 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 --2- 172.21.15.49:0/1760187670 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x556ffe5f7170 0x556ffe605b70 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1760187670 >> 172.21.15.49:0/1760187670 conn(0x556ffe5f26e0 msgr2=0x556ffe5f5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1760187670 shutdown_connections 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 1 -- 172.21.15.49:0/1760187670 wait complete. 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.197+0000 7f5bedb8c740 20 client.17526 trim_cache size 0 max 16384 2024-03-18T00:58:46.202 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:58:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:58:46.203 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:58:46.203 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:58:46.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 -- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bd00a2880 msgr2=0x7f5bd00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bd00a2880 0x7f5bd00a4d40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x556ffe5f83c0 tx=0x7f5bcc005f90 comp rx=0 tx=0).stop 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 -- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41018d0 msgr2=0x7f5be41096a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41018d0 0x7f5be41096a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f5bd400ebc0 tx=0x7f5bd400c2d0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 -- 172.21.15.49:0/3741039157 shutdown_connections 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5bd00a2880 0x7f5bd00a4d40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5be4102290 0x7f5be4109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5be41018d0 0x7f5be41096a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 --2- 172.21.15.49:0/3741039157 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5be41006e0 0x7f5be410b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 -- 172.21.15.49:0/3741039157 >> 172.21.15.49:0/3741039157 conn(0x7f5be40fbc70 msgr2=0x7f5be4104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 -- 172.21.15.49:0/3741039157 shutdown_connections 2024-03-18T00:58:46.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.198+0000 7f5beaec6640 1 -- 172.21.15.49:0/3741039157 wait complete. 2024-03-18T00:58:46.209 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:58:46.209 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:58:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.291+0000 7fdaade6a640 1 -- 172.21.15.49:0/3565176677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100700 msgr2=0x7fdaa8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.291+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3565176677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100700 0x7fdaa8100b20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda98009900 tx=0x7fda9802bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.291+0000 7fdaade6a640 1 -- 172.21.15.49:0/3565176677 shutdown_connections 2024-03-18T00:58:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.291+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3565176677 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaa81022b0 0x7fdaa8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.291+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3565176677 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaa81018f0 0x7fdaa8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.291+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3565176677 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa8100700 0x7fdaa8100b20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.293 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.291+0000 7fdaade6a640 1 -- 172.21.15.49:0/3565176677 >> 172.21.15.49:0/3565176677 conn(0x7fdaa80fbc70 msgr2=0x7fdaa80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.291+0000 7fdaade6a640 1 -- 172.21.15.49:0/3565176677 shutdown_connections 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.291+0000 7fdaade6a640 1 -- 172.21.15.49:0/3565176677 wait complete. 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaade6a640 1 Processor -- start 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaade6a640 1 -- start start 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaade6a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaa8100700 0x7fdaa8104640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaade6a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa81018f0 0x7fdaa8102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaade6a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaa81022b0 0x7fdaa8103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaade6a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa810c380 con 0x7fdaa81022b0 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaade6a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa810c200 con 0x7fdaa8100700 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaade6a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa810c500 con 0x7fdaa81018f0 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaa7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaa81022b0 0x7fdaa8103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.294 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaa7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaa81022b0 0x7fdaa8103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38086/0 (socket says 172.21.15.49:38086) 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaa7fff640 1 -- 172.21.15.49:0/3569032640 learned_addr learned my addr 172.21.15.49:0/3569032640 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa81018f0 0x7fdaa8102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaa8100700 0x7fdaa8104640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaa7fff640 1 -- 172.21.15.49:0/3569032640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa81018f0 msgr2=0x7fdaa8102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaa7fff640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa81018f0 0x7fdaa8102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaa7fff640 1 -- 172.21.15.49:0/3569032640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaa8100700 msgr2=0x7fdaa8104640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.292+0000 7fdaa7fff640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaa8100700 0x7fdaa8104640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.293+0000 7fdaa7fff640 1 -- 172.21.15.49:0/3569032640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa8103ad0 con 0x7fdaa81022b0 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.293+0000 7fdaa7fff640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaa81022b0 0x7fdaa8103210 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000eb10 tx=0x7fdaa000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.293+0000 7fdaa4ff9640 1 -- 172.21.15.49:0/3569032640 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0004b20 con 0x7fdaa81022b0 2024-03-18T00:58:46.295 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.293+0000 7fdaa4ff9640 1 -- 172.21.15.49:0/3569032640 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaa0004cc0 con 0x7fdaa81022b0 2024-03-18T00:58:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.293+0000 7fdaade6a640 1 -- 172.21.15.49:0/3569032640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa81a1c30 con 0x7fdaa81022b0 2024-03-18T00:58:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.293+0000 7fdaa4ff9640 1 -- 172.21.15.49:0/3569032640 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0010760 con 0x7fdaa81022b0 2024-03-18T00:58:46.296 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.293+0000 7fdaade6a640 1 -- 172.21.15.49:0/3569032640 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa81a2150 con 0x7fdaa81022b0 2024-03-18T00:58:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.295+0000 7fdaa4ff9640 1 -- 172.21.15.49:0/3569032640 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdaa0005020 con 0x7fdaa81022b0 2024-03-18T00:58:46.301 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdab0a4d740 1 Processor -- start 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdab0a4d740 1 -- start start 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ab5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1aba830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 0x55abe1abd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55abe1a1aa40 con 0x55abe1ab5340 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55abe19c3920 con 0x55abe1abaea0 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55abe1a16ae0 con 0x55abe1ab5d30 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ab5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa77fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1aba830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.302 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ab5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38094/0 (socket says 172.21.15.49:38094) 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa6ffd640 1 -- 172.21.15.49:0/831498286 learned_addr learned my addr 172.21.15.49:0/831498286 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/831498286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 0x55abe1abd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa4ff9640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda900a28a0 0x7fda900a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa77fe640 1 -- 172.21.15.49:0/831498286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 msgr2=0x55abe1abd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/831498286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 0x55abe1abd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa77fe640 1 -- 172.21.15.49:0/831498286 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 msgr2=0x55abe1ab5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/831498286 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ab5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa77fe640 1 -- 172.21.15.49:0/831498286 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abe19db460 con 0x55abe1ab5d30 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa7fff640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda900a28a0 0x7fda900a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.296+0000 7fdaa4ff9640 1 -- 172.21.15.49:0/3569032640 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fdaa00cf140 con 0x7fdaa81022b0 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.297+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/831498286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1aba830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fda9802c170 tx=0x7fda9802c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bcbf85cc90e7495 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.303 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.297+0000 7fdaa7fff640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda900a28a0 0x7fda900a4d60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0097f20 tx=0x7fdaa0066880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.297+0000 7fda8dffb640 1 -- 172.21.15.49:0/831498286 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9802cb80 con 0x55abe1ab5d30 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.297+0000 7fda8dffb640 1 -- 172.21.15.49:0/831498286 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda98005040 con 0x55abe1ab5d30 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.297+0000 7fda8dffb640 1 -- 172.21.15.49:0/831498286 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98004940 con 0x55abe1ab5d30 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 -- 172.21.15.49:0/831498286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 msgr2=0x55abe1aba830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/831498286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1aba830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fda9802c170 tx=0x7fda9802c660 comp rx=0 tx=0).stop 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 -- 172.21.15.49:0/831498286 shutdown_connections 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/831498286 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 0x55abe1abd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/831498286 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1aba830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/831498286 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ab5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 -- 172.21.15.49:0/831498286 >> 172.21.15.49:0/831498286 conn(0x55abe1ab08b0 msgr2=0x55abe1ab0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.304 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 -- 172.21.15.49:0/831498286 shutdown_connections 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 -- 172.21.15.49:0/831498286 wait complete. 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 -- start start 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 Processor -- start 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5340 0x55abe1abf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5d30 0x55abe1abfb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.298+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1abaea0 0x55abe1ac6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55abe1abddf0 con 0x55abe1abaea0 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55abe19d6d70 con 0x55abe1ab5d30 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55abe1abdf70 con 0x55abe1ab5340 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdab0a4d740 10 client.0 _ll_register_callbacks cb 0x55abe172ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5340 0x55abe1abf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa77fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5d30 0x55abe1abfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.305 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5340 0x55abe1abf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44204/0 (socket says 172.21.15.49:44204) 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa6ffd640 1 -- 172.21.15.49:0/66211056 learned_addr learned my addr 172.21.15.49:0/66211056 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1abaea0 0x55abe1ac6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa77fe640 1 -- 172.21.15.49:0/66211056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5340 msgr2=0x55abe1abf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5340 0x55abe1abf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa77fe640 1 -- 172.21.15.49:0/66211056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1abaea0 msgr2=0x55abe1ac6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1abaea0 0x55abe1ac6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa77fe640 1 -- 172.21.15.49:0/66211056 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abe172f210 con 0x55abe1ab5d30 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5340 0x55abe1abf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5d30 0x55abe1abfb10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55abe1ab6130 tx=0x7fda9802cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61aa304351e5e8ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fda86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fda86ffd640 1 -- 172.21.15.49:0/66211056 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98005600 con 0x55abe1ab5d30 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fda86ffd640 1 -- 172.21.15.49:0/66211056 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fda980046c0 con 0x55abe1ab5d30 2024-03-18T00:58:46.306 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.299+0000 7fdab0a4d740 1 -- 172.21.15.49:0/66211056 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda98005040 con 0x55abe1ab5d30 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fda86ffd640 1 -- 172.21.15.49:0/66211056 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda98004900 con 0x55abe1ab5d30 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fdab0a4d740 10 client.26678 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fdab0a4d740 1 -- 172.21.15.49:0/66211056 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55abe17fb620 con 0x55abe1ab5d30 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fda86ffd640 1 -- 172.21.15.49:0/66211056 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda98005040 con 0x55abe1ab5d30 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fda86ffd640 1 -- 172.21.15.49:0/66211056 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fda98034440 con 0x55abe1ab5d30 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fdab0a4d740 10 client.26678 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fdab0a4d740 10 client.26678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fdab0a4d740 20 client.26678 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55abe1acb040 0x55abe1acd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fdab0a4d740 4 client.26678 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fdab0a4d740 1 -- 172.21.15.49:0/66211056 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abe1a27f80 con 0x55abe1acb040 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.300+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55abe1acb040 0x55abe1acd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.307 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.301+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55abe1acb040 0x55abe1acd430 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1865a7ab1231a20 server_cookie=de1c1c41169923c1 in_seq=0 out_seq=0 2024-03-18T00:58:46.308 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.301+0000 7fda86ffd640 10 client.26678 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fda86ffd640 1 -- 172.21.15.49:0/66211056 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55abe1a27f80 con 0x55abe1acb040 2024-03-18T00:58:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fda86ffd640 10 client.26678 handle_command_reply: tid=0 2024-03-18T00:58:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 client.26678 shutdown 2024-03-18T00:58:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 -- 172.21.15.49:0/66211056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5d30 msgr2=0x55abe1abfb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5d30 0x55abe1abfb10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x55abe1ab6130 tx=0x7fda9802cc30 comp rx=0 tx=0).stop 2024-03-18T00:58:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 -- 172.21.15.49:0/66211056 shutdown_connections 2024-03-18T00:58:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55abe1acb040 0x55abe1acd430 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5d30 0x55abe1abfb10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.309 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1abaea0 0x55abe1ac6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fda86ffd640 0 client.26678 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/66211056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5340 0x55abe1abf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 -- 172.21.15.49:0/66211056 >> 172.21.15.49:0/66211056 conn(0x55abe1ab08b0 msgr2=0x55abe1aafbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 -- 172.21.15.49:0/66211056 shutdown_connections 2024-03-18T00:58:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 1 -- 172.21.15.49:0/66211056 wait complete. 2024-03-18T00:58:46.310 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.303+0000 7fdab0a4d740 20 client.26678 trim_cache size 0 max 16384 2024-03-18T00:58:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.318+0000 7fdab0a4d740 1 Processor -- start 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.318+0000 7fdab0a4d740 1 -- start start 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5340 0x55abe1aaf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5d30 0x55abe1ac6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1abaea0 0x55abe1aaf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81aaf40 con 0x55abe1ab5d30 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81aca80 con 0x55abe1ab5340 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81ab610 con 0x55abe1abaea0 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5d30 0x55abe1ac6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5340 0x55abe1aaf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa77fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5d30 0x55abe1ac6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38120/0 (socket says 172.21.15.49:38120) 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5340 0x55abe1aaf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:58714/0 (socket says 172.21.15.49:58714) 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa77fe640 1 -- 172.21.15.49:0/1028395988 learned_addr learned my addr 172.21.15.49:0/1028395988 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/1028395988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1abaea0 0x55abe1aaf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa77fe640 1 -- 172.21.15.49:0/1028395988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1abaea0 msgr2=0x55abe1aaf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/1028395988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1abaea0 0x55abe1aaf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa77fe640 1 -- 172.21.15.49:0/1028395988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5340 msgr2=0x55abe1aaf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/1028395988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5340 0x55abe1aaf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa77fe640 1 -- 172.21.15.49:0/1028395988 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abe1aa2140 con 0x55abe1ab5d30 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/1028395988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5d30 0x55abe1ac6b10 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fda9802c170 tx=0x7fda98000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac06f0f94753d09f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.319+0000 7fda8d7fa640 1 -- 172.21.15.49:0/1028395988 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980044e0 con 0x55abe1ab5d30 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fda8d7fa640 1 -- 172.21.15.49:0/1028395988 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda9802cca0 con 0x55abe1ab5d30 2024-03-18T00:58:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fda8d7fa640 1 -- 172.21.15.49:0/1028395988 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda98004d00 con 0x55abe1ab5d30 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fdab0a4d740 1 -- 172.21.15.49:0/1028395988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5d30 msgr2=0x55abe1ac6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/1028395988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5d30 0x55abe1ac6b10 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fda9802c170 tx=0x7fda98000bc0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fdab0a4d740 1 -- 172.21.15.49:0/1028395988 shutdown_connections 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/1028395988 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1abaea0 0x55abe1aaf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/1028395988 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5d30 0x55abe1ac6b10 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/1028395988 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1ab5340 0x55abe1aaf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fdab0a4d740 1 -- 172.21.15.49:0/1028395988 >> 172.21.15.49:0/1028395988 conn(0x55abe1ab08b0 msgr2=0x55abe1abf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fdab0a4d740 1 -- 172.21.15.49:0/1028395988 shutdown_connections 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.320+0000 7fdab0a4d740 1 -- 172.21.15.49:0/1028395988 wait complete. 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdab0a4d740 1 -- start start 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdab0a4d740 1 Processor -- start 2024-03-18T00:58:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ac4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1ac4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdab0a4d740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 0x55abe1ac5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81ac3b0 con 0x55abe1ab5340 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81abce0 con 0x55abe1abaea0 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdab0a4d740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa81a8f20 con 0x55abe1ab5d30 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdab0a4d740 10 client.0 _ll_register_callbacks cb 0x7fda9c009870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ac4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ac4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38124/0 (socket says 172.21.15.49:38124) 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdaa6ffd640 1 -- 172.21.15.49:0/217355853 learned_addr learned my addr 172.21.15.49:0/217355853 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1ac4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 0x55abe1ac5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdaa77fe640 1 -- 172.21.15.49:0/217355853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 msgr2=0x55abe1ac5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 0x55abe1ac5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdaa77fe640 1 -- 172.21.15.49:0/217355853 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 msgr2=0x55abe1ac4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.321+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ac4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fdaa77fe640 1 -- 172.21.15.49:0/217355853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abe1ac6010 con 0x55abe1ab5d30 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ac4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fdaa77fe640 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1ac4a80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fda9802c170 tx=0x7fda980346d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c72bb3c2f231a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fda86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fda86ffd640 1 -- 172.21.15.49:0/217355853 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980046c0 con 0x55abe1ab5d30 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fda86ffd640 1 -- 172.21.15.49:0/217355853 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fda98005610 con 0x55abe1ab5d30 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fdab0a4d740 1 -- 172.21.15.49:0/217355853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda9802cca0 con 0x55abe1ab5d30 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fda86ffd640 1 -- 172.21.15.49:0/217355853 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda98004aa0 con 0x55abe1ab5d30 2024-03-18T00:58:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fdab0a4d740 10 client.26776 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fdab0a4d740 1 -- 172.21.15.49:0/217355853 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55abe1b314c0 con 0x55abe1ab5d30 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.322+0000 7fda86ffd640 1 -- 172.21.15.49:0/217355853 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda9802cca0 con 0x55abe1ab5d30 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fda86ffd640 1 -- 172.21.15.49:0/217355853 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fda98003b20 con 0x55abe1ab5d30 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fdab0a4d740 10 client.26776 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fdab0a4d740 10 client.26776 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fdab0a4d740 20 client.26776 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55abe1acb040 0x55abe1b33e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fdab0a4d740 4 client.26776 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fdab0a4d740 1 -- 172.21.15.49:0/217355853 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55abe1a222d0 con 0x55abe1acb040 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55abe1acb040 0x55abe1b33e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fda86ffd640 10 client.26776 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:46.323+0000 7fdaa6ffd640 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55abe1acb040 0x55abe1b33e60 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87fc3e6ee2df301e server_cookie=b70ca311e11c2228 in_seq=0 out_seq=0 2024-03-18T00:58:46.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.325+0000 7fa43a33b640 1 -- 172.21.15.49:0/1633995939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 msgr2=0x7fa434100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.325+0000 7fa43a33b640 1 --2- 172.21.15.49:0/1633995939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 0x7fa434100b20 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fa424009900 tx=0x7fa42402bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.325+0000 7fa43a33b640 1 -- 172.21.15.49:0/1633995939 shutdown_connections 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.325+0000 7fa43a33b640 1 --2- 172.21.15.49:0/1633995939 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4341022b0 0x7fa434108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.325+0000 7fa43a33b640 1 --2- 172.21.15.49:0/1633995939 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4341018f0 0x7fa434101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.325+0000 7fa43a33b640 1 --2- 172.21.15.49:0/1633995939 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 0x7fa434100b20 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.325+0000 7fa43a33b640 1 -- 172.21.15.49:0/1633995939 >> 172.21.15.49:0/1633995939 conn(0x7fa4340fbc70 msgr2=0x7fa4340fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.325+0000 7fa43a33b640 1 -- 172.21.15.49:0/1633995939 shutdown_connections 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.325+0000 7fa43a33b640 1 -- 172.21.15.49:0/1633995939 wait complete. 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa43a33b640 1 Processor -- start 2024-03-18T00:58:46.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa43a33b640 1 -- start start 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa43a33b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 0x7fa434104640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa43a33b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4341018f0 0x7fa434102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa43a33b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4341022b0 0x7fa434103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa43a33b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa43410c350 con 0x7fa434100700 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa43a33b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa43410c1d0 con 0x7fa4341018f0 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa43a33b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa43410c4d0 con 0x7fa4341022b0 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa433fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 0x7fa434104640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.326+0000 7fa433fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 0x7fa434104640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38150/0 (socket says 172.21.15.49:38150) 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa433fff640 1 -- 172.21.15.49:0/2952782180 learned_addr learned my addr 172.21.15.49:0/2952782180 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa4388d8640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4341022b0 0x7fa434103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa4337fe640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4341018f0 0x7fa434102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa433fff640 1 -- 172.21.15.49:0/2952782180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4341022b0 msgr2=0x7fa434103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa433fff640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4341022b0 0x7fa434103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa433fff640 1 -- 172.21.15.49:0/2952782180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4341018f0 msgr2=0x7fa434102cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa433fff640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4341018f0 0x7fa434102cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa433fff640 1 -- 172.21.15.49:0/2952782180 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa434103ad0 con 0x7fa434100700 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa433fff640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 0x7fa434104640 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa424000c00 tx=0x7fa42402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa4317fa640 1 -- 172.21.15.49:0/2952782180 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa424041070 con 0x7fa434100700 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4341a1c00 con 0x7fa434100700 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa4317fa640 1 -- 172.21.15.49:0/2952782180 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa424004450 con 0x7fa434100700 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa4317fa640 1 -- 172.21.15.49:0/2952782180 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa424034440 con 0x7fa434100700 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.327+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4341a2110 con 0x7fa434100700 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.329+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4341a22e0 con 0x7fa434100700 2024-03-18T00:58:46.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.330+0000 7fa4317fa640 1 -- 172.21.15.49:0/2952782180 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa424044070 con 0x7fa434100700 2024-03-18T00:58:46.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.331+0000 7fa4317fa640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa41c0a2800 0x7fa41c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.331+0000 7fa4337fe640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa41c0a2800 0x7fa41c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.331+0000 7fa4317fa640 1 -- 172.21.15.49:0/2952782180 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa4240f0560 con 0x7fa434100700 2024-03-18T00:58:46.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.331+0000 7fa4337fe640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa41c0a2800 0x7fa41c0a4cc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa4341043b0 tx=0x7fa428005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.335+0000 7fa4317fa640 1 -- 172.21.15.49:0/2952782180 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa4240bd020 con 0x7fa434100700 2024-03-18T00:58:46.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.564+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4341a24b0 con 0x7fa434100700 2024-03-18T00:58:46.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.565+0000 7fa4317fa640 1 -- 172.21.15.49:0/2952782180 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11934 (secure 0 0 0) 0x7fa4240c10c0 con 0x7fa434100700 2024-03-18T00:58:46.565 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:58:46.565 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:57:06.597245+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":138,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24499,24506,24481,24470,24463]},"id":1}]} 2024-03-18T00:58:46.565 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 28 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa41c0a2800 msgr2=0x7fa41c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa41c0a2800 0x7fa41c0a4cc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa4341043b0 tx=0x7fa428005f90 comp rx=0 tx=0).stop 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 msgr2=0x7fa434104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 0x7fa434104640 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa424000c00 tx=0x7fa42402c640 comp rx=0 tx=0).stop 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 shutdown_connections 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa41c0a2800 0x7fa41c0a4cc0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa4341018f0 0x7fa434102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4341022b0 0x7fa434103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 --2- 172.21.15.49:0/2952782180 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa434100700 0x7fa434104640 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 >> 172.21.15.49:0/2952782180 conn(0x7fa4340fbc70 msgr2=0x7fa4340fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 shutdown_connections 2024-03-18T00:58:46.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.567+0000 7fa43a33b640 1 -- 172.21.15.49:0/2952782180 wait complete. 2024-03-18T00:58:46.578 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:58:46.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 -- 172.21.15.49:0/3869829839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9260102290 msgr2=0x7f9260108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.695 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 --2- 172.21.15.49:0/3869829839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9260102290 0x7f9260108b20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f9258009ca0 tx=0x7f925802c0f0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 -- 172.21.15.49:0/3869829839 shutdown_connections 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 --2- 172.21.15.49:0/3869829839 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92601018d0 0x7f9260101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 --2- 172.21.15.49:0/3869829839 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9260102290 0x7f9260108b20 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 --2- 172.21.15.49:0/3869829839 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92601006e0 0x7f9260100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 -- 172.21.15.49:0/3869829839 >> 172.21.15.49:0/3869829839 conn(0x7f92600fbc70 msgr2=0x7f92600fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 -- 172.21.15.49:0/3869829839 shutdown_connections 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 -- 172.21.15.49:0/3869829839 wait complete. 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 Processor -- start 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 -- start start 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.694+0000 7f9268560640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92601006e0 0x7f9260104670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f9268560640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92601018d0 0x7f9260102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f9268560640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9260102290 0x7f92601031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f9268560640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f926010c340 con 0x7f92601006e0 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f9268560640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f926010c1c0 con 0x7f92601018d0 2024-03-18T00:58:46.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f9268560640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f926010c4c0 con 0x7f9260102290 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92662fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92601006e0 0x7f9260104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f9265afb640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92601018d0 0x7f9260102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92662fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92601006e0 0x7f9260104670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38160/0 (socket says 172.21.15.49:38160) 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92662fc640 1 -- 172.21.15.49:0/2299380742 learned_addr learned my addr 172.21.15.49:0/2299380742 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f9266afd640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9260102290 0x7f92601031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92662fc640 1 -- 172.21.15.49:0/2299380742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9260102290 msgr2=0x7f92601031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92662fc640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9260102290 0x7f92601031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92662fc640 1 -- 172.21.15.49:0/2299380742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92601018d0 msgr2=0x7f9260102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92662fc640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92601018d0 0x7f9260102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92662fc640 1 -- 172.21.15.49:0/2299380742 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9260103a80 con 0x7f92601006e0 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f9266afd640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9260102290 0x7f92601031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f9265afb640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92601018d0 0x7f9260102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92662fc640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92601006e0 0x7f9260104670 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f925000e9c0 tx=0x7f925000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.695+0000 7f92577fe640 1 -- 172.21.15.49:0/2299380742 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925000cd80 con 0x7f92601006e0 2024-03-18T00:58:46.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.696+0000 7f92577fe640 1 -- 172.21.15.49:0/2299380742 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92500035f0 con 0x7f92601006e0 2024-03-18T00:58:46.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.696+0000 7f9268560640 1 -- 172.21.15.49:0/2299380742 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92601a1bf0 con 0x7f92601006e0 2024-03-18T00:58:46.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.696+0000 7f92577fe640 1 -- 172.21.15.49:0/2299380742 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250010690 con 0x7f92601006e0 2024-03-18T00:58:46.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.696+0000 7f9268560640 1 -- 172.21.15.49:0/2299380742 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92601a2140 con 0x7f92601006e0 2024-03-18T00:58:46.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.698+0000 7f92577fe640 1 -- 172.21.15.49:0/2299380742 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f92500037a0 con 0x7f92601006e0 2024-03-18T00:58:46.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.699+0000 7f92577fe640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f924c0a2800 0x7f924c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.699+0000 7f926b220740 1 Processor -- start 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.699+0000 7f9265afb640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f924c0a2800 0x7f924c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.699+0000 7f92577fe640 1 -- 172.21.15.49:0/2299380742 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f92500ceea0 con 0x7f92601006e0 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.699+0000 7f926b220740 1 -- start start 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f926b220740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc08c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f926b220740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc0dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f926b220740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc10840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f926b220740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55df3db6d490 con 0x55df3dc0e3b0 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f926b220740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55df3db15c90 con 0x55df3dc08850 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f926b220740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55df3db69530 con 0x55df3dc09240 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f924c0a2800 0x7f924c0a4cc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f92601043e0 tx=0x7f925c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc0dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9266afd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc10840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.701 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc0dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44264/0 (socket says 172.21.15.49:44264) 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 -- 172.21.15.49:0/386425290 learned_addr learned my addr 172.21.15.49:0/386425290 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f92662fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc08c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 -- 172.21.15.49:0/386425290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 msgr2=0x55df3dc08c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 --2- 172.21.15.49:0/386425290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc08c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 -- 172.21.15.49:0/386425290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 msgr2=0x55df3dc10840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 --2- 172.21.15.49:0/386425290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc10840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 -- 172.21.15.49:0/386425290 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df3dbd0500 con 0x55df3dc09240 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.700+0000 7f9265afb640 1 --2- 172.21.15.49:0/386425290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc0dd40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f925c009ba0 tx=0x7f925c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10c8cd7d1b32b000 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.701+0000 7f9254bfe640 1 -- 172.21.15.49:0/386425290 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c00f480 con 0x55df3dc09240 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.701+0000 7f9254bfe640 1 -- 172.21.15.49:0/386425290 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f925c00fb00 con 0x55df3dc09240 2024-03-18T00:58:46.702 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.701+0000 7f9254bfe640 1 -- 172.21.15.49:0/386425290 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f925c01dbb0 con 0x55df3dc09240 2024-03-18T00:58:46.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.701+0000 7f926b220740 1 -- 172.21.15.49:0/386425290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 msgr2=0x55df3dc0dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.701+0000 7f926b220740 1 --2- 172.21.15.49:0/386425290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc0dd40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f925c009ba0 tx=0x7f925c014000 comp rx=0 tx=0).stop 2024-03-18T00:58:46.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 -- 172.21.15.49:0/386425290 shutdown_connections 2024-03-18T00:58:46.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 --2- 172.21.15.49:0/386425290 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc10840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 --2- 172.21.15.49:0/386425290 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc0dd40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 --2- 172.21.15.49:0/386425290 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc08c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 -- 172.21.15.49:0/386425290 >> 172.21.15.49:0/386425290 conn(0x55df3dc03dc0 msgr2=0x55df3dc041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 -- 172.21.15.49:0/386425290 shutdown_connections 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 -- 172.21.15.49:0/386425290 wait complete. 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 -- start start 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 Processor -- start 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc08850 0x55df3dc12d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc13260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc0e3b0 0x55df3dc1a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55df3dc11360 con 0x55df3dc08850 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55df3db29230 con 0x55df3dc0e3b0 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55df3dc114e0 con 0x55df3dc09240 2024-03-18T00:58:46.707 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f92662fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc08850 0x55df3dc12d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f926b220740 10 client.0 _ll_register_callbacks cb 0x55df3db65010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f9265afb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc13260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f92662fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc08850 0x55df3dc12d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38178/0 (socket says 172.21.15.49:38178) 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f92662fc640 1 -- 172.21.15.49:0/4293150159 learned_addr learned my addr 172.21.15.49:0/4293150159 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.702+0000 7f9266afd640 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc0e3b0 0x55df3dc1a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f92662fc640 1 -- 172.21.15.49:0/4293150159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 msgr2=0x55df3dc13260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f92662fc640 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc13260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f92662fc640 1 -- 172.21.15.49:0/4293150159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc0e3b0 msgr2=0x55df3dc1a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f92662fc640 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc0e3b0 0x55df3dc1a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f92662fc640 1 -- 172.21.15.49:0/4293150159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df3db38100 con 0x55df3dc08850 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f92662fc640 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc08850 0x55df3dc12d20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9250000c00 tx=0x7f925009c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494b49e73695e38d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f9249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f9249ffb640 1 -- 172.21.15.49:0/4293150159 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500b3070 con 0x55df3dc08850 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f9249ffb640 1 -- 172.21.15.49:0/4293150159 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f92500860b0 con 0x55df3dc08850 2024-03-18T00:58:46.708 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f926b220740 1 -- 172.21.15.49:0/4293150159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f925c00fb00 con 0x55df3dc08850 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f9249ffb640 1 -- 172.21.15.49:0/4293150159 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f925009cb50 con 0x55df3dc08850 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f926b220740 10 client.17562 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.703+0000 7f926b220740 1 -- 172.21.15.49:0/4293150159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df3d8813f0 con 0x55df3dc08850 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f9249ffb640 1 -- 172.21.15.49:0/4293150159 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f925c00fb00 con 0x55df3dc08850 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f9249ffb640 1 -- 172.21.15.49:0/4293150159 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f925009a6e0 con 0x55df3dc08850 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f926b220740 10 client.17562 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f926b220740 10 client.17562 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f926b220740 20 client.17562 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f926b220740 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55df3dc1eaf0 0x55df3dc20f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f926b220740 4 client.17562 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f926b220740 1 -- 172.21.15.49:0/4293150159 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df3db7a9d0 con 0x55df3dc1eaf0 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f9265afb640 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55df3dc1eaf0 0x55df3dc20f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f9249ffb640 10 client.17562 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:58:46.709 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.704+0000 7f9265afb640 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55df3dc1eaf0 0x55df3dc20f90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2a7b3c17d76055d0 server_cookie=206d5ec5d71d1a26 in_seq=0 out_seq=0 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.705+0000 7f9249ffb640 1 -- 172.21.15.49:0/4293150159 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55df3db7a9d0 con 0x55df3dc1eaf0 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.705+0000 7f9249ffb640 10 client.17562 handle_command_reply: tid=0 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.705+0000 7f926b220740 1 client.17562 shutdown 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 -- 172.21.15.49:0/4293150159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc08850 msgr2=0x55df3dc12d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc08850 0x55df3dc12d20 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9250000c00 tx=0x7f925009c060 comp rx=0 tx=0).stop 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 -- 172.21.15.49:0/4293150159 shutdown_connections 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55df3dc1eaf0 0x55df3dc20f90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc0e3b0 0x55df3dc1a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc13260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 --2- 172.21.15.49:0/4293150159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc08850 0x55df3dc12d20 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f9249ffb640 0 client.17562 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 -- 172.21.15.49:0/4293150159 >> 172.21.15.49:0/4293150159 conn(0x55df3dc03dc0 msgr2=0x55df3dc02fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 -- 172.21.15.49:0/4293150159 shutdown_connections 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 1 -- 172.21.15.49:0/4293150159 wait complete. 2024-03-18T00:58:46.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.706+0000 7f926b220740 20 client.17562 trim_cache size 0 max 16384 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f926b220740 1 Processor -- start 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f926b220740 1 -- start start 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f926b220740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc12cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f926b220740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc1a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f926b220740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc13200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f926b220740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f92601ab0d0 con 0x55df3dc0e3b0 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f926b220740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f92601acc10 con 0x55df3dc08850 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f926b220740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f92601ab7a0 con 0x55df3dc09240 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f9266afd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc13200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f92662fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc12cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f9265afb640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc1a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f9266afd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc13200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38190/0 (socket says 172.21.15.49:38190) 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f92662fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc12cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:58806/0 (socket says 172.21.15.49:58806) 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.721+0000 7f9266afd640 1 -- 172.21.15.49:0/3985809468 learned_addr learned my addr 172.21.15.49:0/3985809468 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f9266afd640 1 -- 172.21.15.49:0/3985809468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 msgr2=0x55df3dc1a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f9266afd640 1 --2- 172.21.15.49:0/3985809468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc1a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f9266afd640 1 -- 172.21.15.49:0/3985809468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 msgr2=0x55df3dc12cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f9266afd640 1 --2- 172.21.15.49:0/3985809468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc12cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f9266afd640 1 -- 172.21.15.49:0/3985809468 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df3d94b7d0 con 0x55df3dc0e3b0 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f92662fc640 1 --2- 172.21.15.49:0/3985809468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc12cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f9265afb640 1 --2- 172.21.15.49:0/3985809468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc1a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f9266afd640 1 --2- 172.21.15.49:0/3985809468 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc13200 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f925802c700 tx=0x7f9258032590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7206ee3083d76a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f924bfff640 1 -- 172.21.15.49:0/3985809468 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258039040 con 0x55df3dc0e3b0 2024-03-18T00:58:46.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f924bfff640 1 -- 172.21.15.49:0/3985809468 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9258045070 con 0x55df3dc0e3b0 2024-03-18T00:58:46.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.722+0000 7f924bfff640 1 -- 172.21.15.49:0/3985809468 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580347f0 con 0x55df3dc0e3b0 2024-03-18T00:58:46.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 -- 172.21.15.49:0/3985809468 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 msgr2=0x55df3dc13200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 --2- 172.21.15.49:0/3985809468 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc13200 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f925802c700 tx=0x7f9258032590 comp rx=0 tx=0).stop 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 -- 172.21.15.49:0/3985809468 shutdown_connections 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 --2- 172.21.15.49:0/3985809468 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc13200 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 --2- 172.21.15.49:0/3985809468 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc1a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 --2- 172.21.15.49:0/3985809468 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc12cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 -- 172.21.15.49:0/3985809468 >> 172.21.15.49:0/3985809468 conn(0x55df3dc03dc0 msgr2=0x55df3dc041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 -- 172.21.15.49:0/3985809468 shutdown_connections 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 -- 172.21.15.49:0/3985809468 wait complete. 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 -- start start 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.723+0000 7f926b220740 1 Processor -- start 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f926b220740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc172a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f926b220740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc177e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f926b220740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc18780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f926b220740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f92601ac540 con 0x55df3dc0e3b0 2024-03-18T00:58:46.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f926b220740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f92601abe70 con 0x55df3dc08850 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f926b220740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f92601a90b0 con 0x55df3dc09240 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f926b220740 10 client.0 _ll_register_callbacks cb 0x7f925009bf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9266afd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc18780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f92662fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc172a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9266afd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc18780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38198/0 (socket says 172.21.15.49:38198) 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9266afd640 1 -- 172.21.15.49:0/4170572475 learned_addr learned my addr 172.21.15.49:0/4170572475 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9265afb640 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc177e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9266afd640 1 -- 172.21.15.49:0/4170572475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 msgr2=0x55df3dc177e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9266afd640 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc177e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9266afd640 1 -- 172.21.15.49:0/4170572475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 msgr2=0x55df3dc172a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9266afd640 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc172a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9266afd640 1 -- 172.21.15.49:0/4170572475 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df3dc18d70 con 0x55df3dc0e3b0 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9266afd640 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc18780 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f9258009bf0 tx=0x7f92580323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdeb2b4c5d260d53 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9249ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9249ffb640 1 -- 172.21.15.49:0/4170572475 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258039050 con 0x55df3dc0e3b0 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f9249ffb640 1 -- 172.21.15.49:0/4170572475 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f92580347f0 con 0x55df3dc0e3b0 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.724+0000 7f926b220740 1 -- 172.21.15.49:0/4170572475 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9258045070 con 0x55df3dc0e3b0 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f9249ffb640 1 -- 172.21.15.49:0/4170572475 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9258059070 con 0x55df3dc0e3b0 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f926b220740 10 client.17568 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f926b220740 1 -- 172.21.15.49:0/4170572475 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df3dc1a070 con 0x55df3dc0e3b0 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f9249ffb640 1 -- 172.21.15.49:0/4170572475 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9258045070 con 0x55df3dc0e3b0 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f9249ffb640 1 -- 172.21.15.49:0/4170572475 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9258053470 con 0x55df3dc0e3b0 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f926b220740 10 client.17568 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f926b220740 10 client.17568 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f926b220740 20 client.17568 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f926b220740 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55df3dc8d8d0 0x55df3dc8fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f926b220740 4 client.17568 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f926b220740 1 -- 172.21.15.49:0/4170572475 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55df3db74d20 con 0x55df3dc8d8d0 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.725+0000 7f92662fc640 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55df3dc8d8d0 0x55df3dc8fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f92662fc640 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55df3dc8d8d0 0x55df3dc8fcf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=87b7fcefe1a04422 server_cookie=c9411b5c7dd59d2a in_seq=0 out_seq=0 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f9249ffb640 10 client.17568 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f9249ffb640 1 -- 172.21.15.49:0/4170572475 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55df3db74d20 con 0x55df3dc8d8d0 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f9249ffb640 10 client.17568 handle_command_reply: tid=0 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 client.17568 shutdown 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 -- 172.21.15.49:0/4170572475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 msgr2=0x55df3dc18780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc18780 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f9258009bf0 tx=0x7f92580323c0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 -- 172.21.15.49:0/4170572475 shutdown_connections 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55df3dc8d8d0 0x55df3dc8fcf0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55df3dc0e3b0 0x55df3dc18780 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55df3dc09240 0x55df3dc177e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f9249ffb640 0 client.17568 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 --2- 172.21.15.49:0/4170572475 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55df3dc08850 0x55df3dc172a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 -- 172.21.15.49:0/4170572475 >> 172.21.15.49:0/4170572475 conn(0x55df3dc03dc0 msgr2=0x55df3dc0a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 -- 172.21.15.49:0/4170572475 shutdown_connections 2024-03-18T00:58:46.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 1 -- 172.21.15.49:0/4170572475 wait complete. 2024-03-18T00:58:46.732 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.726+0000 7f926b220740 20 client.17568 trim_cache size 0 max 16384 2024-03-18T00:58:46.732 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 -- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f924c0a2800 msgr2=0x7f924c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f924c0a2800 0x7f924c0a4cc0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f92601043e0 tx=0x7f925c005f90 comp rx=0 tx=0).stop 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 -- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92601006e0 msgr2=0x7f9260104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92601006e0 0x7f9260104670 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f925000e9c0 tx=0x7f925000ee80 comp rx=0 tx=0).stop 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 -- 172.21.15.49:0/2299380742 shutdown_connections 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f924c0a2800 0x7f924c0a4cc0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92601018d0 0x7f9260102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9260102290 0x7f92601031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 --2- 172.21.15.49:0/2299380742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92601006e0 0x7f9260104670 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 -- 172.21.15.49:0/2299380742 >> 172.21.15.49:0/2299380742 conn(0x7f92600fbc70 msgr2=0x7f92600fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 -- 172.21.15.49:0/2299380742 shutdown_connections 2024-03-18T00:58:46.733 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.727+0000 7f9268560640 1 -- 172.21.15.49:0/2299380742 wait complete. 2024-03-18T00:58:46.738 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.854+0000 7f17dee97640 1 -- 172.21.15.49:0/4004598516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8107580 msgr2=0x7f17d8109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.854+0000 7f17dee97640 1 --2- 172.21.15.49:0/4004598516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8107580 0x7f17d8109970 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009ca0 tx=0x7f17d002c0f0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.854+0000 7f17dee97640 1 -- 172.21.15.49:0/4004598516 shutdown_connections 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.854+0000 7f17dee97640 1 --2- 172.21.15.49:0/4004598516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8107580 0x7f17d8109970 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.854+0000 7f17dee97640 1 --2- 172.21.15.49:0/4004598516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17d8104c50 0x7f17d8107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.854+0000 7f17dee97640 1 --2- 172.21.15.49:0/4004598516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8102320 0x7f17d8104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.854+0000 7f17dee97640 1 -- 172.21.15.49:0/4004598516 >> 172.21.15.49:0/4004598516 conn(0x7f17d80fbc70 msgr2=0x7f17d80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.854+0000 7f17dee97640 1 -- 172.21.15.49:0/4004598516 shutdown_connections 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.854+0000 7f17dee97640 1 -- 172.21.15.49:0/4004598516 wait complete. 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dee97640 1 Processor -- start 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dee97640 1 -- start start 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dee97640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8102320 0x7f17d8101c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dee97640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8104c50 0x7f17d81002d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dee97640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17d8107580 0x7f17d8100810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dee97640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f17d810d470 con 0x7f17d8102320 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dee97640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f17d810d2f0 con 0x7f17d8104c50 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dee97640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17d810d5f0 con 0x7f17d8107580 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dcc33640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8102320 0x7f17d8101c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dcc33640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8102320 0x7f17d8101c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38226/0 (socket says 172.21.15.49:38226) 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dcc33640 1 -- 172.21.15.49:0/3601579095 learned_addr learned my addr 172.21.15.49:0/3601579095 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dd434640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17d8107580 0x7f17d8100810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17d7fff640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8104c50 0x7f17d81002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dcc33640 1 -- 172.21.15.49:0/3601579095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17d8107580 msgr2=0x7f17d8100810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dcc33640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17d8107580 0x7f17d8100810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dcc33640 1 -- 172.21.15.49:0/3601579095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8104c50 msgr2=0x7f17d81002d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.855+0000 7f17dcc33640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8104c50 0x7f17d81002d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.856+0000 7f17dcc33640 1 -- 172.21.15.49:0/3601579095 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d81010a0 con 0x7f17d8102320 2024-03-18T00:58:46.857 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.856+0000 7f17dd434640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17d8107580 0x7f17d8100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.856+0000 7f17dcc33640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8102320 0x7f17d8101c90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00ea60 tx=0x7f17cc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.856+0000 7f17d5ffb640 1 -- 172.21.15.49:0/3601579095 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc00cde0 con 0x7f17d8102320 2024-03-18T00:58:46.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.856+0000 7f17d5ffb640 1 -- 172.21.15.49:0/3601579095 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17cc0035f0 con 0x7f17d8102320 2024-03-18T00:58:46.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.856+0000 7f17dee97640 1 -- 172.21.15.49:0/3601579095 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d81a1c30 con 0x7f17d8102320 2024-03-18T00:58:46.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.856+0000 7f17d5ffb640 1 -- 172.21.15.49:0/3601579095 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc010670 con 0x7f17d8102320 2024-03-18T00:58:46.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.856+0000 7f17dee97640 1 -- 172.21.15.49:0/3601579095 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d81a2180 con 0x7f17d8102320 2024-03-18T00:58:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.859+0000 7f17d5ffb640 1 -- 172.21.15.49:0/3601579095 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f17cc0037a0 con 0x7f17d8102320 2024-03-18T00:58:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.859+0000 7f17e1a81740 1 Processor -- start 2024-03-18T00:58:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.859+0000 7f17e1a81740 1 -- start start 2024-03-18T00:58:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.859+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f3673ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.859+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 0x559f36743d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.859+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 0x559f36746840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.859+0000 7f17e1a81740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559f366a3490 con 0x559f3673e850 2024-03-18T00:58:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.859+0000 7f17e1a81740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559f3664bc90 con 0x559f367443b0 2024-03-18T00:58:46.860 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.859+0000 7f17e1a81740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559f3669f530 con 0x559f3673f240 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f3673ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f3673ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38236/0 (socket says 172.21.15.49:38236) 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 -- 172.21.15.49:0/1906962519 learned_addr learned my addr 172.21.15.49:0/1906962519 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17d5ffb640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17c80a2800 0x7f17c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dcc33640 1 --2- 172.21.15.49:0/1906962519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 0x559f36743d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17d7fff640 1 --2- 172.21.15.49:0/1906962519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 0x559f36746840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17c80a2800 0x7f17c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 -- 172.21.15.49:0/1906962519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 msgr2=0x559f36743d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 --2- 172.21.15.49:0/1906962519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 0x559f36743d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 -- 172.21.15.49:0/1906962519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 msgr2=0x559f36746840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17d5ffb640 1 -- 172.21.15.49:0/3601579095 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f17cc0cedd0 con 0x7f17d8102320 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 --2- 172.21.15.49:0/1906962519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 0x559f36746840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 -- 172.21.15.49:0/1906962519 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f36706500 con 0x559f3673e850 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17dd434640 1 --2- 172.21.15.49:0/1906962519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f3673ec70 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f17d00324c0 tx=0x7f17d002cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ecc40290be03a27 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17c2ffd640 1 -- 172.21.15.49:0/1906962519 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0050070 con 0x559f3673e850 2024-03-18T00:58:46.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17c2ffd640 1 -- 172.21.15.49:0/1906962519 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17d0034cd0 con 0x559f3673e850 2024-03-18T00:58:46.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.860+0000 7f17c2ffd640 1 -- 172.21.15.49:0/1906962519 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0039040 con 0x559f3673e850 2024-03-18T00:58:46.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 -- 172.21.15.49:0/1906962519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 msgr2=0x559f3673ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1906962519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f3673ec70 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f17d00324c0 tx=0x7f17d002cef0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17dd434640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17c80a2800 0x7f17c80a4cc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f17d004d000 tx=0x7f17d0048000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 -- 172.21.15.49:0/1906962519 shutdown_connections 2024-03-18T00:58:46.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1906962519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 0x559f36746840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1906962519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 0x559f36743d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1906962519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f3673ec70 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 -- 172.21.15.49:0/1906962519 >> 172.21.15.49:0/1906962519 conn(0x559f36739dc0 msgr2=0x559f3673a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 -- 172.21.15.49:0/1906962519 shutdown_connections 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 -- 172.21.15.49:0/1906962519 wait complete. 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.861+0000 7f17e1a81740 1 -- start start 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17e1a81740 1 Processor -- start 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f36748d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 0x559f36749290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f367443b0 0x559f36750250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17e1a81740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x559f36747360 con 0x559f3673e850 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17e1a81740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x559f3665f230 con 0x559f3673f240 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17e1a81740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x559f367474e0 con 0x559f367443b0 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17e1a81740 10 client.0 _ll_register_callbacks cb 0x559f3669b010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:46.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17dcc33640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f36748d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17dcc33640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f36748d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38252/0 (socket says 172.21.15.49:38252) 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17dcc33640 1 -- 172.21.15.49:0/1368891850 learned_addr learned my addr 172.21.15.49:0/1368891850 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17d7fff640 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 0x559f36749290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17dd434640 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f367443b0 0x559f36750250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17dcc33640 1 -- 172.21.15.49:0/1368891850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f367443b0 msgr2=0x559f36750250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17dcc33640 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f367443b0 0x559f36750250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17dcc33640 1 -- 172.21.15.49:0/1368891850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 msgr2=0x559f36749290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17dcc33640 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 0x559f36749290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.862+0000 7f17dcc33640 1 -- 172.21.15.49:0/1368891850 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f3666e100 con 0x559f3673e850 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17dcc33640 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f36748d50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00ea30 tx=0x7f17cc09b0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc43b8f9b370882 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17b7fff640 1 -- 172.21.15.49:0/1368891850 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc09f8a0 con 0x559f3673e850 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17b7fff640 1 -- 172.21.15.49:0/1368891850 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f17cc003b30 con 0x559f3673e850 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17e1a81740 1 -- 172.21.15.49:0/1368891850 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d0034cd0 con 0x559f3673e850 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17b7fff640 1 -- 172.21.15.49:0/1368891850 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f17cc0a3ef0 con 0x559f3673e850 2024-03-18T00:58:46.868 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17e1a81740 10 client.17580 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17e1a81740 1 -- 172.21.15.49:0/1368891850 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559f363b73f0 con 0x559f3673e850 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17b7fff640 1 -- 172.21.15.49:0/1368891850 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17d0034cd0 con 0x559f3673e850 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17b7fff640 1 -- 172.21.15.49:0/1368891850 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f17cc084ce0 con 0x559f3673e850 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17e1a81740 10 client.17580 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17e1a81740 10 client.17580 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17e1a81740 20 client.17580 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x559f36754bc0 0x559f36757010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17e1a81740 4 client.17580 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.863+0000 7f17e1a81740 1 -- 172.21.15.49:0/1368891850 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f366b09d0 con 0x559f36754bc0 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.864+0000 7f17d7fff640 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x559f36754bc0 0x559f36757010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.864+0000 7f17d7fff640 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x559f36754bc0 0x559f36757010 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dba1e3f954780466 server_cookie=25114a360ff9beee in_seq=0 out_seq=0 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.864+0000 7f17b7fff640 10 client.17580 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17b7fff640 1 -- 172.21.15.49:0/1368891850 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559f366b09d0 con 0x559f36754bc0 2024-03-18T00:58:46.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17b7fff640 10 client.17580 handle_command_reply: tid=0 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 client.17580 shutdown 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 -- 172.21.15.49:0/1368891850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 msgr2=0x559f36748d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f36748d50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00ea30 tx=0x7f17cc09b0b0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 -- 172.21.15.49:0/1368891850 shutdown_connections 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x559f36754bc0 0x559f36757010 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 0x559f36749290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f367443b0 0x559f36750250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17b7fff640 0 client.17580 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 --2- 172.21.15.49:0/1368891850 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f36748d50 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 -- 172.21.15.49:0/1368891850 >> 172.21.15.49:0/1368891850 conn(0x559f36739dc0 msgr2=0x559f36738fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.865+0000 7f17e1a81740 1 -- 172.21.15.49:0/1368891850 shutdown_connections 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.866+0000 7f17e1a81740 1 -- 172.21.15.49:0/1368891850 wait complete. 2024-03-18T00:58:46.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.866+0000 7f17e1a81740 20 client.17580 trim_cache size 0 max 16384 2024-03-18T00:58:46.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:46 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2952782180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:58:46.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.879+0000 7f17e1a81740 1 Processor -- start 2024-03-18T00:58:46.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.879+0000 7f17e1a81740 1 -- start start 2024-03-18T00:58:46.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.879+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f367491f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 0x559f367501e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 0x559f3673d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17e1a81740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f17d81aaf50 con 0x559f3673e850 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17e1a81740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f17d81aca90 con 0x559f367443b0 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17e1a81740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17d81ab620 con 0x559f3673f240 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dcc33640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f367491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 0x559f367501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dcc33640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f367491f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38260/0 (socket says 172.21.15.49:38260) 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17d7fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 0x559f367501e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44330/0 (socket says 172.21.15.49:44330) 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dd434640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 0x559f3673d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dcc33640 1 -- 172.21.15.49:0/2223344080 learned_addr learned my addr 172.21.15.49:0/2223344080 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dcc33640 1 -- 172.21.15.49:0/2223344080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 msgr2=0x559f367501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dcc33640 1 --2- 172.21.15.49:0/2223344080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 0x559f367501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dcc33640 1 -- 172.21.15.49:0/2223344080 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 msgr2=0x559f3673d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dcc33640 1 --2- 172.21.15.49:0/2223344080 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 0x559f3673d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dcc33640 1 -- 172.21.15.49:0/2223344080 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f364817d0 con 0x559f3673e850 2024-03-18T00:58:46.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dd434640 1 --2- 172.21.15.49:0/2223344080 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 0x559f3673d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17dcc33640 1 --2- 172.21.15.49:0/2223344080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f367491f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0ccc20 tx=0x7f17cc09b1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245be53c88dee195 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.880+0000 7f17c27fc640 1 -- 172.21.15.49:0/2223344080 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc09f8a0 con 0x559f3673e850 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17c27fc640 1 -- 172.21.15.49:0/2223344080 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17cc09b750 con 0x559f3673e850 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17c27fc640 1 -- 172.21.15.49:0/2223344080 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17cc097dc0 con 0x559f3673e850 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17e1a81740 1 -- 172.21.15.49:0/2223344080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 msgr2=0x559f367491f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17e1a81740 1 --2- 172.21.15.49:0/2223344080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f367491f0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0ccc20 tx=0x7f17cc09b1e0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17e1a81740 1 -- 172.21.15.49:0/2223344080 shutdown_connections 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17e1a81740 1 --2- 172.21.15.49:0/2223344080 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f367443b0 0x559f3673d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17e1a81740 1 --2- 172.21.15.49:0/2223344080 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673f240 0x559f367501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17e1a81740 1 --2- 172.21.15.49:0/2223344080 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f3673e850 0x559f367491f0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17e1a81740 1 -- 172.21.15.49:0/2223344080 >> 172.21.15.49:0/2223344080 conn(0x559f36739dc0 msgr2=0x559f3673d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17e1a81740 1 -- 172.21.15.49:0/2223344080 shutdown_connections 2024-03-18T00:58:46.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.881+0000 7f17e1a81740 1 -- 172.21.15.49:0/2223344080 wait complete. 2024-03-18T00:58:46.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17e1a81740 1 -- start start 2024-03-18T00:58:46.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17e1a81740 1 Processor -- start 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673e850 0x559f3674d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 0x559f3674dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17e1a81740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f367443b0 0x559f3674ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17e1a81740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f17d81ac3c0 con 0x559f367443b0 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17e1a81740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f17d81abcf0 con 0x559f3673f240 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17e1a81740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f17d81a8f30 con 0x559f3673e850 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17e1a81740 10 client.0 _ll_register_callbacks cb 0x7f17cc003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17dd434640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f367443b0 0x559f3674ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17dd434640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f367443b0 0x559f3674ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38266/0 (socket says 172.21.15.49:38266) 2024-03-18T00:58:46.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17dd434640 1 -- 172.21.15.49:0/779035160 learned_addr learned my addr 172.21.15.49:0/779035160 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17d7fff640 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 0x559f3674dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.882+0000 7f17dcc33640 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673e850 0x559f3674d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17dd434640 1 -- 172.21.15.49:0/779035160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673e850 msgr2=0x559f3674d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17dd434640 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673e850 0x559f3674d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17dd434640 1 -- 172.21.15.49:0/779035160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 msgr2=0x559f3674dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17dd434640 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 0x559f3674dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17dd434640 1 -- 172.21.15.49:0/779035160 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f3674f270 con 0x559f367443b0 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17dd434640 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f367443b0 0x559f3674ec80 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009ca0 tx=0x7f17d00323a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=468ac1cd65493b36 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17b7fff640 1 -- 172.21.15.49:0/779035160 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d0039050 con 0x559f367443b0 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17e1a81740 1 -- 172.21.15.49:0/779035160 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17cc09b750 con 0x559f367443b0 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17b7fff640 1 -- 172.21.15.49:0/779035160 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f17d0003970 con 0x559f367443b0 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17b7fff640 1 -- 172.21.15.49:0/779035160 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f17d0063050 con 0x559f367443b0 2024-03-18T00:58:46.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17e1a81740 10 client.17586 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17e1a81740 1 -- 172.21.15.49:0/779035160 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559f367baf10 con 0x559f367443b0 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17b7fff640 1 -- 172.21.15.49:0/779035160 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17cc09b750 con 0x559f367443b0 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.883+0000 7f17b7fff640 1 -- 172.21.15.49:0/779035160 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f17d003f070 con 0x559f367443b0 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17e1a81740 10 client.17586 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17e1a81740 10 client.17586 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17e1a81740 20 client.17586 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17e1a81740 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x559f367bb420 0x559f367bd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17e1a81740 4 client.17586 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17e1a81740 1 -- 172.21.15.49:0/779035160 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f366aad20 con 0x559f367bb420 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17dcc33640 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x559f367bb420 0x559f367bd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17b7fff640 10 client.17586 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17dcc33640 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x559f367bb420 0x559f367bd810 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ca68839efd93c912 server_cookie=52d8050bfd07f50b in_seq=0 out_seq=0 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17b7fff640 1 -- 172.21.15.49:0/779035160 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f366aad20 con 0x559f367bb420 2024-03-18T00:58:46.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17b7fff640 10 client.17586 handle_command_reply: tid=0 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.884+0000 7f17e1a81740 1 client.17586 shutdown 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 -- 172.21.15.49:0/779035160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f367443b0 msgr2=0x559f3674ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f367443b0 0x559f3674ec80 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009ca0 tx=0x7f17d00323a0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 -- 172.21.15.49:0/779035160 shutdown_connections 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x559f367bb420 0x559f367bd810 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x559f3673f240 0x559f3674dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x559f367443b0 0x559f3674ec80 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17b7fff640 0 client.17586 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 --2- 172.21.15.49:0/779035160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x559f3673e850 0x559f3674d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 -- 172.21.15.49:0/779035160 >> 172.21.15.49:0/779035160 conn(0x559f36739dc0 msgr2=0x559f36748fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 -- 172.21.15.49:0/779035160 shutdown_connections 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 1 -- 172.21.15.49:0/779035160 wait complete. 2024-03-18T00:58:46.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17e1a81740 20 client.17586 trim_cache size 0 max 16384 2024-03-18T00:58:46.891 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:46.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17dee97640 1 -- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17c80a2800 msgr2=0x7f17c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17dee97640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17c80a2800 0x7f17c80a4cc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f17d004d000 tx=0x7f17d0048000 comp rx=0 tx=0).stop 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17dee97640 1 -- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8102320 msgr2=0x7f17d8101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.885+0000 7f17dee97640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8102320 0x7f17d8101c90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00ea60 tx=0x7f17cc00ef20 comp rx=0 tx=0).stop 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.886+0000 7f17dee97640 1 -- 172.21.15.49:0/3601579095 shutdown_connections 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.886+0000 7f17dee97640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17c80a2800 0x7f17c80a4cc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.886+0000 7f17dee97640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f17d8104c50 0x7f17d81002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.886+0000 7f17dee97640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17d8107580 0x7f17d8100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.886+0000 7f17dee97640 1 --2- 172.21.15.49:0/3601579095 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f17d8102320 0x7f17d8101c90 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.886+0000 7f17dee97640 1 -- 172.21.15.49:0/3601579095 >> 172.21.15.49:0/3601579095 conn(0x7f17d80fbc70 msgr2=0x7f17d8102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.886+0000 7f17dee97640 1 -- 172.21.15.49:0/3601579095 shutdown_connections 2024-03-18T00:58:46.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:46.886+0000 7f17dee97640 1 -- 172.21.15.49:0/3601579095 wait complete. 2024-03-18T00:58:46.897 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:58:46.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:46 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2952782180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:58:46.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:46 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2952782180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:58:47.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.011+0000 7f92c93b2640 1 -- 172.21.15.49:0/2993360143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41022b0 msgr2=0x7f92c4108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.011+0000 7f92c93b2640 1 --2- 172.21.15.49:0/2993360143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41022b0 0x7f92c4108b40 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f92bc009ca0 tx=0x7f92bc02c0f0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.011+0000 7f92c93b2640 1 -- 172.21.15.49:0/2993360143 shutdown_connections 2024-03-18T00:58:47.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.011+0000 7f92c93b2640 1 --2- 172.21.15.49:0/2993360143 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92c41018f0 0x7f92c4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.011+0000 7f92c93b2640 1 --2- 172.21.15.49:0/2993360143 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41022b0 0x7f92c4108b40 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.011+0000 7f92c93b2640 1 --2- 172.21.15.49:0/2993360143 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92c4100700 0x7f92c4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.011+0000 7f92c93b2640 1 -- 172.21.15.49:0/2993360143 >> 172.21.15.49:0/2993360143 conn(0x7f92c40fbc90 msgr2=0x7f92c40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.011+0000 7f92c93b2640 1 -- 172.21.15.49:0/2993360143 shutdown_connections 2024-03-18T00:58:47.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.011+0000 7f92c93b2640 1 -- 172.21.15.49:0/2993360143 wait complete. 2024-03-18T00:58:47.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c93b2640 1 Processor -- start 2024-03-18T00:58:47.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c93b2640 1 -- start start 2024-03-18T00:58:47.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c93b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92c4100700 0x7f92c4199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c93b2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41018f0 0x7f92c419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c93b2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92c41022b0 0x7f92c41a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c93b2640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f92c410c380 con 0x7f92c41018f0 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c93b2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f92c410c200 con 0x7f92c41022b0 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c93b2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f92c410c500 con 0x7f92c4100700 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41018f0 0x7f92c419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41018f0 0x7f92c419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38286/0 (socket says 172.21.15.49:38286) 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92c4100700 0x7f92c4199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c27fc640 1 -- 172.21.15.49:0/17283979 learned_addr learned my addr 172.21.15.49:0/17283979 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c37fe640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92c41022b0 0x7f92c41a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c27fc640 1 -- 172.21.15.49:0/17283979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92c4100700 msgr2=0x7f92c4199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c27fc640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92c4100700 0x7f92c4199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c27fc640 1 -- 172.21.15.49:0/17283979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92c41022b0 msgr2=0x7f92c41a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c27fc640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92c41022b0 0x7f92c41a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.012+0000 7f92c27fc640 1 -- 172.21.15.49:0/17283979 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c41a18d0 con 0x7f92c41018f0 2024-03-18T00:58:47.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.013+0000 7f92c27fc640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41018f0 0x7f92c419a1d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f92b800e9f0 tx=0x7f92b800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.013+0000 7f92abfff640 1 -- 172.21.15.49:0/17283979 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b800cd80 con 0x7f92c41018f0 2024-03-18T00:58:47.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.013+0000 7f92abfff640 1 -- 172.21.15.49:0/17283979 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92b8010910 con 0x7f92c41018f0 2024-03-18T00:58:47.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.013+0000 7f92c93b2640 1 -- 172.21.15.49:0/17283979 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c41a1bc0 con 0x7f92c41018f0 2024-03-18T00:58:47.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.013+0000 7f92abfff640 1 -- 172.21.15.49:0/17283979 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8018990 con 0x7f92c41018f0 2024-03-18T00:58:47.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.013+0000 7f92c93b2640 1 -- 172.21.15.49:0/17283979 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c41a2110 con 0x7f92c41018f0 2024-03-18T00:58:47.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.015+0000 7f92abfff640 1 -- 172.21.15.49:0/17283979 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f92b8018b30 con 0x7f92c41018f0 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92cbfa0740 1 Processor -- start 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92cbfa0740 1 -- start start 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f334c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f339d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f33c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x560e6f299490 con 0x560e6f33a3b0 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92abfff640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f92ac0a2800 0x7f92ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560e6f241c90 con 0x560e6f334850 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560e6f295530 con 0x560e6f335240 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92c27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f33c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92c2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f339d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92c37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f334c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92c2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f339d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44380/0 (socket says 172.21.15.49:44380) 2024-03-18T00:58:47.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92c37fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f334c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:58922/0 (socket says 172.21.15.49:58922) 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92c2ffd640 1 -- 172.21.15.49:0/832586625 learned_addr learned my addr 172.21.15.49:0/832586625 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92abfff640 1 -- 172.21.15.49:0/17283979 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f92b80cedb0 con 0x7f92c41018f0 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.016+0000 7f92c37fe640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f92ac0a2800 0x7f92ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92c27fc640 1 -- 172.21.15.49:0/832586625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 msgr2=0x560e6f339d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92c27fc640 1 --2- 172.21.15.49:0/832586625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f339d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92c27fc640 1 -- 172.21.15.49:0/832586625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 msgr2=0x560e6f334c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92c27fc640 1 --2- 172.21.15.49:0/832586625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f334c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92c27fc640 1 -- 172.21.15.49:0/832586625 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e6f2fc500 con 0x560e6f33a3b0 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92c37fe640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f92ac0a2800 0x7f92ac0a4cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f92bc005e30 tx=0x7f92bc032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92c27fc640 1 --2- 172.21.15.49:0/832586625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f33c840 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f92b809b770 tx=0x7f92b8018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e62f6a92d091d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92a97fa640 1 -- 172.21.15.49:0/832586625 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b80ae130 con 0x560e6f33a3b0 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92a97fa640 1 -- 172.21.15.49:0/832586625 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92b8013890 con 0x560e6f33a3b0 2024-03-18T00:58:47.018 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.017+0000 7f92a97fa640 1 -- 172.21.15.49:0/832586625 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8099ae0 con 0x560e6f33a3b0 2024-03-18T00:58:47.022 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 -- 172.21.15.49:0/832586625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 msgr2=0x560e6f33c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/832586625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f33c840 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f92b809b770 tx=0x7f92b8018420 comp rx=0 tx=0).stop 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 -- 172.21.15.49:0/832586625 shutdown_connections 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/832586625 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f33c840 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/832586625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f339d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/832586625 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f334c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 -- 172.21.15.49:0/832586625 >> 172.21.15.49:0/832586625 conn(0x560e6f32fdc0 msgr2=0x560e6f3301c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 -- 172.21.15.49:0/832586625 shutdown_connections 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 -- 172.21.15.49:0/832586625 wait complete. 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 -- start start 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.018+0000 7f92cbfa0740 1 Processor -- start 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f33ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f33a3b0 0x560e6f346290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x560e6f33d300 con 0x560e6f335240 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x560e6f255230 con 0x560e6f334850 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x560e6f33d480 con 0x560e6f33a3b0 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92cbfa0740 10 client.0 _ll_register_callbacks cb 0x560e6f291010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f33ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38298/0 (socket says 172.21.15.49:38298) 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f33ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:58928/0 (socket says 172.21.15.49:58928) 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c27fc640 1 -- 172.21.15.49:0/3135166783 learned_addr learned my addr 172.21.15.49:0/3135166783 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c37fe640 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f33a3b0 0x560e6f346290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c27fc640 1 -- 172.21.15.49:0/3135166783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f33a3b0 msgr2=0x560e6f346290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c27fc640 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f33a3b0 0x560e6f346290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c27fc640 1 -- 172.21.15.49:0/3135166783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 msgr2=0x560e6f33ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c27fc640 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f33ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c27fc640 1 -- 172.21.15.49:0/3135166783 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e6f264100 con 0x560e6f335240 2024-03-18T00:58:47.024 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c2ffd640 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f33ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92c27fc640 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33f290 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c6a0 tx=0x7f92b8005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9c10ec2dc96b01 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3135166783 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b80a6060 con 0x560e6f335240 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.019+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3135166783 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f92b8086110 con 0x560e6f335240 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3135166783 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560e6f2a69d0 con 0x560e6f335240 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3135166783 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f92b80ad050 con 0x560e6f335240 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92cbfa0740 10 client.17598 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3135166783 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560e6efad3f0 con 0x560e6f335240 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3135166783 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560e6f2a69d0 con 0x560e6f335240 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3135166783 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f92b80b3200 con 0x560e6f335240 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92cbfa0740 10 client.17598 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92cbfa0740 10 client.17598 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92cbfa0740 20 client.17598 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x560e6f34a7a0 0x560e6f34cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92cbfa0740 4 client.17598 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:47.025 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3135166783 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e6f2a0d20 con 0x560e6f34a7a0 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.020+0000 7f92c2ffd640 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x560e6f34a7a0 0x560e6f34cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.021+0000 7f92c2ffd640 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x560e6f34a7a0 0x560e6f34cb90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d1fee1fba1546d17 server_cookie=3d259db2ec92f0c3 in_seq=0 out_seq=0 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.021+0000 7f92a6ff5640 10 client.17598 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3135166783 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560e6f2a0d20 con 0x560e6f34a7a0 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92a6ff5640 10 client.17598 handle_command_reply: tid=0 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 client.17598 shutdown 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3135166783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 msgr2=0x560e6f33f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33f290 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f92b800c6a0 tx=0x7f92b8005ab0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3135166783 shutdown_connections 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x560e6f34a7a0 0x560e6f34cb90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f33a3b0 0x560e6f346290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33f290 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3135166783 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f33ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.026 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92a6ff5640 0 client.17598 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:58:47.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3135166783 >> 172.21.15.49:0/3135166783 conn(0x560e6f32fdc0 msgr2=0x560e6f32efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3135166783 shutdown_connections 2024-03-18T00:58:47.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3135166783 wait complete. 2024-03-18T00:58:47.027 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.022+0000 7f92cbfa0740 20 client.17598 trim_cache size 0 max 16384 2024-03-18T00:58:47.036 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92cbfa0740 1 Processor -- start 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92cbfa0740 1 -- start start 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f334850 0x560e6f33d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f33a3b0 0x560e6f346160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41aaf80 con 0x560e6f335240 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41acac0 con 0x560e6f33a3b0 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41ab650 con 0x560e6f334850 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38308/0 (socket says 172.21.15.49:38308) 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c27fc640 1 -- 172.21.15.49:0/4063516404 learned_addr learned my addr 172.21.15.49:0/4063516404 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c37fe640 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f33a3b0 0x560e6f346160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c2ffd640 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f334850 0x560e6f33d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c27fc640 1 -- 172.21.15.49:0/4063516404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f334850 msgr2=0x560e6f33d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c27fc640 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f334850 0x560e6f33d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.037 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c27fc640 1 -- 172.21.15.49:0/4063516404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f33a3b0 msgr2=0x560e6f346160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c27fc640 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f33a3b0 0x560e6f346160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c27fc640 1 -- 172.21.15.49:0/4063516404 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e6f0777d0 con 0x560e6f335240 2024-03-18T00:58:47.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.036+0000 7f92c2ffd640 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f334850 0x560e6f33d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92c27fc640 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33dd60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f92b80cc890 tx=0x7f92b8005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736297aece202430 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92a8ff9640 1 -- 172.21.15.49:0/4063516404 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b80a3a90 con 0x560e6f335240 2024-03-18T00:58:47.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92a8ff9640 1 -- 172.21.15.49:0/4063516404 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92b809b7e0 con 0x560e6f335240 2024-03-18T00:58:47.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92a8ff9640 1 -- 172.21.15.49:0/4063516404 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b8097e50 con 0x560e6f335240 2024-03-18T00:58:47.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92cbfa0740 1 -- 172.21.15.49:0/4063516404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 msgr2=0x560e6f33dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33dd60 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f92b80cc890 tx=0x7f92b8005b20 comp rx=0 tx=0).stop 2024-03-18T00:58:47.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92cbfa0740 1 -- 172.21.15.49:0/4063516404 shutdown_connections 2024-03-18T00:58:47.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f33a3b0 0x560e6f346160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f335240 0x560e6f33dd60 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/4063516404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f334850 0x560e6f33d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.037+0000 7f92cbfa0740 1 -- 172.21.15.49:0/4063516404 >> 172.21.15.49:0/4063516404 conn(0x560e6f32fdc0 msgr2=0x560e6f33ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 -- 172.21.15.49:0/4063516404 shutdown_connections 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 -- 172.21.15.49:0/4063516404 wait complete. 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 -- start start 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 Processor -- start 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f343150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f343690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f3ae920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41ac3f0 con 0x560e6f33a3b0 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41abd20 con 0x560e6f334850 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f92c41a8f60 con 0x560e6f335240 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92cbfa0740 10 client.0 _ll_register_callbacks cb 0x7f92b8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92c37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f3ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92c27fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f343690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92c2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f343150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.043 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92c2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f343150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:58950/0 (socket says 172.21.15.49:58950) 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92c37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f3ae920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38318/0 (socket says 172.21.15.49:38318) 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.038+0000 7f92c2ffd640 1 -- 172.21.15.49:0/3006446088 learned_addr learned my addr 172.21.15.49:0/3006446088 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92c2ffd640 1 -- 172.21.15.49:0/3006446088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 msgr2=0x560e6f343690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92c2ffd640 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f343690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92c2ffd640 1 -- 172.21.15.49:0/3006446088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 msgr2=0x560e6f3ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92c2ffd640 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f3ae920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92c2ffd640 1 -- 172.21.15.49:0/3006446088 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e6f3aee60 con 0x560e6f334850 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92c37fe640 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f3ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92c27fc640 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f343690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92c2ffd640 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f343150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92b4012580 tx=0x7f92b4012af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da9a20f10f6c4962 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92a6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3006446088 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b4022410 con 0x560e6f334850 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3006446088 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92b809b7e0 con 0x560e6f334850 2024-03-18T00:58:47.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3006446088 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f92b40225b0 con 0x560e6f334850 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3006446088 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f92b4032050 con 0x560e6f334850 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92cbfa0740 10 client.26744 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.039+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3006446088 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560e6f3b00f0 con 0x560e6f334850 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3006446088 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92b809b7e0 con 0x560e6f334850 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3006446088 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f92b40105f0 con 0x560e6f334850 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92cbfa0740 10 client.26744 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92cbfa0740 10 client.26744 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92cbfa0740 20 client.26744 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x560e6f3b0a80 0x560e6f3b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92cbfa0740 4 client.26744 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3006446088 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e6f21e6a0 con 0x560e6f3b0a80 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92c27fc640 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x560e6f3b0a80 0x560e6f3b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92c27fc640 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x560e6f3b0a80 0x560e6f3b2ea0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d443f4e7a1faaba3 server_cookie=c4b7922ecce16736 in_seq=0 out_seq=0 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.040+0000 7f92a6ff5640 10 client.26744 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:58:47.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92a6ff5640 1 -- 172.21.15.49:0/3006446088 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e6f21e6a0 con 0x560e6f3b0a80 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92a6ff5640 10 client.26744 handle_command_reply: tid=0 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 client.26744 shutdown 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3006446088 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 msgr2=0x560e6f343150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f343150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92b4012580 tx=0x7f92b4012af0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3006446088 shutdown_connections 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x560e6f3b0a80 0x560e6f3b2ea0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x560e6f33a3b0 0x560e6f3ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560e6f335240 0x560e6f343690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 --2- 172.21.15.49:0/3006446088 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x560e6f334850 0x560e6f343150 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92a6ff5640 0 client.26744 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3006446088 >> 172.21.15.49:0/3006446088 conn(0x560e6f32fdc0 msgr2=0x560e6f33ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3006446088 shutdown_connections 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 1 -- 172.21.15.49:0/3006446088 wait complete. 2024-03-18T00:58:47.046 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92cbfa0740 20 client.26744 trim_cache size 0 max 16384 2024-03-18T00:58:47.047 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:47.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92c93b2640 1 -- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f92ac0a2800 msgr2=0x7f92ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92c93b2640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f92ac0a2800 0x7f92ac0a4cc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f92bc005e30 tx=0x7f92bc032590 comp rx=0 tx=0).stop 2024-03-18T00:58:47.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92c93b2640 1 -- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41018f0 msgr2=0x7f92c419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.047 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.041+0000 7f92c93b2640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41018f0 0x7f92c419a1d0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f92b800e9f0 tx=0x7f92b800eeb0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.042+0000 7f92c93b2640 1 -- 172.21.15.49:0/17283979 shutdown_connections 2024-03-18T00:58:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.042+0000 7f92c93b2640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f92ac0a2800 0x7f92ac0a4cc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.042+0000 7f92c93b2640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f92c41022b0 0x7f92c41a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.042+0000 7f92c93b2640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f92c41018f0 0x7f92c419a1d0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.042+0000 7f92c93b2640 1 --2- 172.21.15.49:0/17283979 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f92c4100700 0x7f92c4199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.042+0000 7f92c93b2640 1 -- 172.21.15.49:0/17283979 >> 172.21.15.49:0/17283979 conn(0x7f92c40fbc90 msgr2=0x7f92c40fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.042+0000 7f92c93b2640 1 -- 172.21.15.49:0/17283979 shutdown_connections 2024-03-18T00:58:47.048 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.042+0000 7f92c93b2640 1 -- 172.21.15.49:0/17283979 wait complete. 2024-03-18T00:58:47.053 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:58:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.169+0000 7f878bfc6640 1 -- 172.21.15.49:0/319651319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 msgr2=0x7f8784108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.169+0000 7f878bfc6640 1 --2- 172.21.15.49:0/319651319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 0x7f8784108b60 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f877c009be0 tx=0x7f877c02bef0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.169+0000 7f878bfc6640 1 -- 172.21.15.49:0/319651319 shutdown_connections 2024-03-18T00:58:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.169+0000 7f878bfc6640 1 --2- 172.21.15.49:0/319651319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8784101910 0x7f8784101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.169+0000 7f878bfc6640 1 --2- 172.21.15.49:0/319651319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 0x7f8784108b60 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.169+0000 7f878bfc6640 1 --2- 172.21.15.49:0/319651319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784100720 0x7f8784100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.169+0000 7f878bfc6640 1 -- 172.21.15.49:0/319651319 >> 172.21.15.49:0/319651319 conn(0x7f87840fbc90 msgr2=0x7f87840fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.169+0000 7f878bfc6640 1 -- 172.21.15.49:0/319651319 shutdown_connections 2024-03-18T00:58:47.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.169+0000 7f878bfc6640 1 -- 172.21.15.49:0/319651319 wait complete. 2024-03-18T00:58:47.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878bfc6640 1 Processor -- start 2024-03-18T00:58:47.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878bfc6640 1 -- start start 2024-03-18T00:58:47.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878bfc6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784100720 0x7f8784197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878bfc6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8784101910 0x7f8784198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878bfc6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 0x7f878419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878bfc6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f878410c3a0 con 0x7f87841022d0 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878bfc6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f878410c220 con 0x7f8784101910 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878bfc6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f878410c520 con 0x7f8784100720 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878a563640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 0x7f878419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f8789561640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8784101910 0x7f8784198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f8789d62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784100720 0x7f8784197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f8789561640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8784101910 0x7f8784198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:58972/0 (socket says 172.21.15.49:58972) 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f8789561640 1 -- 172.21.15.49:0/640694074 learned_addr learned my addr 172.21.15.49:0/640694074 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f8789d62640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784100720 0x7f8784197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44438/0 (socket says 172.21.15.49:44438) 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f878a563640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 0x7f878419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38336/0 (socket says 172.21.15.49:38336) 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f8789561640 1 -- 172.21.15.49:0/640694074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784100720 msgr2=0x7f8784197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f8789561640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784100720 0x7f8784197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f8789561640 1 -- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 msgr2=0x7f878419f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.170+0000 7f8789561640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 0x7f878419f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.171+0000 7f8789561640 1 -- 172.21.15.49:0/640694074 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878419f720 con 0x7f8784101910 2024-03-18T00:58:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.171+0000 7f878a563640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 0x7f878419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.171+0000 7f8789561640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8784101910 0x7f8784198040 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8774009f50 tx=0x7f877400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.171+0000 7f877affd640 1 -- 172.21.15.49:0/640694074 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774020070 con 0x7f8784101910 2024-03-18T00:58:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.171+0000 7f878bfc6640 1 -- 172.21.15.49:0/640694074 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8784102b90 con 0x7f8784101910 2024-03-18T00:58:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.171+0000 7f877affd640 1 -- 172.21.15.49:0/640694074 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8774013cb0 con 0x7f8784101910 2024-03-18T00:58:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.171+0000 7f877affd640 1 -- 172.21.15.49:0/640694074 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877401bb40 con 0x7f8784101910 2024-03-18T00:58:47.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.171+0000 7f878bfc6640 1 -- 172.21.15.49:0/640694074 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8784103130 con 0x7f8784101910 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.173+0000 7f877affd640 1 -- 172.21.15.49:0/640694074 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8774023070 con 0x7f8784101910 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878ebb4740 1 Processor -- start 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878ebb4740 1 -- start start 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 0x55a740328c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740329240 0x55a74032dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a74032e3b0 0x55a740330840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878ebb4740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a74028d490 con 0x55a740329240 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878ebb4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a740235c90 con 0x55a74032e3b0 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878ebb4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a740289530 con 0x55a740328850 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f877affd640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87700a2800 0x7f87700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f8789d62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740329240 0x55a74032dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f8789d62640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740329240 0x55a74032dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38338/0 (socket says 172.21.15.49:38338) 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f8789d62640 1 -- 172.21.15.49:0/564418139 learned_addr learned my addr 172.21.15.49:0/564418139 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878a563640 1 --2- 172.21.15.49:0/564418139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 0x55a740328c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f8789561640 1 --2- 172.21.15.49:0/564418139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a74032e3b0 0x55a740330840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.174+0000 7f878a563640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87700a2800 0x7f87700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f877affd640 1 -- 172.21.15.49:0/640694074 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f87740d2b20 con 0x7f8784101910 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f8789d62640 1 -- 172.21.15.49:0/564418139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 msgr2=0x55a740328c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f8789d62640 1 --2- 172.21.15.49:0/564418139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 0x55a740328c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f8789d62640 1 -- 172.21.15.49:0/564418139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a74032e3b0 msgr2=0x55a740330840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f8789d62640 1 --2- 172.21.15.49:0/564418139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a74032e3b0 0x55a740330840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f8789d62640 1 -- 172.21.15.49:0/564418139 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7402f0500 con 0x55a740329240 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f8789d62640 1 --2- 172.21.15.49:0/564418139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740329240 0x55a74032dd40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8780009ca0 tx=0x7f8780011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697bd088d981d566 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f878a563640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87700a2800 0x7f87700a4cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55a740329aa0 tx=0x7f877c004a80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f876ffff640 1 -- 172.21.15.49:0/564418139 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8780004220 con 0x55a740329240 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f876ffff640 1 -- 172.21.15.49:0/564418139 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87800043c0 con 0x55a740329240 2024-03-18T00:58:47.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.175+0000 7f876ffff640 1 -- 172.21.15.49:0/564418139 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8780019810 con 0x55a740329240 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 -- 172.21.15.49:0/564418139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740329240 msgr2=0x55a74032dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 --2- 172.21.15.49:0/564418139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740329240 0x55a74032dd40 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8780009ca0 tx=0x7f8780011810 comp rx=0 tx=0).stop 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 -- 172.21.15.49:0/564418139 shutdown_connections 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 --2- 172.21.15.49:0/564418139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a74032e3b0 0x55a740330840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 --2- 172.21.15.49:0/564418139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740329240 0x55a74032dd40 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 --2- 172.21.15.49:0/564418139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 0x55a740328c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 -- 172.21.15.49:0/564418139 >> 172.21.15.49:0/564418139 conn(0x55a740323dc0 msgr2=0x55a7403241c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 -- 172.21.15.49:0/564418139 shutdown_connections 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 -- 172.21.15.49:0/564418139 wait complete. 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 -- start start 2024-03-18T00:58:47.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.176+0000 7f878ebb4740 1 Processor -- start 2024-03-18T00:58:47.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 0x55a740332ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a7403333e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740338410 0x55a74033a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f878ebb4740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55a740331360 con 0x55a74032e3b0 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f878ebb4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55a740249230 con 0x55a740338410 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f878ebb4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55a7403314e0 con 0x55a740328850 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f878ebb4740 10 client.0 _ll_register_callbacks cb 0x55a740285010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789561640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a7403333e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789561640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a7403333e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38354/0 (socket says 172.21.15.49:38354) 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789561640 1 -- 172.21.15.49:0/2610830800 learned_addr learned my addr 172.21.15.49:0/2610830800 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789d62640 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 0x55a740332ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f878a563640 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740338410 0x55a74033a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789561640 1 -- 172.21.15.49:0/2610830800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 msgr2=0x55a740332ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789561640 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 0x55a740332ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789561640 1 -- 172.21.15.49:0/2610830800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740338410 msgr2=0x55a74033a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789561640 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740338410 0x55a74033a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789561640 1 -- 172.21.15.49:0/2610830800 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a740258100 con 0x55a74032e3b0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f8789561640 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a7403333e0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f87740d0600 tx=0x7f877406a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36871652e379a839 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.177+0000 7f876d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f876d7fa640 1 -- 172.21.15.49:0/2610830800 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877409eeb0 con 0x55a74032e3b0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f878ebb4740 1 -- 172.21.15.49:0/2610830800 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87800043c0 con 0x55a74032e3b0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f876d7fa640 1 -- 172.21.15.49:0/2610830800 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f877409bbc0 con 0x55a74032e3b0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f876d7fa640 1 -- 172.21.15.49:0/2610830800 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f877409f550 con 0x55a74032e3b0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f878ebb4740 10 client.17616 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f878ebb4740 1 -- 172.21.15.49:0/2610830800 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a73ffa13f0 con 0x55a74032e3b0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f876d7fa640 1 -- 172.21.15.49:0/2610830800 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f87800043c0 con 0x55a74032e3b0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f876d7fa640 1 -- 172.21.15.49:0/2610830800 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f877401bce0 con 0x55a74032e3b0 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f878ebb4740 10 client.17616 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f878ebb4740 10 client.17616 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:58:47.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f878ebb4740 20 client.17616 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.178+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a74033f130 0x55a7403415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.179+0000 7f878ebb4740 4 client.17616 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.179+0000 7f878ebb4740 1 -- 172.21.15.49:0/2610830800 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a74029a9d0 con 0x55a74033f130 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.179+0000 7f8789d62640 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a74033f130 0x55a7403415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.179+0000 7f8789d62640 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a74033f130 0x55a7403415c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86ac064b74f4ca02 server_cookie=476c65e11d7b767 in_seq=0 out_seq=0 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.179+0000 7f876d7fa640 10 client.17616 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f876d7fa640 1 -- 172.21.15.49:0/2610830800 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a74029a9d0 con 0x55a74033f130 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f876d7fa640 10 client.17616 handle_command_reply: tid=0 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f878ebb4740 1 client.17616 shutdown 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f878ebb4740 1 -- 172.21.15.49:0/2610830800 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 msgr2=0x55a7403333e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a7403333e0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f87740d0600 tx=0x7f877406a7b0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f878ebb4740 1 -- 172.21.15.49:0/2610830800 shutdown_connections 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a74033f130 0x55a7403415c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740338410 0x55a74033a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a7403333e0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2610830800 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740328850 0x55a740332ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f876d7fa640 0 client.17616 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:58:47.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.180+0000 7f878ebb4740 1 -- 172.21.15.49:0/2610830800 >> 172.21.15.49:0/2610830800 conn(0x55a740323dc0 msgr2=0x55a740322fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.181+0000 7f878ebb4740 1 -- 172.21.15.49:0/2610830800 shutdown_connections 2024-03-18T00:58:47.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.181+0000 7f878ebb4740 1 -- 172.21.15.49:0/2610830800 wait complete. 2024-03-18T00:58:47.185 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.181+0000 7f878ebb4740 20 client.17616 trim_cache size 0 max 16384 2024-03-18T00:58:47.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.195+0000 7f878ebb4740 1 Processor -- start 2024-03-18T00:58:47.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.195+0000 7f878ebb4740 1 -- start start 2024-03-18T00:58:47.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.195+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740329d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.195+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a74032e3b0 0x55a740322750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.195+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740338410 0x55a740322c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.195+0000 7f878ebb4740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f87841a6bb0 con 0x55a740338410 2024-03-18T00:58:47.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.195+0000 7f878ebb4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f87841a86f0 con 0x55a740328850 2024-03-18T00:58:47.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.195+0000 7f878ebb4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f87841a7280 con 0x55a74032e3b0 2024-03-18T00:58:47.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f878a563640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740338410 0x55a740322c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f8789d62640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740329d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f878a563640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740338410 0x55a740322c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38360/0 (socket says 172.21.15.49:38360) 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f878a563640 1 -- 172.21.15.49:0/2737318247 learned_addr learned my addr 172.21.15.49:0/2737318247 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f8789561640 1 --2- 172.21.15.49:0/2737318247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a74032e3b0 0x55a740322750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f878a563640 1 -- 172.21.15.49:0/2737318247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a74032e3b0 msgr2=0x55a740322750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f878a563640 1 --2- 172.21.15.49:0/2737318247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a74032e3b0 0x55a740322750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f878a563640 1 -- 172.21.15.49:0/2737318247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 msgr2=0x55a740329d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f878a563640 1 --2- 172.21.15.49:0/2737318247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740329d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f878a563640 1 -- 172.21.15.49:0/2737318247 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a74006b7d0 con 0x55a740338410 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f8789d62640 1 --2- 172.21.15.49:0/2737318247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740329d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f878a563640 1 --2- 172.21.15.49:0/2737318247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740338410 0x55a740322c90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f877c036040 tx=0x7f877c033f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246ae1a4385cce0f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f876f7fe640 1 -- 172.21.15.49:0/2737318247 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c02ce20 con 0x55a740338410 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f876f7fe640 1 -- 172.21.15.49:0/2737318247 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f877c003d40 con 0x55a740338410 2024-03-18T00:58:47.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.196+0000 7f876f7fe640 1 -- 172.21.15.49:0/2737318247 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c0043c0 con 0x55a740338410 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.197+0000 7f878ebb4740 1 -- 172.21.15.49:0/2737318247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740338410 msgr2=0x55a740322c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.197+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2737318247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740338410 0x55a740322c90 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f877c036040 tx=0x7f877c033f40 comp rx=0 tx=0).stop 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.197+0000 7f878ebb4740 1 -- 172.21.15.49:0/2737318247 shutdown_connections 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.197+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2737318247 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a740338410 0x55a740322c90 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.197+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2737318247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a74032e3b0 0x55a740322750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.197+0000 7f878ebb4740 1 --2- 172.21.15.49:0/2737318247 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740329d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.197+0000 7f878ebb4740 1 -- 172.21.15.49:0/2737318247 >> 172.21.15.49:0/2737318247 conn(0x55a740323dc0 msgr2=0x55a7403318b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.197+0000 7f878ebb4740 1 -- 172.21.15.49:0/2737318247 shutdown_connections 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.197+0000 7f878ebb4740 1 -- 172.21.15.49:0/2737318247 wait complete. 2024-03-18T00:58:47.202 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878ebb4740 1 -- start start 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878ebb4740 1 Processor -- start 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740350850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a740350d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878ebb4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740338410 0x55a740351be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878ebb4740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f87841a8020 con 0x55a74032e3b0 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878ebb4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f87841a7950 con 0x55a740328850 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878ebb4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f87841a4b90 con 0x55a740338410 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878ebb4740 10 client.0 _ll_register_callbacks cb 0x7f877400cc40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f8789561640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a740350d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f8789d62640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740350850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f8789561640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a740350d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38362/0 (socket says 172.21.15.49:38362) 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f8789561640 1 -- 172.21.15.49:0/3046689448 learned_addr learned my addr 172.21.15.49:0/3046689448 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.198+0000 7f878a563640 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740338410 0x55a740351be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f8789561640 1 -- 172.21.15.49:0/3046689448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740338410 msgr2=0x55a740351be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.203 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f8789561640 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740338410 0x55a740351be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f8789561640 1 -- 172.21.15.49:0/3046689448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 msgr2=0x55a740350850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f8789561640 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740350850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f8789561640 1 -- 172.21.15.49:0/3046689448 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7403522a0 con 0x55a74032e3b0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f8789d62640 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740350850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f8789561640 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a740350d90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f877409f890 tx=0x7f877409f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699878a20bcf7963 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f876d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f876d7fa640 1 -- 172.21.15.49:0/3046689448 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877409eeb0 con 0x55a74032e3b0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f876d7fa640 1 -- 172.21.15.49:0/3046689448 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f877409bbc0 con 0x55a74032e3b0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f878ebb4740 1 -- 172.21.15.49:0/3046689448 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f877c003d40 con 0x55a74032e3b0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f876d7fa640 1 -- 172.21.15.49:0/3046689448 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f87740a7c40 con 0x55a74032e3b0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f878ebb4740 10 client.17622 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f878ebb4740 1 -- 172.21.15.49:0/3046689448 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a7403a58b0 con 0x55a74032e3b0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f876d7fa640 1 -- 172.21.15.49:0/3046689448 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f877c003d40 con 0x55a74032e3b0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f876d7fa640 1 -- 172.21.15.49:0/3046689448 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f877401bce0 con 0x55a74032e3b0 2024-03-18T00:58:47.204 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f878ebb4740 10 client.17622 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f878ebb4740 10 client.17622 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.199+0000 7f878ebb4740 20 client.17622 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.200+0000 7f878ebb4740 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a7403a6260 0x55a7403a8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.200+0000 7f878ebb4740 4 client.17622 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.200+0000 7f878ebb4740 1 -- 172.21.15.49:0/3046689448 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a740294d20 con 0x55a7403a6260 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.200+0000 7f8789d62640 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a7403a6260 0x55a7403a8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.200+0000 7f8789d62640 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a7403a6260 0x55a7403a8680 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73d6b822dc4ce22f server_cookie=56e2a0b595c0b4af in_seq=0 out_seq=0 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.200+0000 7f876d7fa640 10 client.17622 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.200+0000 7f876d7fa640 1 -- 172.21.15.49:0/3046689448 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a740294d20 con 0x55a7403a6260 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.200+0000 7f876d7fa640 10 client.17622 handle_command_reply: tid=0 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.200+0000 7f878ebb4740 1 client.17622 shutdown 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 -- 172.21.15.49:0/3046689448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 msgr2=0x55a740350d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a740350d90 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f877409f890 tx=0x7f877409f150 comp rx=0 tx=0).stop 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 -- 172.21.15.49:0/3046689448 shutdown_connections 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x55a7403a6260 0x55a7403a8680 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.205 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a740338410 0x55a740351be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55a74032e3b0 0x55a740350d90 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 --2- 172.21.15.49:0/3046689448 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55a740328850 0x55a740350850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f876d7fa640 0 client.17622 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:58:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 -- 172.21.15.49:0/3046689448 >> 172.21.15.49:0/3046689448 conn(0x55a740323dc0 msgr2=0x55a7403318b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 -- 172.21.15.49:0/3046689448 shutdown_connections 2024-03-18T00:58:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 1 -- 172.21.15.49:0/3046689448 wait complete. 2024-03-18T00:58:47.206 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878ebb4740 20 client.17622 trim_cache size 0 max 16384 2024-03-18T00:58:47.206 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878bfc6640 1 -- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87700a2800 msgr2=0x7f87700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878bfc6640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87700a2800 0x7f87700a4cc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55a740329aa0 tx=0x7f877c004a80 comp rx=0 tx=0).stop 2024-03-18T00:58:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878bfc6640 1 -- 172.21.15.49:0/640694074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8784101910 msgr2=0x7f8784198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878bfc6640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8784101910 0x7f8784198040 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8774009f50 tx=0x7f877400ed90 comp rx=0 tx=0).stop 2024-03-18T00:58:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878bfc6640 1 -- 172.21.15.49:0/640694074 shutdown_connections 2024-03-18T00:58:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878bfc6640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f87700a2800 0x7f87700a4cc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.207 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878bfc6640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8784101910 0x7f8784198040 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878bfc6640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f87841022d0 0x7f878419f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.201+0000 7f878bfc6640 1 --2- 172.21.15.49:0/640694074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8784100720 0x7f8784197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.202+0000 7f878bfc6640 1 -- 172.21.15.49:0/640694074 >> 172.21.15.49:0/640694074 conn(0x7f87840fbc90 msgr2=0x7f87840fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.202+0000 7f878bfc6640 1 -- 172.21.15.49:0/640694074 shutdown_connections 2024-03-18T00:58:47.208 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.202+0000 7f878bfc6640 1 -- 172.21.15.49:0/640694074 wait complete. 2024-03-18T00:58:47.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T00:58:47.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.329+0000 7f7bf509a640 1 -- 172.21.15.49:0/3665911858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf0102290 msgr2=0x7f7bf0108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.329+0000 7f7bf509a640 1 --2- 172.21.15.49:0/3665911858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf0102290 0x7f7bf0108b20 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f7be8009be0 tx=0x7f7be802bef0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.329+0000 7f7bf509a640 1 -- 172.21.15.49:0/3665911858 shutdown_connections 2024-03-18T00:58:47.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.329+0000 7f7bf509a640 1 --2- 172.21.15.49:0/3665911858 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf0102290 0x7f7bf0108b20 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.329+0000 7f7bf509a640 1 --2- 172.21.15.49:0/3665911858 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bf01018d0 0x7f7bf0101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.329+0000 7f7bf509a640 1 --2- 172.21.15.49:0/3665911858 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf01006e0 0x7f7bf0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.329+0000 7f7bf509a640 1 -- 172.21.15.49:0/3665911858 >> 172.21.15.49:0/3665911858 conn(0x7f7bf00fbc70 msgr2=0x7f7bf00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.329+0000 7f7bf509a640 1 -- 172.21.15.49:0/3665911858 shutdown_connections 2024-03-18T00:58:47.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.329+0000 7f7bf509a640 1 -- 172.21.15.49:0/3665911858 wait complete. 2024-03-18T00:58:47.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.330+0000 7f7bf509a640 1 Processor -- start 2024-03-18T00:58:47.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.330+0000 7f7bf509a640 1 -- start start 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.330+0000 7f7bf509a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf01006e0 0x7f7bf01969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.330+0000 7f7bf509a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bf01018d0 0x7f7bf0196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.330+0000 7f7bf509a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf0102290 0x7f7bf019df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.330+0000 7f7bf509a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf010c360 con 0x7f7bf01006e0 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.330+0000 7f7bf509a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf010c1e0 con 0x7f7bf0102290 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.330+0000 7f7bf509a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf010c4e0 con 0x7f7bf01018d0 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7beed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf01006e0 0x7f7bf01969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7bee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bf01018d0 0x7f7bf0196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7bef59e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf0102290 0x7f7bf019df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7beed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf01006e0 0x7f7bf01969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38386/0 (socket says 172.21.15.49:38386) 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7beed9d640 1 -- 172.21.15.49:0/1977906494 learned_addr learned my addr 172.21.15.49:0/1977906494 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7beed9d640 1 -- 172.21.15.49:0/1977906494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bf01018d0 msgr2=0x7f7bf0196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7beed9d640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bf01018d0 0x7f7bf0196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7beed9d640 1 -- 172.21.15.49:0/1977906494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf0102290 msgr2=0x7f7bf019df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7beed9d640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf0102290 0x7f7bf019df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7beed9d640 1 -- 172.21.15.49:0/1977906494 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf019e5f0 con 0x7f7bf01006e0 2024-03-18T00:58:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7bef59e640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf0102290 0x7f7bf019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7beed9d640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf01006e0 0x7f7bf01969d0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7be000e9f0 tx=0x7f7be000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7bd7fff640 1 -- 172.21.15.49:0/1977906494 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be000dad0 con 0x7f7bf01006e0 2024-03-18T00:58:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7bf509a640 1 -- 172.21.15.49:0/1977906494 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf00689a0 con 0x7f7bf01006e0 2024-03-18T00:58:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7bd7fff640 1 -- 172.21.15.49:0/1977906494 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be000dc70 con 0x7f7bf01006e0 2024-03-18T00:58:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7bd7fff640 1 -- 172.21.15.49:0/1977906494 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be0003da0 con 0x7f7bf01006e0 2024-03-18T00:58:47.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.331+0000 7f7bf509a640 1 -- 172.21.15.49:0/1977906494 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf0068f40 con 0x7f7bf01006e0 2024-03-18T00:58:47.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bd7fff640 1 -- 172.21.15.49:0/1977906494 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7be001d020 con 0x7f7bf01006e0 2024-03-18T00:58:47.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bf7d66740 1 Processor -- start 2024-03-18T00:58:47.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bf7d66740 1 -- start start 2024-03-18T00:58:47.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b00850 0x557826b00c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 0x557826b05d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b08840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557826a65490 con 0x557826b00850 2024-03-18T00:58:47.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557826a0dc90 con 0x557826b01240 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bd7fff640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7bd80a2800 0x7f7bd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.335+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557826a61530 con 0x557826b063b0 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b00850 0x557826b00c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7beed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 0x557826b05d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b00850 0x557826b00c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:38398/0 (socket says 172.21.15.49:38398) 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 -- 172.21.15.49:0/3196644082 learned_addr learned my addr 172.21.15.49:0/3196644082 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bee59c640 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b08840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7bd80a2800 0x7f7bd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bd7fff640 1 -- 172.21.15.49:0/1977906494 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7be00cdfe0 con 0x7f7bf01006e0 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 -- 172.21.15.49:0/3196644082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 msgr2=0x557826b08840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b08840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 -- 172.21.15.49:0/3196644082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 msgr2=0x557826b05d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 0x557826b05d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 -- 172.21.15.49:0/3196644082 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557826ac8500 con 0x557826b00850 2024-03-18T00:58:47.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bee59c640 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b08840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7beed9d640 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 0x557826b05d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b00850 0x557826b00c70 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f7be8005cc0 tx=0x7f7be802da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e1c29837d6a4c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.336+0000 7f7bef59e640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7bd80a2800 0x7f7bd80a4cc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7be80099a0 tx=0x7f7be8049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.337+0000 7f7bd57fa640 1 -- 172.21.15.49:0/3196644082 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be8003e00 con 0x557826b00850 2024-03-18T00:58:47.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.337+0000 7f7bd57fa640 1 -- 172.21.15.49:0/3196644082 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be8051070 con 0x557826b00850 2024-03-18T00:58:47.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.337+0000 7f7bd57fa640 1 -- 172.21.15.49:0/3196644082 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be802ce30 con 0x557826b00850 2024-03-18T00:58:47.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3196644082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b00850 msgr2=0x557826b00c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:47.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b00850 0x557826b00c70 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f7be8005cc0 tx=0x7f7be802da50 comp rx=0 tx=0).stop 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3196644082 shutdown_connections 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 0x557826b05d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b08840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/3196644082 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b00850 0x557826b00c70 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3196644082 >> 172.21.15.49:0/3196644082 conn(0x557826afbdc0 msgr2=0x557826afc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3196644082 shutdown_connections 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3196644082 wait complete. 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 -- start start 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.338+0000 7f7bf7d66740 1 Processor -- start 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b122a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557826b09300 con 0x557826b063b0 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557826a21230 con 0x557826b00850 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557826b09480 con 0x557826b01240 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bf7d66740 10 client.0 _ll_register_callbacks cb 0x557826a5d010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7beed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bef59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b122a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bee59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:44518/0 (socket says 172.21.15.49:44518) 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7beed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:59036/0 (socket says 172.21.15.49:59036) 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bee59c640 1 -- 172.21.15.49:0/2696466912 learned_addr learned my addr 172.21.15.49:0/2696466912 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bef59e640 1 -- 172.21.15.49:0/2696466912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b01240 msgr2=0x557826b0b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bef59e640 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.343 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bef59e640 1 -- 172.21.15.49:0/2696466912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 msgr2=0x557826b0ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bef59e640 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bef59e640 1 -- 172.21.15.49:0/2696466912 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557826a30100 con 0x557826b063b0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7beed9d640 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bee59c640 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bef59e640 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b122a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f7be8005cc0 tx=0x7f7be802dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a997a95f21854d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.339+0000 7f7bd2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/2696466912 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be805b070 con 0x557826b063b0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/2696466912 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7be802cd30 con 0x557826b063b0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bf7d66740 1 -- 172.21.15.49:0/2696466912 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7be8051070 con 0x557826b063b0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/2696466912 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7be8004b40 con 0x557826b063b0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bf7d66740 10 client.17634 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bf7d66740 1 -- 172.21.15.49:0/2696466912 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5578267793f0 con 0x557826b063b0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/2696466912 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be8051070 con 0x557826b063b0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/2696466912 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7be805e030 con 0x557826b063b0 2024-03-18T00:58:47.344 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bf7d66740 10 client.17634 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:47.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bf7d66740 10 client.17634 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:47.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bf7d66740 20 client.17634 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:47.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557826b167b0 0x557826b18ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:47.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bf7d66740 4 client.17634 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:47.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.340+0000 7f7bf7d66740 1 -- 172.21.15.49:0/2696466912 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557826a729d0 con 0x557826b167b0 2024-03-18T00:58:47.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.341+0000 7f7beed9d640 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557826b167b0 0x557826b18ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:47.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.341+0000 7f7bd2ff5640 10 client.17634 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:47.345 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:47.341+0000 7f7beed9d640 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557826b167b0 0x557826b18ba0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd1846a5952e2d24 server_cookie=2f34fb4d41e9befa in_seq=0 out_seq=0 2024-03-18T00:58:47.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:47 smithi049 ceph-mon[32130]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 83 MiB/s wr, 981 op/s 2024-03-18T00:58:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:58:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:58:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:58:47.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:58:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:47 smithi107 ceph-mon[36270]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 83 MiB/s wr, 981 op/s 2024-03-18T00:58:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:58:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:58:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:58:47.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:58:47.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:47 smithi106 ceph-mon[35899]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 83 MiB/s wr, 981 op/s 2024-03-18T00:58:47.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:58:47.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:58:47.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:58:47.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:58:49.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:49 smithi049 ceph-mon[32130]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 78 MiB/s wr, 918 op/s 2024-03-18T00:58:49.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:49 smithi107 ceph-mon[36270]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 78 MiB/s wr, 918 op/s 2024-03-18T00:58:49.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:49 smithi106 ceph-mon[35899]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 78 MiB/s wr, 918 op/s 2024-03-18T00:58:51.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:51 smithi049 ceph-mon[32130]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 66 MiB/s wr, 773 op/s 2024-03-18T00:58:51.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:51 smithi107 ceph-mon[36270]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 66 MiB/s wr, 773 op/s 2024-03-18T00:58:51.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:51 smithi106 ceph-mon[35899]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 66 MiB/s wr, 773 op/s 2024-03-18T00:58:53.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:53 smithi049 ceph-mon[32130]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 62 MiB/s wr, 721 op/s 2024-03-18T00:58:53.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:53 smithi107 ceph-mon[36270]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 62 MiB/s wr, 721 op/s 2024-03-18T00:58:53.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:53 smithi106 ceph-mon[35899]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 62 MiB/s wr, 721 op/s 2024-03-18T00:58:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fda86ffd640 1 -- 172.21.15.49:0/217355853 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55abe1a222d0 con 0x55abe1acb040 2024-03-18T00:58:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fda86ffd640 10 client.26776 handle_command_reply: tid=0 2024-03-18T00:58:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 client.26776 shutdown 2024-03-18T00:58:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 -- 172.21.15.49:0/217355853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 msgr2=0x55abe1ac4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1ac4a80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fda9802c170 tx=0x7fda980346d0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 -- 172.21.15.49:0/217355853 shutdown_connections 2024-03-18T00:58:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55abe1acb040 0x55abe1b33e60 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55abe1abaea0 0x55abe1ac5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55abe1ab5d30 0x55abe1ac4a80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 --2- 172.21.15.49:0/217355853 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55abe1ab5340 0x55abe1ac4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fda86ffd640 0 client.26776 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:54.703 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 -- 172.21.15.49:0/217355853 >> 172.21.15.49:0/217355853 conn(0x55abe1ab08b0 msgr2=0x55abe1aaf320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:54.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.701+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/2696466912 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557826a729d0 con 0x557826b167b0 2024-03-18T00:58:54.703 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.701+0000 7f7bd2ff5640 10 client.17634 handle_command_reply: tid=0 2024-03-18T00:58:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 -- 172.21.15.49:0/217355853 shutdown_connections 2024-03-18T00:58:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 1 -- 172.21.15.49:0/217355853 wait complete. 2024-03-18T00:58:54.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.701+0000 7f7bf7d66740 1 client.17634 shutdown 2024-03-18T00:58:54.704 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.701+0000 7fdab0a4d740 20 client.26776 trim_cache size 0 max 16384 2024-03-18T00:58:54.704 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.701+0000 7f7bf7d66740 1 -- 172.21.15.49:0/2696466912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 msgr2=0x557826b122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:54.705 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.701+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b122a0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f7be8005cc0 tx=0x7f7be802dd60 comp rx=0 tx=0).stop 2024-03-18T00:58:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:58:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:58:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:58:54.705 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bf7d66740 1 -- 172.21.15.49:0/2696466912 shutdown_connections 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557826b167b0 0x557826b18ba0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b122a0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/2696466912 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bd2ff5640 0 client.17634 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bf7d66740 1 -- 172.21.15.49:0/2696466912 >> 172.21.15.49:0/2696466912 conn(0x557826afbdc0 msgr2=0x557826afafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bf7d66740 1 -- 172.21.15.49:0/2696466912 shutdown_connections 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bf7d66740 1 -- 172.21.15.49:0/2696466912 wait complete. 2024-03-18T00:58:54.706 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.702+0000 7f7bf7d66740 20 client.17634 trim_cache size 0 max 16384 2024-03-18T00:58:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 -- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda900a28a0 msgr2=0x7fda900a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda900a28a0 0x7fda900a4d60 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0097f20 tx=0x7fdaa0066880 comp rx=0 tx=0).stop 2024-03-18T00:58:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 -- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaa81022b0 msgr2=0x7fdaa8103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaa81022b0 0x7fdaa8103210 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000eb10 tx=0x7fdaa000c2d0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 -- 172.21.15.49:0/3569032640 shutdown_connections 2024-03-18T00:58:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fda900a28a0 0x7fda900a4d60 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fdaa81022b0 0x7fdaa8103210 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdaa81018f0 0x7fdaa8102cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.707 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 --2- 172.21.15.49:0/3569032640 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fdaa8100700 0x7fdaa8104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.702+0000 7fdaade6a640 1 -- 172.21.15.49:0/3569032640 >> 172.21.15.49:0/3569032640 conn(0x7fdaa80fbc70 msgr2=0x7fdaa80fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.703+0000 7fdaade6a640 1 -- 172.21.15.49:0/3569032640 shutdown_connections 2024-03-18T00:58:54.708 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:58:54.703+0000 7fdaade6a640 1 -- 172.21.15.49:0/3569032640 wait complete. 2024-03-18T00:58:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bf7d66740 1 Processor -- start 2024-03-18T00:58:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bf7d66740 1 -- start start 2024-03-18T00:58:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b12260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf01a6b20 con 0x557826b01240 2024-03-18T00:58:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf01a8660 con 0x557826b00850 2024-03-18T00:58:54.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf01a71f0 con 0x557826b063b0 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bef59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b12260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bee59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7beed9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bef59e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b12260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38790/0 (socket says 172.21.15.49:38790) 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bef59e640 1 -- 172.21.15.49:0/3477040605 learned_addr learned my addr 172.21.15.49:0/3477040605 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bee59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48690/0 (socket says 172.21.15.49:48690) 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.718+0000 7f7bef59e640 1 -- 172.21.15.49:0/3477040605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 msgr2=0x557826b0ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.719+0000 7f7bef59e640 1 --2- 172.21.15.49:0/3477040605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.719+0000 7f7bef59e640 1 -- 172.21.15.49:0/3477040605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b01240 msgr2=0x557826b0b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.719+0000 7f7bef59e640 1 --2- 172.21.15.49:0/3477040605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.719+0000 7f7bef59e640 1 -- 172.21.15.49:0/3477040605 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578268437d0 con 0x557826b063b0 2024-03-18T00:58:54.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.719+0000 7f7bef59e640 1 --2- 172.21.15.49:0/3477040605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b12260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7be802c430 tx=0x7f7be8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=962ee55c68ef07d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.719+0000 7f7bd4ff9640 1 -- 172.21.15.49:0/3477040605 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be805b070 con 0x557826b063b0 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.719+0000 7f7bd4ff9640 1 -- 172.21.15.49:0/3477040605 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be8051070 con 0x557826b063b0 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.719+0000 7f7bd4ff9640 1 -- 172.21.15.49:0/3477040605 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be802cd90 con 0x557826b063b0 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3477040605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 msgr2=0x557826b12260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/3477040605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b12260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7be802c430 tx=0x7f7be8004a40 comp rx=0 tx=0).stop 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3477040605 shutdown_connections 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/3477040605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b063b0 0x557826b12260 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/3477040605 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b01240 0x557826b0b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/3477040605 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b00850 0x557826b0ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3477040605 >> 172.21.15.49:0/3477040605 conn(0x557826afbdc0 msgr2=0x557826afc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3477040605 shutdown_connections 2024-03-18T00:58:54.724 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 -- 172.21.15.49:0/3477040605 wait complete. 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.720+0000 7f7bf7d66740 1 -- start start 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bf7d66740 1 Processor -- start 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b00850 0x557826b0f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 0x557826b0f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bf7d66740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b7aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf01a7f90 con 0x557826b063b0 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf01a78c0 con 0x557826b01240 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bf7d66740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf01a4b00 con 0x557826b00850 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bf7d66740 10 client.0 _ll_register_callbacks cb 0x7f7be009a7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bef59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b7aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7beed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b00850 0x557826b0f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bee59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 0x557826b0f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bef59e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b7aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48698/0 (socket says 172.21.15.49:48698) 2024-03-18T00:58:54.725 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bef59e640 1 -- 172.21.15.49:0/627405454 learned_addr learned my addr 172.21.15.49:0/627405454 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7beed9d640 1 -- 172.21.15.49:0/627405454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 msgr2=0x557826b0f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7beed9d640 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 0x557826b0f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7beed9d640 1 -- 172.21.15.49:0/627405454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 msgr2=0x557826b7aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7beed9d640 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b7aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7beed9d640 1 -- 172.21.15.49:0/627405454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557826b7af70 con 0x557826b00850 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.721+0000 7f7bef59e640 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b7aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7beed9d640 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b00850 0x557826b0f2c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7be00cbbb0 tx=0x7f7be006a520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb22328137bdfd24 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bd2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/627405454 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be00a31a0 con 0x557826b00850 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/627405454 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7be0098d00 con 0x557826b00850 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bf7d66740 1 -- 172.21.15.49:0/627405454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7be8051070 con 0x557826b00850 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/627405454 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f7be0097070 con 0x557826b00850 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bf7d66740 10 client.26872 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:54.726 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bf7d66740 1 -- 172.21.15.49:0/627405454 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557826b7c230 con 0x557826b00850 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/627405454 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7be8051070 con 0x557826b00850 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/627405454 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f7be00a2d10 con 0x557826b00850 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bf7d66740 10 client.26872 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bf7d66740 10 client.26872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bf7d66740 20 client.26872 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557826b7cbe0 0x557826b7f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bf7d66740 4 client.26872 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.722+0000 7f7bf7d66740 1 -- 172.21.15.49:0/627405454 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557826a6cd20 con 0x557826b7cbe0 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.723+0000 7f7bee59c640 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557826b7cbe0 0x557826b7f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.723+0000 7f7bee59c640 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557826b7cbe0 0x557826b7f000 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4abbdb47bb844c6c server_cookie=f12a58f2e1c5d8c1 in_seq=0 out_seq=0 2024-03-18T00:58:54.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.723+0000 7f7bd2ff5640 10 client.26872 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:54.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.725+0000 7f7bd2ff5640 1 -- 172.21.15.49:0/627405454 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557826a6cd20 con 0x557826b7cbe0 2024-03-18T00:58:54.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.725+0000 7f7bd2ff5640 10 client.26872 handle_command_reply: tid=0 2024-03-18T00:58:54.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.725+0000 7f7bf7d66740 1 client.26872 shutdown 2024-03-18T00:58:54.728 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 -- 172.21.15.49:0/627405454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b00850 msgr2=0x557826b0f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b00850 0x557826b0f2c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7be00cbbb0 tx=0x7f7be006a520 comp rx=0 tx=0).stop 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 -- 172.21.15.49:0/627405454 shutdown_connections 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557826b7cbe0 0x557826b7f000 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557826b01240 0x557826b0f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557826b063b0 0x557826b7aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 --2- 172.21.15.49:0/627405454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557826b00850 0x557826b0f2c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bd2ff5640 0 client.26872 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 -- 172.21.15.49:0/627405454 >> 172.21.15.49:0/627405454 conn(0x557826afbdc0 msgr2=0x557826b0ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 -- 172.21.15.49:0/627405454 shutdown_connections 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 1 -- 172.21.15.49:0/627405454 wait complete. 2024-03-18T00:58:54.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf7d66740 20 client.26872 trim_cache size 0 max 16384 2024-03-18T00:58:54.730 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:58:54.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf509a640 1 -- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7bd80a2800 msgr2=0x7f7bd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:54.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf509a640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7bd80a2800 0x7f7bd80a4cc0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7be80099a0 tx=0x7f7be8049000 comp rx=0 tx=0).stop 2024-03-18T00:58:54.730 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf509a640 1 -- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf01006e0 msgr2=0x7f7bf01969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:54.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.726+0000 7f7bf509a640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf01006e0 0x7f7bf01969d0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f7be000e9f0 tx=0x7f7be000eeb0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.727+0000 7f7bf509a640 1 -- 172.21.15.49:0/1977906494 shutdown_connections 2024-03-18T00:58:54.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.727+0000 7f7bf509a640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f7bd80a2800 0x7f7bd80a4cc0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.727+0000 7f7bf509a640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7bf0102290 0x7f7bf019df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.727+0000 7f7bf509a640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7bf01018d0 0x7f7bf0196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.727+0000 7f7bf509a640 1 --2- 172.21.15.49:0/1977906494 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f7bf01006e0 0x7f7bf01969d0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:54.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.727+0000 7f7bf509a640 1 -- 172.21.15.49:0/1977906494 >> 172.21.15.49:0/1977906494 conn(0x7f7bf00fbc70 msgr2=0x7f7bf0104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:54.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.727+0000 7f7bf509a640 1 -- 172.21.15.49:0/1977906494 shutdown_connections 2024-03-18T00:58:54.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:54.727+0000 7f7bf509a640 1 -- 172.21.15.49:0/1977906494 wait complete. 2024-03-18T00:58:55.742 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T00:58:55.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T00:58:55.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T00:58:55.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.870+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3210771146 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 msgr2=0x7f1e04100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:55.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.870+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3210771146 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 0x7f1e04100b00 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009900 tx=0x7f1e0002bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:55.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.870+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3210771146 shutdown_connections 2024-03-18T00:58:55.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.870+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3210771146 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e041018d0 0x7f1e04101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.870+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3210771146 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e04102290 0x7f1e04108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.870+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3210771146 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 0x7f1e04100b00 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.870+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3210771146 >> 172.21.15.49:0/3210771146 conn(0x7f1e040fbc70 msgr2=0x7f1e040fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:55.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.870+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3210771146 shutdown_connections 2024-03-18T00:58:55.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.870+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3210771146 wait complete. 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0c1c4640 1 Processor -- start 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0c1c4640 1 -- start start 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0c1c4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 0x7f1e04104670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0c1c4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e041018d0 0x7f1e04102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0c1c4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e04102290 0x7f1e041031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0c1c4640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0410c360 con 0x7f1e041006e0 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0c1c4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0410c1e0 con 0x7f1e041018d0 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0c1c4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0410c4e0 con 0x7f1e04102290 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e09f60640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 0x7f1e04104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e09f60640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 0x7f1e04104670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48716/0 (socket says 172.21.15.49:48716) 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e09f60640 1 -- 172.21.15.49:0/3772345862 learned_addr learned my addr 172.21.15.49:0/3772345862 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:55.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0a761640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e04102290 0x7f1e041031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.871+0000 7f1e0975f640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e041018d0 0x7f1e04102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1e09f60640 1 -- 172.21.15.49:0/3772345862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e04102290 msgr2=0x7f1e041031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1e09f60640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e04102290 0x7f1e041031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1e09f60640 1 -- 172.21.15.49:0/3772345862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e041018d0 msgr2=0x7f1e04102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1e09f60640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e041018d0 0x7f1e04102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1e09f60640 1 -- 172.21.15.49:0/3772345862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e04103a80 con 0x7f1e041006e0 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1e09f60640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 0x7f1e04104670 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1e00000c00 tx=0x7f1e0002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1dfaffd640 1 -- 172.21.15.49:0/3772345862 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00041070 con 0x7f1e041006e0 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3772345862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e041a1c00 con 0x7f1e041006e0 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1dfaffd640 1 -- 172.21.15.49:0/3772345862 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e00004450 con 0x7f1e041006e0 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1dfaffd640 1 -- 172.21.15.49:0/3772345862 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e00034440 con 0x7f1e041006e0 2024-03-18T00:58:55.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.872+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3772345862 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e041a2150 con 0x7f1e041006e0 2024-03-18T00:58:55.876 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:55 smithi049 ceph-mon[32130]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 29 MiB/s wr, 335 op/s 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.874+0000 7f1dfaffd640 1 -- 172.21.15.49:0/3772345862 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1e00044070 con 0x7f1e041006e0 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.874+0000 7f1e0edaa740 1 Processor -- start 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0edaa740 1 -- start start 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3727db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd372ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 0x555dd372f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555dd368c0d0 con 0x555dd3728380 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555dd3634c70 con 0x555dd3727990 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555dd3688170 con 0x555dd372d4f0 2024-03-18T00:58:55.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e09f60640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd372ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0a761640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3727db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0975f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 0x555dd372f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0a761640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3727db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:54192/0 (socket says 172.21.15.49:54192) 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0a761640 1 -- 172.21.15.49:0/4285859233 learned_addr learned my addr 172.21.15.49:0/4285859233 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0a761640 1 -- 172.21.15.49:0/4285859233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 msgr2=0x555dd372f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0a761640 1 --2- 172.21.15.49:0/4285859233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 0x555dd372f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0a761640 1 -- 172.21.15.49:0/4285859233 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 msgr2=0x555dd372ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0a761640 1 --2- 172.21.15.49:0/4285859233 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd372ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1dfaffd640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1df00a2800 0x7f1df00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.875+0000 7f1e0a761640 1 -- 172.21.15.49:0/4285859233 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555dd34687e0 con 0x555dd3727990 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.876+0000 7f1e0975f640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1df00a2800 0x7f1df00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.876+0000 7f1e0a761640 1 --2- 172.21.15.49:0/4285859233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3727db0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009fd0 tx=0x7f1dfc0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4140e4cf91d99c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.876+0000 7f1dfaffd640 1 -- 172.21.15.49:0/3772345862 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f1e000f0530 con 0x7f1e041006e0 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.876+0000 7f1deffff640 1 -- 172.21.15.49:0/4285859233 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc004d20 con 0x555dd3727990 2024-03-18T00:58:55.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.876+0000 7f1deffff640 1 -- 172.21.15.49:0/4285859233 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dfc004050 con 0x555dd3727990 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 -- 172.21.15.49:0/4285859233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 msgr2=0x555dd3727db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0975f640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1df00a2800 0x7f1df00a4cc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1e041043e0 tx=0x7f1df4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/4285859233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3727db0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009fd0 tx=0x7f1dfc0118d0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1deffff640 1 -- 172.21.15.49:0/4285859233 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc005220 con 0x555dd3727990 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 -- 172.21.15.49:0/4285859233 shutdown_connections 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/4285859233 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 0x555dd372f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/4285859233 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd372ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/4285859233 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3727db0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 -- 172.21.15.49:0/4285859233 >> 172.21.15.49:0/4285859233 conn(0x555dd36efc50 msgr2=0x555dd3723300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 -- 172.21.15.49:0/4285859233 shutdown_connections 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 -- 172.21.15.49:0/4285859233 wait complete. 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 -- start start 2024-03-18T00:58:55.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.877+0000 7f1e0edaa740 1 Processor -- start 2024-03-18T00:58:55.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3727990 0x555dd3731ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd3732400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 0x555dd3739440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555dd37304f0 con 0x555dd3728380 2024-03-18T00:58:55.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555dd36480d0 con 0x555dd372d4f0 2024-03-18T00:58:55.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555dd3730670 con 0x555dd3727990 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0edaa740 10 client.0 _ll_register_callbacks cb 0x555dd3611760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e09f60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3727990 0x555dd3731ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0975f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd3732400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e09f60640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3727990 0x555dd3731ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38822/0 (socket says 172.21.15.49:38822) 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e09f60640 1 -- 172.21.15.49:0/2790532931 learned_addr learned my addr 172.21.15.49:0/2790532931 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0a761640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 0x555dd3739440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0975f640 1 -- 172.21.15.49:0/2790532931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3727990 msgr2=0x555dd3731ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0975f640 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3727990 0x555dd3731ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0975f640 1 -- 172.21.15.49:0/2790532931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 msgr2=0x555dd3739440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0975f640 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 0x555dd3739440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0975f640 1 -- 172.21.15.49:0/2790532931 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555dd34c8f70 con 0x555dd3728380 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e09f60640 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3727990 0x555dd3731ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0a761640 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 0x555dd3739440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1e0975f640 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd3732400 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1df4012cb0 tx=0x7f1df4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb8da37251a3c21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:55.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.878+0000 7f1ded7fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2790532931 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4009e40 con 0x555dd3728380 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2790532931 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f1df4005020 con 0x555dd3728380 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2790532931 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1dfc004050 con 0x555dd3728380 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2790532931 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1df4005220 con 0x555dd3728380 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e0edaa740 10 client.17652 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2790532931 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555dd3360170 con 0x555dd3728380 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2790532931 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1dfc004050 con 0x555dd3728380 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2790532931 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1df4027070 con 0x555dd3728380 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e0edaa740 10 client.17652 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e0edaa740 10 client.17652 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e0edaa740 20 client.17652 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555dd373da50 0x555dd373feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e0edaa740 4 client.17652 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2790532931 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555dd3699610 con 0x555dd373da50 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.879+0000 7f1e09f60640 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555dd373da50 0x555dd373feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.880+0000 7f1e09f60640 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555dd373da50 0x555dd373feb0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79d6ac891259bfd2 server_cookie=e089b38e5eb34980 in_seq=0 out_seq=0 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.880+0000 7f1ded7fa640 10 client.17652 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2790532931 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555dd3699610 con 0x555dd373da50 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1ded7fa640 10 client.17652 handle_command_reply: tid=0 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 client.17652 shutdown 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2790532931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 msgr2=0x555dd3732400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd3732400 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f1df4012cb0 tx=0x7f1df4004730 comp rx=0 tx=0).stop 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2790532931 shutdown_connections 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555dd373da50 0x555dd373feb0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 0x555dd3739440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd3732400 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1ded7fa640 0 client.17652 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:55.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2790532931 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3727990 0x555dd3731ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2790532931 >> 172.21.15.49:0/2790532931 conn(0x555dd36efc50 msgr2=0x555dd37224e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:55.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2790532931 shutdown_connections 2024-03-18T00:58:55.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.883+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2790532931 wait complete. 2024-03-18T00:58:55.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.884+0000 7f1e0edaa740 20 client.17652 trim_cache size 0 max 16384 2024-03-18T00:58:55.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.902+0000 7f1e0edaa740 1 Processor -- start 2024-03-18T00:58:55.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.902+0000 7f1e0edaa740 1 -- start start 2024-03-18T00:58:55.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.902+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3727990 0x555dd3726540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3728380 0x555dd3739400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 0x555dd3726a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041aaf80 con 0x555dd3727990 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041acac0 con 0x555dd372d4f0 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041ab650 con 0x555dd3728380 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e09f60640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3727990 0x555dd3726540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e09f60640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3727990 0x555dd3726540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48738/0 (socket says 172.21.15.49:48738) 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e09f60640 1 -- 172.21.15.49:0/3360908271 learned_addr learned my addr 172.21.15.49:0/3360908271 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e0a761640 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 0x555dd3726a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e0975f640 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3728380 0x555dd3739400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e09f60640 1 -- 172.21.15.49:0/3360908271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3728380 msgr2=0x555dd3739400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e09f60640 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3728380 0x555dd3739400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e09f60640 1 -- 172.21.15.49:0/3360908271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 msgr2=0x555dd3726a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:55.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e09f60640 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 0x555dd3726a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e09f60640 1 -- 172.21.15.49:0/3360908271 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555dd3724250 con 0x555dd3727990 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e0975f640 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3728380 0x555dd3739400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1e09f60640 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3727990 0x555dd3726540 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1e0002cec0 tx=0x7f1e000f3f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69814037a37522e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1def7fe640 1 -- 172.21.15.49:0/3360908271 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0009be60 con 0x555dd3727990 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1def7fe640 1 -- 172.21.15.49:0/3360908271 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e00004c80 con 0x555dd3727990 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.903+0000 7f1def7fe640 1 -- 172.21.15.49:0/3360908271 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0004b040 con 0x555dd3727990 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 -- 172.21.15.49:0/3360908271 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3727990 msgr2=0x555dd3726540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3727990 0x555dd3726540 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f1e0002cec0 tx=0x7f1e000f3f90 comp rx=0 tx=0).stop 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 -- 172.21.15.49:0/3360908271 shutdown_connections 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd372d4f0 0x555dd3726a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd3728380 0x555dd3739400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/3360908271 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3727990 0x555dd3726540 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 -- 172.21.15.49:0/3360908271 >> 172.21.15.49:0/3360908271 conn(0x555dd36efc50 msgr2=0x555dd37297e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 -- 172.21.15.49:0/3360908271 shutdown_connections 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 -- 172.21.15.49:0/3360908271 wait complete. 2024-03-18T00:58:55.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 -- start start 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.904+0000 7f1e0edaa740 1 Processor -- start 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3738c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd37391a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0edaa740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 0x555dd37372e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041ac3f0 con 0x555dd3728380 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041abd20 con 0x555dd3727990 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0edaa740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1e041a8f60 con 0x555dd372d4f0 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0edaa740 10 client.0 _ll_register_callbacks cb 0x7f1df4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0975f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd37391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e09f60640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3738c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0975f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd37391a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48752/0 (socket says 172.21.15.49:48752) 2024-03-18T00:58:55.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0975f640 1 -- 172.21.15.49:0/2001593215 learned_addr learned my addr 172.21.15.49:0/2001593215 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0a761640 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 0x555dd37372e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0975f640 1 -- 172.21.15.49:0/2001593215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 msgr2=0x555dd37372e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0975f640 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 0x555dd37372e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0975f640 1 -- 172.21.15.49:0/2001593215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 msgr2=0x555dd3738c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0975f640 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3738c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0975f640 1 -- 172.21.15.49:0/2001593215 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555dd37378f0 con 0x555dd3728380 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e09f60640 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3738c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0975f640 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd37391a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009730 tx=0x7f1df401f780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2dd494ef07d627f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1ded7fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2001593215 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4036030 con 0x555dd3728380 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2001593215 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f1df4009e40 con 0x555dd3728380 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.905+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2001593215 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e00004c80 con 0x555dd3728380 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2001593215 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1df4041070 con 0x555dd3728380 2024-03-18T00:58:55.911 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1e0edaa740 10 client.17658 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2001593215 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555dd37a4720 con 0x555dd3728380 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2001593215 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e00004c80 con 0x555dd3728380 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2001593215 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f1df4027040 con 0x555dd3728380 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1e0edaa740 10 client.17658 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1e0edaa740 10 client.17658 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1e0edaa740 20 client.17658 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555dd373da50 0x555dd37a7140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1e0edaa740 4 client.17658 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2001593215 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555dd3731cb0 con 0x555dd373da50 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.906+0000 7f1e09f60640 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555dd373da50 0x555dd37a7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.907+0000 7f1e09f60640 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555dd373da50 0x555dd37a7140 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=722ffd761094637b server_cookie=317a4cca62a68816 in_seq=0 out_seq=0 2024-03-18T00:58:55.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:55.907+0000 7f1ded7fa640 10 client.17658 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:55.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:55 smithi107 ceph-mon[36270]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 29 MiB/s wr, 335 op/s 2024-03-18T00:58:55.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:55 smithi106 ceph-mon[35899]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 29 MiB/s wr, 335 op/s 2024-03-18T00:58:56.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.471+0000 7f1ded7fa640 1 -- 172.21.15.49:0/2001593215 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555dd3731cb0 con 0x555dd373da50 2024-03-18T00:58:56.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.471+0000 7f1ded7fa640 10 client.17658 handle_command_reply: tid=0 2024-03-18T00:58:56.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.471+0000 7f1e0edaa740 1 client.17658 shutdown 2024-03-18T00:58:56.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2001593215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 msgr2=0x555dd37391a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd37391a0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009730 tx=0x7f1df401f780 comp rx=0 tx=0).stop 2024-03-18T00:58:56.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2001593215 shutdown_connections 2024-03-18T00:58:56.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555dd373da50 0x555dd37a7140 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555dd372d4f0 0x555dd37372e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.473 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555dd3728380 0x555dd37391a0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 --2- 172.21.15.49:0/2001593215 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555dd3727990 0x555dd3738c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1ded7fa640 0 client.17658 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2001593215 >> 172.21.15.49:0/2001593215 conn(0x555dd36efc50 msgr2=0x555dd3721c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2001593215 shutdown_connections 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 1 -- 172.21.15.49:0/2001593215 wait complete. 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.472+0000 7f1e0edaa740 20 client.17658 trim_cache size 0 max 16384 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "32989747-5ead-4958-8622-ec6c54a01911", 2024-03-18T00:58:56.474 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T00:58:56.475 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:58:56.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1df00a2800 msgr2=0x7f1df00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1df00a2800 0x7f1df00a4cc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1e041043e0 tx=0x7f1df4005f90 comp rx=0 tx=0).stop 2024-03-18T00:58:56.475 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 msgr2=0x7f1e04104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 0x7f1e04104670 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f1e00000c00 tx=0x7f1e0002c640 comp rx=0 tx=0).stop 2024-03-18T00:58:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3772345862 shutdown_connections 2024-03-18T00:58:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1df00a2800 0x7f1df00a4cc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1e041018d0 0x7f1e04102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1e04102290 0x7f1e041031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 --2- 172.21.15.49:0/3772345862 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1e041006e0 0x7f1e04104670 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3772345862 >> 172.21.15.49:0/3772345862 conn(0x7f1e040fbc70 msgr2=0x7f1e040fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3772345862 shutdown_connections 2024-03-18T00:58:56.476 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.473+0000 7f1e0c1c4640 1 -- 172.21.15.49:0/3772345862 wait complete. 2024-03-18T00:58:56.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:58:56.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:58:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.598+0000 7fa7a35df640 1 -- 172.21.15.49:0/672849739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c100700 msgr2=0x7fa79c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.598+0000 7fa7a35df640 1 --2- 172.21.15.49:0/672849739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c100700 0x7fa79c100b20 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa798009900 tx=0x7fa79802bc30 comp rx=0 tx=0).stop 2024-03-18T00:58:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.598+0000 7fa7a35df640 1 -- 172.21.15.49:0/672849739 shutdown_connections 2024-03-18T00:58:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.598+0000 7fa7a35df640 1 --2- 172.21.15.49:0/672849739 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa79c1022b0 0x7fa79c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.598+0000 7fa7a35df640 1 --2- 172.21.15.49:0/672849739 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c1018f0 0x7fa79c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.598+0000 7fa7a35df640 1 --2- 172.21.15.49:0/672849739 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c100700 0x7fa79c100b20 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.598+0000 7fa7a35df640 1 -- 172.21.15.49:0/672849739 >> 172.21.15.49:0/672849739 conn(0x7fa79c0fbc90 msgr2=0x7fa79c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.598+0000 7fa7a35df640 1 -- 172.21.15.49:0/672849739 shutdown_connections 2024-03-18T00:58:56.601 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.598+0000 7fa7a35df640 1 -- 172.21.15.49:0/672849739 wait complete. 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a35df640 1 Processor -- start 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a35df640 1 -- start start 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a35df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100700 0x7fa79c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a35df640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c1018f0 0x7fa79c198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a35df640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa79c1022b0 0x7fa79c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a35df640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c10c350 con 0x7fa79c1018f0 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a35df640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c10c1d0 con 0x7fa79c1022b0 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a35df640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c10c4d0 con 0x7fa79c100700 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a0b7a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c1018f0 0x7fa79c198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a0b7a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c1018f0 0x7fa79c198040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48764/0 (socket says 172.21.15.49:48764) 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/671854694 learned_addr learned my addr 172.21.15.49:0/671854694 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a137b640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100700 0x7fa79c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa79c1022b0 0x7fa79c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.602 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/671854694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100700 msgr2=0x7fa79c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100700 0x7fa79c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/671854694 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa79c1022b0 msgr2=0x7fa79c19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa79c1022b0 0x7fa79c19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/671854694 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c19f720 con 0x7fa79c1018f0 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.599+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c1018f0 0x7fa79c198040 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00e9f0 tx=0x7fa78c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.600+0000 7fa7927fc640 1 -- 172.21.15.49:0/671854694 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78c00cd80 con 0x7fa79c1018f0 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.600+0000 7fa7927fc640 1 -- 172.21.15.49:0/671854694 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa78c010910 con 0x7fa79c1018f0 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.600+0000 7fa7a35df640 1 -- 172.21.15.49:0/671854694 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa79c19fa10 con 0x7fa79c1018f0 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.600+0000 7fa7927fc640 1 -- 172.21.15.49:0/671854694 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78c018990 con 0x7fa79c1018f0 2024-03-18T00:58:56.603 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.600+0000 7fa7a35df640 1 -- 172.21.15.49:0/671854694 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa79c19ff60 con 0x7fa79c1018f0 2024-03-18T00:58:56.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.602+0000 7fa7927fc640 1 -- 172.21.15.49:0/671854694 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa78c010ac0 con 0x7fa79c1018f0 2024-03-18T00:58:56.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.602+0000 7fa7a61c7740 1 Processor -- start 2024-03-18T00:58:56.608 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a61c7740 1 -- start start 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dc6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dcb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f882d2b500 con 0x55f882dcbcd0 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f882cd3be0 con 0x55f882dc6170 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f882d275a0 con 0x55f882dc6b60 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7927fc640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa7880a2800 0x7fa7880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a1b7c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dc6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a137b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dcb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a0b7a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a1b7c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dc6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:54240/0 (socket says 172.21.15.49:54240) 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a1b7c640 1 -- 172.21.15.49:0/1253469185 learned_addr learned my addr 172.21.15.49:0/1253469185 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:56.609 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa7880a2800 0x7fa7880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7927fc640 1 -- 172.21.15.49:0/671854694 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa78c0cedd0 con 0x7fa79c1018f0 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a1b7c640 1 -- 172.21.15.49:0/1253469185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 msgr2=0x55f882dcb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/1253469185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dcb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a1b7c640 1 -- 172.21.15.49:0/1253469185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 msgr2=0x55f882dce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/1253469185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.603+0000 7fa7a1b7c640 1 -- 172.21.15.49:0/1253469185 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f882ce74b0 con 0x55f882dc6170 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.604+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/1253469185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dc6590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa794011e20 tx=0x7fa79400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5479f0c2d1a7851 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.604+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa7880a2800 0x7fa7880a4cc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa79400fc90 tx=0x7fa79401e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.604+0000 7fa7877fe640 1 -- 172.21.15.49:0/1253469185 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79401dcb0 con 0x55f882dc6170 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.604+0000 7fa7877fe640 1 -- 172.21.15.49:0/1253469185 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa79401de50 con 0x55f882dc6170 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.604+0000 7fa7877fe640 1 -- 172.21.15.49:0/1253469185 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa794004280 con 0x55f882dc6170 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.604+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1253469185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 msgr2=0x55f882dc6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.610 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1253469185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dc6590 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa794011e20 tx=0x7fa79400f790 comp rx=0 tx=0).stop 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1253469185 shutdown_connections 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1253469185 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1253469185 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dcb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1253469185 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dc6590 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1253469185 >> 172.21.15.49:0/1253469185 conn(0x55f882dc16e0 msgr2=0x55f882dc1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1253469185 shutdown_connections 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1253469185 wait complete. 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 -- start start 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 Processor -- start 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dd0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f882dcebc0 con 0x55f882dcbcd0 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f882ce7160 con 0x55f882dc6170 2024-03-18T00:58:56.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f882dced40 con 0x55f882dc6b60 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.605+0000 7fa7a61c7740 10 client.0 _ll_register_callbacks cb 0x55f882ccb150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a137b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dd0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a1b7c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a0b7a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a137b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dd0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:54250/0 (socket says 172.21.15.49:54250) 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a137b640 1 -- 172.21.15.49:0/1750191021 learned_addr learned my addr 172.21.15.49:0/1750191021 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/1750191021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 msgr2=0x55f882dd0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dd0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/1750191021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 msgr2=0x55f882dd7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/1750191021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f882b097d0 con 0x55f882dc6b60 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:56.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd0b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa78c09b500 tx=0x7fa78c06b3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a7f569ad222c3d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa784ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa784ff9640 1 -- 172.21.15.49:0/1750191021 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78c094e60 con 0x55f882dc6b60 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa784ff9640 1 -- 172.21.15.49:0/1750191021 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa78c0860e0 con 0x55f882dc6b60 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1750191021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa79401de50 con 0x55f882dc6b60 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa784ff9640 1 -- 172.21.15.49:0/1750191021 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa78c09b800 con 0x55f882dc6b60 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a61c7740 10 client.26902 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.606+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1750191021 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f882a3f3f0 con 0x55f882dc6b60 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa784ff9640 1 -- 172.21.15.49:0/1750191021 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa79401de50 con 0x55f882dc6b60 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa784ff9640 1 -- 172.21.15.49:0/1750191021 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa78c0afb30 con 0x55f882dc6b60 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa7a61c7740 10 client.26902 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa7a61c7740 10 client.26902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa7a61c7740 20 client.26902 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f882ddc110 0x55f882dde550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa7a61c7740 4 client.26902 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:58:56.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1750191021 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f882d38a40 con 0x55f882ddc110 2024-03-18T00:58:56.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa7a137b640 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f882ddc110 0x55f882dde550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa784ff9640 10 client.26902 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:56.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.607+0000 7fa7a137b640 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f882ddc110 0x55f882dde550 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfc2d351e07781b7 server_cookie=68502860abd6c0a3 in_seq=0 out_seq=0 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa784ff9640 1 -- 172.21.15.49:0/1750191021 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f882d38a40 con 0x55f882ddc110 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa784ff9640 10 client.26902 handle_command_reply: tid=0 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 client.26902 shutdown 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1750191021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 msgr2=0x55f882dd0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd0b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa78c09b500 tx=0x7fa78c06b3e0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1750191021 shutdown_connections 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f882ddc110 0x55f882dde550 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd0b60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1750191021 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dd0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa784ff9640 0 client.26902 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:56.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1750191021 >> 172.21.15.49:0/1750191021 conn(0x55f882dc16e0 msgr2=0x55f882dc0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:56.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1750191021 shutdown_connections 2024-03-18T00:58:56.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1750191021 wait complete. 2024-03-18T00:58:56.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.610+0000 7fa7a61c7740 20 client.26902 trim_cache size 0 max 16384 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.628+0000 7fa7a61c7740 1 Processor -- start 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.628+0000 7fa7a61c7740 1 -- start start 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6170 0x55f882dc80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dc6b60 0x55f882dd7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dcbcd0 0x55f882dc8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c1a8d90 con 0x55f882dc6b60 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c1aa8d0 con 0x55f882dcbcd0 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c1a9460 con 0x55f882dc6170 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a0b7a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dc6b60 0x55f882dd7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.634 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a137b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6170 0x55f882dc80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a0b7a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dc6b60 0x55f882dd7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48792/0 (socket says 172.21.15.49:48792) 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a137b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6170 0x55f882dc80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38888/0 (socket says 172.21.15.49:38888) 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/1652501432 learned_addr learned my addr 172.21.15.49:0/1652501432 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/1652501432 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dcbcd0 0x55f882dc8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/1652501432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6170 msgr2=0x55f882dc80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/1652501432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6170 0x55f882dc80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/1652501432 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dcbcd0 msgr2=0x55f882dc8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/1652501432 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dcbcd0 0x55f882dc8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a0b7a640 1 -- 172.21.15.49:0/1652501432 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8829ff0f0 con 0x55f882dc6b60 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a137b640 1 --2- 172.21.15.49:0/1652501432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6170 0x55f882dc80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/1652501432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dc6b60 0x55f882dd7b60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0cc900 tx=0x7fa78c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80e05cfc10a9ca5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa786ffd640 1 -- 172.21.15.49:0/1652501432 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78c0860e0 con 0x55f882dc6b60 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa786ffd640 1 -- 172.21.15.49:0/1652501432 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa78c0afb30 con 0x55f882dc6b60 2024-03-18T00:58:56.635 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.629+0000 7fa786ffd640 1 -- 172.21.15.49:0/1652501432 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa78c06d330 con 0x55f882dc6b60 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.630+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1652501432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dc6b60 msgr2=0x55f882dd7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.630+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1652501432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dc6b60 0x55f882dd7b60 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0cc900 tx=0x7fa78c000bc0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.630+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1652501432 shutdown_connections 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.630+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1652501432 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dcbcd0 0x55f882dc8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.630+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1652501432 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dc6b60 0x55f882dd7b60 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.630+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1652501432 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6170 0x55f882dc80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.630+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1652501432 >> 172.21.15.49:0/1652501432 conn(0x55f882dc16e0 msgr2=0x55f882dc1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.630+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1652501432 shutdown_connections 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.630+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1652501432 wait complete. 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a61c7740 1 -- start start 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a61c7740 1 Processor -- start 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dd4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a61c7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.636 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c1aa200 con 0x55f882dcbcd0 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c1a9b30 con 0x55f882dc6170 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a61c7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c1a6d70 con 0x55f882dc6b60 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a61c7740 10 client.0 _ll_register_callbacks cb 0x7fa78c00e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a1b7c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a1b7c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:48802/0 (socket says 172.21.15.49:48802) 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a1b7c640 1 -- 172.21.15.49:0/1475960845 learned_addr learned my addr 172.21.15.49:0/1475960845 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a137b640 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dd4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a1b7c640 1 -- 172.21.15.49:0/1475960845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 msgr2=0x55f882dd50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a1b7c640 1 -- 172.21.15.49:0/1475960845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 msgr2=0x55f882dd4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dd4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a1b7c640 1 -- 172.21.15.49:0/1475960845 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f882cf29d0 con 0x55f882dcbcd0 2024-03-18T00:58:56.637 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.631+0000 7fa7a0b7a640 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a1b7c640 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd60f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa794011e20 tx=0x7fa7940097a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec435044bdb0f33 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa784ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa784ff9640 1 -- 172.21.15.49:0/1475960845 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79402a040 con 0x55f882dcbcd0 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa784ff9640 1 -- 172.21.15.49:0/1475960845 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa794003c80 con 0x55f882dcbcd0 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1475960845 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa78c0afb30 con 0x55f882dcbcd0 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa784ff9640 1 -- 172.21.15.49:0/1475960845 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa794042050 con 0x55f882dcbcd0 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a61c7740 10 client.17676 fetch_fsmap learned FSMap version 28 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1475960845 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f882dd7820 con 0x55f882dcbcd0 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa784ff9640 1 -- 172.21.15.49:0/1475960845 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa78c0afb30 con 0x55f882dcbcd0 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa784ff9640 1 -- 172.21.15.49:0/1475960845 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa794004b10 con 0x55f882dcbcd0 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a61c7740 10 client.17676 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a61c7740 10 client.17676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a61c7740 20 client.17676 populate_metadata read hostname 'smithi049' 2024-03-18T00:58:56.638 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f882e424a0 0x55f882e44890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:58:56.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a61c7740 4 client.17676 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:58:56.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.632+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1475960845 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f882d32d90 con 0x55f882e424a0 2024-03-18T00:58:56.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.633+0000 7fa7a137b640 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f882e424a0 0x55f882e44890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:58:56.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.633+0000 7fa7a137b640 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f882e424a0 0x55f882e44890 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=204540cec42f0b97 server_cookie=94ab8da743160cf6 in_seq=0 out_seq=0 2024-03-18T00:58:56.639 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.633+0000 7fa784ff9640 10 client.17676 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:56.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.636+0000 7fa784ff9640 1 -- 172.21.15.49:0/1475960845 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55f882d32d90 con 0x55f882e424a0 2024-03-18T00:58:56.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.636+0000 7fa784ff9640 10 client.17676 handle_command_reply: tid=0 2024-03-18T00:58:56.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.636+0000 7fa7a61c7740 1 client.17676 shutdown 2024-03-18T00:58:56.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.636+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1475960845 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 msgr2=0x55f882dd60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.636+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd60f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fa794011e20 tx=0x7fa7940097a0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1475960845 shutdown_connections 2024-03-18T00:58:56.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f882e424a0 0x55f882e44890 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.640 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f882dcbcd0 0x55f882dd60f0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa784ff9640 0 client.17676 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f882dc6b60 0x55f882dd50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a61c7740 1 --2- 172.21.15.49:0/1475960845 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f882dc6170 0x55f882dd4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1475960845 >> 172.21.15.49:0/1475960845 conn(0x55f882dc16e0 msgr2=0x55f882dd04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1475960845 shutdown_connections 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a61c7740 1 -- 172.21.15.49:0/1475960845 wait complete. 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a61c7740 20 client.17676 trim_cache size 0 max 16384 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (2 inodes in the stack)", 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stdout: "32989747-5ead-4958-8622-ec6c54a01911": { 2024-03-18T00:58:56.641 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:58:56.642 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "32989747-5ead-4958-8622-ec6c54a01911", 2024-03-18T00:58:56.642 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:58:56.642 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:58:56.642 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:58:56.642 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 -- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa7880a2800 msgr2=0x7fa7880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa7880a2800 0x7fa7880a4cc0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa79400fc90 tx=0x7fa79401e000 comp rx=0 tx=0).stop 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 -- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c1018f0 msgr2=0x7fa79c198040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c1018f0 0x7fa79c198040 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fa78c00e9f0 tx=0x7fa78c00eeb0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 -- 172.21.15.49:0/671854694 shutdown_connections 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa7880a2800 0x7fa7880a4cc0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa79c1022b0 0x7fa79c19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa79c1018f0 0x7fa79c198040 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 --2- 172.21.15.49:0/671854694 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa79c100700 0x7fa79c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 -- 172.21.15.49:0/671854694 >> 172.21.15.49:0/671854694 conn(0x7fa79c0fbc90 msgr2=0x7fa79c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 -- 172.21.15.49:0/671854694 shutdown_connections 2024-03-18T00:58:56.643 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:58:56.637+0000 7fa7a35df640 1 -- 172.21.15.49:0/671854694 wait complete. 2024-03-18T00:58:56.649 INFO:tasks.cephfs.filesystem:scrub status for tag:32989747-5ead-4958-8622-ec6c54a01911 - {'path': '/', 'tag': '32989747-5ead-4958-8622-ec6c54a01911', 'options': 'recursive,force'} 2024-03-18T00:58:57.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:57 smithi049 ceph-mon[32130]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 196 op/s 2024-03-18T00:58:57.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:57 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T00:58:57.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:57 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:57.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:57 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:58:57.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:57 smithi107 ceph-mon[36270]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 196 op/s 2024-03-18T00:58:57.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:57 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T00:58:57.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:57 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:57.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:57 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:58:57.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:57 smithi106 ceph-mon[35899]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 17 MiB/s wr, 196 op/s 2024-03-18T00:58:57.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:57 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T00:58:57.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:57 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:58:57.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:57 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:58:59.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:58:59 smithi049 ceph-mon[32130]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 12 MiB/s wr, 134 op/s 2024-03-18T00:58:59.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:58:59 smithi107 ceph-mon[36270]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 12 MiB/s wr, 134 op/s 2024-03-18T00:58:59.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:58:59 smithi106 ceph-mon[35899]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 12 MiB/s wr, 134 op/s 2024-03-18T00:59:01.875 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:01 smithi049 ceph-mon[32130]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 55 op/s 2024-03-18T00:59:01.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:01 smithi107 ceph-mon[36270]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 55 op/s 2024-03-18T00:59:01.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:01 smithi106 ceph-mon[35899]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 5.6 MiB/s wr, 55 op/s 2024-03-18T00:59:03.874 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:03 smithi049 ceph-mon[32130]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 27 op/s 2024-03-18T00:59:03.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:03 smithi107 ceph-mon[36270]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 27 op/s 2024-03-18T00:59:03.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:03 smithi106 ceph-mon[35899]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 2.9 MiB/s wr, 27 op/s 2024-03-18T00:59:05.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:05 smithi107 ceph-mon[36270]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-03-18T00:59:05.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:05 smithi106 ceph-mon[35899]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-03-18T00:59:06.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:05 smithi049 ceph-mon[32130]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.807+0000 7f61320ff640 1 -- 172.21.15.49:0/1630549870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c1075d0 msgr2=0x7f612c1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 --2- 172.21.15.49:0/1630549870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c1075d0 0x7f612c1099c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f6124009ca0 tx=0x7f612402c0f0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 -- 172.21.15.49:0/1630549870 shutdown_connections 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 --2- 172.21.15.49:0/1630549870 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612c104ca0 0x7f612c107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 --2- 172.21.15.49:0/1630549870 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c1075d0 0x7f612c1099c0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 --2- 172.21.15.49:0/1630549870 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f612c102370 0x7f612c104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 -- 172.21.15.49:0/1630549870 >> 172.21.15.49:0/1630549870 conn(0x7f612c0fbcc0 msgr2=0x7f612c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 -- 172.21.15.49:0/1630549870 shutdown_connections 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 -- 172.21.15.49:0/1630549870 wait complete. 2024-03-18T00:59:07.808 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 Processor -- start 2024-03-18T00:59:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.808+0000 7f61320ff640 1 -- start start 2024-03-18T00:59:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f61320ff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612c102370 0x7f612c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f61320ff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c104ca0 0x7f612c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f61320ff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f612c1075d0 0x7f612c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f61320ff640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f612c10d4d0 con 0x7f612c104ca0 2024-03-18T00:59:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f61320ff640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f612c10d350 con 0x7f612c102370 2024-03-18T00:59:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f61320ff640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f612c10d650 con 0x7f612c1075d0 2024-03-18T00:59:07.811 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c104ca0 0x7f612c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612c102370 0x7f612c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612c102370 0x7f612c197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:44532/0 (socket says 172.21.15.49:44532) 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c104ca0 0x7f612c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50672/0 (socket says 172.21.15.49:50672) 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612b7fe640 1 -- 172.21.15.49:0/784280152 learned_addr learned my addr 172.21.15.49:0/784280152 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612bfff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f612c1075d0 0x7f612c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612bfff640 1 -- 172.21.15.49:0/784280152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612c102370 msgr2=0x7f612c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612bfff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612c102370 0x7f612c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612bfff640 1 -- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c104ca0 msgr2=0x7f612c197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612bfff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c104ca0 0x7f612c197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612bfff640 1 -- 172.21.15.49:0/784280152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612c19f650 con 0x7f612c1075d0 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612b7fe640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612c102370 0x7f612c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612affd640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c104ca0 0x7f612c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:07.812 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.809+0000 7f612bfff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f612c1075d0 0x7f612c19efe0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f612402c630 tx=0x7f6124032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.810+0000 7f6128ff9640 1 -- 172.21.15.49:0/784280152 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124041020 con 0x7f612c1075d0 2024-03-18T00:59:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.810+0000 7f61320ff640 1 -- 172.21.15.49:0/784280152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612c19f8e0 con 0x7f612c1075d0 2024-03-18T00:59:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.810+0000 7f61320ff640 1 -- 172.21.15.49:0/784280152 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f612c19fdf0 con 0x7f612c1075d0 2024-03-18T00:59:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.810+0000 7f6128ff9640 1 -- 172.21.15.49:0/784280152 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6124034cd0 con 0x7f612c1075d0 2024-03-18T00:59:07.813 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.810+0000 7f6128ff9640 1 -- 172.21.15.49:0/784280152 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124039040 con 0x7f612c1075d0 2024-03-18T00:59:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.812+0000 7f6128ff9640 1 -- 172.21.15.49:0/784280152 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6124044020 con 0x7f612c1075d0 2024-03-18T00:59:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.812+0000 7f6134dca740 1 Processor -- start 2024-03-18T00:59:07.818 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.812+0000 7f6134dca740 1 -- start start 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f6134dca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 0x55dbdc0d3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f6134dca740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d8830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f6134dca740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0db330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f6134dca740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55dbdc038a40 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f6134dca740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55dbdbfe1920 con 0x55dbdc0d3d30 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f6134dca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dbdc034ae0 con 0x55dbdc0d3340 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f6128ff9640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61140a2ab0 0x7f61140a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0db330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0db330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50686/0 (socket says 172.21.15.49:50686) 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 -- 172.21.15.49:0/1622676546 learned_addr learned my addr 172.21.15.49:0/1622676546 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:07.819 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612bfff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61140a2ab0 0x7f61140a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 --2- 172.21.15.49:0/1622676546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 0x55dbdc0d3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f6128ff9640 1 -- 172.21.15.49:0/784280152 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6124004040 con 0x7f612c1075d0 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 -- 172.21.15.49:0/1622676546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 msgr2=0x55dbdc0d3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 --2- 172.21.15.49:0/1622676546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 0x55dbdc0d3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 -- 172.21.15.49:0/1622676546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 msgr2=0x55dbdc0d8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 --2- 172.21.15.49:0/1622676546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 -- 172.21.15.49:0/1622676546 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbdbff9460 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612b7fe640 1 --2- 172.21.15.49:0/1622676546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612affd640 1 --2- 172.21.15.49:0/1622676546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0db330 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x55dbdc0d4590 tx=0x7f6120005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8ab0b25682833f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.813+0000 7f612bfff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61140a2ab0 0x7f61140a4f70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f61240877d0 tx=0x7f612408f770 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6111ffb640 1 -- 172.21.15.49:0/1622676546 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120013d40 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6111ffb640 1 -- 172.21.15.49:0/1622676546 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6120013120 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.820 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6111ffb640 1 -- 172.21.15.49:0/1622676546 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61200037f0 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6134dca740 1 -- 172.21.15.49:0/1622676546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 msgr2=0x55dbdc0db330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6134dca740 1 --2- 172.21.15.49:0/1622676546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0db330 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x55dbdc0d4590 tx=0x7f6120005850 comp rx=0 tx=0).stop 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6134dca740 1 -- 172.21.15.49:0/1622676546 shutdown_connections 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6134dca740 1 --2- 172.21.15.49:0/1622676546 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6134dca740 1 --2- 172.21.15.49:0/1622676546 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0db330 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6134dca740 1 --2- 172.21.15.49:0/1622676546 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 0x55dbdc0d3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6134dca740 1 -- 172.21.15.49:0/1622676546 >> 172.21.15.49:0/1622676546 conn(0x55dbdc0ce8b0 msgr2=0x55dbdc0cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6134dca740 1 -- 172.21.15.49:0/1622676546 shutdown_connections 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.814+0000 7f6134dca740 1 -- 172.21.15.49:0/1622676546 wait complete. 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f6134dca740 1 -- start start 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f6134dca740 1 Processor -- start 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f6134dca740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 0x55dbdc0dd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.821 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f6134dca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0ddb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f6134dca740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f6134dca740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55dbdc0dbdf0 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f6134dca740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55dbdbff4d70 con 0x55dbdc0d3340 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f6134dca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55dbdc0dbf70 con 0x55dbdc0d3d30 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f6134dca740 10 client.0 _ll_register_callbacks cb 0x55dbdbd4ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.815+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50698/0 (socket says 172.21.15.49:50698) 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f612affd640 1 -- 172.21.15.49:0/2248404890 learned_addr learned my addr 172.21.15.49:0/2248404890 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f612b7fe640 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0ddb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f612affd640 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 0x55dbdc0dd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f612b7fe640 1 -- 172.21.15.49:0/2248404890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 msgr2=0x55dbdc0dd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f612b7fe640 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 0x55dbdc0dd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.822 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f612b7fe640 1 -- 172.21.15.49:0/2248404890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 msgr2=0x55dbdc0e4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f612b7fe640 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f612b7fe640 1 -- 172.21.15.49:0/2248404890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbdbd4d210 con 0x55dbdc0d3d30 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f612b7fe640 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0ddb90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55dbdc0d4130 tx=0x7f611c00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e0a8564b14daf13 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f610affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f610affd640 1 -- 172.21.15.49:0/2248404890 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c00fe50 con 0x55dbdc0d3d30 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f610affd640 1 -- 172.21.15.49:0/2248404890 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f611c005020 con 0x55dbdc0d3d30 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f6134dca740 1 -- 172.21.15.49:0/2248404890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6120013120 con 0x55dbdc0d3d30 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f610affd640 1 -- 172.21.15.49:0/2248404890 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f611c005220 con 0x55dbdc0d3d30 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f6134dca740 10 client.26920 fetch_fsmap learned FSMap version 28 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.816+0000 7f6134dca740 1 -- 172.21.15.49:0/2248404890 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dbdbe19620 con 0x55dbdc0d3d30 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f610affd640 1 -- 172.21.15.49:0/2248404890 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6120013120 con 0x55dbdc0d3d30 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f610affd640 1 -- 172.21.15.49:0/2248404890 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f611c01c470 con 0x55dbdc0d3d30 2024-03-18T00:59:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f6134dca740 10 client.26920 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:59:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f6134dca740 10 client.26920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f6134dca740 20 client.26920 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f6134dca740 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55dbdc0e95b0 0x55dbdc0eb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f6134dca740 4 client.26920 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:59:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f6134dca740 1 -- 172.21.15.49:0/2248404890 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbdc045f80 con 0x55dbdc0e95b0 2024-03-18T00:59:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f612affd640 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55dbdc0e95b0 0x55dbdc0eb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f610affd640 10 client.26920 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.817+0000 7f612affd640 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55dbdc0e95b0 0x55dbdc0eb9f0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2a6cebda006136e server_cookie=1e2cfa3a3d0ca504 in_seq=0 out_seq=0 2024-03-18T00:59:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.820+0000 7f610affd640 1 -- 172.21.15.49:0/2248404890 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55dbdc045f80 con 0x55dbdc0e95b0 2024-03-18T00:59:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.820+0000 7f610affd640 10 client.26920 handle_command_reply: tid=0 2024-03-18T00:59:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 client.26920 shutdown 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 -- 172.21.15.49:0/2248404890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 msgr2=0x55dbdc0ddb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0ddb90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x55dbdc0d4130 tx=0x7f611c00f790 comp rx=0 tx=0).stop 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 -- 172.21.15.49:0/2248404890 shutdown_connections 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55dbdc0e95b0 0x55dbdc0eb9f0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0ddb90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f610affd640 0 client.26920 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 --2- 172.21.15.49:0/2248404890 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 0x55dbdc0dd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 -- 172.21.15.49:0/2248404890 >> 172.21.15.49:0/2248404890 conn(0x55dbdc0ce8b0 msgr2=0x55dbdc0cdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 -- 172.21.15.49:0/2248404890 shutdown_connections 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 1 -- 172.21.15.49:0/2248404890 wait complete. 2024-03-18T00:59:07.826 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.821+0000 7f6134dca740 20 client.26920 trim_cache size 0 max 16384 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.836+0000 7f6134dca740 1 Processor -- start 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.836+0000 7f6134dca740 1 -- start start 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f6134dca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 0x55dbdc0d5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f6134dca740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f6134dca740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f6134dca740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1a8c80 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f6134dca740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1aa7c0 con 0x55dbdc0d3d30 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f6134dca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1a9350 con 0x55dbdc0d3340 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50708/0 (socket says 172.21.15.49:50708) 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 -- 172.21.15.49:0/3816690530 learned_addr learned my addr 172.21.15.49:0/3816690530 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:07.841 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 --2- 172.21.15.49:0/3816690530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 0x55dbdc0d5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 -- 172.21.15.49:0/3816690530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 msgr2=0x55dbdc0d5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 --2- 172.21.15.49:0/3816690530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 0x55dbdc0d5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 -- 172.21.15.49:0/3816690530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 msgr2=0x55dbdc0d5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 --2- 172.21.15.49:0/3816690530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 -- 172.21.15.49:0/3816690530 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbdc0ec070 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612b7fe640 1 --2- 172.21.15.49:0/3816690530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f612affd640 1 --2- 172.21.15.49:0/3816690530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4b90 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f612000ff70 tx=0x7f6120024040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52aebf7552ba9a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f61117fa640 1 -- 172.21.15.49:0/3816690530 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6120013120 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f61117fa640 1 -- 172.21.15.49:0/3816690530 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f612000fd30 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.837+0000 7f61117fa640 1 -- 172.21.15.49:0/3816690530 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61200305f0 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 -- 172.21.15.49:0/3816690530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 msgr2=0x55dbdc0e4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 --2- 172.21.15.49:0/3816690530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4b90 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f612000ff70 tx=0x7f6120024040 comp rx=0 tx=0).stop 2024-03-18T00:59:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 -- 172.21.15.49:0/3816690530 shutdown_connections 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 --2- 172.21.15.49:0/3816690530 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0d5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 --2- 172.21.15.49:0/3816690530 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e4b90 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 --2- 172.21.15.49:0/3816690530 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3340 0x55dbdc0d5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 -- 172.21.15.49:0/3816690530 >> 172.21.15.49:0/3816690530 conn(0x55dbdc0ce8b0 msgr2=0x55dbdc0c9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 -- 172.21.15.49:0/3816690530 shutdown_connections 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 -- 172.21.15.49:0/3816690530 wait complete. 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.838+0000 7f6134dca740 1 -- start start 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f6134dca740 1 Processor -- start 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f6134dca740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 0x55dbdc0e2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f6134dca740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0e2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f6134dca740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f6134dca740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1aa0f0 con 0x55dbdc0d8ea0 2024-03-18T00:59:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f6134dca740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1a9a20 con 0x55dbdc0d3340 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f6134dca740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1a6c60 con 0x55dbdc0d3d30 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f6134dca740 10 client.0 _ll_register_callbacks cb 0x7f611c009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0e2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612b7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0e2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:36064/0 (socket says 172.21.15.49:36064) 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612b7fe640 1 -- 172.21.15.49:0/2278281781 learned_addr learned my addr 172.21.15.49:0/2278281781 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:50722/0 (socket says 172.21.15.49:50722) 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612affd640 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 0x55dbdc0e2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612b7fe640 1 -- 172.21.15.49:0/2278281781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 msgr2=0x55dbdc0e2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612b7fe640 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 0x55dbdc0e2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612b7fe640 1 -- 172.21.15.49:0/2278281781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 msgr2=0x55dbdc0e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612b7fe640 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612b7fe640 1 -- 172.21.15.49:0/2278281781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbdc0cf620 con 0x55dbdc0d3d30 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.839+0000 7f612affd640 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f612b7fe640 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0e2c50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x55dbdc0cf7f0 tx=0x7f611c00fd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b54a0049459e2ab server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f610affd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f610affd640 1 -- 172.21.15.49:0/2278281781 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f611c01c680 con 0x55dbdc0d3d30 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f610affd640 1 -- 172.21.15.49:0/2278281781 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f611c0053c0 con 0x55dbdc0d3d30 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f6134dca740 1 -- 172.21.15.49:0/2278281781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f612000fd30 con 0x55dbdc0d3d30 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f610affd640 1 -- 172.21.15.49:0/2278281781 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f611c00fe50 con 0x55dbdc0d3d30 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f6134dca740 10 client.26926 fetch_fsmap learned FSMap version 28 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f6134dca740 1 -- 172.21.15.49:0/2278281781 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dbdc158420 con 0x55dbdc0d3d30 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f610affd640 1 -- 172.21.15.49:0/2278281781 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f612000fd30 con 0x55dbdc0d3d30 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f610affd640 1 -- 172.21.15.49:0/2278281781 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f611c010430 con 0x55dbdc0d3d30 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f6134dca740 10 client.26926 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f6134dca740 10 client.26926 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f6134dca740 20 client.26926 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f6134dca740 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55dbdc158df0 0x55dbdc15b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f6134dca740 4 client.26926 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:59:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.840+0000 7f6134dca740 1 -- 172.21.15.49:0/2278281781 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbdc0402d0 con 0x55dbdc158df0 2024-03-18T00:59:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.841+0000 7f612affd640 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55dbdc158df0 0x55dbdc15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.841+0000 7f612affd640 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55dbdc158df0 0x55dbdc15b210 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=605e0c337b6f4ad5 server_cookie=db15dc598416ef96 in_seq=0 out_seq=0 2024-03-18T00:59:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:07.841+0000 7f610affd640 10 client.26926 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:07.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:07 smithi107 ceph-mon[36270]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 24 op/s 2024-03-18T00:59:07.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:07 smithi106 ceph-mon[35899]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 24 op/s 2024-03-18T00:59:08.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:07 smithi049 ceph-mon[32130]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.5 MiB/s wr, 24 op/s 2024-03-18T00:59:09.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:09 smithi107 ceph-mon[36270]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2024-03-18T00:59:09.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:09 smithi106 ceph-mon[35899]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2024-03-18T00:59:10.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:09 smithi049 ceph-mon[32130]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2024-03-18T00:59:11.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:11 smithi107 ceph-mon[36270]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 21 op/s 2024-03-18T00:59:11.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:11 smithi106 ceph-mon[35899]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 21 op/s 2024-03-18T00:59:12.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:11 smithi049 ceph-mon[32130]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.2 MiB/s wr, 21 op/s 2024-03-18T00:59:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.733+0000 7f610affd640 1 -- 172.21.15.49:0/2278281781 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dbdc0402d0 con 0x55dbdc158df0 2024-03-18T00:59:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.733+0000 7f610affd640 10 client.26926 handle_command_reply: tid=0 2024-03-18T00:59:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.733+0000 7f6134dca740 1 client.26926 shutdown 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.733+0000 7f6134dca740 1 -- 172.21.15.49:0/2278281781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 msgr2=0x55dbdc0e2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.734+0000 7f6134dca740 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0e2c50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x55dbdc0cf7f0 tx=0x7f611c00fd00 comp rx=0 tx=0).stop 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.734+0000 7f6134dca740 1 -- 172.21.15.49:0/2278281781 shutdown_connections 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.734+0000 7f6134dca740 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55dbdc158df0 0x55dbdc15b210 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.734+0000 7f6134dca740 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55dbdc0d8ea0 0x55dbdc0e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.734+0000 7f610affd640 0 client.26926 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.734+0000 7f6134dca740 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55dbdc0d3d30 0x55dbdc0e2c50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.734+0000 7f6134dca740 1 --2- 172.21.15.49:0/2278281781 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55dbdc0d3340 0x55dbdc0e2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.734+0000 7f6134dca740 1 -- 172.21.15.49:0/2278281781 >> 172.21.15.49:0/2278281781 conn(0x55dbdc0ce8b0 msgr2=0x55dbdc0dd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.735+0000 7f6134dca740 1 -- 172.21.15.49:0/2278281781 shutdown_connections 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.735+0000 7f6134dca740 1 -- 172.21.15.49:0/2278281781 wait complete. 2024-03-18T00:59:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.735+0000 7f6134dca740 20 client.26926 trim_cache size 0 max 16384 2024-03-18T00:59:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:59:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:59:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:59:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 -- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61140a2ab0 msgr2=0x7f61140a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61140a2ab0 0x7f61140a4f70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f61240877d0 tx=0x7f612408f770 comp rx=0 tx=0).stop 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 -- 172.21.15.49:0/784280152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f612c1075d0 msgr2=0x7f612c19efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f612c1075d0 0x7f612c19efe0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f612402c630 tx=0x7f6124032f20 comp rx=0 tx=0).stop 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 -- 172.21.15.49:0/784280152 shutdown_connections 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61140a2ab0 0x7f61140a4f70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f612c1075d0 0x7f612c19efe0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f612c104ca0 0x7f612c197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 --2- 172.21.15.49:0/784280152 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f612c102370 0x7f612c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 -- 172.21.15.49:0/784280152 >> 172.21.15.49:0/784280152 conn(0x7f612c0fbcc0 msgr2=0x7f612c102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 -- 172.21.15.49:0/784280152 shutdown_connections 2024-03-18T00:59:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:13.736+0000 7f61320ff640 1 -- 172.21.15.49:0/784280152 wait complete. 2024-03-18T00:59:13.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:13 smithi107 ceph-mon[36270]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 30 op/s 2024-03-18T00:59:13.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:13 smithi106 ceph-mon[35899]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 30 op/s 2024-03-18T00:59:14.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:13 smithi049 ceph-mon[32130]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.2 MiB/s wr, 30 op/s 2024-03-18T00:59:15.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:15 smithi107 ceph-mon[36270]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 24 op/s 2024-03-18T00:59:15.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:15 smithi106 ceph-mon[35899]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 24 op/s 2024-03-18T00:59:16.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:15 smithi049 ceph-mon[32130]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 24 op/s 2024-03-18T00:59:17.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:17 smithi107 ceph-mon[36270]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 26 op/s 2024-03-18T00:59:17.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:17 smithi106 ceph-mon[35899]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 26 op/s 2024-03-18T00:59:18.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:17 smithi049 ceph-mon[32130]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 26 op/s 2024-03-18T00:59:19.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:19 smithi107 ceph-mon[36270]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 25 op/s 2024-03-18T00:59:19.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:19 smithi106 ceph-mon[35899]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 25 op/s 2024-03-18T00:59:20.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:19 smithi049 ceph-mon[32130]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 3.1 MiB/s wr, 25 op/s 2024-03-18T00:59:21.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:21 smithi107 ceph-mon[36270]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 23 op/s 2024-03-18T00:59:21.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:21 smithi106 ceph-mon[35899]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 23 op/s 2024-03-18T00:59:22.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:21 smithi049 ceph-mon[32130]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.8 MiB/s wr, 23 op/s 2024-03-18T00:59:23.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:23 smithi107 ceph-mon[36270]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 26 op/s 2024-03-18T00:59:23.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:23 smithi106 ceph-mon[35899]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 26 op/s 2024-03-18T00:59:24.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:23 smithi049 ceph-mon[32130]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 26 op/s 2024-03-18T00:59:25.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:25 smithi107 ceph-mon[36270]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-03-18T00:59:25.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:25 smithi106 ceph-mon[35899]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-03-18T00:59:26.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:25 smithi049 ceph-mon[32130]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.0 MiB/s wr, 17 op/s 2024-03-18T00:59:26.650 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:59:26.651 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:59:26.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.775+0000 7fc33475b640 1 -- 172.21.15.49:0/597272874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc32c1006e0 msgr2=0x7fc32c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.775+0000 7fc33475b640 1 --2- 172.21.15.49:0/597272874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc32c1006e0 0x7fc32c100b00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc328009860 tx=0x7fc32802bb70 comp rx=0 tx=0).stop 2024-03-18T00:59:26.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.775+0000 7fc33475b640 1 -- 172.21.15.49:0/597272874 shutdown_connections 2024-03-18T00:59:26.777 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.775+0000 7fc33475b640 1 --2- 172.21.15.49:0/597272874 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc32c102290 0x7fc32c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.775+0000 7fc33475b640 1 --2- 172.21.15.49:0/597272874 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc32c1018d0 0x7fc32c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.775+0000 7fc33475b640 1 --2- 172.21.15.49:0/597272874 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc32c1006e0 0x7fc32c100b00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.775+0000 7fc33475b640 1 -- 172.21.15.49:0/597272874 >> 172.21.15.49:0/597272874 conn(0x7fc32c0fbc70 msgr2=0x7fc32c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 -- 172.21.15.49:0/597272874 shutdown_connections 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 -- 172.21.15.49:0/597272874 wait complete. 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 Processor -- start 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 -- start start 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc32c1006e0 0x7fc32c104670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc32c1018d0 0x7fc32c102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc32c102290 0x7fc32c1031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c10c360 con 0x7fc32c1018d0 2024-03-18T00:59:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c10c1e0 con 0x7fc32c102290 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc33475b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c10c4e0 con 0x7fc32c1006e0 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc331cf6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc32c1018d0 0x7fc32c102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc331cf6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc32c1018d0 0x7fc32c102cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37670/0 (socket says 172.21.15.49:37670) 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.776+0000 7fc331cf6640 1 -- 172.21.15.49:0/4001817647 learned_addr learned my addr 172.21.15.49:0/4001817647 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc32c1006e0 0x7fc32c104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc332cf8640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc32c102290 0x7fc32c1031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3324f7640 1 -- 172.21.15.49:0/4001817647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc32c102290 msgr2=0x7fc32c1031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc32c102290 0x7fc32c1031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3324f7640 1 -- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc32c1018d0 msgr2=0x7fc32c102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc32c1018d0 0x7fc32c102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3324f7640 1 -- 172.21.15.49:0/4001817647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32c103a80 con 0x7fc32c1006e0 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc331cf6640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc32c1018d0 0x7fc32c102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc332cf8640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc32c102290 0x7fc32c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc32c1006e0 0x7fc32c104670 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc328009780 tx=0x7fc328032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3237fe640 1 -- 172.21.15.49:0/4001817647 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328041020 con 0x7fc32c1006e0 2024-03-18T00:59:26.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3237fe640 1 -- 172.21.15.49:0/4001817647 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc32802c9f0 con 0x7fc32c1006e0 2024-03-18T00:59:26.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc33475b640 1 -- 172.21.15.49:0/4001817647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc32c1a1c20 con 0x7fc32c1006e0 2024-03-18T00:59:26.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc3237fe640 1 -- 172.21.15.49:0/4001817647 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc328039040 con 0x7fc32c1006e0 2024-03-18T00:59:26.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.777+0000 7fc33475b640 1 -- 172.21.15.49:0/4001817647 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32c1a2140 con 0x7fc32c1006e0 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.779+0000 7fc3237fe640 1 -- 172.21.15.49:0/4001817647 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc328044020 con 0x7fc32c1006e0 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc337426740 1 Processor -- start 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc337426740 1 -- start start 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc3237fe640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3180a2800 0x7fc3180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc337426740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037ba170 0x5649037ba590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc337426740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037bf660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc337426740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037bfcd0 0x5649037c2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc337426740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x56490371f500 con 0x5649037bab60 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc337426740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5649036c7be0 con 0x5649037ba170 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc337426740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x56490371b5a0 con 0x5649037bfcd0 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc3237fe640 1 -- 172.21.15.49:0/4001817647 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc3280f0260 con 0x7fc32c1006e0 2024-03-18T00:59:26.785 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc332cf8640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3180a2800 0x7fc3180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc3324f7640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037ba170 0x5649037ba590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc331cf6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037bf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.780+0000 7fc331cf6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037bf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37678/0 (socket says 172.21.15.49:37678) 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc331cf6640 1 -- 172.21.15.49:0/4251757782 learned_addr learned my addr 172.21.15.49:0/4251757782 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc332cf8640 1 --2- 172.21.15.49:0/4251757782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037bfcd0 0x5649037c2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc3324f7640 1 -- 172.21.15.49:0/4251757782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037bfcd0 msgr2=0x5649037c2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4251757782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037bfcd0 0x5649037c2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc3324f7640 1 -- 172.21.15.49:0/4251757782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 msgr2=0x5649037bf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4251757782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037bf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc3324f7640 1 -- 172.21.15.49:0/4251757782 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649036db4b0 con 0x5649037ba170 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc332cf8640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3180a2800 0x7fc3180a4cc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc324005fd0 tx=0x7fc32400feb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.786 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4251757782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037ba170 0x5649037ba590 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc3280bc8e0 tx=0x7fc328005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91a631cea5897222 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc320ff9640 1 -- 172.21.15.49:0/4251757782 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280a7070 con 0x5649037ba170 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc320ff9640 1 -- 172.21.15.49:0/4251757782 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3280bd2e0 con 0x5649037ba170 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.781+0000 7fc320ff9640 1 -- 172.21.15.49:0/4251757782 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280c0d30 con 0x5649037ba170 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 -- 172.21.15.49:0/4251757782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037ba170 msgr2=0x5649037ba590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 --2- 172.21.15.49:0/4251757782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037ba170 0x5649037ba590 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc3280bc8e0 tx=0x7fc328005f90 comp rx=0 tx=0).stop 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 -- 172.21.15.49:0/4251757782 shutdown_connections 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 --2- 172.21.15.49:0/4251757782 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037bfcd0 0x5649037c2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 --2- 172.21.15.49:0/4251757782 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037bf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 --2- 172.21.15.49:0/4251757782 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037ba170 0x5649037ba590 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 -- 172.21.15.49:0/4251757782 >> 172.21.15.49:0/4251757782 conn(0x5649037b56e0 msgr2=0x5649037b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 -- 172.21.15.49:0/4251757782 shutdown_connections 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 -- 172.21.15.49:0/4251757782 wait complete. 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.782+0000 7fc337426740 1 -- start start 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc337426740 1 Processor -- start 2024-03-18T00:59:26.787 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc337426740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc337426740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc337426740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc337426740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5649037c2a40 con 0x5649037bab60 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc337426740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5649036db160 con 0x5649037bfcd0 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc337426740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5649037c2bc0 con 0x5649037ba170 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc337426740 10 client.0 _ll_register_callbacks cb 0x5649036bf150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc331cf6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc3324f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc3324f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:45704/0 (socket says 172.21.15.49:45704) 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc3324f7640 1 -- 172.21.15.49:0/969999692 learned_addr learned my addr 172.21.15.49:0/969999692 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc332cf8640 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc3324f7640 1 -- 172.21.15.49:0/969999692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 msgr2=0x5649037cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc3324f7640 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc3324f7640 1 -- 172.21.15.49:0/969999692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 msgr2=0x5649037c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc3324f7640 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc3324f7640 1 -- 172.21.15.49:0/969999692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649034fd7d0 con 0x5649037ba170 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc3324f7640 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc32802cbc0 tx=0x7fc3280bc720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9313e7aeb351a659 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.783+0000 7fc315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc315ffb640 1 -- 172.21.15.49:0/969999692 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280da060 con 0x5649037ba170 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc315ffb640 1 -- 172.21.15.49:0/969999692 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc3280ede50 con 0x5649037ba170 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc337426740 1 -- 172.21.15.49:0/969999692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3280bd2e0 con 0x5649037ba170 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc315ffb640 1 -- 172.21.15.49:0/969999692 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc3280d4030 con 0x5649037ba170 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc337426740 10 client.26938 fetch_fsmap learned FSMap version 28 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc337426740 1 -- 172.21.15.49:0/969999692 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649034333f0 con 0x5649037ba170 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc315ffb640 1 -- 172.21.15.49:0/969999692 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3280bd2e0 con 0x5649037ba170 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc315ffb640 1 -- 172.21.15.49:0/969999692 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc32803c030 con 0x5649037ba170 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc337426740 10 client.26938 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc337426740 10 client.26938 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:26.789 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc337426740 20 client.26938 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:26.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc337426740 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649037cfb40 0x5649037d1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc337426740 4 client.26938 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:59:26.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc337426740 1 -- 172.21.15.49:0/969999692 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56490372ca40 con 0x5649037cfb40 2024-03-18T00:59:26.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.784+0000 7fc331cf6640 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649037cfb40 0x5649037d1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.785+0000 7fc331cf6640 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649037cfb40 0x5649037d1f30 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=475cf2fbb1363eff server_cookie=c2af90761daf7c4c in_seq=0 out_seq=0 2024-03-18T00:59:26.790 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.785+0000 7fc315ffb640 10 client.26938 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:26.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc315ffb640 1 -- 172.21.15.49:0/969999692 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56490372ca40 con 0x5649037cfb40 2024-03-18T00:59:26.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc315ffb640 10 client.26938 handle_command_reply: tid=0 2024-03-18T00:59:26.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 client.26938 shutdown 2024-03-18T00:59:26.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 -- 172.21.15.49:0/969999692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 msgr2=0x5649037c4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc32802cbc0 tx=0x7fc3280bc720 comp rx=0 tx=0).stop 2024-03-18T00:59:26.791 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 -- 172.21.15.49:0/969999692 shutdown_connections 2024-03-18T00:59:26.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5649037cfb40 0x5649037d1f30 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc315ffb640 0 client.26938 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:26.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 --2- 172.21.15.49:0/969999692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4460 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 -- 172.21.15.49:0/969999692 >> 172.21.15.49:0/969999692 conn(0x5649037b56e0 msgr2=0x5649037b4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 -- 172.21.15.49:0/969999692 shutdown_connections 2024-03-18T00:59:26.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 1 -- 172.21.15.49:0/969999692 wait complete. 2024-03-18T00:59:26.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.787+0000 7fc337426740 20 client.26938 trim_cache size 0 max 16384 2024-03-18T00:59:26.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.824+0000 7fc337426740 1 Processor -- start 2024-03-18T00:59:26.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.824+0000 7fc337426740 1 -- start start 2024-03-18T00:59:26.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc337426740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc337426740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037cb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc337426740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc337426740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1aaf70 con 0x5649037bab60 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc337426740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1acab0 con 0x5649037bfcd0 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc337426740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1ab640 con 0x5649037ba170 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc3324f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc331cf6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037cb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc3324f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:45718/0 (socket says 172.21.15.49:45718) 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc3324f7640 1 -- 172.21.15.49:0/4928688 learned_addr learned my addr 172.21.15.49:0/4928688 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc332cf8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc3324f7640 1 -- 172.21.15.49:0/4928688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 msgr2=0x5649037b5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4928688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037b5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc3324f7640 1 -- 172.21.15.49:0/4928688 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 msgr2=0x5649037cb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4928688 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037cb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc3324f7640 1 -- 172.21.15.49:0/4928688 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649037c3590 con 0x5649037ba170 2024-03-18T00:59:26.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc331cf6640 1 --2- 172.21.15.49:0/4928688 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:26.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.825+0000 7fc3324f7640 1 --2- 172.21.15.49:0/4928688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4900 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc32804a030 tx=0x7fc3280bb0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a13ad20cb47fcfaf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.826+0000 7fc317fff640 1 -- 172.21.15.49:0/4928688 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280da070 con 0x5649037ba170 2024-03-18T00:59:26.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.826+0000 7fc317fff640 1 -- 172.21.15.49:0/4928688 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3280bd2e0 con 0x5649037ba170 2024-03-18T00:59:26.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.826+0000 7fc317fff640 1 -- 172.21.15.49:0/4928688 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280a7040 con 0x5649037ba170 2024-03-18T00:59:26.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 -- 172.21.15.49:0/4928688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 msgr2=0x5649037c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 --2- 172.21.15.49:0/4928688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4900 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc32804a030 tx=0x7fc3280bb0f0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 -- 172.21.15.49:0/4928688 shutdown_connections 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 --2- 172.21.15.49:0/4928688 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037b5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 --2- 172.21.15.49:0/4928688 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037cb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 --2- 172.21.15.49:0/4928688 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c4900 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 -- 172.21.15.49:0/4928688 >> 172.21.15.49:0/4928688 conn(0x5649037b56e0 msgr2=0x5649037b8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 -- 172.21.15.49:0/4928688 shutdown_connections 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 -- 172.21.15.49:0/4928688 wait complete. 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 -- start start 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 Processor -- start 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1ac3e0 con 0x5649037bab60 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1abd10 con 0x5649037bfcd0 2024-03-18T00:59:26.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.827+0000 7fc337426740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc32c1a8f50 con 0x5649037ba170 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc337426740 10 client.0 _ll_register_callbacks cb 0x7fc31c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc331cf6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc3324f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc3324f7640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:45730/0 (socket says 172.21.15.49:45730) 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc332cf8640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc3324f7640 1 -- 172.21.15.49:0/704773156 learned_addr learned my addr 172.21.15.49:0/704773156 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc331cf6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37704/0 (socket says 172.21.15.49:37704) 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc3324f7640 1 -- 172.21.15.49:0/704773156 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 msgr2=0x5649037ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc3324f7640 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc3324f7640 1 -- 172.21.15.49:0/704773156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 msgr2=0x5649037c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc3324f7640 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc3324f7640 1 -- 172.21.15.49:0/704773156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649033f30f0 con 0x5649037ba170 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc331cf6640 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc3324f7640 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c8fb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc3280bae90 tx=0x7fc3280044f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a86aa2ab402a54c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc315ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc315ffb640 1 -- 172.21.15.49:0/704773156 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3280ca060 con 0x5649037ba170 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc337426740 1 -- 172.21.15.49:0/704773156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3280bd2e0 con 0x5649037ba170 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.828+0000 7fc315ffb640 1 -- 172.21.15.49:0/704773156 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc3280db450 con 0x5649037ba170 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc315ffb640 1 -- 172.21.15.49:0/704773156 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc3280da050 con 0x5649037ba170 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc337426740 10 client.26944 fetch_fsmap learned FSMap version 28 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc337426740 1 -- 172.21.15.49:0/704773156 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5649036e69d0 con 0x5649037ba170 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc315ffb640 1 -- 172.21.15.49:0/704773156 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3280bd2e0 con 0x5649037ba170 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc315ffb640 1 -- 172.21.15.49:0/704773156 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc32803c040 con 0x5649037ba170 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc337426740 10 client.26944 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc337426740 10 client.26944 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc337426740 20 client.26944 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc337426740 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564903835f00 0x564903838320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc337426740 4 client.26944 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc337426740 1 -- 172.21.15.49:0/704773156 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564903726d90 con 0x564903835f00 2024-03-18T00:59:26.833 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.829+0000 7fc331cf6640 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564903835f00 0x564903838320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.830+0000 7fc315ffb640 10 client.26944 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:26.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.830+0000 7fc331cf6640 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564903835f00 0x564903838320 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c9577ec5f94677d server_cookie=b6ec268137fa2ec2 in_seq=0 out_seq=0 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc315ffb640 1 -- 172.21.15.49:0/704773156 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x564903726d90 con 0x564903835f00 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc315ffb640 10 client.26944 handle_command_reply: tid=0 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 client.26944 shutdown 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 -- 172.21.15.49:0/704773156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 msgr2=0x5649037c8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c8fb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc3280bae90 tx=0x7fc3280044f0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 -- 172.21.15.49:0/704773156 shutdown_connections 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564903835f00 0x564903838320 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5649037bfcd0 0x5649037ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5649037bab60 0x5649037c94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 --2- 172.21.15.49:0/704773156 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5649037ba170 0x5649037c8fb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc315ffb640 0 client.26944 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 -- 172.21.15.49:0/704773156 >> 172.21.15.49:0/704773156 conn(0x5649037b56e0 msgr2=0x5649037c4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 -- 172.21.15.49:0/704773156 shutdown_connections 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 1 -- 172.21.15.49:0/704773156 wait complete. 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.833+0000 7fc337426740 20 client.26944 trim_cache size 0 max 16384 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "scrub active (1 inodes in the stack)", 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stdout: "32989747-5ead-4958-8622-ec6c54a01911": { 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "32989747-5ead-4958-8622-ec6c54a01911", 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T00:59:26.836 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:59:26.837 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T00:59:26.837 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:59:26.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 -- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3180a2800 msgr2=0x7fc3180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3180a2800 0x7fc3180a4cc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc324005fd0 tx=0x7fc32400feb0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 -- 172.21.15.49:0/4001817647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc32c1006e0 msgr2=0x7fc32c104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc32c1006e0 0x7fc32c104670 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc328009780 tx=0x7fc328032430 comp rx=0 tx=0).stop 2024-03-18T00:59:26.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 -- 172.21.15.49:0/4001817647 shutdown_connections 2024-03-18T00:59:26.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc3180a2800 0x7fc3180a4cc0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc32c102290 0x7fc32c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc32c1018d0 0x7fc32c102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 --2- 172.21.15.49:0/4001817647 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc32c1006e0 0x7fc32c104670 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 -- 172.21.15.49:0/4001817647 >> 172.21.15.49:0/4001817647 conn(0x7fc32c0fbc70 msgr2=0x7fc32c0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 -- 172.21.15.49:0/4001817647 shutdown_connections 2024-03-18T00:59:26.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:26.834+0000 7fc33475b640 1 -- 172.21.15.49:0/4001817647 wait complete. 2024-03-18T00:59:26.845 INFO:tasks.cephfs.filesystem:scrub status for tag:32989747-5ead-4958-8622-ec6c54a01911 - {'path': '/', 'tag': '32989747-5ead-4958-8622-ec6c54a01911', 'options': 'recursive,force'} 2024-03-18T00:59:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.851+0000 7f8dbca81640 1 -- 172.21.15.49:0/2540731260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db81018f0 msgr2=0x7f8db8101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.851+0000 7f8dbca81640 1 --2- 172.21.15.49:0/2540731260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db81018f0 0x7f8db8101d70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009900 tx=0x7f8da002bc30 comp rx=0 tx=0).stop 2024-03-18T00:59:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.851+0000 7f8dbca81640 1 -- 172.21.15.49:0/2540731260 shutdown_connections 2024-03-18T00:59:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.851+0000 7f8dbca81640 1 --2- 172.21.15.49:0/2540731260 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8db81022b0 0x7f8db8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.851+0000 7f8dbca81640 1 --2- 172.21.15.49:0/2540731260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db81018f0 0x7f8db8101d70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.851+0000 7f8dbca81640 1 --2- 172.21.15.49:0/2540731260 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8db8100700 0x7f8db8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.851+0000 7f8dbca81640 1 -- 172.21.15.49:0/2540731260 >> 172.21.15.49:0/2540731260 conn(0x7f8db80fbc70 msgr2=0x7f8db80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.851+0000 7f8dbca81640 1 -- 172.21.15.49:0/2540731260 shutdown_connections 2024-03-18T00:59:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.851+0000 7f8dbca81640 1 -- 172.21.15.49:0/2540731260 wait complete. 2024-03-18T00:59:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dbca81640 1 Processor -- start 2024-03-18T00:59:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dbca81640 1 -- start start 2024-03-18T00:59:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dbca81640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db8100700 0x7f8db80699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dbca81640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8db81018f0 0x7f8db819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dbca81640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8db81022b0 0x7f8db819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dbca81640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8db810c360 con 0x7f8db81018f0 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dbca81640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8db810c1e0 con 0x7f8db81022b0 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dbca81640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8db810c4e0 con 0x7f8db8100700 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8db81018f0 0x7f8db819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8db659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db8100700 0x7f8db80699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8db81018f0 0x7f8db819c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37720/0 (socket says 172.21.15.49:37720) 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8db659c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db8100700 0x7f8db80699d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:45740/0 (socket says 172.21.15.49:45740) 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dadd9b640 1 -- 172.21.15.49:0/3891416676 learned_addr learned my addr 172.21.15.49:0/3891416676 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8db81022b0 0x7f8db819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8db659c640 1 -- 172.21.15.49:0/3891416676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8db81022b0 msgr2=0x7f8db819ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.854 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8db659c640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8db81022b0 0x7f8db819ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8db659c640 1 -- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8db81018f0 msgr2=0x7f8db819c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8db659c640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8db81018f0 0x7f8db819c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8db659c640 1 -- 172.21.15.49:0/3891416676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db819f6f0 con 0x7f8db8100700 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.852+0000 7f8dadd9b640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8db81018f0 0x7f8db819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.853+0000 7f8db659c640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db8100700 0x7f8db80699d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8da800e9c0 tx=0x7f8da800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.853+0000 7f8daffff640 1 -- 172.21.15.49:0/3891416676 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da800ccc0 con 0x7f8db8100700 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.853+0000 7f8dbca81640 1 -- 172.21.15.49:0/3891416676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db819f980 con 0x7f8db8100700 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.853+0000 7f8daffff640 1 -- 172.21.15.49:0/3891416676 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8da80035f0 con 0x7f8db8100700 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.853+0000 7f8daffff640 1 -- 172.21.15.49:0/3891416676 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da800ce60 con 0x7f8db8100700 2024-03-18T00:59:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.853+0000 7f8dbca81640 1 -- 172.21.15.49:0/3891416676 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db819fef0 con 0x7f8db8100700 2024-03-18T00:59:26.860 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.855+0000 7f8daffff640 1 -- 172.21.15.49:0/3891416676 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8da80037a0 con 0x7f8db8100700 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.855+0000 7f8dbf66b740 1 Processor -- start 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8dbf66b740 1 -- start start 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a8203760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8daffff640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8da40a28a0 0x7f8da40a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203d30 0x55b8a8208830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 0x55b8a820b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b8a8168a40 con 0x55b8a8203d30 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b8a8111920 con 0x55b8a8208ea0 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b8a8164ae0 con 0x55b8a8203340 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203d30 0x55b8a8208830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db659c640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8da40a28a0 0x7f8da40a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a8203760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a8203760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:45746/0 (socket says 172.21.15.49:45746) 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db6d9d640 1 -- 172.21.15.49:0/3256991110 learned_addr learned my addr 172.21.15.49:0/3256991110 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.861 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/3256991110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 0x55b8a820b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8daffff640 1 -- 172.21.15.49:0/3891416676 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8da80ceea0 con 0x7f8db8100700 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db6d9d640 1 -- 172.21.15.49:0/3256991110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 msgr2=0x55b8a820b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/3256991110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 0x55b8a820b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db6d9d640 1 -- 172.21.15.49:0/3256991110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203d30 msgr2=0x55b8a8208830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/3256991110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203d30 0x55b8a8208830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.856+0000 7f8db6d9d640 1 -- 172.21.15.49:0/3256991110 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8a8129460 con 0x55b8a8203340 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8db659c640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8da40a28a0 0x7f8da40a4d60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8da8097c80 tx=0x7f8da80665e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/3256991110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a8203760 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8db0005f40 tx=0x7f8db0009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39f9ed02268788fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dacd99640 1 -- 172.21.15.49:0/3256991110 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0011070 con 0x55b8a8203340 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dacd99640 1 -- 172.21.15.49:0/3256991110 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db0003e50 con 0x55b8a8203340 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dacd99640 1 -- 172.21.15.49:0/3256991110 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db00042b0 con 0x55b8a8203340 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3256991110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 msgr2=0x55b8a8203760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3256991110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a8203760 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8db0005f40 tx=0x7f8db0009b80 comp rx=0 tx=0).stop 2024-03-18T00:59:26.862 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3256991110 shutdown_connections 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3256991110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 0x55b8a820b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3256991110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203d30 0x55b8a8208830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3256991110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a8203760 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.857+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3256991110 >> 172.21.15.49:0/3256991110 conn(0x55b8a81fe8b0 msgr2=0x55b8a81fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3256991110 shutdown_connections 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3256991110 wait complete. 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 -- start start 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 Processor -- start 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a820d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203d30 0x55b8a820db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 0x55b8a8214bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55b8a820be50 con 0x55b8a8203340 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55b8a8124d70 con 0x55b8a8208ea0 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55b8a820bfd0 con 0x55b8a8203d30 2024-03-18T00:59:26.863 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dbf66b740 10 client.0 _ll_register_callbacks cb 0x55b8a7e7ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203d30 0x55b8a820db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a820d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dadd9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203d30 0x55b8a820db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:45750/0 (socket says 172.21.15.49:45750) 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8dadd9b640 1 -- 172.21.15.49:0/1698859160 learned_addr learned my addr 172.21.15.49:0/1698859160 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a820d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37742/0 (socket says 172.21.15.49:37742) 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.858+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 0x55b8a8214bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8dadd9b640 1 -- 172.21.15.49:0/1698859160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 msgr2=0x55b8a8214bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8dadd9b640 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 0x55b8a8214bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8dadd9b640 1 -- 172.21.15.49:0/1698859160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 msgr2=0x55b8a820d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8dadd9b640 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a820d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8dadd9b640 1 -- 172.21.15.49:0/1698859160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8a7e7d210 con 0x55b8a8203d30 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8dadd9b640 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203d30 0x55b8a820db70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009230 tx=0x7f8da00048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10dba62cc58762f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8d9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8d9dffb640 1 -- 172.21.15.49:0/1698859160 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0005d50 con 0x55b8a8203d30 2024-03-18T00:59:26.864 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8d9dffb640 1 -- 172.21.15.49:0/1698859160 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8da0004ef0 con 0x55b8a8203d30 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1698859160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8db0003e50 con 0x55b8a8203d30 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8d9dffb640 1 -- 172.21.15.49:0/1698859160 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8da0004ce0 con 0x55b8a8203d30 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8dbf66b740 10 client.26956 fetch_fsmap learned FSMap version 28 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1698859160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b8a7f49620 con 0x55b8a8203d30 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.859+0000 7f8d9dffb640 1 -- 172.21.15.49:0/1698859160 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db0003e50 con 0x55b8a8203d30 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8d9dffb640 1 -- 172.21.15.49:0/1698859160 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8da0003ab0 con 0x55b8a8203d30 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8dbf66b740 10 client.26956 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8dbf66b740 10 client.26956 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8dbf66b740 20 client.26956 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b8a82194b0 0x55b8a821b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8dbf66b740 4 client.26956 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1698859160 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b8a8175f80 con 0x55b8a82194b0 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b8a82194b0 0x55b8a821b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.865 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b8a82194b0 0x55b8a821b920 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bdcec631ad39d7f server_cookie=5bb472ae58d9d69b in_seq=0 out_seq=0 2024-03-18T00:59:26.866 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.860+0000 7f8d9dffb640 10 client.26956 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.862+0000 7f8d9dffb640 1 -- 172.21.15.49:0/1698859160 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b8a8175f80 con 0x55b8a82194b0 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.862+0000 7f8d9dffb640 10 client.26956 handle_command_reply: tid=0 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.862+0000 7f8dbf66b740 1 client.26956 shutdown 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.862+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1698859160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203d30 msgr2=0x55b8a820db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.862+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203d30 0x55b8a820db70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8da0009230 tx=0x7f8da00048f0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1698859160 shutdown_connections 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b8a82194b0 0x55b8a821b920 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8208ea0 0x55b8a8214bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8d9dffb640 0 client.26956 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203d30 0x55b8a820db70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1698859160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a820d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1698859160 >> 172.21.15.49:0/1698859160 conn(0x55b8a81fe8b0 msgr2=0x55b8a81fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1698859160 shutdown_connections 2024-03-18T00:59:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1698859160 wait complete. 2024-03-18T00:59:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.863+0000 7f8dbf66b740 20 client.26956 trim_cache size 0 max 16384 2024-03-18T00:59:26.881 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.877+0000 7f8dbf66b740 1 Processor -- start 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.877+0000 7f8dbf66b740 1 -- start start 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a820b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 0x55b8a8214b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8208ea0 0x55b8a81fd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81a8c40 con 0x55b8a8208ea0 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81aa780 con 0x55b8a8203d30 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81a9310 con 0x55b8a8203340 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8208ea0 0x55b8a81fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8208ea0 0x55b8a81fd320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37758/0 (socket says 172.21.15.49:37758) 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 -- 172.21.15.49:0/1431397172 learned_addr learned my addr 172.21.15.49:0/1431397172 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8dadd9b640 1 --2- 172.21.15.49:0/1431397172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 0x55b8a8214b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/1431397172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a820b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 -- 172.21.15.49:0/1431397172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 msgr2=0x55b8a820b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/1431397172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a820b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 -- 172.21.15.49:0/1431397172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 msgr2=0x55b8a8214b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/1431397172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 0x55b8a8214b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 -- 172.21.15.49:0/1431397172 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b8a8202f80 con 0x55b8a8208ea0 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/1431397172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8208ea0 0x55b8a81fd320 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f8db0006010 tx=0x7f8db0009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=281198b0963920eb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8d9ffff640 1 -- 172.21.15.49:0/1431397172 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db002eb40 con 0x55b8a8208ea0 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8d9ffff640 1 -- 172.21.15.49:0/1431397172 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db0024070 con 0x55b8a8208ea0 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.878+0000 7f8d9ffff640 1 -- 172.21.15.49:0/1431397172 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db002e590 con 0x55b8a8208ea0 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1431397172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8208ea0 msgr2=0x55b8a81fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1431397172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8208ea0 0x55b8a81fd320 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f8db0006010 tx=0x7f8db0009fd0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1431397172 shutdown_connections 2024-03-18T00:59:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1431397172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 0x55b8a8214b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1431397172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8208ea0 0x55b8a81fd320 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/1431397172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8203340 0x55b8a820b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1431397172 >> 172.21.15.49:0/1431397172 conn(0x55b8a81fe8b0 msgr2=0x55b8a81f9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1431397172 shutdown_connections 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 -- 172.21.15.49:0/1431397172 wait complete. 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.879+0000 7f8dbf66b740 1 -- start start 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dbf66b740 1 Processor -- start 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a8212370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 0x55b8a82128b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dbf66b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8208ea0 0x55b8a82137b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81aa0b0 con 0x55b8a8203340 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81a99e0 con 0x55b8a8203d30 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dbf66b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8db81a6c20 con 0x55b8a8208ea0 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dbf66b740 10 client.0 _ll_register_callbacks cb 0x7f8da0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:26.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a8212370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8db6d9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a8212370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37766/0 (socket says 172.21.15.49:37766) 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8db6d9d640 1 -- 172.21.15.49:0/3923256676 learned_addr learned my addr 172.21.15.49:0/3923256676 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dadd9b640 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 0x55b8a82128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8208ea0 0x55b8a82137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dadd9b640 1 -- 172.21.15.49:0/3923256676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8208ea0 msgr2=0x55b8a82137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dadd9b640 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8208ea0 0x55b8a82137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dadd9b640 1 -- 172.21.15.49:0/3923256676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 msgr2=0x55b8a8212370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dadd9b640 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a8212370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dadd9b640 1 -- 172.21.15.49:0/3923256676 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b8a8213da0 con 0x55b8a8203d30 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a8212370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8dadd9b640 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 0x55b8a82128b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8da0036040 tx=0x7f8da0009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cea2f33f9d97334d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.880+0000 7f8d9dffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:59:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8d9dffb640 1 -- 172.21.15.49:0/3923256676 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0061060 con 0x55b8a8203d30 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3923256676 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8db0024070 con 0x55b8a8203d30 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8d9dffb640 1 -- 172.21.15.49:0/3923256676 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8da0004960 con 0x55b8a8203d30 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8d9dffb640 1 -- 172.21.15.49:0/3923256676 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8da0058040 con 0x55b8a8203d30 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8dbf66b740 10 client.26870 fetch_fsmap learned FSMap version 28 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3923256676 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b8a827f8d0 con 0x55b8a8203d30 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8d9dffb640 1 -- 172.21.15.49:0/3923256676 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8db0024070 con 0x55b8a8203d30 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8d9dffb640 1 -- 172.21.15.49:0/3923256676 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8da0003710 con 0x55b8a8203d30 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8dbf66b740 10 client.26870 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8dbf66b740 10 client.26870 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8dbf66b740 20 client.26870 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b8a82194b0 0x55b8a82822b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8dbf66b740 4 client.26870 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3923256676 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b8a81702d0 con 0x55b8a82194b0 2024-03-18T00:59:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.881+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b8a82194b0 0x55b8a82822b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.882+0000 7f8db6d9d640 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b8a82194b0 0x55b8a82822b0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bef7a562ed3a48b1 server_cookie=ea7a3b6c53c133de in_seq=0 out_seq=0 2024-03-18T00:59:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:26.882+0000 7f8d9dffb640 10 client.26870 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:27.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:27 smithi107 ceph-mon[36270]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 21 op/s 2024-03-18T00:59:27.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:27 smithi106 ceph-mon[35899]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 21 op/s 2024-03-18T00:59:28.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:27 smithi049 ceph-mon[32130]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 21 op/s 2024-03-18T00:59:29.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:29 smithi107 ceph-mon[36270]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 21 op/s 2024-03-18T00:59:29.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:29 smithi106 ceph-mon[35899]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 21 op/s 2024-03-18T00:59:30.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:29 smithi049 ceph-mon[32130]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.8 MiB/s wr, 21 op/s 2024-03-18T00:59:31.927 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:31 smithi049 ceph-mon[32130]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-03-18T00:59:31.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:31 smithi107 ceph-mon[36270]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-03-18T00:59:31.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:31 smithi106 ceph-mon[35899]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2024-03-18T00:59:33.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:33 smithi107 ceph-mon[36270]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 31 op/s 2024-03-18T00:59:33.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:33 smithi106 ceph-mon[35899]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 31 op/s 2024-03-18T00:59:34.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:33 smithi049 ceph-mon[32130]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 3.1 MiB/s wr, 31 op/s 2024-03-18T00:59:35.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:35 smithi107 ceph-mon[36270]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2024-03-18T00:59:35.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:35 smithi106 ceph-mon[35899]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2024-03-18T00:59:36.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:35 smithi049 ceph-mon[32130]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2024-03-18T00:59:37.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:37 smithi107 ceph-mon[36270]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 32 op/s 2024-03-18T00:59:37.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:37 smithi106 ceph-mon[35899]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 32 op/s 2024-03-18T00:59:38.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:37 smithi049 ceph-mon[32130]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 2.9 MiB/s wr, 32 op/s 2024-03-18T00:59:39.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:39 smithi107 ceph-mon[36270]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-03-18T00:59:39.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:39 smithi106 ceph-mon[35899]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-03-18T00:59:40.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:39 smithi049 ceph-mon[32130]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-03-18T00:59:41.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:41 smithi107 ceph-mon[36270]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2024-03-18T00:59:41.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:41 smithi106 ceph-mon[35899]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2024-03-18T00:59:42.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:41 smithi049 ceph-mon[32130]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.0 MiB/s wr, 31 op/s 2024-03-18T00:59:43.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:43 smithi107 ceph-mon[36270]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 37 op/s 2024-03-18T00:59:43.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:43 smithi106 ceph-mon[35899]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 37 op/s 2024-03-18T00:59:44.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:43 smithi049 ceph-mon[32130]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.3 MiB/s wr, 37 op/s 2024-03-18T00:59:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.230+0000 7f8d9dffb640 1 -- 172.21.15.49:0/3923256676 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b8a81702d0 con 0x55b8a82194b0 2024-03-18T00:59:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.230+0000 7f8d9dffb640 10 client.26870 handle_command_reply: tid=0 2024-03-18T00:59:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.230+0000 7f8dbf66b740 1 client.26870 shutdown 2024-03-18T00:59:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.230+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3923256676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 msgr2=0x55b8a82128b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.230+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 0x55b8a82128b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8da0036040 tx=0x7f8da0009260 comp rx=0 tx=0).stop 2024-03-18T00:59:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.230+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3923256676 shutdown_connections 2024-03-18T00:59:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.230+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55b8a82194b0 0x55b8a82822b0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:45.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.231+0000 7f8d9dffb640 0 client.26870 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.231+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55b8a8203d30 0x55b8a82128b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.231+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55b8a8208ea0 0x55b8a82137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.231+0000 7f8dbf66b740 1 --2- 172.21.15.49:0/3923256676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55b8a8203340 0x55b8a8212370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.231+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3923256676 >> 172.21.15.49:0/3923256676 conn(0x55b8a81fe8b0 msgr2=0x55b8a8201b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.231+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3923256676 shutdown_connections 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.231+0000 7f8dbf66b740 1 -- 172.21.15.49:0/3923256676 wait complete. 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.231+0000 7f8dbf66b740 20 client.26870 trim_cache size 0 max 16384 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:59:45.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:59:45.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 -- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8da40a28a0 msgr2=0x7f8da40a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:45.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8da40a28a0 0x7f8da40a4d60 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8da8097c80 tx=0x7f8da80665e0 comp rx=0 tx=0).stop 2024-03-18T00:59:45.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 -- 172.21.15.49:0/3891416676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db8100700 msgr2=0x7f8db80699d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db8100700 0x7f8db80699d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8da800e9c0 tx=0x7f8da800ee80 comp rx=0 tx=0).stop 2024-03-18T00:59:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 -- 172.21.15.49:0/3891416676 shutdown_connections 2024-03-18T00:59:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8da40a28a0 0x7f8da40a4d60 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8db81022b0 0x7f8db819ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8db81018f0 0x7f8db819c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 --2- 172.21.15.49:0/3891416676 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8db8100700 0x7f8db80699d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 -- 172.21.15.49:0/3891416676 >> 172.21.15.49:0/3891416676 conn(0x7f8db80fbc70 msgr2=0x7f8db8104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 -- 172.21.15.49:0/3891416676 shutdown_connections 2024-03-18T00:59:45.234 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:45.232+0000 7f8dbca81640 1 -- 172.21.15.49:0/3891416676 wait complete. 2024-03-18T00:59:45.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:45 smithi107 ceph-mon[36270]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 24 op/s 2024-03-18T00:59:45.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:45 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T00:59:45.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:45 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:59:45.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:45 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T00:59:45.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:45 smithi106 ceph-mon[35899]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 24 op/s 2024-03-18T00:59:45.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:45 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T00:59:45.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:45 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:59:45.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:45 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T00:59:46.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:45 smithi049 ceph-mon[32130]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 24 op/s 2024-03-18T00:59:46.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:45 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T00:59:46.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:45 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:59:46.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:45 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T00:59:46.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:46 smithi106 ceph-mon[35899]: MDS health message cleared (mds.?): Client smithi049:0 failing to advance its oldest client/flush tid. 2024-03-18T00:59:47.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:46 smithi049 ceph-mon[32130]: MDS health message cleared (mds.?): Client smithi049:0 failing to advance its oldest client/flush tid. 2024-03-18T00:59:47.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:46 smithi107 ceph-mon[36270]: MDS health message cleared (mds.?): Client smithi049:0 failing to advance its oldest client/flush tid. 2024-03-18T00:59:47.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:47 smithi106 ceph-mon[35899]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.0 MiB/s wr, 40 op/s 2024-03-18T00:59:47.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:47 smithi106 ceph-mon[35899]: Health check cleared: MDS_CLIENT_OLDEST_TID (was: 1 clients failing to advance oldest client/flush tid) 2024-03-18T00:59:47.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:47 smithi106 ceph-mon[35899]: Cluster is now healthy 2024-03-18T00:59:47.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:47 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:active 2024-03-18T00:59:47.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:47 smithi106 ceph-mon[35899]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:59:47.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:47 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:59:48.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:47 smithi049 ceph-mon[32130]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.0 MiB/s wr, 40 op/s 2024-03-18T00:59:48.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:47 smithi049 ceph-mon[32130]: Health check cleared: MDS_CLIENT_OLDEST_TID (was: 1 clients failing to advance oldest client/flush tid) 2024-03-18T00:59:48.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:47 smithi049 ceph-mon[32130]: Cluster is now healthy 2024-03-18T00:59:48.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:47 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:active 2024-03-18T00:59:48.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:47 smithi049 ceph-mon[32130]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:59:48.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:47 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:59:48.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:47 smithi107 ceph-mon[36270]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 2.0 MiB/s wr, 40 op/s 2024-03-18T00:59:48.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:47 smithi107 ceph-mon[36270]: Health check cleared: MDS_CLIENT_OLDEST_TID (was: 1 clients failing to advance oldest client/flush tid) 2024-03-18T00:59:48.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:47 smithi107 ceph-mon[36270]: Cluster is now healthy 2024-03-18T00:59:48.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:47 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] up:active 2024-03-18T00:59:48.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:47 smithi107 ceph-mon[36270]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:59:48.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:47 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T00:59:49.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:49 smithi106 ceph-mon[35899]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s rd, 4.4 MiB/s wr, 309 op/s 2024-03-18T00:59:49.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:59:49.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:59:49.946 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:59:50.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:49 smithi049 ceph-mon[32130]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s rd, 4.4 MiB/s wr, 309 op/s 2024-03-18T00:59:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:59:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:59:50.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:59:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:49 smithi107 ceph-mon[36270]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s rd, 4.4 MiB/s wr, 309 op/s 2024-03-18T00:59:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T00:59:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T00:59:50.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T00:59:51.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:51 smithi106 ceph-mon[35899]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s rd, 14 MiB/s wr, 539 op/s 2024-03-18T00:59:51.981 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:51 smithi107 ceph-mon[36270]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s rd, 14 MiB/s wr, 539 op/s 2024-03-18T00:59:52.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:51 smithi049 ceph-mon[32130]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 88 MiB/s rd, 14 MiB/s wr, 539 op/s 2024-03-18T00:59:54.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:53 smithi049 ceph-mon[32130]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 MiB/s rd, 142 MiB/s wr, 2.16k op/s 2024-03-18T00:59:54.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:53 smithi107 ceph-mon[36270]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 MiB/s rd, 142 MiB/s wr, 2.16k op/s 2024-03-18T00:59:54.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:53 smithi106 ceph-mon[35899]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 191 MiB/s rd, 142 MiB/s wr, 2.16k op/s 2024-03-18T00:59:55.997 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:55 smithi106 ceph-mon[35899]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 204 MiB/s rd, 223 MiB/s wr, 3.04k op/s 2024-03-18T00:59:55.997 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:55 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:59:55.997 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:55 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T00:59:56.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:55 smithi049 ceph-mon[32130]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 204 MiB/s rd, 223 MiB/s wr, 3.04k op/s 2024-03-18T00:59:56.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:55 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:59:56.126 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:55 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T00:59:56.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:55 smithi107 ceph-mon[36270]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 204 MiB/s rd, 223 MiB/s wr, 3.04k op/s 2024-03-18T00:59:56.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:55 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:59:56.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:55 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T00:59:56.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T00:59:56.847 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T00:59:57.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.115+0000 7fa8214b2640 1 -- 172.21.15.49:0/1943325822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa81c1097e0 msgr2=0x7fa81c10e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.115 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.115+0000 7fa8214b2640 1 --2- 172.21.15.49:0/1943325822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa81c1097e0 0x7fa81c10e080 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa81001a910 tx=0x7fa81003cc80 comp rx=0 tx=0).stop 2024-03-18T00:59:57.124 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.124+0000 7fa8214b2640 1 -- 172.21.15.49:0/1943325822 shutdown_connections 2024-03-18T00:59:57.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.124+0000 7fa8214b2640 1 --2- 172.21.15.49:0/1943325822 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa81c1097e0 0x7fa81c10e080 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.124+0000 7fa8214b2640 1 --2- 172.21.15.49:0/1943325822 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa81c108e20 0x7fa81c1092a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.124+0000 7fa8214b2640 1 --2- 172.21.15.49:0/1943325822 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81c072b50 0x7fa81c072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.125 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.124+0000 7fa8214b2640 1 -- 172.21.15.49:0/1943325822 >> 172.21.15.49:0/1943325822 conn(0x7fa81c06e600 msgr2=0x7fa81c070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:57.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:56 smithi049 ceph-mon[32130]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:59:57.129 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.127+0000 7fa8214b2640 1 -- 172.21.15.49:0/1943325822 shutdown_connections 2024-03-18T00:59:57.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.131+0000 7fa8214b2640 1 -- 172.21.15.49:0/1943325822 wait complete. 2024-03-18T00:59:57.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.131+0000 7fa8214b2640 1 Processor -- start 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.134+0000 7fa8214b2640 1 -- start start 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.135+0000 7fa8214b2640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81c072b50 0x7fa81c19df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.135+0000 7fa8214b2640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa81c108e20 0x7fa81c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.135+0000 7fa8214b2640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa81c1097e0 0x7fa81c1a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.135+0000 7fa8214b2640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c111a90 con 0x7fa81c072b50 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.135+0000 7fa8214b2640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c111910 con 0x7fa81c1097e0 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.135+0000 7fa8214b2640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c111c10 con 0x7fa81c108e20 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.135+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa81c108e20 0x7fa81c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.135+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa81c108e20 0x7fa81c19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38698/0 (socket says 172.21.15.49:38698) 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.135+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2405034755 learned_addr learned my addr 172.21.15.49:0/2405034755 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.136+0000 7fa81affd640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81c072b50 0x7fa81c19df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.136+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2405034755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa81c1097e0 msgr2=0x7fa81c1a54b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:59:57.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.136+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa81c1097e0 0x7fa81c1a54b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.136+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81c072b50 msgr2=0x7fa81c19df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.136+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81c072b50 0x7fa81c19df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.136+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2405034755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c1a5b20 con 0x7fa81c108e20 2024-03-18T00:59:57.138 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.137+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa81c108e20 0x7fa81c19e440 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa81400ea80 tx=0x7fa81400ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:57.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.139+0000 7fa803fff640 1 -- 172.21.15.49:0/2405034755 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81400ce40 con 0x7fa81c108e20 2024-03-18T00:59:57.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.139+0000 7fa803fff640 1 -- 172.21.15.49:0/2405034755 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa814011950 con 0x7fa81c108e20 2024-03-18T00:59:57.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.139+0000 7fa803fff640 1 -- 172.21.15.49:0/2405034755 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81401abb0 con 0x7fa81c108e20 2024-03-18T00:59:57.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.140+0000 7fa8214b2640 1 -- 172.21.15.49:0/2405034755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81c1a5e10 con 0x7fa81c108e20 2024-03-18T00:59:57.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.140+0000 7fa8214b2640 1 -- 172.21.15.49:0/2405034755 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81c1a6300 con 0x7fa81c108e20 2024-03-18T00:59:57.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.148+0000 7fa82417c740 1 Processor -- start 2024-03-18T00:59:57.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.150+0000 7fa803fff640 1 -- 172.21.15.49:0/2405034755 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa814011470 con 0x7fa81c108e20 2024-03-18T00:59:57.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.151+0000 7fa803fff640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa8040a2a00 0x7fa8040a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.151+0000 7fa803fff640 1 -- 172.21.15.49:0/2405034755 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa8140cef00 con 0x7fa81c108e20 2024-03-18T00:59:57.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.151+0000 7fa82417c740 1 -- start start 2024-03-18T00:59:57.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.151+0000 7fa82417c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1360 0x55c2b3df1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.151 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.152+0000 7fa82417c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3df6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.152+0000 7fa82417c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3df9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.152+0000 7fa82417c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c2b3d56500 con 0x55c2b3df6ec0 2024-03-18T00:59:57.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.152+0000 7fa82417c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c2b3cfebe0 con 0x55c2b3df1d50 2024-03-18T00:59:57.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.152+0000 7fa82417c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c2b3d525a0 con 0x55c2b3df1360 2024-03-18T00:59:57.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.152+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3df6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.152+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3df6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:35276/0 (socket says 172.21.15.49:35276) 2024-03-18T00:59:57.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.152+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2258622450 learned_addr learned my addr 172.21.15.49:0/2258622450 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:57.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.153+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/2258622450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3df9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.155+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2258622450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1360 msgr2=0x55c2b3df1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:57.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.155+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2258622450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1360 0x55c2b3df1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.155+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2258622450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 msgr2=0x55c2b3df9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:57.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.155+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2258622450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3df9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.155+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2258622450 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2b3d124b0 con 0x55c2b3df1d50 2024-03-18T00:59:57.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.155+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2258622450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3df6850 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa8140a3a90 tx=0x7fa81409c060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c01a61fd07ec068 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:57.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.161+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/2258622450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3df9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T00:59:57.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.161+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa8040a2a00 0x7fa8040a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.163+0000 7fa8017fa640 1 -- 172.21.15.49:0/2258622450 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8140a3fc0 con 0x55c2b3df1d50 2024-03-18T00:59:57.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.163+0000 7fa8017fa640 1 -- 172.21.15.49:0/2258622450 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa814011760 con 0x55c2b3df1d50 2024-03-18T00:59:57.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.164+0000 7fa82417c740 1 -- 172.21.15.49:0/2258622450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 msgr2=0x55c2b3df6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.164+0000 7fa82417c740 1 --2- 172.21.15.49:0/2258622450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3df6850 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa8140a3a90 tx=0x7fa81409c060 comp rx=0 tx=0).stop 2024-03-18T00:59:57.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.167+0000 7fa82417c740 1 -- 172.21.15.49:0/2258622450 shutdown_connections 2024-03-18T00:59:57.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.167+0000 7fa82417c740 1 --2- 172.21.15.49:0/2258622450 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3df6850 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.167+0000 7fa82417c740 1 --2- 172.21.15.49:0/2258622450 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3df9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.167+0000 7fa82417c740 1 --2- 172.21.15.49:0/2258622450 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1360 0x55c2b3df1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.167+0000 7fa82417c740 1 -- 172.21.15.49:0/2258622450 >> 172.21.15.49:0/2258622450 conn(0x55c2b3dec8d0 msgr2=0x55c2b3deccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:57.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.168+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa8040a2a00 0x7fa8040a4ec0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa8100047e0 tx=0x7fa810005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:57.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.170+0000 7fa82417c740 1 -- 172.21.15.49:0/2258622450 shutdown_connections 2024-03-18T00:59:57.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.171+0000 7fa82417c740 1 -- 172.21.15.49:0/2258622450 wait complete. 2024-03-18T00:59:57.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.171+0000 7fa82417c740 1 -- start start 2024-03-18T00:59:57.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.171+0000 7fa82417c740 1 Processor -- start 2024-03-18T00:59:57.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa82417c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1360 0x55c2b3dfbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa82417c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1d50 0x55c2b3dfc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa82417c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e030e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa82417c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c2b3df9e60 con 0x55c2b3df6ec0 2024-03-18T00:59:57.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa82417c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c2b3d12160 con 0x55c2b3df1360 2024-03-18T00:59:57.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa82417c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c2b3df9fe0 con 0x55c2b3df1d50 2024-03-18T00:59:57.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa82417c740 10 client.0 _ll_register_callbacks cb 0x55c2b3cf6150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1d50 0x55c2b3dfc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1d50 0x55c2b3dfc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38716/0 (socket says 172.21.15.49:38716) 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2100391116 learned_addr learned my addr 172.21.15.49:0/2100391116 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.172+0000 7fa81affd640 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1360 0x55c2b3dfbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.173+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.173+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2100391116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1360 msgr2=0x55c2b3dfbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.173+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1360 0x55c2b3dfbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.173+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2100391116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 msgr2=0x55c2b3e030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.173+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.173+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2100391116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2b3b347d0 con 0x55c2b3df1d50 2024-03-18T00:59:57.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.173+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:59:57.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.176+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1d50 0x55c2b3dfc0a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa81409b520 tx=0x7fa814011760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e40e27412a38446 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:57.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.177+0000 7fa7feff5640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:57.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:56 smithi107 ceph-mon[36270]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:59:57.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.179+0000 7fa7feff5640 1 -- 172.21.15.49:0/2100391116 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81409cb50 con 0x55c2b3df1d50 2024-03-18T00:59:57.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.179+0000 7fa7feff5640 1 -- 172.21.15.49:0/2100391116 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa81409ccf0 con 0x55c2b3df1d50 2024-03-18T00:59:57.182 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.182+0000 7fa82417c740 1 -- 172.21.15.49:0/2100391116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c2b3d63a40 con 0x55c2b3df1d50 2024-03-18T00:59:57.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.188+0000 7fa7feff5640 1 -- 172.21.15.49:0/2100391116 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa81409ac70 con 0x55c2b3df1d50 2024-03-18T00:59:57.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.189+0000 7fa82417c740 10 client.26971 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:57.189 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.189+0000 7fa82417c740 1 -- 172.21.15.49:0/2100391116 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c2b3a6a3f0 con 0x55c2b3df1d50 2024-03-18T00:59:57.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.193+0000 7fa7feff5640 1 -- 172.21.15.49:0/2100391116 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55c2b3d63a40 con 0x55c2b3df1d50 2024-03-18T00:59:57.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.194+0000 7fa7feff5640 1 -- 172.21.15.49:0/2100391116 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fa8140a7030 con 0x55c2b3df1d50 2024-03-18T00:59:57.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:56 smithi106 ceph-mon[35899]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T00:59:57.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.199+0000 7fa82417c740 10 client.26971 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:57.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.199+0000 7fa82417c740 10 client.26971 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:57.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.199+0000 7fa82417c740 20 client.26971 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:57.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.199+0000 7fa82417c740 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c2b3e07240 0x55c2b3e09630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.199+0000 7fa82417c740 4 client.26971 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:59:57.199 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.199+0000 7fa82417c740 1 -- 172.21.15.49:0/2100391116 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c2b3d5dd90 con 0x55c2b3e07240 2024-03-18T00:59:57.201 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.201+0000 7fa81affd640 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c2b3e07240 0x55c2b3e09630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.209+0000 7fa81affd640 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c2b3e07240 0x55c2b3e09630 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=377a18ea040badd8 server_cookie=60db492bb5d614ab in_seq=0 out_seq=0 2024-03-18T00:59:57.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.209+0000 7fa7feff5640 10 client.26971 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:57.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.210+0000 7fa7feff5640 1 -- 172.21.15.49:0/2100391116 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c2b3d5dd90 con 0x55c2b3e07240 2024-03-18T00:59:57.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.210+0000 7fa7feff5640 10 client.26971 handle_command_reply: tid=0 2024-03-18T00:59:57.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.210+0000 7fa82417c740 1 client.26971 shutdown 2024-03-18T00:59:57.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.211+0000 7fa82417c740 1 -- 172.21.15.49:0/2100391116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1d50 msgr2=0x55c2b3dfc0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.211 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.211+0000 7fa82417c740 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1d50 0x55c2b3dfc0a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa81409b520 tx=0x7fa814011760 comp rx=0 tx=0).stop 2024-03-18T00:59:57.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.214+0000 7fa82417c740 1 -- 172.21.15.49:0/2100391116 shutdown_connections 2024-03-18T00:59:57.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.214+0000 7fa82417c740 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c2b3e07240 0x55c2b3e09630 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.214+0000 7fa82417c740 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.214+0000 7fa82417c740 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1d50 0x55c2b3dfc0a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.214+0000 7fa82417c740 1 --2- 172.21.15.49:0/2100391116 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1360 0x55c2b3dfbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.214+0000 7fa82417c740 1 -- 172.21.15.49:0/2100391116 >> 172.21.15.49:0/2100391116 conn(0x55c2b3dec8d0 msgr2=0x55c2b3debb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:57.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.215+0000 7fa7feff5640 0 client.26971 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:57.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.215+0000 7fa82417c740 1 -- 172.21.15.49:0/2100391116 shutdown_connections 2024-03-18T00:59:57.215 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.215+0000 7fa82417c740 1 -- 172.21.15.49:0/2100391116 wait complete. 2024-03-18T00:59:57.218 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.217+0000 7fa82417c740 20 client.26971 trim_cache size 0 max 16384 2024-03-18T00:59:57.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.264+0000 7fa82417c740 1 Processor -- start 2024-03-18T00:59:57.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa82417c740 1 -- start start 2024-03-18T00:59:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa82417c740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1360 0x55c2b3dfae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa82417c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1d50 0x55c2b3e030a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa82417c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df6ec0 0x55c2b3dfb390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa82417c740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa80800e690 con 0x55c2b3df1d50 2024-03-18T00:59:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa82417c740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa808010200 con 0x55c2b3df6ec0 2024-03-18T00:59:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa82417c740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa80800ed60 con 0x55c2b3df1360 2024-03-18T00:59:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1d50 0x55c2b3e030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa81a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1d50 0x55c2b3e030a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57534/0 (socket says 172.21.15.49:57534) 2024-03-18T00:59:57.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa81a7fc640 1 -- 172.21.15.49:0/3594779070 learned_addr learned my addr 172.21.15.49:0/3594779070 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/3594779070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df6ec0 0x55c2b3dfb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.265+0000 7fa81affd640 1 --2- 172.21.15.49:0/3594779070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1360 0x55c2b3dfae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.266+0000 7fa81b7fe640 1 -- 172.21.15.49:0/3594779070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1360 msgr2=0x55c2b3dfae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.266+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/3594779070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1360 0x55c2b3dfae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.266+0000 7fa81b7fe640 1 -- 172.21.15.49:0/3594779070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1d50 msgr2=0x55c2b3e030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.266+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/3594779070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1d50 0x55c2b3e030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.266+0000 7fa81b7fe640 1 -- 172.21.15.49:0/3594779070 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2b3a2a0f0 con 0x55c2b3df6ec0 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.266+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/3594779070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df6ec0 0x55c2b3dfb390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa810004880 tx=0x7fa81004e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e7d40bee857dbbb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.266+0000 7fa800ff9640 1 -- 172.21.15.49:0/3594779070 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810055070 con 0x55c2b3df6ec0 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.266+0000 7fa800ff9640 1 -- 172.21.15.49:0/3594779070 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa810060070 con 0x55c2b3df6ec0 2024-03-18T00:59:57.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.266+0000 7fa800ff9640 1 -- 172.21.15.49:0/3594779070 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81003e440 con 0x55c2b3df6ec0 2024-03-18T00:59:57.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.268+0000 7fa82417c740 1 -- 172.21.15.49:0/3594779070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df6ec0 msgr2=0x55c2b3dfb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.268+0000 7fa82417c740 1 --2- 172.21.15.49:0/3594779070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df6ec0 0x55c2b3dfb390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa810004880 tx=0x7fa81004e000 comp rx=0 tx=0).stop 2024-03-18T00:59:57.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.268+0000 7fa82417c740 1 -- 172.21.15.49:0/3594779070 shutdown_connections 2024-03-18T00:59:57.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.268+0000 7fa82417c740 1 --2- 172.21.15.49:0/3594779070 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df6ec0 0x55c2b3dfb390 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.268+0000 7fa82417c740 1 --2- 172.21.15.49:0/3594779070 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1d50 0x55c2b3e030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.268+0000 7fa82417c740 1 --2- 172.21.15.49:0/3594779070 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df1360 0x55c2b3dfae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.268+0000 7fa82417c740 1 -- 172.21.15.49:0/3594779070 >> 172.21.15.49:0/3594779070 conn(0x55c2b3dec8d0 msgr2=0x55c2b3df34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:57.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.269+0000 7fa82417c740 1 -- 172.21.15.49:0/3594779070 shutdown_connections 2024-03-18T00:59:57.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.269+0000 7fa82417c740 1 -- 172.21.15.49:0/3594779070 wait complete. 2024-03-18T00:59:57.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.269+0000 7fa82417c740 1 -- start start 2024-03-18T00:59:57.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.269+0000 7fa82417c740 1 Processor -- start 2024-03-18T00:59:57.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.270+0000 7fa82417c740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1360 0x55c2b3dffcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.270+0000 7fa82417c740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3e001f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.270+0000 7fa81affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1360 0x55c2b3dffcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.270+0000 7fa81affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1360 0x55c2b3dffcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:57542/0 (socket says 172.21.15.49:57542) 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.270+0000 7fa81affd640 1 -- 172.21.15.49:0/2304748331 learned_addr learned my addr 172.21.15.49:0/2304748331 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.270+0000 7fa82417c740 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e6b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa80800fb00 con 0x55c2b3df1360 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa80800f430 con 0x55c2b3df1d50 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3e001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa80800c670 con 0x55c2b3df6ec0 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e6b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2304748331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df6ec0 msgr2=0x55c2b3e6b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e6b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2304748331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1360 msgr2=0x55c2b3dffcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1360 0x55c2b3dffcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa81a7fc640 1 -- 172.21.15.49:0/2304748331 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2b3d1d9d0 con 0x55c2b3df1d50 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa81b7fe640 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e6b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa82417c740 10 client.0 _ll_register_callbacks cb 0x7fa81409b210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:57.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa810060070 con 0x55c2b3df1d50 2024-03-18T00:59:57.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa81a7fc640 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3e001f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa814019000 tx=0x7fa81401af80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7093800805630c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:57.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa7feff5640 10 client.26885 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:59:57.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa7feff5640 1 -- 172.21.15.49:0/2304748331 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81409b5f0 con 0x55c2b3df1d50 2024-03-18T00:59:57.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.271+0000 7fa7feff5640 1 -- 172.21.15.49:0/2304748331 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa81409cd50 con 0x55c2b3df1d50 2024-03-18T00:59:57.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa7feff5640 1 -- 172.21.15.49:0/2304748331 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa814011b00 con 0x55c2b3df1d50 2024-03-18T00:59:57.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa82417c740 10 client.26885 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:57.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55c2b3e6c590 con 0x55c2b3df1d50 2024-03-18T00:59:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa7feff5640 1 -- 172.21.15.49:0/2304748331 <== mon.2 v2:172.21.15.106:3300/0 4 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fa8140a7040 con 0x55c2b3df1d50 2024-03-18T00:59:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa82417c740 10 client.26885 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa82417c740 10 client.26885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa82417c740 20 client.26885 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa82417c740 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c2b3e6cc60 0x55c2b3e6f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa82417c740 4 client.26885 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T00:59:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.272+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c2b3cdb540 con 0x55c2b3e6cc60 2024-03-18T00:59:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.273+0000 7fa81affd640 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c2b3e6cc60 0x55c2b3e6f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.275 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.273+0000 7fa7feff5640 10 client.26885 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.273+0000 7fa81affd640 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c2b3e6cc60 0x55c2b3e6f080 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26434395fb07f5e8 server_cookie=1e132909ff8a5b64 in_seq=0 out_seq=0 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa7feff5640 1 -- 172.21.15.49:0/2304748331 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c2b3cdb540 con 0x55c2b3e6cc60 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa7feff5640 10 client.26885 handle_command_reply: tid=0 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 client.26885 shutdown 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 msgr2=0x55c2b3e001f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3e001f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa814019000 tx=0x7fa81401af80 comp rx=0 tx=0).stop 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 shutdown_connections 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c2b3e6cc60 0x55c2b3e6f080 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa7feff5640 0 client.26885 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c2b3df1d50 0x55c2b3e001f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c2b3df6ec0 0x55c2b3e6b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 --2- 172.21.15.49:0/2304748331 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c2b3df1360 0x55c2b3dffcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 >> 172.21.15.49:0/2304748331 conn(0x55c2b3dec8d0 msgr2=0x55c2b3df03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 shutdown_connections 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 1 -- 172.21.15.49:0/2304748331 wait complete. 2024-03-18T00:59:57.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.274+0000 7fa82417c740 20 client.26885 trim_cache size 0 max 16384 2024-03-18T00:59:57.277 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T00:59:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T00:59:57.277 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T00:59:57.277 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 -- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa8040a2a00 msgr2=0x7fa8040a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa8040a2a00 0x7fa8040a4ec0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa8100047e0 tx=0x7fa810005fd0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 -- 172.21.15.49:0/2405034755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa81c108e20 msgr2=0x7fa81c19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa81c108e20 0x7fa81c19e440 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa81400ea80 tx=0x7fa81400ef40 comp rx=0 tx=0).stop 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 -- 172.21.15.49:0/2405034755 shutdown_connections 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa8040a2a00 0x7fa8040a4ec0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa81c1097e0 0x7fa81c1a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa81c108e20 0x7fa81c19e440 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 --2- 172.21.15.49:0/2405034755 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa81c072b50 0x7fa81c19df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 -- 172.21.15.49:0/2405034755 >> 172.21.15.49:0/2405034755 conn(0x7fa81c06e600 msgr2=0x7fa81c10c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 -- 172.21.15.49:0/2405034755 shutdown_connections 2024-03-18T00:59:57.278 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.276+0000 7fa8214b2640 1 -- 172.21.15.49:0/2405034755 wait complete. 2024-03-18T00:59:57.300 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T00:59:57.300 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T00:59:57.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.548+0000 7f113cbbc640 1 -- 172.21.15.49:0/207196516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113810db20 msgr2=0x7f113810dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.548+0000 7f113cbbc640 1 --2- 172.21.15.49:0/207196516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113810db20 0x7f113810dfd0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f1130009ca0 tx=0x7f113002c140 comp rx=0 tx=0).stop 2024-03-18T00:59:57.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.549+0000 7f113cbbc640 1 -- 172.21.15.49:0/207196516 shutdown_connections 2024-03-18T00:59:57.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.549+0000 7f113cbbc640 1 --2- 172.21.15.49:0/207196516 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f113810b150 0x7f113810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.549+0000 7f113cbbc640 1 --2- 172.21.15.49:0/207196516 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113810db20 0x7f113810dfd0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.549+0000 7f113cbbc640 1 --2- 172.21.15.49:0/207196516 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f11381087e0 0x7f113810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.549+0000 7f113cbbc640 1 -- 172.21.15.49:0/207196516 >> 172.21.15.49:0/207196516 conn(0x7f113806e650 msgr2=0x7f1138070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:57.551 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.551+0000 7f113cbbc640 1 -- 172.21.15.49:0/207196516 shutdown_connections 2024-03-18T00:59:57.553 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.553+0000 7f113cbbc640 1 -- 172.21.15.49:0/207196516 wait complete. 2024-03-18T00:59:57.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.555+0000 7f113cbbc640 1 Processor -- start 2024-03-18T00:59:57.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.558+0000 7f113cbbc640 1 -- start start 2024-03-18T00:59:57.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.559+0000 7f113cbbc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11381087e0 0x7f11381a6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.559+0000 7f113cbbc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f113810b150 0x7f11381a6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.559+0000 7f113cbbc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113810db20 0x7f11381adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.559+0000 7f113cbbc640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f113811c2f0 con 0x7f113810db20 2024-03-18T00:59:57.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.559+0000 7f113cbbc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f113811c170 con 0x7f11381087e0 2024-03-18T00:59:57.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.559+0000 7f113cbbc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f113811c470 con 0x7f113810b150 2024-03-18T00:59:57.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.560+0000 7f113659c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11381087e0 0x7f11381a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.562+0000 7f113659c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11381087e0 0x7f11381a6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:35316/0 (socket says 172.21.15.49:35316) 2024-03-18T00:59:57.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.562+0000 7f113659c640 1 -- 172.21.15.49:0/1971840572 learned_addr learned my addr 172.21.15.49:0/1971840572 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:57.562 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.562+0000 7f1135d9b640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f113810b150 0x7f11381a6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.563+0000 7f1136d9d640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113810db20 0x7f11381adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.565+0000 7f113659c640 1 -- 172.21.15.49:0/1971840572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f113810b150 msgr2=0x7f11381a6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.565+0000 7f113659c640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f113810b150 0x7f11381a6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.565+0000 7f113659c640 1 -- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113810db20 msgr2=0x7f11381adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:57.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.565+0000 7f113659c640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113810db20 0x7f11381adec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:57.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.565+0000 7f113659c640 1 -- 172.21.15.49:0/1971840572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11381ae530 con 0x7f11381087e0 2024-03-18T00:59:57.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.566+0000 7f1136d9d640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113810db20 0x7f11381adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:59:57.579 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.577+0000 7f113659c640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11381087e0 0x7f11381a6960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f112c00d770 tx=0x7f112c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:57.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.580+0000 7f11277fe640 1 -- 172.21.15.49:0/1971840572 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c017070 con 0x7f11381087e0 2024-03-18T00:59:57.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.580+0000 7f11277fe640 1 -- 172.21.15.49:0/1971840572 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f112c009c30 con 0x7f11381087e0 2024-03-18T00:59:57.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.580+0000 7f11277fe640 1 -- 172.21.15.49:0/1971840572 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c003da0 con 0x7f11381087e0 2024-03-18T00:59:57.581 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.581+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11381ae820 con 0x7f11381087e0 2024-03-18T00:59:57.582 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.581+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11381aeda0 con 0x7f11381087e0 2024-03-18T00:59:57.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.584+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11180052c0 con 0x7f11381087e0 2024-03-18T00:59:57.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.593+0000 7f11277fe640 1 -- 172.21.15.49:0/1971840572 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f112c003f40 con 0x7f11381087e0 2024-03-18T00:59:57.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.594+0000 7f11277fe640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11200a29d0 0x7f11200a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:57.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.595+0000 7f11277fe640 1 -- 172.21.15.49:0/1971840572 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f112c0ce750 con 0x7f11381087e0 2024-03-18T00:59:57.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.598+0000 7f11277fe640 1 -- 172.21.15.49:0/1971840572 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f112c09b870 con 0x7f11381087e0 2024-03-18T00:59:57.600 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.598+0000 7f1135d9b640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11200a29d0 0x7f11200a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:57.605 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:57.605+0000 7f1135d9b640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11200a29d0 0x7f11200a4e90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f11381a7e20 tx=0x7f1128008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.008 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T00:59:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.006+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11180055b0 con 0x7f11381087e0 2024-03-18T00:59:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.008+0000 7f11277fe640 1 -- 172.21.15.49:0/1971840572 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11935 (secure 0 0 0) 0x7f112c009de0 con 0x7f11381087e0 2024-03-18T00:59:58.009 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:59:55.206381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":178,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24506,24499,24481,24463,24470]},"id":1}]} 2024-03-18T00:59:58.009 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 30 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.013+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11200a29d0 msgr2=0x7f11200a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.013+0000 7f113cbbc640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11200a29d0 0x7f11200a4e90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f11381a7e20 tx=0x7f1128008040 comp rx=0 tx=0).stop 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.013+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11381087e0 msgr2=0x7f11381a6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.013+0000 7f113cbbc640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11381087e0 0x7f11381a6960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f112c00d770 tx=0x7f112c00dc30 comp rx=0 tx=0).stop 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.015+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 shutdown_connections 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.015+0000 7f113cbbc640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f11200a29d0 0x7f11200a4e90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.015+0000 7f113cbbc640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f113810db20 0x7f11381adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.015+0000 7f113cbbc640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f113810b150 0x7f11381a6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.015+0000 7f113cbbc640 1 --2- 172.21.15.49:0/1971840572 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f11381087e0 0x7f11381a6960 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.015+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 >> 172.21.15.49:0/1971840572 conn(0x7f113806e650 msgr2=0x7f1138108c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.021 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.020+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 shutdown_connections 2024-03-18T00:59:58.023 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.022+0000 7f113cbbc640 1 -- 172.21.15.49:0/1971840572 wait complete. 2024-03-18T00:59:58.062 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:57 smithi049 ceph-mon[32130]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 213 MiB/s rd, 346 MiB/s wr, 4.36k op/s 2024-03-18T00:59:58.064 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T00:59:58.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:57 smithi107 ceph-mon[36270]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 213 MiB/s rd, 346 MiB/s wr, 4.36k op/s 2024-03-18T00:59:58.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:57 smithi106 ceph-mon[35899]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 213 MiB/s rd, 346 MiB/s wr, 4.36k op/s 2024-03-18T00:59:58.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.294+0000 7ff538ed1640 1 -- 172.21.15.49:0/2186094564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534075820 msgr2=0x7ff534073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.294+0000 7ff538ed1640 1 --2- 172.21.15.49:0/2186094564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534075820 0x7ff534073c70 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff524009930 tx=0x7ff52402bc30 comp rx=0 tx=0).stop 2024-03-18T00:59:58.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.294+0000 7ff538ed1640 1 -- 172.21.15.49:0/2186094564 shutdown_connections 2024-03-18T00:59:58.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.294+0000 7ff538ed1640 1 --2- 172.21.15.49:0/2186094564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff534074240 0x7ff5340746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.294+0000 7ff538ed1640 1 --2- 172.21.15.49:0/2186094564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53410dde0 0x7ff5341101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.294+0000 7ff538ed1640 1 --2- 172.21.15.49:0/2186094564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534075820 0x7ff534073c70 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.294+0000 7ff538ed1640 1 -- 172.21.15.49:0/2186094564 >> 172.21.15.49:0/2186094564 conn(0x7ff5340fcd80 msgr2=0x7ff5340ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.295+0000 7ff538ed1640 1 -- 172.21.15.49:0/2186094564 shutdown_connections 2024-03-18T00:59:58.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.295+0000 7ff538ed1640 1 -- 172.21.15.49:0/2186094564 wait complete. 2024-03-18T00:59:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.295+0000 7ff538ed1640 1 Processor -- start 2024-03-18T00:59:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.295+0000 7ff538ed1640 1 -- start start 2024-03-18T00:59:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff538ed1640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534074240 0x7ff53419e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff538ed1640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff534075820 0x7ff53419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff538ed1640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53410dde0 0x7ff5341a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff538ed1640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff534113bb0 con 0x7ff534074240 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff538ed1640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff534113a30 con 0x7ff534075820 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff538ed1640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff534113d30 con 0x7ff53410dde0 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff53259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534074240 0x7ff53419e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff53259c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534074240 0x7ff53419e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40966/0 (socket says 172.21.15.49:40966) 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff53259c640 1 -- 172.21.15.49:0/98678707 learned_addr learned my addr 172.21.15.49:0/98678707 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff532d9d640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53410dde0 0x7ff5341a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.296+0000 7ff531d9b640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff534075820 0x7ff53419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.297+0000 7ff53259c640 1 -- 172.21.15.49:0/98678707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53410dde0 msgr2=0x7ff5341a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.297+0000 7ff53259c640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53410dde0 0x7ff5341a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.297+0000 7ff53259c640 1 -- 172.21.15.49:0/98678707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff534075820 msgr2=0x7ff53419e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.297+0000 7ff53259c640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff534075820 0x7ff53419e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.297+0000 7ff53259c640 1 -- 172.21.15.49:0/98678707 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5341a5e50 con 0x7ff534074240 2024-03-18T00:59:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.297+0000 7ff53259c640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534074240 0x7ff53419e230 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7ff524000c00 tx=0x7ff524004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.298+0000 7ff5237fe640 1 -- 172.21.15.49:0/98678707 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff524043020 con 0x7ff534074240 2024-03-18T00:59:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.298+0000 7ff5237fe640 1 -- 172.21.15.49:0/98678707 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff524004440 con 0x7ff534074240 2024-03-18T00:59:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.298+0000 7ff5237fe640 1 -- 172.21.15.49:0/98678707 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52403e450 con 0x7ff534074240 2024-03-18T00:59:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.298+0000 7ff538ed1640 1 -- 172.21.15.49:0/98678707 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5341a60e0 con 0x7ff534074240 2024-03-18T00:59:58.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.298+0000 7ff538ed1640 1 -- 172.21.15.49:0/98678707 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5341a65e0 con 0x7ff534074240 2024-03-18T00:59:58.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.301+0000 7ff53bb96740 1 Processor -- start 2024-03-18T00:59:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff53bb96740 1 -- start start 2024-03-18T00:59:58.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318af2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318af3240 0x55c318af7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff53bb96740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c318a57490 con 0x55c318af83b0 2024-03-18T00:59:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff53bb96740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c3189ffc90 con 0x55c318af2850 2024-03-18T00:59:58.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff53bb96740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c318a53530 con 0x55c318af3240 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff531d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff531d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afa840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40978/0 (socket says 172.21.15.49:40978) 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.302+0000 7ff531d9b640 1 -- 172.21.15.49:0/340569228 learned_addr learned my addr 172.21.15.49:0/340569228 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.303+0000 7ff531d9b640 1 -- 172.21.15.49:0/340569228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318af3240 msgr2=0x55c318af7d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.303+0000 7ff531d9b640 1 --2- 172.21.15.49:0/340569228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318af3240 0x55c318af7d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.303+0000 7ff531d9b640 1 -- 172.21.15.49:0/340569228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 msgr2=0x55c318af2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.303+0000 7ff531d9b640 1 --2- 172.21.15.49:0/340569228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318af2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.303+0000 7ff531d9b640 1 -- 172.21.15.49:0/340569228 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c318aba500 con 0x55c318af83b0 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.303+0000 7ff531d9b640 1 --2- 172.21.15.49:0/340569228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afa840 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff528009a40 tx=0x7ff528009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a16fb6e6da075 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.303+0000 7ff520ff9640 1 -- 172.21.15.49:0/340569228 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528017070 con 0x55c318af83b0 2024-03-18T00:59:58.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.303+0000 7ff520ff9640 1 -- 172.21.15.49:0/340569228 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff528013470 con 0x55c318af83b0 2024-03-18T00:59:58.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.303+0000 7ff520ff9640 1 -- 172.21.15.49:0/340569228 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528012630 con 0x55c318af83b0 2024-03-18T00:59:58.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.305+0000 7ff53bb96740 1 -- 172.21.15.49:0/340569228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 msgr2=0x55c318afa840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.305+0000 7ff53bb96740 1 --2- 172.21.15.49:0/340569228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afa840 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff528009a40 tx=0x7ff528009f00 comp rx=0 tx=0).stop 2024-03-18T00:59:58.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.305+0000 7ff53bb96740 1 -- 172.21.15.49:0/340569228 shutdown_connections 2024-03-18T00:59:58.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.305+0000 7ff53bb96740 1 --2- 172.21.15.49:0/340569228 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afa840 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.305+0000 7ff53bb96740 1 --2- 172.21.15.49:0/340569228 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318af3240 0x55c318af7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.305+0000 7ff53bb96740 1 --2- 172.21.15.49:0/340569228 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318af2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.305 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.305+0000 7ff53bb96740 1 -- 172.21.15.49:0/340569228 >> 172.21.15.49:0/340569228 conn(0x55c318aeddc0 msgr2=0x55c318aee1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.306+0000 7ff5237fe640 1 -- 172.21.15.49:0/98678707 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff524046020 con 0x7ff534074240 2024-03-18T00:59:58.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.306+0000 7ff53bb96740 1 -- 172.21.15.49:0/340569228 shutdown_connections 2024-03-18T00:59:58.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.306+0000 7ff53bb96740 1 -- 172.21.15.49:0/340569228 wait complete. 2024-03-18T00:59:58.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.306+0000 7ff53bb96740 1 -- start start 2024-03-18T00:59:58.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.307+0000 7ff53bb96740 1 Processor -- start 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.307+0000 7ff5237fe640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff51c0a2810 0x7ff51c0a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318afcd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff532d9d640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff51c0a2810 0x7ff51c0a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318b02250 0x55c318b04640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff53bb96740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c318afb360 con 0x55c318af83b0 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff531d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff531d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:40982/0 (socket says 172.21.15.49:40982) 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff531d9b640 1 -- 172.21.15.49:0/3691571209 learned_addr learned my addr 172.21.15.49:0/3691571209 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c318a13230 con 0x55c318af2850 2024-03-18T00:59:58.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c318afb4e0 con 0x55c318b02250 2024-03-18T00:59:58.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.308+0000 7ff53bb96740 10 client.0 _ll_register_callbacks cb 0x55c318a4f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:58.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.309+0000 7ff5237fe640 1 -- 172.21.15.49:0/98678707 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff5240f0e60 con 0x7ff534074240 2024-03-18T00:59:58.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.309+0000 7ff532d9d640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff51c0a2810 0x7ff51c0a4cd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x55c318af3aa0 tx=0x7ff52c0093f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.309+0000 7ff531d9b640 1 -- 172.21.15.49:0/3691571209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318b02250 msgr2=0x55c318b04640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.309+0000 7ff531d9b640 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318b02250 0x55c318b04640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.309+0000 7ff531d9b640 1 -- 172.21.15.49:0/3691571209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 msgr2=0x55c318afcd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.309+0000 7ff531d9b640 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318afcd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.309+0000 7ff531d9b640 1 -- 172.21.15.49:0/3691571209 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c318a22100 con 0x55c318af83b0 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.309+0000 7ff531d9b640 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afd2a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff528012630 tx=0x7ff528012660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7a2fb511bba75ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.309+0000 7ff519ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.310+0000 7ff519ffb640 1 -- 172.21.15.49:0/3691571209 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff528017050 con 0x55c318af83b0 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.310+0000 7ff519ffb640 1 -- 172.21.15.49:0/3691571209 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff528012880 con 0x55c318af83b0 2024-03-18T00:59:58.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.310+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff528013470 con 0x55c318af83b0 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.310+0000 7ff519ffb640 1 -- 172.21.15.49:0/3691571209 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff528006d60 con 0x55c318af83b0 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.310+0000 7ff53bb96740 10 client.17757 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.311+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c31876b3f0 con 0x55c318af83b0 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.311+0000 7ff519ffb640 1 -- 172.21.15.49:0/3691571209 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff528013470 con 0x55c318af83b0 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.311+0000 7ff519ffb640 1 -- 172.21.15.49:0/3691571209 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7ff52801ccc0 con 0x55c318af83b0 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.311+0000 7ff53bb96740 10 client.17757 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.311+0000 7ff53bb96740 10 client.17757 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.311+0000 7ff53bb96740 20 client.17757 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.311+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c318b08b50 0x55c318b0af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.311+0000 7ff53bb96740 4 client.17757 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.311+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c318a649d0 con 0x55c318b08b50 2024-03-18T00:59:58.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.312+0000 7ff53259c640 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c318b08b50 0x55c318b0af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.314+0000 7ff53259c640 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c318b08b50 0x55c318b0af40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8fd358d9399118cc server_cookie=1159d708fd02170d in_seq=0 out_seq=0 2024-03-18T00:59:58.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.314+0000 7ff519ffb640 10 client.17757 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:59:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.316+0000 7ff519ffb640 1 -- 172.21.15.49:0/3691571209 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c318a649d0 con 0x55c318b08b50 2024-03-18T00:59:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.317+0000 7ff519ffb640 10 client.17757 handle_command_reply: tid=0 2024-03-18T00:59:58.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.317+0000 7ff53bb96740 1 client.17757 shutdown 2024-03-18T00:59:58.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.318+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 msgr2=0x55c318afd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.318+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afd2a0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff528012630 tx=0x7ff528012660 comp rx=0 tx=0).stop 2024-03-18T00:59:58.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 shutdown_connections 2024-03-18T00:59:58.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c318b08b50 0x55c318b0af40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318b02250 0x55c318b04640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afd2a0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3691571209 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318afcd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff519ffb640 0 client.17757 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:59:58.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 >> 172.21.15.49:0/3691571209 conn(0x55c318aeddc0 msgr2=0x55c318aecfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 shutdown_connections 2024-03-18T00:59:58.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff53bb96740 1 -- 172.21.15.49:0/3691571209 wait complete. 2024-03-18T00:59:58.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.319+0000 7ff53bb96740 20 client.17757 trim_cache size 0 max 16384 2024-03-18T00:59:58.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.348+0000 7ff53bb96740 1 Processor -- start 2024-03-18T00:59:58.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.348+0000 7ff53bb96740 1 -- start start 2024-03-18T00:59:58.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.348+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318afb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.348+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.348+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318b02250 0x55c318af4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.349+0000 7ff53bb96740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff5341af4b0 con 0x55c318af83b0 2024-03-18T00:59:58.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.349+0000 7ff53bb96740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff5341b1020 con 0x55c318af2850 2024-03-18T00:59:58.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.349+0000 7ff53259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318afb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.349+0000 7ff532d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318b02250 0x55c318af4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.349+0000 7ff53259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318afb8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52446/0 (socket says 172.21.15.49:52446) 2024-03-18T00:59:58.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.349+0000 7ff53259c640 1 -- 172.21.15.49:0/3731275964 learned_addr learned my addr 172.21.15.49:0/3731275964 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.349+0000 7ff53bb96740 1 -- 172.21.15.49:0/3731275964 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff5341afb80 con 0x55c318b02250 2024-03-18T00:59:58.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.349+0000 7ff531d9b640 1 --2- 172.21.15.49:0/3731275964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff53259c640 1 -- 172.21.15.49:0/3731275964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318b02250 msgr2=0x55c318af4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff53259c640 1 --2- 172.21.15.49:0/3731275964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318b02250 0x55c318af4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff53259c640 1 -- 172.21.15.49:0/3731275964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 msgr2=0x55c318afbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff53259c640 1 --2- 172.21.15.49:0/3731275964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff53259c640 1 -- 172.21.15.49:0/3731275964 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3188357d0 con 0x55c318af2850 2024-03-18T00:59:58.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff531d9b640 1 --2- 172.21.15.49:0/3731275964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T00:59:58.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff53259c640 1 --2- 172.21.15.49:0/3731275964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318afb8b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff5240ba150 tx=0x7ff5240a8040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13b3c652b073e135 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff51bfff640 1 -- 172.21.15.49:0/3731275964 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5240bd0e0 con 0x55c318af2850 2024-03-18T00:59:58.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff51bfff640 1 -- 172.21.15.49:0/3731275964 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff524004c80 con 0x55c318af2850 2024-03-18T00:59:58.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.350+0000 7ff51bfff640 1 -- 172.21.15.49:0/3731275964 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5240d6840 con 0x55c318af2850 2024-03-18T00:59:58.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.351+0000 7ff53bb96740 1 -- 172.21.15.49:0/3731275964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 msgr2=0x55c318afb8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.351+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3731275964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318afb8b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff5240ba150 tx=0x7ff5240a8040 comp rx=0 tx=0).stop 2024-03-18T00:59:58.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.352+0000 7ff53bb96740 1 -- 172.21.15.49:0/3731275964 shutdown_connections 2024-03-18T00:59:58.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.352+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3731275964 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318b02250 0x55c318af4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.352+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3731275964 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318af83b0 0x55c318afbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.352+0000 7ff53bb96740 1 --2- 172.21.15.49:0/3731275964 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318afb8b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.352+0000 7ff53bb96740 1 -- 172.21.15.49:0/3731275964 >> 172.21.15.49:0/3731275964 conn(0x55c318aeddc0 msgr2=0x55c318afcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.352+0000 7ff53bb96740 1 -- 172.21.15.49:0/3731275964 shutdown_connections 2024-03-18T00:59:58.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.352+0000 7ff53bb96740 1 -- 172.21.15.49:0/3731275964 wait complete. 2024-03-18T00:59:58.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.352+0000 7ff53bb96740 1 -- start start 2024-03-18T00:59:58.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.353+0000 7ff53bb96740 1 Processor -- start 2024-03-18T00:59:58.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318b19400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318af83b0 0x55c318b19940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53bb96740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318b02250 0x55c318b1a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53bb96740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff5341b0920 con 0x55c318b02250 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53bb96740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff5341b0250 con 0x55c318af2850 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53bb96740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff5341ad490 con 0x55c318af83b0 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53bb96740 10 client.0 _ll_register_callbacks cb 0x7ff528009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff531d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318af83b0 0x55c318b19940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318b19400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53259c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318b19400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52448/0 (socket says 172.21.15.49:52448) 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.354+0000 7ff53259c640 1 -- 172.21.15.49:0/951703052 learned_addr learned my addr 172.21.15.49:0/951703052 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.355+0000 7ff532d9d640 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318b02250 0x55c318b1a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.355+0000 7ff53259c640 1 -- 172.21.15.49:0/951703052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318af83b0 msgr2=0x55c318b19940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.355+0000 7ff53259c640 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318af83b0 0x55c318b19940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.355+0000 7ff53259c640 1 -- 172.21.15.49:0/951703052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318b02250 msgr2=0x55c318b1a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.355+0000 7ff53259c640 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318b02250 0x55c318b1a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.355+0000 7ff53259c640 1 -- 172.21.15.49:0/951703052 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c318b1af40 con 0x55c318af2850 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.355+0000 7ff53259c640 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318b19400 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff5240bd480 tx=0x7ff5240045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79980050ee1b4f62 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.355+0000 7ff519ffb640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.355+0000 7ff519ffb640 1 -- 172.21.15.49:0/951703052 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5240bd0e0 con 0x55c318af2850 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.356+0000 7ff519ffb640 1 -- 172.21.15.49:0/951703052 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff5240a80b0 con 0x55c318af2850 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.356+0000 7ff53bb96740 1 -- 172.21.15.49:0/951703052 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff524004c80 con 0x55c318af2850 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.356+0000 7ff519ffb640 1 -- 172.21.15.49:0/951703052 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7ff5240bd780 con 0x55c318af2850 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.357+0000 7ff53bb96740 10 client.26897 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.357+0000 7ff53bb96740 1 -- 172.21.15.49:0/951703052 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55c318b1c2b0 con 0x55c318af2850 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.357+0000 7ff519ffb640 1 -- 172.21.15.49:0/951703052 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5240bde80 con 0x55c318af2850 2024-03-18T00:59:58.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.357+0000 7ff519ffb640 1 -- 172.21.15.49:0/951703052 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7ff5240b9040 con 0x55c318af2850 2024-03-18T00:59:58.358 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.359+0000 7ff53bb96740 10 client.26897 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:58.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.359+0000 7ff53bb96740 10 client.26897 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T00:59:58.359 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.360+0000 7ff53bb96740 20 client.26897 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:58.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.360+0000 7ff53bb96740 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c318b6ede0 0x55c318b71200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.360+0000 7ff53bb96740 4 client.26897 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:59:58.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.360+0000 7ff531d9b640 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c318b6ede0 0x55c318b71200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.360+0000 7ff53bb96740 1 -- 172.21.15.49:0/951703052 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ff524004c80 con 0x55c318b6ede0 2024-03-18T00:59:58.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.361+0000 7ff519ffb640 10 client.26897 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T00:59:58.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.361+0000 7ff531d9b640 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c318b6ede0 0x55c318b71200 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3915ef06b9a86ae1 server_cookie=26887032f4db3291 in_seq=0 out_seq=0 2024-03-18T00:59:58.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.361+0000 7ff519ffb640 1 -- 172.21.15.49:0/951703052 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ff524004c80 con 0x55c318b6ede0 2024-03-18T00:59:58.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.361+0000 7ff519ffb640 10 client.26897 handle_command_reply: tid=0 2024-03-18T00:59:58.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.361+0000 7ff53bb96740 1 client.26897 shutdown 2024-03-18T00:59:58.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.362+0000 7ff53bb96740 1 -- 172.21.15.49:0/951703052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 msgr2=0x55c318b19400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.362+0000 7ff53bb96740 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318b19400 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff5240bd480 tx=0x7ff5240045f0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.362+0000 7ff53bb96740 1 -- 172.21.15.49:0/951703052 shutdown_connections 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.362+0000 7ff53bb96740 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55c318b6ede0 0x55c318b71200 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.362+0000 7ff53bb96740 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c318b02250 0x55c318b1a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.362+0000 7ff53bb96740 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c318af83b0 0x55c318b19940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.362+0000 7ff53bb96740 1 --2- 172.21.15.49:0/951703052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c318af2850 0x55c318b19400 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.362+0000 7ff519ffb640 0 client.26897 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.362+0000 7ff53bb96740 1 -- 172.21.15.49:0/951703052 >> 172.21.15.49:0/951703052 conn(0x55c318aeddc0 msgr2=0x55c318af4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.363+0000 7ff53bb96740 1 -- 172.21.15.49:0/951703052 shutdown_connections 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.363+0000 7ff53bb96740 1 -- 172.21.15.49:0/951703052 wait complete. 2024-03-18T00:59:58.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.363+0000 7ff53bb96740 20 client.26897 trim_cache size 0 max 16384 2024-03-18T00:59:58.364 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:59:58.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.364+0000 7ff538ed1640 1 -- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff51c0a2810 msgr2=0x7ff51c0a4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.364+0000 7ff538ed1640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff51c0a2810 0x7ff51c0a4cd0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x55c318af3aa0 tx=0x7ff52c0093f0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.364+0000 7ff538ed1640 1 -- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534074240 msgr2=0x7ff53419e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.364+0000 7ff538ed1640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534074240 0x7ff53419e230 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7ff524000c00 tx=0x7ff524004040 comp rx=0 tx=0).stop 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.365+0000 7ff538ed1640 1 -- 172.21.15.49:0/98678707 shutdown_connections 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.365+0000 7ff538ed1640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff51c0a2810 0x7ff51c0a4cd0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.365+0000 7ff538ed1640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff534075820 0x7ff53419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.365+0000 7ff538ed1640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff53410dde0 0x7ff5341a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.365+0000 7ff538ed1640 1 --2- 172.21.15.49:0/98678707 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff534074240 0x7ff53419e230 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.365+0000 7ff538ed1640 1 -- 172.21.15.49:0/98678707 >> 172.21.15.49:0/98678707 conn(0x7ff5340fcd80 msgr2=0x7ff534101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.365+0000 7ff538ed1640 1 -- 172.21.15.49:0/98678707 shutdown_connections 2024-03-18T00:59:58.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.365+0000 7ff538ed1640 1 -- 172.21.15.49:0/98678707 wait complete. 2024-03-18T00:59:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.399+0000 7fabade1f640 1 -- 172.21.15.49:0/2390376172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faba81087e0 msgr2=0x7faba810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.403 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.399+0000 7fabade1f640 1 --2- 172.21.15.49:0/2390376172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faba81087e0 0x7faba810ac10 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7faba001acb0 tx=0x7faba003d170 comp rx=0 tx=0).stop 2024-03-18T00:59:58.406 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T00:59:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.406+0000 7fabade1f640 1 -- 172.21.15.49:0/2390376172 shutdown_connections 2024-03-18T00:59:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.406+0000 7fabade1f640 1 --2- 172.21.15.49:0/2390376172 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faba810b150 0x7faba810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.406+0000 7fabade1f640 1 --2- 172.21.15.49:0/2390376172 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba810db00 0x7faba810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.406+0000 7fabade1f640 1 --2- 172.21.15.49:0/2390376172 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faba81087e0 0x7faba810ac10 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.407 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.406+0000 7fabade1f640 1 -- 172.21.15.49:0/2390376172 >> 172.21.15.49:0/2390376172 conn(0x7faba806e820 msgr2=0x7faba8070c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.408+0000 7fabade1f640 1 -- 172.21.15.49:0/2390376172 shutdown_connections 2024-03-18T00:59:58.409 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.408+0000 7fabade1f640 1 -- 172.21.15.49:0/2390376172 wait complete. 2024-03-18T00:59:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.409+0000 7fabade1f640 1 Processor -- start 2024-03-18T00:59:58.410 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.410+0000 7fabade1f640 1 -- start start 2024-03-18T00:59:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.411+0000 7fabade1f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faba81087e0 0x7faba819def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.411+0000 7fabade1f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba810b150 0x7faba819e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.413 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.411+0000 7fabade1f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faba810db00 0x7faba81a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.411+0000 7fabade1f640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7faba8111ab0 con 0x7faba810db00 2024-03-18T00:59:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.411+0000 7fabade1f640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7faba8111930 con 0x7faba81087e0 2024-03-18T00:59:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.411+0000 7fabade1f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faba8111c30 con 0x7faba810b150 2024-03-18T00:59:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.415+0000 7faba6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba810b150 0x7faba819e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.415+0000 7faba6ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba810b150 0x7faba819e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59926/0 (socket says 172.21.15.49:59926) 2024-03-18T00:59:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.415+0000 7faba6ffd640 1 -- 172.21.15.49:0/1442844092 learned_addr learned my addr 172.21.15.49:0/1442844092 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.414 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.415+0000 7faba77fe640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faba81087e0 0x7faba819def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.415 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.415+0000 7faba7fff640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faba810db00 0x7faba81a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.416+0000 7faba77fe640 1 -- 172.21.15.49:0/1442844092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba810b150 msgr2=0x7faba819e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.416+0000 7faba77fe640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba810b150 0x7faba819e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.416+0000 7faba77fe640 1 -- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faba810db00 msgr2=0x7faba81a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.416+0000 7faba77fe640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faba810db00 0x7faba81a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.416+0000 7faba77fe640 1 -- 172.21.15.49:0/1442844092 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba81a5b10 con 0x7faba81087e0 2024-03-18T00:59:58.417 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.416+0000 7faba77fe640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faba81087e0 0x7faba819def0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faba001ac00 tx=0x7faba0045670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.419 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.418+0000 7faba4ff9640 1 -- 172.21.15.49:0/1442844092 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0052020 con 0x7faba81087e0 2024-03-18T00:59:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.418+0000 7faba4ff9640 1 -- 172.21.15.49:0/1442844092 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faba0045d40 con 0x7faba81087e0 2024-03-18T00:59:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.418+0000 7faba4ff9640 1 -- 172.21.15.49:0/1442844092 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba004a040 con 0x7faba81087e0 2024-03-18T00:59:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.420+0000 7fabade1f640 1 -- 172.21.15.49:0/1442844092 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba81a5da0 con 0x7faba81087e0 2024-03-18T00:59:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.420+0000 7fabade1f640 1 -- 172.21.15.49:0/1442844092 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba81a6150 con 0x7faba81087e0 2024-03-18T00:59:58.435 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.436+0000 7faba4ff9640 1 -- 172.21.15.49:0/1442844092 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7faba0055020 con 0x7faba81087e0 2024-03-18T00:59:58.435 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.436+0000 7fabb0a05740 1 Processor -- start 2024-03-18T00:59:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.436+0000 7fabb0a05740 1 -- start start 2024-03-18T00:59:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217ba4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217ba9830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba9ea0 0x565217bac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7fabb0a05740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x565217b09a40 con 0x565217ba4340 2024-03-18T00:59:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7fabb0a05740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x565217ab2920 con 0x565217ba4d30 2024-03-18T00:59:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7fabb0a05740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x565217b05ae0 con 0x565217ba9ea0 2024-03-18T00:59:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7faba7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217ba4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7faba7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217ba4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41030/0 (socket says 172.21.15.49:41030) 2024-03-18T00:59:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7faba7fff640 1 -- 172.21.15.49:0/2287686429 learned_addr learned my addr 172.21.15.49:0/2287686429 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.437+0000 7faba77fe640 1 --2- 172.21.15.49:0/2287686429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217ba9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.438+0000 7faba7fff640 1 -- 172.21.15.49:0/2287686429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba9ea0 msgr2=0x565217bac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.438+0000 7faba7fff640 1 --2- 172.21.15.49:0/2287686429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba9ea0 0x565217bac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.438+0000 7faba7fff640 1 -- 172.21.15.49:0/2287686429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 msgr2=0x565217ba9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.438+0000 7faba7fff640 1 --2- 172.21.15.49:0/2287686429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217ba9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.438+0000 7faba7fff640 1 -- 172.21.15.49:0/2287686429 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565217aca460 con 0x565217ba4340 2024-03-18T00:59:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.446+0000 7faba7fff640 1 --2- 172.21.15.49:0/2287686429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217ba4760 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00cb40 tx=0x7fab9c00e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d42fe05ed5aae29a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.450+0000 7fab8dffb640 1 -- 172.21.15.49:0/2287686429 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c004020 con 0x565217ba4340 2024-03-18T00:59:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.450+0000 7fab8dffb640 1 -- 172.21.15.49:0/2287686429 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9c0041c0 con 0x565217ba4340 2024-03-18T00:59:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.450+0000 7fab8dffb640 1 -- 172.21.15.49:0/2287686429 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c0107b0 con 0x565217ba4340 2024-03-18T00:59:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.451+0000 7faba4ff9640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fab900a2a60 0x7fab900a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.451+0000 7fabb0a05740 1 -- 172.21.15.49:0/2287686429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 msgr2=0x565217ba4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.451+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2287686429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217ba4760 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00cb40 tx=0x7fab9c00e350 comp rx=0 tx=0).stop 2024-03-18T00:59:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.451+0000 7fabb0a05740 1 -- 172.21.15.49:0/2287686429 shutdown_connections 2024-03-18T00:59:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.451+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2287686429 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217ba9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.451+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2287686429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba9ea0 0x565217bac330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.451+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2287686429 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217ba4760 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.451+0000 7fabb0a05740 1 -- 172.21.15.49:0/2287686429 >> 172.21.15.49:0/2287686429 conn(0x565217b9f8b0 msgr2=0x565217b9fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.451+0000 7faba4ff9640 1 -- 172.21.15.49:0/1442844092 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7faba0101300 con 0x7faba81087e0 2024-03-18T00:59:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.453+0000 7faba77fe640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fab900a2a60 0x7fab900a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.455+0000 7fabb0a05740 1 -- 172.21.15.49:0/2287686429 shutdown_connections 2024-03-18T00:59:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.457+0000 7fabb0a05740 1 -- 172.21.15.49:0/2287686429 wait complete. 2024-03-18T00:59:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.457+0000 7fabb0a05740 1 -- start start 2024-03-18T00:59:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.458+0000 7fabb0a05740 1 Processor -- start 2024-03-18T00:59:58.458 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.459+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217bae700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.459+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4d30 0x565217baec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.459+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 0x565217bb5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.459+0000 7fabb0a05740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x565217bace50 con 0x565217ba4340 2024-03-18T00:59:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.459+0000 7fabb0a05740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x565217ac5d70 con 0x565217ba9ea0 2024-03-18T00:59:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.459+0000 7fabb0a05740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x565217bacfd0 con 0x565217ba4d30 2024-03-18T00:59:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.460+0000 7faba77fe640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fab900a2a60 0x7fab900a4f20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7faba00ff150 tx=0x7faba003d170 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.460+0000 7fabb0a05740 10 client.0 _ll_register_callbacks cb 0x56521781da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.461+0000 7faba7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217bae700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.461+0000 7faba7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217bae700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41042/0 (socket says 172.21.15.49:41042) 2024-03-18T00:59:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.461+0000 7faba7fff640 1 -- 172.21.15.49:0/2670028092 learned_addr learned my addr 172.21.15.49:0/2670028092 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.461+0000 7faba6ffd640 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4d30 0x565217baec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.464+0000 7faba6ffd640 1 -- 172.21.15.49:0/2670028092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 msgr2=0x565217bb5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.464+0000 7faba6ffd640 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 0x565217bb5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.464+0000 7faba6ffd640 1 -- 172.21.15.49:0/2670028092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 msgr2=0x565217bae700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.464+0000 7faba6ffd640 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217bae700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.464+0000 7faba6ffd640 1 -- 172.21.15.49:0/2670028092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56521781e210 con 0x565217ba4d30 2024-03-18T00:59:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.467+0000 7faba6ffd640 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4d30 0x565217baec40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faba819f400 tx=0x7fab980107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=175b8f849ae8a369 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.468 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.467+0000 7fab86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.472+0000 7fab86ffd640 1 -- 172.21.15.49:0/2670028092 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab98004230 con 0x565217ba4d30 2024-03-18T00:59:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.472+0000 7fabb0a05740 1 -- 172.21.15.49:0/2670028092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab9c0041c0 con 0x565217ba4d30 2024-03-18T00:59:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.472+0000 7fab86ffd640 1 -- 172.21.15.49:0/2670028092 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fab980043d0 con 0x565217ba4d30 2024-03-18T00:59:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.473+0000 7fab86ffd640 1 -- 172.21.15.49:0/2670028092 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fab98021070 con 0x565217ba4d30 2024-03-18T00:59:58.478 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.475+0000 7fabb0a05740 10 client.26998 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:58.478 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.475+0000 7fabb0a05740 1 -- 172.21.15.49:0/2670028092 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5652178ea620 con 0x565217ba4d30 2024-03-18T00:59:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.481+0000 7fab86ffd640 1 -- 172.21.15.49:0/2670028092 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9c0041c0 con 0x565217ba4d30 2024-03-18T00:59:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.482+0000 7fab86ffd640 1 -- 172.21.15.49:0/2670028092 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fab98003970 con 0x565217ba4d30 2024-03-18T00:59:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.483+0000 7fabb0a05740 10 client.26998 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.483+0000 7fabb0a05740 10 client.26998 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.483+0000 7fabb0a05740 20 client.26998 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.483+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x565217bba450 0x565217bbc860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.483+0000 7fabb0a05740 4 client.26998 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:59:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.483+0000 7fabb0a05740 1 -- 172.21.15.49:0/2670028092 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565217b16f80 con 0x565217bba450 2024-03-18T00:59:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.486+0000 7faba7fff640 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x565217bba450 0x565217bbc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.503+0000 7faba7fff640 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x565217bba450 0x565217bbc860 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f28c80ac02783bc server_cookie=bb0bdeaa246269d7 in_seq=0 out_seq=0 2024-03-18T00:59:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.504+0000 7fab86ffd640 10 client.26998 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.507+0000 7fab86ffd640 1 -- 172.21.15.49:0/2670028092 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x565217b16f80 con 0x565217bba450 2024-03-18T00:59:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.507+0000 7fab86ffd640 10 client.26998 handle_command_reply: tid=0 2024-03-18T00:59:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.508+0000 7fabb0a05740 1 client.26998 shutdown 2024-03-18T00:59:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.515+0000 7fabb0a05740 1 -- 172.21.15.49:0/2670028092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4d30 msgr2=0x565217baec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.515+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4d30 0x565217baec40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faba819f400 tx=0x7fab980107a0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.521+0000 7fabb0a05740 1 -- 172.21.15.49:0/2670028092 shutdown_connections 2024-03-18T00:59:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.521+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x565217bba450 0x565217bbc860 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.521+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 0x565217bb5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.521+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4d30 0x565217baec40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.521+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2670028092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4340 0x565217bae700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.521+0000 7fabb0a05740 1 -- 172.21.15.49:0/2670028092 >> 172.21.15.49:0/2670028092 conn(0x565217b9f8b0 msgr2=0x565217b9ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.521 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.521+0000 7fab86ffd640 0 client.26998 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.524+0000 7fabb0a05740 1 -- 172.21.15.49:0/2670028092 shutdown_connections 2024-03-18T00:59:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.527+0000 7fabb0a05740 1 -- 172.21.15.49:0/2670028092 wait complete. 2024-03-18T00:59:58.530 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.530+0000 7fabb0a05740 20 client.26998 trim_cache size 0 max 16384 2024-03-18T00:59:58.587 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.587+0000 7fabb0a05740 1 Processor -- start 2024-03-18T00:59:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.589+0000 7fabb0a05740 1 -- start start 2024-03-18T00:59:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.590+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4340 0x565217b9e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.590+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4d30 0x565217bb5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.590+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 0x565217b9e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.590+0000 7fabb0a05740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7faba81af060 con 0x565217ba4d30 2024-03-18T00:59:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.590+0000 7fabb0a05740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7faba81b0bd0 con 0x565217ba9ea0 2024-03-18T00:59:58.590 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.590+0000 7fabb0a05740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faba81af730 con 0x565217ba4340 2024-03-18T00:59:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.591+0000 7faba7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 0x565217b9e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.591+0000 7faba7fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 0x565217b9e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52510/0 (socket says 172.21.15.49:52510) 2024-03-18T00:59:58.591 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.592+0000 7faba7fff640 1 -- 172.21.15.49:0/411925779 learned_addr learned my addr 172.21.15.49:0/411925779 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.592+0000 7faba7fff640 1 --2- 172.21.15.49:0/411925779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4340 0x565217b9e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.593+0000 7faba7fff640 1 -- 172.21.15.49:0/411925779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4340 msgr2=0x565217b9e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.593+0000 7faba7fff640 1 --2- 172.21.15.49:0/411925779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4340 0x565217b9e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.593+0000 7faba7fff640 1 -- 172.21.15.49:0/411925779 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4d30 msgr2=0x565217bb5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.593+0000 7faba7fff640 1 --2- 172.21.15.49:0/411925779 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4d30 0x565217bb5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.592 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.593+0000 7faba7fff640 1 -- 172.21.15.49:0/411925779 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565217b91140 con 0x565217ba9ea0 2024-03-18T00:59:58.593 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.594+0000 7faba7fff640 1 --2- 172.21.15.49:0/411925779 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 0x565217b9e860 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00cb40 tx=0x7fab9c005cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d32393463ca90ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.597+0000 7fab8d7fa640 1 -- 172.21.15.49:0/411925779 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c0043c0 con 0x565217ba9ea0 2024-03-18T00:59:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.597+0000 7fab8d7fa640 1 -- 172.21.15.49:0/411925779 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9c0035f0 con 0x565217ba9ea0 2024-03-18T00:59:58.596 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.597+0000 7fab8d7fa640 1 -- 172.21.15.49:0/411925779 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c0106d0 con 0x565217ba9ea0 2024-03-18T00:59:58.601 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.601+0000 7fabb0a05740 1 -- 172.21.15.49:0/411925779 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 msgr2=0x565217b9e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.601 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.601+0000 7fabb0a05740 1 --2- 172.21.15.49:0/411925779 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 0x565217b9e860 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00cb40 tx=0x7fab9c005cc0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.601 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.602+0000 7fabb0a05740 1 -- 172.21.15.49:0/411925779 shutdown_connections 2024-03-18T00:59:58.601 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.602+0000 7fabb0a05740 1 --2- 172.21.15.49:0/411925779 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba9ea0 0x565217b9e860 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.602+0000 7fabb0a05740 1 --2- 172.21.15.49:0/411925779 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba4d30 0x565217bb5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.602+0000 7fabb0a05740 1 --2- 172.21.15.49:0/411925779 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4340 0x565217b9e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.602+0000 7fabb0a05740 1 -- 172.21.15.49:0/411925779 >> 172.21.15.49:0/411925779 conn(0x565217b9f8b0 msgr2=0x565217ba60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.608+0000 7fabb0a05740 1 -- 172.21.15.49:0/411925779 shutdown_connections 2024-03-18T00:59:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.609+0000 7fabb0a05740 1 -- 172.21.15.49:0/411925779 wait complete. 2024-03-18T00:59:58.608 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.609+0000 7fabb0a05740 1 -- start start 2024-03-18T00:59:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.609+0000 7fabb0a05740 1 Processor -- start 2024-03-18T00:59:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4340 0x565217bb3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217bb39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7fabb0a05740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba9ea0 0x565217bb49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7fabb0a05740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7faba81b04d0 con 0x565217ba9ea0 2024-03-18T00:59:58.609 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7fabb0a05740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7faba81afe00 con 0x565217ba4d30 2024-03-18T00:59:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7fabb0a05740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7faba81ad0f0 con 0x565217ba4340 2024-03-18T00:59:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7fabb0a05740 10 client.0 _ll_register_callbacks cb 0x7fab9800baf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7faba6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217bb39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7faba6ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217bb39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52514/0 (socket says 172.21.15.49:52514) 2024-03-18T00:59:58.610 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.610+0000 7faba6ffd640 1 -- 172.21.15.49:0/2032990888 learned_addr learned my addr 172.21.15.49:0/2032990888 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.611 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.611+0000 7faba6ffd640 1 -- 172.21.15.49:0/2032990888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4340 msgr2=0x565217bb3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.611 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.611+0000 7faba6ffd640 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4340 0x565217bb3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.611 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.611+0000 7faba6ffd640 1 -- 172.21.15.49:0/2032990888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba9ea0 msgr2=0x565217bb49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.611 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.611+0000 7faba6ffd640 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba9ea0 0x565217bb49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.611 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.611+0000 7faba6ffd640 1 -- 172.21.15.49:0/2032990888 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565217bb4fc0 con 0x565217ba4d30 2024-03-18T00:59:58.612 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.613+0000 7faba6ffd640 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217bb39c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fab9800bfd0 tx=0x7fab9801b6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5faa97d3f33b32b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.612 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.613+0000 7fab86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:59:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.617+0000 7fab86ffd640 1 -- 172.21.15.49:0/2032990888 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab980041e0 con 0x565217ba4d30 2024-03-18T00:59:58.616 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.617+0000 7fab86ffd640 1 -- 172.21.15.49:0/2032990888 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fab98004380 con 0x565217ba4d30 2024-03-18T00:59:58.620 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.618+0000 7fabb0a05740 1 -- 172.21.15.49:0/2032990888 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab9c0035f0 con 0x565217ba4d30 2024-03-18T00:59:58.621 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.621+0000 7fab86ffd640 1 -- 172.21.15.49:0/2032990888 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fab980035f0 con 0x565217ba4d30 2024-03-18T00:59:58.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.625+0000 7fabb0a05740 10 client.26912 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:58.625 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.625+0000 7fabb0a05740 1 -- 172.21.15.49:0/2032990888 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565217c208f0 con 0x565217ba4d30 2024-03-18T00:59:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.626+0000 7fab86ffd640 1 -- 172.21.15.49:0/2032990888 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9c0035f0 con 0x565217ba4d30 2024-03-18T00:59:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.626+0000 7fab86ffd640 1 -- 172.21.15.49:0/2032990888 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fab98011650 con 0x565217ba4d30 2024-03-18T00:59:58.627 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.626+0000 7fabb0a05740 10 client.26912 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:58.628 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.626+0000 7fabb0a05740 10 client.26912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T00:59:58.628 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.627+0000 7fabb0a05740 20 client.26912 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:58.628 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.627+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x565217c21250 0x565217c23670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.628 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.627+0000 7fabb0a05740 4 client.26912 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T00:59:58.628 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.627+0000 7fabb0a05740 1 -- 172.21.15.49:0/2032990888 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x565217b112d0 con 0x565217c21250 2024-03-18T00:59:58.628 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.628+0000 7faba7fff640 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x565217c21250 0x565217c23670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.633+0000 7faba7fff640 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x565217c21250 0x565217c23670 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1aa0bd72f368f935 server_cookie=f2bfce74521b1650 in_seq=0 out_seq=0 2024-03-18T00:59:58.635 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.634+0000 7fab86ffd640 10 client.26912 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:58.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.764+0000 7fe222423640 1 -- 172.21.15.49:0/3587926555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 msgr2=0x7fe21c108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.764+0000 7fe222423640 1 --2- 172.21.15.49:0/3587926555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 0x7fe21c108120 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009860 tx=0x7fe20c02bb90 comp rx=0 tx=0).stop 2024-03-18T00:59:58.765 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:58 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1971840572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:59:58.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.774+0000 7fe222423640 1 -- 172.21.15.49:0/3587926555 shutdown_connections 2024-03-18T00:59:58.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.774+0000 7fe222423640 1 --2- 172.21.15.49:0/3587926555 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe21c108ef0 0x7fe21c109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.774+0000 7fe222423640 1 --2- 172.21.15.49:0/3587926555 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe21c1098b0 0x7fe21c110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.774+0000 7fe222423640 1 --2- 172.21.15.49:0/3587926555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 0x7fe21c108120 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.776 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.774+0000 7fe222423640 1 -- 172.21.15.49:0/3587926555 >> 172.21.15.49:0/3587926555 conn(0x7fe21c076cb0 msgr2=0x7fe21c0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.792 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.792+0000 7fe222423640 1 -- 172.21.15.49:0/3587926555 shutdown_connections 2024-03-18T00:59:58.797 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.797+0000 7fe222423640 1 -- 172.21.15.49:0/3587926555 wait complete. 2024-03-18T00:59:58.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.798+0000 7fe222423640 1 Processor -- start 2024-03-18T00:59:58.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe222423640 1 -- start start 2024-03-18T00:59:58.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe222423640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 0x7fe21c07a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe222423640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe21c108ef0 0x7fe21c07a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe222423640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe21c1098b0 0x7fe21c081380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.801 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe222423640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c06de50 con 0x7fe21c1098b0 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe222423640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c06dcd0 con 0x7fe21c108ef0 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe222423640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c06dfd0 con 0x7fe21c107d00 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe221421640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 0x7fe21c07a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe220c20640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe21c108ef0 0x7fe21c07a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe221421640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 0x7fe21c07a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59976/0 (socket says 172.21.15.49:59976) 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.800+0000 7fe221421640 1 -- 172.21.15.49:0/100582519 learned_addr learned my addr 172.21.15.49:0/100582519 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.801+0000 7fe221421640 1 -- 172.21.15.49:0/100582519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe21c108ef0 msgr2=0x7fe21c07a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.801+0000 7fe221421640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe21c108ef0 0x7fe21c07a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.801+0000 7fe221421640 1 -- 172.21.15.49:0/100582519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe21c1098b0 msgr2=0x7fe21c081380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.801+0000 7fe221421640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe21c1098b0 0x7fe21c081380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.801+0000 7fe221421640 1 -- 172.21.15.49:0/100582519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21c0819f0 con 0x7fe21c107d00 2024-03-18T00:59:58.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.801+0000 7fe221421640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 0x7fe21c07a190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe20c034a10 tx=0x7fe20c034930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.803 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.802+0000 7fe2127fc640 1 -- 172.21.15.49:0/100582519 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c039040 con 0x7fe21c107d00 2024-03-18T00:59:58.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.802+0000 7fe2127fc640 1 -- 172.21.15.49:0/100582519 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe20c042070 con 0x7fe21c107d00 2024-03-18T00:59:58.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.802+0000 7fe2127fc640 1 -- 172.21.15.49:0/100582519 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0046c0 con 0x7fe21c107d00 2024-03-18T00:59:58.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.802+0000 7fe222423640 1 -- 172.21.15.49:0/100582519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe21c081c80 con 0x7fe21c107d00 2024-03-18T00:59:58.804 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.802+0000 7fe222423640 1 -- 172.21.15.49:0/100582519 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe21c082170 con 0x7fe21c107d00 2024-03-18T00:59:58.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.808+0000 7fe22627a740 1 Processor -- start 2024-03-18T00:59:58.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.808+0000 7fe2127fc640 1 -- 172.21.15.49:0/100582519 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe20c004880 con 0x7fe21c107d00 2024-03-18T00:59:58.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.810+0000 7fe22627a740 1 -- start start 2024-03-18T00:59:58.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.810+0000 7fe22627a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91ba40 0x55cffd91be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.810+0000 7fe22627a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 0x55cffd920f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.810+0000 7fe22627a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd9215a0 0x55cffd923a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.810+0000 7fe22627a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55cffd880490 con 0x55cffd91c430 2024-03-18T00:59:58.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.810+0000 7fe22627a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55cffd828c90 con 0x55cffd9215a0 2024-03-18T00:59:58.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.810+0000 7fe22627a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cffd87c530 con 0x55cffd91ba40 2024-03-18T00:59:58.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.810+0000 7fe220c20640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd9215a0 0x55cffd923a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.810 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.810+0000 7fe220c20640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd9215a0 0x55cffd923a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52532/0 (socket says 172.21.15.49:52532) 2024-03-18T00:59:58.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.811+0000 7fe220c20640 1 -- 172.21.15.49:0/3614686365 learned_addr learned my addr 172.21.15.49:0/3614686365 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.811+0000 7fe220c20640 1 -- 172.21.15.49:0/3614686365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91ba40 msgr2=0x55cffd91be60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:59:58.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.811+0000 7fe220c20640 1 --2- 172.21.15.49:0/3614686365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91ba40 0x55cffd91be60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.811+0000 7fe220c20640 1 -- 172.21.15.49:0/3614686365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 msgr2=0x55cffd920f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.811+0000 7fe220c20640 1 --2- 172.21.15.49:0/3614686365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 0x55cffd920f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.811+0000 7fe220c20640 1 -- 172.21.15.49:0/3614686365 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cffd8e3500 con 0x55cffd9215a0 2024-03-18T00:59:58.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.811+0000 7fe220c20640 1 --2- 172.21.15.49:0/3614686365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd9215a0 0x55cffd923a30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe218009c90 tx=0x7fe218011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77285df699c20f39 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.811+0000 7fe2127fc640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2080a2a00 0x7fe2080a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.812 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.812+0000 7fe2127fc640 1 -- 172.21.15.49:0/100582519 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe20c0f01f0 con 0x7fe21c107d00 2024-03-18T00:59:58.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.814+0000 7fe2077fe640 1 -- 172.21.15.49:0/3614686365 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218005220 con 0x55cffd9215a0 2024-03-18T00:59:58.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.814+0000 7fe2077fe640 1 -- 172.21.15.49:0/3614686365 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe218004c80 con 0x55cffd9215a0 2024-03-18T00:59:58.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.814+0000 7fe2077fe640 1 -- 172.21.15.49:0/3614686365 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218019940 con 0x55cffd9215a0 2024-03-18T00:59:58.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.814+0000 7fe221c22640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2080a2a00 0x7fe2080a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.815+0000 7fe22627a740 1 -- 172.21.15.49:0/3614686365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd9215a0 msgr2=0x55cffd923a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.815+0000 7fe22627a740 1 --2- 172.21.15.49:0/3614686365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd9215a0 0x55cffd923a30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe218009c90 tx=0x7fe218011810 comp rx=0 tx=0).stop 2024-03-18T00:59:58.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.818+0000 7fe22627a740 1 -- 172.21.15.49:0/3614686365 shutdown_connections 2024-03-18T00:59:58.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.818+0000 7fe22627a740 1 --2- 172.21.15.49:0/3614686365 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd9215a0 0x55cffd923a30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.818+0000 7fe22627a740 1 --2- 172.21.15.49:0/3614686365 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 0x55cffd920f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.818+0000 7fe22627a740 1 --2- 172.21.15.49:0/3614686365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91ba40 0x55cffd91be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.818+0000 7fe22627a740 1 -- 172.21.15.49:0/3614686365 >> 172.21.15.49:0/3614686365 conn(0x55cffd916fb0 msgr2=0x55cffd9173b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.820+0000 7fe22627a740 1 -- 172.21.15.49:0/3614686365 shutdown_connections 2024-03-18T00:59:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.823+0000 7fe22627a740 1 -- 172.21.15.49:0/3614686365 wait complete. 2024-03-18T00:59:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.823+0000 7fe221c22640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2080a2a00 0x7fe2080a4ec0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x55cffd91cc90 tx=0x7fe21401a5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.823+0000 7fe22627a740 1 -- start start 2024-03-18T00:59:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.823+0000 7fe22627a740 1 Processor -- start 2024-03-18T00:59:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.824+0000 7fe22627a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 0x55cffd9260a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.824+0000 7fe22627a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91c430 0x55cffd9265e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.825 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.824+0000 7fe22627a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 0x55cffd92d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.824+0000 7fe22627a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55cffd924550 con 0x55cffd9215a0 2024-03-18T00:59:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.824+0000 7fe22627a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55cffd83c230 con 0x55cffd91ba40 2024-03-18T00:59:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.824+0000 7fe22627a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55cffd9246d0 con 0x55cffd91c430 2024-03-18T00:59:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.824+0000 7fe22627a740 10 client.0 _ll_register_callbacks cb 0x55cffd878010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.825+0000 7fe221421640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 0x55cffd9260a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.825+0000 7fe221421640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 0x55cffd9260a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52538/0 (socket says 172.21.15.49:52538) 2024-03-18T00:59:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.825+0000 7fe221421640 1 -- 172.21.15.49:0/998179592 learned_addr learned my addr 172.21.15.49:0/998179592 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.826+0000 7fe221c22640 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 0x55cffd92d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.826 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.826+0000 7fe220c20640 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91c430 0x55cffd9265e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.827+0000 7fe221c22640 1 -- 172.21.15.49:0/998179592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91c430 msgr2=0x55cffd9265e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.827+0000 7fe221c22640 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91c430 0x55cffd9265e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.827+0000 7fe221c22640 1 -- 172.21.15.49:0/998179592 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 msgr2=0x55cffd9260a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.827+0000 7fe221c22640 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 0x55cffd9260a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.827 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.827+0000 7fe221c22640 1 -- 172.21.15.49:0/998179592 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cffd84b100 con 0x55cffd9215a0 2024-03-18T00:59:58.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.828+0000 7fe221c22640 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 0x55cffd92d620 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe21402ad50 tx=0x7fe214004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ff72bd9f8ecb80 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.828+0000 7fe204ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T00:59:58.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.828+0000 7fe204ff9640 1 -- 172.21.15.49:0/998179592 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21401cd00 con 0x55cffd9215a0 2024-03-18T00:59:58.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.829+0000 7fe22627a740 1 -- 172.21.15.49:0/998179592 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe218004c80 con 0x55cffd9215a0 2024-03-18T00:59:58.828 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.829+0000 7fe204ff9640 1 -- 172.21.15.49:0/998179592 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe214036480 con 0x55cffd9215a0 2024-03-18T00:59:58.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.829+0000 7fe204ff9640 1 -- 172.21.15.49:0/998179592 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe21401cea0 con 0x55cffd9215a0 2024-03-18T00:59:58.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.830+0000 7fe22627a740 10 client.17781 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:58.829 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.830+0000 7fe22627a740 1 -- 172.21.15.49:0/998179592 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cffd5943f0 con 0x55cffd9215a0 2024-03-18T00:59:58.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.830+0000 7fe204ff9640 1 -- 172.21.15.49:0/998179592 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe218004c80 con 0x55cffd9215a0 2024-03-18T00:59:58.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.831+0000 7fe204ff9640 1 -- 172.21.15.49:0/998179592 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fe2140418a0 con 0x55cffd9215a0 2024-03-18T00:59:58.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.831+0000 7fe22627a740 10 client.17781 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:58.830 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.831+0000 7fe22627a740 10 client.17781 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:59:58.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.831+0000 7fe22627a740 20 client.17781 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:58.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.831+0000 7fe22627a740 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55cffd932440 0x55cffd934860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.831+0000 7fe22627a740 4 client.17781 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:59:58.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.831+0000 7fe22627a740 1 -- 172.21.15.49:0/998179592 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cffd88d9d0 con 0x55cffd932440 2024-03-18T00:59:58.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.831+0000 7fe221421640 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55cffd932440 0x55cffd934860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.831 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.832+0000 7fe204ff9640 10 client.17781 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:59:58.832 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.833+0000 7fe221421640 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55cffd932440 0x55cffd934860 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8ac0809f1d78e4d4 server_cookie=781c955b58b2c266 in_seq=0 out_seq=0 2024-03-18T00:59:58.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.835+0000 7fe204ff9640 1 -- 172.21.15.49:0/998179592 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cffd88d9d0 con 0x55cffd932440 2024-03-18T00:59:58.834 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.835+0000 7fe204ff9640 10 client.17781 handle_command_reply: tid=0 2024-03-18T00:59:58.835 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.835+0000 7fe22627a740 1 client.17781 shutdown 2024-03-18T00:59:58.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.837+0000 7fe22627a740 1 -- 172.21.15.49:0/998179592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 msgr2=0x55cffd92d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.837+0000 7fe22627a740 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 0x55cffd92d620 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fe21402ad50 tx=0x7fe214004a40 comp rx=0 tx=0).stop 2024-03-18T00:59:58.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.838+0000 7fe22627a740 1 -- 172.21.15.49:0/998179592 shutdown_connections 2024-03-18T00:59:58.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.838+0000 7fe22627a740 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55cffd932440 0x55cffd934860 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.837 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.838+0000 7fe22627a740 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 0x55cffd92d620 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.838+0000 7fe22627a740 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91c430 0x55cffd9265e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.838+0000 7fe22627a740 1 --2- 172.21.15.49:0/998179592 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 0x55cffd9260a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.838+0000 7fe22627a740 1 -- 172.21.15.49:0/998179592 >> 172.21.15.49:0/998179592 conn(0x55cffd916fb0 msgr2=0x55cffd916140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.839+0000 7fe204ff9640 0 client.17781 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:59:58.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.839+0000 7fe22627a740 1 -- 172.21.15.49:0/998179592 shutdown_connections 2024-03-18T00:59:58.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.840+0000 7fe22627a740 1 -- 172.21.15.49:0/998179592 wait complete. 2024-03-18T00:59:58.839 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.840+0000 7fe22627a740 20 client.17781 trim_cache size 0 max 16384 2024-03-18T00:59:58.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.869+0000 7fe22627a740 1 Processor -- start 2024-03-18T00:59:58.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.869+0000 7fe22627a740 1 -- start start 2024-03-18T00:59:58.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.870+0000 7fe22627a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91ba40 0x55cffd911fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.870+0000 7fe22627a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91c430 0x55cffd92d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.870+0000 7fe22627a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 0x55cffd924a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.870+0000 7fe220c20640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91c430 0x55cffd92d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.870+0000 7fe220c20640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91c430 0x55cffd92d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52548/0 (socket says 172.21.15.49:52548) 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.870+0000 7fe220c20640 1 -- 172.21.15.49:0/4198010225 learned_addr learned my addr 172.21.15.49:0/4198010225 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.870+0000 7fe22627a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c1af260 con 0x55cffd9215a0 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.870+0000 7fe22627a740 1 -- 172.21.15.49:0/4198010225 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c1b0dd0 con 0x55cffd91c430 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.870+0000 7fe22627a740 1 -- 172.21.15.49:0/4198010225 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c1af930 con 0x55cffd91ba40 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.871+0000 7fe221421640 1 --2- 172.21.15.49:0/4198010225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91ba40 0x55cffd911fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.871+0000 7fe220c20640 1 -- 172.21.15.49:0/4198010225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91ba40 msgr2=0x55cffd911fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.871+0000 7fe220c20640 1 --2- 172.21.15.49:0/4198010225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91ba40 0x55cffd911fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.871+0000 7fe220c20640 1 -- 172.21.15.49:0/4198010225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 msgr2=0x55cffd924a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.871+0000 7fe220c20640 1 --2- 172.21.15.49:0/4198010225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 0x55cffd924a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.871 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.871+0000 7fe220c20640 1 -- 172.21.15.49:0/4198010225 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cffd65e7d0 con 0x55cffd91c430 2024-03-18T00:59:58.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.871+0000 7fe220c20640 1 --2- 172.21.15.49:0/4198010225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91c430 0x55cffd92d5e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe21801b040 tx=0x7fe218009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abfc54b1d64a4f0f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.872+0000 7fe206ffd640 1 -- 172.21.15.49:0/4198010225 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218029020 con 0x55cffd91c430 2024-03-18T00:59:58.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.872+0000 7fe206ffd640 1 -- 172.21.15.49:0/4198010225 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe218004680 con 0x55cffd91c430 2024-03-18T00:59:58.872 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.872+0000 7fe206ffd640 1 -- 172.21.15.49:0/4198010225 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218019670 con 0x55cffd91c430 2024-03-18T00:59:58.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.873+0000 7fe22627a740 1 -- 172.21.15.49:0/4198010225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91c430 msgr2=0x55cffd92d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.873+0000 7fe22627a740 1 --2- 172.21.15.49:0/4198010225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91c430 0x55cffd92d5e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe21801b040 tx=0x7fe218009c00 comp rx=0 tx=0).stop 2024-03-18T00:59:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.876+0000 7fe22627a740 1 -- 172.21.15.49:0/4198010225 shutdown_connections 2024-03-18T00:59:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.876+0000 7fe22627a740 1 --2- 172.21.15.49:0/4198010225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91c430 0x55cffd92d5e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.876+0000 7fe22627a740 1 --2- 172.21.15.49:0/4198010225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd9215a0 0x55cffd924a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.876+0000 7fe22627a740 1 --2- 172.21.15.49:0/4198010225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd91ba40 0x55cffd911fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.876+0000 7fe22627a740 1 -- 172.21.15.49:0/4198010225 >> 172.21.15.49:0/4198010225 conn(0x55cffd916fb0 msgr2=0x55cffd91da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.877+0000 7fe22627a740 1 -- 172.21.15.49:0/4198010225 shutdown_connections 2024-03-18T00:59:58.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.877+0000 7fe22627a740 1 -- 172.21.15.49:0/4198010225 wait complete. 2024-03-18T00:59:58.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.877+0000 7fe22627a740 1 -- start start 2024-03-18T00:59:58.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.877+0000 7fe22627a740 1 Processor -- start 2024-03-18T00:59:58.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe22627a740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 0x55cffd92a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe22627a740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 0x55cffd92a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe22627a740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd9215a0 0x55cffd996790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe22627a740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c1b06d0 con 0x55cffd91c430 2024-03-18T00:59:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe22627a740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c1b0000 con 0x55cffd91ba40 2024-03-18T00:59:58.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe22627a740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c1ad240 con 0x55cffd9215a0 2024-03-18T00:59:58.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe22627a740 10 client.0 _ll_register_callbacks cb 0x7fe21402aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:58.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe221c22640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd9215a0 0x55cffd996790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe221c22640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd9215a0 0x55cffd996790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60008/0 (socket says 172.21.15.49:60008) 2024-03-18T00:59:58.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe221c22640 1 -- 172.21.15.49:0/1324455487 learned_addr learned my addr 172.21.15.49:0/1324455487 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe221421640 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 0x55cffd92a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.878+0000 7fe220c20640 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 0x55cffd92a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fe221c22640 1 -- 172.21.15.49:0/1324455487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 msgr2=0x55cffd92a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fe221c22640 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 0x55cffd92a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fe221c22640 1 -- 172.21.15.49:0/1324455487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 msgr2=0x55cffd92a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fe221c22640 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 0x55cffd92a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fe221c22640 1 -- 172.21.15.49:0/1324455487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cffd996cd0 con 0x55cffd9215a0 2024-03-18T00:59:58.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fe220c20640 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 0x55cffd92a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:59:58.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fe221c22640 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd9215a0 0x55cffd996790 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe214041cc0 tx=0x7fe21401abd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35a0ac6cb8ae58d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:58.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fe204ff9640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:58.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fab86ffd640 1 -- 172.21.15.49:0/2032990888 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x565217b112d0 con 0x565217c21250 2024-03-18T00:59:58.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fab86ffd640 10 client.26912 handle_command_reply: tid=0 2024-03-18T00:59:58.883 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.879+0000 7fabb0a05740 1 client.26912 shutdown 2024-03-18T00:59:58.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.880+0000 7fe204ff9640 1 -- 172.21.15.49:0/1324455487 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe214004e50 con 0x55cffd9215a0 2024-03-18T00:59:58.883 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.880+0000 7fe204ff9640 1 -- 172.21.15.49:0/1324455487 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe214041420 con 0x55cffd9215a0 2024-03-18T00:59:58.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fe22627a740 1 -- 172.21.15.49:0/1324455487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe218004680 con 0x55cffd9215a0 2024-03-18T00:59:58.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fabb0a05740 1 -- 172.21.15.49:0/2032990888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 msgr2=0x565217bb39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217bb39c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fab9800bfd0 tx=0x7fab9801b6a0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fabb0a05740 1 -- 172.21.15.49:0/2032990888 shutdown_connections 2024-03-18T00:59:58.884 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x565217c21250 0x565217c23670 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x565217ba4d30 0x565217bb39c0 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fab9800bfd0 tx=0x7fab9801b6a0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x565217ba9ea0 0x565217bb49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fabb0a05740 1 --2- 172.21.15.49:0/2032990888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x565217ba4340 0x565217bb3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fabb0a05740 1 -- 172.21.15.49:0/2032990888 >> 172.21.15.49:0/2032990888 conn(0x565217b9f8b0 msgr2=0x565217ba2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.881+0000 7fab86ffd640 0 client.26912 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T00:59:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.883+0000 7fabb0a05740 1 -- 172.21.15.49:0/2032990888 shutdown_connections 2024-03-18T00:59:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.883+0000 7fabb0a05740 1 -- 172.21.15.49:0/2032990888 wait complete. 2024-03-18T00:59:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.884+0000 7fabb0a05740 20 client.26912 trim_cache size 0 max 16384 2024-03-18T00:59:58.885 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.884+0000 7fe204ff9640 1 -- 172.21.15.49:0/1324455487 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe214041660 con 0x55cffd9215a0 2024-03-18T00:59:58.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.885+0000 7fe22627a740 10 client.27013 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:58.886 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.885+0000 7fe22627a740 1 -- 172.21.15.49:0/1324455487 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cffd997e70 con 0x55cffd9215a0 2024-03-18T00:59:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T00:59:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T00:59:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T00:59:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T00:59:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fabade1f640 1 -- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fab900a2a60 msgr2=0x7fab900a4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fabade1f640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fab900a2a60 0x7fab900a4f20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7faba00ff150 tx=0x7faba003d170 comp rx=0 tx=0).stop 2024-03-18T00:59:58.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fabade1f640 1 -- 172.21.15.49:0/1442844092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faba81087e0 msgr2=0x7faba819def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.888 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fabade1f640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faba81087e0 0x7faba819def0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faba001ac00 tx=0x7faba0045670 comp rx=0 tx=0).stop 2024-03-18T00:59:58.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fe204ff9640 1 -- 172.21.15.49:0/1324455487 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe218004680 con 0x55cffd9215a0 2024-03-18T00:59:58.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fe204ff9640 1 -- 172.21.15.49:0/1324455487 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fe214036480 con 0x55cffd9215a0 2024-03-18T00:59:58.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fe22627a740 10 client.27013 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:58.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fe22627a740 10 client.27013 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T00:59:58.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fe22627a740 20 client.27013 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:58.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fe22627a740 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55cffd998800 0x55cffd99ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:58.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fe22627a740 4 client.27013 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:59:58.889 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.887+0000 7fe22627a740 1 -- 172.21.15.49:0/1324455487 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cffd887d20 con 0x55cffd998800 2024-03-18T00:59:58.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.888+0000 7fabade1f640 1 -- 172.21.15.49:0/1442844092 shutdown_connections 2024-03-18T00:59:58.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.888+0000 7fabade1f640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fab900a2a60 0x7fab900a4f20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.888+0000 7fabade1f640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7faba810db00 0x7faba81a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.889 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.888+0000 7fabade1f640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faba810b150 0x7faba819e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.888+0000 7fe221421640 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55cffd998800 0x55cffd99ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:58.890 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.888+0000 7fabade1f640 1 --2- 172.21.15.49:0/1442844092 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7faba81087e0 0x7faba819def0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.890 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.888+0000 7fabade1f640 1 -- 172.21.15.49:0/1442844092 >> 172.21.15.49:0/1442844092 conn(0x7faba806e820 msgr2=0x7faba8108c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.889+0000 7fe221421640 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55cffd998800 0x55cffd99ac20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f613755c96b210c9 server_cookie=65898fc1be5542d4 in_seq=0 out_seq=0 2024-03-18T00:59:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.889+0000 7fe204ff9640 10 client.27013 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T00:59:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.889+0000 7fe204ff9640 1 -- 172.21.15.49:0/1324455487 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cffd887d20 con 0x55cffd998800 2024-03-18T00:59:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.889+0000 7fe204ff9640 10 client.27013 handle_command_reply: tid=0 2024-03-18T00:59:58.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.890+0000 7fe22627a740 1 client.27013 shutdown 2024-03-18T00:59:58.891 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.890+0000 7fabade1f640 1 -- 172.21.15.49:0/1442844092 shutdown_connections 2024-03-18T00:59:58.892 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T00:59:58.890+0000 7fabade1f640 1 -- 172.21.15.49:0/1442844092 wait complete. 2024-03-18T00:59:58.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 -- 172.21.15.49:0/1324455487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd9215a0 msgr2=0x55cffd996790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd9215a0 0x55cffd996790 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe214041cc0 tx=0x7fe21401abd0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 -- 172.21.15.49:0/1324455487 shutdown_connections 2024-03-18T00:59:58.892 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55cffd998800 0x55cffd99ac20 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55cffd9215a0 0x55cffd996790 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55cffd91c430 0x55cffd92a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 --2- 172.21.15.49:0/1324455487 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55cffd91ba40 0x55cffd92a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe204ff9640 0 client.27013 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T00:59:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 -- 172.21.15.49:0/1324455487 >> 172.21.15.49:0/1324455487 conn(0x55cffd916fb0 msgr2=0x55cffd924a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 -- 172.21.15.49:0/1324455487 shutdown_connections 2024-03-18T00:59:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 1 -- 172.21.15.49:0/1324455487 wait complete. 2024-03-18T00:59:58.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.891+0000 7fe22627a740 20 client.27013 trim_cache size 0 max 16384 2024-03-18T00:59:58.893 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:59:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 -- 172.21.15.49:0/100582519 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2080a2a00 msgr2=0x7fe2080a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2080a2a00 0x7fe2080a4ec0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x55cffd91cc90 tx=0x7fe21401a5d0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 -- 172.21.15.49:0/100582519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 msgr2=0x7fe21c07a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 0x7fe21c07a190 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe20c034a10 tx=0x7fe20c034930 comp rx=0 tx=0).stop 2024-03-18T00:59:58.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 -- 172.21.15.49:0/100582519 shutdown_connections 2024-03-18T00:59:58.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fe2080a2a00 0x7fe2080a4ec0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fe21c108ef0 0x7fe21c07a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fe21c1098b0 0x7fe21c081380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 --2- 172.21.15.49:0/100582519 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe21c107d00 0x7fe21c07a190 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:58.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 -- 172.21.15.49:0/100582519 >> 172.21.15.49:0/100582519 conn(0x7fe21c076cb0 msgr2=0x7fe21c0709b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:58.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 -- 172.21.15.49:0/100582519 shutdown_connections 2024-03-18T00:59:58.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:58.892+0000 7fe222423640 1 -- 172.21.15.49:0/100582519 wait complete. 2024-03-18T00:59:58.939 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T00:59:59.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:58 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1971840572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:59:59.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:58 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1971840572' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T00:59:59.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.312+0000 7f0625c63640 1 -- 172.21.15.49:0/1525162719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06201006e0 msgr2=0x7f0620100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.312+0000 7f0625c63640 1 --2- 172.21.15.49:0/1525162719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06201006e0 0x7f0620100b00 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f0610009860 tx=0x7f061002bb90 comp rx=0 tx=0).stop 2024-03-18T00:59:59.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.312+0000 7f0625c63640 1 -- 172.21.15.49:0/1525162719 shutdown_connections 2024-03-18T00:59:59.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.312+0000 7f0625c63640 1 --2- 172.21.15.49:0/1525162719 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06201018d0 0x7f0620101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.312+0000 7f0625c63640 1 --2- 172.21.15.49:0/1525162719 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0620102290 0x7f0620108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.312+0000 7f0625c63640 1 --2- 172.21.15.49:0/1525162719 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f06201006e0 0x7f0620100b00 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.312+0000 7f0625c63640 1 -- 172.21.15.49:0/1525162719 >> 172.21.15.49:0/1525162719 conn(0x7f06200fbc70 msgr2=0x7f06200fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.313+0000 7f0625c63640 1 -- 172.21.15.49:0/1525162719 shutdown_connections 2024-03-18T00:59:59.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.313+0000 7f0625c63640 1 -- 172.21.15.49:0/1525162719 wait complete. 2024-03-18T00:59:59.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f0625c63640 1 Processor -- start 2024-03-18T00:59:59.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f0625c63640 1 -- start start 2024-03-18T00:59:59.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f0625c63640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06201006e0 0x7f06200694d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f0625c63640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06201018d0 0x7f062019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f0625c63640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0620102290 0x7f062019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f0625c63640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f062010c360 con 0x7f0620102290 2024-03-18T00:59:59.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f0625c63640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f062010c1e0 con 0x7f06201018d0 2024-03-18T00:59:59.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f0625c63640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f062010c4e0 con 0x7f06201006e0 2024-03-18T00:59:59.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f061f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06201006e0 0x7f06200694d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.314+0000 7f061f7fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06201006e0 0x7f06200694d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60032/0 (socket says 172.21.15.49:60032) 2024-03-18T00:59:59.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061f7fe640 1 -- 172.21.15.49:0/1452325742 learned_addr learned my addr 172.21.15.49:0/1452325742 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061ffff640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0620102290 0x7f062019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061effd640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06201018d0 0x7f062019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061f7fe640 1 -- 172.21.15.49:0/1452325742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06201018d0 msgr2=0x7f062019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061f7fe640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06201018d0 0x7f062019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061f7fe640 1 -- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0620102290 msgr2=0x7f062019ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061f7fe640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0620102290 0x7f062019ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061f7fe640 1 -- 172.21.15.49:0/1452325742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062019f750 con 0x7f06201006e0 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061effd640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06201018d0 0x7f062019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061f7fe640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06201006e0 0x7f06200694d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f06100349e0 tx=0x7f0610032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f061cff9640 1 -- 172.21.15.49:0/1452325742 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610041020 con 0x7f06201006e0 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.315+0000 7f0625c63640 1 -- 172.21.15.49:0/1452325742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062019f9e0 con 0x7f06201006e0 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.316+0000 7f0625c63640 1 -- 172.21.15.49:0/1452325742 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f062019ff20 con 0x7f06201006e0 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.316+0000 7f061cff9640 1 -- 172.21.15.49:0/1452325742 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f061002ca10 con 0x7f06201006e0 2024-03-18T00:59:59.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.316+0000 7f061cff9640 1 -- 172.21.15.49:0/1452325742 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0610039040 con 0x7f06201006e0 2024-03-18T00:59:59.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.321+0000 7f061cff9640 1 -- 172.21.15.49:0/1452325742 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0610044020 con 0x7f06201006e0 2024-03-18T00:59:59.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.321+0000 7f0628849740 1 Processor -- start 2024-03-18T00:59:59.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.321+0000 7f0628849740 1 -- start start 2024-03-18T00:59:59.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.322+0000 7f0628849740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d12c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.322+0000 7f0628849740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 0x55deb8d17d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.322+0000 7f0628849740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8d1a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.322+0000 7f0628849740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55deb8c77490 con 0x55deb8d13240 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.322+0000 7f0628849740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55deb8c1fc90 con 0x55deb8d12850 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.322+0000 7f0628849740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55deb8c73530 con 0x55deb8d183b0 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.322+0000 7f061ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8d1a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.322+0000 7f061effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 0x55deb8d17d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.323+0000 7f061effd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 0x55deb8d17d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41164/0 (socket says 172.21.15.49:41164) 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.323+0000 7f061effd640 1 -- 172.21.15.49:0/1065974796 learned_addr learned my addr 172.21.15.49:0/1065974796 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.323+0000 7f061cff9640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06080a2880 0x7f06080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.323+0000 7f061ffff640 1 --2- 172.21.15.49:0/1065974796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d12c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.323+0000 7f061f7fe640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06080a2880 0x7f06080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.324+0000 7f061ffff640 1 -- 172.21.15.49:0/1065974796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 msgr2=0x55deb8d1a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.324+0000 7f061ffff640 1 --2- 172.21.15.49:0/1065974796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8d1a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.324+0000 7f061ffff640 1 -- 172.21.15.49:0/1065974796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 msgr2=0x55deb8d17d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.324+0000 7f061ffff640 1 --2- 172.21.15.49:0/1065974796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 0x55deb8d17d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.324+0000 7f061ffff640 1 -- 172.21.15.49:0/1065974796 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deb8cda500 con 0x55deb8d12850 2024-03-18T00:59:59.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.324+0000 7f061ffff640 1 --2- 172.21.15.49:0/1065974796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d12c70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0618009270 tx=0x7f0618009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aa071a5ac8ae414 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.324+0000 7f0605ffb640 1 -- 172.21.15.49:0/1065974796 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618012250 con 0x55deb8d12850 2024-03-18T00:59:59.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.324+0000 7f0605ffb640 1 -- 172.21.15.49:0/1065974796 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06180126e0 con 0x55deb8d12850 2024-03-18T00:59:59.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.324+0000 7f0605ffb640 1 -- 172.21.15.49:0/1065974796 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618004060 con 0x55deb8d12850 2024-03-18T00:59:59.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.325+0000 7f061cff9640 1 -- 172.21.15.49:0/1452325742 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f06100f0400 con 0x7f06201006e0 2024-03-18T00:59:59.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.325+0000 7f0628849740 1 -- 172.21.15.49:0/1065974796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 msgr2=0x55deb8d12c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.325+0000 7f061f7fe640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06080a2880 0x7f06080a4d40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f06100b8a60 tx=0x7f06100b89f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.325+0000 7f0628849740 1 --2- 172.21.15.49:0/1065974796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d12c70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0618009270 tx=0x7f0618009500 comp rx=0 tx=0).stop 2024-03-18T00:59:59.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.326+0000 7f0628849740 1 -- 172.21.15.49:0/1065974796 shutdown_connections 2024-03-18T00:59:59.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.326+0000 7f0628849740 1 --2- 172.21.15.49:0/1065974796 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8d1a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.326 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.326+0000 7f0628849740 1 --2- 172.21.15.49:0/1065974796 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 0x55deb8d17d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.326+0000 7f0628849740 1 --2- 172.21.15.49:0/1065974796 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d12c70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.326+0000 7f0628849740 1 -- 172.21.15.49:0/1065974796 >> 172.21.15.49:0/1065974796 conn(0x55deb8d0ddc0 msgr2=0x55deb8d0e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.326+0000 7f0628849740 1 -- 172.21.15.49:0/1065974796 shutdown_connections 2024-03-18T00:59:59.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.326+0000 7f0628849740 1 -- 172.21.15.49:0/1065974796 wait complete. 2024-03-18T00:59:59.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.326+0000 7f0628849740 1 -- start start 2024-03-18T00:59:59.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.327+0000 7f0628849740 1 Processor -- start 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.327+0000 7f0628849740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d1cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.327+0000 7f0628849740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d13240 0x55deb8d1d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f0628849740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 0x55deb8d24280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f0628849740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55deb8d1b2a0 con 0x55deb8d183b0 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d13240 0x55deb8d1d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061effd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d13240 0x55deb8d1d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60050/0 (socket says 172.21.15.49:60050) 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061effd640 1 -- 172.21.15.49:0/3494625041 learned_addr learned my addr 172.21.15.49:0/3494625041 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061ffff640 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d1cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55deb8c33230 con 0x55deb8d12850 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55deb8d1b420 con 0x55deb8d13240 2024-03-18T00:59:59.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f0628849740 10 client.0 _ll_register_callbacks cb 0x55deb8c6f010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061ffff640 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 0x55deb8d24280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061effd640 1 -- 172.21.15.49:0/3494625041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 msgr2=0x55deb8d1cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061effd640 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d1cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061effd640 1 -- 172.21.15.49:0/3494625041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 msgr2=0x55deb8d24280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061effd640 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 0x55deb8d24280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.328+0000 7f061effd640 1 -- 172.21.15.49:0/3494625041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deb8c42100 con 0x55deb8d13240 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.329+0000 7f061effd640 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d13240 0x55deb8d1d280 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0614012730 tx=0x7f0614012bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19dbe40c7f42ad4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.329+0000 7f05feffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.329+0000 7f05feffd640 1 -- 172.21.15.49:0/3494625041 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614004240 con 0x55deb8d13240 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.329+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06180126e0 con 0x55deb8d13240 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.329+0000 7f05feffd640 1 -- 172.21.15.49:0/3494625041 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f06140043e0 con 0x55deb8d13240 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.329+0000 7f05feffd640 1 -- 172.21.15.49:0/3494625041 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f06140119a0 con 0x55deb8d13240 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.329+0000 7f0628849740 10 client.27025 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:59.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.329+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55deb898b3f0 con 0x55deb8d13240 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.330+0000 7f05feffd640 1 -- 172.21.15.49:0/3494625041 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06140035f0 con 0x55deb8d13240 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.330+0000 7f05feffd640 1 -- 172.21.15.49:0/3494625041 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f061400f620 con 0x55deb8d13240 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.330+0000 7f0628849740 10 client.27025 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.330+0000 7f0628849740 10 client.27025 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.330+0000 7f0628849740 20 client.27025 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.330+0000 7f0628849740 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55deb8d28c20 0x55deb8d2b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.330+0000 7f0628849740 4 client.27025 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.330+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f06180126e0 con 0x55deb8d28c20 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.330+0000 7f061ffff640 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55deb8d28c20 0x55deb8d2b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.331+0000 7f061ffff640 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55deb8d28c20 0x55deb8d2b080 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=656fb8ae7dce5e3c server_cookie=7d49fd62940eb061 in_seq=0 out_seq=0 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.331+0000 7f05feffd640 10 client.27025 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.332+0000 7f05feffd640 1 -- 172.21.15.49:0/3494625041 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f06180126e0 con 0x55deb8d28c20 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.332+0000 7f05feffd640 10 client.27025 handle_command_reply: tid=0 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.332+0000 7f0628849740 1 client.27025 shutdown 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.332+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d13240 msgr2=0x55deb8d1d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.332+0000 7f0628849740 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d13240 0x55deb8d1d280 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0614012730 tx=0x7f0614012bf0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.332+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 shutdown_connections 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.333+0000 7f0628849740 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55deb8d28c20 0x55deb8d2b080 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.333+0000 7f0628849740 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 0x55deb8d24280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.333+0000 7f0628849740 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d13240 0x55deb8d1d280 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.333+0000 7f0628849740 1 --2- 172.21.15.49:0/3494625041 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d1cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.333+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 >> 172.21.15.49:0/3494625041 conn(0x55deb8d0ddc0 msgr2=0x55deb8d0cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.333+0000 7f05feffd640 0 client.27025 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:59:59.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.333+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 shutdown_connections 2024-03-18T00:59:59.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.333+0000 7f0628849740 1 -- 172.21.15.49:0/3494625041 wait complete. 2024-03-18T00:59:59.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.333+0000 7f0628849740 20 client.27025 trim_cache size 0 max 16384 2024-03-18T00:59:59.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.350+0000 7f0628849740 1 Processor -- start 2024-03-18T00:59:59.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.352+0000 7f0628849740 1 -- start start 2024-03-18T00:59:59.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.353+0000 7f0628849740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d12850 0x55deb8d1b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.353+0000 7f0628849740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d13240 0x55deb8d24240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.353+0000 7f0628849740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 0x55deb8d1bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.353+0000 7f0628849740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f06201a6b50 con 0x55deb8d183b0 2024-03-18T00:59:59.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.353+0000 7f0628849740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f06201a86c0 con 0x55deb8d13240 2024-03-18T00:59:59.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.353+0000 7f0628849740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f06201a7220 con 0x55deb8d12850 2024-03-18T00:59:59.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.356+0000 7f061ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d12850 0x55deb8d1b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.356+0000 7f061ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 0x55deb8d1bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.356+0000 7f061ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d12850 0x55deb8d1b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60060/0 (socket says 172.21.15.49:60060) 2024-03-18T00:59:59.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.356+0000 7f061ffff640 1 -- 172.21.15.49:0/1693978557 learned_addr learned my addr 172.21.15.49:0/1693978557 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.361+0000 7f061ffff640 1 -- 172.21.15.49:0/1693978557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d13240 msgr2=0x55deb8d24240 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:59:59.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.361+0000 7f061ffff640 1 --2- 172.21.15.49:0/1693978557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d13240 0x55deb8d24240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.361+0000 7f061ffff640 1 -- 172.21.15.49:0/1693978557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 msgr2=0x55deb8d1bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.361+0000 7f061ffff640 1 --2- 172.21.15.49:0/1693978557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 0x55deb8d1bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.361+0000 7f061ffff640 1 -- 172.21.15.49:0/1693978557 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deb8a557d0 con 0x55deb8d12850 2024-03-18T00:59:59.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.367+0000 7f061ffff640 1 --2- 172.21.15.49:0/1693978557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d12850 0x55deb8d1b7f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f06180129f0 tx=0x7f0618009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caacb99778e98cad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.372+0000 7f06057fa640 1 -- 172.21.15.49:0/1693978557 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06180125c0 con 0x55deb8d12850 2024-03-18T00:59:59.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.372+0000 7f06057fa640 1 -- 172.21.15.49:0/1693978557 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f061800db70 con 0x55deb8d12850 2024-03-18T00:59:59.372 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.372+0000 7f06057fa640 1 -- 172.21.15.49:0/1693978557 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618005560 con 0x55deb8d12850 2024-03-18T00:59:59.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.375+0000 7f0628849740 1 -- 172.21.15.49:0/1693978557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d12850 msgr2=0x55deb8d1b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.375 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.375+0000 7f0628849740 1 --2- 172.21.15.49:0/1693978557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d12850 0x55deb8d1b7f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f06180129f0 tx=0x7f0618009500 comp rx=0 tx=0).stop 2024-03-18T00:59:59.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.381+0000 7f0628849740 1 -- 172.21.15.49:0/1693978557 shutdown_connections 2024-03-18T00:59:59.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.381+0000 7f0628849740 1 --2- 172.21.15.49:0/1693978557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d13240 0x55deb8d24240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.381+0000 7f0628849740 1 --2- 172.21.15.49:0/1693978557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d183b0 0x55deb8d1bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.381+0000 7f0628849740 1 --2- 172.21.15.49:0/1693978557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d12850 0x55deb8d1b7f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.383 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.381+0000 7f0628849740 1 -- 172.21.15.49:0/1693978557 >> 172.21.15.49:0/1693978557 conn(0x55deb8d0ddc0 msgr2=0x55deb8d1cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.387 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.387+0000 7f0628849740 1 -- 172.21.15.49:0/1693978557 shutdown_connections 2024-03-18T00:59:59.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.388+0000 7f0628849740 1 -- 172.21.15.49:0/1693978557 wait complete. 2024-03-18T00:59:59.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.389+0000 7f0628849740 1 -- start start 2024-03-18T00:59:59.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.389+0000 7f0628849740 1 Processor -- start 2024-03-18T00:59:59.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f0628849740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d1eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.389 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f0628849740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 0x55deb8d1f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f0628849740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8da6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f0628849740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f06201a7fc0 con 0x55deb8d13240 2024-03-18T00:59:59.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f0628849740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f06201a78f0 con 0x55deb8d12850 2024-03-18T00:59:59.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f0628849740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f06201a4b30 con 0x55deb8d183b0 2024-03-18T00:59:59.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f0628849740 10 client.0 _ll_register_callbacks cb 0x7f0614009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:59.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f061ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8da6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f061ffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8da6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60070/0 (socket says 172.21.15.49:60070) 2024-03-18T00:59:59.390 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f061ffff640 1 -- 172.21.15.49:0/3956205661 learned_addr learned my addr 172.21.15.49:0/3956205661 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.390+0000 7f061ffff640 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d1eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.391+0000 7f061ffff640 1 -- 172.21.15.49:0/3956205661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 msgr2=0x55deb8d1eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.391+0000 7f061ffff640 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d1eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.391+0000 7f061ffff640 1 -- 172.21.15.49:0/3956205661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 msgr2=0x55deb8d1f420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:59:59.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.391+0000 7f061ffff640 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 0x55deb8d1f420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.391+0000 7f061ffff640 1 -- 172.21.15.49:0/3956205661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deb8d244b0 con 0x55deb8d183b0 2024-03-18T00:59:59.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.392+0000 7f061ffff640 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8da6790 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f06180124f0 tx=0x7f0618032040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a700745781c0df4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.391 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.392+0000 7f05feffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T00:59:59.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.395+0000 7f05feffd640 1 -- 172.21.15.49:0/3956205661 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0618012230 con 0x55deb8d183b0 2024-03-18T00:59:59.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.395+0000 7f05feffd640 1 -- 172.21.15.49:0/3956205661 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0618003a30 con 0x55deb8d183b0 2024-03-18T00:59:59.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.395+0000 7f0628849740 1 -- 172.21.15.49:0/3956205661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f061800db70 con 0x55deb8d183b0 2024-03-18T00:59:59.395 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.396+0000 7f05feffd640 1 -- 172.21.15.49:0/3956205661 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0618003c30 con 0x55deb8d183b0 2024-03-18T00:59:59.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.397+0000 7f0628849740 10 client.27031 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:59.396 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.397+0000 7f0628849740 1 -- 172.21.15.49:0/3956205661 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55deb8da7d30 con 0x55deb8d183b0 2024-03-18T00:59:59.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.399+0000 7f05feffd640 1 -- 172.21.15.49:0/3956205661 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f061800db70 con 0x55deb8d183b0 2024-03-18T00:59:59.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.399+0000 7f05feffd640 1 -- 172.21.15.49:0/3956205661 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f0618046780 con 0x55deb8d183b0 2024-03-18T00:59:59.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.400+0000 7f0628849740 10 client.27031 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:59.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.400+0000 7f0628849740 10 client.27031 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T00:59:59.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.400+0000 7f0628849740 20 client.27031 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:59.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.400+0000 7f0628849740 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55deb8da8440 0x55deb8daa860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.400+0000 7f0628849740 4 client.27031 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:59:59.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.400+0000 7f0628849740 1 -- 172.21.15.49:0/3956205661 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55deb8c849d0 con 0x55deb8da8440 2024-03-18T00:59:59.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.402+0000 7f061effd640 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55deb8da8440 0x55deb8daa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.422+0000 7f061effd640 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55deb8da8440 0x55deb8daa860 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a172115509da345d server_cookie=fc31c406ddb613d9 in_seq=0 out_seq=0 2024-03-18T00:59:59.421 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.422+0000 7f05feffd640 10 client.27031 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T00:59:59.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.423+0000 7f05feffd640 1 -- 172.21.15.49:0/3956205661 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55deb8c849d0 con 0x55deb8da8440 2024-03-18T00:59:59.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.423+0000 7f05feffd640 10 client.27031 handle_command_reply: tid=0 2024-03-18T00:59:59.422 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.423+0000 7f0628849740 1 client.27031 shutdown 2024-03-18T00:59:59.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.427+0000 7f0628849740 1 -- 172.21.15.49:0/3956205661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 msgr2=0x55deb8da6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.427+0000 7f0628849740 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8da6790 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f06180124f0 tx=0x7f0618032040 comp rx=0 tx=0).stop 2024-03-18T00:59:59.426 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.427+0000 7f0628849740 1 -- 172.21.15.49:0/3956205661 shutdown_connections 2024-03-18T00:59:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.427+0000 7f0628849740 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x55deb8da8440 0x55deb8daa860 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.427+0000 7f0628849740 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55deb8d183b0 0x55deb8da6790 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.427+0000 7f0628849740 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55deb8d13240 0x55deb8d1f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.427+0000 7f0628849740 1 --2- 172.21.15.49:0/3956205661 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55deb8d12850 0x55deb8d1eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.427+0000 7f0628849740 1 -- 172.21.15.49:0/3956205661 >> 172.21.15.49:0/3956205661 conn(0x55deb8d0ddc0 msgr2=0x55deb8d147e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.427 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.427+0000 7f05feffd640 0 client.27031 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T00:59:59.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.430+0000 7f0628849740 1 -- 172.21.15.49:0/3956205661 shutdown_connections 2024-03-18T00:59:59.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.430+0000 7f0628849740 1 -- 172.21.15.49:0/3956205661 wait complete. 2024-03-18T00:59:59.431 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.432+0000 7f0628849740 20 client.27031 trim_cache size 0 max 16384 2024-03-18T00:59:59.436 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:59:59.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.442+0000 7f0625c63640 1 -- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06080a2880 msgr2=0x7f06080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.443+0000 7f0625c63640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06080a2880 0x7f06080a4d40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f06100b8a60 tx=0x7f06100b89f0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.443+0000 7f0625c63640 1 -- 172.21.15.49:0/1452325742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06201006e0 msgr2=0x7f06200694d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.443+0000 7f0625c63640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06201006e0 0x7f06200694d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f06100349e0 tx=0x7f0610032430 comp rx=0 tx=0).stop 2024-03-18T00:59:59.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.450+0000 7f0625c63640 1 -- 172.21.15.49:0/1452325742 shutdown_connections 2024-03-18T00:59:59.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.450+0000 7f0625c63640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06080a2880 0x7f06080a4d40 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.450+0000 7f0625c63640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f06201018d0 0x7f062019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.450+0000 7f0625c63640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0620102290 0x7f062019ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.450+0000 7f0625c63640 1 --2- 172.21.15.49:0/1452325742 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06201006e0 0x7f06200694d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.450 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.450+0000 7f0625c63640 1 -- 172.21.15.49:0/1452325742 >> 172.21.15.49:0/1452325742 conn(0x7f06200fbc70 msgr2=0x7f0620104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.457+0000 7f0625c63640 1 -- 172.21.15.49:0/1452325742 shutdown_connections 2024-03-18T00:59:59.456 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.457+0000 7f0625c63640 1 -- 172.21.15.49:0/1452325742 wait complete. 2024-03-18T00:59:59.490 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T00:59:59.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.792+0000 7f357ed22640 1 -- 172.21.15.49:0/1376859210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357810b150 msgr2=0x7f357810d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.792+0000 7f357ed22640 1 --2- 172.21.15.49:0/1376859210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357810b150 0x7f357810d5c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3570009ca0 tx=0x7f357002c140 comp rx=0 tx=0).stop 2024-03-18T00:59:59.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.798+0000 7f357ed22640 1 -- 172.21.15.49:0/1376859210 shutdown_connections 2024-03-18T00:59:59.798 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.798+0000 7f357ed22640 1 --2- 172.21.15.49:0/1376859210 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f357810db00 0x7f357810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.799 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.798+0000 7f357ed22640 1 --2- 172.21.15.49:0/1376859210 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f357810b150 0x7f357810d5c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.798+0000 7f357ed22640 1 --2- 172.21.15.49:0/1376859210 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f35781087e0 0x7f357810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.800 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.798+0000 7f357ed22640 1 -- 172.21.15.49:0/1376859210 >> 172.21.15.49:0/1376859210 conn(0x7f357806e640 msgr2=0x7f3578070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.802 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.801+0000 7f357ed22640 1 -- 172.21.15.49:0/1376859210 shutdown_connections 2024-03-18T00:59:59.805 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.804+0000 7f357ed22640 1 -- 172.21.15.49:0/1376859210 wait complete. 2024-03-18T00:59:59.806 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.806+0000 7f357ed22640 1 Processor -- start 2024-03-18T00:59:59.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.806+0000 7f357ed22640 1 -- start start 2024-03-18T00:59:59.807 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.807+0000 7f357ed22640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35781087e0 0x7f35781a6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.807+0000 7f357ed22640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f357810b150 0x7f35781a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.807+0000 7f357ed22640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f357810db00 0x7f35781adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.807+0000 7f357ed22640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f357811c290 con 0x7f35781087e0 2024-03-18T00:59:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.807+0000 7f357ed22640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f357811c110 con 0x7f357810db00 2024-03-18T00:59:59.808 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.807+0000 7f357ed22640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f357811c410 con 0x7f357810b150 2024-03-18T00:59:59.809 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.809+0000 7f357d2bf640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f357810db00 0x7f35781adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.810+0000 7f357cabe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35781087e0 0x7f35781a6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.810+0000 7f357cabe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35781087e0 0x7f35781a6940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41206/0 (socket says 172.21.15.49:41206) 2024-03-18T00:59:59.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.810+0000 7f357cabe640 1 -- 172.21.15.49:0/882131330 learned_addr learned my addr 172.21.15.49:0/882131330 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.811 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.810+0000 7f3577fff640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f357810b150 0x7f35781a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.815+0000 7f357cabe640 1 -- 172.21.15.49:0/882131330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f357810b150 msgr2=0x7f35781a6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.815+0000 7f357cabe640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f357810b150 0x7f35781a6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.815+0000 7f357cabe640 1 -- 172.21.15.49:0/882131330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f357810db00 msgr2=0x7f35781adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.815+0000 7f357cabe640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f357810db00 0x7f35781adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.815+0000 7f357cabe640 1 -- 172.21.15.49:0/882131330 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3578110040 con 0x7f35781087e0 2024-03-18T00:59:59.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.819+0000 7f357cabe640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35781087e0 0x7f35781a6940 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f356800e9f0 tx=0x7f356800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.823+0000 7f3575ffb640 1 -- 172.21.15.49:0/882131330 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356800cd80 con 0x7f35781087e0 2024-03-18T00:59:59.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.823+0000 7f3575ffb640 1 -- 172.21.15.49:0/882131330 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3568010910 con 0x7f35781087e0 2024-03-18T00:59:59.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.823+0000 7f3575ffb640 1 -- 172.21.15.49:0/882131330 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568018990 con 0x7f35781087e0 2024-03-18T00:59:59.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.824+0000 7f357ed22640 1 -- 172.21.15.49:0/882131330 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3578110330 con 0x7f35781087e0 2024-03-18T00:59:59.824 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.824+0000 7f357ed22640 1 -- 172.21.15.49:0/882131330 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3578110800 con 0x7f35781087e0 2024-03-18T00:59:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.836+0000 7f3575ffb640 1 -- 172.21.15.49:0/882131330 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3568010ac0 con 0x7f35781087e0 2024-03-18T00:59:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.837+0000 7f3575ffb640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f35600a29d0 0x7f35600a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.837+0000 7f3575ffb640 1 -- 172.21.15.49:0/882131330 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f35680ceda0 con 0x7f35781087e0 2024-03-18T00:59:59.838 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.838+0000 7f35819ec740 1 Processor -- start 2024-03-18T00:59:59.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.843+0000 7f3577fff640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f35600a29d0 0x7f35600a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.844+0000 7f35819ec740 1 -- start start 2024-03-18T00:59:59.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.844+0000 7f35819ec740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 0x5617ba131c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.844+0000 7f35819ec740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba136d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.844+0000 7f35819ec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba1373b0 0x5617ba139840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.844+0000 7f35819ec740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5617ba096490 con 0x5617ba132240 2024-03-18T00:59:59.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.844+0000 7f35819ec740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5617ba03ec90 con 0x5617ba131850 2024-03-18T00:59:59.845 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.844+0000 7f35819ec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5617ba092530 con 0x5617ba1373b0 2024-03-18T00:59:59.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.846+0000 7f357cabe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 0x5617ba131c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.847+0000 7f357cabe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 0x5617ba131c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52648/0 (socket says 172.21.15.49:52648) 2024-03-18T00:59:59.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.847+0000 7f357cabe640 1 -- 172.21.15.49:0/2125495444 learned_addr learned my addr 172.21.15.49:0/2125495444 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.849+0000 7f3577fff640 1 --2- 172.21.15.49:0/2125495444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.850+0000 7f3577fff640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f35600a29d0 0x7f35600a4e90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f357002c750 tx=0x7f3570032dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.850+0000 7f3577fff640 1 -- 172.21.15.49:0/2125495444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba1373b0 msgr2=0x5617ba139840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:59:59.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.850+0000 7f3577fff640 1 --2- 172.21.15.49:0/2125495444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba1373b0 0x5617ba139840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.851+0000 7f3577fff640 1 -- 172.21.15.49:0/2125495444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 msgr2=0x5617ba131c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.851+0000 7f3577fff640 1 --2- 172.21.15.49:0/2125495444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 0x5617ba131c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.851+0000 7f3577fff640 1 -- 172.21.15.49:0/2125495444 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617ba0f9500 con 0x5617ba132240 2024-03-18T00:59:59.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.851+0000 7f3577fff640 1 --2- 172.21.15.49:0/2125495444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba136d40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3570037040 tx=0x7f3570049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aade14cf08f4d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.854+0000 7f355effd640 1 -- 172.21.15.49:0/2125495444 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570052070 con 0x5617ba132240 2024-03-18T00:59:59.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.854+0000 7f355effd640 1 -- 172.21.15.49:0/2125495444 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3570034ca0 con 0x5617ba132240 2024-03-18T00:59:59.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.855+0000 7f35819ec740 1 -- 172.21.15.49:0/2125495444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 msgr2=0x5617ba136d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.855+0000 7f35819ec740 1 --2- 172.21.15.49:0/2125495444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba136d40 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3570037040 tx=0x7f3570049000 comp rx=0 tx=0).stop 2024-03-18T00:59:59.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.856+0000 7f35819ec740 1 -- 172.21.15.49:0/2125495444 shutdown_connections 2024-03-18T00:59:59.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.856+0000 7f35819ec740 1 --2- 172.21.15.49:0/2125495444 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba1373b0 0x5617ba139840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.856+0000 7f35819ec740 1 --2- 172.21.15.49:0/2125495444 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba136d40 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.856+0000 7f35819ec740 1 --2- 172.21.15.49:0/2125495444 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 0x5617ba131c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.856+0000 7f35819ec740 1 -- 172.21.15.49:0/2125495444 >> 172.21.15.49:0/2125495444 conn(0x5617ba12cdc0 msgr2=0x5617ba12d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.859+0000 7f35819ec740 1 -- 172.21.15.49:0/2125495444 shutdown_connections 2024-03-18T00:59:59.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.860+0000 7f35819ec740 1 -- 172.21.15.49:0/2125495444 wait complete. 2024-03-18T00:59:59.861 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.861+0000 7f35819ec740 1 -- start start 2024-03-18T00:59:59.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.861+0000 7f35819ec740 1 Processor -- start 2024-03-18T00:59:59.863 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.863+0000 7f35819ec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 0x5617ba13bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.863+0000 7f35819ec740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba132240 0x5617ba13c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.863+0000 7f35819ec740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba1373b0 0x5617ba1432b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.863+0000 7f35819ec740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5617ba13a350 con 0x5617ba1373b0 2024-03-18T00:59:59.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.863+0000 7f35819ec740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5617ba052230 con 0x5617ba132240 2024-03-18T00:59:59.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.863+0000 7f35819ec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5617ba13a4d0 con 0x5617ba131850 2024-03-18T00:59:59.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.863+0000 7f35819ec740 10 client.0 _ll_register_callbacks cb 0x5617ba08e010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:59.864 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.864+0000 7f357cabe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 0x5617ba13bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.864+0000 7f357d2bf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba1373b0 0x5617ba1432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.864+0000 7f357d2bf640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba1373b0 0x5617ba1432b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41232/0 (socket says 172.21.15.49:41232) 2024-03-18T00:59:59.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.864+0000 7f357d2bf640 1 -- 172.21.15.49:0/3217913943 learned_addr learned my addr 172.21.15.49:0/3217913943 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.867+0000 7f3577fff640 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba132240 0x5617ba13c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.870+0000 7f3577fff640 1 -- 172.21.15.49:0/3217913943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 msgr2=0x5617ba13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.870+0000 7f3577fff640 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 0x5617ba13bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.870+0000 7f3577fff640 1 -- 172.21.15.49:0/3217913943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba1373b0 msgr2=0x5617ba1432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.870+0000 7f3577fff640 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba1373b0 0x5617ba1432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.870+0000 7f3577fff640 1 -- 172.21.15.49:0/3217913943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617ba061100 con 0x5617ba132240 2024-03-18T00:59:59.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.873+0000 7f3577fff640 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba132240 0x5617ba13c2b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3570009ca0 tx=0x7f357004d6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e6f543884490bc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.873+0000 7f3557fff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:59:59.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.874+0000 7f3557fff640 1 -- 172.21.15.49:0/3217913943 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357005b060 con 0x5617ba132240 2024-03-18T00:59:59.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.875+0000 7f3557fff640 1 -- 172.21.15.49:0/3217913943 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f35700347c0 con 0x5617ba132240 2024-03-18T00:59:59.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.875+0000 7f35819ec740 1 -- 172.21.15.49:0/3217913943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3570034ca0 con 0x5617ba132240 2024-03-18T00:59:59.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.877+0000 7f3557fff640 1 -- 172.21.15.49:0/3217913943 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3570052040 con 0x5617ba132240 2024-03-18T00:59:59.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.878+0000 7f35819ec740 10 client.26957 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:59.878 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.878+0000 7f35819ec740 1 -- 172.21.15.49:0/3217913943 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5617b9daa3f0 con 0x5617ba132240 2024-03-18T00:59:59.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.881+0000 7f3557fff640 1 -- 172.21.15.49:0/3217913943 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3570034ca0 con 0x5617ba132240 2024-03-18T00:59:59.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.881+0000 7f3557fff640 1 -- 172.21.15.49:0/3217913943 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f3570041440 con 0x5617ba132240 2024-03-18T00:59:59.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.882+0000 7f35819ec740 10 client.26957 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:59.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.882+0000 7f35819ec740 10 client.26957 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:59:59.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.882+0000 7f35819ec740 20 client.26957 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:59.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.882+0000 7f35819ec740 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5617ba1477c0 0x5617ba149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.882+0000 7f35819ec740 4 client.26957 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T00:59:59.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.882+0000 7f35819ec740 1 -- 172.21.15.49:0/3217913943 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617ba0a39d0 con 0x5617ba1477c0 2024-03-18T00:59:59.890 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.890+0000 7f357cabe640 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5617ba1477c0 0x5617ba149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.904+0000 7f3557fff640 10 client.26957 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:59:59.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.904+0000 7f357cabe640 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5617ba1477c0 0x5617ba149bb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af9797e5548f1a25 server_cookie=1ee8ab6a7ef96da4 in_seq=0 out_seq=0 2024-03-18T00:59:59.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.908+0000 7f3557fff640 1 -- 172.21.15.49:0/3217913943 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5617ba0a39d0 con 0x5617ba1477c0 2024-03-18T00:59:59.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.908+0000 7f3557fff640 10 client.26957 handle_command_reply: tid=0 2024-03-18T00:59:59.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.908+0000 7f35819ec740 1 client.26957 shutdown 2024-03-18T00:59:59.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.909+0000 7f35819ec740 1 -- 172.21.15.49:0/3217913943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba132240 msgr2=0x5617ba13c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.910 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.909+0000 7f35819ec740 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba132240 0x5617ba13c2b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3570009ca0 tx=0x7f357004d6b0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.913+0000 7f35819ec740 1 -- 172.21.15.49:0/3217913943 shutdown_connections 2024-03-18T00:59:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.913+0000 7f35819ec740 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5617ba1477c0 0x5617ba149bb0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.913+0000 7f35819ec740 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba132240 0x5617ba13c2b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.913+0000 7f35819ec740 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba1373b0 0x5617ba1432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.913+0000 7f35819ec740 1 --2- 172.21.15.49:0/3217913943 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 0x5617ba13bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.913+0000 7f3557fff640 0 client.26957 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:59:59.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.913+0000 7f35819ec740 1 -- 172.21.15.49:0/3217913943 >> 172.21.15.49:0/3217913943 conn(0x5617ba12cdc0 msgr2=0x5617ba12bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.916+0000 7f35819ec740 1 -- 172.21.15.49:0/3217913943 shutdown_connections 2024-03-18T00:59:59.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.917+0000 7f35819ec740 1 -- 172.21.15.49:0/3217913943 wait complete. 2024-03-18T00:59:59.918 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.917+0000 7f35819ec740 20 client.26957 trim_cache size 0 max 16384 2024-03-18T00:59:59.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.947+0000 7f35819ec740 1 Processor -- start 2024-03-18T00:59:59.950 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.950+0000 7f35819ec740 1 -- start start 2024-03-18T00:59:59.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.951+0000 7f35819ec740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 0x5617ba132990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.951+0000 7f35819ec740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.951+0000 7f35819ec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba1373b0 0x5617ba130730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.951+0000 7f35819ec740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f35781c04c0 con 0x5617ba132240 2024-03-18T00:59:59.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.951+0000 7f35819ec740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f35781c2030 con 0x5617ba131850 2024-03-18T00:59:59.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.951+0000 7f35819ec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f35781c0b90 con 0x5617ba1373b0 2024-03-18T00:59:59.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.951+0000 7f3577fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.951+0000 7f3577fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41238/0 (socket says 172.21.15.49:41238) 2024-03-18T00:59:59.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.951+0000 7f3577fff640 1 -- 172.21.15.49:0/3070018616 learned_addr learned my addr 172.21.15.49:0/3070018616 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.952+0000 7f3577fff640 1 -- 172.21.15.49:0/3070018616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba1373b0 msgr2=0x5617ba130730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T00:59:59.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.952+0000 7f3577fff640 1 --2- 172.21.15.49:0/3070018616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba1373b0 0x5617ba130730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.952+0000 7f3577fff640 1 -- 172.21.15.49:0/3070018616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 msgr2=0x5617ba132990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T00:59:59.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.952+0000 7f3577fff640 1 --2- 172.21.15.49:0/3070018616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 0x5617ba132990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.952+0000 7f3577fff640 1 -- 172.21.15.49:0/3070018616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617b9e747d0 con 0x5617ba132240 2024-03-18T00:59:59.952 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.953+0000 7f3577fff640 1 --2- 172.21.15.49:0/3070018616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143270 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f3570000f80 tx=0x7f3570032e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dc52bd5806e52ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.953+0000 7f355e7fc640 1 -- 172.21.15.49:0/3070018616 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357005b040 con 0x5617ba132240 2024-03-18T00:59:59.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.954+0000 7f355e7fc640 1 -- 172.21.15.49:0/3070018616 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f357004d820 con 0x5617ba132240 2024-03-18T00:59:59.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.955+0000 7f355e7fc640 1 -- 172.21.15.49:0/3070018616 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570065080 con 0x5617ba132240 2024-03-18T00:59:59.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.955+0000 7f35819ec740 1 -- 172.21.15.49:0/3070018616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 msgr2=0x5617ba143270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.955+0000 7f35819ec740 1 --2- 172.21.15.49:0/3070018616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143270 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f3570000f80 tx=0x7f3570032e80 comp rx=0 tx=0).stop 2024-03-18T00:59:59.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.956+0000 7f35819ec740 1 -- 172.21.15.49:0/3070018616 shutdown_connections 2024-03-18T00:59:59.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.956+0000 7f35819ec740 1 --2- 172.21.15.49:0/3070018616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba1373b0 0x5617ba130730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.956+0000 7f35819ec740 1 --2- 172.21.15.49:0/3070018616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143270 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.956+0000 7f35819ec740 1 --2- 172.21.15.49:0/3070018616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba131850 0x5617ba132990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.956+0000 7f35819ec740 1 -- 172.21.15.49:0/3070018616 >> 172.21.15.49:0/3070018616 conn(0x5617ba12cdc0 msgr2=0x5617ba130020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.956+0000 7f35819ec740 1 -- 172.21.15.49:0/3070018616 shutdown_connections 2024-03-18T00:59:59.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.956+0000 7f35819ec740 1 -- 172.21.15.49:0/3070018616 wait complete. 2024-03-18T00:59:59.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.956+0000 7f35819ec740 1 -- start start 2024-03-18T00:59:59.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.957+0000 7f35819ec740 1 Processor -- start 2024-03-18T00:59:59.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.958+0000 7f35819ec740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 0x5617ba142c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.958+0000 7f35819ec740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.958+0000 7f35819ec740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba1373b0 0x5617ba141270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.958+0000 7f35819ec740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f35781c1930 con 0x5617ba132240 2024-03-18T00:59:59.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.958+0000 7f35819ec740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f35781c1260 con 0x5617ba1373b0 2024-03-18T00:59:59.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.958+0000 7f35819ec740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f3578117770 con 0x5617ba131850 2024-03-18T00:59:59.958 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.958+0000 7f35819ec740 10 client.0 _ll_register_callbacks cb 0x7f3570050070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f3577fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f3577fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41244/0 (socket says 172.21.15.49:41244) 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f3577fff640 1 -- 172.21.15.49:0/2096415090 learned_addr learned my addr 172.21.15.49:0/2096415090 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f357d2bf640 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba1373b0 0x5617ba141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f357cabe640 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 0x5617ba142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f357d2bf640 1 -- 172.21.15.49:0/2096415090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 msgr2=0x5617ba142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f357d2bf640 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 0x5617ba142c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f357d2bf640 1 -- 172.21.15.49:0/2096415090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 msgr2=0x5617ba143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f357d2bf640 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.959 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.959+0000 7f357d2bf640 1 -- 172.21.15.49:0/2096415090 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617ba141960 con 0x5617ba1373b0 2024-03-18T00:59:59.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.960+0000 7f357d2bf640 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba1373b0 0x5617ba141270 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x5617ba13ca40 tx=0x7f356c00bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6532aa5b2dd37f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T00:59:59.960 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.961+0000 7f3557fff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T00:59:59.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.961+0000 7f3557fff640 1 -- 172.21.15.49:0/2096415090 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c00be40 con 0x5617ba1373b0 2024-03-18T00:59:59.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.961+0000 7f3557fff640 1 -- 172.21.15.49:0/2096415090 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f356c003cb0 con 0x5617ba1373b0 2024-03-18T00:59:59.961 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.961+0000 7f35819ec740 1 -- 172.21.15.49:0/2096415090 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f357004d820 con 0x5617ba1373b0 2024-03-18T00:59:59.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.962+0000 7f3557fff640 1 -- 172.21.15.49:0/2096415090 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f356c011440 con 0x5617ba1373b0 2024-03-18T00:59:59.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.962+0000 7f35819ec740 10 client.26960 fetch_fsmap learned FSMap version 30 2024-03-18T00:59:59.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.962+0000 7f35819ec740 1 -- 172.21.15.49:0/2096415090 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5617ba1ae370 con 0x5617ba1373b0 2024-03-18T00:59:59.962 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.963+0000 7f3557fff640 1 -- 172.21.15.49:0/2096415090 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f357004d820 con 0x5617ba1373b0 2024-03-18T00:59:59.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.964+0000 7f3557fff640 1 -- 172.21.15.49:0/2096415090 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f356c025760 con 0x5617ba1373b0 2024-03-18T00:59:59.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.964+0000 7f35819ec740 10 client.26960 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T00:59:59.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.964+0000 7f35819ec740 10 client.26960 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T00:59:59.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.964+0000 7f35819ec740 20 client.26960 populate_metadata read hostname 'smithi049' 2024-03-18T00:59:59.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.964+0000 7f35819ec740 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5617ba1aed30 0x5617ba1b1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T00:59:59.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.964+0000 7f35819ec740 4 client.26960 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T00:59:59.964 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.964+0000 7f35819ec740 1 -- 172.21.15.49:0/2096415090 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617ba09dd20 con 0x5617ba1aed30 2024-03-18T00:59:59.965 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.965+0000 7f357cabe640 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5617ba1aed30 0x5617ba1b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T00:59:59.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.966+0000 7f357cabe640 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5617ba1aed30 0x5617ba1b1150 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e28d0b31a999592 server_cookie=473d0ca3551e5fc9 in_seq=0 out_seq=0 2024-03-18T00:59:59.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.966+0000 7f3557fff640 10 client.26960 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T00:59:59.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.967+0000 7f3557fff640 1 -- 172.21.15.49:0/2096415090 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617ba09dd20 con 0x5617ba1aed30 2024-03-18T00:59:59.966 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.967+0000 7f3557fff640 10 client.26960 handle_command_reply: tid=0 2024-03-18T00:59:59.967 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.967+0000 7f35819ec740 1 client.26960 shutdown 2024-03-18T00:59:59.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f35819ec740 1 -- 172.21.15.49:0/2096415090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba1373b0 msgr2=0x5617ba141270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f35819ec740 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba1373b0 0x5617ba141270 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x5617ba13ca40 tx=0x7f356c00bd10 comp rx=0 tx=0).stop 2024-03-18T00:59:59.968 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f35819ec740 1 -- 172.21.15.49:0/2096415090 shutdown_connections 2024-03-18T00:59:59.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f35819ec740 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5617ba1aed30 0x5617ba1b1150 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f35819ec740 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5617ba1373b0 0x5617ba141270 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f35819ec740 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5617ba132240 0x5617ba143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f35819ec740 1 --2- 172.21.15.49:0/2096415090 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5617ba131850 0x5617ba142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f3557fff640 0 client.26960 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T00:59:59.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f35819ec740 1 -- 172.21.15.49:0/2096415090 >> 172.21.15.49:0/2096415090 conn(0x5617ba12cdc0 msgr2=0x5617ba1308b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.969 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.969+0000 7f35819ec740 1 -- 172.21.15.49:0/2096415090 shutdown_connections 2024-03-18T00:59:59.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.970+0000 7f35819ec740 1 -- 172.21.15.49:0/2096415090 wait complete. 2024-03-18T00:59:59.970 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.971+0000 7f35819ec740 20 client.26960 trim_cache size 0 max 16384 2024-03-18T00:59:59.971 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T00:59:59.973 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 -- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f35600a29d0 msgr2=0x7f35600a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f35600a29d0 0x7f35600a4e90 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f357002c750 tx=0x7f3570032dc0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 -- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35781087e0 msgr2=0x7f35781a6940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35781087e0 0x7f35781a6940 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f356800e9f0 tx=0x7f356800eeb0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 -- 172.21.15.49:0/882131330 shutdown_connections 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f35600a29d0 0x7f35600a4e90 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f357810db00 0x7f35781adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f357810b150 0x7f35781a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 --2- 172.21.15.49:0/882131330 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f35781087e0 0x7f35781a6940 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.973+0000 7f357ed22640 1 -- 172.21.15.49:0/882131330 >> 172.21.15.49:0/882131330 conn(0x7f357806e640 msgr2=0x7f3578108c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.974+0000 7f357ed22640 1 -- 172.21.15.49:0/882131330 shutdown_connections 2024-03-18T00:59:59.974 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T00:59:59.974+0000 7f357ed22640 1 -- 172.21.15.49:0/882131330 wait complete. 2024-03-18T00:59:59.999 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T01:00:00.040 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 00:59:59 smithi049 ceph-mon[32130]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 218 MiB/s rd, 472 MiB/s wr, 5.58k op/s 2024-03-18T01:00:00.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 -- 172.21.15.49:0/1150443528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 msgr2=0x7eff68108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:00.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 --2- 172.21.15.49:0/1150443528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 0x7eff68108b60 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7eff60009ca0 tx=0x7eff6002c0f0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 -- 172.21.15.49:0/1150443528 shutdown_connections 2024-03-18T01:00:00.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 --2- 172.21.15.49:0/1150443528 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 0x7eff68108b60 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 --2- 172.21.15.49:0/1150443528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68101910 0x7eff68101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 --2- 172.21.15.49:0/1150443528 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff68100720 0x7eff68100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 -- 172.21.15.49:0/1150443528 >> 172.21.15.49:0/1150443528 conn(0x7eff680fbc90 msgr2=0x7eff680fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:00.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 -- 172.21.15.49:0/1150443528 shutdown_connections 2024-03-18T01:00:00.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 -- 172.21.15.49:0/1150443528 wait complete. 2024-03-18T01:00:00.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.163+0000 7eff7047e640 1 Processor -- start 2024-03-18T01:00:00.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff7047e640 1 -- start start 2024-03-18T01:00:00.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff7047e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68100720 0x7eff68104660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff7047e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff68101910 0x7eff68102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff7047e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 0x7eff68103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff7047e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7eff6810c3a0 con 0x7eff681022d0 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff7047e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7eff6810c220 con 0x7eff68101910 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff7047e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7eff6810c520 con 0x7eff68100720 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff6ea1b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 0x7eff68103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff6ea1b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 0x7eff68103230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41260/0 (socket says 172.21.15.49:41260) 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff6ea1b640 1 -- 172.21.15.49:0/756759238 learned_addr learned my addr 172.21.15.49:0/756759238 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff6ea1b640 1 -- 172.21.15.49:0/756759238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68100720 msgr2=0x7eff68104660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68100720 0x7eff68104660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff6ea1b640 1 -- 172.21.15.49:0/756759238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff68101910 msgr2=0x7eff68102cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T01:00:00.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff68101910 0x7eff68102cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.164+0000 7eff6ea1b640 1 -- 172.21.15.49:0/756759238 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff68103af0 con 0x7eff681022d0 2024-03-18T01:00:00.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.165+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 0x7eff68103230 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7eff60005cc0 tx=0x7eff60032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:00.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.165+0000 7eff5f7fe640 1 -- 172.21.15.49:0/756759238 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff60041020 con 0x7eff681022d0 2024-03-18T01:00:00.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.165+0000 7eff7047e640 1 -- 172.21.15.49:0/756759238 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff681a1c30 con 0x7eff681022d0 2024-03-18T01:00:00.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.165+0000 7eff5f7fe640 1 -- 172.21.15.49:0/756759238 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff60034cd0 con 0x7eff681022d0 2024-03-18T01:00:00.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.165+0000 7eff5f7fe640 1 -- 172.21.15.49:0/756759238 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff60039040 con 0x7eff681022d0 2024-03-18T01:00:00.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.165+0000 7eff7047e640 1 -- 172.21.15.49:0/756759238 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff681a20f0 con 0x7eff681022d0 2024-03-18T01:00:00.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.167+0000 7eff5f7fe640 1 -- 172.21.15.49:0/756759238 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7eff60044020 con 0x7eff681022d0 2024-03-18T01:00:00.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.168+0000 7eff73069740 1 Processor -- start 2024-03-18T01:00:00.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.168+0000 7eff5f7fe640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7eff540a2800 0x7eff540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.168+0000 7eff73069740 1 -- start start 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff5f7fe640 1 -- 172.21.15.49:0/756759238 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7eff600f09e0 con 0x7eff681022d0 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff73069740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6beec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff73069740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bef240 0x562df6bf3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff73069740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bf6840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff73069740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x562df6b53490 con 0x562df6bf43b0 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff73069740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x562df6afbc90 con 0x562df6bee850 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff73069740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562df6b4f530 con 0x562df6bef240 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6da19640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7eff540a2800 0x7eff540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6ea1b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bf6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6ea1b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bf6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41262/0 (socket says 172.21.15.49:41262) 2024-03-18T01:00:00.169 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6ea1b640 1 -- 172.21.15.49:0/2514876685 learned_addr learned my addr 172.21.15.49:0/2514876685 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6e21a640 1 --2- 172.21.15.49:0/2514876685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6beec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6ea1b640 1 -- 172.21.15.49:0/2514876685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bef240 msgr2=0x562df6bf3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/2514876685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bef240 0x562df6bf3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6ea1b640 1 -- 172.21.15.49:0/2514876685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 msgr2=0x562df6beec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/2514876685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6beec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6ea1b640 1 -- 172.21.15.49:0/2514876685 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562df6bb6500 con 0x562df6bf43b0 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.169+0000 7eff6e21a640 1 --2- 172.21.15.49:0/2514876685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6beec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.170+0000 7eff6da19640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7eff540a2800 0x7eff540a4cc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7eff681043d0 tx=0x7eff58009e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.170+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/2514876685 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bf6840 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7eff600bd380 tx=0x7eff600323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3989ad9b95a37fc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.170+0000 7eff5cbfe640 1 -- 172.21.15.49:0/2514876685 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff600dbe60 con 0x562df6bf43b0 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.170+0000 7eff5cbfe640 1 -- 172.21.15.49:0/2514876685 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff600bdaf0 con 0x562df6bf43b0 2024-03-18T01:00:00.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.170+0000 7eff5cbfe640 1 -- 172.21.15.49:0/2514876685 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff60003970 con 0x562df6bf43b0 2024-03-18T01:00:00.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 00:59:59 smithi107 ceph-mon[36270]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 218 MiB/s rd, 472 MiB/s wr, 5.58k op/s 2024-03-18T01:00:00.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.173+0000 7eff73069740 1 -- 172.21.15.49:0/2514876685 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 msgr2=0x562df6bf6840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:00.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.173+0000 7eff73069740 1 --2- 172.21.15.49:0/2514876685 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bf6840 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7eff600bd380 tx=0x7eff600323c0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.178+0000 7eff73069740 1 -- 172.21.15.49:0/2514876685 shutdown_connections 2024-03-18T01:00:00.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.178+0000 7eff73069740 1 --2- 172.21.15.49:0/2514876685 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bf6840 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.178+0000 7eff73069740 1 --2- 172.21.15.49:0/2514876685 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bef240 0x562df6bf3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.178+0000 7eff73069740 1 --2- 172.21.15.49:0/2514876685 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6beec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.186 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.178+0000 7eff73069740 1 -- 172.21.15.49:0/2514876685 >> 172.21.15.49:0/2514876685 conn(0x562df6be9dc0 msgr2=0x562df6bea1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:00.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.186+0000 7eff73069740 1 -- 172.21.15.49:0/2514876685 shutdown_connections 2024-03-18T01:00:00.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.187+0000 7eff73069740 1 -- 172.21.15.49:0/2514876685 wait complete. 2024-03-18T01:00:00.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 00:59:59 smithi106 ceph-mon[35899]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 218 MiB/s rd, 472 MiB/s wr, 5.58k op/s 2024-03-18T01:00:00.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.188+0000 7eff73069740 1 -- start start 2024-03-18T01:00:00.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.188+0000 7eff73069740 1 Processor -- start 2024-03-18T01:00:00.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.189+0000 7eff73069740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6bf7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.189+0000 7eff73069740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bef240 0x562df6bf7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.196 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.189+0000 7eff73069740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bfeec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.189+0000 7eff73069740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x562df6bf7120 con 0x562df6bf43b0 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.189+0000 7eff73069740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x562df6b0f230 con 0x562df6bee850 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.189+0000 7eff73069740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x562df6bf72a0 con 0x562df6bef240 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.189+0000 7eff73069740 10 client.0 _ll_register_callbacks cb 0x562df6b4b010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.190+0000 7eff6da19640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bef240 0x562df6bf7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.191+0000 7eff6ea1b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bfeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.191+0000 7eff6ea1b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bfeec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41276/0 (socket says 172.21.15.49:41276) 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.191+0000 7eff6ea1b640 1 -- 172.21.15.49:0/2231846472 learned_addr learned my addr 172.21.15.49:0/2231846472 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.192+0000 7eff6ea1b640 1 -- 172.21.15.49:0/2231846472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bef240 msgr2=0x562df6bf7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.192+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bef240 0x562df6bf7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.192+0000 7eff6ea1b640 1 -- 172.21.15.49:0/2231846472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 msgr2=0x562df6bf7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.192+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6bf7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.192+0000 7eff6ea1b640 1 -- 172.21.15.49:0/2231846472 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562df6b1e100 con 0x562df6bf43b0 2024-03-18T01:00:00.197 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.193+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bfeec0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7eff600325f0 tx=0x7eff600bd050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dab1a07c1c91567 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:00.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.193+0000 7eff51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:00.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.195+0000 7eff51ffb640 1 -- 172.21.15.49:0/2231846472 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff600dbe60 con 0x562df6bf43b0 2024-03-18T01:00:00.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.195+0000 7eff73069740 1 -- 172.21.15.49:0/2231846472 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff600bdaf0 con 0x562df6bf43b0 2024-03-18T01:00:00.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.195+0000 7eff51ffb640 1 -- 172.21.15.49:0/2231846472 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7eff600d6150 con 0x562df6bf43b0 2024-03-18T01:00:00.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.197+0000 7eff51ffb640 1 -- 172.21.15.49:0/2231846472 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7eff600bd3f0 con 0x562df6bf43b0 2024-03-18T01:00:00.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.198+0000 7eff73069740 10 client.17829 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:00.198 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.198+0000 7eff73069740 1 -- 172.21.15.49:0/2231846472 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562df68673f0 con 0x562df6bf43b0 2024-03-18T01:00:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.206+0000 7eff51ffb640 1 -- 172.21.15.49:0/2231846472 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff600bdaf0 con 0x562df6bf43b0 2024-03-18T01:00:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.206+0000 7eff51ffb640 1 -- 172.21.15.49:0/2231846472 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7eff6003c030 con 0x562df6bf43b0 2024-03-18T01:00:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.206+0000 7eff73069740 10 client.17829 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.206+0000 7eff73069740 10 client.17829 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.206+0000 7eff73069740 20 client.17829 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.206+0000 7eff73069740 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562df6c023a0 0x562df6c04800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.206+0000 7eff73069740 4 client.17829 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:00.209 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.206+0000 7eff73069740 1 -- 172.21.15.49:0/2231846472 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562df6b609d0 con 0x562df6c023a0 2024-03-18T01:00:00.214 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.211+0000 7eff6e21a640 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562df6c023a0 0x562df6c04800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.233+0000 7eff6e21a640 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562df6c023a0 0x562df6c04800 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9741ed4c75b581d server_cookie=73f22e49f1b2fe91 in_seq=0 out_seq=0 2024-03-18T01:00:00.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.234+0000 7eff51ffb640 10 client.17829 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:00.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.236+0000 7eff51ffb640 1 -- 172.21.15.49:0/2231846472 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562df6b609d0 con 0x562df6c023a0 2024-03-18T01:00:00.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.236+0000 7eff51ffb640 10 client.17829 handle_command_reply: tid=0 2024-03-18T01:00:00.236 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.236+0000 7eff73069740 1 client.17829 shutdown 2024-03-18T01:00:00.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff73069740 1 -- 172.21.15.49:0/2231846472 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 msgr2=0x562df6bfeec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:00.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff73069740 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bfeec0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7eff600325f0 tx=0x7eff600bd050 comp rx=0 tx=0).stop 2024-03-18T01:00:00.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff73069740 1 -- 172.21.15.49:0/2231846472 shutdown_connections 2024-03-18T01:00:00.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff73069740 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562df6c023a0 0x562df6c04800 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff73069740 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bf43b0 0x562df6bfeec0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff73069740 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bef240 0x562df6bf7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff73069740 1 --2- 172.21.15.49:0/2231846472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6bf7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff73069740 1 -- 172.21.15.49:0/2231846472 >> 172.21.15.49:0/2231846472 conn(0x562df6be9dc0 msgr2=0x562df6be8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:00.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff51ffb640 0 client.17829 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:00.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.242+0000 7eff73069740 1 -- 172.21.15.49:0/2231846472 shutdown_connections 2024-03-18T01:00:00.245 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.245+0000 7eff73069740 1 -- 172.21.15.49:0/2231846472 wait complete. 2024-03-18T01:00:00.247 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.248+0000 7eff73069740 20 client.17829 trim_cache size 0 max 16384 2024-03-18T01:00:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.290+0000 7eff73069740 1 Processor -- start 2024-03-18T01:00:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.290+0000 7eff73069740 1 -- start start 2024-03-18T01:00:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.291+0000 7eff73069740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6c00020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.291+0000 7eff73069740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bef240 0x562df6bfee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.291+0000 7eff73069740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bf43b0 0x562df6be8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.291+0000 7eff73069740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7eff681a6c80 con 0x562df6bef240 2024-03-18T01:00:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.291+0000 7eff73069740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7eff681a87f0 con 0x562df6bee850 2024-03-18T01:00:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.291+0000 7eff73069740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7eff681a7350 con 0x562df6bf43b0 2024-03-18T01:00:00.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.291+0000 7eff6da19640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bef240 0x562df6bfee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.291+0000 7eff6da19640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bef240 0x562df6bfee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41292/0 (socket says 172.21.15.49:41292) 2024-03-18T01:00:00.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.291+0000 7eff6da19640 1 -- 172.21.15.49:0/544923251 learned_addr learned my addr 172.21.15.49:0/544923251 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:00.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.292+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/544923251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bf43b0 0x562df6be8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.292+0000 7eff6da19640 1 -- 172.21.15.49:0/544923251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bf43b0 msgr2=0x562df6be8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:00.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.292+0000 7eff6da19640 1 --2- 172.21.15.49:0/544923251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bf43b0 0x562df6be8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.292+0000 7eff6da19640 1 -- 172.21.15.49:0/544923251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 msgr2=0x562df6c00020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:00:00.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.292+0000 7eff6da19640 1 --2- 172.21.15.49:0/544923251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6c00020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.292+0000 7eff6da19640 1 -- 172.21.15.49:0/544923251 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562df69317d0 con 0x562df6bef240 2024-03-18T01:00:00.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.293+0000 7eff6da19640 1 --2- 172.21.15.49:0/544923251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bef240 0x562df6bfee80 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7eff5800ec50 tx=0x7eff58013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10b07dab237c70a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:00.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.293+0000 7eff53fff640 1 -- 172.21.15.49:0/544923251 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58023070 con 0x562df6bef240 2024-03-18T01:00:00.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.293+0000 7eff53fff640 1 -- 172.21.15.49:0/544923251 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff5800ec80 con 0x562df6bef240 2024-03-18T01:00:00.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.293+0000 7eff53fff640 1 -- 172.21.15.49:0/544923251 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58010680 con 0x562df6bef240 2024-03-18T01:00:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.294+0000 7eff73069740 1 -- 172.21.15.49:0/544923251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bef240 msgr2=0x562df6bfee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.294+0000 7eff73069740 1 --2- 172.21.15.49:0/544923251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bef240 0x562df6bfee80 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7eff5800ec50 tx=0x7eff58013000 comp rx=0 tx=0).stop 2024-03-18T01:00:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.294+0000 7eff73069740 1 -- 172.21.15.49:0/544923251 shutdown_connections 2024-03-18T01:00:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.294+0000 7eff73069740 1 --2- 172.21.15.49:0/544923251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bf43b0 0x562df6be8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.294+0000 7eff73069740 1 --2- 172.21.15.49:0/544923251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bef240 0x562df6bfee80 secure :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7eff5800ec50 tx=0x7eff58013000 comp rx=0 tx=0).stop 2024-03-18T01:00:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.295+0000 7eff73069740 1 --2- 172.21.15.49:0/544923251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bee850 0x562df6c00020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.295+0000 7eff73069740 1 -- 172.21.15.49:0/544923251 >> 172.21.15.49:0/544923251 conn(0x562df6be9dc0 msgr2=0x562df6bff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.295+0000 7eff73069740 1 -- 172.21.15.49:0/544923251 shutdown_connections 2024-03-18T01:00:00.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.295+0000 7eff73069740 1 -- 172.21.15.49:0/544923251 wait complete. 2024-03-18T01:00:00.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.296+0000 7eff73069740 1 -- start start 2024-03-18T01:00:00.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.296+0000 7eff73069740 1 Processor -- start 2024-03-18T01:00:00.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.297+0000 7eff73069740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bee850 0x562df6bfb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.297+0000 7eff73069740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bf43b0 0x562df6bfb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.297+0000 7eff73069740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bfe390 0x562df6bfe810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.297+0000 7eff73069740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7eff681a80f0 con 0x562df6bee850 2024-03-18T01:00:00.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.297+0000 7eff73069740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7eff681a7a20 con 0x562df6bf43b0 2024-03-18T01:00:00.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.297+0000 7eff73069740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7eff681a4c80 con 0x562df6bfe390 2024-03-18T01:00:00.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.297+0000 7eff73069740 10 client.0 _ll_register_callbacks cb 0x7eff60009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:00.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.299+0000 7eff6e21a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bee850 0x562df6bfb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.299+0000 7eff6e21a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bee850 0x562df6bfb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41308/0 (socket says 172.21.15.49:41308) 2024-03-18T01:00:00.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.299+0000 7eff6e21a640 1 -- 172.21.15.49:0/2383251654 learned_addr learned my addr 172.21.15.49:0/2383251654 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:00.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.299+0000 7eff6ea1b640 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bfe390 0x562df6bfe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.299+0000 7eff6da19640 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bf43b0 0x562df6bfb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.300+0000 7eff6e21a640 1 -- 172.21.15.49:0/2383251654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bfe390 msgr2=0x562df6bfe810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:00.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.300+0000 7eff6e21a640 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bfe390 0x562df6bfe810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.300+0000 7eff6e21a640 1 -- 172.21.15.49:0/2383251654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bf43b0 msgr2=0x562df6bfb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:00.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.300+0000 7eff6e21a640 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bf43b0 0x562df6bfb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.300+0000 7eff6e21a640 1 -- 172.21.15.49:0/2383251654 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562df6bfed80 con 0x562df6bee850 2024-03-18T01:00:00.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.305+0000 7eff6e21a640 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bee850 0x562df6bfb010 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x562df6c05070 tx=0x7eff64012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b6ca28b04c8480 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:00.304 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.305+0000 7eff51ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:00.307 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.308+0000 7eff51ffb640 1 -- 172.21.15.49:0/2383251654 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6401e020 con 0x562df6bee850 2024-03-18T01:00:00.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.308+0000 7eff73069740 1 -- 172.21.15.49:0/2383251654 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff5800ec80 con 0x562df6bee850 2024-03-18T01:00:00.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.308+0000 7eff51ffb640 1 -- 172.21.15.49:0/2383251654 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7eff6401aca0 con 0x562df6bee850 2024-03-18T01:00:00.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.315+0000 7eff51ffb640 1 -- 172.21.15.49:0/2383251654 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7eff64029050 con 0x562df6bee850 2024-03-18T01:00:00.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.319+0000 7eff73069740 10 client.17835 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:00.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.319+0000 7eff73069740 1 -- 172.21.15.49:0/2383251654 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562df6c6b1d0 con 0x562df6bee850 2024-03-18T01:00:00.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.326+0000 7eff51ffb640 1 -- 172.21.15.49:0/2383251654 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff5800ec80 con 0x562df6bee850 2024-03-18T01:00:00.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.326+0000 7eff51ffb640 1 -- 172.21.15.49:0/2383251654 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7eff6400f040 con 0x562df6bee850 2024-03-18T01:00:00.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.327+0000 7eff73069740 10 client.17835 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:00.327 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.327+0000 7eff73069740 10 client.17835 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:00.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.327+0000 7eff73069740 20 client.17835 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:00.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.327+0000 7eff73069740 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562df6c6bc30 0x562df6c6e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:00.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.327+0000 7eff73069740 4 client.17835 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:00:00.328 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.327+0000 7eff73069740 1 -- 172.21.15.49:0/2383251654 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562df6b5ad20 con 0x562df6c6bc30 2024-03-18T01:00:00.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.330+0000 7eff6da19640 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562df6c6bc30 0x562df6c6e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:00.340 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.340+0000 7eff6da19640 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562df6c6bc30 0x562df6c6e050 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dcba84ed401813d server_cookie=5fb0285fcd963480 in_seq=0 out_seq=0 2024-03-18T01:00:00.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.345+0000 7eff51ffb640 10 client.17835 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:00.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.345+0000 7eff51ffb640 1 -- 172.21.15.49:0/2383251654 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562df6b5ad20 con 0x562df6c6bc30 2024-03-18T01:00:00.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.345+0000 7eff51ffb640 10 client.17835 handle_command_reply: tid=0 2024-03-18T01:00:00.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.347+0000 7eff73069740 1 client.17835 shutdown 2024-03-18T01:00:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.350+0000 7eff73069740 1 -- 172.21.15.49:0/2383251654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bee850 msgr2=0x562df6bfb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.350+0000 7eff73069740 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bee850 0x562df6bfb010 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x562df6c05070 tx=0x7eff64012820 comp rx=0 tx=0).stop 2024-03-18T01:00:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.350+0000 7eff73069740 1 -- 172.21.15.49:0/2383251654 shutdown_connections 2024-03-18T01:00:00.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.350+0000 7eff73069740 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x562df6c6bc30 0x562df6c6e050 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.350+0000 7eff73069740 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562df6bf43b0 0x562df6bfb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.350+0000 7eff73069740 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562df6bfe390 0x562df6bfe810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.350+0000 7eff73069740 1 --2- 172.21.15.49:0/2383251654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562df6bee850 0x562df6bfb010 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.350+0000 7eff73069740 1 -- 172.21.15.49:0/2383251654 >> 172.21.15.49:0/2383251654 conn(0x562df6be9dc0 msgr2=0x562df6bed020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:00.352 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.350+0000 7eff51ffb640 0 client.17835 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:00.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.354+0000 7eff73069740 1 -- 172.21.15.49:0/2383251654 shutdown_connections 2024-03-18T01:00:00.354 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.354+0000 7eff73069740 1 -- 172.21.15.49:0/2383251654 wait complete. 2024-03-18T01:00:00.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.355+0000 7eff73069740 20 client.17835 trim_cache size 0 max 16384 2024-03-18T01:00:00.359 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:00:00.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.366+0000 7eff7047e640 1 -- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7eff540a2800 msgr2=0x7eff540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:00.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.366+0000 7eff7047e640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7eff540a2800 0x7eff540a4cc0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7eff681043d0 tx=0x7eff58009e50 comp rx=0 tx=0).stop 2024-03-18T01:00:00.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.366+0000 7eff7047e640 1 -- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 msgr2=0x7eff68103230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:00.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.366+0000 7eff7047e640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 0x7eff68103230 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7eff60005cc0 tx=0x7eff60032f40 comp rx=0 tx=0).stop 2024-03-18T01:00:00.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.370+0000 7eff7047e640 1 -- 172.21.15.49:0/756759238 shutdown_connections 2024-03-18T01:00:00.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.370+0000 7eff7047e640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7eff540a2800 0x7eff540a4cc0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.370+0000 7eff7047e640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7eff68101910 0x7eff68102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.370+0000 7eff7047e640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7eff681022d0 0x7eff68103230 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.370+0000 7eff7047e640 1 --2- 172.21.15.49:0/756759238 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7eff68100720 0x7eff68104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:00.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.370+0000 7eff7047e640 1 -- 172.21.15.49:0/756759238 >> 172.21.15.49:0/756759238 conn(0x7eff680fbc90 msgr2=0x7eff680fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:00.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.375+0000 7eff7047e640 1 -- 172.21.15.49:0/756759238 shutdown_connections 2024-03-18T01:00:00.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:00.375+0000 7eff7047e640 1 -- 172.21.15.49:0/756759238 wait complete. 2024-03-18T01:00:01.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:00 smithi049 ceph-mon[32130]: overall HEALTH_OK 2024-03-18T01:00:01.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:00 smithi107 ceph-mon[36270]: overall HEALTH_OK 2024-03-18T01:00:01.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:00 smithi106 ceph-mon[35899]: overall HEALTH_OK 2024-03-18T01:00:01.416 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T01:00:01.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:00:01.416 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T01:00:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.710+0000 7f4a06dd4640 1 -- 172.21.15.49:0/2382844679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010dda0 msgr2=0x7f4a00110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:01.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.710+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/2382844679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010dda0 0x7f4a00110260 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f49f8005fd0 tx=0x7f49f800aec0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.714+0000 7f4a06dd4640 1 -- 172.21.15.49:0/2382844679 shutdown_connections 2024-03-18T01:00:01.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.714+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/2382844679 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a0010b250 0x7f4a0010d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.714+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/2382844679 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010dda0 0x7f4a00110260 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.714+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/2382844679 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a00108870 0x7f4a0010ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.714+0000 7f4a06dd4640 1 -- 172.21.15.49:0/2382844679 >> 172.21.15.49:0/2382844679 conn(0x7f4a0006eb40 msgr2=0x7f4a00070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.725+0000 7f4a06dd4640 1 -- 172.21.15.49:0/2382844679 shutdown_connections 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.726+0000 7f4a06dd4640 1 -- 172.21.15.49:0/2382844679 wait complete. 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f4a06dd4640 1 Processor -- start 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f4a06dd4640 1 -- start start 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f4a06dd4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a00108870 0x7f4a001aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f4a06dd4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010b250 0x7f4a001af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f4a06dd4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a0010dda0 0x7f4a001b6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f4a06dd4640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4a00113980 con 0x7f4a00108870 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f4a06dd4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4a00113800 con 0x7f4a0010dda0 2024-03-18T01:00:01.727 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f4a06dd4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4a00113b00 con 0x7f4a0010b250 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f49fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010b250 0x7f4a001af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f49fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010b250 0x7f4a001af500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60164/0 (socket says 172.21.15.49:60164) 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.727+0000 7f49fffff640 1 -- 172.21.15.49:0/3995312384 learned_addr learned my addr 172.21.15.49:0/3995312384 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.728+0000 7f4a04b70640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a00108870 0x7f4a001aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.728+0000 7f49fffff640 1 -- 172.21.15.49:0/3995312384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a0010dda0 msgr2=0x7f4a001b6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.728+0000 7f49fffff640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a0010dda0 0x7f4a001b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.728+0000 7f49fffff640 1 -- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a00108870 msgr2=0x7f4a001aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.728+0000 7f49fffff640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a00108870 0x7f4a001aefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.728+0000 7f49fffff640 1 -- 172.21.15.49:0/3995312384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a001b6be0 con 0x7f4a0010b250 2024-03-18T01:00:01.731 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.729+0000 7f49fffff640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010b250 0x7f4a001af500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f49f000daf0 tx=0x7f49f000af60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.733+0000 7f49fdffb640 1 -- 172.21.15.49:0/3995312384 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f001a070 con 0x7f4a0010b250 2024-03-18T01:00:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.734+0000 7f49fdffb640 1 -- 172.21.15.49:0/3995312384 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f000cc70 con 0x7f4a0010b250 2024-03-18T01:00:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.734+0000 7f49fdffb640 1 -- 172.21.15.49:0/3995312384 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f00146a0 con 0x7f4a0010b250 2024-03-18T01:00:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.734+0000 7f4a06dd4640 1 -- 172.21.15.49:0/3995312384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a001b6ed0 con 0x7f4a0010b250 2024-03-18T01:00:01.735 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.734+0000 7f4a06dd4640 1 -- 172.21.15.49:0/3995312384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a001b7340 con 0x7f4a0010b250 2024-03-18T01:00:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.740+0000 7f49fdffb640 1 -- 172.21.15.49:0/3995312384 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f49f001d070 con 0x7f4a0010b250 2024-03-18T01:00:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.741+0000 7f4a099c0740 1 Processor -- start 2024-03-18T01:00:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.741+0000 7f49fdffb640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49e80a2980 0x7f49e80a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.742 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.741+0000 7f49fdffb640 1 -- 172.21.15.49:0/3995312384 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f49f00cefa0 con 0x7f4a0010b250 2024-03-18T01:00:01.748 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.743+0000 7f4a099c0740 1 -- start start 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.743+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3247b80 0x55f9e3247fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.743+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 0x55f9e324d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.743+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e324fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.743+0000 7f4a099c0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f9e31ac0d0 con 0x55f9e3248570 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.743+0000 7f4a099c0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f9e3154c70 con 0x55f9e324d6e0 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.743+0000 7f4a099c0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f9e31a8170 con 0x55f9e3247b80 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.745+0000 7f4a05371640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49e80a2980 0x7f49e80a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.745+0000 7f49fffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 0x55f9e324d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.745+0000 7f49fffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 0x55f9e324d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41330/0 (socket says 172.21.15.49:41330) 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.745+0000 7f49fffff640 1 -- 172.21.15.49:0/43383961 learned_addr learned my addr 172.21.15.49:0/43383961 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.747+0000 7f4a05371640 1 --2- 172.21.15.49:0/43383961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e324fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.747+0000 7f4a05371640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49e80a2980 0x7f49e80a4e40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f49f8004af0 tx=0x7f49f802d550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:01.749 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.747+0000 7f4a05371640 1 -- 172.21.15.49:0/43383961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3247b80 msgr2=0x55f9e3247fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:01.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.747+0000 7f4a05371640 1 --2- 172.21.15.49:0/43383961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3247b80 0x55f9e3247fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.747+0000 7f4a05371640 1 -- 172.21.15.49:0/43383961 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 msgr2=0x55f9e324d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:01.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.747+0000 7f4a05371640 1 --2- 172.21.15.49:0/43383961 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 0x55f9e324d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.747+0000 7f4a05371640 1 -- 172.21.15.49:0/43383961 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9e2f887e0 con 0x55f9e324d6e0 2024-03-18T01:00:01.750 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.748+0000 7f4a05371640 1 --2- 172.21.15.49:0/43383961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e324fb70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f49f802d8f0 tx=0x7f49f8046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9864c12b9c22fed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.752+0000 7f49e6ffd640 1 -- 172.21.15.49:0/43383961 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8040040 con 0x55f9e324d6e0 2024-03-18T01:00:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.752+0000 7f49e6ffd640 1 -- 172.21.15.49:0/43383961 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f804a070 con 0x55f9e324d6e0 2024-03-18T01:00:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.754+0000 7f4a099c0740 1 -- 172.21.15.49:0/43383961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 msgr2=0x55f9e324fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:01.753 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.754+0000 7f4a099c0740 1 --2- 172.21.15.49:0/43383961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e324fb70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f49f802d8f0 tx=0x7f49f8046000 comp rx=0 tx=0).stop 2024-03-18T01:00:01.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.756+0000 7f4a099c0740 1 -- 172.21.15.49:0/43383961 shutdown_connections 2024-03-18T01:00:01.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.756+0000 7f4a099c0740 1 --2- 172.21.15.49:0/43383961 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e324fb70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.756+0000 7f4a099c0740 1 --2- 172.21.15.49:0/43383961 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 0x55f9e324d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.756+0000 7f4a099c0740 1 --2- 172.21.15.49:0/43383961 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3247b80 0x55f9e3247fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.756+0000 7f4a099c0740 1 -- 172.21.15.49:0/43383961 >> 172.21.15.49:0/43383961 conn(0x55f9e320fc50 msgr2=0x55f9e32434a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:01.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.756+0000 7f4a099c0740 1 -- 172.21.15.49:0/43383961 shutdown_connections 2024-03-18T01:00:01.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.756+0000 7f4a099c0740 1 -- 172.21.15.49:0/43383961 wait complete. 2024-03-18T01:00:01.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.756+0000 7f4a099c0740 1 -- start start 2024-03-18T01:00:01.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.757+0000 7f4a099c0740 1 Processor -- start 2024-03-18T01:00:01.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.757+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3247b80 0x55f9e3252020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.757+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e3252560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.757+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e32595a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.757+0000 7f4a099c0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f9e3250680 con 0x55f9e3247b80 2024-03-18T01:00:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.757+0000 7f4a099c0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f9e31680d0 con 0x55f9e324d6e0 2024-03-18T01:00:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.757+0000 7f4a099c0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f9e3250800 con 0x55f9e3248570 2024-03-18T01:00:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.757+0000 7f4a099c0740 10 client.0 _ll_register_callbacks cb 0x55f9e3131760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.758+0000 7f49fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e3252560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.758+0000 7f49fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e3252560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60180/0 (socket says 172.21.15.49:60180) 2024-03-18T01:00:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.758+0000 7f49fffff640 1 -- 172.21.15.49:0/2734709904 learned_addr learned my addr 172.21.15.49:0/2734709904 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:01.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.758+0000 7f4a05371640 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e32595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.758+0000 7f4a05371640 1 -- 172.21.15.49:0/2734709904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 msgr2=0x55f9e3252560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:01.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.758+0000 7f4a05371640 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e3252560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.758+0000 7f4a05371640 1 -- 172.21.15.49:0/2734709904 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3247b80 msgr2=0x55f9e3252020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:01.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.758+0000 7f4a05371640 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3247b80 0x55f9e3252020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.758+0000 7f4a05371640 1 -- 172.21.15.49:0/2734709904 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9e2fe8f70 con 0x55f9e324d6e0 2024-03-18T01:00:01.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.759+0000 7f4a05371640 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e32595a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f49f802c170 tx=0x7f49f800afd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e225b6f0aba465c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:01.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.760+0000 7f49dffff640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T01:00:01.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.760+0000 7f49dffff640 1 -- 172.21.15.49:0/2734709904 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8058020 con 0x55f9e324d6e0 2024-03-18T01:00:01.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.760+0000 7f4a099c0740 1 -- 172.21.15.49:0/2734709904 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49f804a070 con 0x55f9e324d6e0 2024-03-18T01:00:01.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.760+0000 7f49dffff640 1 -- 172.21.15.49:0/2734709904 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f49f8046ce0 con 0x55f9e324d6e0 2024-03-18T01:00:01.761 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.761+0000 7f49dffff640 1 -- 172.21.15.49:0/2734709904 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f49f8040040 con 0x55f9e324d6e0 2024-03-18T01:00:01.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.762+0000 7f4a099c0740 10 client.26975 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:01.762 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.762+0000 7f4a099c0740 1 -- 172.21.15.49:0/2734709904 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f9e2e80170 con 0x55f9e324d6e0 2024-03-18T01:00:01.765 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.762+0000 7f49dffff640 1 -- 172.21.15.49:0/2734709904 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f804a070 con 0x55f9e324d6e0 2024-03-18T01:00:01.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.767+0000 7f49dffff640 1 -- 172.21.15.49:0/2734709904 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f49f8043030 con 0x55f9e324d6e0 2024-03-18T01:00:01.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.768+0000 7f4a099c0740 10 client.26975 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:01.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.768+0000 7f4a099c0740 10 client.26975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:01.767 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.768+0000 7f4a099c0740 20 client.26975 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:01.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.768+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f9e325d700 0x55f9e325faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.768+0000 7f4a099c0740 4 client.26975 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:01.768 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.768+0000 7f4a099c0740 1 -- 172.21.15.49:0/2734709904 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9e31b9610 con 0x55f9e325d700 2024-03-18T01:00:01.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.769+0000 7f4a04b70640 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f9e325d700 0x55f9e325faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.774+0000 7f4a04b70640 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f9e325d700 0x55f9e325faf0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf932cd7b0825d6 server_cookie=ddcf8065ccb2c37f in_seq=0 out_seq=0 2024-03-18T01:00:01.774 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.774+0000 7f49dffff640 10 client.26975 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:01.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.780+0000 7f49dffff640 1 -- 172.21.15.49:0/2734709904 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f9e31b9610 con 0x55f9e325d700 2024-03-18T01:00:01.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.780+0000 7f49dffff640 10 client.26975 handle_command_reply: tid=0 2024-03-18T01:00:01.779 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.780+0000 7f4a099c0740 1 client.26975 shutdown 2024-03-18T01:00:01.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.780+0000 7f4a099c0740 1 -- 172.21.15.49:0/2734709904 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 msgr2=0x55f9e32595a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:01.780 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.780+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e32595a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f49f802c170 tx=0x7f49f800afd0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.783+0000 7f4a099c0740 1 -- 172.21.15.49:0/2734709904 shutdown_connections 2024-03-18T01:00:01.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.783+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f9e325d700 0x55f9e325faf0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.783+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e32595a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.783+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e3252560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.783 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.783+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2734709904 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3247b80 0x55f9e3252020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.784 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.783+0000 7f4a099c0740 1 -- 172.21.15.49:0/2734709904 >> 172.21.15.49:0/2734709904 conn(0x55f9e320fc50 msgr2=0x55f9e32426e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:01.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.787+0000 7f49dffff640 0 client.26975 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:01.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.789+0000 7f4a099c0740 1 -- 172.21.15.49:0/2734709904 shutdown_connections 2024-03-18T01:00:01.788 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.789+0000 7f4a099c0740 1 -- 172.21.15.49:0/2734709904 wait complete. 2024-03-18T01:00:01.794 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.792+0000 7f4a099c0740 20 client.26975 trim_cache size 0 max 16384 2024-03-18T01:00:01.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.840+0000 7f4a099c0740 1 Processor -- start 2024-03-18T01:00:01.840 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.840+0000 7f4a099c0740 1 -- start start 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e3247b80 0x55f9e32436d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 0x55f9e3259560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e324d6e0 0x55f9e3246730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a04b70640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e3247b80 0x55f9e32436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a04b70640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e3247b80 0x55f9e32436d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:52770/0 (socket says 172.21.15.49:52770) 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a099c0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec00e640 con 0x55f9e3248570 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a099c0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec0101b0 con 0x55f9e3247b80 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a099c0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec00ed10 con 0x55f9e324d6e0 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a04b70640 1 -- 172.21.15.49:0/1311377520 learned_addr learned my addr 172.21.15.49:0/1311377520 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:01.841 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.841+0000 7f4a05371640 1 --2- 172.21.15.49:0/1311377520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e324d6e0 0x55f9e3246730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.842+0000 7f4a05371640 1 -- 172.21.15.49:0/1311377520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e3247b80 msgr2=0x55f9e32436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:01.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.842+0000 7f4a05371640 1 --2- 172.21.15.49:0/1311377520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e3247b80 0x55f9e32436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.842+0000 7f4a05371640 1 -- 172.21.15.49:0/1311377520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 msgr2=0x55f9e3259560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:01.842 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.842+0000 7f4a05371640 1 --2- 172.21.15.49:0/1311377520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 0x55f9e3259560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.842+0000 7f4a05371640 1 -- 172.21.15.49:0/1311377520 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9e324a050 con 0x55f9e324d6e0 2024-03-18T01:00:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.842+0000 7f4a05371640 1 --2- 172.21.15.49:0/1311377520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e324d6e0 0x55f9e3246730 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f49f8005c40 tx=0x7f49f802d9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c500ca3ed9bbcc21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.843+0000 7f49e67fc640 1 -- 172.21.15.49:0/1311377520 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f8058030 con 0x55f9e324d6e0 2024-03-18T01:00:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.843+0000 7f49e67fc640 1 -- 172.21.15.49:0/1311377520 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f804a070 con 0x55f9e324d6e0 2024-03-18T01:00:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.844+0000 7f4a099c0740 1 -- 172.21.15.49:0/1311377520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e324d6e0 msgr2=0x55f9e3246730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:01.843 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.844+0000 7f4a099c0740 1 --2- 172.21.15.49:0/1311377520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e324d6e0 0x55f9e3246730 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f49f8005c40 tx=0x7f49f802d9c0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.844+0000 7f4a099c0740 1 -- 172.21.15.49:0/1311377520 shutdown_connections 2024-03-18T01:00:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.844+0000 7f4a099c0740 1 --2- 172.21.15.49:0/1311377520 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e324d6e0 0x55f9e3246730 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.844+0000 7f4a099c0740 1 --2- 172.21.15.49:0/1311377520 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3248570 0x55f9e3259560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.844+0000 7f4a099c0740 1 --2- 172.21.15.49:0/1311377520 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e3247b80 0x55f9e32436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.844 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.844+0000 7f4a099c0740 1 -- 172.21.15.49:0/1311377520 >> 172.21.15.49:0/1311377520 conn(0x55f9e320fc50 msgr2=0x55f9e3242fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:01.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.847+0000 7f4a099c0740 1 -- 172.21.15.49:0/1311377520 shutdown_connections 2024-03-18T01:00:01.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.848+0000 7f4a099c0740 1 -- 172.21.15.49:0/1311377520 wait complete. 2024-03-18T01:00:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.848+0000 7f4a099c0740 1 -- start start 2024-03-18T01:00:01.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.848+0000 7f4a099c0740 1 Processor -- start 2024-03-18T01:00:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3247b80 0x55f9e3258ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e32593e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f4a099c0740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e32574b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f4a099c0740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec00fab0 con 0x55f9e3247b80 2024-03-18T01:00:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f4a099c0740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec00f3e0 con 0x55f9e324d6e0 2024-03-18T01:00:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f4a099c0740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec00c620 con 0x55f9e3248570 2024-03-18T01:00:01.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f4a099c0740 10 client.0 _ll_register_callbacks cb 0x7f49f0097bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:01.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f49fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e32593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f49fffff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e32593e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60198/0 (socket says 172.21.15.49:60198) 2024-03-18T01:00:01.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f49fffff640 1 -- 172.21.15.49:0/2115897812 learned_addr learned my addr 172.21.15.49:0/2115897812 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:01.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.849+0000 7f4a05371640 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e32574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.850+0000 7f49fffff640 1 -- 172.21.15.49:0/2115897812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 msgr2=0x55f9e32574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.850+0000 7f49fffff640 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e32574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.850+0000 7f49fffff640 1 -- 172.21.15.49:0/2115897812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3247b80 msgr2=0x55f9e3258ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.850+0000 7f49fffff640 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3247b80 0x55f9e3258ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.850+0000 7f49fffff640 1 -- 172.21.15.49:0/2115897812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9e3257ce0 con 0x55f9e3248570 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.850+0000 7f49fffff640 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e32593e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f49f0015f00 tx=0x7f49f009c180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a101a8e6e3eb7878 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.851+0000 7f49dffff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.851+0000 7f49dffff640 1 -- 172.21.15.49:0/2115897812 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f00b3070 con 0x55f9e3248570 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.851+0000 7f4a099c0740 1 -- 172.21.15.49:0/2115897812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49f804a070 con 0x55f9e3248570 2024-03-18T01:00:01.851 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.851+0000 7f49dffff640 1 -- 172.21.15.49:0/2115897812 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f49f009af70 con 0x55f9e3248570 2024-03-18T01:00:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.851+0000 7f49dffff640 1 -- 172.21.15.49:0/2115897812 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f49f00a40c0 con 0x55f9e3248570 2024-03-18T01:00:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.853+0000 7f4a099c0740 10 client.27070 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:01.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.853+0000 7f4a099c0740 1 -- 172.21.15.49:0/2115897812 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f9e32c44c0 con 0x55f9e3248570 2024-03-18T01:00:01.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.856+0000 7f49dffff640 1 -- 172.21.15.49:0/2115897812 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49f804a070 con 0x55f9e3248570 2024-03-18T01:00:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.856+0000 7f49dffff640 1 -- 172.21.15.49:0/2115897812 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f49f00a0a80 con 0x55f9e3248570 2024-03-18T01:00:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.856+0000 7f4a099c0740 10 client.27070 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.856+0000 7f4a099c0740 10 client.27070 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.856+0000 7f4a099c0740 20 client.27070 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.856+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f9e32c4f70 0x55f9e32c7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.856+0000 7f4a099c0740 4 client.27070 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T01:00:01.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.856+0000 7f4a099c0740 1 -- 172.21.15.49:0/2115897812 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f9e31b3960 con 0x55f9e32c4f70 2024-03-18T01:00:01.862 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.862+0000 7f4a04b70640 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f9e32c4f70 0x55f9e32c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.868+0000 7f4a04b70640 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f9e32c4f70 0x55f9e32c7390 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb66442c207599d8 server_cookie=6ed316fb8dc9111f in_seq=0 out_seq=0 2024-03-18T01:00:01.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.869+0000 7f49dffff640 10 client.27070 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:01.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.877+0000 7f49dffff640 1 -- 172.21.15.49:0/2115897812 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f9e31b3960 con 0x55f9e32c4f70 2024-03-18T01:00:01.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.877+0000 7f49dffff640 10 client.27070 handle_command_reply: tid=0 2024-03-18T01:00:01.877 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.877+0000 7f4a099c0740 1 client.27070 shutdown 2024-03-18T01:00:01.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.885+0000 7f4a099c0740 1 -- 172.21.15.49:0/2115897812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 msgr2=0x55f9e32593e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:01.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.885+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e32593e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f49f0015f00 tx=0x7f49f009c180 comp rx=0 tx=0).stop 2024-03-18T01:00:01.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.887+0000 7f4a099c0740 1 -- 172.21.15.49:0/2115897812 shutdown_connections 2024-03-18T01:00:01.887 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.887+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55f9e32c4f70 0x55f9e32c7390 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.887+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f9e324d6e0 0x55f9e32574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.887+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f9e3248570 0x55f9e32593e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.887+0000 7f4a099c0740 1 --2- 172.21.15.49:0/2115897812 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f9e3247b80 0x55f9e3258ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.887+0000 7f4a099c0740 1 -- 172.21.15.49:0/2115897812 >> 172.21.15.49:0/2115897812 conn(0x55f9e320fc50 msgr2=0x55f9e3246730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.888+0000 7f49dffff640 0 client.27070 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:01.888 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.888+0000 7f4a099c0740 1 -- 172.21.15.49:0/2115897812 shutdown_connections 2024-03-18T01:00:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.890+0000 7f4a099c0740 1 -- 172.21.15.49:0/2115897812 wait complete. 2024-03-18T01:00:01.891 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.891+0000 7f4a099c0740 20 client.27070 trim_cache size 0 max 16384 2024-03-18T01:00:01.894 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:00:01.894 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T01:00:01.894 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "cc5d7be3-a24e-4c1b-a8f7-5c208fb563b0", 2024-03-18T01:00:01.894 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T01:00:01.894 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:00:01.897 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.898+0000 7f4a06dd4640 1 -- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49e80a2980 msgr2=0x7f49e80a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.898+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49e80a2980 0x7f49e80a4e40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f49f8004af0 tx=0x7f49f802d550 comp rx=0 tx=0).stop 2024-03-18T01:00:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.898+0000 7f4a06dd4640 1 -- 172.21.15.49:0/3995312384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010b250 msgr2=0x7f4a001af500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:01.898 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.898+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010b250 0x7f4a001af500 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f49f000daf0 tx=0x7f49f000af60 comp rx=0 tx=0).stop 2024-03-18T01:00:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.900+0000 7f4a06dd4640 1 -- 172.21.15.49:0/3995312384 shutdown_connections 2024-03-18T01:00:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.900+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f49e80a2980 0x7f49e80a4e40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.900+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a0010dda0 0x7f4a001b6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.900+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a0010b250 0x7f4a001af500 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.900+0000 7f4a06dd4640 1 --2- 172.21.15.49:0/3995312384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a00108870 0x7f4a001aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:01.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.900+0000 7f4a06dd4640 1 -- 172.21.15.49:0/3995312384 >> 172.21.15.49:0/3995312384 conn(0x7f4a0006eb40 msgr2=0x7f4a00109e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:01.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.905+0000 7f4a06dd4640 1 -- 172.21.15.49:0/3995312384 shutdown_connections 2024-03-18T01:00:01.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:01.906+0000 7f4a06dd4640 1 -- 172.21.15.49:0/3995312384 wait complete. 2024-03-18T01:00:01.938 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:00:01.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:00:02.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:01 smithi049 ceph-mon[32130]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 171 MiB/s rd, 520 MiB/s wr, 5.77k op/s 2024-03-18T01:00:02.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:01 smithi107 ceph-mon[36270]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 171 MiB/s rd, 520 MiB/s wr, 5.77k op/s 2024-03-18T01:00:02.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:01 smithi106 ceph-mon[35899]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 171 MiB/s rd, 520 MiB/s wr, 5.77k op/s 2024-03-18T01:00:02.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.227+0000 7fb77135b640 1 -- 172.21.15.49:0/2018938795 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb76c10db00 msgr2=0x7fb76c10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:02.228 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.227+0000 7fb77135b640 1 --2- 172.21.15.49:0/2018938795 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb76c10db00 0x7fb76c10df80 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fb760009900 tx=0x7fb76002bc60 comp rx=0 tx=0).stop 2024-03-18T01:00:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.229+0000 7fb77135b640 1 -- 172.21.15.49:0/2018938795 shutdown_connections 2024-03-18T01:00:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.229+0000 7fb77135b640 1 --2- 172.21.15.49:0/2018938795 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb76c10b150 0x7fb76c10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.229+0000 7fb77135b640 1 --2- 172.21.15.49:0/2018938795 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb76c10db00 0x7fb76c10df80 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.229+0000 7fb77135b640 1 --2- 172.21.15.49:0/2018938795 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb76c1087e0 0x7fb76c10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.229+0000 7fb77135b640 1 -- 172.21.15.49:0/2018938795 >> 172.21.15.49:0/2018938795 conn(0x7fb76c06e640 msgr2=0x7fb76c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.230+0000 7fb77135b640 1 -- 172.21.15.49:0/2018938795 shutdown_connections 2024-03-18T01:00:02.233 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.232+0000 7fb77135b640 1 -- 172.21.15.49:0/2018938795 wait complete. 2024-03-18T01:00:02.234 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.233+0000 7fb77135b640 1 Processor -- start 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.238+0000 7fb77135b640 1 -- start start 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.238+0000 7fb77135b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb76c1087e0 0x7fb76c117df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.239+0000 7fb77135b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb76c10b150 0x7fb76c118330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.239+0000 7fb77135b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb76c10db00 0x7fb76c119370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.239+0000 7fb77135b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c110520 con 0x7fb76c10db00 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.239+0000 7fb77135b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1103a0 con 0x7fb76c10b150 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.239+0000 7fb77135b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1106a0 con 0x7fb76c1087e0 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.240+0000 7fb76affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb76c1087e0 0x7fb76c117df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.240+0000 7fb76affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb76c1087e0 0x7fb76c117df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:60214/0 (socket says 172.21.15.49:60214) 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.240+0000 7fb76affd640 1 -- 172.21.15.49:0/2430895485 learned_addr learned my addr 172.21.15.49:0/2430895485 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:02.241 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.241+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb76c10db00 0x7fb76c119370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.241+0000 7fb76affd640 1 -- 172.21.15.49:0/2430895485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb76c10b150 msgr2=0x7fb76c118330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:02.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.241+0000 7fb76affd640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb76c10b150 0x7fb76c118330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.241+0000 7fb76affd640 1 -- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb76c10db00 msgr2=0x7fb76c119370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:02.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.241+0000 7fb76affd640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb76c10db00 0x7fb76c119370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.242 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.241+0000 7fb76affd640 1 -- 172.21.15.49:0/2430895485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76c1199e0 con 0x7fb76c1087e0 2024-03-18T01:00:02.243 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.243+0000 7fb76affd640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb76c1087e0 0x7fb76c117df0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb76400cce0 tx=0x7fb76400a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:02.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.244+0000 7fb753fff640 1 -- 172.21.15.49:0/2430895485 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb764004b20 con 0x7fb76c1087e0 2024-03-18T01:00:02.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.244+0000 7fb753fff640 1 -- 172.21.15.49:0/2430895485 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb764004cc0 con 0x7fb76c1087e0 2024-03-18T01:00:02.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.244+0000 7fb77135b640 1 -- 172.21.15.49:0/2430895485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb76c119cd0 con 0x7fb76c1087e0 2024-03-18T01:00:02.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.244+0000 7fb753fff640 1 -- 172.21.15.49:0/2430895485 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76400f7b0 con 0x7fb76c1087e0 2024-03-18T01:00:02.244 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.244+0000 7fb77135b640 1 -- 172.21.15.49:0/2430895485 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb76c11a220 con 0x7fb76c1087e0 2024-03-18T01:00:02.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.254+0000 7fb774023740 1 Processor -- start 2024-03-18T01:00:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.255+0000 7fb753fff640 1 -- 172.21.15.49:0/2430895485 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb764005600 con 0x7fb76c1087e0 2024-03-18T01:00:02.255 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.256+0000 7fb774023740 1 -- start start 2024-03-18T01:00:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.256+0000 7fb774023740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc5216170 0x55afc5216590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.256 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.257+0000 7fb753fff640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7540a2a10 0x7fb7540a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.257+0000 7fb774023740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc521b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.257+0000 7fb774023740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc521bcd0 0x55afc521e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.257+0000 7fb753fff640 1 -- 172.21.15.49:0/2430895485 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb7640ceb40 con 0x7fb76c1087e0 2024-03-18T01:00:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.257+0000 7fb774023740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55afc517b500 con 0x55afc5216b60 2024-03-18T01:00:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.257+0000 7fb774023740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55afc5123be0 con 0x55afc5216170 2024-03-18T01:00:02.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.257+0000 7fb774023740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55afc51775a0 con 0x55afc521bcd0 2024-03-18T01:00:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.258+0000 7fb76b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc5216170 0x55afc5216590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.258+0000 7fb76affd640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7540a2a10 0x7fb7540a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.260+0000 7fb76a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc521b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.260+0000 7fb76a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc521b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41386/0 (socket says 172.21.15.49:41386) 2024-03-18T01:00:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.260+0000 7fb76a7fc640 1 -- 172.21.15.49:0/935187472 learned_addr learned my addr 172.21.15.49:0/935187472 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:02.260 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.261+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/935187472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc521bcd0 0x55afc521e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.261+0000 7fb76a7fc640 1 -- 172.21.15.49:0/935187472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc521bcd0 msgr2=0x55afc521e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.261+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/935187472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc521bcd0 0x55afc521e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.261+0000 7fb76a7fc640 1 -- 172.21.15.49:0/935187472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc5216170 msgr2=0x55afc5216590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.261+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/935187472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc5216170 0x55afc5216590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.261 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.261+0000 7fb76a7fc640 1 -- 172.21.15.49:0/935187472 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afc51374b0 con 0x55afc5216b60 2024-03-18T01:00:02.262 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.262+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/935187472 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc521b660 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0097b0 tx=0x7fb75c009c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff802d4d723f991e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.263+0000 7fb7517fa640 1 -- 172.21.15.49:0/935187472 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c015020 con 0x55afc5216b60 2024-03-18T01:00:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.263+0000 7fb7517fa640 1 -- 172.21.15.49:0/935187472 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb75c011420 con 0x55afc5216b60 2024-03-18T01:00:02.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.263+0000 7fb7517fa640 1 -- 172.21.15.49:0/935187472 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c0105e0 con 0x55afc5216b60 2024-03-18T01:00:02.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.265+0000 7fb774023740 1 -- 172.21.15.49:0/935187472 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 msgr2=0x55afc521b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:02.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.265+0000 7fb774023740 1 --2- 172.21.15.49:0/935187472 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc521b660 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0097b0 tx=0x7fb75c009c70 comp rx=0 tx=0).stop 2024-03-18T01:00:02.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.265+0000 7fb774023740 1 -- 172.21.15.49:0/935187472 shutdown_connections 2024-03-18T01:00:02.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.265+0000 7fb774023740 1 --2- 172.21.15.49:0/935187472 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc521bcd0 0x55afc521e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.265+0000 7fb774023740 1 --2- 172.21.15.49:0/935187472 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc521b660 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.265+0000 7fb774023740 1 --2- 172.21.15.49:0/935187472 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc5216170 0x55afc5216590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.265+0000 7fb774023740 1 -- 172.21.15.49:0/935187472 >> 172.21.15.49:0/935187472 conn(0x55afc52116e0 msgr2=0x55afc5211ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:02.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.266+0000 7fb76affd640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7540a2a10 0x7fb7540a4ed0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb764005950 tx=0x7fb7640059a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:02.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.269+0000 7fb774023740 1 -- 172.21.15.49:0/935187472 shutdown_connections 2024-03-18T01:00:02.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.270+0000 7fb774023740 1 -- 172.21.15.49:0/935187472 wait complete. 2024-03-18T01:00:02.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.271+0000 7fb774023740 1 -- start start 2024-03-18T01:00:02.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.271+0000 7fb774023740 1 Processor -- start 2024-03-18T01:00:02.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb774023740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc52206e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb774023740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 0x55afc5220c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb774023740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5227c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb774023740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55afc521ec80 con 0x55afc5216170 2024-03-18T01:00:02.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb774023740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55afc5137160 con 0x55afc521bcd0 2024-03-18T01:00:02.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb774023740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55afc521ee00 con 0x55afc5216b60 2024-03-18T01:00:02.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb774023740 10 client.0 _ll_register_callbacks cb 0x55afc511b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:02.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb76b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc52206e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb76b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc52206e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41402/0 (socket says 172.21.15.49:41402) 2024-03-18T01:00:02.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.272+0000 7fb76b7fe640 1 -- 172.21.15.49:0/3349401650 learned_addr learned my addr 172.21.15.49:0/3349401650 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:02.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.273+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5227c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.273+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 0x55afc5220c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.273+0000 7fb76b7fe640 1 -- 172.21.15.49:0/3349401650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 msgr2=0x55afc5220c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:02.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.273+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 0x55afc5220c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.273+0000 7fb76b7fe640 1 -- 172.21.15.49:0/3349401650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 msgr2=0x55afc5227c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:02.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.273+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5227c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.273+0000 7fb76b7fe640 1 -- 172.21.15.49:0/3349401650 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afc4f597d0 con 0x55afc5216170 2024-03-18T01:00:02.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.274+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc52206e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb76002c910 tx=0x7fb760005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9603cbdc4e6d28a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:02.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.274+0000 7fb74eff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:02.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.275+0000 7fb74eff5640 1 -- 172.21.15.49:0/3349401650 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760004220 con 0x55afc5216170 2024-03-18T01:00:02.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.275+0000 7fb774023740 1 -- 172.21.15.49:0/3349401650 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb75c011420 con 0x55afc5216170 2024-03-18T01:00:02.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.275+0000 7fb74eff5640 1 -- 172.21.15.49:0/3349401650 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb7600043c0 con 0x55afc5216170 2024-03-18T01:00:02.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.275+0000 7fb74eff5640 1 -- 172.21.15.49:0/3349401650 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb760005000 con 0x55afc5216170 2024-03-18T01:00:02.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.275+0000 7fb774023740 10 client.17856 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:02.276 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.275+0000 7fb774023740 1 -- 172.21.15.49:0/3349401650 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55afc4e8f3f0 con 0x55afc5216170 2024-03-18T01:00:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.279+0000 7fb74eff5640 1 -- 172.21.15.49:0/3349401650 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb75c011420 con 0x55afc5216170 2024-03-18T01:00:02.280 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.279+0000 7fb74eff5640 1 -- 172.21.15.49:0/3349401650 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fb760041460 con 0x55afc5216170 2024-03-18T01:00:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.279+0000 7fb774023740 10 client.17856 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.279+0000 7fb774023740 10 client.17856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.279+0000 7fb774023740 20 client.17856 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.280+0000 7fb774023740 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55afc522c1f0 0x55afc522e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.280+0000 7fb774023740 4 client.17856 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:02.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.280+0000 7fb774023740 1 -- 172.21.15.49:0/3349401650 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afc5188a40 con 0x55afc522c1f0 2024-03-18T01:00:02.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.284+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55afc522c1f0 0x55afc522e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.291+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55afc522c1f0 0x55afc522e660 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=397df39f9213c8e0 server_cookie=47b07dfd097e3244 in_seq=0 out_seq=0 2024-03-18T01:00:02.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.291+0000 7fb74eff5640 10 client.17856 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:02.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.296+0000 7fb74eff5640 1 -- 172.21.15.49:0/3349401650 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55afc5188a40 con 0x55afc522c1f0 2024-03-18T01:00:02.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.296+0000 7fb74eff5640 10 client.17856 handle_command_reply: tid=0 2024-03-18T01:00:02.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.296+0000 7fb774023740 1 client.17856 shutdown 2024-03-18T01:00:02.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.300+0000 7fb774023740 1 -- 172.21.15.49:0/3349401650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 msgr2=0x55afc52206e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:02.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.300+0000 7fb774023740 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc52206e0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb76002c910 tx=0x7fb760005fd0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.302 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.302+0000 7fb774023740 1 -- 172.21.15.49:0/3349401650 shutdown_connections 2024-03-18T01:00:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.303+0000 7fb774023740 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55afc522c1f0 0x55afc522e660 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.303+0000 7fb774023740 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5227c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.303+0000 7fb774023740 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 0x55afc5220c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.303+0000 7fb774023740 1 --2- 172.21.15.49:0/3349401650 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc52206e0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.303+0000 7fb774023740 1 -- 172.21.15.49:0/3349401650 >> 172.21.15.49:0/3349401650 conn(0x55afc52116e0 msgr2=0x55afc5210900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.303+0000 7fb74eff5640 0 client.17856 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.303+0000 7fb774023740 1 -- 172.21.15.49:0/3349401650 shutdown_connections 2024-03-18T01:00:02.303 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.303+0000 7fb774023740 1 -- 172.21.15.49:0/3349401650 wait complete. 2024-03-18T01:00:02.306 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.304+0000 7fb774023740 20 client.17856 trim_cache size 0 max 16384 2024-03-18T01:00:02.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.349+0000 7fb774023740 1 Processor -- start 2024-03-18T01:00:02.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.349+0000 7fb774023740 1 -- start start 2024-03-18T01:00:02.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.350+0000 7fb774023740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc5220a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.350+0000 7fb774023740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 0x55afc5227bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.351+0000 7fb76b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc5220a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.351+0000 7fb76b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc5220a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41414/0 (socket says 172.21.15.49:41414) 2024-03-18T01:00:02.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.351+0000 7fb76b7fe640 1 -- 172.21.15.49:0/697697384 learned_addr learned my addr 172.21.15.49:0/697697384 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:02.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.351+0000 7fb774023740 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5210070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.351+0000 7fb774023740 1 -- 172.21.15.49:0/697697384 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1c0350 con 0x55afc5216170 2024-03-18T01:00:02.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.351+0000 7fb774023740 1 -- 172.21.15.49:0/697697384 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1c1ec0 con 0x55afc521bcd0 2024-03-18T01:00:02.351 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.351+0000 7fb774023740 1 -- 172.21.15.49:0/697697384 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1c0a20 con 0x55afc5216b60 2024-03-18T01:00:02.353 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.354+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 0x55afc5227bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.355+0000 7fb76b7fe640 1 -- 172.21.15.49:0/697697384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 msgr2=0x55afc5227bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:02.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.356+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 0x55afc5227bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.356+0000 7fb76b7fe640 1 -- 172.21.15.49:0/697697384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 msgr2=0x55afc5210070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:02.355 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.356+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5210070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.356+0000 7fb76b7fe640 1 -- 172.21.15.49:0/697697384 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afc4e4f0f0 con 0x55afc5216170 2024-03-18T01:00:02.356 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.356+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 0x55afc5227bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:00:02.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.356+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc5220a00 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb760009850 tx=0x7fb76002bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2623b4454f8788 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:02.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.357+0000 7fb750ff9640 1 -- 172.21.15.49:0/697697384 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760041d10 con 0x55afc5216170 2024-03-18T01:00:02.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.357+0000 7fb750ff9640 1 -- 172.21.15.49:0/697697384 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb760041460 con 0x55afc5216170 2024-03-18T01:00:02.357 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.357+0000 7fb750ff9640 1 -- 172.21.15.49:0/697697384 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7600035f0 con 0x55afc5216170 2024-03-18T01:00:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.358+0000 7fb774023740 1 -- 172.21.15.49:0/697697384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 msgr2=0x55afc5220a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:02.360 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.359+0000 7fb774023740 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc5220a00 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fb760009850 tx=0x7fb76002bc60 comp rx=0 tx=0).stop 2024-03-18T01:00:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.362+0000 7fb774023740 1 -- 172.21.15.49:0/697697384 shutdown_connections 2024-03-18T01:00:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.362+0000 7fb774023740 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5210070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.362+0000 7fb774023740 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216b60 0x55afc5227bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.362+0000 7fb774023740 1 --2- 172.21.15.49:0/697697384 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216170 0x55afc5220a00 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.362+0000 7fb774023740 1 -- 172.21.15.49:0/697697384 >> 172.21.15.49:0/697697384 conn(0x55afc52116e0 msgr2=0x55afc52202f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:02.363 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.363+0000 7fb774023740 1 -- 172.21.15.49:0/697697384 shutdown_connections 2024-03-18T01:00:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.363+0000 7fb774023740 1 -- 172.21.15.49:0/697697384 wait complete. 2024-03-18T01:00:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.364+0000 7fb774023740 1 -- start start 2024-03-18T01:00:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.364+0000 7fb774023740 1 Processor -- start 2024-03-18T01:00:02.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.364+0000 7fb774023740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216170 0x55afc5225550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.364+0000 7fb774023740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc5225a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.364+0000 7fb774023740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5226aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.364+0000 7fb774023740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1c17c0 con 0x55afc5216b60 2024-03-18T01:00:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.364+0000 7fb774023740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1c10f0 con 0x55afc521bcd0 2024-03-18T01:00:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.364+0000 7fb774023740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb76c1be330 con 0x55afc5216170 2024-03-18T01:00:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.364+0000 7fb774023740 10 client.0 _ll_register_callbacks cb 0x7fb760009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.365+0000 7fb76a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc5225a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.365+0000 7fb76a7fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc5225a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:41416/0 (socket says 172.21.15.49:41416) 2024-03-18T01:00:02.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.365+0000 7fb76a7fc640 1 -- 172.21.15.49:0/2262325309 learned_addr learned my addr 172.21.15.49:0/2262325309 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.365+0000 7fb76a7fc640 1 -- 172.21.15.49:0/2262325309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216170 msgr2=0x55afc5225550 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.365+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216170 0x55afc5225550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.365+0000 7fb76a7fc640 1 -- 172.21.15.49:0/2262325309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 msgr2=0x55afc5226aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.365+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5226aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.365+0000 7fb76a7fc640 1 -- 172.21.15.49:0/2262325309 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afc51429d0 con 0x55afc5216b60 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.366+0000 7fb76a7fc640 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc5225a90 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0097b0 tx=0x7fb75c006e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc36a3d55da46a77 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.366+0000 7fb74eff5640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.366+0000 7fb74eff5640 1 -- 172.21.15.49:0/2262325309 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75c015040 con 0x55afc5216b60 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.366+0000 7fb74eff5640 1 -- 172.21.15.49:0/2262325309 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb75c00bd80 con 0x55afc5216b60 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.366+0000 7fb774023740 1 -- 172.21.15.49:0/2262325309 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb760041460 con 0x55afc5216b60 2024-03-18T01:00:02.367 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.366+0000 7fb74eff5640 1 -- 172.21.15.49:0/2262325309 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb75c0093a0 con 0x55afc5216b60 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb774023740 10 client.17862 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb774023740 1 -- 172.21.15.49:0/2262325309 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55afc52926c0 con 0x55afc5216b60 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb74eff5640 1 -- 172.21.15.49:0/2262325309 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb760041460 con 0x55afc5216b60 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb74eff5640 1 -- 172.21.15.49:0/2262325309 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fb75c02d030 con 0x55afc5216b60 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb774023740 10 client.17862 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb774023740 10 client.17862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb774023740 20 client.17862 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb774023740 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55afc5293110 0x55afc5295530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb774023740 4 client.17862 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:00:02.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.367+0000 7fb774023740 1 -- 172.21.15.49:0/2262325309 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55afc5182d90 con 0x55afc5293110 2024-03-18T01:00:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.368+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55afc5293110 0x55afc5295530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.369+0000 7fb76b7fe640 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55afc5293110 0x55afc5295530 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b7177d47f1ebeba server_cookie=b65ec90738837c in_seq=0 out_seq=0 2024-03-18T01:00:02.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.369+0000 7fb74eff5640 10 client.17862 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:02.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb74eff5640 1 -- 172.21.15.49:0/2262325309 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55afc5182d90 con 0x55afc5293110 2024-03-18T01:00:02.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb74eff5640 10 client.17862 handle_command_reply: tid=0 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb774023740 1 client.17862 shutdown 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb774023740 1 -- 172.21.15.49:0/2262325309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 msgr2=0x55afc5225a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb774023740 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc5225a90 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb75c0097b0 tx=0x7fb75c006e10 comp rx=0 tx=0).stop 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb774023740 1 -- 172.21.15.49:0/2262325309 shutdown_connections 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb774023740 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55afc5293110 0x55afc5295530 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb774023740 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55afc521bcd0 0x55afc5226aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb774023740 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55afc5216b60 0x55afc5225a90 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb74eff5640 0 client.17862 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb774023740 1 --2- 172.21.15.49:0/2262325309 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55afc5216170 0x55afc5225550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.370+0000 7fb774023740 1 -- 172.21.15.49:0/2262325309 >> 172.21.15.49:0/2262325309 conn(0x55afc52116e0 msgr2=0x55afc5214940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.371+0000 7fb774023740 1 -- 172.21.15.49:0/2262325309 shutdown_connections 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.371+0000 7fb774023740 1 -- 172.21.15.49:0/2262325309 wait complete. 2024-03-18T01:00:02.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.371+0000 7fb774023740 20 client.17862 trim_cache size 0 max 16384 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout: "cc5d7be3-a24e-4c1b-a8f7-5c208fb563b0": { 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "cc5d7be3-a24e-4c1b-a8f7-5c208fb563b0", 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:00:02.372 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:00:02.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.371+0000 7fb77135b640 1 -- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7540a2a10 msgr2=0x7fb7540a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:02.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.371+0000 7fb77135b640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7540a2a10 0x7fb7540a4ed0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb764005950 tx=0x7fb7640059a0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.372+0000 7fb77135b640 1 -- 172.21.15.49:0/2430895485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb76c1087e0 msgr2=0x7fb76c117df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:02.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.372+0000 7fb77135b640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb76c1087e0 0x7fb76c117df0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb76400cce0 tx=0x7fb76400a620 comp rx=0 tx=0).stop 2024-03-18T01:00:02.373 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.372+0000 7fb77135b640 1 -- 172.21.15.49:0/2430895485 shutdown_connections 2024-03-18T01:00:02.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.372+0000 7fb77135b640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb7540a2a10 0x7fb7540a4ed0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.372+0000 7fb77135b640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb76c10b150 0x7fb76c118330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.372+0000 7fb77135b640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb76c10db00 0x7fb76c119370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.372+0000 7fb77135b640 1 --2- 172.21.15.49:0/2430895485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb76c1087e0 0x7fb76c117df0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:02.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.372+0000 7fb77135b640 1 -- 172.21.15.49:0/2430895485 >> 172.21.15.49:0/2430895485 conn(0x7fb76c06e640 msgr2=0x7fb76c108f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:02.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.373+0000 7fb77135b640 1 -- 172.21.15.49:0/2430895485 shutdown_connections 2024-03-18T01:00:02.374 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:02.373+0000 7fb77135b640 1 -- 172.21.15.49:0/2430895485 wait complete. 2024-03-18T01:00:02.390 INFO:tasks.cephfs.filesystem:scrub status for tag:cc5d7be3-a24e-4c1b-a8f7-5c208fb563b0 - {'path': '/', 'tag': 'cc5d7be3-a24e-4c1b-a8f7-5c208fb563b0', 'options': 'recursive,force'} 2024-03-18T01:00:02.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:02 smithi049 ceph-mon[32130]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 MiB/s rd, 685 MiB/s wr, 7.24k op/s 2024-03-18T01:00:02.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:02 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T01:00:02.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:02 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:02.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:02 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T01:00:02.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:02 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T01:00:02.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:02 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:02.797 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:02 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:03.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:02 smithi107 ceph-mon[36270]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 MiB/s rd, 685 MiB/s wr, 7.24k op/s 2024-03-18T01:00:03.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:02 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T01:00:03.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:02 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:03.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:02 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T01:00:03.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:02 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T01:00:03.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:02 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:03.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:02 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:03.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:02 smithi106 ceph-mon[35899]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 MiB/s rd, 685 MiB/s wr, 7.24k op/s 2024-03-18T01:00:03.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:02 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T01:00:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:02 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:02 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T01:00:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:02 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T01:00:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:02 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:03.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:02 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:05.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:04 smithi107 ceph-mon[36270]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 650 MiB/s wr, 6.41k op/s 2024-03-18T01:00:05.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:04 smithi106 ceph-mon[35899]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 650 MiB/s wr, 6.41k op/s 2024-03-18T01:00:05.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:04 smithi049 ceph-mon[32130]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s rd, 650 MiB/s wr, 6.41k op/s 2024-03-18T01:00:07.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:06 smithi049 ceph-mon[32130]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s rd, 663 MiB/s wr, 6.23k op/s 2024-03-18T01:00:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:06 smithi107 ceph-mon[36270]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s rd, 663 MiB/s wr, 6.23k op/s 2024-03-18T01:00:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:06 smithi106 ceph-mon[35899]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s rd, 663 MiB/s wr, 6.23k op/s 2024-03-18T01:00:09.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:09 smithi049 ceph-mon[32130]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s rd, 730 MiB/s wr, 5.85k op/s 2024-03-18T01:00:09.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:09 smithi107 ceph-mon[36270]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s rd, 730 MiB/s wr, 5.85k op/s 2024-03-18T01:00:09.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:09 smithi106 ceph-mon[35899]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 MiB/s rd, 730 MiB/s wr, 5.85k op/s 2024-03-18T01:00:11.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:11 smithi049 ceph-mon[32130]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 658 MiB/s wr, 4.92k op/s 2024-03-18T01:00:11.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:11 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T01:00:11.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:11 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T01:00:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:11 smithi107 ceph-mon[36270]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 658 MiB/s wr, 4.92k op/s 2024-03-18T01:00:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:11 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T01:00:11.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:11 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T01:00:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:11 smithi106 ceph-mon[35899]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 MiB/s rd, 658 MiB/s wr, 4.92k op/s 2024-03-18T01:00:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:11 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T01:00:11.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:11 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T01:00:12.005 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.005+0000 7febac1df640 1 -- 172.21.15.49:0/2605560568 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba41022b0 msgr2=0x7feba4108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:12.006 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.005+0000 7febac1df640 1 --2- 172.21.15.49:0/2605560568 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba41022b0 0x7feba4108b40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009ca0 tx=0x7feb9c02c0f0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.008 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 -- 172.21.15.49:0/2605560568 shutdown_connections 2024-03-18T01:00:12.008 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 --2- 172.21.15.49:0/2605560568 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba41018f0 0x7feba4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.008 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 --2- 172.21.15.49:0/2605560568 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba41022b0 0x7feba4108b40 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.008 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 --2- 172.21.15.49:0/2605560568 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba4100700 0x7feba4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.008 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 -- 172.21.15.49:0/2605560568 >> 172.21.15.49:0/2605560568 conn(0x7feba40fbc70 msgr2=0x7feba40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 -- 172.21.15.49:0/2605560568 shutdown_connections 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 -- 172.21.15.49:0/2605560568 wait complete. 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 Processor -- start 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 -- start start 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.006+0000 7febac1df640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4100700 0x7feba4199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7febac1df640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba41018f0 0x7feba419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7febac1df640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba41022b0 0x7feba41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7febac1df640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7feba410b280 con 0x7feba4100700 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7febac1df640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7feba410b100 con 0x7feba41018f0 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7febac1df640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feba410b400 con 0x7feba41022b0 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba9f7b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4100700 0x7feba4199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba977a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba41018f0 0x7feba419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.009 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7febaa77c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba41022b0 0x7feba41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba9f7b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4100700 0x7feba4199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53940/0 (socket says 172.21.15.49:53940) 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba9f7b640 1 -- 172.21.15.49:0/2037278767 learned_addr learned my addr 172.21.15.49:0/2037278767 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba9f7b640 1 -- 172.21.15.49:0/2037278767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba41022b0 msgr2=0x7feba41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba9f7b640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba41022b0 0x7feba41a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba9f7b640 1 -- 172.21.15.49:0/2037278767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba41018f0 msgr2=0x7feba419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba9f7b640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba41018f0 0x7feba419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba9f7b640 1 -- 172.21.15.49:0/2037278767 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba41a1810 con 0x7feba4100700 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba977a640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba41018f0 0x7feba419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feba9f7b640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4100700 0x7feba4199bf0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7feba000b3c0 tx=0x7feba0009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.007+0000 7feb9affd640 1 -- 172.21.15.49:0/2037278767 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0004230 con 0x7feba4100700 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.008+0000 7feb9affd640 1 -- 172.21.15.49:0/2037278767 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feba00043d0 con 0x7feba4100700 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.008+0000 7febac1df640 1 -- 172.21.15.49:0/2037278767 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba41a1b00 con 0x7feba4100700 2024-03-18T01:00:12.010 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.008+0000 7feb9affd640 1 -- 172.21.15.49:0/2037278767 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba0003cd0 con 0x7feba4100700 2024-03-18T01:00:12.011 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.008+0000 7febac1df640 1 -- 172.21.15.49:0/2037278767 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba41a2000 con 0x7feba4100700 2024-03-18T01:00:12.017 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.010+0000 7febaedc7740 1 Processor -- start 2024-03-18T01:00:12.017 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.010+0000 7feb9affd640 1 -- 172.21.15.49:0/2037278767 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7feba000e6a0 con 0x7feba4100700 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7febaedc7740 1 -- start start 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7febaedc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db2575760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7febaedc7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7febaedc7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db257d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7febaedc7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558db24daa40 con 0x558db2575d30 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7febaedc7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558db2483920 con 0x558db257aea0 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7febaedc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558db24d6ae0 con 0x558db2575340 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba9f7b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7febaa77c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db2575760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba9f7b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53942/0 (socket says 172.21.15.49:53942) 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba9f7b640 1 -- 172.21.15.49:0/252589540 learned_addr learned my addr 172.21.15.49:0/252589540 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba977a640 1 --2- 172.21.15.49:0/252589540 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db257d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba9f7b640 1 -- 172.21.15.49:0/252589540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 msgr2=0x558db2575760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba9f7b640 1 --2- 172.21.15.49:0/252589540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db2575760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba9f7b640 1 -- 172.21.15.49:0/252589540 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 msgr2=0x558db257d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba9f7b640 1 --2- 172.21.15.49:0/252589540 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db257d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba9f7b640 1 -- 172.21.15.49:0/252589540 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558db249b460 con 0x558db2575d30 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feb9affd640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a28a0 0x7feb900a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7feba977a640 1 --2- 172.21.15.49:0/252589540 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db257d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.011+0000 7febaa77c640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a28a0 0x7feb900a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.012+0000 7feba9f7b640 1 --2- 172.21.15.49:0/252589540 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257a830 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7feba00d4a90 tx=0x7feba00d4f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f803999388a4afac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.012+0000 7feb9affd640 1 -- 172.21.15.49:0/2037278767 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7feba00ce7d0 con 0x7feba4100700 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.012+0000 7feb8ffff640 1 -- 172.21.15.49:0/252589540 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba009aec0 con 0x558db2575d30 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.012+0000 7feb8ffff640 1 -- 172.21.15.49:0/252589540 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feba00a3890 con 0x558db2575d30 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.012+0000 7feb8ffff640 1 -- 172.21.15.49:0/252589540 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba009aa50 con 0x558db2575d30 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 -- 172.21.15.49:0/252589540 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 msgr2=0x558db257a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaa77c640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a28a0 0x7feb900a4d60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7feb9c02c630 tx=0x7feb9c032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 --2- 172.21.15.49:0/252589540 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257a830 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7feba00d4a90 tx=0x7feba00d4f50 comp rx=0 tx=0).stop 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 -- 172.21.15.49:0/252589540 shutdown_connections 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 --2- 172.21.15.49:0/252589540 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db257d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 --2- 172.21.15.49:0/252589540 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257a830 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 --2- 172.21.15.49:0/252589540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db2575760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 -- 172.21.15.49:0/252589540 >> 172.21.15.49:0/252589540 conn(0x558db25708b0 msgr2=0x558db2570cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 -- 172.21.15.49:0/252589540 shutdown_connections 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 -- 172.21.15.49:0/252589540 wait complete. 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 -- start start 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.013+0000 7febaedc7740 1 Processor -- start 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7febaedc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db257f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7febaedc7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7febaedc7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db2586c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7febaedc7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558db257de50 con 0x558db2575d30 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7febaedc7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558db2496d70 con 0x558db257aea0 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7febaedc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558db257dfd0 con 0x558db2575340 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7febaedc7740 10 client.0 _ll_register_callbacks cb 0x558db21eea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba977a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba977a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53946/0 (socket says 172.21.15.49:53946) 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba977a640 1 -- 172.21.15.49:0/426795225 learned_addr learned my addr 172.21.15.49:0/426795225 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7febaa77c640 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db2586c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba9f7b640 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db257f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba977a640 1 -- 172.21.15.49:0/426795225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 msgr2=0x558db257f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba977a640 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db257f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba977a640 1 -- 172.21.15.49:0/426795225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 msgr2=0x558db2586c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba977a640 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db2586c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba977a640 1 -- 172.21.15.49:0/426795225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558db21ef210 con 0x558db2575d30 2024-03-18T01:00:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feba977a640 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257fbc0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7feb94012b10 tx=0x7feb9400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa7c6fb8f529431 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feb8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feb8d7fa640 1 -- 172.21.15.49:0/426795225 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9400fae0 con 0x558db2575d30 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7feb8d7fa640 1 -- 172.21.15.49:0/426795225 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7feb9400fc80 con 0x558db2575d30 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.014+0000 7febaedc7740 1 -- 172.21.15.49:0/426795225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558db24e7f80 con 0x558db2575d30 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7feb8d7fa640 1 -- 172.21.15.49:0/426795225 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feb940047a0 con 0x558db2575d30 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7febaedc7740 10 client.17874 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7febaedc7740 1 -- 172.21.15.49:0/426795225 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558db22bb620 con 0x558db2575d30 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7feb8d7fa640 1 -- 172.21.15.49:0/426795225 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558db24e7f80 con 0x558db2575d30 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7feb8d7fa640 1 -- 172.21.15.49:0/426795225 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7feb940105f0 con 0x558db2575d30 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7febaedc7740 10 client.17874 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7febaedc7740 10 client.17874 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7febaedc7740 20 client.17874 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7febaedc7740 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558db258b600 0x558db258da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7febaedc7740 4 client.17874 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7febaedc7740 1 -- 172.21.15.49:0/426795225 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558db24e22d0 con 0x558db258b600 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.015+0000 7feba9f7b640 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558db258b600 0x558db258da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7feba9f7b640 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558db258b600 0x558db258da40 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e0595ef695942f4 server_cookie=74da0792d8e82999 in_seq=0 out_seq=0 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7feb8d7fa640 10 client.17874 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7feb8d7fa640 1 -- 172.21.15.49:0/426795225 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558db24e22d0 con 0x558db258b600 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7feb8d7fa640 10 client.17874 handle_command_reply: tid=0 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7febaedc7740 1 client.17874 shutdown 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7febaedc7740 1 -- 172.21.15.49:0/426795225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 msgr2=0x558db257fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7febaedc7740 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257fbc0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7feb94012b10 tx=0x7feb9400f3c0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7febaedc7740 1 -- 172.21.15.49:0/426795225 shutdown_connections 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7febaedc7740 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558db258b600 0x558db258da40 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.016+0000 7febaedc7740 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db2586c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.017+0000 7feb8d7fa640 0 client.17874 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.017+0000 7febaedc7740 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575d30 0x558db257fbc0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.017+0000 7febaedc7740 1 --2- 172.21.15.49:0/426795225 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db257f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.017+0000 7febaedc7740 1 -- 172.21.15.49:0/426795225 >> 172.21.15.49:0/426795225 conn(0x558db25708b0 msgr2=0x558db256fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.017+0000 7febaedc7740 1 -- 172.21.15.49:0/426795225 shutdown_connections 2024-03-18T01:00:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.017+0000 7febaedc7740 1 -- 172.21.15.49:0/426795225 wait complete. 2024-03-18T01:00:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.017+0000 7febaedc7740 20 client.17874 trim_cache size 0 max 16384 2024-03-18T01:00:12.035 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7febaedc7740 1 Processor -- start 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7febaedc7740 1 -- start start 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7febaedc7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575340 0x558db256f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7febaedc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575d30 0x558db2586bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7febaedc7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db256f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7febaedc7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7feba41aaf60 con 0x558db2575340 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7febaedc7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7feba41acad0 con 0x558db257aea0 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7febaedc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feba41ab630 con 0x558db2575d30 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7feba9f7b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575340 0x558db256f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7feba977a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575d30 0x558db2586bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7feba9f7b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575340 0x558db256f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53950/0 (socket says 172.21.15.49:53950) 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7feba9f7b640 1 -- 172.21.15.49:0/4217217159 learned_addr learned my addr 172.21.15.49:0/4217217159 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7febaa77c640 1 --2- 172.21.15.49:0/4217217159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db256f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.036 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7feba9f7b640 1 -- 172.21.15.49:0/4217217159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575d30 msgr2=0x558db2586bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7feba9f7b640 1 --2- 172.21.15.49:0/4217217159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575d30 0x558db2586bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7feba9f7b640 1 -- 172.21.15.49:0/4217217159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 msgr2=0x558db256f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7feba9f7b640 1 --2- 172.21.15.49:0/4217217159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db256f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.034+0000 7feba9f7b640 1 -- 172.21.15.49:0/4217217159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558db2571820 con 0x558db2575340 2024-03-18T01:00:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.035+0000 7feba977a640 1 --2- 172.21.15.49:0/4217217159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575d30 0x558db2586bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.035+0000 7feba9f7b640 1 --2- 172.21.15.49:0/4217217159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575340 0x558db256f320 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7feba009b080 tx=0x7feba00bb0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd8e7a0753ddafa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.035+0000 7feb8f7fe640 1 -- 172.21.15.49:0/4217217159 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba009ac30 con 0x558db2575340 2024-03-18T01:00:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.035+0000 7feb8f7fe640 1 -- 172.21.15.49:0/4217217159 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feba000ecf0 con 0x558db2575340 2024-03-18T01:00:12.037 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.035+0000 7feb8f7fe640 1 -- 172.21.15.49:0/4217217159 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba009b0f0 con 0x558db2575340 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.036+0000 7febaedc7740 1 -- 172.21.15.49:0/4217217159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575340 msgr2=0x558db256f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.036+0000 7febaedc7740 1 --2- 172.21.15.49:0/4217217159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575340 0x558db256f320 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7feba009b080 tx=0x7feba00bb0d0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.036+0000 7febaedc7740 1 -- 172.21.15.49:0/4217217159 shutdown_connections 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.036+0000 7febaedc7740 1 --2- 172.21.15.49:0/4217217159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db257aea0 0x558db256f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.036+0000 7febaedc7740 1 --2- 172.21.15.49:0/4217217159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575d30 0x558db2586bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.036+0000 7febaedc7740 1 --2- 172.21.15.49:0/4217217159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db2575340 0x558db256f320 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.036+0000 7febaedc7740 1 -- 172.21.15.49:0/4217217159 >> 172.21.15.49:0/4217217159 conn(0x558db25708b0 msgr2=0x558db25770b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.037+0000 7febaedc7740 1 -- 172.21.15.49:0/4217217159 shutdown_connections 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.037+0000 7febaedc7740 1 -- 172.21.15.49:0/4217217159 wait complete. 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.037+0000 7febaedc7740 1 -- start start 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.037+0000 7febaedc7740 1 Processor -- start 2024-03-18T01:00:12.041 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7febaedc7740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db25863b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7febaedc7740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db2575d30 0x558db25868f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7febaedc7740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db257aea0 0x558db2584a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7febaedc7740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7feba41ac3d0 con 0x558db257aea0 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7febaedc7740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7feba41abd00 con 0x558db2575d30 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7febaedc7740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7feba41a8f40 con 0x558db2575340 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7febaedc7740 10 client.0 _ll_register_callbacks cb 0x7feb94009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7feba9f7b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db25863b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7feba9f7b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db25863b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:33926/0 (socket says 172.21.15.49:33926) 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7feba977a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db2575d30 0x558db25868f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7feba9f7b640 1 -- 172.21.15.49:0/1891845319 learned_addr learned my addr 172.21.15.49:0/1891845319 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7febaa77c640 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db257aea0 0x558db2584a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7feba977a640 1 -- 172.21.15.49:0/1891845319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 msgr2=0x558db25863b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:12.042 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7feba977a640 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db25863b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7feba977a640 1 -- 172.21.15.49:0/1891845319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db257aea0 msgr2=0x558db2584a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7feba977a640 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db257aea0 0x558db2584a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.038+0000 7feba977a640 1 -- 172.21.15.49:0/1891845319 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558db2585080 con 0x558db2575d30 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7feba977a640 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db2575d30 0x558db25868f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feb94010c50 tx=0x7feb940106e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91a4b464e1ee3522 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7feb8d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7feb8d7fa640 1 -- 172.21.15.49:0/1891845319 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb94034060 con 0x558db2575d30 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7febaedc7740 1 -- 172.21.15.49:0/1891845319 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558db2460770 con 0x558db2575d30 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7feb8d7fa640 1 -- 172.21.15.49:0/1891845319 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7feb9401c470 con 0x558db2575d30 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7feb8d7fa640 1 -- 172.21.15.49:0/1891845319 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7feb9401c690 con 0x558db2575d30 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7febaedc7740 10 client.27005 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7febaedc7740 1 -- 172.21.15.49:0/1891845319 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558db25f22d0 con 0x558db2575d30 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7feb8d7fa640 1 -- 172.21.15.49:0/1891845319 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558db2460770 con 0x558db2575d30 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7feb8d7fa640 1 -- 172.21.15.49:0/1891845319 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7feb9402f640 con 0x558db2575d30 2024-03-18T01:00:12.043 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7febaedc7740 10 client.27005 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7febaedc7740 10 client.27005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7febaedc7740 20 client.27005 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7febaedc7740 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558db25f2c50 0x558db25f5070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7febaedc7740 4 client.27005 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T01:00:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.039+0000 7febaedc7740 1 -- 172.21.15.49:0/1891845319 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558db24ac560 con 0x558db25f2c50 2024-03-18T01:00:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.040+0000 7feba9f7b640 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558db25f2c50 0x558db25f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.040+0000 7feb8d7fa640 10 client.27005 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:12.044 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.040+0000 7feba9f7b640 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558db25f2c50 0x558db25f5070 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=878775a97e61a00 server_cookie=9e1292e7328d5b1b in_seq=0 out_seq=0 2024-03-18T01:00:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7feb8d7fa640 1 -- 172.21.15.49:0/1891845319 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558db24ac560 con 0x558db25f2c50 2024-03-18T01:00:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7feb8d7fa640 10 client.27005 handle_command_reply: tid=0 2024-03-18T01:00:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 client.27005 shutdown 2024-03-18T01:00:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 -- 172.21.15.49:0/1891845319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db2575d30 msgr2=0x558db25868f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db2575d30 0x558db25868f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7feb94010c50 tx=0x7feb940106e0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.063 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 -- 172.21.15.49:0/1891845319 shutdown_connections 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558db25f2c50 0x558db25f5070 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558db2575d30 0x558db25868f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558db257aea0 0x558db2584a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7feb8d7fa640 0 client.27005 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 --2- 172.21.15.49:0/1891845319 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558db2575340 0x558db25863b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 -- 172.21.15.49:0/1891845319 >> 172.21.15.49:0/1891845319 conn(0x558db25708b0 msgr2=0x558db2573b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 -- 172.21.15.49:0/1891845319 shutdown_connections 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 1 -- 172.21.15.49:0/1891845319 wait complete. 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.062+0000 7febaedc7740 20 client.27005 trim_cache size 0 max 16384 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T01:00:12.064 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T01:00:12.065 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T01:00:12.065 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T01:00:12.065 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 -- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a28a0 msgr2=0x7feb900a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a28a0 0x7feb900a4d60 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7feb9c02c630 tx=0x7feb9c032590 comp rx=0 tx=0).stop 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 -- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4100700 msgr2=0x7feba4199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4100700 0x7feba4199bf0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7feba000b3c0 tx=0x7feba0009ba0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 -- 172.21.15.49:0/2037278767 shutdown_connections 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feb900a28a0 0x7feb900a4d60 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7feba41018f0 0x7feba419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feba41022b0 0x7feba41a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 --2- 172.21.15.49:0/2037278767 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7feba4100700 0x7feba4199bf0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 -- 172.21.15.49:0/2037278767 >> 172.21.15.49:0/2037278767 conn(0x7feba40fbc70 msgr2=0x7feba4104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 -- 172.21.15.49:0/2037278767 shutdown_connections 2024-03-18T01:00:12.066 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:12.063+0000 7febac1df640 1 -- 172.21.15.49:0/2037278767 wait complete. 2024-03-18T01:00:13.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:13 smithi049 ceph-mon[32130]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s rd, 688 MiB/s wr, 4.95k op/s 2024-03-18T01:00:13.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:13 smithi107 ceph-mon[36270]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s rd, 688 MiB/s wr, 4.95k op/s 2024-03-18T01:00:13.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:13 smithi106 ceph-mon[35899]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s rd, 688 MiB/s wr, 4.95k op/s 2024-03-18T01:00:15.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:15 smithi049 ceph-mon[32130]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s rd, 545 MiB/s wr, 3.46k op/s 2024-03-18T01:00:15.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:15 smithi107 ceph-mon[36270]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s rd, 545 MiB/s wr, 3.46k op/s 2024-03-18T01:00:15.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:15 smithi106 ceph-mon[35899]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 169 MiB/s rd, 545 MiB/s wr, 3.46k op/s 2024-03-18T01:00:17.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:17 smithi049 ceph-mon[32130]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 198 MiB/s rd, 453 MiB/s wr, 2.71k op/s 2024-03-18T01:00:17.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:17 smithi107 ceph-mon[36270]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 198 MiB/s rd, 453 MiB/s wr, 2.71k op/s 2024-03-18T01:00:17.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:17 smithi106 ceph-mon[35899]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 198 MiB/s rd, 453 MiB/s wr, 2.71k op/s 2024-03-18T01:00:19.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:19 smithi049 ceph-mon[32130]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 292 MiB/s rd, 362 MiB/s wr, 2.24k op/s 2024-03-18T01:00:19.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:19 smithi107 ceph-mon[36270]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 292 MiB/s rd, 362 MiB/s wr, 2.24k op/s 2024-03-18T01:00:19.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:19 smithi106 ceph-mon[35899]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 292 MiB/s rd, 362 MiB/s wr, 2.24k op/s 2024-03-18T01:00:21.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:21 smithi049 ceph-mon[32130]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 323 MiB/s rd, 174 MiB/s wr, 1.34k op/s 2024-03-18T01:00:21.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:21 smithi107 ceph-mon[36270]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 323 MiB/s rd, 174 MiB/s wr, 1.34k op/s 2024-03-18T01:00:21.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:21 smithi106 ceph-mon[35899]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 323 MiB/s rd, 174 MiB/s wr, 1.34k op/s 2024-03-18T01:00:23.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:23 smithi049 ceph-mon[32130]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 400 MiB/s rd, 122 MiB/s wr, 1.22k op/s 2024-03-18T01:00:23.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:23 smithi107 ceph-mon[36270]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 400 MiB/s rd, 122 MiB/s wr, 1.22k op/s 2024-03-18T01:00:23.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:23 smithi106 ceph-mon[35899]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 400 MiB/s rd, 122 MiB/s wr, 1.22k op/s 2024-03-18T01:00:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 -- 172.21.15.49:0/3408365153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 msgr2=0x7f554c109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 --2- 172.21.15.49:0/3408365153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 0x7f554c109970 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f5544009ca0 tx=0x7f554402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 -- 172.21.15.49:0/3408365153 shutdown_connections 2024-03-18T01:00:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 --2- 172.21.15.49:0/3408365153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f554c104c50 0x7f554c107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 --2- 172.21.15.49:0/3408365153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 0x7f554c109970 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 --2- 172.21.15.49:0/3408365153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554c102320 0x7f554c104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 -- 172.21.15.49:0/3408365153 >> 172.21.15.49:0/3408365153 conn(0x7f554c0fbc70 msgr2=0x7f554c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 -- 172.21.15.49:0/3408365153 shutdown_connections 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 -- 172.21.15.49:0/3408365153 wait complete. 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.181+0000 7f5552236640 1 Processor -- start 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f5552236640 1 -- start start 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f5552236640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554c102320 0x7f554c197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f5552236640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f554c104c50 0x7f554c197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f5552236640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 0x7f554c19f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f5552236640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f554c10d4a0 con 0x7f554c107580 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f5552236640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f554c10d320 con 0x7f554c104c50 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f5552236640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f554c10d620 con 0x7f554c102320 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 0x7f554c19f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 0x7f554c19f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53738/0 (socket says 172.21.15.49:53738) 2024-03-18T01:00:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f554c104c50 0x7f554c197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554bfff640 1 -- 172.21.15.49:0/563732983 learned_addr learned my addr 172.21.15.49:0/563732983 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554b7fe640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554c102320 0x7f554c197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554bfff640 1 -- 172.21.15.49:0/563732983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554c102320 msgr2=0x7f554c197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554bfff640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554c102320 0x7f554c197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554bfff640 1 -- 172.21.15.49:0/563732983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f554c104c50 msgr2=0x7f554c197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554bfff640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f554c104c50 0x7f554c197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554bfff640 1 -- 172.21.15.49:0/563732983 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f554c19f6b0 con 0x7f554c107580 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554affd640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f554c104c50 0x7f554c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.182+0000 7f554bfff640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 0x7f554c19f040 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f5544037040 tx=0x7f5544032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.183+0000 7f5548ff9640 1 -- 172.21.15.49:0/563732983 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544041020 con 0x7f554c107580 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.183+0000 7f5552236640 1 -- 172.21.15.49:0/563732983 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f554c19f940 con 0x7f554c107580 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.183+0000 7f5548ff9640 1 -- 172.21.15.49:0/563732983 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5544034d90 con 0x7f554c107580 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.183+0000 7f5548ff9640 1 -- 172.21.15.49:0/563732983 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5544039040 con 0x7f554c107580 2024-03-18T01:00:25.185 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.183+0000 7f5552236640 1 -- 172.21.15.49:0/563732983 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f554c19fe30 con 0x7f554c107580 2024-03-18T01:00:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.185+0000 7f5548ff9640 1 -- 172.21.15.49:0/563732983 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5544044020 con 0x7f554c107580 2024-03-18T01:00:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5554e27740 1 Processor -- start 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5554e27740 1 -- start start 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5554e27740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1b8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5554e27740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1bd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5554e27740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5554e27740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563ead11da40 con 0x563ead1bdea0 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5554e27740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563ead0c6920 con 0x563ead1b8d30 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5554e27740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563ead119ae0 con 0x563ead1b8340 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5548ff9640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f55340a28a0 0x7f55340a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1bd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53754/0 (socket says 172.21.15.49:53754) 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554affd640 1 -- 172.21.15.49:0/2591301557 learned_addr learned my addr 172.21.15.49:0/2591301557 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554affd640 1 --2- 172.21.15.49:0/2591301557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1b8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554bfff640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f55340a28a0 0x7f55340a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f5548ff9640 1 -- 172.21.15.49:0/563732983 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f55440f0960 con 0x7f554c107580 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554b7fe640 1 -- 172.21.15.49:0/2591301557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 msgr2=0x563ead1b8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554b7fe640 1 --2- 172.21.15.49:0/2591301557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1b8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554b7fe640 1 -- 172.21.15.49:0/2591301557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 msgr2=0x563ead1c0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554b7fe640 1 --2- 172.21.15.49:0/2591301557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554b7fe640 1 -- 172.21.15.49:0/2591301557 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ead0de460 con 0x563ead1b8d30 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.186+0000 7f554affd640 1 --2- 172.21.15.49:0/2591301557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.187+0000 7f554bfff640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f55340a28a0 0x7f55340a4d60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f55440bd270 tx=0x7f55440323c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.187+0000 7f554b7fe640 1 --2- 172.21.15.49:0/2591301557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1bd830 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f553c004630 tx=0x7f553c00bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d82106c66006f710 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.187+0000 7f5531ffb640 1 -- 172.21.15.49:0/2591301557 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c003bd0 con 0x563ead1b8d30 2024-03-18T01:00:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.187+0000 7f5531ffb640 1 -- 172.21.15.49:0/2591301557 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f553c015070 con 0x563ead1b8d30 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.187+0000 7f5531ffb640 1 -- 172.21.15.49:0/2591301557 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c004d20 con 0x563ead1b8d30 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 -- 172.21.15.49:0/2591301557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 msgr2=0x563ead1bd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 --2- 172.21.15.49:0/2591301557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1bd830 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f553c004630 tx=0x7f553c00bb20 comp rx=0 tx=0).stop 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 -- 172.21.15.49:0/2591301557 shutdown_connections 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 --2- 172.21.15.49:0/2591301557 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1bd830 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 --2- 172.21.15.49:0/2591301557 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 --2- 172.21.15.49:0/2591301557 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1b8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 -- 172.21.15.49:0/2591301557 >> 172.21.15.49:0/2591301557 conn(0x563ead1b38b0 msgr2=0x563ead1b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 -- 172.21.15.49:0/2591301557 shutdown_connections 2024-03-18T01:00:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 -- 172.21.15.49:0/2591301557 wait complete. 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 -- start start 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 Processor -- start 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1c26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563ead1c0e50 con 0x563ead1bdea0 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563ead0d9d70 con 0x563ead1b8d30 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.188+0000 7f5554e27740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563ead1c0fd0 con 0x563ead1b8340 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f5554e27740 10 client.0 _ll_register_callbacks cb 0x563eace31a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554b7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53756/0 (socket says 172.21.15.49:53756) 2024-03-18T01:00:25.196 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 -- 172.21.15.49:0/3419307231 learned_addr learned my addr 172.21.15.49:0/3419307231 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1c26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 -- 172.21.15.49:0/3419307231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 msgr2=0x563ead1c26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1c26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 -- 172.21.15.49:0/3419307231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 msgr2=0x563ead1c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1c2be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 -- 172.21.15.49:0/3419307231 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563eace32210 con 0x563ead1bdea0 2024-03-18T01:00:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f554affd640 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c9c20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5540013e10 tx=0x7f5540009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1bc3fe935547ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f552affd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f552affd640 1 -- 172.21.15.49:0/3419307231 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5540004230 con 0x563ead1bdea0 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f552affd640 1 -- 172.21.15.49:0/3419307231 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f55400043d0 con 0x563ead1bdea0 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f5554e27740 1 -- 172.21.15.49:0/3419307231 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f553c015070 con 0x563ead1bdea0 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f552affd640 1 -- 172.21.15.49:0/3419307231 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5540011950 con 0x563ead1bdea0 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f5554e27740 10 client.17892 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.189+0000 7f5554e27740 1 -- 172.21.15.49:0/3419307231 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563eacefe620 con 0x563ead1bdea0 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f552affd640 1 -- 172.21.15.49:0/3419307231 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f553c015070 con 0x563ead1bdea0 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f552affd640 1 -- 172.21.15.49:0/3419307231 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f5540005020 con 0x563ead1bdea0 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f5554e27740 10 client.17892 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f5554e27740 10 client.17892 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f5554e27740 20 client.17892 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f5554e27740 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563ead1ce520 0x563ead1d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f5554e27740 4 client.17892 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f5554e27740 1 -- 172.21.15.49:0/3419307231 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ead12af80 con 0x563ead1ce520 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f554b7fe640 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563ead1ce520 0x563ead1d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f552affd640 10 client.17892 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.190+0000 7f554b7fe640 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563ead1ce520 0x563ead1d09a0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dc451eb10ad9df3 server_cookie=95e22f6291a37f42 in_seq=0 out_seq=0 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f552affd640 1 -- 172.21.15.49:0/3419307231 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ead12af80 con 0x563ead1ce520 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f552affd640 10 client.17892 handle_command_reply: tid=0 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 client.17892 shutdown 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 -- 172.21.15.49:0/3419307231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 msgr2=0x563ead1c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c9c20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5540013e10 tx=0x7f5540009f10 comp rx=0 tx=0).stop 2024-03-18T01:00:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 -- 172.21.15.49:0/3419307231 shutdown_connections 2024-03-18T01:00:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563ead1ce520 0x563ead1d09a0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f552affd640 0 client.17892 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1bdea0 0x563ead1c9c20 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 --2- 172.21.15.49:0/3419307231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1c26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 -- 172.21.15.49:0/3419307231 >> 172.21.15.49:0/3419307231 conn(0x563ead1b38b0 msgr2=0x563ead1b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 -- 172.21.15.49:0/3419307231 shutdown_connections 2024-03-18T01:00:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.191+0000 7f5554e27740 1 -- 172.21.15.49:0/3419307231 wait complete. 2024-03-18T01:00:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.192+0000 7f5554e27740 20 client.17892 trim_cache size 0 max 16384 2024-03-18T01:00:25.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f5554e27740 1 Processor -- start 2024-03-18T01:00:25.211 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f5554e27740 1 -- start start 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f5554e27740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b38b0 0x563ead1c1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f5554e27740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f5554e27740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f5554e27740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f554c1a8c50 con 0x563ead1b38b0 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f5554e27740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f554c1aa7c0 con 0x563ead1b8d30 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f5554e27740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f554c1a9320 con 0x563ead1b8340 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b38b0 0x563ead1c1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b38b0 0x563ead1c1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53766/0 (socket says 172.21.15.49:53766) 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554affd640 1 -- 172.21.15.49:0/2278879687 learned_addr learned my addr 172.21.15.49:0/2278879687 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554b7fe640 1 --2- 172.21.15.49:0/2278879687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554affd640 1 --2- 172.21.15.49:0/2278879687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554affd640 1 -- 172.21.15.49:0/2278879687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 msgr2=0x563ead1c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554affd640 1 --2- 172.21.15.49:0/2278879687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.212 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554affd640 1 -- 172.21.15.49:0/2278879687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 msgr2=0x563ead1b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554affd640 1 --2- 172.21.15.49:0/2278879687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.207+0000 7f554affd640 1 -- 172.21.15.49:0/2278879687 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ead1b4300 con 0x563ead1b38b0 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.208+0000 7f554affd640 1 --2- 172.21.15.49:0/2278879687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b38b0 0x563ead1c1b10 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f554000f940 tx=0x7f554000f760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a0a47e449c51f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.208+0000 7f55317fa640 1 -- 172.21.15.49:0/2278879687 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5540009cc0 con 0x563ead1b38b0 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.208+0000 7f55317fa640 1 -- 172.21.15.49:0/2278879687 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5540033030 con 0x563ead1b38b0 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.208+0000 7f55317fa640 1 -- 172.21.15.49:0/2278879687 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f554000ebc0 con 0x563ead1b38b0 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.208+0000 7f5554e27740 1 -- 172.21.15.49:0/2278879687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b38b0 msgr2=0x563ead1c1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.208+0000 7f5554e27740 1 --2- 172.21.15.49:0/2278879687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b38b0 0x563ead1c1b10 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f554000f940 tx=0x7f554000f760 comp rx=0 tx=0).stop 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 -- 172.21.15.49:0/2278879687 shutdown_connections 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 --2- 172.21.15.49:0/2278879687 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b8d30 0x563ead1b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 --2- 172.21.15.49:0/2278879687 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8340 0x563ead1c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 --2- 172.21.15.49:0/2278879687 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b38b0 0x563ead1c1b10 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 -- 172.21.15.49:0/2278879687 >> 172.21.15.49:0/2278879687 conn(0x563ead1bd9a0 msgr2=0x563ead1c1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:25.213 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 -- 172.21.15.49:0/2278879687 shutdown_connections 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 -- 172.21.15.49:0/2278879687 wait complete. 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 -- start start 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 Processor -- start 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b38b0 0x563ead1c7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b8340 0x563ead1c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8d30 0x563ead1c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f554c1aa0c0 con 0x563ead1b8340 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f554c1a99f0 con 0x563ead1b38b0 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.209+0000 7f5554e27740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f554c1a6c30 con 0x563ead1b8d30 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f5554e27740 10 client.0 _ll_register_callbacks cb 0x7f5540013970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b8340 0x563ead1c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b38b0 0x563ead1c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.214 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b8340 0x563ead1c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:53782/0 (socket says 172.21.15.49:53782) 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554b7fe640 1 -- 172.21.15.49:0/2957641776 learned_addr learned my addr 172.21.15.49:0/2957641776 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554affd640 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8d30 0x563ead1c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554b7fe640 1 -- 172.21.15.49:0/2957641776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8d30 msgr2=0x563ead1c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554b7fe640 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8d30 0x563ead1c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554b7fe640 1 -- 172.21.15.49:0/2957641776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b38b0 msgr2=0x563ead1c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554b7fe640 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b38b0 0x563ead1c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554b7fe640 1 -- 172.21.15.49:0/2957641776 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ead1c9480 con 0x563ead1b8340 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f554b7fe640 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b8340 0x563ead1c7e80 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f553c0043f0 tx=0x7f553c005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d074f6881bfa41a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f552affd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f552affd640 1 -- 172.21.15.49:0/2957641776 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553c030060 con 0x563ead1b8340 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f552affd640 1 -- 172.21.15.49:0/2957641776 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f553c005040 con 0x563ead1b8340 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f5554e27740 1 -- 172.21.15.49:0/2957641776 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5540033030 con 0x563ead1b8340 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f552affd640 1 -- 172.21.15.49:0/2957641776 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f553c005240 con 0x563ead1b8340 2024-03-18T01:00:25.215 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f5554e27740 10 client.17898 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.210+0000 7f5554e27740 1 -- 172.21.15.49:0/2957641776 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563ead234ff0 con 0x563ead1b8340 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.211+0000 7f552affd640 1 -- 172.21.15.49:0/2957641776 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5540033030 con 0x563ead1b8340 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.211+0000 7f552affd640 1 -- 172.21.15.49:0/2957641776 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f553c004860 con 0x563ead1b8340 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.211+0000 7f5554e27740 10 client.17898 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.211+0000 7f5554e27740 10 client.17898 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.211+0000 7f5554e27740 20 client.17898 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.211+0000 7f5554e27740 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563ead1bdea0 0x563ead237870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.211+0000 7f5554e27740 4 client.17898 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.211+0000 7f5554e27740 1 -- 172.21.15.49:0/2957641776 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563ead1252d0 con 0x563ead1bdea0 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.211+0000 7f554affd640 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563ead1bdea0 0x563ead237870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.212+0000 7f554affd640 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563ead1bdea0 0x563ead237870 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dbffa6936a47a4b server_cookie=d7bd6cc24b6c6582 in_seq=0 out_seq=0 2024-03-18T01:00:25.216 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.212+0000 7f552affd640 10 client.17898 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f552affd640 1 -- 172.21.15.49:0/2957641776 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563ead1252d0 con 0x563ead1bdea0 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f552affd640 10 client.17898 handle_command_reply: tid=0 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 client.17898 shutdown 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 -- 172.21.15.49:0/2957641776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b8340 msgr2=0x563ead1c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b8340 0x563ead1c7e80 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f553c0043f0 tx=0x7f553c005c80 comp rx=0 tx=0).stop 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 -- 172.21.15.49:0/2957641776 shutdown_connections 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563ead1bdea0 0x563ead237870 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563ead1b8d30 0x563ead1c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563ead1b8340 0x563ead1c7e80 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f552affd640 0 client.17898 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 --2- 172.21.15.49:0/2957641776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563ead1b38b0 0x563ead1c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 -- 172.21.15.49:0/2957641776 >> 172.21.15.49:0/2957641776 conn(0x563ead1bd9a0 msgr2=0x563ead1b6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 -- 172.21.15.49:0/2957641776 shutdown_connections 2024-03-18T01:00:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 1 -- 172.21.15.49:0/2957641776 wait complete. 2024-03-18T01:00:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.228+0000 7f5554e27740 20 client.17898 trim_cache size 0 max 16384 2024-03-18T01:00:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T01:00:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T01:00:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T01:00:25.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T01:00:25.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 -- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f55340a28a0 msgr2=0x7f55340a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:25.231 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f55340a28a0 0x7f55340a4d60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f55440bd270 tx=0x7f55440323c0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 -- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 msgr2=0x7f554c19f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 0x7f554c19f040 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f5544037040 tx=0x7f5544032f20 comp rx=0 tx=0).stop 2024-03-18T01:00:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 -- 172.21.15.49:0/563732983 shutdown_connections 2024-03-18T01:00:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f55340a28a0 0x7f55340a4d60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f554c104c50 0x7f554c197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f554c107580 0x7f554c19f040 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.232 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 --2- 172.21.15.49:0/563732983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f554c102320 0x7f554c197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 -- 172.21.15.49:0/563732983 >> 172.21.15.49:0/563732983 conn(0x7f554c0fbc70 msgr2=0x7f554c102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 -- 172.21.15.49:0/563732983 shutdown_connections 2024-03-18T01:00:25.233 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:25.229+0000 7f5552236640 1 -- 172.21.15.49:0/563732983 wait complete. 2024-03-18T01:00:25.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:25 smithi049 ceph-mon[32130]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 411 MiB/s rd, 44 MiB/s wr, 835 op/s 2024-03-18T01:00:25.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:25 smithi107 ceph-mon[36270]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 411 MiB/s rd, 44 MiB/s wr, 835 op/s 2024-03-18T01:00:25.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:25 smithi106 ceph-mon[35899]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 411 MiB/s rd, 44 MiB/s wr, 835 op/s 2024-03-18T01:00:27.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:27 smithi049 ceph-mon[32130]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 413 MiB/s rd, 14 MiB/s wr, 675 op/s 2024-03-18T01:00:27.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:27 smithi107 ceph-mon[36270]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 413 MiB/s rd, 14 MiB/s wr, 675 op/s 2024-03-18T01:00:27.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:27 smithi106 ceph-mon[35899]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 413 MiB/s rd, 14 MiB/s wr, 675 op/s 2024-03-18T01:00:29.571 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:29 smithi106 ceph-mon[35899]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 447 MiB/s rd, 16 MiB/s wr, 820 op/s 2024-03-18T01:00:29.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:29 smithi049 ceph-mon[32130]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 447 MiB/s rd, 16 MiB/s wr, 820 op/s 2024-03-18T01:00:29.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:29 smithi107 ceph-mon[36270]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 447 MiB/s rd, 16 MiB/s wr, 820 op/s 2024-03-18T01:00:31.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:31 smithi049 ceph-mon[32130]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 406 MiB/s rd, 13 MiB/s wr, 633 op/s 2024-03-18T01:00:31.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:31 smithi107 ceph-mon[36270]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 406 MiB/s rd, 13 MiB/s wr, 633 op/s 2024-03-18T01:00:31.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:31 smithi106 ceph-mon[35899]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 406 MiB/s rd, 13 MiB/s wr, 633 op/s 2024-03-18T01:00:32.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:00:32.391 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 -- 172.21.15.49:0/864909929 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62101018d0 msgr2=0x7f6210101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 --2- 172.21.15.49:0/864909929 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62101018d0 0x7f6210101d50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f6200009860 tx=0x7f620002bb70 comp rx=0 tx=0).stop 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 -- 172.21.15.49:0/864909929 shutdown_connections 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 --2- 172.21.15.49:0/864909929 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6210102290 0x7f6210108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 --2- 172.21.15.49:0/864909929 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f62101018d0 0x7f6210101d50 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 --2- 172.21.15.49:0/864909929 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62101006e0 0x7f6210100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 -- 172.21.15.49:0/864909929 >> 172.21.15.49:0/864909929 conn(0x7f62100fbc70 msgr2=0x7f62100fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 -- 172.21.15.49:0/864909929 shutdown_connections 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 -- 172.21.15.49:0/864909929 wait complete. 2024-03-18T01:00:32.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.524+0000 7f6217dda640 1 Processor -- start 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6217dda640 1 -- start start 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6217dda640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62101006e0 0x7f6210196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6217dda640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62101018d0 0x7f6210196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6217dda640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6210102290 0x7f621019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6217dda640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f621010c360 con 0x7f6210102290 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6217dda640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f621010c1e0 con 0x7f62101006e0 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6217dda640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f621010c4e0 con 0x7f62101018d0 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6216377640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6210102290 0x7f621019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6215b76640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62101006e0 0x7f6210196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6216377640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6210102290 0x7f621019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37406/0 (socket says 172.21.15.49:37406) 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6216377640 1 -- 172.21.15.49:0/2132996299 learned_addr learned my addr 172.21.15.49:0/2132996299 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6215375640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62101018d0 0x7f6210196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6216377640 1 -- 172.21.15.49:0/2132996299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62101018d0 msgr2=0x7f6210196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.527 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6216377640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62101018d0 0x7f6210196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6216377640 1 -- 172.21.15.49:0/2132996299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62101006e0 msgr2=0x7f6210196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6216377640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62101006e0 0x7f6210196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6216377640 1 -- 172.21.15.49:0/2132996299 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621019e620 con 0x7f6210102290 2024-03-18T01:00:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6216377640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6210102290 0x7f621019dfb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f620800eb10 tx=0x7f620800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.525+0000 7f6206ffd640 1 -- 172.21.15.49:0/2132996299 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208004b20 con 0x7f6210102290 2024-03-18T01:00:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.526+0000 7f6217dda640 1 -- 172.21.15.49:0/2132996299 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f621019e910 con 0x7f6210102290 2024-03-18T01:00:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.526+0000 7f6206ffd640 1 -- 172.21.15.49:0/2132996299 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6208004cc0 con 0x7f6210102290 2024-03-18T01:00:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.526+0000 7f6206ffd640 1 -- 172.21.15.49:0/2132996299 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208010760 con 0x7f6210102290 2024-03-18T01:00:32.528 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.526+0000 7f6217dda640 1 -- 172.21.15.49:0/2132996299 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f621019ee30 con 0x7f6210102290 2024-03-18T01:00:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.528+0000 7f6206ffd640 1 -- 172.21.15.49:0/2132996299 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6208005020 con 0x7f6210102290 2024-03-18T01:00:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.528+0000 7f6206ffd640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61fc0a2800 0x7f61fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f6215b76640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61fc0a2800 0x7f61fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f6206ffd640 1 -- 172.21.15.49:0/2132996299 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f62080cf160 con 0x7f6210102290 2024-03-18T01:00:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f621a9bd740 1 Processor -- start 2024-03-18T01:00:32.531 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f6215b76640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61fc0a2800 0x7f61fc0a4cc0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f620c009530 tx=0x7f620c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f621a9bd740 1 -- start start 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612977f0 0x562661295f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562661296480 0x562661296900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612a0490 0x5626612a2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f621a9bd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5626611fb500 con 0x5626612a0490 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f621a9bd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5626611a3be0 con 0x5626612977f0 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.529+0000 7f621a9bd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5626611f75a0 con 0x562661296480 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6216377640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612a0490 0x5626612a2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6215375640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562661296480 0x562661296900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6215b76640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612977f0 0x562661295f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6216377640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612a0490 0x5626612a2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37408/0 (socket says 172.21.15.49:37408) 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6215b76640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612977f0 0x562661295f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:55796/0 (socket says 172.21.15.49:55796) 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6216377640 1 -- 172.21.15.49:0/251792052 learned_addr learned my addr 172.21.15.49:0/251792052 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:32.532 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6216377640 1 -- 172.21.15.49:0/251792052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562661296480 msgr2=0x562661296900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6216377640 1 --2- 172.21.15.49:0/251792052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562661296480 0x562661296900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6216377640 1 -- 172.21.15.49:0/251792052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612977f0 msgr2=0x562661295f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6216377640 1 --2- 172.21.15.49:0/251792052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612977f0 0x562661295f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6216377640 1 -- 172.21.15.49:0/251792052 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626611b74b0 con 0x5626612a0490 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f6216377640 1 --2- 172.21.15.49:0/251792052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612a0490 0x5626612a2880 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6208097f40 tx=0x7f620809b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edacebaab09824b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f61fbfff640 1 -- 172.21.15.49:0/251792052 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620809bae0 con 0x5626612a0490 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f61fbfff640 1 -- 172.21.15.49:0/251792052 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62080b4070 con 0x5626612a0490 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.530+0000 7f61fbfff640 1 -- 172.21.15.49:0/251792052 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62080b4ba0 con 0x5626612a0490 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.531+0000 7f621a9bd740 1 -- 172.21.15.49:0/251792052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612a0490 msgr2=0x5626612a2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.531+0000 7f621a9bd740 1 --2- 172.21.15.49:0/251792052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612a0490 0x5626612a2880 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f6208097f40 tx=0x7f620809b620 comp rx=0 tx=0).stop 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.531+0000 7f621a9bd740 1 -- 172.21.15.49:0/251792052 shutdown_connections 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.531+0000 7f621a9bd740 1 --2- 172.21.15.49:0/251792052 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612a0490 0x5626612a2880 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.531+0000 7f621a9bd740 1 --2- 172.21.15.49:0/251792052 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562661296480 0x562661296900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.533 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.531+0000 7f621a9bd740 1 --2- 172.21.15.49:0/251792052 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612977f0 0x562661295f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.531+0000 7f621a9bd740 1 -- 172.21.15.49:0/251792052 >> 172.21.15.49:0/251792052 conn(0x5626612916e0 msgr2=0x562661291ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.531+0000 7f621a9bd740 1 -- 172.21.15.49:0/251792052 shutdown_connections 2024-03-18T01:00:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.531+0000 7f621a9bd740 1 -- 172.21.15.49:0/251792052 wait complete. 2024-03-18T01:00:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f621a9bd740 1 -- start start 2024-03-18T01:00:32.534 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f621a9bd740 1 Processor -- start 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x5626612a4b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 0x5626612a5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 0x5626612abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f621a9bd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5626612a3060 con 0x562661296480 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f621a9bd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5626611b7160 con 0x5626612a0490 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f621a9bd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5626612a31e0 con 0x5626612977f0 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f621a9bd740 10 client.0 _ll_register_callbacks cb 0x56266119b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f6215b76640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x5626612a4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f6215b76640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x5626612a4b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37410/0 (socket says 172.21.15.49:37410) 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f6215b76640 1 -- 172.21.15.49:0/2670457622 learned_addr learned my addr 172.21.15.49:0/2670457622 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f6215375640 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 0x5626612a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.532+0000 7f6216377640 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 0x5626612abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f6215b76640 1 -- 172.21.15.49:0/2670457622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 msgr2=0x5626612a5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f6215b76640 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 0x5626612a5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f6215b76640 1 -- 172.21.15.49:0/2670457622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 msgr2=0x5626612abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f6215b76640 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 0x5626612abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f6215b76640 1 -- 172.21.15.49:0/2670457622 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562660fd97d0 con 0x562661296480 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f6216377640 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 0x5626612abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f6215b76640 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x5626612a4b40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f620c01b9e0 tx=0x7f620c013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb9d5d99c27c3b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f61f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f61f97fa640 1 -- 172.21.15.49:0/2670457622 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c02c060 con 0x562661296480 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f61f97fa640 1 -- 172.21.15.49:0/2670457622 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f620c00ed40 con 0x562661296480 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f621a9bd740 1 -- 172.21.15.49:0/2670457622 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62080b4070 con 0x562661296480 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f61f97fa640 1 -- 172.21.15.49:0/2670457622 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f620c027430 con 0x562661296480 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f621a9bd740 10 client.17910 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.533+0000 7f621a9bd740 1 -- 172.21.15.49:0/2670457622 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562660f0f3f0 con 0x562661296480 2024-03-18T01:00:32.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.534+0000 7f61f97fa640 1 -- 172.21.15.49:0/2670457622 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62080b4070 con 0x562661296480 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.534+0000 7f61f97fa640 1 -- 172.21.15.49:0/2670457622 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f620c026070 con 0x562661296480 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.534+0000 7f621a9bd740 10 client.17910 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.534+0000 7f621a9bd740 10 client.17910 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.534+0000 7f621a9bd740 20 client.17910 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.534+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5626612b0570 0x5626612b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.534+0000 7f621a9bd740 4 client.17910 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.534+0000 7f621a9bd740 1 -- 172.21.15.49:0/2670457622 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562661208a40 con 0x5626612b0570 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.534+0000 7f6215375640 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5626612b0570 0x5626612b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.535+0000 7f6215375640 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5626612b0570 0x5626612b29f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7377cb77adf5061 server_cookie=1a4764d7213dc967 in_seq=0 out_seq=0 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.535+0000 7f61f97fa640 10 client.17910 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f61f97fa640 1 -- 172.21.15.49:0/2670457622 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562661208a40 con 0x5626612b0570 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f61f97fa640 10 client.17910 handle_command_reply: tid=0 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 client.17910 shutdown 2024-03-18T01:00:32.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 -- 172.21.15.49:0/2670457622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 msgr2=0x5626612a4b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x5626612a4b40 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f620c01b9e0 tx=0x7f620c013040 comp rx=0 tx=0).stop 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 -- 172.21.15.49:0/2670457622 shutdown_connections 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5626612b0570 0x5626612b29f0 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 0x5626612abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 0x5626612a5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f61f97fa640 0 client.17910 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2670457622 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x5626612a4b40 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 -- 172.21.15.49:0/2670457622 >> 172.21.15.49:0/2670457622 conn(0x5626612916e0 msgr2=0x562661290900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 -- 172.21.15.49:0/2670457622 shutdown_connections 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 1 -- 172.21.15.49:0/2670457622 wait complete. 2024-03-18T01:00:32.541 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.536+0000 7f621a9bd740 20 client.17910 trim_cache size 0 max 16384 2024-03-18T01:00:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.556+0000 7f621a9bd740 1 Processor -- start 2024-03-18T01:00:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.556+0000 7f621a9bd740 1 -- start start 2024-03-18T01:00:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.556+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x562661295e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.556+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 0x5626612abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 0x5626612a4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f621a9bd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f62101a6bb0 con 0x562661296480 2024-03-18T01:00:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f621a9bd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f62101a8720 con 0x5626612a0490 2024-03-18T01:00:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f621a9bd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62101a7280 con 0x5626612977f0 2024-03-18T01:00:32.558 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215b76640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x562661295e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215b76640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x562661295e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37424/0 (socket says 172.21.15.49:37424) 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215b76640 1 -- 172.21.15.49:0/1949275829 learned_addr learned my addr 172.21.15.49:0/1949275829 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6216377640 1 --2- 172.21.15.49:0/1949275829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 0x5626612a4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215375640 1 --2- 172.21.15.49:0/1949275829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 0x5626612abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215b76640 1 -- 172.21.15.49:0/1949275829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 msgr2=0x5626612abfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215b76640 1 --2- 172.21.15.49:0/1949275829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 0x5626612abfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215b76640 1 -- 172.21.15.49:0/1949275829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 msgr2=0x5626612a4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215b76640 1 --2- 172.21.15.49:0/1949275829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 0x5626612a4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215b76640 1 -- 172.21.15.49:0/1949275829 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562660ecf0f0 con 0x562661296480 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f6215b76640 1 --2- 172.21.15.49:0/1949275829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x562661295e20 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f620c011d10 tx=0x7f620c0092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef7e9961033537d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f61fb7fe640 1 -- 172.21.15.49:0/1949275829 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c03f070 con 0x562661296480 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f61fb7fe640 1 -- 172.21.15.49:0/1949275829 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f620c01b7a0 con 0x562661296480 2024-03-18T01:00:32.559 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.557+0000 7f61fb7fe640 1 -- 172.21.15.49:0/1949275829 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c004680 con 0x562661296480 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 -- 172.21.15.49:0/1949275829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 msgr2=0x562661295e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 --2- 172.21.15.49:0/1949275829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x562661295e20 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f620c011d10 tx=0x7f620c0092d0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 -- 172.21.15.49:0/1949275829 shutdown_connections 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 --2- 172.21.15.49:0/1949275829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5626612a0490 0x5626612a4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 --2- 172.21.15.49:0/1949275829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612977f0 0x5626612abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 --2- 172.21.15.49:0/1949275829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x562661296480 0x562661295e20 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 -- 172.21.15.49:0/1949275829 >> 172.21.15.49:0/1949275829 conn(0x5626612916e0 msgr2=0x56266128c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 -- 172.21.15.49:0/1949275829 shutdown_connections 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 -- 172.21.15.49:0/1949275829 wait complete. 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.558+0000 7f621a9bd740 1 -- start start 2024-03-18T01:00:32.560 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f621a9bd740 1 Processor -- start 2024-03-18T01:00:32.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562661296480 0x5626612a8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612977f0 0x5626612a94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f621a9bd740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612a0490 0x5626613148e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f621a9bd740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f62101a8020 con 0x5626612977f0 2024-03-18T01:00:32.564 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f621a9bd740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f62101a7950 con 0x562661296480 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f621a9bd740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f62101a4b90 con 0x5626612a0490 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f621a9bd740 10 client.0 _ll_register_callbacks cb 0x7f620c009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215375640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612977f0 0x5626612a94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215b76640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562661296480 0x5626612a8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215375640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612977f0 0x5626612a94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37432/0 (socket says 172.21.15.49:37432) 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215375640 1 -- 172.21.15.49:0/2759309390 learned_addr learned my addr 172.21.15.49:0/2759309390 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6216377640 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612a0490 0x5626613148e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215375640 1 -- 172.21.15.49:0/2759309390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612a0490 msgr2=0x5626613148e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215375640 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612a0490 0x5626613148e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215375640 1 -- 172.21.15.49:0/2759309390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562661296480 msgr2=0x5626612a8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215375640 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562661296480 0x5626612a8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215375640 1 -- 172.21.15.49:0/2759309390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626611c29d0 con 0x5626612977f0 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215b76640 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562661296480 0x5626612a8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f6215375640 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612977f0 0x5626612a94e0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f6200037040 tx=0x7f6200005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ad8589c57e8ec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:32.565 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.559+0000 7f61f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f61f97fa640 1 -- 172.21.15.49:0/2759309390 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200039040 con 0x5626612977f0 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f61f97fa640 1 -- 172.21.15.49:0/2759309390 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6200004040 con 0x5626612977f0 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f621a9bd740 1 -- 172.21.15.49:0/2759309390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562661202d90 con 0x5626612977f0 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f61f97fa640 1 -- 172.21.15.49:0/2759309390 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6200004240 con 0x5626612977f0 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f621a9bd740 10 client.17916 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f621a9bd740 1 -- 172.21.15.49:0/2759309390 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562661316030 con 0x5626612977f0 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f61f97fa640 1 -- 172.21.15.49:0/2759309390 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562661202d90 con 0x5626612977f0 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f61f97fa640 1 -- 172.21.15.49:0/2759309390 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f62000327f0 con 0x5626612977f0 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f621a9bd740 10 client.17916 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f621a9bd740 10 client.17916 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f621a9bd740 20 client.17916 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5626613169c0 0x562661318de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f621a9bd740 4 client.17916 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:00:32.566 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f621a9bd740 1 -- 172.21.15.49:0/2759309390 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562661180540 con 0x5626613169c0 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.560+0000 7f6215b76640 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5626613169c0 0x562661318de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f6215b76640 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5626613169c0 0x562661318de0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ffdd1422174edc9 server_cookie=dcafc03d3a307d23 in_seq=0 out_seq=0 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f61f97fa640 10 client.17916 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f61f97fa640 1 -- 172.21.15.49:0/2759309390 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562661180540 con 0x5626613169c0 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f61f97fa640 10 client.17916 handle_command_reply: tid=0 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 client.17916 shutdown 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 -- 172.21.15.49:0/2759309390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612977f0 msgr2=0x5626612a94e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612977f0 0x5626612a94e0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f6200037040 tx=0x7f6200005780 comp rx=0 tx=0).stop 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 -- 172.21.15.49:0/2759309390 shutdown_connections 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x5626613169c0 0x562661318de0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5626612a0490 0x5626613148e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f61f97fa640 0 client.17916 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:32.567 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5626612977f0 0x5626612a94e0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 --2- 172.21.15.49:0/2759309390 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x562661296480 0x5626612a8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 -- 172.21.15.49:0/2759309390 >> 172.21.15.49:0/2759309390 conn(0x5626612916e0 msgr2=0x5626612a4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:32.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 -- 172.21.15.49:0/2759309390 shutdown_connections 2024-03-18T01:00:32.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 1 -- 172.21.15.49:0/2759309390 wait complete. 2024-03-18T01:00:32.568 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.561+0000 7f621a9bd740 20 client.17916 trim_cache size 0 max 16384 2024-03-18T01:00:32.568 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:00:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T01:00:32.568 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T01:00:32.568 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:00:32.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 -- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61fc0a2800 msgr2=0x7f61fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61fc0a2800 0x7f61fc0a4cc0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f620c009530 tx=0x7f620c009260 comp rx=0 tx=0).stop 2024-03-18T01:00:32.569 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 -- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6210102290 msgr2=0x7f621019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6210102290 0x7f621019dfb0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f620800eb10 tx=0x7f620800c2d0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 -- 172.21.15.49:0/2132996299 shutdown_connections 2024-03-18T01:00:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f61fc0a2800 0x7f61fc0a4cc0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f6210102290 0x7f621019dfb0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f62101018d0 0x7f6210196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 --2- 172.21.15.49:0/2132996299 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f62101006e0 0x7f6210196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 -- 172.21.15.49:0/2132996299 >> 172.21.15.49:0/2132996299 conn(0x7f62100fbc70 msgr2=0x7f6210104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 -- 172.21.15.49:0/2132996299 shutdown_connections 2024-03-18T01:00:32.570 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.562+0000 7f6217dda640 1 -- 172.21.15.49:0/2132996299 wait complete. 2024-03-18T01:00:32.576 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T01:00:32.576 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:00:32.710 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.710+0000 7f5a1cb77640 1 -- 172.21.15.49:0/2301680212 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a181033e0 msgr2=0x7f5a18109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.710+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/2301680212 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a181033e0 0x7f5a18109c70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5a10009900 tx=0x7f5a1002bc30 comp rx=0 tx=0).stop 2024-03-18T01:00:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.711+0000 7f5a1cb77640 1 -- 172.21.15.49:0/2301680212 shutdown_connections 2024-03-18T01:00:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.711+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/2301680212 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a18102a20 0x7f5a18102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.711+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/2301680212 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a181033e0 0x7f5a18109c70 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.711+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/2301680212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a18101830 0x7f5a18101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.711+0000 7f5a1cb77640 1 -- 172.21.15.49:0/2301680212 >> 172.21.15.49:0/2301680212 conn(0x7f5a180fcd80 msgr2=0x7f5a180ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.711+0000 7f5a1cb77640 1 -- 172.21.15.49:0/2301680212 shutdown_connections 2024-03-18T01:00:32.711 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.711+0000 7f5a1cb77640 1 -- 172.21.15.49:0/2301680212 wait complete. 2024-03-18T01:00:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1cb77640 1 Processor -- start 2024-03-18T01:00:32.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1cb77640 1 -- start start 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1cb77640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a18101830 0x7f5a18197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1cb77640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a18102a20 0x7f5a18198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1cb77640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a181033e0 0x7f5a1819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1cb77640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1810d560 con 0x7f5a18101830 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1cb77640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1810d3e0 con 0x7f5a181033e0 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1cb77640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1810d6e0 con 0x7f5a18102a20 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a18101830 0x7f5a18197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a18101830 0x7f5a18197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37462/0 (socket says 172.21.15.49:37462) 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a1659c640 1 -- 172.21.15.49:0/1384183034 learned_addr learned my addr 172.21.15.49:0/1384183034 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a16d9d640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a181033e0 0x7f5a1819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.714 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.712+0000 7f5a15d9b640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a18102a20 0x7f5a18198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a16d9d640 1 -- 172.21.15.49:0/1384183034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a18102a20 msgr2=0x7f5a18198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a16d9d640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a18102a20 0x7f5a18198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a16d9d640 1 -- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a18101830 msgr2=0x7f5a18197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a16d9d640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a18101830 0x7f5a18197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a16d9d640 1 -- 172.21.15.49:0/1384183034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1819f730 con 0x7f5a181033e0 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a15d9b640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a18102a20 0x7f5a18198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a1659c640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a18101830 0x7f5a18197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a16d9d640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a181033e0 0x7f5a1819f0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5a1002c170 tx=0x7f5a1002ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a077fe640 1 -- 172.21.15.49:0/1384183034 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10043020 con 0x7f5a181033e0 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a1819f9c0 con 0x7f5a181033e0 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a077fe640 1 -- 172.21.15.49:0/1384183034 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a100054e0 con 0x7f5a181033e0 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a077fe640 1 -- 172.21.15.49:0/1384183034 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a10003b50 con 0x7f5a181033e0 2024-03-18T01:00:32.715 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.713+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a1819ff30 con 0x7f5a181033e0 2024-03-18T01:00:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.715+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a18108a80 con 0x7f5a181033e0 2024-03-18T01:00:32.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.717+0000 7f5a077fe640 1 -- 172.21.15.49:0/1384183034 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5a10046020 con 0x7f5a181033e0 2024-03-18T01:00:32.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.717+0000 7f5a077fe640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a000a2800 0x7f5a000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:32.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.718+0000 7f5a1659c640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a000a2800 0x7f5a000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:32.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.718+0000 7f5a077fe640 1 -- 172.21.15.49:0/1384183034 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5a100f0f40 con 0x7f5a181033e0 2024-03-18T01:00:32.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.718+0000 7f5a1659c640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a000a2800 0x7f5a000a4cc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800f7b0 tx=0x7f5a08009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:32.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.724+0000 7f5a077fe640 1 -- 172.21.15.49:0/1384183034 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5a1004d020 con 0x7f5a181033e0 2024-03-18T01:00:32.986 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.986+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a1802d1b0 con 0x7f5a181033e0 2024-03-18T01:00:32.988 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.988+0000 7f5a077fe640 1 -- 172.21.15.49:0/1384183034 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11935 (secure 0 0 0) 0x7f5a100bdfa0 con 0x7f5a181033e0 2024-03-18T01:00:32.988 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T01:00:32.988 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T00:59:55.206381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":178,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24506,24499,24481,24463,24470]},"id":1}]} 2024-03-18T01:00:32.988 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 30 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a000a2800 msgr2=0x7f5a000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a000a2800 0x7f5a000a4cc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f5a0800f7b0 tx=0x7f5a08009500 comp rx=0 tx=0).stop 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a181033e0 msgr2=0x7f5a1819f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a181033e0 0x7f5a1819f0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5a1002c170 tx=0x7f5a1002ce60 comp rx=0 tx=0).stop 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 shutdown_connections 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5a000a2800 0x7f5a000a4cc0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5a181033e0 0x7f5a1819f0c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5a18102a20 0x7f5a18198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 --2- 172.21.15.49:0/1384183034 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5a18101830 0x7f5a18197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 >> 172.21.15.49:0/1384183034 conn(0x7f5a180fcd80 msgr2=0x7f5a18105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 shutdown_connections 2024-03-18T01:00:32.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:32.990+0000 7f5a1cb77640 1 -- 172.21.15.49:0/1384183034 wait complete. 2024-03-18T01:00:33.004 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T01:00:33.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.139+0000 7efdb15d4640 1 -- 172.21.15.49:0/2891799126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac100700 msgr2=0x7efdac100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.139+0000 7efdb15d4640 1 --2- 172.21.15.49:0/2891799126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac100700 0x7efdac100b20 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7efd9c009900 tx=0x7efd9c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:00:33.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.140+0000 7efdb15d4640 1 -- 172.21.15.49:0/2891799126 shutdown_connections 2024-03-18T01:00:33.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.140+0000 7efdb15d4640 1 --2- 172.21.15.49:0/2891799126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdac1018f0 0x7efdac101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.140+0000 7efdb15d4640 1 --2- 172.21.15.49:0/2891799126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdac1022b0 0x7efdac108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.140+0000 7efdb15d4640 1 --2- 172.21.15.49:0/2891799126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac100700 0x7efdac100b20 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.140+0000 7efdb15d4640 1 -- 172.21.15.49:0/2891799126 >> 172.21.15.49:0/2891799126 conn(0x7efdac0fbc90 msgr2=0x7efdac0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.140+0000 7efdb15d4640 1 -- 172.21.15.49:0/2891799126 shutdown_connections 2024-03-18T01:00:33.140 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.140+0000 7efdb15d4640 1 -- 172.21.15.49:0/2891799126 wait complete. 2024-03-18T01:00:33.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.140+0000 7efdb15d4640 1 Processor -- start 2024-03-18T01:00:33.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.140+0000 7efdb15d4640 1 -- start start 2024-03-18T01:00:33.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdb15d4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdac100700 0x7efdac06a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdb15d4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdac1018f0 0x7efdac19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdb15d4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac1022b0 0x7efdac19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdb15d4640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efdac10c3f0 con 0x7efdac1022b0 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdb15d4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efdac10c270 con 0x7efdac1018f0 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdb15d4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efdac10c570 con 0x7efdac100700 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdab7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac1022b0 0x7efdac19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdab7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac1022b0 0x7efdac19ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37484/0 (socket says 172.21.15.49:37484) 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdab7fe640 1 -- 172.21.15.49:0/532162682 learned_addr learned my addr 172.21.15.49:0/532162682 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdaaffd640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdac100700 0x7efdac06a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efda3fff640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdac1018f0 0x7efdac19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdab7fe640 1 -- 172.21.15.49:0/532162682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdac100700 msgr2=0x7efdac06a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdab7fe640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdac100700 0x7efdac06a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdab7fe640 1 -- 172.21.15.49:0/532162682 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdac1018f0 msgr2=0x7efdac19c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdab7fe640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdac1018f0 0x7efdac19c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdab7fe640 1 -- 172.21.15.49:0/532162682 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdac19f7e0 con 0x7efdac1022b0 2024-03-18T01:00:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdab7fe640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac1022b0 0x7efdac19ef20 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7efda400eb10 tx=0x7efda400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efda8ff9640 1 -- 172.21.15.49:0/532162682 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda400ce20 con 0x7efdac1022b0 2024-03-18T01:00:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efdb15d4640 1 -- 172.21.15.49:0/532162682 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdac19fad0 con 0x7efdac1022b0 2024-03-18T01:00:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efda8ff9640 1 -- 172.21.15.49:0/532162682 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efda4005020 con 0x7efdac1022b0 2024-03-18T01:00:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.141+0000 7efda8ff9640 1 -- 172.21.15.49:0/532162682 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4010690 con 0x7efdac1022b0 2024-03-18T01:00:33.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.142+0000 7efdb15d4640 1 -- 172.21.15.49:0/532162682 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdac1a0020 con 0x7efdac1022b0 2024-03-18T01:00:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.144+0000 7efda8ff9640 1 -- 172.21.15.49:0/532162682 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efda40108f0 con 0x7efdac1022b0 2024-03-18T01:00:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.144+0000 7efdb41c1740 1 Processor -- start 2024-03-18T01:00:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.144+0000 7efdb41c1740 1 -- start start 2024-03-18T01:00:33.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.144+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f79c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.144+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f7ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.144+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807f81840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.144+0000 7efdb41c1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f807ede490 con 0x55f807f79850 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.144+0000 7efdb41c1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f807e86c90 con 0x55f807f7a240 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.144+0000 7efdb41c1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f807eda530 con 0x55f807f7f3b0 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efda3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f79c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efdaaffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f7ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efda3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f79c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37494/0 (socket says 172.21.15.49:37494) 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efda3fff640 1 -- 172.21.15.49:0/3055305067 learned_addr learned my addr 172.21.15.49:0/3055305067 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efda3fff640 1 --2- 172.21.15.49:0/3055305067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807f81840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efdaaffd640 1 -- 172.21.15.49:0/3055305067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 msgr2=0x55f807f81840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efdaaffd640 1 --2- 172.21.15.49:0/3055305067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807f81840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efdaaffd640 1 -- 172.21.15.49:0/3055305067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 msgr2=0x55f807f79c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efdaaffd640 1 --2- 172.21.15.49:0/3055305067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f79c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efdaaffd640 1 -- 172.21.15.49:0/3055305067 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f807f41500 con 0x55f807f7a240 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efda8ff9640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd980a2800 0x7efd980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efda3fff640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd980a2800 0x7efd980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efdaaffd640 1 --2- 172.21.15.49:0/3055305067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f7ed40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efd9c02c170 tx=0x7efd9c02c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e85a1e23477c5f96 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efda8ff9640 1 -- 172.21.15.49:0/532162682 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7efda40cf150 con 0x7efdac1022b0 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efda17fa640 1 -- 172.21.15.49:0/3055305067 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c042070 con 0x55f807f7a240 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.145+0000 7efda17fa640 1 -- 172.21.15.49:0/3055305067 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd9c02cc40 con 0x55f807f7a240 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.146+0000 7efda17fa640 1 -- 172.21.15.49:0/3055305067 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c004920 con 0x55f807f7a240 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.146+0000 7efdb41c1740 1 -- 172.21.15.49:0/3055305067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 msgr2=0x55f807f7ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.146+0000 7efdb41c1740 1 --2- 172.21.15.49:0/3055305067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f7ed40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efd9c02c170 tx=0x7efd9c02c6a0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.146+0000 7efda3fff640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd980a2800 0x7efd980a4cc0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x55f807f7aaa0 tx=0x7efd94008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.147+0000 7efdb41c1740 1 -- 172.21.15.49:0/3055305067 shutdown_connections 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.147+0000 7efdb41c1740 1 --2- 172.21.15.49:0/3055305067 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f7ed40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.147+0000 7efdb41c1740 1 --2- 172.21.15.49:0/3055305067 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807f81840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.147+0000 7efdb41c1740 1 --2- 172.21.15.49:0/3055305067 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f79c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.147+0000 7efdb41c1740 1 -- 172.21.15.49:0/3055305067 >> 172.21.15.49:0/3055305067 conn(0x55f807f74dc0 msgr2=0x55f807f751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.147+0000 7efdb41c1740 1 -- 172.21.15.49:0/3055305067 shutdown_connections 2024-03-18T01:00:33.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.147+0000 7efdb41c1740 1 -- 172.21.15.49:0/3055305067 wait complete. 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.147+0000 7efdb41c1740 1 -- start start 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.147+0000 7efdb41c1740 1 Processor -- start 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f84230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807f8b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdb41c1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55f807f82300 con 0x55f807f79850 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdb41c1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55f807e9a230 con 0x55f807f7a240 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdb41c1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55f807f82480 con 0x55f807f7f3b0 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdb41c1740 10 client.0 _ll_register_callbacks cb 0x55f807ed6010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdaaffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.152 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efda3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f84230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdaaffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37510/0 (socket says 172.21.15.49:37510) 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdaaffd640 1 -- 172.21.15.49:0/998509957 learned_addr learned my addr 172.21.15.49:0/998509957 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efda3fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f84230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:55878/0 (socket says 172.21.15.49:55878) 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdab7fe640 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807f8b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdaaffd640 1 -- 172.21.15.49:0/998509957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 msgr2=0x55f807f8b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdaaffd640 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807f8b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdaaffd640 1 -- 172.21.15.49:0/998509957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 msgr2=0x55f807f84230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdaaffd640 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f84230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdaaffd640 1 -- 172.21.15.49:0/998509957 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f807ea9100 con 0x55f807f79850 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efda3fff640 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f84230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efdaaffd640 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83cf0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7efd9c036040 tx=0x7efd9c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e934945354a070 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.148+0000 7efd927fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.149+0000 7efd927fc640 1 -- 172.21.15.49:0/998509957 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c042040 con 0x55f807f79850 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.149+0000 7efd927fc640 1 -- 172.21.15.49:0/998509957 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7efd9c0046c0 con 0x55f807f79850 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.149+0000 7efdb41c1740 1 -- 172.21.15.49:0/998509957 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd9c02cc40 con 0x55f807f79850 2024-03-18T01:00:33.153 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.149+0000 7efd927fc640 1 -- 172.21.15.49:0/998509957 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efd9c055050 con 0x55f807f79850 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.149+0000 7efdb41c1740 10 client.17934 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.149+0000 7efdb41c1740 1 -- 172.21.15.49:0/998509957 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f807bf23f0 con 0x55f807f79850 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efd927fc640 1 -- 172.21.15.49:0/998509957 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd9c02cc40 con 0x55f807f79850 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efd927fc640 1 -- 172.21.15.49:0/998509957 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7efd9c003820 con 0x55f807f79850 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efdb41c1740 10 client.17934 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efdb41c1740 10 client.17934 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efdb41c1740 20 client.17934 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efdb41c1740 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f807f8f750 0x55f807f91b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efdb41c1740 4 client.17934 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efdb41c1740 1 -- 172.21.15.49:0/998509957 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f807eeb9d0 con 0x55f807f8f750 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efda3fff640 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f807f8f750 0x55f807f91b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.154 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efda3fff640 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f807f8f750 0x55f807f91b40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e2a6165c817a9ab server_cookie=2753ab532edbaaa1 in_seq=0 out_seq=0 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.150+0000 7efd927fc640 10 client.17934 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.151+0000 7efd927fc640 1 -- 172.21.15.49:0/998509957 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f807eeb9d0 con 0x55f807f8f750 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.151+0000 7efd927fc640 10 client.17934 handle_command_reply: tid=0 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.151+0000 7efdb41c1740 1 client.17934 shutdown 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 -- 172.21.15.49:0/998509957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 msgr2=0x55f807f83cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83cf0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7efd9c036040 tx=0x7efd9c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 -- 172.21.15.49:0/998509957 shutdown_connections 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f807f8f750 0x55f807f91b40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7a240 0x55f807f84230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807f8b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 --2- 172.21.15.49:0/998509957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83cf0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efd927fc640 0 client.17934 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 -- 172.21.15.49:0/998509957 >> 172.21.15.49:0/998509957 conn(0x55f807f74dc0 msgr2=0x55f807f73fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 -- 172.21.15.49:0/998509957 shutdown_connections 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 1 -- 172.21.15.49:0/998509957 wait complete. 2024-03-18T01:00:33.155 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.152+0000 7efdb41c1740 20 client.17934 trim_cache size 0 max 16384 2024-03-18T01:00:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.168+0000 7efdb41c1740 1 Processor -- start 2024-03-18T01:00:33.168 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.168+0000 7efdb41c1740 1 -- start start 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7a240 0x55f807f8b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7f3b0 0x55f807f83850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdb41c1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efdac1aaf70 con 0x55f807f79850 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdb41c1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efdac1acae0 con 0x55f807f7f3b0 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdb41c1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efdac1ab640 con 0x55f807f7a240 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdaaffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdaaffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37512/0 (socket says 172.21.15.49:37512) 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdab7fe640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7f3b0 0x55f807f83850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdaaffd640 1 -- 172.21.15.49:0/4189879305 learned_addr learned my addr 172.21.15.49:0/4189879305 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efda3fff640 1 --2- 172.21.15.49:0/4189879305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7a240 0x55f807f8b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdaaffd640 1 -- 172.21.15.49:0/4189879305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7a240 msgr2=0x55f807f8b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdaaffd640 1 --2- 172.21.15.49:0/4189879305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7a240 0x55f807f8b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdaaffd640 1 -- 172.21.15.49:0/4189879305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7f3b0 msgr2=0x55f807f83850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdaaffd640 1 --2- 172.21.15.49:0/4189879305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7f3b0 0x55f807f83850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.169+0000 7efdaaffd640 1 -- 172.21.15.49:0/4189879305 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f807cbc7d0 con 0x55f807f79850 2024-03-18T01:00:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.170+0000 7efdab7fe640 1 --2- 172.21.15.49:0/4189879305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7f3b0 0x55f807f83850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.170+0000 7efdaaffd640 1 --2- 172.21.15.49:0/4189879305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83310 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7efd9c001eb0 tx=0x7efd9c005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4b7dcb9b34df51 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.170+0000 7efda0ff9640 1 -- 172.21.15.49:0/4189879305 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c042070 con 0x55f807f79850 2024-03-18T01:00:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.170+0000 7efda0ff9640 1 -- 172.21.15.49:0/4189879305 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd9c034440 con 0x55f807f79850 2024-03-18T01:00:33.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.170+0000 7efda0ff9640 1 -- 172.21.15.49:0/4189879305 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c003970 con 0x55f807f79850 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 -- 172.21.15.49:0/4189879305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 msgr2=0x55f807f83310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 --2- 172.21.15.49:0/4189879305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83310 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7efd9c001eb0 tx=0x7efd9c005bd0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 -- 172.21.15.49:0/4189879305 shutdown_connections 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 --2- 172.21.15.49:0/4189879305 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f7f3b0 0x55f807f83850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 --2- 172.21.15.49:0/4189879305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7a240 0x55f807f8b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 --2- 172.21.15.49:0/4189879305 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f79850 0x55f807f83310 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 -- 172.21.15.49:0/4189879305 >> 172.21.15.49:0/4189879305 conn(0x55f807f74dc0 msgr2=0x55f807f78020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 -- 172.21.15.49:0/4189879305 shutdown_connections 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 -- 172.21.15.49:0/4189879305 wait complete. 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 -- start start 2024-03-18T01:00:33.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 Processor -- start 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f79850 0x55f807f881b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f7a240 0x55f807f886f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807ffc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7efdac1ac3e0 con 0x55f807f7a240 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7efdac1abd10 con 0x55f807f79850 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7efdac1a8f50 con 0x55f807f7f3b0 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efdb41c1740 10 client.0 _ll_register_callbacks cb 0x7efd94009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efda3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f7a240 0x55f807f886f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efda3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f7a240 0x55f807f886f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37526/0 (socket says 172.21.15.49:37526) 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.171+0000 7efda3fff640 1 -- 172.21.15.49:0/1887094955 learned_addr learned my addr 172.21.15.49:0/1887094955 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efdaaffd640 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f79850 0x55f807f881b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efda3fff640 1 -- 172.21.15.49:0/1887094955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 msgr2=0x55f807ffc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efda3fff640 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807ffc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efda3fff640 1 -- 172.21.15.49:0/1887094955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f79850 msgr2=0x55f807f881b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efda3fff640 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f79850 0x55f807f881b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efda3fff640 1 -- 172.21.15.49:0/1887094955 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f807ffc760 con 0x55f807f7a240 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efdaaffd640 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f79850 0x55f807f881b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efda3fff640 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f7a240 0x55f807f886f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7efd94017e90 tx=0x7efd94009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9e4326ee1c81c8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efd927fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efd927fc640 1 -- 172.21.15.49:0/1887094955 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9400f040 con 0x55f807f7a240 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efd927fc640 1 -- 172.21.15.49:0/1887094955 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7efd940269b0 con 0x55f807f7a240 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efdb41c1740 1 -- 172.21.15.49:0/1887094955 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd9c034440 con 0x55f807f7a240 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efd927fc640 1 -- 172.21.15.49:0/1887094955 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efd94026bb0 con 0x55f807f7a240 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efdb41c1740 10 client.17940 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.172+0000 7efdb41c1740 1 -- 172.21.15.49:0/1887094955 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f807ffdad0 con 0x55f807f7a240 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.173+0000 7efd927fc640 1 -- 172.21.15.49:0/1887094955 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd9c034440 con 0x55f807f7a240 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.173+0000 7efd927fc640 1 -- 172.21.15.49:0/1887094955 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7efd94015040 con 0x55f807f7a240 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.173+0000 7efdb41c1740 10 client.17940 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.173+0000 7efdb41c1740 10 client.17940 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T01:00:33.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.173+0000 7efdb41c1740 20 client.17940 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.173+0000 7efdb41c1740 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f807ffe4b0 0x55f8080008d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.173+0000 7efdb41c1740 4 client.17940 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:00:33.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.173+0000 7efdb41c1740 1 -- 172.21.15.49:0/1887094955 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f807ee5d20 con 0x55f807ffe4b0 2024-03-18T01:00:33.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.174+0000 7efdaaffd640 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f807ffe4b0 0x55f8080008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.174+0000 7efd927fc640 10 client.17940 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.174+0000 7efdaaffd640 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f807ffe4b0 0x55f8080008d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=414c30b65cf4802f server_cookie=b7fdc6108bb87c30 in_seq=0 out_seq=0 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efd927fc640 1 -- 172.21.15.49:0/1887094955 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f807ee5d20 con 0x55f807ffe4b0 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efd927fc640 10 client.17940 handle_command_reply: tid=0 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 client.17940 shutdown 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 -- 172.21.15.49:0/1887094955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f7a240 msgr2=0x55f807f886f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f7a240 0x55f807f886f0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7efd94017e90 tx=0x7efd94009520 comp rx=0 tx=0).stop 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 -- 172.21.15.49:0/1887094955 shutdown_connections 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x55f807ffe4b0 0x55f8080008d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.178 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f807f7f3b0 0x55f807ffc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55f807f7a240 0x55f807f886f0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efd927fc640 0 client.17940 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T01:00:33.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 --2- 172.21.15.49:0/1887094955 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55f807f79850 0x55f807f881b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 -- 172.21.15.49:0/1887094955 >> 172.21.15.49:0/1887094955 conn(0x55f807f74dc0 msgr2=0x55f807f73750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 -- 172.21.15.49:0/1887094955 shutdown_connections 2024-03-18T01:00:33.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 1 -- 172.21.15.49:0/1887094955 wait complete. 2024-03-18T01:00:33.179 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.175+0000 7efdb41c1740 20 client.17940 trim_cache size 0 max 16384 2024-03-18T01:00:33.179 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:00:33.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.176+0000 7efdb15d4640 1 -- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd980a2800 msgr2=0x7efd980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.176+0000 7efdb15d4640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd980a2800 0x7efd980a4cc0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x55f807f7aaa0 tx=0x7efd94008040 comp rx=0 tx=0).stop 2024-03-18T01:00:33.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 -- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac1022b0 msgr2=0x7efdac19ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac1022b0 0x7efdac19ef20 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7efda400eb10 tx=0x7efda400ef10 comp rx=0 tx=0).stop 2024-03-18T01:00:33.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 -- 172.21.15.49:0/532162682 shutdown_connections 2024-03-18T01:00:33.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7efd980a2800 0x7efd980a4cc0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7efdac1018f0 0x7efdac19c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7efdac1022b0 0x7efdac19ef20 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 --2- 172.21.15.49:0/532162682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efdac100700 0x7efdac06a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 -- 172.21.15.49:0/532162682 >> 172.21.15.49:0/532162682 conn(0x7efdac0fbc90 msgr2=0x7efdac104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 -- 172.21.15.49:0/532162682 shutdown_connections 2024-03-18T01:00:33.181 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.177+0000 7efdb15d4640 1 -- 172.21.15.49:0/532162682 wait complete. 2024-03-18T01:00:33.193 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T01:00:33.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.333+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/1664623584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ed01022a0 msgr2=0x7f5ed0108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.333+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/1664623584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ed01022a0 0x7f5ed0108b30 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8009be0 tx=0x7f5ec802bef0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.333+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/1664623584 shutdown_connections 2024-03-18T01:00:33.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.333+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/1664623584 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ed01022a0 0x7f5ed0108b30 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.333+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/1664623584 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ed01018e0 0x7f5ed0101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.333+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/1664623584 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed01006f0 0x7f5ed0100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.333+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/1664623584 >> 172.21.15.49:0/1664623584 conn(0x7f5ed00fbc80 msgr2=0x7f5ed00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.333+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/1664623584 shutdown_connections 2024-03-18T01:00:33.334 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.333+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/1664623584 wait complete. 2024-03-18T01:00:33.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.334+0000 7f5ed4ccc640 1 Processor -- start 2024-03-18T01:00:33.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.334+0000 7f5ed4ccc640 1 -- start start 2024-03-18T01:00:33.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ed4ccc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ed01006f0 0x7f5ed01969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ed4ccc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed01018e0 0x7f5ed0196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ed4ccc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ed01022a0 0x7f5ed019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ed4ccc640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed010c370 con 0x7f5ed01022a0 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ed4ccc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed010c1f0 con 0x7f5ed01018e0 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ed4ccc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed010c4f0 con 0x7f5ed01006f0 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ece59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ed01006f0 0x7f5ed01969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ec5d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed01018e0 0x7f5ed0196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ece59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ed01006f0 0x7f5ed01969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59550/0 (socket says 172.21.15.49:59550) 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ece59c640 1 -- 172.21.15.49:0/403063635 learned_addr learned my addr 172.21.15.49:0/403063635 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5eced9d640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ed01022a0 0x7f5ed019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ec5d9b640 1 -- 172.21.15.49:0/403063635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ed01006f0 msgr2=0x7f5ed01969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ed01006f0 0x7f5ed01969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ec5d9b640 1 -- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ed01022a0 msgr2=0x7f5ed019df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ed01022a0 0x7f5ed019df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.335+0000 7f5ec5d9b640 1 -- 172.21.15.49:0/403063635 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed0069e50 con 0x7f5ed01018e0 2024-03-18T01:00:33.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.336+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed01018e0 0x7f5ed0196f20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800e9f0 tx=0x7f5eb800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.336+0000 7f5ec7fff640 1 -- 172.21.15.49:0/403063635 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb800ccc0 con 0x7f5ed01018e0 2024-03-18T01:00:33.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.336+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/403063635 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed006a140 con 0x7f5ed01018e0 2024-03-18T01:00:33.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.336+0000 7f5ec7fff640 1 -- 172.21.15.49:0/403063635 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb8005020 con 0x7f5ed01018e0 2024-03-18T01:00:33.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.336+0000 7f5ec7fff640 1 -- 172.21.15.49:0/403063635 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8004a00 con 0x7f5ed01018e0 2024-03-18T01:00:33.337 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.336+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/403063635 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed006a690 con 0x7f5ed01018e0 2024-03-18T01:00:33.338 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.339+0000 7f5ec7fff640 1 -- 172.21.15.49:0/403063635 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5eb80051d0 con 0x7f5ed01018e0 2024-03-18T01:00:33.339 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.339+0000 7f5ed798b740 1 Processor -- start 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ed798b740 1 -- start start 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21a850 0x558c3d21ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d21b240 0x558c3d21fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ec7fff640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5ebc0a2850 0x7f5ebc0a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d2203b0 0x558c3d222840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ed798b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558c3d17f490 con 0x558c3d2203b0 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ed798b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558c3d127c90 con 0x558c3d21a850 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ed798b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558c3d17b530 con 0x558c3d21b240 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ece59c640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5ebc0a2850 0x7f5ebc0a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ec5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d21b240 0x558c3d21fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ec5d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d21b240 0x558c3d21fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59556/0 (socket says 172.21.15.49:59556) 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5eced9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21a850 0x558c3d21ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.341 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ec5d9b640 1 -- 172.21.15.49:0/1211967897 learned_addr learned my addr 172.21.15.49:0/1211967897 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ec7fff640 1 -- 172.21.15.49:0/403063635 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5eb80cebc0 con 0x7f5ed01018e0 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5eced9d640 1 --2- 172.21.15.49:0/1211967897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d2203b0 0x558c3d222840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ec5d9b640 1 -- 172.21.15.49:0/1211967897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21a850 msgr2=0x558c3d21ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.340+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/1211967897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21a850 0x558c3d21ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.341+0000 7f5ec5d9b640 1 -- 172.21.15.49:0/1211967897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d2203b0 msgr2=0x558c3d222840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.341+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/1211967897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d2203b0 0x558c3d222840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.341+0000 7f5ec5d9b640 1 -- 172.21.15.49:0/1211967897 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c3d1e2500 con 0x558c3d21b240 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.341+0000 7f5eced9d640 1 --2- 172.21.15.49:0/1211967897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21a850 0x558c3d21ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.341+0000 7f5ece59c640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5ebc0a2850 0x7f5ebc0a4d10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0009c60 tx=0x7f5ec0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.341+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/1211967897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d21b240 0x558c3d21fd40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5eb809b4d0 tx=0x7f5eb8090a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b939b4b0123a936 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.341+0000 7f5ec4d99640 1 -- 172.21.15.49:0/1211967897 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb809aea0 con 0x558c3d21b240 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.341+0000 7f5ec4d99640 1 -- 172.21.15.49:0/1211967897 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb809bc40 con 0x558c3d21b240 2024-03-18T01:00:33.342 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.341+0000 7f5ec4d99640 1 -- 172.21.15.49:0/1211967897 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8099840 con 0x558c3d21b240 2024-03-18T01:00:33.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 -- 172.21.15.49:0/1211967897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d21b240 msgr2=0x558c3d21fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1211967897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d21b240 0x558c3d21fd40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5eb809b4d0 tx=0x7f5eb8090a20 comp rx=0 tx=0).stop 2024-03-18T01:00:33.346 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 -- 172.21.15.49:0/1211967897 shutdown_connections 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1211967897 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d2203b0 0x558c3d222840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1211967897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d21b240 0x558c3d21fd40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1211967897 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21a850 0x558c3d21ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 -- 172.21.15.49:0/1211967897 >> 172.21.15.49:0/1211967897 conn(0x558c3d215dc0 msgr2=0x558c3d2161c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 -- 172.21.15.49:0/1211967897 shutdown_connections 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 -- 172.21.15.49:0/1211967897 wait complete. 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 -- start start 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.342+0000 7f5ed798b740 1 Processor -- start 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d224da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d2252e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d22c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ed798b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558c3d223360 con 0x558c3d21a850 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ed798b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558c3d13b230 con 0x558c3d21b240 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ed798b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558c3d2234e0 con 0x558c3d2203b0 2024-03-18T01:00:33.347 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ed798b740 10 client.0 _ll_register_callbacks cb 0x558c3d177010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ece59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d224da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ece59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d224da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37568/0 (socket says 172.21.15.49:37568) 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ece59c640 1 -- 172.21.15.49:0/3750942318 learned_addr learned my addr 172.21.15.49:0/3750942318 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d2252e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5eced9d640 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d22c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ece59c640 1 -- 172.21.15.49:0/3750942318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 msgr2=0x558c3d22c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ece59c640 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d22c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ece59c640 1 -- 172.21.15.49:0/3750942318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 msgr2=0x558c3d2252e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ece59c640 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d2252e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ece59c640 1 -- 172.21.15.49:0/3750942318 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c3d14a100 con 0x558c3d21a850 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ece59c640 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d224da0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00045f0 tx=0x7f5ec0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36802185d231fdf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5eb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/3750942318 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0026470 con 0x558c3d21a850 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/3750942318 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5ec0026610 con 0x558c3d21a850 2024-03-18T01:00:33.348 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.343+0000 7f5ed798b740 1 -- 172.21.15.49:0/3750942318 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5eb809bc40 con 0x558c3d21a850 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/3750942318 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5ec002c070 con 0x558c3d21a850 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5ed798b740 10 client.17952 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5ed798b740 1 -- 172.21.15.49:0/3750942318 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558c3ce933f0 con 0x558c3d21a850 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/3750942318 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5eb809bc40 con 0x558c3d21a850 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/3750942318 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f5ec001ca10 con 0x558c3d21a850 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5ed798b740 10 client.17952 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5ed798b740 10 client.17952 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5ed798b740 20 client.17952 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5ed798b740 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x558c3d230c40 0x558c3d233100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5ed798b740 4 client.17952 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5ed798b740 1 -- 172.21.15.49:0/3750942318 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c3d18c9d0 con 0x558c3d230c40 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.344+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x558c3d230c40 0x558c3d233100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.345+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x558c3d230c40 0x558c3d233100 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a16c178df94eea2b server_cookie=703ca21ff87ef89f in_seq=0 out_seq=0 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.345+0000 7f5eb5ffb640 10 client.17952 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.345+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/3750942318 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558c3d18c9d0 con 0x558c3d230c40 2024-03-18T01:00:33.349 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.345+0000 7f5eb5ffb640 10 client.17952 handle_command_reply: tid=0 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.345+0000 7f5ed798b740 1 client.17952 shutdown 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.345+0000 7f5ed798b740 1 -- 172.21.15.49:0/3750942318 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 msgr2=0x558c3d224da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.345+0000 7f5ed798b740 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d224da0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00045f0 tx=0x7f5ec0004730 comp rx=0 tx=0).stop 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5ed798b740 1 -- 172.21.15.49:0/3750942318 shutdown_connections 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5ed798b740 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x558c3d230c40 0x558c3d233100 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5ed798b740 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d2252e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5ed798b740 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d22c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5eb5ffb640 0 client.17952 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5ed798b740 1 --2- 172.21.15.49:0/3750942318 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d224da0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5ed798b740 1 -- 172.21.15.49:0/3750942318 >> 172.21.15.49:0/3750942318 conn(0x558c3d215dc0 msgr2=0x558c3d214fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5ed798b740 1 -- 172.21.15.49:0/3750942318 shutdown_connections 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5ed798b740 1 -- 172.21.15.49:0/3750942318 wait complete. 2024-03-18T01:00:33.350 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.346+0000 7f5ed798b740 20 client.17952 trim_cache size 0 max 16384 2024-03-18T01:00:33.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.360+0000 7f5ed798b740 1 Processor -- start 2024-03-18T01:00:33.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.360+0000 7f5ed798b740 1 -- start start 2024-03-18T01:00:33.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d225240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d22c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d214e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ed798b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed01a8de0 con 0x558c3d21a850 2024-03-18T01:00:33.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ed798b740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed01aa950 con 0x558c3d21b240 2024-03-18T01:00:33.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ed798b740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed01a94b0 con 0x558c3d2203b0 2024-03-18T01:00:33.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ece59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d225240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ece59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d225240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37578/0 (socket says 172.21.15.49:37578) 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ece59c640 1 -- 172.21.15.49:0/844594471 learned_addr learned my addr 172.21.15.49:0/844594471 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d22c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5eced9d640 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d214e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ece59c640 1 -- 172.21.15.49:0/844594471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 msgr2=0x558c3d214e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ece59c640 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d214e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ece59c640 1 -- 172.21.15.49:0/844594471 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 msgr2=0x558c3d22c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ece59c640 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d22c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ece59c640 1 -- 172.21.15.49:0/844594471 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c3d211640 con 0x558c3d21a850 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5eced9d640 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d214e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5ece59c640 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d225240 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00058c0 tx=0x7f5ec0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c019f01ed0aecf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5eb7fff640 1 -- 172.21.15.49:0/844594471 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec001d510 con 0x558c3d21a850 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5eb7fff640 1 -- 172.21.15.49:0/844594471 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ec0021070 con 0x558c3d21a850 2024-03-18T01:00:33.362 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.361+0000 7f5eb7fff640 1 -- 172.21.15.49:0/844594471 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec001ca70 con 0x558c3d21a850 2024-03-18T01:00:33.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.362+0000 7f5ed798b740 1 -- 172.21.15.49:0/844594471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 msgr2=0x558c3d225240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.362+0000 7f5ed798b740 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d225240 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00058c0 tx=0x7f5ec0011da0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.362+0000 7f5ed798b740 1 -- 172.21.15.49:0/844594471 shutdown_connections 2024-03-18T01:00:33.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.362+0000 7f5ed798b740 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d22c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.362+0000 7f5ed798b740 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d214e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.362+0000 7f5ed798b740 1 --2- 172.21.15.49:0/844594471 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d225240 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.362+0000 7f5ed798b740 1 -- 172.21.15.49:0/844594471 >> 172.21.15.49:0/844594471 conn(0x558c3d215dc0 msgr2=0x558c3d214750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.362+0000 7f5ed798b740 1 -- 172.21.15.49:0/844594471 shutdown_connections 2024-03-18T01:00:33.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.362+0000 7f5ed798b740 1 -- 172.21.15.49:0/844594471 wait complete. 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ed798b740 1 -- start start 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ed798b740 1 Processor -- start 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d229530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d229a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ed798b740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d22aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ed798b740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed01aa250 con 0x558c3d21a850 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ece59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d229530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ece59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d229530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37590/0 (socket says 172.21.15.49:37590) 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ece59c640 1 -- 172.21.15.49:0/1107017593 learned_addr learned my addr 172.21.15.49:0/1107017593 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed01a9b80 con 0x558c3d21b240 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed01a6dc0 con 0x558c3d2203b0 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ed798b740 10 client.0 _ll_register_callbacks cb 0x7f5ec00098b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.363+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d229a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5ece59c640 1 -- 172.21.15.49:0/1107017593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 msgr2=0x558c3d22aa80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5ece59c640 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d22aa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5ece59c640 1 -- 172.21.15.49:0/1107017593 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 msgr2=0x558c3d229a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5ece59c640 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d229a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5ece59c640 1 -- 172.21.15.49:0/1107017593 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c3cf5d7d0 con 0x558c3d21a850 2024-03-18T01:00:33.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d229a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5ece59c640 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d229530 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0004630 tx=0x7f5ec001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca270f226c285b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5eb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:33.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/1107017593 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ec003e850 con 0x558c3d21a850 2024-03-18T01:00:33.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.364+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/1107017593 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f5ec003e9f0 con 0x558c3d21a850 2024-03-18T01:00:33.366 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.365+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ec0021070 con 0x558c3d21a850 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.365+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/1107017593 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5ec003ebf0 con 0x558c3d21a850 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.365+0000 7f5ed798b740 10 client.17958 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.365+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558c3d22c1a0 con 0x558c3d21a850 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.365+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/1107017593 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ec0021070 con 0x558c3d21a850 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/1107017593 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f5ec0031480 con 0x558c3d21a850 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5ed798b740 10 client.17958 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5ed798b740 10 client.17958 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5ed798b740 20 client.17958 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x558c3d297150 0x558c3d299570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5ed798b740 4 client.17958 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:00:33.368 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558c3d186d20 con 0x558c3d297150 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x558c3d297150 0x558c3d299570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5eb5ffb640 10 client.17958 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.366+0000 7f5ec5d9b640 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x558c3d297150 0x558c3d299570 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6415426b4f23a2d5 server_cookie=8bcca51421a34eb9 in_seq=0 out_seq=0 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5eb5ffb640 1 -- 172.21.15.49:0/1107017593 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558c3d186d20 con 0x558c3d297150 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5eb5ffb640 10 client.17958 handle_command_reply: tid=0 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 client.17958 shutdown 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 msgr2=0x558c3d229530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d229530 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0004630 tx=0x7f5ec001c4b0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 shutdown_connections 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x558c3d297150 0x558c3d299570 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558c3d21b240 0x558c3d229a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558c3d2203b0 0x558c3d22aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5eb5ffb640 0 client.17958 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 --2- 172.21.15.49:0/1107017593 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558c3d21a850 0x558c3d229530 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 >> 172.21.15.49:0/1107017593 conn(0x558c3d215dc0 msgr2=0x558c3d2198b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.369 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 shutdown_connections 2024-03-18T01:00:33.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 1 -- 172.21.15.49:0/1107017593 wait complete. 2024-03-18T01:00:33.370 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.367+0000 7f5ed798b740 20 client.17958 trim_cache size 0 max 16384 2024-03-18T01:00:33.370 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5ebc0a2850 msgr2=0x7f5ebc0a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5ebc0a2850 0x7f5ebc0a4d10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5ec0009c60 tx=0x7f5ec0009260 comp rx=0 tx=0).stop 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/403063635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed01018e0 msgr2=0x7f5ed0196f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed01018e0 0x7f5ed0196f20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800e9f0 tx=0x7f5eb800eeb0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/403063635 shutdown_connections 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f5ebc0a2850 0x7f5ebc0a4d10 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f5ed01018e0 0x7f5ed0196f20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5ed01022a0 0x7f5ed019df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 --2- 172.21.15.49:0/403063635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5ed01006f0 0x7f5ed01969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/403063635 >> 172.21.15.49:0/403063635 conn(0x7f5ed00fbc80 msgr2=0x7f5ed0104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/403063635 shutdown_connections 2024-03-18T01:00:33.371 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.368+0000 7f5ed4ccc640 1 -- 172.21.15.49:0/403063635 wait complete. 2024-03-18T01:00:33.379 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.506+0000 7f7247fb9640 1 -- 172.21.15.49:0/1166314087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 msgr2=0x7f7240101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.506+0000 7f7247fb9640 1 --2- 172.21.15.49:0/1166314087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 0x7f7240101d50 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f7230009860 tx=0x7f723002bb70 comp rx=0 tx=0).stop 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 -- 172.21.15.49:0/1166314087 shutdown_connections 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 --2- 172.21.15.49:0/1166314087 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f7240102290 0x7f7240108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 --2- 172.21.15.49:0/1166314087 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 0x7f7240101d50 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 --2- 172.21.15.49:0/1166314087 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f72401006e0 0x7f7240100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 -- 172.21.15.49:0/1166314087 >> 172.21.15.49:0/1166314087 conn(0x7f72400fbc70 msgr2=0x7f72400fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 -- 172.21.15.49:0/1166314087 shutdown_connections 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 -- 172.21.15.49:0/1166314087 wait complete. 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 Processor -- start 2024-03-18T01:00:33.508 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 -- start start 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72401006e0 0x7f7240197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 0x7f72401980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7240102290 0x7f724019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f724010c360 con 0x7f72401018d0 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f724010c1e0 con 0x7f72401006e0 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.507+0000 7f7247fb9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f724010c4e0 con 0x7f7240102290 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245554640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 0x7f72401980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245554640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 0x7f72401980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37616/0 (socket says 172.21.15.49:37616) 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245d55640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72401006e0 0x7f7240197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245554640 1 -- 172.21.15.49:0/4149341690 learned_addr learned my addr 172.21.15.49:0/4149341690 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7246556640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7240102290 0x7f724019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245554640 1 -- 172.21.15.49:0/4149341690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7240102290 msgr2=0x7f724019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245554640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7240102290 0x7f724019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245554640 1 -- 172.21.15.49:0/4149341690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72401006e0 msgr2=0x7f7240197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245554640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72401006e0 0x7f7240197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245554640 1 -- 172.21.15.49:0/4149341690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724019f750 con 0x7f72401018d0 2024-03-18T01:00:33.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7245554640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 0x7f72401980c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7230005940 tx=0x7f7230032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7236ffd640 1 -- 172.21.15.49:0/4149341690 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230041020 con 0x7f72401018d0 2024-03-18T01:00:33.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7236ffd640 1 -- 172.21.15.49:0/4149341690 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f723002c9f0 con 0x7f72401018d0 2024-03-18T01:00:33.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7247fb9640 1 -- 172.21.15.49:0/4149341690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f724019f9e0 con 0x7f72401018d0 2024-03-18T01:00:33.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7236ffd640 1 -- 172.21.15.49:0/4149341690 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230039040 con 0x7f72401018d0 2024-03-18T01:00:33.510 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.508+0000 7f7247fb9640 1 -- 172.21.15.49:0/4149341690 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f724019fed0 con 0x7f72401018d0 2024-03-18T01:00:33.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.511+0000 7f724aba4740 1 Processor -- start 2024-03-18T01:00:33.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.511+0000 7f724aba4740 1 -- start start 2024-03-18T01:00:33.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.511+0000 7f7236ffd640 1 -- 172.21.15.49:0/4149341690 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7230044020 con 0x7f72401018d0 2024-03-18T01:00:33.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.511+0000 7f724aba4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb20bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.512 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.511+0000 7f724aba4740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 0x555bfb210d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.511+0000 7f724aba4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb213840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.511+0000 7f724aba4740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555bfb170490 con 0x555bfb20c240 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.511+0000 7f724aba4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555bfb118c90 con 0x555bfb20b850 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.511+0000 7f724aba4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555bfb16c530 con 0x555bfb2113b0 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7246556640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb20bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7245554640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb213840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7246556640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb20bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:55974/0 (socket says 172.21.15.49:55974) 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7246556640 1 -- 172.21.15.49:0/3789060261 learned_addr learned my addr 172.21.15.49:0/3789060261 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7245d55640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 0x555bfb210d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7245554640 1 -- 172.21.15.49:0/3789060261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 msgr2=0x555bfb20bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7245554640 1 --2- 172.21.15.49:0/3789060261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb20bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7245554640 1 -- 172.21.15.49:0/3789060261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 msgr2=0x555bfb210d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7245554640 1 --2- 172.21.15.49:0/3789060261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 0x555bfb210d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7245554640 1 -- 172.21.15.49:0/3789060261 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bfb1d3500 con 0x555bfb2113b0 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7245d55640 1 --2- 172.21.15.49:0/3789060261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 0x555bfb210d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.513 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7245554640 1 --2- 172.21.15.49:0/3789060261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb213840 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f72300ede80 tx=0x7f72300f8490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cff5948347ff9e51 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.512+0000 7f7236ffd640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f722c0a2800 0x7f722c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.513+0000 7f722bfff640 1 -- 172.21.15.49:0/3789060261 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72300b8ff0 con 0x555bfb2113b0 2024-03-18T01:00:33.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.513+0000 7f7246556640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f722c0a2800 0x7f722c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.513+0000 7f722bfff640 1 -- 172.21.15.49:0/3789060261 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72300fd030 con 0x555bfb2113b0 2024-03-18T01:00:33.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.513+0000 7f722bfff640 1 -- 172.21.15.49:0/3789060261 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72300bc1a0 con 0x555bfb2113b0 2024-03-18T01:00:33.514 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.513+0000 7f7236ffd640 1 -- 172.21.15.49:0/4149341690 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f72300f03d0 con 0x7f72401018d0 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 -- 172.21.15.49:0/3789060261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 msgr2=0x555bfb213840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f7246556640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f722c0a2800 0x7f722c0a4cc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x555bfb20caa0 tx=0x7f7238010040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 --2- 172.21.15.49:0/3789060261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb213840 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f72300ede80 tx=0x7f72300f8490 comp rx=0 tx=0).stop 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 -- 172.21.15.49:0/3789060261 shutdown_connections 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 --2- 172.21.15.49:0/3789060261 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb213840 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 --2- 172.21.15.49:0/3789060261 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 0x555bfb210d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 --2- 172.21.15.49:0/3789060261 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb20bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 -- 172.21.15.49:0/3789060261 >> 172.21.15.49:0/3789060261 conn(0x555bfb206dc0 msgr2=0x555bfb2071c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 -- 172.21.15.49:0/3789060261 shutdown_connections 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 -- 172.21.15.49:0/3789060261 wait complete. 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.514+0000 7f724aba4740 1 -- start start 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f724aba4740 1 Processor -- start 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f724aba4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb218f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.517 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f724aba4740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 0x555bfb219440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f724aba4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb2159a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f724aba4740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555bfb214120 con 0x555bfb20c240 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f724aba4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555bfb12c230 con 0x555bfb20b850 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f724aba4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555bfb2142a0 con 0x555bfb2113b0 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f724aba4740 10 client.0 _ll_register_callbacks cb 0x555bfb168010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f7245d55640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb218f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f7246556640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb2159a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f7245554640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 0x555bfb219440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f7246556640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb2159a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59616/0 (socket says 172.21.15.49:59616) 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f7245d55640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb218f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:55988/0 (socket says 172.21.15.49:55988) 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.515+0000 7f7246556640 1 -- 172.21.15.49:0/37888382 learned_addr learned my addr 172.21.15.49:0/37888382 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f7245d55640 1 -- 172.21.15.49:0/37888382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 msgr2=0x555bfb2159a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f7245d55640 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb2159a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f7245d55640 1 -- 172.21.15.49:0/37888382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 msgr2=0x555bfb219440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.518 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f7245d55640 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 0x555bfb219440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f7245d55640 1 -- 172.21.15.49:0/37888382 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bfb13b100 con 0x555bfb20b850 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f7246556640 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb2159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f7245d55640 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb218f00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x555bfb20c640 tx=0x7f723c012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82722ae420f56341 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f72297fa640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f72297fa640 1 -- 172.21.15.49:0/37888382 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c004030 con 0x555bfb20b850 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f72297fa640 1 -- 172.21.15.49:0/37888382 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f723c0041d0 con 0x555bfb20b850 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f724aba4740 1 -- 172.21.15.49:0/37888382 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72300fd030 con 0x555bfb20b850 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f72297fa640 1 -- 172.21.15.49:0/37888382 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f723c022070 con 0x555bfb20b850 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f724aba4740 10 client.27089 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.516+0000 7f724aba4740 1 -- 172.21.15.49:0/37888382 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555bfae843f0 con 0x555bfb20b850 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.517+0000 7f72297fa640 1 -- 172.21.15.49:0/37888382 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72300fd030 con 0x555bfb20b850 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.517+0000 7f72297fa640 1 -- 172.21.15.49:0/37888382 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f723c003a90 con 0x555bfb20b850 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.517+0000 7f724aba4740 10 client.27089 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.517+0000 7f724aba4740 10 client.27089 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T01:00:33.519 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.517+0000 7f724aba4740 20 client.27089 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.517+0000 7f724aba4740 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555bfb22a120 0x555bfb22c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.517+0000 7f724aba4740 4 client.27089 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:33.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.517+0000 7f724aba4740 1 -- 172.21.15.49:0/37888382 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bfb17d9d0 con 0x555bfb22a120 2024-03-18T01:00:33.520 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.517+0000 7f7245554640 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555bfb22a120 0x555bfb22c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.518+0000 7f7245554640 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555bfb22a120 0x555bfb22c5e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e29c94fda19eac8 server_cookie=832b62989e00dbac in_seq=0 out_seq=0 2024-03-18T01:00:33.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.518+0000 7f72297fa640 10 client.27089 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T01:00:33.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f72297fa640 1 -- 172.21.15.49:0/37888382 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555bfb17d9d0 con 0x555bfb22a120 2024-03-18T01:00:33.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f72297fa640 10 client.27089 handle_command_reply: tid=0 2024-03-18T01:00:33.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 client.27089 shutdown 2024-03-18T01:00:33.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 -- 172.21.15.49:0/37888382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 msgr2=0x555bfb218f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb218f00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x555bfb20c640 tx=0x7f723c012b60 comp rx=0 tx=0).stop 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 -- 172.21.15.49:0/37888382 shutdown_connections 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555bfb22a120 0x555bfb22c5e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb2113b0 0x555bfb2159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb20c240 0x555bfb219440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 --2- 172.21.15.49:0/37888382 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb218f00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 -- 172.21.15.49:0/37888382 >> 172.21.15.49:0/37888382 conn(0x555bfb206dc0 msgr2=0x555bfb205fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f72297fa640 0 client.27089 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 -- 172.21.15.49:0/37888382 shutdown_connections 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 1 -- 172.21.15.49:0/37888382 wait complete. 2024-03-18T01:00:33.522 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.519+0000 7f724aba4740 20 client.27089 trim_cache size 0 max 16384 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f724aba4740 1 Processor -- start 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f724aba4740 1 -- start start 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f724aba4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb20cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f724aba4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb20c240 0x555bfb20d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f724aba4740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb20dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f724aba4740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f72401a6b40 con 0x555bfb2113b0 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f724aba4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f72401a86b0 con 0x555bfb20b850 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f724aba4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f72401a7210 con 0x555bfb20c240 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f7246556640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb20dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f7246556640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb20dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37648/0 (socket says 172.21.15.49:37648) 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f7246556640 1 -- 172.21.15.49:0/618712807 learned_addr learned my addr 172.21.15.49:0/618712807 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f7245d55640 1 --2- 172.21.15.49:0/618712807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb20cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.538+0000 7f7245554640 1 --2- 172.21.15.49:0/618712807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb20c240 0x555bfb20d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.539 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.539+0000 7f7246556640 1 -- 172.21.15.49:0/618712807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb20c240 msgr2=0x555bfb20d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.539+0000 7f7246556640 1 --2- 172.21.15.49:0/618712807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb20c240 0x555bfb20d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.539+0000 7f7246556640 1 -- 172.21.15.49:0/618712807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 msgr2=0x555bfb20cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.539+0000 7f7246556640 1 --2- 172.21.15.49:0/618712807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb20cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.539+0000 7f7246556640 1 -- 172.21.15.49:0/618712807 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bfaf4e7d0 con 0x555bfb2113b0 2024-03-18T01:00:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.539+0000 7f7246556640 1 --2- 172.21.15.49:0/618712807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb20dbe0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7238004360 tx=0x7f7238016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7baa88a7033abc5e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.539+0000 7f722b7fe640 1 -- 172.21.15.49:0/618712807 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238004820 con 0x555bfb2113b0 2024-03-18T01:00:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.539+0000 7f722b7fe640 1 -- 172.21.15.49:0/618712807 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f723802b070 con 0x555bfb2113b0 2024-03-18T01:00:33.540 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.539+0000 7f722b7fe640 1 -- 172.21.15.49:0/618712807 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72380268e0 con 0x555bfb2113b0 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 -- 172.21.15.49:0/618712807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 msgr2=0x555bfb20dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 --2- 172.21.15.49:0/618712807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb20dbe0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f7238004360 tx=0x7f7238016000 comp rx=0 tx=0).stop 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 -- 172.21.15.49:0/618712807 shutdown_connections 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 --2- 172.21.15.49:0/618712807 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb20dbe0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 --2- 172.21.15.49:0/618712807 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb20c240 0x555bfb20d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 --2- 172.21.15.49:0/618712807 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20b850 0x555bfb20cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 -- 172.21.15.49:0/618712807 >> 172.21.15.49:0/618712807 conn(0x555bfb206dc0 msgr2=0x555bfb218ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 -- 172.21.15.49:0/618712807 shutdown_connections 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 -- 172.21.15.49:0/618712807 wait complete. 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 -- start start 2024-03-18T01:00:33.543 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.540+0000 7f724aba4740 1 Processor -- start 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f724aba4740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb20b850 0x555bfb23d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f724aba4740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20c240 0x555bfb23a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f724aba4740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb23a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f724aba4740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f72401a7fb0 con 0x555bfb2113b0 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f724aba4740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f72401a78e0 con 0x555bfb20c240 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f724aba4740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f72401a4b20 con 0x555bfb20b850 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f724aba4740 10 client.0 _ll_register_callbacks cb 0x7f723c009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f7246556640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb23a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f7246556640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb23a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37656/0 (socket says 172.21.15.49:37656) 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f7246556640 1 -- 172.21.15.49:0/694442940 learned_addr learned my addr 172.21.15.49:0/694442940 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f7245554640 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20c240 0x555bfb23a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f7246556640 1 -- 172.21.15.49:0/694442940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb20b850 msgr2=0x555bfb23d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f7246556640 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb20b850 0x555bfb23d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f7246556640 1 -- 172.21.15.49:0/694442940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20c240 msgr2=0x555bfb23a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.544 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f7246556640 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20c240 0x555bfb23a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.541+0000 7f7246556640 1 -- 172.21.15.49:0/694442940 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bfb28de90 con 0x555bfb2113b0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.542+0000 7f7246556640 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb23a550 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7238005bf0 tx=0x7f7238005c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b694ea39faea2b3d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.542+0000 7f72297fa640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.542+0000 7f72297fa640 1 -- 172.21.15.49:0/694442940 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7238004e50 con 0x555bfb2113b0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.542+0000 7f724aba4740 1 -- 172.21.15.49:0/694442940 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f723802b070 con 0x555bfb2113b0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.542+0000 7f72297fa640 1 -- 172.21.15.49:0/694442940 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7238026660 con 0x555bfb2113b0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.542+0000 7f72297fa640 1 -- 172.21.15.49:0/694442940 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7238004000 con 0x555bfb2113b0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.542+0000 7f724aba4740 10 client.17976 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.542+0000 7f724aba4740 1 -- 172.21.15.49:0/694442940 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555bfb28f0e0 con 0x555bfb2113b0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.542+0000 7f72297fa640 1 -- 172.21.15.49:0/694442940 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f723802b070 con 0x555bfb2113b0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.543+0000 7f72297fa640 1 -- 172.21.15.49:0/694442940 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f723800b040 con 0x555bfb2113b0 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.543+0000 7f724aba4740 10 client.17976 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.543+0000 7f724aba4740 10 client.17976 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.543+0000 7f724aba4740 20 client.17976 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.543+0000 7f724aba4740 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555bfb28fa20 0x555bfb291e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.545 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.543+0000 7f724aba4740 4 client.17976 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:00:33.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.543+0000 7f724aba4740 1 -- 172.21.15.49:0/694442940 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555bfb177d20 con 0x555bfb28fa20 2024-03-18T01:00:33.546 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.543+0000 7f7245d55640 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555bfb28fa20 0x555bfb291e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f72297fa640 10 client.17976 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f7245d55640 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555bfb28fa20 0x555bfb291e40 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=51b79d8c94b3ca88 server_cookie=76a82955d9c4b6d1 in_seq=0 out_seq=0 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f72297fa640 1 -- 172.21.15.49:0/694442940 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555bfb177d20 con 0x555bfb28fa20 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f72297fa640 10 client.17976 handle_command_reply: tid=0 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f724aba4740 1 client.17976 shutdown 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f724aba4740 1 -- 172.21.15.49:0/694442940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 msgr2=0x555bfb23a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f724aba4740 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb23a550 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f7238005bf0 tx=0x7f7238005c20 comp rx=0 tx=0).stop 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f724aba4740 1 -- 172.21.15.49:0/694442940 shutdown_connections 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f724aba4740 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x555bfb28fa20 0x555bfb291e40 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f724aba4740 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555bfb20c240 0x555bfb23a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f724aba4740 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555bfb2113b0 0x555bfb23a550 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f72297fa640 0 client.17976 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T01:00:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f724aba4740 1 --2- 172.21.15.49:0/694442940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555bfb20b850 0x555bfb23d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.544+0000 7f724aba4740 1 -- 172.21.15.49:0/694442940 >> 172.21.15.49:0/694442940 conn(0x555bfb206dc0 msgr2=0x555bfb218ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.546+0000 7f724aba4740 1 -- 172.21.15.49:0/694442940 shutdown_connections 2024-03-18T01:00:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.546+0000 7f724aba4740 1 -- 172.21.15.49:0/694442940 wait complete. 2024-03-18T01:00:33.548 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.546+0000 7f724aba4740 20 client.17976 trim_cache size 0 max 16384 2024-03-18T01:00:33.548 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:00:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 -- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f722c0a2800 msgr2=0x7f722c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f722c0a2800 0x7f722c0a4cc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x555bfb20caa0 tx=0x7f7238010040 comp rx=0 tx=0).stop 2024-03-18T01:00:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 -- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 msgr2=0x7f72401980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 0x7f72401980c0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7230005940 tx=0x7f7230032470 comp rx=0 tx=0).stop 2024-03-18T01:00:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 -- 172.21.15.49:0/4149341690 shutdown_connections 2024-03-18T01:00:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f722c0a2800 0x7f722c0a4cc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.549 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7240102290 0x7f724019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f72401018d0 0x7f72401980c0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 --2- 172.21.15.49:0/4149341690 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f72401006e0 0x7f7240197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 -- 172.21.15.49:0/4149341690 >> 172.21.15.49:0/4149341690 conn(0x7f72400fbc70 msgr2=0x7f72400fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 -- 172.21.15.49:0/4149341690 shutdown_connections 2024-03-18T01:00:33.550 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.547+0000 7f7247fb9640 1 -- 172.21.15.49:0/4149341690 wait complete. 2024-03-18T01:00:33.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:33 smithi049 ceph-mon[32130]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 433 MiB/s rd, 14 MiB/s wr, 762 op/s 2024-03-18T01:00:33.559 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:33 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1384183034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:00:33.560 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T01:00:33.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:33 smithi107 ceph-mon[36270]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 433 MiB/s rd, 14 MiB/s wr, 762 op/s 2024-03-18T01:00:33.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:33 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1384183034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:00:33.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.684+0000 7f05e9f59640 1 -- 172.21.15.49:0/364077878 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41006f0 msgr2=0x7f05e4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.684+0000 7f05e9f59640 1 --2- 172.21.15.49:0/364077878 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41006f0 0x7f05e4100b10 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f05d4009900 tx=0x7f05d402bc30 comp rx=0 tx=0).stop 2024-03-18T01:00:33.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.684+0000 7f05e9f59640 1 -- 172.21.15.49:0/364077878 shutdown_connections 2024-03-18T01:00:33.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.684+0000 7f05e9f59640 1 --2- 172.21.15.49:0/364077878 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05e41018e0 0x7f05e4101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.684+0000 7f05e9f59640 1 --2- 172.21.15.49:0/364077878 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05e41022a0 0x7f05e4108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.684+0000 7f05e9f59640 1 --2- 172.21.15.49:0/364077878 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41006f0 0x7f05e4100b10 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.684+0000 7f05e9f59640 1 -- 172.21.15.49:0/364077878 >> 172.21.15.49:0/364077878 conn(0x7f05e40fbc80 msgr2=0x7f05e40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.684+0000 7f05e9f59640 1 -- 172.21.15.49:0/364077878 shutdown_connections 2024-03-18T01:00:33.686 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.684+0000 7f05e9f59640 1 -- 172.21.15.49:0/364077878 wait complete. 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e9f59640 1 Processor -- start 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e9f59640 1 -- start start 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e9f59640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05e41006f0 0x7f05e406a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e9f59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05e41018e0 0x7f05e419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e9f59640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41022a0 0x7f05e419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e9f59640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f05e410c370 con 0x7f05e41022a0 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e9f59640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f05e410c1f0 con 0x7f05e41006f0 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e9f59640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f05e410c4f0 con 0x7f05e41018e0 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41022a0 0x7f05e419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e3fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41022a0 0x7f05e419eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37668/0 (socket says 172.21.15.49:37668) 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e3fff640 1 -- 172.21.15.49:0/1571527449 learned_addr learned my addr 172.21.15.49:0/1571527449 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.685+0000 7f05e37fe640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05e41006f0 0x7f05e406a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05e41018e0 0x7f05e419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e3fff640 1 -- 172.21.15.49:0/1571527449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05e41018e0 msgr2=0x7f05e419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e3fff640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05e41018e0 0x7f05e419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e3fff640 1 -- 172.21.15.49:0/1571527449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05e41006f0 msgr2=0x7f05e406a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e3fff640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05e41006f0 0x7f05e406a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e3fff640 1 -- 172.21.15.49:0/1571527449 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e419f760 con 0x7f05e41022a0 2024-03-18T01:00:33.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e3fff640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41022a0 0x7f05e419eea0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00eb10 tx=0x7f05dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e0ff9640 1 -- 172.21.15.49:0/1571527449 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc004b20 con 0x7f05e41022a0 2024-03-18T01:00:33.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e9f59640 1 -- 172.21.15.49:0/1571527449 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e419fa50 con 0x7f05e41022a0 2024-03-18T01:00:33.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e0ff9640 1 -- 172.21.15.49:0/1571527449 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05dc004cc0 con 0x7f05e41022a0 2024-03-18T01:00:33.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e9f59640 1 -- 172.21.15.49:0/1571527449 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e419fff0 con 0x7f05e41022a0 2024-03-18T01:00:33.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.686+0000 7f05e0ff9640 1 -- 172.21.15.49:0/1571527449 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc010760 con 0x7f05e41022a0 2024-03-18T01:00:33.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05ecc14740 1 Processor -- start 2024-03-18T01:00:33.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05e0ff9640 1 -- 172.21.15.49:0/1571527449 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f05dc005020 con 0x7f05e41022a0 2024-03-18T01:00:33.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05ecc14740 1 -- start start 2024-03-18T01:00:33.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be244080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.690 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be244650 0x5618be249150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be2497c0 0x5618be24bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05ecc14740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5618be1a4490 con 0x5618be244650 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05ecc14740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5618be14cc90 con 0x5618be243c60 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05ecc14740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5618be1a0530 con 0x5618be2497c0 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05e37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be244650 0x5618be249150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be244080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be244080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:56040/0 (socket says 172.21.15.49:56040) 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05e37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be244650 0x5618be249150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37682/0 (socket says 172.21.15.49:37682) 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05e2ffd640 1 -- 172.21.15.49:0/2296755372 learned_addr learned my addr 172.21.15.49:0/2296755372 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/2296755372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be2497c0 0x5618be24bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.689+0000 7f05e0ff9640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05cc0a2800 0x7f05cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05e2ffd640 1 -- 172.21.15.49:0/2296755372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be2497c0 msgr2=0x5618be24bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/2296755372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be2497c0 0x5618be24bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05e2ffd640 1 -- 172.21.15.49:0/2296755372 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be244650 msgr2=0x5618be249150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.691 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/2296755372 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be244650 0x5618be249150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05e2ffd640 1 -- 172.21.15.49:0/2296755372 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618be207500 con 0x5618be243c60 2024-03-18T01:00:33.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05e3fff640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05cc0a2800 0x7f05cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/2296755372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be244080 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f05d80058c0 tx=0x7f05d8005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24b025abbe324f01 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05e0ff9640 1 -- 172.21.15.49:0/1571527449 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f05dc0cf190 con 0x7f05e41022a0 2024-03-18T01:00:33.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05c9ffb640 1 -- 172.21.15.49:0/2296755372 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d8004030 con 0x5618be243c60 2024-03-18T01:00:33.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05c9ffb640 1 -- 172.21.15.49:0/2296755372 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05d8005500 con 0x5618be243c60 2024-03-18T01:00:33.692 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.690+0000 7f05c9ffb640 1 -- 172.21.15.49:0/2296755372 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d8004ce0 con 0x5618be243c60 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05ecc14740 1 -- 172.21.15.49:0/2296755372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 msgr2=0x5618be244080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05e3fff640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05cc0a2800 0x7f05cc0a4cc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f05dc091c50 tx=0x7f05dc097690 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05ecc14740 1 --2- 172.21.15.49:0/2296755372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be244080 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f05d80058c0 tx=0x7f05d8005850 comp rx=0 tx=0).stop 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05ecc14740 1 -- 172.21.15.49:0/2296755372 shutdown_connections 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05ecc14740 1 --2- 172.21.15.49:0/2296755372 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be2497c0 0x5618be24bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05ecc14740 1 --2- 172.21.15.49:0/2296755372 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be244650 0x5618be249150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05ecc14740 1 --2- 172.21.15.49:0/2296755372 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be244080 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05ecc14740 1 -- 172.21.15.49:0/2296755372 >> 172.21.15.49:0/2296755372 conn(0x5618be23adc0 msgr2=0x5618be23b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05ecc14740 1 -- 172.21.15.49:0/2296755372 shutdown_connections 2024-03-18T01:00:33.696 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.691+0000 7f05ecc14740 1 -- 172.21.15.49:0/2296755372 wait complete. 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05ecc14740 1 -- start start 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05ecc14740 1 Processor -- start 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be24e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2556f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05ecc14740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5618be24c770 con 0x5618be2497c0 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05ecc14740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5618be160230 con 0x5618be243c60 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05ecc14740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5618be24c8f0 con 0x5618be244650 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05ecc14740 10 client.0 _ll_register_callbacks cb 0x5618be19c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:56044/0 (socket says 172.21.15.49:56044) 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05e2ffd640 1 -- 172.21.15.49:0/1232647339 learned_addr learned my addr 172.21.15.49:0/1232647339 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.692+0000 7f05e37fe640 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be24e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05e2ffd640 1 -- 172.21.15.49:0/1232647339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 msgr2=0x5618be24e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.697 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be24e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05e2ffd640 1 -- 172.21.15.49:0/1232647339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 msgr2=0x5618be2556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05e2ffd640 1 -- 172.21.15.49:0/1232647339 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618be16f100 con 0x5618be243c60 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24e170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f05d8011660 tx=0x7f05d80115f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e21c7d41f05f1e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.106:3300/0 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05c2ffd640 1 -- 172.21.15.49:0/1232647339 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d8004a60 con 0x5618be243c60 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05c2ffd640 1 -- 172.21.15.49:0/1232647339 <== mon.2 v2:172.21.15.106:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f05d802a040 con 0x5618be243c60 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05ecc14740 1 -- 172.21.15.49:0/1232647339 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05d8005500 con 0x5618be243c60 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05c2ffd640 1 -- 172.21.15.49:0/1232647339 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f05d8036050 con 0x5618be243c60 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05ecc14740 10 client.27107 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05ecc14740 1 -- 172.21.15.49:0/1232647339 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5618bdeb83f0 con 0x5618be243c60 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.693+0000 7f05c2ffd640 1 -- 172.21.15.49:0/1232647339 <== mon.2 v2:172.21.15.106:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05d8005500 con 0x5618be243c60 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05c2ffd640 1 -- 172.21.15.49:0/1232647339 <== mon.2 v2:172.21.15.106:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f05d801f440 con 0x5618be243c60 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05ecc14740 10 client.27107 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.698 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05ecc14740 10 client.27107 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05ecc14740 20 client.27107 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5618be259c30 0x5618be25c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05ecc14740 4 client.27107 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05ecc14740 1 -- 172.21.15.49:0/1232647339 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618be1b19d0 con 0x5618be259c30 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05e37fe640 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5618be259c30 0x5618be25c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05e37fe640 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5618be259c30 0x5618be25c020 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=393d21e9d40749a7 server_cookie=470596d1e3cc1c81 in_seq=0 out_seq=0 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.694+0000 7f05c2ffd640 10 client.27107 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05c2ffd640 1 -- 172.21.15.49:0/1232647339 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5618be1b19d0 con 0x5618be259c30 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05c2ffd640 10 client.27107 handle_command_reply: tid=0 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 client.27107 shutdown 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 -- 172.21.15.49:0/1232647339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 msgr2=0x5618be24e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24e170 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f05d8011660 tx=0x7f05d80115f0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 -- 172.21.15.49:0/1232647339 shutdown_connections 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5618be259c30 0x5618be25c020 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.699 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be24e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05c2ffd640 0 client.27107 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T01:00:33.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1232647339 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24e170 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 -- 172.21.15.49:0/1232647339 >> 172.21.15.49:0/1232647339 conn(0x5618be23adc0 msgr2=0x5618be239fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 -- 172.21.15.49:0/1232647339 shutdown_connections 2024-03-18T01:00:33.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.696+0000 7f05ecc14740 1 -- 172.21.15.49:0/1232647339 wait complete. 2024-03-18T01:00:33.700 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.697+0000 7f05ecc14740 20 client.27107 trim_cache size 0 max 16384 2024-03-18T01:00:33.700 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:33 smithi106 ceph-mon[35899]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 433 MiB/s rd, 14 MiB/s wr, 762 op/s 2024-03-18T01:00:33.700 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:33 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1384183034' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05ecc14740 1 Processor -- start 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05ecc14740 1 -- start start 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be2556b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be24e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05ecc14740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f05e41a6cb0 con 0x5618be2497c0 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05ecc14740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f05e41a8820 con 0x5618be243c60 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05ecc14740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f05e41a7380 con 0x5618be244650 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be24e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be2556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:56056/0 (socket says 172.21.15.49:56056) 2024-03-18T01:00:33.712 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05e2ffd640 1 -- 172.21.15.49:0/1965111099 learned_addr learned my addr 172.21.15.49:0/1965111099 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05e2ffd640 1 -- 172.21.15.49:0/1965111099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 msgr2=0x5618be2556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/1965111099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be2556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.711+0000 7f05e2ffd640 1 -- 172.21.15.49:0/1965111099 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 msgr2=0x5618be24e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.712+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/1965111099 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be24e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.712+0000 7f05e2ffd640 1 -- 172.21.15.49:0/1965111099 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618bdf827d0 con 0x5618be243c60 2024-03-18T01:00:33.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.712+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/1965111099 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24dd80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f05d801f890 tx=0x7f05d8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5b431bf3b8e276e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.712+0000 7f05c97fa640 1 -- 172.21.15.49:0/1965111099 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d80053c0 con 0x5618be243c60 2024-03-18T01:00:33.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.712+0000 7f05c97fa640 1 -- 172.21.15.49:0/1965111099 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05d800fc70 con 0x5618be243c60 2024-03-18T01:00:33.713 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.712+0000 7f05c97fa640 1 -- 172.21.15.49:0/1965111099 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d801fce0 con 0x5618be243c60 2024-03-18T01:00:33.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 -- 172.21.15.49:0/1965111099 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 msgr2=0x5618be24dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1965111099 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24dd80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f05d801f890 tx=0x7f05d8000bc0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 -- 172.21.15.49:0/1965111099 shutdown_connections 2024-03-18T01:00:33.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1965111099 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be24e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1965111099 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be2556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.716 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 --2- 172.21.15.49:0/1965111099 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be24dd80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 -- 172.21.15.49:0/1965111099 >> 172.21.15.49:0/1965111099 conn(0x5618be23adc0 msgr2=0x5618be245b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 -- 172.21.15.49:0/1965111099 shutdown_connections 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 -- 172.21.15.49:0/1965111099 wait complete. 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 -- start start 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 Processor -- start 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be255470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be251f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f05e41a8120 con 0x5618be2497c0 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f05e41a7a50 con 0x5618be243c60 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f05e41a4c90 con 0x5618be244650 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.713+0000 7f05ecc14740 10 client.0 _ll_register_callbacks cb 0x7f05dc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be255470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37718/0 (socket says 172.21.15.49:37718) 2024-03-18T01:00:33.717 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 -- 172.21.15.49:0/4216542164 learned_addr learned my addr 172.21.15.49:0/4216542164 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be251f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 -- 172.21.15.49:0/4216542164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 msgr2=0x5618be251f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be251f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 -- 172.21.15.49:0/4216542164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 msgr2=0x5618be255470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be255470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 -- 172.21.15.49:0/4216542164 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618be2bdec0 con 0x5618be2497c0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05e2ffd640 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2524c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f05d801fa00 tx=0x7f05d8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84cb34825e9f4fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05c2ffd640 1 -- 172.21.15.49:0/4216542164 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d8005480 con 0x5618be2497c0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05c2ffd640 1 -- 172.21.15.49:0/4216542164 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f05d802a040 con 0x5618be2497c0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05ecc14740 1 -- 172.21.15.49:0/4216542164 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f05d800fc70 con 0x5618be2497c0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05c2ffd640 1 -- 172.21.15.49:0/4216542164 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f05d800f790 con 0x5618be2497c0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05ecc14740 10 client.17994 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.714+0000 7f05ecc14740 1 -- 172.21.15.49:0/4216542164 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5618be2bf000 con 0x5618be2497c0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05c2ffd640 1 -- 172.21.15.49:0/4216542164 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05d800fc70 con 0x5618be2497c0 2024-03-18T01:00:33.718 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05c2ffd640 1 -- 172.21.15.49:0/4216542164 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f05d80038f0 con 0x5618be2497c0 2024-03-18T01:00:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05ecc14740 10 client.17994 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05ecc14740 10 client.17994 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T01:00:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05ecc14740 20 client.17994 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05ecc14740 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5618be2bf900 0x5618be2c1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05ecc14740 4 client.17994 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:00:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05ecc14740 1 -- 172.21.15.49:0/4216542164 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5618be1abd20 con 0x5618be2bf900 2024-03-18T01:00:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05e37fe640 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5618be2bf900 0x5618be2c1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05e37fe640 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5618be2bf900 0x5618be2c1d20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf2cd776353abd0b server_cookie=41f9d5fb6ab895d3 in_seq=0 out_seq=0 2024-03-18T01:00:33.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.715+0000 7f05c2ffd640 10 client.17994 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.717+0000 7f05c2ffd640 1 -- 172.21.15.49:0/4216542164 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5618be1abd20 con 0x5618be2bf900 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.717+0000 7f05c2ffd640 10 client.17994 handle_command_reply: tid=0 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.717+0000 7f05ecc14740 1 client.17994 shutdown 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 -- 172.21.15.49:0/4216542164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 msgr2=0x5618be2524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2524c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f05d801fa00 tx=0x7f05d8005f90 comp rx=0 tx=0).stop 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 -- 172.21.15.49:0/4216542164 shutdown_connections 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5618be2bf900 0x5618be2c1d20 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5618be2497c0 0x5618be2524c0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5618be244650 0x5618be251f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05c2ffd640 0 client.17994 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T01:00:33.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 --2- 172.21.15.49:0/4216542164 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5618be243c60 0x5618be255470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 -- 172.21.15.49:0/4216542164 >> 172.21.15.49:0/4216542164 conn(0x5618be23adc0 msgr2=0x5618be24dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 -- 172.21.15.49:0/4216542164 shutdown_connections 2024-03-18T01:00:33.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 1 -- 172.21.15.49:0/4216542164 wait complete. 2024-03-18T01:00:33.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.718+0000 7f05ecc14740 20 client.17994 trim_cache size 0 max 16384 2024-03-18T01:00:33.721 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 -- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05cc0a2800 msgr2=0x7f05cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05cc0a2800 0x7f05cc0a4cc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f05dc091c50 tx=0x7f05dc097690 comp rx=0 tx=0).stop 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 -- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41022a0 msgr2=0x7f05e419eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41022a0 0x7f05e419eea0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00eb10 tx=0x7f05dc00c2d0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 -- 172.21.15.49:0/1571527449 shutdown_connections 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f05cc0a2800 0x7f05cc0a4cc0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f05e41022a0 0x7f05e419eea0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05e41018e0 0x7f05e419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 --2- 172.21.15.49:0/1571527449 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f05e41006f0 0x7f05e406a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 -- 172.21.15.49:0/1571527449 >> 172.21.15.49:0/1571527449 conn(0x7f05e40fbc80 msgr2=0x7f05e4104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 -- 172.21.15.49:0/1571527449 shutdown_connections 2024-03-18T01:00:33.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.719+0000 7f05e9f59640 1 -- 172.21.15.49:0/1571527449 wait complete. 2024-03-18T01:00:33.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T01:00:33.866 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 -- 172.21.15.49:0/1485911139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41033a0 msgr2=0x7f77d4109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 --2- 172.21.15.49:0/1485911139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41033a0 0x7f77d4109c30 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f77cc009ca0 tx=0x7f77cc02c0f0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 -- 172.21.15.49:0/1485911139 shutdown_connections 2024-03-18T01:00:33.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 --2- 172.21.15.49:0/1485911139 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41033a0 0x7f77d4109c30 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 --2- 172.21.15.49:0/1485911139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d41029e0 0x7f77d4102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 --2- 172.21.15.49:0/1485911139 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77d41017f0 0x7f77d4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 -- 172.21.15.49:0/1485911139 >> 172.21.15.49:0/1485911139 conn(0x7f77d40fcd60 msgr2=0x7f77d40ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 -- 172.21.15.49:0/1485911139 shutdown_connections 2024-03-18T01:00:33.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 -- 172.21.15.49:0/1485911139 wait complete. 2024-03-18T01:00:33.867 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.867+0000 7f77d9875640 1 Processor -- start 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d9875640 1 -- start start 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d9875640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41017f0 0x7f77d4075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d9875640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77d41029e0 0x7f77d4073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d9875640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d41033a0 0x7f77d4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d9875640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f77d410d470 con 0x7f77d41017f0 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d9875640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f77d410d2f0 con 0x7f77d41029e0 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d9875640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f77d410d5f0 con 0x7f77d41033a0 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41017f0 0x7f77d4075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41017f0 0x7f77d4075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37742/0 (socket says 172.21.15.49:37742) 2024-03-18T01:00:33.869 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d2ffd640 1 -- 172.21.15.49:0/2375671952 learned_addr learned my addr 172.21.15.49:0/2375671952 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d27fc640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77d41029e0 0x7f77d4073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d37fe640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d41033a0 0x7f77d4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d2ffd640 1 -- 172.21.15.49:0/2375671952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d41033a0 msgr2=0x7f77d4074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d41033a0 0x7f77d4074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d2ffd640 1 -- 172.21.15.49:0/2375671952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77d41029e0 msgr2=0x7f77d4073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77d41029e0 0x7f77d4073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d2ffd640 1 -- 172.21.15.49:0/2375671952 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d40749a0 con 0x7f77d41017f0 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d27fc640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77d41029e0 0x7f77d4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.868+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41017f0 0x7f77d4075590 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00e9c0 tx=0x7f77bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.869+0000 7f77d8873640 1 -- 172.21.15.49:0/2375671952 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc00cd80 con 0x7f77d41017f0 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.869+0000 7f77d9875640 1 -- 172.21.15.49:0/2375671952 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d41a1c00 con 0x7f77d41017f0 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.869+0000 7f77d8873640 1 -- 172.21.15.49:0/2375671952 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77bc0035f0 con 0x7f77d41017f0 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.869+0000 7f77d9875640 1 -- 172.21.15.49:0/2375671952 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d41a21a0 con 0x7f77d41017f0 2024-03-18T01:00:33.870 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.869+0000 7f77d8873640 1 -- 172.21.15.49:0/2375671952 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc010690 con 0x7f77d41017f0 2024-03-18T01:00:33.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.871+0000 7f77d8873640 1 -- 172.21.15.49:0/2375671952 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f77bc003fa0 con 0x7f77d41017f0 2024-03-18T01:00:33.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.871+0000 7f77dc459740 1 Processor -- start 2024-03-18T01:00:33.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.871+0000 7f77dc459740 1 -- start start 2024-03-18T01:00:33.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77dc459740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1b7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77dc459740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b8240 0x564eec1bcd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77dc459740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1bf840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.873 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77dc459740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564eec11c490 con 0x564eec1b8240 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77dc459740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x564eec0c4c90 con 0x564eec1bd3b0 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77dc459740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564eec118530 con 0x564eec1b7850 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d27fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1bf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d27fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1bf840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:56102/0 (socket says 172.21.15.49:56102) 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b8240 0x564eec1bcd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d27fc640 1 -- 172.21.15.49:0/1776940110 learned_addr learned my addr 172.21.15.49:0/1776940110 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d37fe640 1 --2- 172.21.15.49:0/1776940110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1b7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d8873640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77c00a2800 0x7f77c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d27fc640 1 -- 172.21.15.49:0/1776940110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 msgr2=0x564eec1b7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d27fc640 1 --2- 172.21.15.49:0/1776940110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1b7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d37fe640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77c00a2800 0x7f77c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d27fc640 1 -- 172.21.15.49:0/1776940110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b8240 msgr2=0x564eec1bcd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d27fc640 1 --2- 172.21.15.49:0/1776940110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b8240 0x564eec1bcd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.874 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d27fc640 1 -- 172.21.15.49:0/1776940110 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eec17f500 con 0x564eec1bd3b0 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/1776940110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b8240 0x564eec1bcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d8873640 1 -- 172.21.15.49:0/2375671952 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f77bc0cee00 con 0x7f77d41017f0 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.872+0000 7f77d27fc640 1 --2- 172.21.15.49:0/1776940110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1bf840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f77c40114f0 tx=0x7f77c40119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6370c68d74357ba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77d37fe640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77c00a2800 0x7f77c00a4cc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f77cc005e30 tx=0x7f77cc02c0f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77c9ffb640 1 -- 172.21.15.49:0/1776940110 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c4004020 con 0x564eec1bd3b0 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77c9ffb640 1 -- 172.21.15.49:0/1776940110 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c40041c0 con 0x564eec1bd3b0 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77c9ffb640 1 -- 172.21.15.49:0/1776940110 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c400f950 con 0x564eec1bd3b0 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77dc459740 1 -- 172.21.15.49:0/1776940110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 msgr2=0x564eec1bf840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77dc459740 1 --2- 172.21.15.49:0/1776940110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1bf840 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f77c40114f0 tx=0x7f77c40119b0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77dc459740 1 -- 172.21.15.49:0/1776940110 shutdown_connections 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77dc459740 1 --2- 172.21.15.49:0/1776940110 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1bf840 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77dc459740 1 --2- 172.21.15.49:0/1776940110 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b8240 0x564eec1bcd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77dc459740 1 --2- 172.21.15.49:0/1776940110 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1b7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77dc459740 1 -- 172.21.15.49:0/1776940110 >> 172.21.15.49:0/1776940110 conn(0x564eec1b2dc0 msgr2=0x564eec1b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.873+0000 7f77dc459740 1 -- 172.21.15.49:0/1776940110 shutdown_connections 2024-03-18T01:00:33.875 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.874+0000 7f77dc459740 1 -- 172.21.15.49:0/1776940110 wait complete. 2024-03-18T01:00:33.876 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.874+0000 7f77dc459740 1 -- start start 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.874+0000 7f77dc459740 1 Processor -- start 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.874+0000 7f77dc459740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.874+0000 7f77dc459740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.874+0000 7f77dc459740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1bd3b0 0x564eec1c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.874+0000 7f77dc459740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564eec1c0360 con 0x564eec1bd3b0 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.874+0000 7f77dc459740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x564eec0d8230 con 0x564eec1b8240 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.874+0000 7f77dc459740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564eec1c04e0 con 0x564eec1b7850 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77dc459740 10 client.0 _ll_register_callbacks cb 0x564eec114010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1c1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:59708/0 (socket says 172.21.15.49:59708) 2024-03-18T01:00:33.879 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d2ffd640 1 -- 172.21.15.49:0/3698408005 learned_addr learned my addr 172.21.15.49:0/3698408005 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d27fc640 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d37fe640 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1bd3b0 0x564eec1c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d2ffd640 1 -- 172.21.15.49:0/3698408005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 msgr2=0x564eec1c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d2ffd640 1 -- 172.21.15.49:0/3698408005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1bd3b0 msgr2=0x564eec1c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1bd3b0 0x564eec1c91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d2ffd640 1 -- 172.21.15.49:0/3698408005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eec0e7100 con 0x564eec1b7850 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d37fe640 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1bd3b0 0x564eec1c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d27fc640 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1c1ca0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0058c0 tx=0x7f77bc09b610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39163f93e2f7daa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.875+0000 7f77b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.876+0000 7f77b2ffd640 1 -- 172.21.15.49:0/3698408005 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc09be80 con 0x564eec1b7850 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.876+0000 7f77dc459740 1 -- 172.21.15.49:0/3698408005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77c40041c0 con 0x564eec1b7850 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.876+0000 7f77b2ffd640 1 -- 172.21.15.49:0/3698408005 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f77bc0a3a90 con 0x564eec1b7850 2024-03-18T01:00:33.880 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.876+0000 7f77b2ffd640 1 -- 172.21.15.49:0/3698408005 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f77bc09f8d0 con 0x564eec1b7850 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.876+0000 7f77dc459740 10 client.27211 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.876+0000 7f77dc459740 1 -- 172.21.15.49:0/3698408005 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564eebe303f0 con 0x564eec1b7850 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.876+0000 7f77b2ffd640 1 -- 172.21.15.49:0/3698408005 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77c40041c0 con 0x564eec1b7850 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.877+0000 7f77b2ffd640 1 -- 172.21.15.49:0/3698408005 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f77bc09abd0 con 0x564eec1b7850 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.877+0000 7f77dc459740 10 client.27211 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.877+0000 7f77dc459740 10 client.27211 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.877+0000 7f77dc459740 20 client.27211 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.877+0000 7f77dc459740 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564eec1cdff0 0x564eec1d0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.877+0000 7f77dc459740 4 client.27211 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.877+0000 7f77dc459740 1 -- 172.21.15.49:0/3698408005 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564eec1299d0 con 0x564eec1cdff0 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.877+0000 7f77d27fc640 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564eec1cdff0 0x564eec1d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.878+0000 7f77b2ffd640 10 client.27211 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.878+0000 7f77d27fc640 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564eec1cdff0 0x564eec1d0490 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d383dbcd9db47e9 server_cookie=6c70a3d57c6753b1 in_seq=0 out_seq=0 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77b2ffd640 1 -- 172.21.15.49:0/3698408005 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564eec1299d0 con 0x564eec1cdff0 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77b2ffd640 10 client.27211 handle_command_reply: tid=0 2024-03-18T01:00:33.881 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 client.27211 shutdown 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 -- 172.21.15.49:0/3698408005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 msgr2=0x564eec1c1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1c1ca0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0058c0 tx=0x7f77bc09b610 comp rx=0 tx=0).stop 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 -- 172.21.15.49:0/3698408005 shutdown_connections 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564eec1cdff0 0x564eec1d0490 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1bd3b0 0x564eec1c91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 --2- 172.21.15.49:0/3698408005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b7850 0x564eec1c1ca0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77b2ffd640 0 client.27211 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 -- 172.21.15.49:0/3698408005 >> 172.21.15.49:0/3698408005 conn(0x564eec1b2dc0 msgr2=0x564eec1b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 -- 172.21.15.49:0/3698408005 shutdown_connections 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 1 -- 172.21.15.49:0/3698408005 wait complete. 2024-03-18T01:00:33.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.879+0000 7f77dc459740 20 client.27211 trim_cache size 0 max 16384 2024-03-18T01:00:33.893 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77dc459740 1 Processor -- start 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77dc459740 1 -- start start 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77dc459740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1b1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77dc459740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77dc459740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1bd3b0 0x564eec1b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77dc459740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f77b400e530 con 0x564eec1b7850 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77dc459740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f77b40100a0 con 0x564eec1b8240 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77dc459740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f77b400ec00 con 0x564eec1bd3b0 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1b1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37764/0 (socket says 172.21.15.49:37764) 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d2ffd640 1 -- 172.21.15.49:0/4112552564 learned_addr learned my addr 172.21.15.49:0/4112552564 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d27fc640 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d37fe640 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1bd3b0 0x564eec1b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d2ffd640 1 -- 172.21.15.49:0/4112552564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1bd3b0 msgr2=0x564eec1b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.894 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1bd3b0 0x564eec1b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d2ffd640 1 -- 172.21.15.49:0/4112552564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 msgr2=0x564eec1c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d2ffd640 1 -- 172.21.15.49:0/4112552564 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564eebefa7d0 con 0x564eec1b7850 2024-03-18T01:00:33.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.893+0000 7f77d27fc640 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:33.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.894+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1b1750 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0ccc50 tx=0x7f77bc099c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21847aed7b33b6d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.894+0000 7f77c97fa640 1 -- 172.21.15.49:0/4112552564 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc09b0e0 con 0x564eec1b7850 2024-03-18T01:00:33.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.894+0000 7f77c97fa640 1 -- 172.21.15.49:0/4112552564 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77bc09be80 con 0x564eec1b7850 2024-03-18T01:00:33.895 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.894+0000 7f77c97fa640 1 -- 172.21.15.49:0/4112552564 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc09cd90 con 0x564eec1b7850 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 -- 172.21.15.49:0/4112552564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 msgr2=0x564eec1b1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1b1750 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0ccc50 tx=0x7f77bc099c90 comp rx=0 tx=0).stop 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 -- 172.21.15.49:0/4112552564 shutdown_connections 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1b8240 0x564eec1c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1bd3b0 0x564eec1b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 --2- 172.21.15.49:0/4112552564 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1b1750 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 -- 172.21.15.49:0/4112552564 >> 172.21.15.49:0/4112552564 conn(0x564eec1b2dc0 msgr2=0x564eec1c1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 -- 172.21.15.49:0/4112552564 shutdown_connections 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 -- 172.21.15.49:0/4112552564 wait complete. 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 -- start start 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.895+0000 7f77dc459740 1 Processor -- start 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77dc459740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1c6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.899 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77dc459740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b8240 0x564eec1c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77dc459740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77dc459740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f77b400f9a0 con 0x564eec1b7850 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77dc459740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f77b400f2d0 con 0x564eec1bd3b0 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77dc459740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f77b400c510 con 0x564eec1b8240 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77dc459740 10 client.0 _ll_register_callbacks cb 0x7f77bc0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1c6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d2ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1c6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37778/0 (socket says 172.21.15.49:37778) 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d2ffd640 1 -- 172.21.15.49:0/162231346 learned_addr learned my addr 172.21.15.49:0/162231346 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d37fe640 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d27fc640 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b8240 0x564eec1c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d2ffd640 1 -- 172.21.15.49:0/162231346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b8240 msgr2=0x564eec1c6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b8240 0x564eec1c6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.900 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d2ffd640 1 -- 172.21.15.49:0/162231346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 msgr2=0x564eec1c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1c7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d2ffd640 1 -- 172.21.15.49:0/162231346 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564eec1c8050 con 0x564eec1b7850 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77d2ffd640 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1c6510 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f77bc09b480 tx=0x7f77bc0cf190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1ce9b009f4b657 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77b2ffd640 1 -- 172.21.15.49:0/162231346 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc09b0e0 con 0x564eec1b7850 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77b2ffd640 1 -- 172.21.15.49:0/162231346 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f77bc0974d0 con 0x564eec1b7850 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.896+0000 7f77dc459740 1 -- 172.21.15.49:0/162231346 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f77bc09be80 con 0x564eec1b7850 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77b2ffd640 1 -- 172.21.15.49:0/162231346 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f77bc0a3f20 con 0x564eec1b7850 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77dc459740 10 client.18012 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77dc459740 1 -- 172.21.15.49:0/162231346 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564eec1c9350 con 0x564eec1b7850 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77b2ffd640 1 -- 172.21.15.49:0/162231346 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77bc0ba030 con 0x564eec1b7850 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77b2ffd640 1 -- 172.21.15.49:0/162231346 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f77bc0bb2f0 con 0x564eec1b7850 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77dc459740 10 client.18012 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77dc459740 10 client.18012 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:33.901 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77dc459740 20 client.18012 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77dc459740 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564eec2347e0 0x564eec236c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77dc459740 4 client.18012 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77dc459740 1 -- 172.21.15.49:0/162231346 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564eec123d20 con 0x564eec2347e0 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77d27fc640 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564eec2347e0 0x564eec236c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.897+0000 7f77b2ffd640 10 client.18012 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77d27fc640 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564eec2347e0 0x564eec236c00 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c36cb98720610ef8 server_cookie=8ae8186f6b9a78c2 in_seq=0 out_seq=0 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77b2ffd640 1 -- 172.21.15.49:0/162231346 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564eec123d20 con 0x564eec2347e0 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77b2ffd640 10 client.18012 handle_command_reply: tid=0 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 client.18012 shutdown 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 -- 172.21.15.49:0/162231346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 msgr2=0x564eec1c6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1c6510 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f77bc09b480 tx=0x7f77bc0cf190 comp rx=0 tx=0).stop 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 -- 172.21.15.49:0/162231346 shutdown_connections 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x564eec2347e0 0x564eec236c00 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77b2ffd640 0 client.18012 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564eec1bd3b0 0x564eec1c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564eec1b8240 0x564eec1c6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 --2- 172.21.15.49:0/162231346 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564eec1b7850 0x564eec1c6510 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 -- 172.21.15.49:0/162231346 >> 172.21.15.49:0/162231346 conn(0x564eec1b2dc0 msgr2=0x564eec1b6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 -- 172.21.15.49:0/162231346 shutdown_connections 2024-03-18T01:00:33.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 1 -- 172.21.15.49:0/162231346 wait complete. 2024-03-18T01:00:33.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.898+0000 7f77dc459740 20 client.18012 trim_cache size 0 max 16384 2024-03-18T01:00:33.903 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:00:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 -- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77c00a2800 msgr2=0x7f77c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77c00a2800 0x7f77c00a4cc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f77cc005e30 tx=0x7f77cc02c0f0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 -- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41017f0 msgr2=0x7f77d4075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41017f0 0x7f77d4075590 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f77bc00e9c0 tx=0x7f77bc00ee80 comp rx=0 tx=0).stop 2024-03-18T01:00:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 -- 172.21.15.49:0/2375671952 shutdown_connections 2024-03-18T01:00:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f77c00a2800 0x7f77c00a4cc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f77d41029e0 0x7f77d4073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77d41033a0 0x7f77d4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 --2- 172.21.15.49:0/2375671952 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f77d41017f0 0x7f77d4075590 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 -- 172.21.15.49:0/2375671952 >> 172.21.15.49:0/2375671952 conn(0x7f77d40fcd60 msgr2=0x7f77d4105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 -- 172.21.15.49:0/2375671952 shutdown_connections 2024-03-18T01:00:33.905 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:33.899+0000 7f77d9875640 1 -- 172.21.15.49:0/2375671952 wait complete. 2024-03-18T01:00:34.912 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T01:00:34.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:00:34.912 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T01:00:35.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.055+0000 7fad30c36640 1 -- 172.21.15.49:0/3788192746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad2c101910 msgr2=0x7fad2c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.055+0000 7fad30c36640 1 --2- 172.21.15.49:0/3788192746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad2c101910 0x7fad2c101d90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fad14009900 tx=0x7fad1402bc30 comp rx=0 tx=0).stop 2024-03-18T01:00:35.055 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.056+0000 7fad30c36640 1 -- 172.21.15.49:0/3788192746 shutdown_connections 2024-03-18T01:00:35.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.056+0000 7fad30c36640 1 --2- 172.21.15.49:0/3788192746 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad2c101910 0x7fad2c101d90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.056+0000 7fad30c36640 1 --2- 172.21.15.49:0/3788192746 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad2c1022d0 0x7fad2c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.056+0000 7fad30c36640 1 --2- 172.21.15.49:0/3788192746 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c100720 0x7fad2c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.056+0000 7fad30c36640 1 -- 172.21.15.49:0/3788192746 >> 172.21.15.49:0/3788192746 conn(0x7fad2c0fbc90 msgr2=0x7fad2c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.056+0000 7fad30c36640 1 -- 172.21.15.49:0/3788192746 shutdown_connections 2024-03-18T01:00:35.056 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.056+0000 7fad30c36640 1 -- 172.21.15.49:0/3788192746 wait complete. 2024-03-18T01:00:35.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.056+0000 7fad30c36640 1 Processor -- start 2024-03-18T01:00:35.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad30c36640 1 -- start start 2024-03-18T01:00:35.057 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad30c36640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c100720 0x7fad2c06a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad30c36640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad2c101910 0x7fad2c19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad30c36640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad2c1022d0 0x7fad2c19ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad30c36640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c10c380 con 0x7fad2c101910 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad30c36640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c10c200 con 0x7fad2c1022d0 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad30c36640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c10c500 con 0x7fad2c100720 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad21d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad2c101910 0x7fad2c19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad2ad9d640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad2c1022d0 0x7fad2c19ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad21d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad2c101910 0x7fad2c19c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37790/0 (socket says 172.21.15.49:37790) 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.057+0000 7fad21d9b640 1 -- 172.21.15.49:0/2881640863 learned_addr learned my addr 172.21.15.49:0/2881640863 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.058+0000 7fad21d9b640 1 -- 172.21.15.49:0/2881640863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c100720 msgr2=0x7fad2c06a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.058+0000 7fad21d9b640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c100720 0x7fad2c06a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.058+0000 7fad21d9b640 1 -- 172.21.15.49:0/2881640863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad2c1022d0 msgr2=0x7fad2c19ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.058+0000 7fad21d9b640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad2c1022d0 0x7fad2c19ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.058 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.058+0000 7fad21d9b640 1 -- 172.21.15.49:0/2881640863 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c19f750 con 0x7fad2c101910 2024-03-18T01:00:35.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.058+0000 7fad2ad9d640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad2c1022d0 0x7fad2c19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:35.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.058+0000 7fad21d9b640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad2c101910 0x7fad2c19c850 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fad14006010 tx=0x7fad14004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.059 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.059+0000 7fad23fff640 1 -- 172.21.15.49:0/2881640863 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14041070 con 0x7fad2c101910 2024-03-18T01:00:35.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.059+0000 7fad23fff640 1 -- 172.21.15.49:0/2881640863 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad140044d0 con 0x7fad2c101910 2024-03-18T01:00:35.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.059+0000 7fad30c36640 1 -- 172.21.15.49:0/2881640863 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2c19f9e0 con 0x7fad2c101910 2024-03-18T01:00:35.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.059+0000 7fad23fff640 1 -- 172.21.15.49:0/2881640863 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14034440 con 0x7fad2c101910 2024-03-18T01:00:35.060 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.059+0000 7fad30c36640 1 -- 172.21.15.49:0/2881640863 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2c19fed0 con 0x7fad2c101910 2024-03-18T01:00:35.061 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.061+0000 7fad23fff640 1 -- 172.21.15.49:0/2881640863 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fad14044070 con 0x7fad2c101910 2024-03-18T01:00:35.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.062+0000 7fad23fff640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad180a2950 0x7fad180a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.062+0000 7fad2a59c640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad180a2950 0x7fad180a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.062+0000 7fad23fff640 1 -- 172.21.15.49:0/2881640863 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fad140f05a0 con 0x7fad2c101910 2024-03-18T01:00:35.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.063+0000 7fad2a59c640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad180a2950 0x7fad180a4e10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fad2c101770 tx=0x7fad1c005c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.062 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.063+0000 7fad3381e740 1 Processor -- start 2024-03-18T01:00:35.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.063+0000 7fad3381e740 1 -- start start 2024-03-18T01:00:35.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad3381e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 0x558bf251bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad3381e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251c380 0x558bf2520e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad3381e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf25214f0 0x558bf2523980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad3381e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558bf24800d0 con 0x558bf251c380 2024-03-18T01:00:35.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad3381e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558bf2428c70 con 0x558bf25214f0 2024-03-18T01:00:35.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad3381e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558bf247c170 con 0x558bf251b990 2024-03-18T01:00:35.064 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad21d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251c380 0x558bf2520e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad21d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251c380 0x558bf2520e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37796/0 (socket says 172.21.15.49:37796) 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad21d9b640 1 -- 172.21.15.49:0/3068526051 learned_addr learned my addr 172.21.15.49:0/3068526051 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad2ad9d640 1 --2- 172.21.15.49:0/3068526051 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf25214f0 0x558bf2523980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad21d9b640 1 -- 172.21.15.49:0/3068526051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 msgr2=0x558bf251bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad21d9b640 1 --2- 172.21.15.49:0/3068526051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 0x558bf251bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad21d9b640 1 -- 172.21.15.49:0/3068526051 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf25214f0 msgr2=0x558bf2523980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad21d9b640 1 --2- 172.21.15.49:0/3068526051 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf25214f0 0x558bf2523980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad21d9b640 1 -- 172.21.15.49:0/3068526051 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bf225c7e0 con 0x558bf251c380 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.064+0000 7fad21d9b640 1 --2- 172.21.15.49:0/3068526051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251c380 0x558bf2520e80 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fad140bceb0 tx=0x7fad14003990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c896109ec98afa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.065+0000 7fad20d99640 1 -- 172.21.15.49:0/3068526051 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1404b040 con 0x558bf251c380 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.065+0000 7fad20d99640 1 -- 172.21.15.49:0/3068526051 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad140bcf20 con 0x558bf251c380 2024-03-18T01:00:35.065 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.065+0000 7fad20d99640 1 -- 172.21.15.49:0/3068526051 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad140035f0 con 0x558bf251c380 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 -- 172.21.15.49:0/3068526051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251c380 msgr2=0x558bf2520e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 --2- 172.21.15.49:0/3068526051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251c380 0x558bf2520e80 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fad140bceb0 tx=0x7fad14003990 comp rx=0 tx=0).stop 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 -- 172.21.15.49:0/3068526051 shutdown_connections 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 --2- 172.21.15.49:0/3068526051 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf25214f0 0x558bf2523980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 --2- 172.21.15.49:0/3068526051 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251c380 0x558bf2520e80 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 --2- 172.21.15.49:0/3068526051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 0x558bf251bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 -- 172.21.15.49:0/3068526051 >> 172.21.15.49:0/3068526051 conn(0x558bf24e3c50 msgr2=0x558bf2517300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 -- 172.21.15.49:0/3068526051 shutdown_connections 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 -- 172.21.15.49:0/3068526051 wait complete. 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 -- start start 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.066+0000 7fad3381e740 1 Processor -- start 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad3381e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 0x558bf2525d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad3381e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf25214f0 0x558bf25262b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.067 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad3381e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 0x558bf252d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad3381e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x558bf2524260 con 0x558bf25214f0 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad3381e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x558bf243c0d0 con 0x558bf252b2e0 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad3381e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x558bf25243e0 con 0x558bf251b990 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad3381e740 10 client.0 _ll_register_callbacks cb 0x558bf2405760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad21d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf25214f0 0x558bf25262b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad21d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf25214f0 0x558bf25262b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37806/0 (socket says 172.21.15.49:37806) 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad21d9b640 1 -- 172.21.15.49:0/3564397126 learned_addr learned my addr 172.21.15.49:0/3564397126 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad2ad9d640 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 0x558bf252d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad21d9b640 1 -- 172.21.15.49:0/3564397126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 msgr2=0x558bf2525d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad21d9b640 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 0x558bf2525d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad21d9b640 1 -- 172.21.15.49:0/3564397126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 msgr2=0x558bf252d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad21d9b640 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 0x558bf252d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad21d9b640 1 -- 172.21.15.49:0/3564397126 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bf22bcf70 con 0x558bf25214f0 2024-03-18T01:00:35.068 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.067+0000 7fad2ad9d640 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 0x558bf252d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.068+0000 7fad21d9b640 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf25214f0 0x558bf25262b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fad140ee0d0 tx=0x7fad140bb220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a3fe79eaa006213 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.068+0000 7fad11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.068+0000 7fad11ffb640 1 -- 172.21.15.49:0/3564397126 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1404b070 con 0x558bf25214f0 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.068+0000 7fad3381e740 1 -- 172.21.15.49:0/3564397126 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad140bcf20 con 0x558bf25214f0 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.068+0000 7fad11ffb640 1 -- 172.21.15.49:0/3564397126 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fad140b9670 con 0x558bf25214f0 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.068+0000 7fad11ffb640 1 -- 172.21.15.49:0/3564397126 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad140c56c0 con 0x558bf25214f0 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.068+0000 7fad3381e740 10 client.18024 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.068+0000 7fad3381e740 1 -- 172.21.15.49:0/3564397126 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558bf2154170 con 0x558bf25214f0 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.068+0000 7fad11ffb640 1 -- 172.21.15.49:0/3564397126 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad140bcf20 con 0x558bf25214f0 2024-03-18T01:00:35.071 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.069+0000 7fad11ffb640 1 -- 172.21.15.49:0/3564397126 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fad140bc370 con 0x558bf25214f0 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.069+0000 7fad3381e740 10 client.18024 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.069+0000 7fad3381e740 10 client.18024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.069+0000 7fad3381e740 20 client.18024 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.069+0000 7fad3381e740 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558bf2531ac0 0x558bf2533f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.069+0000 7fad3381e740 4 client.18024 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.069+0000 7fad3381e740 1 -- 172.21.15.49:0/3564397126 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bf248d610 con 0x558bf2531ac0 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.069+0000 7fad2a59c640 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558bf2531ac0 0x558bf2533f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.070+0000 7fad11ffb640 10 client.18024 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.070+0000 7fad2a59c640 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558bf2531ac0 0x558bf2533f20 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64f5bfe0703b0833 server_cookie=32546336bb2a89d3 in_seq=0 out_seq=0 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad11ffb640 1 -- 172.21.15.49:0/3564397126 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558bf248d610 con 0x558bf2531ac0 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad11ffb640 10 client.18024 handle_command_reply: tid=0 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 client.18024 shutdown 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 -- 172.21.15.49:0/3564397126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf25214f0 msgr2=0x558bf25262b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf25214f0 0x558bf25262b0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fad140ee0d0 tx=0x7fad140bb220 comp rx=0 tx=0).stop 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 -- 172.21.15.49:0/3564397126 shutdown_connections 2024-03-18T01:00:35.072 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558bf2531ac0 0x558bf2533f20 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 0x558bf252d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf25214f0 0x558bf25262b0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad11ffb640 0 client.18024 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:35.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 --2- 172.21.15.49:0/3564397126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 0x558bf2525d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 -- 172.21.15.49:0/3564397126 >> 172.21.15.49:0/3564397126 conn(0x558bf24e3c50 msgr2=0x558bf25164e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 -- 172.21.15.49:0/3564397126 shutdown_connections 2024-03-18T01:00:35.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 1 -- 172.21.15.49:0/3564397126 wait complete. 2024-03-18T01:00:35.073 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.071+0000 7fad3381e740 20 client.18024 trim_cache size 0 max 16384 2024-03-18T01:00:35.096 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.096+0000 7fad3381e740 1 Processor -- start 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.096+0000 7fad3381e740 1 -- start start 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.096+0000 7fad3381e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251b990 0x558bf2517530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.096+0000 7fad3381e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf25214f0 0x558bf25246d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.096+0000 7fad3381e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 0x558bf2524c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.096+0000 7fad3381e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1a6c00 con 0x558bf251b990 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.096+0000 7fad3381e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1a8770 con 0x558bf252b2e0 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.096+0000 7fad3381e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1a72d0 con 0x558bf25214f0 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad2a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251b990 0x558bf2517530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad2a59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251b990 0x558bf2517530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37818/0 (socket says 172.21.15.49:37818) 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad2a59c640 1 -- 172.21.15.49:0/668017326 learned_addr learned my addr 172.21.15.49:0/668017326 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad2a59c640 1 -- 172.21.15.49:0/668017326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf25214f0 msgr2=0x558bf25246d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad2a59c640 1 --2- 172.21.15.49:0/668017326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf25214f0 0x558bf25246d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.097 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad2a59c640 1 -- 172.21.15.49:0/668017326 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 msgr2=0x558bf2524c10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T01:00:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad2a59c640 1 --2- 172.21.15.49:0/668017326 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 0x558bf2524c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad2a59c640 1 -- 172.21.15.49:0/668017326 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bf2516c30 con 0x558bf251b990 2024-03-18T01:00:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad2a59c640 1 --2- 172.21.15.49:0/668017326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251b990 0x558bf2517530 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00a2c0 tx=0x7fad1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced2616070a7ec08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad13fff640 1 -- 172.21.15.49:0/668017326 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c022250 con 0x558bf251b990 2024-03-18T01:00:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad13fff640 1 -- 172.21.15.49:0/668017326 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c015050 con 0x558bf251b990 2024-03-18T01:00:35.098 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.097+0000 7fad13fff640 1 -- 172.21.15.49:0/668017326 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c004a00 con 0x558bf251b990 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.098+0000 7fad3381e740 1 -- 172.21.15.49:0/668017326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251b990 msgr2=0x558bf2517530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.098+0000 7fad3381e740 1 --2- 172.21.15.49:0/668017326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251b990 0x558bf2517530 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00a2c0 tx=0x7fad1c0046c0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.098+0000 7fad3381e740 1 -- 172.21.15.49:0/668017326 shutdown_connections 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.098+0000 7fad3381e740 1 --2- 172.21.15.49:0/668017326 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf252b2e0 0x558bf2524c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.098+0000 7fad3381e740 1 --2- 172.21.15.49:0/668017326 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf25214f0 0x558bf25246d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.098+0000 7fad3381e740 1 --2- 172.21.15.49:0/668017326 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf251b990 0x558bf2517530 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.098+0000 7fad3381e740 1 -- 172.21.15.49:0/668017326 >> 172.21.15.49:0/668017326 conn(0x558bf24e3c50 msgr2=0x558bf2516e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.098+0000 7fad3381e740 1 -- 172.21.15.49:0/668017326 shutdown_connections 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.098+0000 7fad3381e740 1 -- 172.21.15.49:0/668017326 wait complete. 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad3381e740 1 -- start start 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad3381e740 1 Processor -- start 2024-03-18T01:00:35.101 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad3381e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 0x558bf2542760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad3381e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf25214f0 0x558bf2542ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad3381e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf252b2e0 0x558bf2543cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad3381e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1a8070 con 0x558bf252b2e0 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad3381e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1a79a0 con 0x558bf25214f0 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad3381e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1a4be0 con 0x558bf251b990 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad3381e740 10 client.0 _ll_register_callbacks cb 0x7fad140bccf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad2ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf252b2e0 0x558bf2543cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad2ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf252b2e0 0x558bf2543cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37834/0 (socket says 172.21.15.49:37834) 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.099+0000 7fad2ad9d640 1 -- 172.21.15.49:0/2193064768 learned_addr learned my addr 172.21.15.49:0/2193064768 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad2ad9d640 1 -- 172.21.15.49:0/2193064768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 msgr2=0x558bf2542760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad2ad9d640 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 0x558bf2542760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad2ad9d640 1 -- 172.21.15.49:0/2193064768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf25214f0 msgr2=0x558bf2542ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.102 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad2ad9d640 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf25214f0 0x558bf2542ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad2ad9d640 1 -- 172.21.15.49:0/2193064768 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bf25442a0 con 0x558bf252b2e0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad2ad9d640 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf252b2e0 0x558bf2543cb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fad24010cf0 tx=0x7fad2400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4540ab779c7104ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad11ffb640 1 -- 172.21.15.49:0/2193064768 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad24020060 con 0x558bf252b2e0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad11ffb640 1 -- 172.21.15.49:0/2193064768 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fad24003b70 con 0x558bf252b2e0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad3381e740 1 -- 172.21.15.49:0/2193064768 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad1c015050 con 0x558bf252b2e0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad11ffb640 1 -- 172.21.15.49:0/2193064768 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad24003d70 con 0x558bf252b2e0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad3381e740 10 client.18030 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.100+0000 7fad3381e740 1 -- 172.21.15.49:0/2193064768 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558bf2597970 con 0x558bf252b2e0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad11ffb640 1 -- 172.21.15.49:0/2193064768 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c015050 con 0x558bf252b2e0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad11ffb640 1 -- 172.21.15.49:0/2193064768 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fad240266d0 con 0x558bf252b2e0 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad3381e740 10 client.18030 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad3381e740 10 client.18030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad3381e740 20 client.18030 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad3381e740 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558bf2598400 0x558bf259a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.103 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad3381e740 4 client.18030 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T01:00:35.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad3381e740 1 -- 172.21.15.49:0/2193064768 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558bf2525b60 con 0x558bf2598400 2024-03-18T01:00:35.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad2a59c640 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558bf2598400 0x558bf259a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad2a59c640 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558bf2598400 0x558bf259a820 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33129cd449a2ca9d server_cookie=9a4fcb96f041887a in_seq=0 out_seq=0 2024-03-18T01:00:35.104 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.101+0000 7fad11ffb640 10 client.18030 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad11ffb640 1 -- 172.21.15.49:0/2193064768 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558bf2525b60 con 0x558bf2598400 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad11ffb640 10 client.18030 handle_command_reply: tid=0 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad3381e740 1 client.18030 shutdown 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad3381e740 1 -- 172.21.15.49:0/2193064768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf252b2e0 msgr2=0x558bf2543cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad3381e740 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf252b2e0 0x558bf2543cb0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fad24010cf0 tx=0x7fad2400f790 comp rx=0 tx=0).stop 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad3381e740 1 -- 172.21.15.49:0/2193064768 shutdown_connections 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad3381e740 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x558bf2598400 0x558bf259a820 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad3381e740 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x558bf25214f0 0x558bf2542ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad3381e740 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x558bf252b2e0 0x558bf2543cb0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad11ffb640 0 client.18030 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad3381e740 1 --2- 172.21.15.49:0/2193064768 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x558bf251b990 0x558bf2542760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.102+0000 7fad3381e740 1 -- 172.21.15.49:0/2193064768 >> 172.21.15.49:0/2193064768 conn(0x558bf24e3c50 msgr2=0x558bf25246d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.105 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.103+0000 7fad3381e740 1 -- 172.21.15.49:0/2193064768 shutdown_connections 2024-03-18T01:00:35.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.103+0000 7fad3381e740 1 -- 172.21.15.49:0/2193064768 wait complete. 2024-03-18T01:00:35.106 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.103+0000 7fad3381e740 20 client.18030 trim_cache size 0 max 16384 2024-03-18T01:00:35.106 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:00:35.106 INFO:teuthology.orchestra.run.smithi049.stdout: "return_code": 0, 2024-03-18T01:00:35.106 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_tag": "b514c819-3a26-4f02-b8b0-f5f6188b46c9", 2024-03-18T01:00:35.106 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": "asynchronous" 2024-03-18T01:00:35.106 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.103+0000 7fad30c36640 1 -- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad180a2950 msgr2=0x7fad180a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.103+0000 7fad30c36640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad180a2950 0x7fad180a4e10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fad2c101770 tx=0x7fad1c005c90 comp rx=0 tx=0).stop 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.103+0000 7fad30c36640 1 -- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad2c101910 msgr2=0x7fad2c19c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.103+0000 7fad30c36640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad2c101910 0x7fad2c19c850 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fad14006010 tx=0x7fad14004ab0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.104+0000 7fad30c36640 1 -- 172.21.15.49:0/2881640863 shutdown_connections 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.104+0000 7fad30c36640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fad180a2950 0x7fad180a4e10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.104+0000 7fad30c36640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fad2c1022d0 0x7fad2c19ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.104+0000 7fad30c36640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fad2c101910 0x7fad2c19c850 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.104+0000 7fad30c36640 1 --2- 172.21.15.49:0/2881640863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fad2c100720 0x7fad2c06a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.107 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.104+0000 7fad30c36640 1 -- 172.21.15.49:0/2881640863 >> 172.21.15.49:0/2881640863 conn(0x7fad2c0fbc90 msgr2=0x7fad2c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.104+0000 7fad30c36640 1 -- 172.21.15.49:0/2881640863 shutdown_connections 2024-03-18T01:00:35.108 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.104+0000 7fad30c36640 1 -- 172.21.15.49:0/2881640863 wait complete. 2024-03-18T01:00:35.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:00:35.122 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:00:35.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.249+0000 7f1752061640 1 -- 172.21.15.49:0/170783182 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c102290 msgr2=0x7f174c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.249+0000 7f1752061640 1 --2- 172.21.15.49:0/170783182 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c102290 0x7f174c108b20 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f1744009ca0 tx=0x7f174402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.249+0000 7f1752061640 1 -- 172.21.15.49:0/170783182 shutdown_connections 2024-03-18T01:00:35.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.249+0000 7f1752061640 1 --2- 172.21.15.49:0/170783182 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c102290 0x7f174c108b20 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.249+0000 7f1752061640 1 --2- 172.21.15.49:0/170783182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c1018d0 0x7f174c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.249+0000 7f1752061640 1 --2- 172.21.15.49:0/170783182 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f174c1006e0 0x7f174c100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.249 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.249+0000 7f1752061640 1 -- 172.21.15.49:0/170783182 >> 172.21.15.49:0/170783182 conn(0x7f174c0fbc70 msgr2=0x7f174c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.249+0000 7f1752061640 1 -- 172.21.15.49:0/170783182 shutdown_connections 2024-03-18T01:00:35.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.249+0000 7f1752061640 1 -- 172.21.15.49:0/170783182 wait complete. 2024-03-18T01:00:35.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f1752061640 1 Processor -- start 2024-03-18T01:00:35.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f1752061640 1 -- start start 2024-03-18T01:00:35.251 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f1752061640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c1006e0 0x7f174c104620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f1752061640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c1018d0 0x7f174c102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f1752061640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f174c102290 0x7f174c1031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f1752061640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f174c10c360 con 0x7f174c1006e0 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f1752061640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f174c10c1e0 con 0x7f174c102290 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f1752061640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f174c10c4e0 con 0x7f174c1018d0 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f174b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c1006e0 0x7f174c104620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f174b7fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c1006e0 0x7f174c104620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37856/0 (socket says 172.21.15.49:37856) 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f174b7fe640 1 -- 172.21.15.49:0/3087509857 learned_addr learned my addr 172.21.15.49:0/3087509857 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.250+0000 7f174affd640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c1018d0 0x7f174c102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f174bfff640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f174c102290 0x7f174c1031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f174b7fe640 1 -- 172.21.15.49:0/3087509857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c1018d0 msgr2=0x7f174c102cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f174b7fe640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c1018d0 0x7f174c102cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f174b7fe640 1 -- 172.21.15.49:0/3087509857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f174c102290 msgr2=0x7f174c1031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f174b7fe640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f174c102290 0x7f174c1031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f174b7fe640 1 -- 172.21.15.49:0/3087509857 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174c103ab0 con 0x7f174c1006e0 2024-03-18T01:00:35.252 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f174b7fe640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c1006e0 0x7f174c104620 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f173c00e9c0 tx=0x7f173c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f1748ff9640 1 -- 172.21.15.49:0/3087509857 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c00cde0 con 0x7f174c1006e0 2024-03-18T01:00:35.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f1748ff9640 1 -- 172.21.15.49:0/3087509857 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f173c010910 con 0x7f174c1006e0 2024-03-18T01:00:35.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f1748ff9640 1 -- 172.21.15.49:0/3087509857 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173c018a00 con 0x7f174c1006e0 2024-03-18T01:00:35.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f1752061640 1 -- 172.21.15.49:0/3087509857 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174c1a1c80 con 0x7f174c1006e0 2024-03-18T01:00:35.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.251+0000 7f1752061640 1 -- 172.21.15.49:0/3087509857 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174c1a2150 con 0x7f174c1006e0 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.255+0000 7f1754c52740 1 Processor -- start 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.255+0000 7f1754c52740 1 -- start start 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.255+0000 7f1754c52740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 0x563fd465c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.255+0000 7f1754c52740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465cb60 0x563fd4661660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.255+0000 7f1754c52740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd4664160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.255+0000 7f1754c52740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563fd45c1500 con 0x563fd4661cd0 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.255+0000 7f1754c52740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563fd4569be0 con 0x563fd465c170 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f1754c52740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563fd45bd5a0 con 0x563fd465cb60 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 0x563fd465c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.257 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd4664160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174bfff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 0x563fd465c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:56202/0 (socket says 172.21.15.49:56202) 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174bfff640 1 -- 172.21.15.49:0/1172703206 learned_addr learned my addr 172.21.15.49:0/1172703206 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174affd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd4664160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37866/0 (socket says 172.21.15.49:37866) 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174bfff640 1 -- 172.21.15.49:0/1172703206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465cb60 msgr2=0x563fd4661660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174bfff640 1 --2- 172.21.15.49:0/1172703206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465cb60 0x563fd4661660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174bfff640 1 -- 172.21.15.49:0/1172703206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 msgr2=0x563fd4664160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174bfff640 1 --2- 172.21.15.49:0/1172703206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd4664160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174bfff640 1 -- 172.21.15.49:0/1172703206 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fd457d4b0 con 0x563fd465c170 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f174bfff640 1 --2- 172.21.15.49:0/1172703206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 0x563fd465c590 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1744009ca0 tx=0x7f17440323c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed917eba1feb9596 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f1748ff9640 1 -- 172.21.15.49:0/3087509857 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f173c01f020 con 0x7f174c1006e0 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f1731ffb640 1 -- 172.21.15.49:0/1172703206 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744043070 con 0x563fd465c170 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f1731ffb640 1 -- 172.21.15.49:0/1172703206 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1744032c90 con 0x563fd465c170 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.256+0000 7f1731ffb640 1 -- 172.21.15.49:0/1172703206 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744039040 con 0x563fd465c170 2024-03-18T01:00:35.258 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1748ff9640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17340a2800 0x7f17340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f174b7fe640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17340a2800 0x7f17340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1748ff9640 1 -- 172.21.15.49:0/3087509857 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f173c0cec00 con 0x7f174c1006e0 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1754c52740 1 -- 172.21.15.49:0/1172703206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 msgr2=0x563fd465c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1754c52740 1 --2- 172.21.15.49:0/1172703206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 0x563fd465c590 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1744009ca0 tx=0x7f17440323c0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1754c52740 1 -- 172.21.15.49:0/1172703206 shutdown_connections 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1754c52740 1 --2- 172.21.15.49:0/1172703206 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd4664160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1754c52740 1 --2- 172.21.15.49:0/1172703206 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465cb60 0x563fd4661660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1754c52740 1 --2- 172.21.15.49:0/1172703206 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 0x563fd465c590 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1754c52740 1 -- 172.21.15.49:0/1172703206 >> 172.21.15.49:0/1172703206 conn(0x563fd46576e0 msgr2=0x563fd4657ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1754c52740 1 -- 172.21.15.49:0/1172703206 shutdown_connections 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f1754c52740 1 -- 172.21.15.49:0/1172703206 wait complete. 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.257+0000 7f174b7fe640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17340a2800 0x7f17340a4cc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f173c09b510 tx=0x7f173c06a3e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.259 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f1754c52740 1 -- start start 2024-03-18T01:00:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f1754c52740 1 Processor -- start 2024-03-18T01:00:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f1754c52740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465c170 0x563fd4666630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f1754c52740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd4666b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f1754c52740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f1754c52740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563fd4664c20 con 0x563fd4661cd0 2024-03-18T01:00:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f1754c52740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563fd457d160 con 0x563fd465cb60 2024-03-18T01:00:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f1754c52740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563fd4664da0 con 0x563fd465c170 2024-03-18T01:00:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f1754c52740 10 client.0 _ll_register_callbacks cb 0x563fd4561150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:35.263 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f174bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f174affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd4666b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f174bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37872/0 (socket says 172.21.15.49:37872) 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f174bfff640 1 -- 172.21.15.49:0/590872588 learned_addr learned my addr 172.21.15.49:0/590872588 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.258+0000 7f174bfff640 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465c170 0x563fd4666630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f174bfff640 1 -- 172.21.15.49:0/590872588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465c170 msgr2=0x563fd4666630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f174bfff640 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465c170 0x563fd4666630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f174bfff640 1 -- 172.21.15.49:0/590872588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 msgr2=0x563fd4666b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f174bfff640 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd4666b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f174bfff640 1 -- 172.21.15.49:0/590872588 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fd439f7d0 con 0x563fd4661cd0 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f174affd640 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd4666b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f174bfff640 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466dbb0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1744009a60 tx=0x7f1744036040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e980efaf79efd8e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f172affd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f172affd640 1 -- 172.21.15.49:0/590872588 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744043070 con 0x563fd4661cd0 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f172affd640 1 -- 172.21.15.49:0/590872588 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f174402cca0 con 0x563fd4661cd0 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f1754c52740 1 -- 172.21.15.49:0/590872588 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1744032c90 con 0x563fd4661cd0 2024-03-18T01:00:35.264 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f172affd640 1 -- 172.21.15.49:0/590872588 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1744039040 con 0x563fd4661cd0 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f1754c52740 10 client.18042 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f1754c52740 1 -- 172.21.15.49:0/590872588 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563fd42d53f0 con 0x563fd4661cd0 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.259+0000 7f172affd640 1 -- 172.21.15.49:0/590872588 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1744032c90 con 0x563fd4661cd0 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f172affd640 1 -- 172.21.15.49:0/590872588 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f174403f440 con 0x563fd4661cd0 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f1754c52740 10 client.18042 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f1754c52740 10 client.18042 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f1754c52740 20 client.18042 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f1754c52740 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fd4671d10 0x563fd4674140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f1754c52740 4 client.18042 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f1754c52740 1 -- 172.21.15.49:0/590872588 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fd45cea40 con 0x563fd4671d10 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f174affd640 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fd4671d10 0x563fd4674140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f172affd640 10 client.18042 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.260+0000 7f174affd640 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fd4671d10 0x563fd4674140 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d23b60c9a7b47ee1 server_cookie=1a7a2b202b6275b5 in_seq=0 out_seq=0 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f172affd640 1 -- 172.21.15.49:0/590872588 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563fd45cea40 con 0x563fd4671d10 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f172affd640 10 client.18042 handle_command_reply: tid=0 2024-03-18T01:00:35.265 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 client.18042 shutdown 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 -- 172.21.15.49:0/590872588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 msgr2=0x563fd466dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466dbb0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1744009a60 tx=0x7f1744036040 comp rx=0 tx=0).stop 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 -- 172.21.15.49:0/590872588 shutdown_connections 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fd4671d10 0x563fd4674140 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd4666b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466dbb0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 --2- 172.21.15.49:0/590872588 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465c170 0x563fd4666630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 -- 172.21.15.49:0/590872588 >> 172.21.15.49:0/590872588 conn(0x563fd46576e0 msgr2=0x563fd4656900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f172affd640 0 client.18042 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 -- 172.21.15.49:0/590872588 shutdown_connections 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.261+0000 7f1754c52740 1 -- 172.21.15.49:0/590872588 wait complete. 2024-03-18T01:00:35.266 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.262+0000 7f1754c52740 20 client.18042 trim_cache size 0 max 16384 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.284+0000 7f1754c52740 1 Processor -- start 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.284+0000 7f1754c52740 1 -- start start 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f1754c52740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd465c170 0x563fd4657ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f1754c52740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd466db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f1754c52740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd4661cd0 0x563fd4656070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f1754c52740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f174c1ab020 con 0x563fd465c170 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f1754c52740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f174c1acb90 con 0x563fd465cb60 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f1754c52740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f174c1ab6f0 con 0x563fd4661cd0 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd466db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd465c170 0x563fd4657ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.285 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174affd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd466db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:56220/0 (socket says 172.21.15.49:56220) 2024-03-18T01:00:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174affd640 1 -- 172.21.15.49:0/3170432776 learned_addr learned my addr 172.21.15.49:0/3170432776 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174bfff640 1 --2- 172.21.15.49:0/3170432776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd4661cd0 0x563fd4656070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174affd640 1 -- 172.21.15.49:0/3170432776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd4661cd0 msgr2=0x563fd4656070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174affd640 1 --2- 172.21.15.49:0/3170432776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd4661cd0 0x563fd4656070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174affd640 1 -- 172.21.15.49:0/3170432776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd465c170 msgr2=0x563fd4657ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174affd640 1 --2- 172.21.15.49:0/3170432776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd465c170 0x563fd4657ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174affd640 1 -- 172.21.15.49:0/3170432776 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fd42950f0 con 0x563fd465cb60 2024-03-18T01:00:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.285+0000 7f174bfff640 1 --2- 172.21.15.49:0/3170432776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd465c170 0x563fd4657ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:35.286 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.286+0000 7f174affd640 1 --2- 172.21.15.49:0/3170432776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd466db70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1740004700 tx=0x7f1740005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17b2b99ecea65eec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.290+0000 7f17317fa640 1 -- 172.21.15.49:0/3170432776 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17400200f0 con 0x563fd465cb60 2024-03-18T01:00:35.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.290+0000 7f17317fa640 1 -- 172.21.15.49:0/3170432776 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1740020770 con 0x563fd465cb60 2024-03-18T01:00:35.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.290+0000 7f17317fa640 1 -- 172.21.15.49:0/3170432776 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1740010430 con 0x563fd465cb60 2024-03-18T01:00:35.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 -- 172.21.15.49:0/3170432776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 msgr2=0x563fd466db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 --2- 172.21.15.49:0/3170432776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd466db70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1740004700 tx=0x7f1740005f90 comp rx=0 tx=0).stop 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 -- 172.21.15.49:0/3170432776 shutdown_connections 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 --2- 172.21.15.49:0/3170432776 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465cb60 0x563fd466db70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 --2- 172.21.15.49:0/3170432776 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd4661cd0 0x563fd4656070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 --2- 172.21.15.49:0/3170432776 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd465c170 0x563fd4657ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 -- 172.21.15.49:0/3170432776 >> 172.21.15.49:0/3170432776 conn(0x563fd46576e0 msgr2=0x563fd465e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 -- 172.21.15.49:0/3170432776 shutdown_connections 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 -- 172.21.15.49:0/3170432776 wait complete. 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.291+0000 7f1754c52740 1 -- start start 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f1754c52740 1 Processor -- start 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f1754c52740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 0x563fd466b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f1754c52740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465cb60 0x563fd466b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f1754c52740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f1754c52740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f174c1ac490 con 0x563fd4661cd0 2024-03-18T01:00:35.292 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f1754c52740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f174c1abdc0 con 0x563fd465c170 2024-03-18T01:00:35.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f1754c52740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f174c1a9000 con 0x563fd465cb60 2024-03-18T01:00:35.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f1754c52740 10 client.0 _ll_register_callbacks cb 0x7f1740009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:35.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f174bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f174bfff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:37888/0 (socket says 172.21.15.49:37888) 2024-03-18T01:00:35.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f174affd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465cb60 0x563fd466b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f174bfff640 1 -- 172.21.15.49:0/1874954610 learned_addr learned my addr 172.21.15.49:0/1874954610 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:35.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f174bfff640 1 -- 172.21.15.49:0/1874954610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465cb60 msgr2=0x563fd466b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f174bfff640 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465cb60 0x563fd466b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f174bfff640 1 -- 172.21.15.49:0/1874954610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 msgr2=0x563fd466b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f174bfff640 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 0x563fd466b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.292+0000 7f174bfff640 1 -- 172.21.15.49:0/1874954610 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fd45889d0 con 0x563fd4661cd0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.293+0000 7f174bfff640 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466c620 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f174403fa20 tx=0x7f1744005fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef321a91c823b5e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.293+0000 7f172affd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.293+0000 7f172affd640 1 -- 172.21.15.49:0/1874954610 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744043070 con 0x563fd4661cd0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.293+0000 7f172affd640 1 -- 172.21.15.49:0/1874954610 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f17440039e0 con 0x563fd4661cd0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.293+0000 7f1754c52740 1 -- 172.21.15.49:0/1874954610 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1740020770 con 0x563fd4661cd0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.293+0000 7f172affd640 1 -- 172.21.15.49:0/1874954610 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1744066050 con 0x563fd4661cd0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.293+0000 7f1754c52740 10 client.18048 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.293+0000 7f1754c52740 1 -- 172.21.15.49:0/1874954610 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563fd466dd50 con 0x563fd4661cd0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.294+0000 7f172affd640 1 -- 172.21.15.49:0/1874954610 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1740020770 con 0x563fd4661cd0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.294+0000 7f172affd640 1 -- 172.21.15.49:0/1874954610 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f1744004410 con 0x563fd4661cd0 2024-03-18T01:00:35.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.294+0000 7f1754c52740 10 client.18048 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:35.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.294+0000 7f1754c52740 10 client.18048 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:35.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.294+0000 7f1754c52740 20 client.18048 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:35.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.294+0000 7f1754c52740 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fd46d8610 0x563fd46daa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:35.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.294+0000 7f1754c52740 4 client.18048 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:00:35.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.294+0000 7f1754c52740 1 -- 172.21.15.49:0/1874954610 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563fd45c8d90 con 0x563fd46d8610 2024-03-18T01:00:35.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.294+0000 7f174affd640 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fd46d8610 0x563fd46daa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:35.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.296+0000 7f172affd640 10 client.18048 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:35.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.296+0000 7f174affd640 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fd46d8610 0x563fd46daa30 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=907cd951274da42a server_cookie=5a4aad62d0512ad5 in_seq=0 out_seq=0 2024-03-18T01:00:35.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f172affd640 1 -- 172.21.15.49:0/1874954610 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563fd45c8d90 con 0x563fd46d8610 2024-03-18T01:00:35.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f172affd640 10 client.18048 handle_command_reply: tid=0 2024-03-18T01:00:35.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 client.18048 shutdown 2024-03-18T01:00:35.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 -- 172.21.15.49:0/1874954610 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 msgr2=0x563fd466c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466c620 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f174403fa20 tx=0x7f1744005fc0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 -- 172.21.15.49:0/1874954610 shutdown_connections 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x563fd46d8610 0x563fd46daa30 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563fd4661cd0 0x563fd466c620 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563fd465cb60 0x563fd466b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f172affd640 0 client.18048 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 --2- 172.21.15.49:0/1874954610 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563fd465c170 0x563fd466b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 -- 172.21.15.49:0/1874954610 >> 172.21.15.49:0/1874954610 conn(0x563fd46576e0 msgr2=0x563fd465a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.297+0000 7f1754c52740 1 -- 172.21.15.49:0/1874954610 shutdown_connections 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.298+0000 7f1754c52740 1 -- 172.21.15.49:0/1874954610 wait complete. 2024-03-18T01:00:35.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.298+0000 7f1754c52740 20 client.18048 trim_cache size 0 max 16384 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": { 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "b514c819-3a26-4f02-b8b0-f5f6188b46c9": { 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "path": "/", 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "tag": "b514c819-3a26-4f02-b8b0-f5f6188b46c9", 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: "options": "recursive,force" 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:00:35.299 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:00:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 -- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17340a2800 msgr2=0x7f17340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17340a2800 0x7f17340a4cc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f173c09b510 tx=0x7f173c06a3e0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 -- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c1006e0 msgr2=0x7f174c104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c1006e0 0x7f174c104620 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f173c00e9c0 tx=0x7f173c00ee80 comp rx=0 tx=0).stop 2024-03-18T01:00:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 -- 172.21.15.49:0/3087509857 shutdown_connections 2024-03-18T01:00:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f17340a2800 0x7f17340a4cc0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f174c102290 0x7f174c1031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f174c1018d0 0x7f174c102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.300 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 --2- 172.21.15.49:0/3087509857 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f174c1006e0 0x7f174c104620 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:35.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 -- 172.21.15.49:0/3087509857 >> 172.21.15.49:0/3087509857 conn(0x7f174c0fbc70 msgr2=0x7f174c0fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:35.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 -- 172.21.15.49:0/3087509857 shutdown_connections 2024-03-18T01:00:35.301 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:00:35.299+0000 7f1752061640 1 -- 172.21.15.49:0/3087509857 wait complete. 2024-03-18T01:00:35.315 INFO:tasks.cephfs.filesystem:scrub status for tag:b514c819-3a26-4f02-b8b0-f5f6188b46c9 - {'path': '/', 'tag': 'b514c819-3a26-4f02-b8b0-f5f6188b46c9', 'options': 'recursive,force'} 2024-03-18T01:00:35.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:35 smithi049 ceph-mon[32130]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 417 MiB/s rd, 14 MiB/s wr, 672 op/s 2024-03-18T01:00:35.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:35 smithi107 ceph-mon[36270]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 417 MiB/s rd, 14 MiB/s wr, 672 op/s 2024-03-18T01:00:35.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:35 smithi106 ceph-mon[35899]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 417 MiB/s rd, 14 MiB/s wr, 672 op/s 2024-03-18T01:00:36.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:36 smithi049 ceph-mon[32130]: scrub queued for path: / 2024-03-18T01:00:36.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:36 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:36.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:36 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T01:00:36.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:36 smithi049 ceph-mon[32130]: scrub summary: active paths [/] 2024-03-18T01:00:36.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:36 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:36.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:36 smithi049 ceph-mon[32130]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:36.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:36 smithi107 ceph-mon[36270]: scrub queued for path: / 2024-03-18T01:00:36.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:36 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:36.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:36 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T01:00:36.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:36 smithi107 ceph-mon[36270]: scrub summary: active paths [/] 2024-03-18T01:00:36.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:36 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:36.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:36 smithi107 ceph-mon[36270]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:36.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:36 smithi106 ceph-mon[35899]: scrub queued for path: / 2024-03-18T01:00:36.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:36 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:36.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:36 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T01:00:36.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:36 smithi106 ceph-mon[35899]: scrub summary: active paths [/] 2024-03-18T01:00:36.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:36 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:36.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:36 smithi106 ceph-mon[35899]: scrub summary: idle+waiting paths [/] 2024-03-18T01:00:37.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:37 smithi049 ceph-mon[32130]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 385 MiB/s rd, 12 MiB/s wr, 603 op/s 2024-03-18T01:00:37.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:37 smithi107 ceph-mon[36270]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 385 MiB/s rd, 12 MiB/s wr, 603 op/s 2024-03-18T01:00:37.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:37 smithi106 ceph-mon[35899]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 385 MiB/s rd, 12 MiB/s wr, 603 op/s 2024-03-18T01:00:38.335 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.335+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3934891691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cd4100700 msgr2=0x7f9cd4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:38.335 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.335+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3934891691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cd4100700 0x7f9cd4100b20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4009900 tx=0x7f9cc402bc30 comp rx=0 tx=0).stop 2024-03-18T01:00:38.335 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.335+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3934891691 shutdown_connections 2024-03-18T01:00:38.335 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.335+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3934891691 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd41022b0 0x7f9cd4108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.335+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3934891691 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cd41018f0 0x7f9cd4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.335+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3934891691 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cd4100700 0x7f9cd4100b20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.335+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3934891691 >> 172.21.15.49:0/3934891691 conn(0x7f9cd40fbc70 msgr2=0x7f9cd40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.335+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3934891691 shutdown_connections 2024-03-18T01:00:38.336 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.336+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3934891691 wait complete. 2024-03-18T01:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.336+0000 7f9cd95cd640 1 Processor -- start 2024-03-18T01:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.336+0000 7f9cd95cd640 1 -- start start 2024-03-18T01:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd95cd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cd4100700 0x7f9cd4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.338 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd95cd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cd41018f0 0x7f9cd4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd95cd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd41022b0 0x7f9cd419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd95cd640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd410d4a0 con 0x7f9cd41018f0 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd95cd640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd410d320 con 0x7f9cd4100700 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd95cd640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd410d620 con 0x7f9cd41022b0 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cd41018f0 0x7f9cd4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd41022b0 0x7f9cd419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd37fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd41022b0 0x7f9cd419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56536/0 (socket says 172.21.15.49:56536) 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd37fe640 1 -- 172.21.15.49:0/3011071911 learned_addr learned my addr 172.21.15.49:0/3011071911 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cd41018f0 0x7f9cd4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34058/0 (socket says 172.21.15.49:34058) 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd37fe640 1 -- 172.21.15.49:0/3011071911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cd4100700 msgr2=0x7f9cd4197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd37fe640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cd4100700 0x7f9cd4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd37fe640 1 -- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cd41018f0 msgr2=0x7f9cd4198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd37fe640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cd41018f0 0x7f9cd4198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.337+0000 7f9cd37fe640 1 -- 172.21.15.49:0/3011071911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cd419f690 con 0x7f9cd41022b0 2024-03-18T01:00:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.338+0000 7f9cd37fe640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd41022b0 0x7f9cd419f020 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00e970 tx=0x7f9ccc00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.338+0000 7f9ccbfff640 1 -- 172.21.15.49:0/3011071911 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc00ccc0 con 0x7f9cd41022b0 2024-03-18T01:00:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.338+0000 7f9ccbfff640 1 -- 172.21.15.49:0/3011071911 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ccc00ce60 con 0x7f9cd41022b0 2024-03-18T01:00:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.338+0000 7f9ccbfff640 1 -- 172.21.15.49:0/3011071911 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ccc010490 con 0x7f9cd41022b0 2024-03-18T01:00:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.338+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3011071911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cd419f980 con 0x7f9cd41022b0 2024-03-18T01:00:38.341 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.338+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3011071911 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cd419fe80 con 0x7f9cd41022b0 2024-03-18T01:00:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9ccbfff640 1 -- 172.21.15.49:0/3011071911 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9ccc005020 con 0x7f9cd41022b0 2024-03-18T01:00:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cdc1b9740 1 Processor -- start 2024-03-18T01:00:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cdc1b9740 1 -- start start 2024-03-18T01:00:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 0x555e21612b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21617c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e2161a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555e21573a40 con 0x555e216182b0 2024-03-18T01:00:38.346 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555e2151c920 con 0x555e21612750 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555e2156fae0 con 0x555e21613140 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cd27fc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 0x555e21612b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cd2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21617c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cd27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e2161a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cd2ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21617c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:56544/0 (socket says 172.21.15.49:56544) 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.341+0000 7f9cd2ffd640 1 -- 172.21.15.49:0/2964144160 learned_addr learned my addr 172.21.15.49:0/2964144160 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9ccbfff640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9cc00a2920 0x7f9cc00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cd2ffd640 1 -- 172.21.15.49:0/2964144160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 msgr2=0x555e21612b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/2964144160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 0x555e21612b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cd2ffd640 1 -- 172.21.15.49:0/2964144160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 msgr2=0x555e2161a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/2964144160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e2161a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cd2ffd640 1 -- 172.21.15.49:0/2964144160 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e21534460 con 0x555e21613140 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/2964144160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 0x555e21612b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:00:38.347 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cd37fe640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9cc00a2920 0x7f9cc00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9ccbfff640 1 -- 172.21.15.49:0/3011071911 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9ccc0ceef0 con 0x7f9cd41022b0 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/2964144160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21617c40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9cc402c8e0 tx=0x7f9cc4005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3576b8c734e03a4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cc97fa640 1 -- 172.21.15.49:0/2964144160 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4041020 con 0x555e21613140 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cc97fa640 1 -- 172.21.15.49:0/2964144160 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cc402cb80 con 0x555e21613140 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.342+0000 7f9cc97fa640 1 -- 172.21.15.49:0/2964144160 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4003d20 con 0x555e21613140 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/2964144160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 msgr2=0x555e21617c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/2964144160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21617c40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9cc402c8e0 tx=0x7f9cc4005780 comp rx=0 tx=0).stop 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cd37fe640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9cc00a2920 0x7f9cc00a4de0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00d010 tx=0x7f9ccc066580 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/2964144160 shutdown_connections 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/2964144160 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e2161a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/2964144160 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21617c40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/2964144160 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 0x555e21612b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/2964144160 >> 172.21.15.49:0/2964144160 conn(0x555e216098b0 msgr2=0x555e21609cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/2964144160 shutdown_connections 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/2964144160 wait complete. 2024-03-18T01:00:38.348 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 -- start start 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.343+0000 7f9cdc1b9740 1 Processor -- start 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 0x555e2161ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e2161cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21623fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x555e2161b200 con 0x555e216182b0 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x555e2152fd70 con 0x555e21612750 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x555e2161b380 con 0x555e21613140 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cdc1b9740 10 client.0 _ll_register_callbacks cb 0x555e21287a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21623fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21623fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34074/0 (socket says 172.21.15.49:34074) 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 -- 172.21.15.49:0/1680427775 learned_addr learned my addr 172.21.15.49:0/1680427775 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 0x555e2161ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.349 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e2161cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 -- 172.21.15.49:0/1680427775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 msgr2=0x555e2161cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e2161cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 -- 172.21.15.49:0/1680427775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 msgr2=0x555e2161ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 0x555e2161ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 -- 172.21.15.49:0/1680427775 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e21288210 con 0x555e216182b0 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e2161cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21623fe0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00f760 tx=0x7f9cbc009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59bd698bde0214c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.344+0000 7f9cba7fc640 1 -- 172.21.15.49:0/1680427775 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc019040 con 0x555e216182b0 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1680427775 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cc402cb80 con 0x555e216182b0 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cba7fc640 1 -- 172.21.15.49:0/1680427775 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9cbc004d20 con 0x555e216182b0 2024-03-18T01:00:38.350 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cba7fc640 1 -- 172.21.15.49:0/1680427775 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9cbc028050 con 0x555e216182b0 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cdc1b9740 10 client.18060 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1680427775 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555e21354620 con 0x555e216182b0 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cba7fc640 1 -- 172.21.15.49:0/1680427775 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cc402cb80 con 0x555e216182b0 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cba7fc640 1 -- 172.21.15.49:0/1680427775 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f9cbc01f070 con 0x555e216182b0 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cdc1b9740 10 client.18060 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cdc1b9740 10 client.18060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cdc1b9740 20 client.18060 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555e21628890 0x555e2162ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cdc1b9740 4 client.18060 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1680427775 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e21580f80 con 0x555e21628890 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.345+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555e21628890 0x555e2162ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.346+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555e21628890 0x555e2162ad00 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5829a78ecccd948 server_cookie=6068bf03f807aca5 in_seq=0 out_seq=0 2024-03-18T01:00:38.351 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.346+0000 7f9cba7fc640 10 client.18060 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cba7fc640 1 -- 172.21.15.49:0/1680427775 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555e21580f80 con 0x555e21628890 2024-03-18T01:00:38.352 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cba7fc640 10 client.18060 handle_command_reply: tid=0 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 client.18060 shutdown 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1680427775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 msgr2=0x555e21623fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21623fe0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00f760 tx=0x7f9cbc009ef0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1680427775 shutdown_connections 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555e21628890 0x555e2162ad00 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21623fe0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e2161cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1680427775 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21612750 0x555e2161ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1680427775 >> 172.21.15.49:0/1680427775 conn(0x555e216098b0 msgr2=0x555e21608bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cba7fc640 0 client.18060 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1680427775 shutdown_connections 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1680427775 wait complete. 2024-03-18T01:00:38.353 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.347+0000 7f9cdc1b9740 20 client.18060 trim_cache size 0 max 16384 2024-03-18T01:00:38.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cdc1b9740 1 Processor -- start 2024-03-18T01:00:38.364 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cdc1b9740 1 -- start start 2024-03-18T01:00:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21612750 0x555e21608320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21613140 0x555e21623fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21608860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd41a6bb0 con 0x555e216182b0 2024-03-18T01:00:38.365 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd41a8720 con 0x555e21613140 2024-03-18T01:00:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd41a7280 con 0x555e21612750 2024-03-18T01:00:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cd27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21608860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.365+0000 7f9cd27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21608860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34082/0 (socket says 172.21.15.49:34082) 2024-03-18T01:00:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cd27fc640 1 -- 172.21.15.49:0/1666196686 learned_addr learned my addr 172.21.15.49:0/1666196686 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/1666196686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21613140 0x555e21623fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.366 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/1666196686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21612750 0x555e21608320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cd27fc640 1 -- 172.21.15.49:0/1666196686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21612750 msgr2=0x555e21608320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/1666196686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21612750 0x555e21608320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cd27fc640 1 -- 172.21.15.49:0/1666196686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21613140 msgr2=0x555e21623fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/1666196686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21613140 0x555e21623fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cd27fc640 1 -- 172.21.15.49:0/1666196686 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e2161b500 con 0x555e216182b0 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/1666196686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21608860 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00f460 tx=0x7f9cbc009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82458dd6bfb214ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cc8ff9640 1 -- 172.21.15.49:0/1666196686 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc005990 con 0x555e216182b0 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cc8ff9640 1 -- 172.21.15.49:0/1666196686 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cbc004f20 con 0x555e216182b0 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.366+0000 7f9cc8ff9640 1 -- 172.21.15.49:0/1666196686 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc00ca00 con 0x555e216182b0 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.367+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1666196686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 msgr2=0x555e21608860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:38.370 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.367+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1666196686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21608860 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00f460 tx=0x7f9cbc009820 comp rx=0 tx=0).stop 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.367+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1666196686 shutdown_connections 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.367+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1666196686 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e21613140 0x555e21623fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.367+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1666196686 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e216182b0 0x555e21608860 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.367+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/1666196686 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21612750 0x555e21608320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.367+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1666196686 >> 172.21.15.49:0/1666196686 conn(0x555e216098b0 msgr2=0x555e2162a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.367+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1666196686 shutdown_connections 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.367+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/1666196686 wait complete. 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cdc1b9740 1 -- start start 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cdc1b9740 1 Processor -- start 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e21612750 0x555e21621de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21622320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cdc1b9740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e216182b0 0x555e21623330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd41a8020 con 0x555e21612750 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd41a7950 con 0x555e216182b0 2024-03-18T01:00:38.371 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cdc1b9740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f9cd41a4b90 con 0x555e21613140 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cdc1b9740 10 client.0 _ll_register_callbacks cb 0x7f9cbc010590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e21612750 0x555e21621de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd27fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e21612750 0x555e21621de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34098/0 (socket says 172.21.15.49:34098) 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd27fc640 1 -- 172.21.15.49:0/497560537 learned_addr learned my addr 172.21.15.49:0/497560537 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21622320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e216182b0 0x555e21623330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd27fc640 1 -- 172.21.15.49:0/497560537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 msgr2=0x555e21622320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21622320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd27fc640 1 -- 172.21.15.49:0/497560537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e216182b0 msgr2=0x555e21623330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e216182b0 0x555e21623330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.368+0000 7f9cd27fc640 1 -- 172.21.15.49:0/497560537 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e21623920 con 0x555e21612750 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cd27fc640 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e21612750 0x555e21621de0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc010010 tx=0x7f9cbc009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3527a360a6105537 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cba7fc640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cba7fc640 1 -- 172.21.15.49:0/497560537 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc004a00 con 0x555e21612750 2024-03-18T01:00:38.372 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/497560537 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cbc004f20 con 0x555e21612750 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cba7fc640 1 -- 172.21.15.49:0/497560537 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9cbc004ba0 con 0x555e21612750 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cba7fc640 1 -- 172.21.15.49:0/497560537 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9cbc041070 con 0x555e21612750 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cdc1b9740 10 client.18066 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/497560537 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555e2168f350 con 0x555e21612750 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cba7fc640 1 -- 172.21.15.49:0/497560537 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cbc004f20 con 0x555e21612750 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cba7fc640 1 -- 172.21.15.49:0/497560537 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f9cbc01f070 con 0x555e21612750 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cdc1b9740 10 client.18066 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cdc1b9740 10 client.18066 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cdc1b9740 20 client.18066 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555e2168fce0 0x555e21692100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cdc1b9740 4 client.18066 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.369+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/497560537 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555e2157b2d0 con 0x555e2168fce0 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.370+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555e2168fce0 0x555e21692100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.370+0000 7f9cd2ffd640 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555e2168fce0 0x555e21692100 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=619d740d49e7df21 server_cookie=6f44db48e61f787b in_seq=0 out_seq=0 2024-03-18T01:00:38.373 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.370+0000 7f9cba7fc640 10 client.18066 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.380+0000 7f9cba7fc640 1 -- 172.21.15.49:0/497560537 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555e2157b2d0 con 0x555e2168fce0 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cba7fc640 10 client.18066 handle_command_reply: tid=0 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 client.18066 shutdown 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/497560537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e21612750 msgr2=0x555e21621de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e21612750 0x555e21621de0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc010010 tx=0x7f9cbc009ef0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/497560537 shutdown_connections 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x555e2168fce0 0x555e21692100 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x555e216182b0 0x555e21623330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cba7fc640 0 client.18066 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:38.381 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x555e21613140 0x555e21622320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 --2- 172.21.15.49:0/497560537 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x555e21612750 0x555e21621de0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/497560537 >> 172.21.15.49:0/497560537 conn(0x555e216098b0 msgr2=0x555e2160cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/497560537 shutdown_connections 2024-03-18T01:00:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 1 -- 172.21.15.49:0/497560537 wait complete. 2024-03-18T01:00:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cdc1b9740 20 client.18066 trim_cache size 0 max 16384 2024-03-18T01:00:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T01:00:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T01:00:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T01:00:38.382 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T01:00:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9cc00a2920 msgr2=0x7f9cc00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9cc00a2920 0x7f9cc00a4de0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00d010 tx=0x7f9ccc066580 comp rx=0 tx=0).stop 2024-03-18T01:00:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3011071911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd41022b0 msgr2=0x7f9cd419f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.381+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd41022b0 0x7f9cd419f020 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc00e970 tx=0x7f9ccc00ed70 comp rx=0 tx=0).stop 2024-03-18T01:00:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.382+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3011071911 shutdown_connections 2024-03-18T01:00:38.383 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.382+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f9cc00a2920 0x7f9cc00a4de0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.382+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9cd41022b0 0x7f9cd419f020 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.382+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f9cd41018f0 0x7f9cd4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.382+0000 7f9cd95cd640 1 --2- 172.21.15.49:0/3011071911 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f9cd4100700 0x7f9cd4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:38.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.382+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3011071911 >> 172.21.15.49:0/3011071911 conn(0x7f9cd40fbc70 msgr2=0x7f9cd4104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:38.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.382+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3011071911 shutdown_connections 2024-03-18T01:00:38.384 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:38.382+0000 7f9cd95cd640 1 -- 172.21.15.49:0/3011071911 wait complete. 2024-03-18T01:00:39.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:39 smithi049 ceph-mon[32130]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 415 MiB/s rd, 15 MiB/s wr, 756 op/s 2024-03-18T01:00:39.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:39 smithi107 ceph-mon[36270]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 415 MiB/s rd, 15 MiB/s wr, 756 op/s 2024-03-18T01:00:39.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:39 smithi106 ceph-mon[35899]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 415 MiB/s rd, 15 MiB/s wr, 756 op/s 2024-03-18T01:00:40.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:40 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T01:00:40.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:40 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T01:00:40.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:40 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T01:00:41.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:41 smithi049 ceph-mon[32130]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 354 MiB/s rd, 12 MiB/s wr, 574 op/s 2024-03-18T01:00:41.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:41 smithi049 ceph-mon[32130]: scrub summary: idle 2024-03-18T01:00:41.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:41 smithi107 ceph-mon[36270]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 354 MiB/s rd, 12 MiB/s wr, 574 op/s 2024-03-18T01:00:41.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:41 smithi107 ceph-mon[36270]: scrub summary: idle 2024-03-18T01:00:41.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:41 smithi106 ceph-mon[35899]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 354 MiB/s rd, 12 MiB/s wr, 574 op/s 2024-03-18T01:00:41.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:41 smithi106 ceph-mon[35899]: scrub summary: idle 2024-03-18T01:00:43.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:43 smithi049 ceph-mon[32130]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 364 MiB/s rd, 13 MiB/s wr, 680 op/s 2024-03-18T01:00:43.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:43 smithi107 ceph-mon[36270]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 364 MiB/s rd, 13 MiB/s wr, 680 op/s 2024-03-18T01:00:43.696 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:43 smithi106 ceph-mon[35899]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 364 MiB/s rd, 13 MiB/s wr, 680 op/s 2024-03-18T01:00:45.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:45 smithi049 ceph-mon[32130]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 MiB/s rd, 13 MiB/s wr, 609 op/s 2024-03-18T01:00:45.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:45 smithi107 ceph-mon[36270]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 MiB/s rd, 13 MiB/s wr, 609 op/s 2024-03-18T01:00:45.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:45 smithi106 ceph-mon[35899]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 380 MiB/s rd, 13 MiB/s wr, 609 op/s 2024-03-18T01:00:47.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:47 smithi049 ceph-mon[32130]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 356 MiB/s rd, 12 MiB/s wr, 563 op/s 2024-03-18T01:00:47.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:47 smithi107 ceph-mon[36270]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 356 MiB/s rd, 12 MiB/s wr, 563 op/s 2024-03-18T01:00:47.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:47 smithi106 ceph-mon[35899]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 356 MiB/s rd, 12 MiB/s wr, 563 op/s 2024-03-18T01:00:49.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:49 smithi049 ceph-mon[32130]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 452 MiB/s rd, 15 MiB/s wr, 778 op/s 2024-03-18T01:00:49.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:49 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:00:49.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:49 smithi107 ceph-mon[36270]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 452 MiB/s rd, 15 MiB/s wr, 778 op/s 2024-03-18T01:00:49.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:49 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:00:49.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:49 smithi106 ceph-mon[35899]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 452 MiB/s rd, 15 MiB/s wr, 778 op/s 2024-03-18T01:00:49.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:49 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:00:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.502+0000 7f5209997640 1 -- 172.21.15.49:0/3191756646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 msgr2=0x7f5204104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:51.502 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.502+0000 7f5209997640 1 --2- 172.21.15.49:0/3191756646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 0x7f5204104730 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f51f4009900 tx=0x7f51f402bc30 comp rx=0 tx=0).stop 2024-03-18T01:00:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.503+0000 7f5209997640 1 -- 172.21.15.49:0/3191756646 shutdown_connections 2024-03-18T01:00:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.503+0000 7f5209997640 1 --2- 172.21.15.49:0/3191756646 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52041075a0 0x7f5204109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.503+0000 7f5209997640 1 --2- 172.21.15.49:0/3191756646 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5204104c70 0x7f5204107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.503+0000 7f5209997640 1 --2- 172.21.15.49:0/3191756646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 0x7f5204104730 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.503+0000 7f5209997640 1 -- 172.21.15.49:0/3191756646 >> 172.21.15.49:0/3191756646 conn(0x7f52040fbc70 msgr2=0x7f52040fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.503+0000 7f5209997640 1 -- 172.21.15.49:0/3191756646 shutdown_connections 2024-03-18T01:00:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.503+0000 7f5209997640 1 -- 172.21.15.49:0/3191756646 wait complete. 2024-03-18T01:00:51.503 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.504+0000 7f5209997640 1 Processor -- start 2024-03-18T01:00:51.504 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5209997640 1 -- start start 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5209997640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 0x7f5204197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5209997640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5204104c70 0x7f5204197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5209997640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52041075a0 0x7f520419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f52027fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5204104c70 0x7f5204197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 0x7f5204197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 0x7f5204197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38596/0 (socket says 172.21.15.49:38596) 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f52027fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5204104c70 0x7f5204197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36698/0 (socket says 172.21.15.49:36698) 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5202ffd640 1 -- 172.21.15.49:0/3440596615 learned_addr learned my addr 172.21.15.49:0/3440596615 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f520410d4b0 con 0x7f5204104c70 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f520410d330 con 0x7f52041075a0 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f520410d630 con 0x7f5204102340 2024-03-18T01:00:51.505 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.505+0000 7f52037fe640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52041075a0 0x7f520419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f52037fe640 1 -- 172.21.15.49:0/3440596615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 msgr2=0x7f5204197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f52037fe640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 0x7f5204197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f52037fe640 1 -- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5204104c70 msgr2=0x7f5204197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f52037fe640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5204104c70 0x7f5204197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f52037fe640 1 -- 172.21.15.49:0/3440596615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520419f6a0 con 0x7f52041075a0 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f52027fc640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5204104c70 0x7f5204197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f5202ffd640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 0x7f5204197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f52037fe640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52041075a0 0x7f520419f030 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009fd0 tx=0x7f51fc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f5208995640 1 -- 172.21.15.49:0/3440596615 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc00ce20 con 0x7f52041075a0 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520419f990 con 0x7f52041075a0 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f5208995640 1 -- 172.21.15.49:0/3440596615 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51fc005020 con 0x7f52041075a0 2024-03-18T01:00:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f5208995640 1 -- 172.21.15.49:0/3440596615 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc010720 con 0x7f52041075a0 2024-03-18T01:00:51.508 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.506+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520419ff30 con 0x7f52041075a0 2024-03-18T01:00:51.511 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.510+0000 7f520c583740 1 Processor -- start 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.510+0000 7f5208995640 1 -- 172.21.15.49:0/3440596615 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f51fc005540 con 0x7f52041075a0 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.510+0000 7f520c583740 1 -- start start 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f520c583740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05a340 0x55fa1d05a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f520c583740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f520c583740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d062330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f520c583740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55fa1cfbfa40 con 0x55fa1d05fea0 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f520c583740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55fa1cf68920 con 0x55fa1d05a340 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f520c583740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fa1cfbbae0 con 0x55fa1d05ad30 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d062330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d062330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36706/0 (socket says 172.21.15.49:36706) 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 -- 172.21.15.49:0/2163666891 learned_addr learned my addr 172.21.15.49:0/2163666891 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:51.512 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05a340 0x55fa1d05a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f5202ffd640 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f5208995640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51ec0a6d30 0x7f51ec0a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 -- 172.21.15.49:0/2163666891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 msgr2=0x55fa1d05f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 -- 172.21.15.49:0/2163666891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05a340 msgr2=0x55fa1d05a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05a340 0x55fa1d05a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 -- 172.21.15.49:0/2163666891 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa1cf80460 con 0x55fa1d05fea0 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f5202ffd640 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52037fe640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51ec0a6d30 0x7f51ec0a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.511+0000 7f52027fc640 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d062330 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f51f8011b80 tx=0x7f51f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7003095e6eadcbf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.512+0000 7f5208995640 1 -- 172.21.15.49:0/3440596615 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f51fc0cf010 con 0x7f52041075a0 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.512+0000 7f51e9ffb640 1 -- 172.21.15.49:0/2163666891 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f800fc50 con 0x55fa1d05fea0 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.512+0000 7f51e9ffb640 1 -- 172.21.15.49:0/2163666891 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f800fdf0 con 0x55fa1d05fea0 2024-03-18T01:00:51.513 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.512+0000 7f51e9ffb640 1 -- 172.21.15.49:0/2163666891 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8003b90 con 0x55fa1d05fea0 2024-03-18T01:00:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.512+0000 7f520c583740 1 -- 172.21.15.49:0/2163666891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 msgr2=0x55fa1d062330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.512+0000 7f520c583740 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d062330 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f51f8011b80 tx=0x7f51f8004730 comp rx=0 tx=0).stop 2024-03-18T01:00:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.512+0000 7f52037fe640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51ec0a6d30 0x7f51ec0a91f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f51fc097df0 tx=0x7f51fc06d610 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 -- 172.21.15.49:0/2163666891 shutdown_connections 2024-03-18T01:00:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d062330 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 --2- 172.21.15.49:0/2163666891 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05a340 0x55fa1d05a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.517 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 -- 172.21.15.49:0/2163666891 >> 172.21.15.49:0/2163666891 conn(0x55fa1d0558b0 msgr2=0x55fa1d055cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 -- 172.21.15.49:0/2163666891 shutdown_connections 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 -- 172.21.15.49:0/2163666891 wait complete. 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 -- start start 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 Processor -- start 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.513+0000 7f520c583740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05a340 0x55fa1d064640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f520c583740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d064b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f520c583740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f520c583740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55fa1d062e50 con 0x55fa1d05a340 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f520c583740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55fa1cf7bd70 con 0x55fa1d05fea0 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f520c583740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55fa1d062fd0 con 0x55fa1d05ad30 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f520c583740 10 client.0 _ll_register_callbacks cb 0x55fa1ccd3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f52027fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05a340 0x55fa1d064640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.518 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d064b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d064b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38620/0 (socket says 172.21.15.49:38620) 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f52027fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05a340 0x55fa1d064640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36712/0 (socket says 172.21.15.49:36712) 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f5202ffd640 1 -- 172.21.15.49:0/64349074 learned_addr learned my addr 172.21.15.49:0/64349074 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f52027fc640 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f52027fc640 1 -- 172.21.15.49:0/64349074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 msgr2=0x55fa1d064b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f52027fc640 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d064b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f52027fc640 1 -- 172.21.15.49:0/64349074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05fea0 msgr2=0x55fa1d06bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f52027fc640 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f52027fc640 1 -- 172.21.15.49:0/64349074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa1ccd4210 con 0x55fa1d05a340 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f5202ffd640 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d064b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f51e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.514+0000 7f52027fc640 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05a340 0x55fa1d064640 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f51f8006010 tx=0x7f51f80035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28ae1ccf5b12eddc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.515+0000 7f51e2ffd640 1 -- 172.21.15.49:0/64349074 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f800fc20 con 0x55fa1d05a340 2024-03-18T01:00:51.519 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.515+0000 7f520c583740 1 -- 172.21.15.49:0/64349074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51f800fdf0 con 0x55fa1d05a340 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.515+0000 7f51e2ffd640 1 -- 172.21.15.49:0/64349074 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f51f8019040 con 0x55fa1d05a340 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.515+0000 7f51e2ffd640 1 -- 172.21.15.49:0/64349074 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f51f8034050 con 0x55fa1d05a340 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.515+0000 7f520c583740 10 client.18078 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.515+0000 7f520c583740 1 -- 172.21.15.49:0/64349074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fa1cda0620 con 0x55fa1d05a340 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.515+0000 7f51e2ffd640 1 -- 172.21.15.49:0/64349074 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f800fdf0 con 0x55fa1d05a340 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.515+0000 7f51e2ffd640 1 -- 172.21.15.49:0/64349074 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f51f80047a0 con 0x55fa1d05a340 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.515+0000 7f520c583740 10 client.18078 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.516+0000 7f520c583740 10 client.18078 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.516+0000 7f520c583740 20 client.18078 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.516+0000 7f520c583740 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fa1d070660 0x55fa1d072a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.516+0000 7f520c583740 4 client.18078 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.516+0000 7f520c583740 1 -- 172.21.15.49:0/64349074 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa1cfccf80 con 0x55fa1d070660 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.516+0000 7f5202ffd640 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fa1d070660 0x55fa1d072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.520 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.516+0000 7f51e2ffd640 10 client.18078 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.516+0000 7f5202ffd640 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fa1d070660 0x55fa1d072a80 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6ce0ce0482b2eb server_cookie=fbc06c23bcf982c7 in_seq=0 out_seq=0 2024-03-18T01:00:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.517+0000 7f51e2ffd640 1 -- 172.21.15.49:0/64349074 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fa1cfccf80 con 0x55fa1d070660 2024-03-18T01:00:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.517+0000 7f51e2ffd640 10 client.18078 handle_command_reply: tid=0 2024-03-18T01:00:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.517+0000 7f520c583740 1 client.18078 shutdown 2024-03-18T01:00:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 -- 172.21.15.49:0/64349074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05a340 msgr2=0x55fa1d064640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05a340 0x55fa1d064640 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f51f8006010 tx=0x7f51f80035f0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 -- 172.21.15.49:0/64349074 shutdown_connections 2024-03-18T01:00:51.521 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fa1d070660 0x55fa1d072a80 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d064b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 --2- 172.21.15.49:0/64349074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05a340 0x55fa1d064640 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 -- 172.21.15.49:0/64349074 >> 172.21.15.49:0/64349074 conn(0x55fa1d0558b0 msgr2=0x55fa1d054bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f51e2ffd640 0 client.18078 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 -- 172.21.15.49:0/64349074 shutdown_connections 2024-03-18T01:00:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 1 -- 172.21.15.49:0/64349074 wait complete. 2024-03-18T01:00:51.522 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.518+0000 7f520c583740 20 client.18078 trim_cache size 0 max 16384 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.535+0000 7f520c583740 1 Processor -- start 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f520c583740 1 -- start start 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f520c583740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f520c583740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f520c583740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d055c90 0x55fa1d064250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f5202ffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36728/0 (socket says 172.21.15.49:36728) 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f52027fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f52027fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38626/0 (socket says 172.21.15.49:38626) 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f5202ffd640 1 -- 172.21.15.49:0/604541124 learned_addr learned my addr 172.21.15.49:0/604541124 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:51.536 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f520c583740 1 -- 172.21.15.49:0/604541124 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f52041a6b40 con 0x55fa1d05fea0 2024-03-18T01:00:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f520c583740 1 -- 172.21.15.49:0/604541124 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f52041a86b0 con 0x55fa1d055c90 2024-03-18T01:00:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f520c583740 1 -- 172.21.15.49:0/604541124 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f52041a7210 con 0x55fa1d05ad30 2024-03-18T01:00:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.536+0000 7f52027fc640 1 --2- 172.21.15.49:0/604541124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d055c90 0x55fa1d064250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f5202ffd640 1 -- 172.21.15.49:0/604541124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 msgr2=0x55fa1d05b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:51.537 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f5202ffd640 1 --2- 172.21.15.49:0/604541124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f5202ffd640 1 -- 172.21.15.49:0/604541124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d055c90 msgr2=0x55fa1d064250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f5202ffd640 1 --2- 172.21.15.49:0/604541124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d055c90 0x55fa1d064250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f5202ffd640 1 -- 172.21.15.49:0/604541124 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa1d05b110 con 0x55fa1d05fea0 2024-03-18T01:00:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f52027fc640 1 --2- 172.21.15.49:0/604541124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:00:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f5202ffd640 1 --2- 172.21.15.49:0/604541124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bb80 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f51f4005ab0 tx=0x7f51f402c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6e82280e90503 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f51e97fa640 1 -- 172.21.15.49:0/604541124 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f404f070 con 0x55fa1d05fea0 2024-03-18T01:00:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f51e97fa640 1 -- 172.21.15.49:0/604541124 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f4004ba0 con 0x55fa1d05fea0 2024-03-18T01:00:51.538 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.537+0000 7f51e97fa640 1 -- 172.21.15.49:0/604541124 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f40054b0 con 0x55fa1d05fea0 2024-03-18T01:00:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.538+0000 7f520c583740 1 -- 172.21.15.49:0/604541124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 msgr2=0x55fa1d06bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.538+0000 7f520c583740 1 --2- 172.21.15.49:0/604541124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bb80 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f51f4005ab0 tx=0x7f51f402c880 comp rx=0 tx=0).stop 2024-03-18T01:00:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.539+0000 7f520c583740 1 -- 172.21.15.49:0/604541124 shutdown_connections 2024-03-18T01:00:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.539+0000 7f520c583740 1 --2- 172.21.15.49:0/604541124 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d055c90 0x55fa1d064250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.539 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.539+0000 7f520c583740 1 --2- 172.21.15.49:0/604541124 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d05fea0 0x55fa1d06bb80 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.539+0000 7f520c583740 1 --2- 172.21.15.49:0/604541124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05ad30 0x55fa1d05b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.539+0000 7f520c583740 1 -- 172.21.15.49:0/604541124 >> 172.21.15.49:0/604541124 conn(0x55fa1d0558b0 msgr2=0x55fa1d050a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:51.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.539+0000 7f520c583740 1 -- 172.21.15.49:0/604541124 shutdown_connections 2024-03-18T01:00:51.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.539+0000 7f520c583740 1 -- 172.21.15.49:0/604541124 wait complete. 2024-03-18T01:00:51.540 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.540+0000 7f520c583740 1 -- start start 2024-03-18T01:00:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.540+0000 7f520c583740 1 Processor -- start 2024-03-18T01:00:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f520c583740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d055c90 0x55fa1d068e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.541 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f520c583740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05ad30 0x55fa1d0693b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f520c583740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05fea0 0x55fa1d06a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f520c583740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f52041a7fb0 con 0x55fa1d055c90 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f520c583740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f52041a78e0 con 0x55fa1d05ad30 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f520c583740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f52041a4b20 con 0x55fa1d05fea0 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f520c583740 10 client.0 _ll_register_callbacks cb 0x7f51f8011670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f52027fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d055c90 0x55fa1d068e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f52027fc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d055c90 0x55fa1d068e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:36742/0 (socket says 172.21.15.49:36742) 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f52027fc640 1 -- 172.21.15.49:0/957966058 learned_addr learned my addr 172.21.15.49:0/957966058 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f52027fc640 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05fea0 0x55fa1d06a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f5202ffd640 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05ad30 0x55fa1d0693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.541+0000 7f52027fc640 1 -- 172.21.15.49:0/957966058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05fea0 msgr2=0x55fa1d06a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.542+0000 7f52027fc640 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05fea0 0x55fa1d06a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.542+0000 7f52027fc640 1 -- 172.21.15.49:0/957966058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05ad30 msgr2=0x55fa1d0693b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:00:51.542 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.542+0000 7f52027fc640 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05ad30 0x55fa1d0693b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.542+0000 7f52027fc640 1 -- 172.21.15.49:0/957966058 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa1d06a880 con 0x55fa1d055c90 2024-03-18T01:00:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.542+0000 7f52027fc640 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d055c90 0x55fa1d068e70 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f51f80059f0 tx=0x7f51f80217f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e29bd1dd3fe80b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:00:51.543 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.542+0000 7f51e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:00:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.542+0000 7f51e2ffd640 1 -- 172.21.15.49:0/957966058 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f800fe30 con 0x55fa1d055c90 2024-03-18T01:00:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.542+0000 7f51e2ffd640 1 -- 172.21.15.49:0/957966058 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f51f8003660 con 0x55fa1d055c90 2024-03-18T01:00:51.544 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.542+0000 7f520c583740 1 -- 172.21.15.49:0/957966058 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51f4004ba0 con 0x55fa1d055c90 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f51e2ffd640 1 -- 172.21.15.49:0/957966058 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f51f8003860 con 0x55fa1d055c90 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f520c583740 10 client.18084 fetch_fsmap learned FSMap version 30 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f520c583740 1 -- 172.21.15.49:0/957966058 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fa1d06bb90 con 0x55fa1d055c90 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f51e2ffd640 1 -- 172.21.15.49:0/957966058 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f4004ba0 con 0x55fa1d055c90 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f51e2ffd640 1 -- 172.21.15.49:0/957966058 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f51f80051b0 con 0x55fa1d055c90 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f520c583740 10 client.18084 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f520c583740 10 client.18084 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f520c583740 20 client.18084 populate_metadata read hostname 'smithi049' 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f520c583740 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fa1d0d6e60 0x55fa1d0d9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f520c583740 4 client.18084 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.543+0000 7f520c583740 1 -- 172.21.15.49:0/957966058 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fa1cfc72d0 con 0x55fa1d0d6e60 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.544+0000 7f5202ffd640 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fa1d0d6e60 0x55fa1d0d9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.545+0000 7f51e2ffd640 10 client.18084 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:51.545 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.545+0000 7f5202ffd640 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fa1d0d6e60 0x55fa1d0d9280 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c58b1b7ac948fc07 server_cookie=e51a199914a86e5d in_seq=0 out_seq=0 2024-03-18T01:00:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.557+0000 7f51e2ffd640 1 -- 172.21.15.49:0/957966058 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fa1cfc72d0 con 0x55fa1d0d6e60 2024-03-18T01:00:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.557+0000 7f51e2ffd640 10 client.18084 handle_command_reply: tid=0 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.557+0000 7f520c583740 1 client.18084 shutdown 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 -- 172.21.15.49:0/957966058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d055c90 msgr2=0x55fa1d068e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d055c90 0x55fa1d068e70 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f51f80059f0 tx=0x7f51f80217f0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 -- 172.21.15.49:0/957966058 shutdown_connections 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55fa1d0d6e60 0x55fa1d0d9280 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55fa1d05ad30 0x55fa1d0693b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fa1d05fea0 0x55fa1d06a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 --2- 172.21.15.49:0/957966058 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55fa1d055c90 0x55fa1d068e70 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 -- 172.21.15.49:0/957966058 >> 172.21.15.49:0/957966058 conn(0x55fa1d0558b0 msgr2=0x55fa1d064250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f51e2ffd640 0 client.18084 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:00:51.559 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 -- 172.21.15.49:0/957966058 shutdown_connections 2024-03-18T01:00:51.559 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 1 -- 172.21.15.49:0/957966058 wait complete. 2024-03-18T01:00:51.559 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f520c583740 20 client.18084 trim_cache size 0 max 16384 2024-03-18T01:00:51.559 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:{ 2024-03-18T01:00:51.559 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "message": "", 2024-03-18T01:00:51.559 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout: "return_code": 0 2024-03-18T01:00:51.559 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stdout:} 2024-03-18T01:00:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.558+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51ec0a6d30 msgr2=0x7f51ec0a91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51ec0a6d30 0x7f51ec0a91f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f51fc097df0 tx=0x7f51fc06d610 comp rx=0 tx=0).stop 2024-03-18T01:00:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52041075a0 msgr2=0x7f520419f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:00:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52041075a0 0x7f520419f030 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009fd0 tx=0x7f51fc00ef10 comp rx=0 tx=0).stop 2024-03-18T01:00:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 shutdown_connections 2024-03-18T01:00:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f51ec0a6d30 0x7f51ec0a91f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f52041075a0 0x7f520419f030 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f5204104c70 0x7f5204197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.560 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 --2- 172.21.15.49:0/3440596615 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5204102340 0x7f5204197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:00:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 >> 172.21.15.49:0/3440596615 conn(0x7f52040fbc70 msgr2=0x7f5204102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:00:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 shutdown_connections 2024-03-18T01:00:51.561 INFO:teuthology.task.background_exec.ubuntu@smithi049.front.sepia.ceph.com.smithi049.stderr:2024-03-18T01:00:51.559+0000 7f5209997640 1 -- 172.21.15.49:0/3440596615 wait complete. 2024-03-18T01:00:51.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:51 smithi049 ceph-mon[32130]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 353 MiB/s rd, 12 MiB/s wr, 581 op/s 2024-03-18T01:00:51.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:51.624 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:51 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:51.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:51 smithi106 ceph-mon[35899]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 353 MiB/s rd, 12 MiB/s wr, 581 op/s 2024-03-18T01:00:51.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:51.695 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:51 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:52.032 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:51 smithi107 ceph-mon[36270]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 353 MiB/s rd, 12 MiB/s wr, 581 op/s 2024-03-18T01:00:52.033 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:52.033 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:51 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:52.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:52.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:52 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:52.992 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:52.992 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:52 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:53.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:53.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:52 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:53.930 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:53 smithi107 ceph-mon[36270]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 414 MiB/s rd, 14 MiB/s wr, 747 op/s 2024-03-18T01:00:53.945 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:53 smithi106 ceph-mon[35899]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 414 MiB/s rd, 14 MiB/s wr, 747 op/s 2024-03-18T01:00:54.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:53 smithi049 ceph-mon[32130]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 414 MiB/s rd, 14 MiB/s wr, 747 op/s 2024-03-18T01:00:55.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:54 smithi049 ceph-mon[32130]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 424 MiB/s rd, 15 MiB/s wr, 698 op/s 2024-03-18T01:00:55.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:55.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:54 smithi107 ceph-mon[36270]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 424 MiB/s rd, 15 MiB/s wr, 698 op/s 2024-03-18T01:00:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:55.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:55.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:54 smithi106 ceph-mon[35899]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 424 MiB/s rd, 15 MiB/s wr, 698 op/s 2024-03-18T01:00:55.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:55.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:56.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:55 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:00:56.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:55 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:00:56.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:55 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:56.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:55 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:00:56.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:55 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:00:56.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:55 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:56.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:55 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T01:00:56.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:55 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T01:00:56.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:55 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' 2024-03-18T01:00:57.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:56 smithi049 ceph-mon[32130]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 MiB/s rd, 13 MiB/s wr, 644 op/s 2024-03-18T01:00:57.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:56 smithi107 ceph-mon[36270]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 MiB/s rd, 13 MiB/s wr, 644 op/s 2024-03-18T01:00:57.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:56 smithi106 ceph-mon[35899]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 377 MiB/s rd, 13 MiB/s wr, 644 op/s 2024-03-18T01:00:58.843 INFO:tasks.workunit.client.0.smithi049.stdout:stopping iogen 2024-03-18T01:00:58.843 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo 'stopping iogen' 2024-03-18T01:00:58.843 INFO:tasks.workunit.client.0.smithi049.stderr:+ ./iogen -k 2024-03-18T01:00:58.843 INFO:tasks.workunit.client.0.smithi049.stdout:OK 2024-03-18T01:00:58.844 INFO:tasks.workunit.client.0.smithi049.stderr:+ echo OK 2024-03-18T01:00:59.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:00:58 smithi049 ceph-mon[32130]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 448 MiB/s rd, 16 MiB/s wr, 830 op/s 2024-03-18T01:00:59.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:00:58 smithi107 ceph-mon[36270]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 448 MiB/s rd, 16 MiB/s wr, 830 op/s 2024-03-18T01:00:59.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:00:58 smithi106 ceph-mon[35899]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 448 MiB/s rd, 16 MiB/s wr, 830 op/s 2024-03-18T01:01:01.030 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-18T01:01:01.030 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T01:01:01.224 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:00 smithi049 ceph-mon[32130]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 317 MiB/s rd, 12 MiB/s wr, 586 op/s 2024-03-18T01:01:01.238 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-03-18T01:01:01.238 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T01:01:01.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:00 smithi107 ceph-mon[36270]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 317 MiB/s rd, 12 MiB/s wr, 586 op/s 2024-03-18T01:01:01.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:00 smithi106 ceph-mon[35899]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 317 MiB/s rd, 12 MiB/s wr, 586 op/s 2024-03-18T01:01:01.601 DEBUG:teuthology.parallel:result is None 2024-03-18T01:01:01.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T01:01:01.645 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T01:01:01.646 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-18T01:01:01.656 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi049.front.sepia.ceph.com 2024-03-18T01:01:01.664 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-18T01:01:01.684 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-18T01:01:03.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:02 smithi049 ceph-mon[32130]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 364 MiB/s rd, 14 MiB/s wr, 690 op/s 2024-03-18T01:01:03.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:02 smithi049 ceph-mon[32130]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:01:03.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:02 smithi107 ceph-mon[36270]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 364 MiB/s rd, 14 MiB/s wr, 690 op/s 2024-03-18T01:01:03.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:02 smithi107 ceph-mon[36270]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:01:03.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:02 smithi106 ceph-mon[35899]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 364 MiB/s rd, 14 MiB/s wr, 690 op/s 2024-03-18T01:01:03.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:02 smithi106 ceph-mon[35899]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T01:01:05.315 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T01:01:05.315 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T01:01:05.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:04 smithi049 ceph-mon[32130]: pgmap v679: 97 pgs: 97 active+clean; 9.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 354 MiB/s rd, 15 MiB/s wr, 711 op/s 2024-03-18T01:01:05.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.429+0000 7f06837a9640 1 -- 172.21.15.49:0/1977782585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c1022b0 msgr2=0x7f067c108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.429+0000 7f06837a9640 1 --2- 172.21.15.49:0/1977782585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c1022b0 0x7f067c108b40 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f0674009ca0 tx=0x7f067402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.429+0000 7f06837a9640 1 -- 172.21.15.49:0/1977782585 shutdown_connections 2024-03-18T01:01:05.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.429+0000 7f06837a9640 1 --2- 172.21.15.49:0/1977782585 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c1022b0 0x7f067c108b40 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.429+0000 7f06837a9640 1 --2- 172.21.15.49:0/1977782585 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f067c1018f0 0x7f067c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.429+0000 7f06837a9640 1 --2- 172.21.15.49:0/1977782585 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f067c100700 0x7f067c100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.429 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.429+0000 7f06837a9640 1 -- 172.21.15.49:0/1977782585 >> 172.21.15.49:0/1977782585 conn(0x7f067c0fbc90 msgr2=0x7f067c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.429+0000 7f06837a9640 1 -- 172.21.15.49:0/1977782585 shutdown_connections 2024-03-18T01:01:05.430 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f06837a9640 1 -- 172.21.15.49:0/1977782585 wait complete. 2024-03-18T01:01:05.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:04 smithi107 ceph-mon[36270]: pgmap v679: 97 pgs: 97 active+clean; 9.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 354 MiB/s rd, 15 MiB/s wr, 711 op/s 2024-03-18T01:01:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f06837a9640 1 Processor -- start 2024-03-18T01:01:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f06837a9640 1 -- start start 2024-03-18T01:01:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f06837a9640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c100700 0x7f067c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f06837a9640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f067c1018f0 0x7f067c198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f06837a9640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f067c1022b0 0x7f067c19f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f06837a9640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f067c10c380 con 0x7f067c100700 2024-03-18T01:01:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f06837a9640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f067c10c200 con 0x7f067c1018f0 2024-03-18T01:01:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f06837a9640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f067c10c500 con 0x7f067c1022b0 2024-03-18T01:01:05.432 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f0681545640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c100700 0x7f067c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f0681545640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c100700 0x7f067c197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34238/0 (socket says 172.21.15.49:34238) 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f0681545640 1 -- 172.21.15.49:0/4180751827 learned_addr learned my addr 172.21.15.49:0/4180751827 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f0680d44640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f067c1018f0 0x7f067c198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.430+0000 7f0681d46640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f067c1022b0 0x7f067c19f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f0681545640 1 -- 172.21.15.49:0/4180751827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f067c1022b0 msgr2=0x7f067c19f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f0681545640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f067c1022b0 0x7f067c19f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f0681545640 1 -- 172.21.15.49:0/4180751827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f067c1018f0 msgr2=0x7f067c198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f0681545640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f067c1018f0 0x7f067c198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f0681545640 1 -- 172.21.15.49:0/4180751827 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067c19f6e0 con 0x7f067c100700 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f0681545640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c100700 0x7f067c197ac0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f067800e9c0 tx=0x7f067800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f06727fc640 1 -- 172.21.15.49:0/4180751827 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067800cd80 con 0x7f067c100700 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f06727fc640 1 -- 172.21.15.49:0/4180751827 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06780035f0 con 0x7f067c100700 2024-03-18T01:01:05.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f06837a9640 1 -- 172.21.15.49:0/4180751827 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067c19f9d0 con 0x7f067c100700 2024-03-18T01:01:05.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f06727fc640 1 -- 172.21.15.49:0/4180751827 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0678010690 con 0x7f067c100700 2024-03-18T01:01:05.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.431+0000 7f06837a9640 1 -- 172.21.15.49:0/4180751827 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f067c19ff20 con 0x7f067c100700 2024-03-18T01:01:05.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.433+0000 7f06727fc640 1 -- 172.21.15.49:0/4180751827 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f06780037a0 con 0x7f067c100700 2024-03-18T01:01:05.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f068638e740 1 Processor -- start 2024-03-18T01:01:05.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f068638e740 1 -- start start 2024-03-18T01:01:05.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f068638e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 0x55c62a530590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f068638e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 0x55c62a535660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f068638e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a538160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f068638e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c62a495500 con 0x55c62a535cd0 2024-03-18T01:01:05.440 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f068638e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c62a43dbe0 con 0x55c62a530170 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f068638e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c62a4915a0 con 0x55c62a530b60 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f06727fc640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06680a2800 0x7f06680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f0680d44640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a538160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f0681545640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 0x55c62a535660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f0681d46640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 0x55c62a530590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f0680d44640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a538160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34244/0 (socket says 172.21.15.49:34244) 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f0680d44640 1 -- 172.21.15.49:0/2342927076 learned_addr learned my addr 172.21.15.49:0/2342927076 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.434+0000 7f0681d46640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06680a2800 0x7f06680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f0681d46640 1 -- 172.21.15.49:0/2342927076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 msgr2=0x55c62a535660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f0681d46640 1 --2- 172.21.15.49:0/2342927076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 0x55c62a535660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f0681d46640 1 -- 172.21.15.49:0/2342927076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 msgr2=0x55c62a538160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f0681d46640 1 --2- 172.21.15.49:0/2342927076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a538160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f0681d46640 1 -- 172.21.15.49:0/2342927076 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c62a4514b0 con 0x55c62a530170 2024-03-18T01:01:05.441 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f06727fc640 1 -- 172.21.15.49:0/4180751827 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f06780cee30 con 0x7f067c100700 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f0680d44640 1 --2- 172.21.15.49:0/2342927076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a538160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f0681d46640 1 --2- 172.21.15.49:0/2342927076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 0x55c62a530590 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f067402c630 tx=0x7f067402ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500bf158bff5a76f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f0681d46640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06680a2800 0x7f06680a4cc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f067403b000 tx=0x7f0674047000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f06677fe640 1 -- 172.21.15.49:0/2342927076 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674050070 con 0x55c62a530170 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f06677fe640 1 -- 172.21.15.49:0/2342927076 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0674047070 con 0x55c62a530170 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.435+0000 7f06677fe640 1 -- 172.21.15.49:0/2342927076 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674039040 con 0x55c62a530170 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 -- 172.21.15.49:0/2342927076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 msgr2=0x55c62a530590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 --2- 172.21.15.49:0/2342927076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 0x55c62a530590 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f067402c630 tx=0x7f067402ce30 comp rx=0 tx=0).stop 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 -- 172.21.15.49:0/2342927076 shutdown_connections 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 --2- 172.21.15.49:0/2342927076 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a538160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 --2- 172.21.15.49:0/2342927076 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 0x55c62a535660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 --2- 172.21.15.49:0/2342927076 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 0x55c62a530590 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 -- 172.21.15.49:0/2342927076 >> 172.21.15.49:0/2342927076 conn(0x55c62a52b6e0 msgr2=0x55c62a52bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 -- 172.21.15.49:0/2342927076 shutdown_connections 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 -- 172.21.15.49:0/2342927076 wait complete. 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 -- start start 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.436+0000 7f068638e740 1 Processor -- start 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f068638e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530170 0x55c62a53a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f068638e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530b60 0x55c62a53abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f068638e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a535cd0 0x55c62a541c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f068638e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c62a538c20 con 0x55c62a530b60 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f068638e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c62a451160 con 0x55c62a535cd0 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f068638e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c62a538da0 con 0x55c62a530170 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f068638e740 10 client.0 _ll_register_callbacks cb 0x55c62a435150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0681545640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530170 0x55c62a53a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0681d46640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a535cd0 0x55c62a541c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0680d44640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530b60 0x55c62a53abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0681545640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530170 0x55c62a53a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58384/0 (socket says 172.21.15.49:58384) 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0681545640 1 -- 172.21.15.49:0/1303125654 learned_addr learned my addr 172.21.15.49:0/1303125654 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0681d46640 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a535cd0 0x55c62a541c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:35172/0 (socket says 172.21.15.49:35172) 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0680d44640 1 -- 172.21.15.49:0/1303125654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530170 msgr2=0x55c62a53a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0680d44640 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530170 0x55c62a53a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0680d44640 1 -- 172.21.15.49:0/1303125654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a535cd0 msgr2=0x55c62a541c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0680d44640 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a535cd0 0x55c62a541c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0680d44640 1 -- 172.21.15.49:0/1303125654 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c62a2737d0 con 0x55c62a530b60 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0681d46640 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a535cd0 0x55c62a541c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0681545640 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530170 0x55c62a53a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0680d44640 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530b60 0x55c62a53abc0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f066c0126b0 tx=0x7f066c012b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c00e696ee423a9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0664ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:01:05.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0664ff9640 1 -- 172.21.15.49:0/1303125654 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c004050 con 0x55c62a530b60 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f0664ff9640 1 -- 172.21.15.49:0/1303125654 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f066c0041f0 con 0x55c62a530b60 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.437+0000 7f068638e740 1 -- 172.21.15.49:0/1303125654 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0674047070 con 0x55c62a530b60 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f0664ff9640 1 -- 172.21.15.49:0/1303125654 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f066c0043f0 con 0x55c62a530b60 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f068638e740 10 client.18096 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f068638e740 1 -- 172.21.15.49:0/1303125654 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c62a1a93f0 con 0x55c62a530b60 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f0664ff9640 1 -- 172.21.15.49:0/1303125654 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0674047070 con 0x55c62a530b60 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f0664ff9640 1 -- 172.21.15.49:0/1303125654 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f066c003c30 con 0x55c62a530b60 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f068638e740 10 client.18096 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f068638e740 10 client.18096 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f068638e740 20 client.18096 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f068638e740 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c62a546210 0x55c62a548690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f068638e740 4 client.18096 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f068638e740 1 -- 172.21.15.49:0/1303125654 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c62a4a2a40 con 0x55c62a546210 2024-03-18T01:01:05.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.438+0000 7f0681545640 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c62a546210 0x55c62a548690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.439+0000 7f0681545640 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c62a546210 0x55c62a548690 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc96ad69de4ed166 server_cookie=29c00941d63fdf8e in_seq=0 out_seq=0 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.439+0000 7f0664ff9640 10 client.18096 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f0664ff9640 1 -- 172.21.15.49:0/1303125654 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c62a4a2a40 con 0x55c62a546210 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f0664ff9640 10 client.18096 handle_command_reply: tid=0 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 client.18096 shutdown 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 -- 172.21.15.49:0/1303125654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530b60 msgr2=0x55c62a53abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530b60 0x55c62a53abc0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f066c0126b0 tx=0x7f066c012b70 comp rx=0 tx=0).stop 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 -- 172.21.15.49:0/1303125654 shutdown_connections 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c62a546210 0x55c62a548690 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a535cd0 0x55c62a541c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530b60 0x55c62a53abc0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f0664ff9640 0 client.18096 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:01:05.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 --2- 172.21.15.49:0/1303125654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530170 0x55c62a53a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 -- 172.21.15.49:0/1303125654 >> 172.21.15.49:0/1303125654 conn(0x55c62a52b6e0 msgr2=0x55c62a52a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 -- 172.21.15.49:0/1303125654 shutdown_connections 2024-03-18T01:01:05.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 1 -- 172.21.15.49:0/1303125654 wait complete. 2024-03-18T01:01:05.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.440+0000 7f068638e740 20 client.18096 trim_cache size 0 max 16384 2024-03-18T01:01:05.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:04 smithi106 ceph-mon[35899]: pgmap v679: 97 pgs: 97 active+clean; 9.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 354 MiB/s rd, 15 MiB/s wr, 711 op/s 2024-03-18T01:01:05.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.457+0000 7f068638e740 1 Processor -- start 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.457+0000 7f068638e740 1 -- start start 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f068638e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 0x55c62a531580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f068638e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 0x55c62a541bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f068638e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a52bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f068638e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f067c1a8d70 con 0x55c62a535cd0 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f068638e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f067c1aa910 con 0x55c62a530170 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f068638e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f067c1a9440 con 0x55c62a530b60 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681d46640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a52bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681d46640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a52bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34260/0 (socket says 172.21.15.49:34260) 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681d46640 1 -- 172.21.15.49:0/4022788303 learned_addr learned my addr 172.21.15.49:0/4022788303 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0680d44640 1 --2- 172.21.15.49:0/4022788303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 0x55c62a541bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681545640 1 --2- 172.21.15.49:0/4022788303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 0x55c62a531580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681d46640 1 -- 172.21.15.49:0/4022788303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 msgr2=0x55c62a541bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681d46640 1 --2- 172.21.15.49:0/4022788303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 0x55c62a541bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681d46640 1 -- 172.21.15.49:0/4022788303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 msgr2=0x55c62a531580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681d46640 1 --2- 172.21.15.49:0/4022788303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 0x55c62a531580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681d46640 1 -- 172.21.15.49:0/4022788303 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c62a1690f0 con 0x55c62a535cd0 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0681d46640 1 --2- 172.21.15.49:0/4022788303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a52bac0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0674032420 tx=0x7f0674032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc137be2d60acd52 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0666ffd640 1 -- 172.21.15.49:0/4022788303 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674050030 con 0x55c62a535cd0 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0666ffd640 1 -- 172.21.15.49:0/4022788303 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0674059070 con 0x55c62a535cd0 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.458+0000 7f0666ffd640 1 -- 172.21.15.49:0/4022788303 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0674039040 con 0x55c62a535cd0 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 -- 172.21.15.49:0/4022788303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 msgr2=0x55c62a52bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 --2- 172.21.15.49:0/4022788303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a52bac0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f0674032420 tx=0x7f0674032c90 comp rx=0 tx=0).stop 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 -- 172.21.15.49:0/4022788303 shutdown_connections 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 --2- 172.21.15.49:0/4022788303 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a535cd0 0x55c62a52bac0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 --2- 172.21.15.49:0/4022788303 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a530b60 0x55c62a541bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 --2- 172.21.15.49:0/4022788303 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530170 0x55c62a531580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 -- 172.21.15.49:0/4022788303 >> 172.21.15.49:0/4022788303 conn(0x55c62a52b6e0 msgr2=0x55c62a539140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 -- 172.21.15.49:0/4022788303 shutdown_connections 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 -- 172.21.15.49:0/4022788303 wait complete. 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 -- start start 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.459+0000 7f068638e740 1 Processor -- start 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f068638e740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530170 0x55c62a53ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f068638e740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530b60 0x55c62a53efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f068638e740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a535cd0 0x55c62a5aa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f068638e740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f067c1aa210 con 0x55c62a530170 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f068638e740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f067c1a9b10 con 0x55c62a530b60 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f068638e740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f067c1a6d50 con 0x55c62a535cd0 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f068638e740 10 client.0 _ll_register_callbacks cb 0x7f066c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681545640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530170 0x55c62a53ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681545640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530170 0x55c62a53ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34276/0 (socket says 172.21.15.49:34276) 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681545640 1 -- 172.21.15.49:0/1615400300 learned_addr learned my addr 172.21.15.49:0/1615400300 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:05.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0680d44640 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530b60 0x55c62a53efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681d46640 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a535cd0 0x55c62a5aa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681545640 1 -- 172.21.15.49:0/1615400300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a535cd0 msgr2=0x55c62a5aa5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681545640 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a535cd0 0x55c62a5aa5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681545640 1 -- 172.21.15.49:0/1615400300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530b60 msgr2=0x55c62a53efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681545640 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530b60 0x55c62a53efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681545640 1 -- 172.21.15.49:0/1615400300 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c62a45c9d0 con 0x55c62a530170 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0680d44640 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530b60 0x55c62a53efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0681545640 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530170 0x55c62a53ea80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0678098180 tx=0x7f0678000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e57c190dd6b4e3ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0664ff9640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0664ff9640 1 -- 172.21.15.49:0/1615400300 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0678003b30 con 0x55c62a530170 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.460+0000 7f0664ff9640 1 -- 172.21.15.49:0/1615400300 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f06780860b0 con 0x55c62a530170 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f068638e740 1 -- 172.21.15.49:0/1615400300 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0674059070 con 0x55c62a530170 2024-03-18T01:01:05.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f0664ff9640 1 -- 172.21.15.49:0/1615400300 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f06780a3ac0 con 0x55c62a530170 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f068638e740 10 client.18102 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f068638e740 1 -- 172.21.15.49:0/1615400300 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c62a5abc20 con 0x55c62a530170 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f0664ff9640 1 -- 172.21.15.49:0/1615400300 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0674059070 con 0x55c62a530170 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f0664ff9640 1 -- 172.21.15.49:0/1615400300 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f067809ac00 con 0x55c62a530170 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f068638e740 10 client.18102 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f068638e740 10 client.18102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f068638e740 20 client.18102 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f068638e740 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c62a5ac570 0x55c62a5ae960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f068638e740 4 client.18102 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f068638e740 1 -- 172.21.15.49:0/1615400300 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c62a49cd90 con 0x55c62a5ac570 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.461+0000 7f0680d44640 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c62a5ac570 0x55c62a5ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f0680d44640 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c62a5ac570 0x55c62a5ae960 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e9254ad77a30cba server_cookie=1bf98ed8ca5074 in_seq=0 out_seq=0 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f0664ff9640 10 client.18102 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f0664ff9640 1 -- 172.21.15.49:0/1615400300 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55c62a49cd90 con 0x55c62a5ac570 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f0664ff9640 10 client.18102 handle_command_reply: tid=0 2024-03-18T01:01:05.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 client.18102 shutdown 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 -- 172.21.15.49:0/1615400300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530170 msgr2=0x55c62a53ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530170 0x55c62a53ea80 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f0678098180 tx=0x7f0678000bc0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 -- 172.21.15.49:0/1615400300 shutdown_connections 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x55c62a5ac570 0x55c62a5ae960 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c62a530b60 0x55c62a53efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c62a535cd0 0x55c62a5aa5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f0664ff9640 0 client.18102 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 --2- 172.21.15.49:0/1615400300 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c62a530170 0x55c62a53ea80 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 -- 172.21.15.49:0/1615400300 >> 172.21.15.49:0/1615400300 conn(0x55c62a52b6e0 msgr2=0x55c62a52bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 -- 172.21.15.49:0/1615400300 shutdown_connections 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 1 -- 172.21.15.49:0/1615400300 wait complete. 2024-03-18T01:01:05.468 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.462+0000 7f068638e740 20 client.18102 trim_cache size 0 max 16384 2024-03-18T01:01:05.469 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:01:05.469 INFO:teuthology.orchestra.run.smithi049.stdout: "status": "no active scrubs running", 2024-03-18T01:01:05.469 INFO:teuthology.orchestra.run.smithi049.stdout: "scrubs": {} 2024-03-18T01:01:05.469 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:01:05.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 -- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06680a2800 msgr2=0x7f06680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06680a2800 0x7f06680a4cc0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f067403b000 tx=0x7f0674047000 comp rx=0 tx=0).stop 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 -- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c100700 msgr2=0x7f067c197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c100700 0x7f067c197ac0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f067800e9c0 tx=0x7f067800ee80 comp rx=0 tx=0).stop 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 -- 172.21.15.49:0/4180751827 shutdown_connections 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f06680a2800 0x7f06680a4cc0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f067c1018f0 0x7f067c198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f067c1022b0 0x7f067c19f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 --2- 172.21.15.49:0/4180751827 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f067c100700 0x7f067c197ac0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 -- 172.21.15.49:0/4180751827 >> 172.21.15.49:0/4180751827 conn(0x7f067c0fbc90 msgr2=0x7f067c0fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 -- 172.21.15.49:0/4180751827 shutdown_connections 2024-03-18T01:01:05.470 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.463+0000 7f06837a9640 1 -- 172.21.15.49:0/4180751827 wait complete. 2024-03-18T01:01:05.473 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T01:01:05.473 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:01:05.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.623+0000 7f2a5b47e640 1 -- 172.21.15.49:0/3165807548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 msgr2=0x7f2a54101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.623+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/3165807548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 0x7f2a54101d70 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009900 tx=0x7f2a4802bc30 comp rx=0 tx=0).stop 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.623+0000 7f2a5b47e640 1 -- 172.21.15.49:0/3165807548 shutdown_connections 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.623+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/3165807548 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a541022b0 0x7f2a54108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.623+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/3165807548 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 0x7f2a54101d70 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.623+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/3165807548 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a54100700 0x7f2a54100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.623+0000 7f2a5b47e640 1 -- 172.21.15.49:0/3165807548 >> 172.21.15.49:0/3165807548 conn(0x7f2a540fbc90 msgr2=0x7f2a540fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 -- 172.21.15.49:0/3165807548 shutdown_connections 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 -- 172.21.15.49:0/3165807548 wait complete. 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 Processor -- start 2024-03-18T01:01:05.626 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 -- start start 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a54100700 0x7f2a54196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 0x7f2a54196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a541022b0 0x7f2a5419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5410c360 con 0x7f2a541018f0 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5410c1e0 con 0x7f2a541022b0 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5b47e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5410c4e0 con 0x7f2a54100700 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a58a19640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 0x7f2a54196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a58a19640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 0x7f2a54196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34300/0 (socket says 172.21.15.49:34300) 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a5921a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a54100700 0x7f2a54196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a58a19640 1 -- 172.21.15.49:0/529879248 learned_addr learned my addr 172.21.15.49:0/529879248 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.624+0000 7f2a59a1b640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a541022b0 0x7f2a5419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a58a19640 1 -- 172.21.15.49:0/529879248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a54100700 msgr2=0x7f2a54196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a58a19640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a54100700 0x7f2a54196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a58a19640 1 -- 172.21.15.49:0/529879248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a541022b0 msgr2=0x7f2a5419dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.627 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a58a19640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a541022b0 0x7f2a5419dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a58a19640 1 -- 172.21.15.49:0/529879248 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5419e620 con 0x7f2a541018f0 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a5921a640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a54100700 0x7f2a54196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a59a1b640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a541022b0 0x7f2a5419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a58a19640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 0x7f2a54196f40 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009230 tx=0x7f2a4802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a527fc640 1 -- 172.21.15.49:0/529879248 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a48041070 con 0x7f2a541018f0 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a527fc640 1 -- 172.21.15.49:0/529879248 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a48004450 con 0x7f2a541018f0 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5419e8b0 con 0x7f2a541018f0 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a527fc640 1 -- 172.21.15.49:0/529879248 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a48003e20 con 0x7f2a541018f0 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.625+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5419edf0 con 0x7f2a541018f0 2024-03-18T01:01:05.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.626+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a54107a20 con 0x7f2a541018f0 2024-03-18T01:01:05.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.627+0000 7f2a527fc640 1 -- 172.21.15.49:0/529879248 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2a48044070 con 0x7f2a541018f0 2024-03-18T01:01:05.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.628+0000 7f2a527fc640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2a3c0a2880 0x7f2a3c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.628+0000 7f2a527fc640 1 -- 172.21.15.49:0/529879248 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2a480f0590 con 0x7f2a541018f0 2024-03-18T01:01:05.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.628+0000 7f2a5921a640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2a3c0a2880 0x7f2a3c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.629 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.628+0000 7f2a5921a640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2a3c0a2880 0x7f2a3c0a4d40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2a54101750 tx=0x7f2a4c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.632 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.632+0000 7f2a527fc640 1 -- 172.21.15.49:0/529879248 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a480bcfc0 con 0x7f2a541018f0 2024-03-18T01:01:05.855 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.856+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a5402d1b0 con 0x7f2a541018f0 2024-03-18T01:01:05.856 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.856+0000 7f2a527fc640 1 -- 172.21.15.49:0/529879248 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11936 (secure 0 0 0) 0x7f2a480c1060 con 0x7f2a541018f0 2024-03-18T01:01:05.856 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T01:01:05.856 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T01:01:01.508865+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":178,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24470,24463,24481,24506,24499]},"id":1}]} 2024-03-18T01:01:05.857 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 31 2024-03-18T01:01:05.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2a3c0a2880 msgr2=0x7f2a3c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2a3c0a2880 0x7f2a3c0a4d40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f2a54101750 tx=0x7f2a4c005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:05.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 msgr2=0x7f2a54196f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 0x7f2a54196f40 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009230 tx=0x7f2a4802c650 comp rx=0 tx=0).stop 2024-03-18T01:01:05.858 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 shutdown_connections 2024-03-18T01:01:05.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f2a3c0a2880 0x7f2a3c0a4d40 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2a541022b0 0x7f2a5419dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f2a541018f0 0x7f2a54196f40 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 --2- 172.21.15.49:0/529879248 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2a54100700 0x7f2a54196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 >> 172.21.15.49:0/529879248 conn(0x7f2a540fbc90 msgr2=0x7f2a54104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 shutdown_connections 2024-03-18T01:01:05.859 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.858+0000 7f2a5b47e640 1 -- 172.21.15.49:0/529879248 wait complete. 2024-03-18T01:01:05.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T01:01:05.981 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.980+0000 7ff12ef37640 1 -- 172.21.15.49:0/2760150863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 msgr2=0x7ff128101d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.980+0000 7ff12ef37640 1 --2- 172.21.15.49:0/2760150863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 0x7ff128101d70 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7ff114009900 tx=0x7ff11402bc30 comp rx=0 tx=0).stop 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.980+0000 7ff12ef37640 1 -- 172.21.15.49:0/2760150863 shutdown_connections 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.980+0000 7ff12ef37640 1 --2- 172.21.15.49:0/2760150863 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff1281022b0 0x7ff128108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.980+0000 7ff12ef37640 1 --2- 172.21.15.49:0/2760150863 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 0x7ff128101d70 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.980+0000 7ff12ef37640 1 --2- 172.21.15.49:0/2760150863 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff128100700 0x7ff128100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.980+0000 7ff12ef37640 1 -- 172.21.15.49:0/2760150863 >> 172.21.15.49:0/2760150863 conn(0x7ff1280fbc90 msgr2=0x7ff1280fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.980+0000 7ff12ef37640 1 -- 172.21.15.49:0/2760150863 shutdown_connections 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.980+0000 7ff12ef37640 1 -- 172.21.15.49:0/2760150863 wait complete. 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ef37640 1 Processor -- start 2024-03-18T01:01:05.982 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ef37640 1 -- start start 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ef37640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff128100700 0x7ff128104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ef37640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 0x7ff128102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ef37640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1281022b0 0x7ff128103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ef37640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff12810c360 con 0x7ff1281018f0 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ef37640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff12810c1e0 con 0x7ff128100700 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ef37640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff12810c4e0 con 0x7ff1281022b0 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff127fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 0x7ff128102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff127fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 0x7ff128102cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34326/0 (socket says 172.21.15.49:34326) 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff127fff640 1 -- 172.21.15.49:0/1358934908 learned_addr learned my addr 172.21.15.49:0/1358934908 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff128100700 0x7ff128104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12d4d4640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1281022b0 0x7ff128103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff127fff640 1 -- 172.21.15.49:0/1358934908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1281022b0 msgr2=0x7ff128103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff127fff640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1281022b0 0x7ff128103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.983 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff127fff640 1 -- 172.21.15.49:0/1358934908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff128100700 msgr2=0x7ff128104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff127fff640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff128100700 0x7ff128104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff127fff640 1 -- 172.21.15.49:0/1358934908 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff128103aa0 con 0x7ff1281018f0 2024-03-18T01:01:05.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.981+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff128100700 0x7ff128104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:05.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.982+0000 7ff127fff640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 0x7ff128102cd0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff114009230 tx=0x7ff11402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.982+0000 7ff125ffb640 1 -- 172.21.15.49:0/1358934908 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114041070 con 0x7ff1281018f0 2024-03-18T01:01:05.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.982+0000 7ff125ffb640 1 -- 172.21.15.49:0/1358934908 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff114004450 con 0x7ff1281018f0 2024-03-18T01:01:05.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.982+0000 7ff12ef37640 1 -- 172.21.15.49:0/1358934908 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1281a1bf0 con 0x7ff1281018f0 2024-03-18T01:01:05.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.982+0000 7ff125ffb640 1 -- 172.21.15.49:0/1358934908 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114003db0 con 0x7ff1281018f0 2024-03-18T01:01:05.984 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.982+0000 7ff12ef37640 1 -- 172.21.15.49:0/1358934908 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1281a2140 con 0x7ff1281018f0 2024-03-18T01:01:05.990 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.984+0000 7ff125ffb640 1 -- 172.21.15.49:0/1358934908 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff114044070 con 0x7ff1281018f0 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.984+0000 7ff131bf8740 1 Processor -- start 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.984+0000 7ff131bf8740 1 -- start start 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff125ffb640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1180a2800 0x7ff1180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563caceedc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 0x563cacef2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 0x563cacef5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff131bf8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563cace52490 con 0x563caceed850 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff131bf8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563cacdfac90 con 0x563caceee240 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff131bf8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563cace4e530 con 0x563cacef33b0 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12d4d4640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1180a2800 0x7ff1180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12ccd3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563caceedc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff125ffb640 1 -- 172.21.15.49:0/1358934908 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff1140f0620 con 0x7ff1281018f0 2024-03-18T01:01:05.991 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12ccd3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563caceedc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34328/0 (socket says 172.21.15.49:34328) 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12ccd3640 1 -- 172.21.15.49:0/3335430541 learned_addr learned my addr 172.21.15.49:0/3335430541 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff127fff640 1 --2- 172.21.15.49:0/3335430541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 0x563cacef2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12d4d4640 1 --2- 172.21.15.49:0/3335430541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 0x563cacef5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12ccd3640 1 -- 172.21.15.49:0/3335430541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 msgr2=0x563cacef5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/3335430541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 0x563cacef5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12ccd3640 1 -- 172.21.15.49:0/3335430541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 msgr2=0x563cacef2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/3335430541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 0x563cacef2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12ccd3640 1 -- 172.21.15.49:0/3335430541 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563caceb5500 con 0x563caceed850 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12d4d4640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1180a2800 0x7ff1180a4cc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff120005fd0 tx=0x7ff120009e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.985+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/3335430541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563caceedc70 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0114c0 tx=0x7ff11c0118c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef63f3b958b73c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff112ffd640 1 -- 172.21.15.49:0/3335430541 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c004020 con 0x563caceed850 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff112ffd640 1 -- 172.21.15.49:0/3335430541 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff11c005500 con 0x563caceed850 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff112ffd640 1 -- 172.21.15.49:0/3335430541 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c004cb0 con 0x563caceed850 2024-03-18T01:01:05.992 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff131bf8740 1 -- 172.21.15.49:0/3335430541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 msgr2=0x563caceedc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff131bf8740 1 --2- 172.21.15.49:0/3335430541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563caceedc70 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0114c0 tx=0x7ff11c0118c0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff131bf8740 1 -- 172.21.15.49:0/3335430541 shutdown_connections 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff131bf8740 1 --2- 172.21.15.49:0/3335430541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 0x563cacef2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff131bf8740 1 --2- 172.21.15.49:0/3335430541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 0x563cacef5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff131bf8740 1 --2- 172.21.15.49:0/3335430541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563caceedc70 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff131bf8740 1 -- 172.21.15.49:0/3335430541 >> 172.21.15.49:0/3335430541 conn(0x563cacee8dc0 msgr2=0x563cacee91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff131bf8740 1 -- 172.21.15.49:0/3335430541 shutdown_connections 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.986+0000 7ff131bf8740 1 -- 172.21.15.49:0/3335430541 wait complete. 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff131bf8740 1 -- start start 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff131bf8740 1 Processor -- start 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563cacef7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceee240 0x563cacef8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cacef33b0 0x563caceff240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.993 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff131bf8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x563cacef6360 con 0x563caceed850 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff131bf8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x563cace0e230 con 0x563cacef33b0 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff131bf8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x563cacef64e0 con 0x563caceee240 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff131bf8740 10 client.0 _ll_register_callbacks cb 0x563cace4a010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff12ccd3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563cacef7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff12ccd3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563cacef7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34344/0 (socket says 172.21.15.49:34344) 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff12ccd3640 1 -- 172.21.15.49:0/27883946 learned_addr learned my addr 172.21.15.49:0/27883946 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff127fff640 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceee240 0x563cacef8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.987+0000 7ff12d4d4640 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cacef33b0 0x563caceff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff12ccd3640 1 -- 172.21.15.49:0/27883946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceee240 msgr2=0x563cacef8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceee240 0x563cacef8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff12ccd3640 1 -- 172.21.15.49:0/27883946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cacef33b0 msgr2=0x563caceff240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cacef33b0 0x563caceff240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.994 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff12ccd3640 1 -- 172.21.15.49:0/27883946 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cace1d100 con 0x563caceed850 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff127fff640 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceee240 0x563cacef8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563cacef7cd0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009730 tx=0x7ff11c005120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdfc0371e0a02b9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff107fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff107fff640 1 -- 172.21.15.49:0/27883946 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c028020 con 0x563caceed850 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff107fff640 1 -- 172.21.15.49:0/27883946 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff11c00f4e0 con 0x563caceed850 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff131bf8740 1 -- 172.21.15.49:0/27883946 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff11c005500 con 0x563caceed850 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff107fff640 1 -- 172.21.15.49:0/27883946 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff11c033050 con 0x563caceed850 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff131bf8740 10 client.18120 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff131bf8740 1 -- 172.21.15.49:0/27883946 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563cacb663f0 con 0x563caceed850 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff107fff640 1 -- 172.21.15.49:0/27883946 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff11c005500 con 0x563caceed850 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff107fff640 1 -- 172.21.15.49:0/27883946 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff11c023470 con 0x563caceed850 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff131bf8740 10 client.18120 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.988+0000 7ff131bf8740 10 client.18120 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.989+0000 7ff131bf8740 20 client.18120 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.989+0000 7ff131bf8740 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x563cacf039b0 0x563cacf05e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:05.995 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.989+0000 7ff131bf8740 4 client.18120 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.989+0000 7ff131bf8740 1 -- 172.21.15.49:0/27883946 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cace5f9d0 con 0x563cacf039b0 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.989+0000 7ff127fff640 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x563cacf039b0 0x563cacf05e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.989+0000 7ff127fff640 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x563cacf039b0 0x563cacf05e30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=84b9ad55e1612ee6 server_cookie=cae087308a18461c in_seq=0 out_seq=0 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.989+0000 7ff107fff640 10 client.18120 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff107fff640 1 -- 172.21.15.49:0/27883946 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563cace5f9d0 con 0x563cacf039b0 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff107fff640 10 client.18120 handle_command_reply: tid=0 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 client.18120 shutdown 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 -- 172.21.15.49:0/27883946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 msgr2=0x563cacef7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563cacef7cd0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009730 tx=0x7ff11c005120 comp rx=0 tx=0).stop 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 -- 172.21.15.49:0/27883946 shutdown_connections 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x563cacf039b0 0x563cacf05e30 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563cacef33b0 0x563caceff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.996 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceee240 0x563cacef8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff107fff640 0 client.18120 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T01:01:05.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 --2- 172.21.15.49:0/27883946 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceed850 0x563cacef7cd0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:05.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 -- 172.21.15.49:0/27883946 >> 172.21.15.49:0/27883946 conn(0x563cacee8dc0 msgr2=0x563cacee7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:05.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 -- 172.21.15.49:0/27883946 shutdown_connections 2024-03-18T01:01:05.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 1 -- 172.21.15.49:0/27883946 wait complete. 2024-03-18T01:01:05.997 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:05.990+0000 7ff131bf8740 20 client.18120 trim_cache size 0 max 16384 2024-03-18T01:01:06.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.003+0000 7ff131bf8740 1 Processor -- start 2024-03-18T01:01:06.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.003+0000 7ff131bf8740 1 -- start start 2024-03-18T01:01:06.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceed850 0x563cacee7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.008 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceee240 0x563caceff200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 0x563cacef72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff131bf8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281aafb0 con 0x563caceee240 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff131bf8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281acb50 con 0x563caceed850 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff131bf8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281ab680 con 0x563cacef33b0 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff127fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceee240 0x563caceff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff127fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceee240 0x563caceff200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34350/0 (socket says 172.21.15.49:34350) 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff127fff640 1 -- 172.21.15.49:0/2675321363 learned_addr learned my addr 172.21.15.49:0/2675321363 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceed850 0x563cacee7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff12d4d4640 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 0x563cacef72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff12ccd3640 1 -- 172.21.15.49:0/2675321363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 msgr2=0x563cacef72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 0x563cacef72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.009 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff12ccd3640 1 -- 172.21.15.49:0/2675321363 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceee240 msgr2=0x563caceff200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceee240 0x563caceff200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff12ccd3640 1 -- 172.21.15.49:0/2675321363 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cacc307d0 con 0x563caceed850 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff127fff640 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceee240 0x563caceff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceed850 0x563cacee7e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff11c000c00 tx=0x7ff11c011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6640330968ae4e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff1127fc640 1 -- 172.21.15.49:0/2675321363 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c03c070 con 0x563caceed850 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff1127fc640 1 -- 172.21.15.49:0/2675321363 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff11c009b50 con 0x563caceed850 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.004+0000 7ff1127fc640 1 -- 172.21.15.49:0/2675321363 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c028040 con 0x563caceed850 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.005+0000 7ff131bf8740 1 -- 172.21.15.49:0/2675321363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceed850 msgr2=0x563cacee7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.005+0000 7ff131bf8740 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceed850 0x563cacee7e60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff11c000c00 tx=0x7ff11c011dc0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.005+0000 7ff131bf8740 1 -- 172.21.15.49:0/2675321363 shutdown_connections 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.005+0000 7ff131bf8740 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563cacef33b0 0x563cacef72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.010 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.005+0000 7ff131bf8740 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563caceee240 0x563caceff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.005+0000 7ff131bf8740 1 --2- 172.21.15.49:0/2675321363 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceed850 0x563cacee7e60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.005+0000 7ff131bf8740 1 -- 172.21.15.49:0/2675321363 >> 172.21.15.49:0/2675321363 conn(0x563cacee8dc0 msgr2=0x563cacee7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.005+0000 7ff131bf8740 1 -- 172.21.15.49:0/2675321363 shutdown_connections 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.005+0000 7ff131bf8740 1 -- 172.21.15.49:0/2675321363 wait complete. 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff131bf8740 1 -- start start 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff131bf8740 1 Processor -- start 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceed850 0x563cacefc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 0x563cacefc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff131bf8740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563cacef33b0 0x563cacf67d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff131bf8740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281ac450 con 0x563cacef33b0 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff131bf8740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281abd50 con 0x563caceee240 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff131bf8740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281a8f90 con 0x563caceed850 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff131bf8740 10 client.0 _ll_register_callbacks cb 0x7ff1200137c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff12ccd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceed850 0x563cacefc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.011 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff12d4d4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563cacef33b0 0x563cacf67d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff127fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 0x563cacefc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff12ccd3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceed850 0x563cacefc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58452/0 (socket says 172.21.15.49:58452) 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff12ccd3640 1 -- 172.21.15.49:0/1951670140 learned_addr learned my addr 172.21.15.49:0/1951670140 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff12ccd3640 1 -- 172.21.15.49:0/1951670140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 msgr2=0x563cacefc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 0x563cacefc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff12ccd3640 1 -- 172.21.15.49:0/1951670140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563cacef33b0 msgr2=0x563cacf67d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563cacef33b0 0x563cacf67d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.006+0000 7ff12ccd3640 1 -- 172.21.15.49:0/1951670140 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cacf68290 con 0x563caceed850 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff12ccd3640 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceed850 0x563cacefc1a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0239d0 tx=0x7ff11c009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eca42b03e97b4007 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff107fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff107fff640 1 -- 172.21.15.49:0/1951670140 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c03c070 con 0x563caceed850 2024-03-18T01:01:06.012 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff131bf8740 1 -- 172.21.15.49:0/1951670140 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff11c009b50 con 0x563caceed850 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff107fff640 1 -- 172.21.15.49:0/1951670140 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff11c005120 con 0x563caceed850 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff107fff640 1 -- 172.21.15.49:0/1951670140 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff11c028070 con 0x563caceed850 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff131bf8740 10 client.27310 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff131bf8740 1 -- 172.21.15.49:0/1951670140 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563cacf69570 con 0x563caceed850 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff107fff640 1 -- 172.21.15.49:0/1951670140 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff11c009b50 con 0x563caceed850 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff107fff640 1 -- 172.21.15.49:0/1951670140 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7ff11c00f9c0 con 0x563caceed850 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff131bf8740 10 client.27310 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff131bf8740 10 client.27310 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff131bf8740 20 client.27310 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff131bf8740 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x563cacf69e00 0x563cacf6c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff131bf8740 4 client.27310 mds_command: new command op to 24463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.007+0000 7ff131bf8740 1 -- 172.21.15.49:0/1951670140 --> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563cace59d20 con 0x563cacf69e00 2024-03-18T01:01:06.013 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff127fff640 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x563cacf69e00 0x563cacf6c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff127fff640 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x563cacf69e00 0x563cacf6c1f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=255a545c854861da server_cookie=63b492b6b9477ac5 in_seq=0 out_seq=0 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff107fff640 10 client.27310 ms_handle_connect on v2:172.21.15.107:6834/863606455 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff107fff640 1 -- 172.21.15.49:0/1951670140 <== mds.4 v2:172.21.15.107:6834/863606455 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563cace59d20 con 0x563cacf69e00 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff107fff640 10 client.27310 handle_command_reply: tid=0 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 client.27310 shutdown 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 -- 172.21.15.49:0/1951670140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceed850 msgr2=0x563cacefc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceed850 0x563cacefc1a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0239d0 tx=0x7ff11c009f50 comp rx=0 tx=0).stop 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 -- 172.21.15.49:0/1951670140 shutdown_connections 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.107:6834/863606455,v1:172.21.15.107:6835/863606455] conn(0x563cacf69e00 0x563cacf6c1f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x563caceee240 0x563cacefc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x563cacef33b0 0x563cacf67d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 --2- 172.21.15.49:0/1951670140 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x563caceed850 0x563cacefc1a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff107fff640 0 client.27310 ms_handle_reset on v2:172.21.15.107:6834/863606455 2024-03-18T01:01:06.014 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 -- 172.21.15.49:0/1951670140 >> 172.21.15.49:0/1951670140 conn(0x563cacee8dc0 msgr2=0x563caceec8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 -- 172.21.15.49:0/1951670140 shutdown_connections 2024-03-18T01:01:06.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 1 -- 172.21.15.49:0/1951670140 wait complete. 2024-03-18T01:01:06.015 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.008+0000 7ff131bf8740 20 client.27310 trim_cache size 0 max 16384 2024-03-18T01:01:06.015 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 -- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1180a2800 msgr2=0x7ff1180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1180a2800 0x7ff1180a4cc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff120005fd0 tx=0x7ff120009e80 comp rx=0 tx=0).stop 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 -- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 msgr2=0x7ff128102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 0x7ff128102cd0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff114009230 tx=0x7ff11402c650 comp rx=0 tx=0).stop 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 -- 172.21.15.49:0/1358934908 shutdown_connections 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7ff1180a2800 0x7ff1180a4cc0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1281022b0 0x7ff128103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7ff1281018f0 0x7ff128102cd0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 --2- 172.21.15.49:0/1358934908 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7ff128100700 0x7ff128104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 -- 172.21.15.49:0/1358934908 >> 172.21.15.49:0/1358934908 conn(0x7ff1280fbc90 msgr2=0x7ff1280fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.016 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 -- 172.21.15.49:0/1358934908 shutdown_connections 2024-03-18T01:01:06.017 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.009+0000 7ff12ef37640 1 -- 172.21.15.49:0/1358934908 wait complete. 2024-03-18T01:01:06.020 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T01:01:06.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:06 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/529879248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:01:06.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.130+0000 7fea64a9a640 1 -- 172.21.15.49:0/81168488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea601022b0 msgr2=0x7fea60108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.130+0000 7fea64a9a640 1 --2- 172.21.15.49:0/81168488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea601022b0 0x7fea60108b40 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fea58009900 tx=0x7fea5802bc30 comp rx=0 tx=0).stop 2024-03-18T01:01:06.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.130+0000 7fea64a9a640 1 -- 172.21.15.49:0/81168488 shutdown_connections 2024-03-18T01:01:06.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.130+0000 7fea64a9a640 1 --2- 172.21.15.49:0/81168488 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea601022b0 0x7fea60108b40 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.130+0000 7fea64a9a640 1 --2- 172.21.15.49:0/81168488 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea601018f0 0x7fea60101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.130+0000 7fea64a9a640 1 --2- 172.21.15.49:0/81168488 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea60100700 0x7fea60100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.130+0000 7fea64a9a640 1 -- 172.21.15.49:0/81168488 >> 172.21.15.49:0/81168488 conn(0x7fea600fbc90 msgr2=0x7fea600fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.130+0000 7fea64a9a640 1 -- 172.21.15.49:0/81168488 shutdown_connections 2024-03-18T01:01:06.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.130+0000 7fea64a9a640 1 -- 172.21.15.49:0/81168488 wait complete. 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea64a9a640 1 Processor -- start 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea64a9a640 1 -- start start 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea64a9a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea60100700 0x7fea60197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea64a9a640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea601018f0 0x7fea60198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea64a9a640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea601022b0 0x7fea6019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea64a9a640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fea6010c380 con 0x7fea60100700 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea64a9a640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fea6010c200 con 0x7fea601018f0 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea64a9a640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea6010c500 con 0x7fea601022b0 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea5e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea60100700 0x7fea60197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea5e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea60100700 0x7fea60197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34384/0 (socket says 172.21.15.49:34384) 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea5e59c640 1 -- 172.21.15.49:0/3804736229 learned_addr learned my addr 172.21.15.49:0/3804736229 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea5ed9d640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea601022b0 0x7fea6019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.131+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea601018f0 0x7fea60198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea5e59c640 1 -- 172.21.15.49:0/3804736229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea601022b0 msgr2=0x7fea6019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea5e59c640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea601022b0 0x7fea6019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea5e59c640 1 -- 172.21.15.49:0/3804736229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea601018f0 msgr2=0x7fea60198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea5e59c640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea601018f0 0x7fea60198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea5e59c640 1 -- 172.21.15.49:0/3804736229 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6019f6f0 con 0x7fea60100700 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea601018f0 0x7fea60198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea5e59c640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea60100700 0x7fea60197ad0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fea50009d20 tx=0x7fea5000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea4f7fe640 1 -- 172.21.15.49:0/3804736229 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50010630 con 0x7fea60100700 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea4f7fe640 1 -- 172.21.15.49:0/3804736229 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea50010cb0 con 0x7fea60100700 2024-03-18T01:01:06.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea64a9a640 1 -- 172.21.15.49:0/3804736229 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea6019f9e0 con 0x7fea60100700 2024-03-18T01:01:06.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea4f7fe640 1 -- 172.21.15.49:0/3804736229 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50018de0 con 0x7fea60100700 2024-03-18T01:01:06.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.132+0000 7fea64a9a640 1 -- 172.21.15.49:0/3804736229 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea6019ff30 con 0x7fea60100700 2024-03-18T01:01:06.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.134+0000 7fea4f7fe640 1 -- 172.21.15.49:0/3804736229 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fea5001f020 con 0x7fea60100700 2024-03-18T01:01:06.136 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.134+0000 7fea67765740 1 Processor -- start 2024-03-18T01:01:06.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea67765740 1 -- start start 2024-03-18T01:01:06.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea67765740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d24080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea67765740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d29150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea67765740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d2bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea67765740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c647c84490 con 0x55c647d297c0 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea67765740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c647c2cc90 con 0x55c647d24650 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea67765740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c647c80530 con 0x55c647d23c60 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea4f7fe640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea480a2800 0x7fea480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d24080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d2bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5e59c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d29150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d24080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58484/0 (socket says 172.21.15.49:58484) 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5ed9d640 1 -- 172.21.15.49:0/3319761980 learned_addr learned my addr 172.21.15.49:0/3319761980 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5ed9d640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea480a2800 0x7fea480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea4f7fe640 1 -- 172.21.15.49:0/3804736229 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fea500ceef0 con 0x7fea60100700 2024-03-18T01:01:06.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5ed9d640 1 -- 172.21.15.49:0/3319761980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 msgr2=0x55c647d29150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5ed9d640 1 --2- 172.21.15.49:0/3319761980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d29150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5ed9d640 1 -- 172.21.15.49:0/3319761980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 msgr2=0x55c647d2bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5ed9d640 1 --2- 172.21.15.49:0/3319761980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d2bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5ed9d640 1 -- 172.21.15.49:0/3319761980 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c647ce7500 con 0x55c647d23c60 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.135+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/3319761980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d2bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.136+0000 7fea5ed9d640 1 --2- 172.21.15.49:0/3319761980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d24080 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fea5802cfd0 tx=0x7fea5802cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc5845271ceb08e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.136+0000 7fea5ed9d640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea480a2800 0x7fea480a4cc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fea58048000 tx=0x7fea58049000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.136+0000 7fea4cff9640 1 -- 172.21.15.49:0/3319761980 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58004980 con 0x55c647d23c60 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.136+0000 7fea4cff9640 1 -- 172.21.15.49:0/3319761980 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea58004b20 con 0x55c647d23c60 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.136+0000 7fea4cff9640 1 -- 172.21.15.49:0/3319761980 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea580039d0 con 0x55c647d23c60 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 -- 172.21.15.49:0/3319761980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 msgr2=0x55c647d24080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 --2- 172.21.15.49:0/3319761980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d24080 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fea5802cfd0 tx=0x7fea5802cc50 comp rx=0 tx=0).stop 2024-03-18T01:01:06.144 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 -- 172.21.15.49:0/3319761980 shutdown_connections 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 --2- 172.21.15.49:0/3319761980 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d29150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 --2- 172.21.15.49:0/3319761980 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d2bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 --2- 172.21.15.49:0/3319761980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d24080 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 -- 172.21.15.49:0/3319761980 >> 172.21.15.49:0/3319761980 conn(0x55c647d1adc0 msgr2=0x55c647d1b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 -- 172.21.15.49:0/3319761980 shutdown_connections 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 -- 172.21.15.49:0/3319761980 wait complete. 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 -- start start 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 Processor -- start 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d2e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d2e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d35720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x55c647d2c710 con 0x55c647d297c0 2024-03-18T01:01:06.145 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x55c647c40230 con 0x55c647d24650 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x55c647d2c890 con 0x55c647d23c60 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.137+0000 7fea67765740 10 client.0 _ll_register_callbacks cb 0x55c647c7c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d2e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d35720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5e59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d2e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58494/0 (socket says 172.21.15.49:58494) 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5ed9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d35720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34410/0 (socket says 172.21.15.49:34410) 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5e59c640 1 -- 172.21.15.49:0/2618822386 learned_addr learned my addr 172.21.15.49:0/2618822386 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d2e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5e59c640 1 -- 172.21.15.49:0/2618822386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 msgr2=0x55c647d2e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5e59c640 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d2e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5e59c640 1 -- 172.21.15.49:0/2618822386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 msgr2=0x55c647d35720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5e59c640 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d35720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.146 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5e59c640 1 -- 172.21.15.49:0/2618822386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c647c4f100 con 0x55c647d23c60 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5ed9d640 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d35720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea5e59c640 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d2e1a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fea500cc9d0 tx=0x7fea50009900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96e73d7a734fdb3e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea45ffb640 1 -- 172.21.15.49:0/2618822386 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50010e60 con 0x55c647d23c60 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea45ffb640 1 -- 172.21.15.49:0/2618822386 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fea500184b0 con 0x55c647d23c60 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea67765740 1 -- 172.21.15.49:0/2618822386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea58004b20 con 0x55c647d23c60 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea45ffb640 1 -- 172.21.15.49:0/2618822386 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fea5009bf70 con 0x55c647d23c60 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea67765740 10 client.27322 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.138+0000 7fea67765740 1 -- 172.21.15.49:0/2618822386 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c6479983f0 con 0x55c647d23c60 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea45ffb640 1 -- 172.21.15.49:0/2618822386 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea58004b20 con 0x55c647d23c60 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea45ffb640 1 -- 172.21.15.49:0/2618822386 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fea50086070 con 0x55c647d23c60 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea67765740 10 client.27322 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea67765740 10 client.27322 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:01:06.147 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea67765740 20 client.27322 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea67765740 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55c647d3a000 0x55c647d3c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea67765740 4 client.27322 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea67765740 1 -- 172.21.15.49:0/2618822386 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c647c919d0 con 0x55c647d3a000 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55c647d3a000 0x55c647d3c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55c647d3a000 0x55c647d3c480 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1e90e033f89bcc09 server_cookie=3cd2521834cf0645 in_seq=0 out_seq=0 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.139+0000 7fea45ffb640 10 client.27322 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea45ffb640 1 -- 172.21.15.49:0/2618822386 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c647c919d0 con 0x55c647d3a000 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea45ffb640 10 client.27322 handle_command_reply: tid=0 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 client.27322 shutdown 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 -- 172.21.15.49:0/2618822386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 msgr2=0x55c647d2e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d2e1a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fea500cc9d0 tx=0x7fea50009900 comp rx=0 tx=0).stop 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 -- 172.21.15.49:0/2618822386 shutdown_connections 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55c647d3a000 0x55c647d3c480 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.148 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d2e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea45ffb640 0 client.27322 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T01:01:06.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d297c0 0x55c647d35720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 --2- 172.21.15.49:0/2618822386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d23c60 0x55c647d2e1a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 -- 172.21.15.49:0/2618822386 >> 172.21.15.49:0/2618822386 conn(0x55c647d1adc0 msgr2=0x55c647d19fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 -- 172.21.15.49:0/2618822386 shutdown_connections 2024-03-18T01:01:06.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 1 -- 172.21.15.49:0/2618822386 wait complete. 2024-03-18T01:01:06.149 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.140+0000 7fea67765740 20 client.27322 trim_cache size 0 max 16384 2024-03-18T01:01:06.158 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea67765740 1 Processor -- start 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea67765740 1 -- start start 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea67765740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d23c60 0x55c647d2e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea67765740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea67765740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 0x55c647d1e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea67765740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fea601a6b10 con 0x55c647d23c60 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea67765740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fea601a86b0 con 0x55c647d24650 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea67765740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea601a71e0 con 0x55c647d297c0 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d23c60 0x55c647d2e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5e59c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d23c60 0x55c647d2e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34414/0 (socket says 172.21.15.49:34414) 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5e59c640 1 -- 172.21.15.49:0/4017634619 learned_addr learned my addr 172.21.15.49:0/4017634619 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.159 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5ed9d640 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 0x55c647d1e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5e59c640 1 -- 172.21.15.49:0/4017634619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 msgr2=0x55c647d1e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5e59c640 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 0x55c647d1e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5e59c640 1 -- 172.21.15.49:0/4017634619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 msgr2=0x55c647d356e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5e59c640 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d356e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5e59c640 1 -- 172.21.15.49:0/4017634619 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c647a627d0 con 0x55c647d23c60 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5ed9d640 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 0x55c647d1e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea5e59c640 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d23c60 0x55c647d2e640 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fea50000f80 tx=0x7fea50099d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bfd3d03cd744c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea47fff640 1 -- 172.21.15.49:0/4017634619 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50010e60 con 0x55c647d23c60 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea47fff640 1 -- 172.21.15.49:0/4017634619 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea500a4010 con 0x55c647d23c60 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.154+0000 7fea47fff640 1 -- 172.21.15.49:0/4017634619 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea500a3c80 con 0x55c647d23c60 2024-03-18T01:01:06.160 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 -- 172.21.15.49:0/4017634619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d23c60 msgr2=0x55c647d2e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d23c60 0x55c647d2e640 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fea50000f80 tx=0x7fea50099d80 comp rx=0 tx=0).stop 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 -- 172.21.15.49:0/4017634619 shutdown_connections 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d24650 0x55c647d356e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 0x55c647d1e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 --2- 172.21.15.49:0/4017634619 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d23c60 0x55c647d2e640 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 -- 172.21.15.49:0/4017634619 >> 172.21.15.49:0/4017634619 conn(0x55c647d1adc0 msgr2=0x55c647d1e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 -- 172.21.15.49:0/4017634619 shutdown_connections 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 -- 172.21.15.49:0/4017634619 wait complete. 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 -- start start 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.155+0000 7fea67765740 1 Processor -- start 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea67765740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d23c60 0x55c647d32970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea67765740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d24650 0x55c647d32eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea67765740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 0x55c647d33e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.161 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea67765740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fea601a7fb0 con 0x55c647d24650 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea67765740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fea601a78b0 con 0x55c647d23c60 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea67765740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fea601a4af0 con 0x55c647d297c0 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea67765740 10 client.0 _ll_register_callbacks cb 0x7fea50004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d24650 0x55c647d32eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5dd9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d24650 0x55c647d32eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34416/0 (socket says 172.21.15.49:34416) 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5ed9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 0x55c647d33e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5dd9b640 1 -- 172.21.15.49:0/3043703159 learned_addr learned my addr 172.21.15.49:0/3043703159 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5e59c640 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d23c60 0x55c647d32970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5dd9b640 1 -- 172.21.15.49:0/3043703159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 msgr2=0x55c647d33e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 0x55c647d33e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5dd9b640 1 -- 172.21.15.49:0/3043703159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d23c60 msgr2=0x55c647d32970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d23c60 0x55c647d32970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.162 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5dd9b640 1 -- 172.21.15.49:0/3043703159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c647d34410 con 0x55c647d24650 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea5dd9b640 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d24650 0x55c647d32eb0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fea540125c0 tx=0x7fea54013ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6083217f0010ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea45ffb640 1 -- 172.21.15.49:0/3043703159 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54024070 con 0x55c647d24650 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea45ffb640 1 -- 172.21.15.49:0/3043703159 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fea54005220 con 0x55c647d24650 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.156+0000 7fea67765740 1 -- 172.21.15.49:0/3043703159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea500a4010 con 0x55c647d24650 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea45ffb640 1 -- 172.21.15.49:0/3043703159 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fea5402f050 con 0x55c647d24650 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea67765740 10 client.18144 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea67765740 1 -- 172.21.15.49:0/3043703159 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c647d35810 con 0x55c647d24650 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea45ffb640 1 -- 172.21.15.49:0/3043703159 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea500a4010 con 0x55c647d24650 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea45ffb640 1 -- 172.21.15.49:0/3043703159 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7fea540200e0 con 0x55c647d24650 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea67765740 10 client.18144 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea67765740 10 client.18144 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea67765740 20 client.18144 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:06.163 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea67765740 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55c647da09f0 0x55c647da2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea67765740 4 client.18144 mds_command: new command op to 24470 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea67765740 1 -- 172.21.15.49:0/3043703159 --> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c647c8bd20 con 0x55c647da09f0 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea5e59c640 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55c647da09f0 0x55c647da2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea5e59c640 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55c647da09f0 0x55c647da2e10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=381ea37c6684b45a server_cookie=cba03dfd5c0e0523 in_seq=0 out_seq=0 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.157+0000 7fea45ffb640 10 client.18144 ms_handle_connect on v2:172.21.15.106:6836/3542707642 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea45ffb640 1 -- 172.21.15.49:0/3043703159 <== mds.3 v2:172.21.15.106:6836/3542707642 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c647c8bd20 con 0x55c647da09f0 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea45ffb640 10 client.18144 handle_command_reply: tid=0 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 client.18144 shutdown 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 -- 172.21.15.49:0/3043703159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d24650 msgr2=0x55c647d32eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d24650 0x55c647d32eb0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fea540125c0 tx=0x7fea54013ab0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 -- 172.21.15.49:0/3043703159 shutdown_connections 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.106:6836/3542707642,v1:172.21.15.106:6837/3542707642] conn(0x55c647da09f0 0x55c647da2e10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55c647d297c0 0x55c647d33e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.164 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x55c647d24650 0x55c647d32eb0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 --2- 172.21.15.49:0/3043703159 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x55c647d23c60 0x55c647d32970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea45ffb640 0 client.18144 ms_handle_reset on v2:172.21.15.106:6836/3542707642 2024-03-18T01:01:06.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 -- 172.21.15.49:0/3043703159 >> 172.21.15.49:0/3043703159 conn(0x55c647d1adc0 msgr2=0x55c647d1e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 -- 172.21.15.49:0/3043703159 shutdown_connections 2024-03-18T01:01:06.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 1 -- 172.21.15.49:0/3043703159 wait complete. 2024-03-18T01:01:06.165 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea67765740 20 client.18144 trim_cache size 0 max 16384 2024-03-18T01:01:06.165 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:01:06.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 -- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea480a2800 msgr2=0x7fea480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea480a2800 0x7fea480a4cc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fea58048000 tx=0x7fea58049000 comp rx=0 tx=0).stop 2024-03-18T01:01:06.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 -- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea60100700 msgr2=0x7fea60197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea60100700 0x7fea60197ad0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fea50009d20 tx=0x7fea5000c6a0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.166 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 -- 172.21.15.49:0/3804736229 shutdown_connections 2024-03-18T01:01:06.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fea480a2800 0x7fea480a4cc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fea601018f0 0x7fea60198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fea601022b0 0x7fea6019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 --2- 172.21.15.49:0/3804736229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fea60100700 0x7fea60197ad0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.158+0000 7fea64a9a640 1 -- 172.21.15.49:0/3804736229 >> 172.21.15.49:0/3804736229 conn(0x7fea600fbc90 msgr2=0x7fea60069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.159+0000 7fea64a9a640 1 -- 172.21.15.49:0/3804736229 shutdown_connections 2024-03-18T01:01:06.167 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.159+0000 7fea64a9a640 1 -- 172.21.15.49:0/3804736229 wait complete. 2024-03-18T01:01:06.169 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T01:01:06.281 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/850430829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a681022b0 msgr2=0x7f8a68108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/850430829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a681022b0 0x7f8a68108b40 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009ca0 tx=0x7f8a6002c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/850430829 shutdown_connections 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/850430829 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a681018f0 0x7f8a68101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/850430829 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a681022b0 0x7f8a68108b40 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/850430829 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a68100700 0x7f8a68100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/850430829 >> 172.21.15.49:0/850430829 conn(0x7f8a680fbc90 msgr2=0x7f8a680fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/850430829 shutdown_connections 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/850430829 wait complete. 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 Processor -- start 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 -- start start 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a68100700 0x7f8a68197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a681018f0 0x7f8a68198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a681022b0 0x7f8a6819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810c380 con 0x7f8a68100700 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810c200 con 0x7f8a681022b0 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.280+0000 7f8a6ebe6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810c500 con 0x7f8a681018f0 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6c982640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a68100700 0x7f8a68197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6c982640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a68100700 0x7f8a68197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34444/0 (socket says 172.21.15.49:34444) 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6c982640 1 -- 172.21.15.49:0/1563937229 learned_addr learned my addr 172.21.15.49:0/1563937229 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6d183640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a681022b0 0x7f8a6819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a67fff640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a681018f0 0x7f8a68198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6c982640 1 -- 172.21.15.49:0/1563937229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a681018f0 msgr2=0x7f8a68198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.283 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6c982640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a681018f0 0x7f8a68198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6c982640 1 -- 172.21.15.49:0/1563937229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a681022b0 msgr2=0x7f8a6819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6c982640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a681022b0 0x7f8a6819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6c982640 1 -- 172.21.15.49:0/1563937229 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6819f770 con 0x7f8a68100700 2024-03-18T01:01:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6c982640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a68100700 0x7f8a68197b50 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800e9c0 tx=0x7f8a5800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a65ffb640 1 -- 172.21.15.49:0/1563937229 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5800cd80 con 0x7f8a68100700 2024-03-18T01:01:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/1563937229 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6819fa60 con 0x7f8a68100700 2024-03-18T01:01:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a65ffb640 1 -- 172.21.15.49:0/1563937229 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a580035f0 con 0x7f8a68100700 2024-03-18T01:01:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a65ffb640 1 -- 172.21.15.49:0/1563937229 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58010690 con 0x7f8a68100700 2024-03-18T01:01:06.284 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.281+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/1563937229 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6819ff60 con 0x7f8a68100700 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.283+0000 7f8a65ffb640 1 -- 172.21.15.49:0/1563937229 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8a580037a0 con 0x7f8a68100700 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a717d1740 1 Processor -- start 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a717d1740 1 -- start start 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 0x564bfddcf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfddd4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfddd6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a717d1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564bfdd2f490 con 0x564bfddcf650 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a717d1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x564bfdcd7c90 con 0x564bfddcec60 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a717d1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564bfdd2b530 con 0x564bfddd47c0 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a65ffb640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a500a2800 0x7f8a500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a6d183640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 0x564bfddcf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a6c982640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfddd4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.293 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a6d183640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 0x564bfddcf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:35338/0 (socket says 172.21.15.49:35338) 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a6d183640 1 -- 172.21.15.49:0/2157474965 learned_addr learned my addr 172.21.15.49:0/2157474965 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a67fff640 1 --2- 172.21.15.49:0/2157474965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfddd6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a6d183640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a500a2800 0x7f8a500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.284+0000 7f8a65ffb640 1 -- 172.21.15.49:0/1563937229 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8a580cede0 con 0x7f8a68100700 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a6c982640 1 -- 172.21.15.49:0/2157474965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 msgr2=0x564bfddd6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a6c982640 1 --2- 172.21.15.49:0/2157474965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfddd6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a6c982640 1 -- 172.21.15.49:0/2157474965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 msgr2=0x564bfddcf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a6c982640 1 --2- 172.21.15.49:0/2157474965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 0x564bfddcf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a6c982640 1 -- 172.21.15.49:0/2157474965 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bfdd92500 con 0x564bfddcf650 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a67fff640 1 --2- 172.21.15.49:0/2157474965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfddd6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a6d183640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a500a2800 0x7f8a500a4cc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x564bfddcfeb0 tx=0x7f8a60032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a6c982640 1 --2- 172.21.15.49:0/2157474965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfddd4150 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f8a5809b460 tx=0x7f8a5806e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5d15c3930df55 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a4effd640 1 -- 172.21.15.49:0/2157474965 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5809b2a0 con 0x564bfddcf650 2024-03-18T01:01:06.294 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a4effd640 1 -- 172.21.15.49:0/2157474965 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a5809be60 con 0x564bfddcf650 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.285+0000 7f8a4effd640 1 -- 172.21.15.49:0/2157474965 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5809b0c0 con 0x564bfddcf650 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 -- 172.21.15.49:0/2157474965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 msgr2=0x564bfddd4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 --2- 172.21.15.49:0/2157474965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfddd4150 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f8a5809b460 tx=0x7f8a5806e490 comp rx=0 tx=0).stop 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 -- 172.21.15.49:0/2157474965 shutdown_connections 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 --2- 172.21.15.49:0/2157474965 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfddd6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 --2- 172.21.15.49:0/2157474965 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfddd4150 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 --2- 172.21.15.49:0/2157474965 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 0x564bfddcf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 -- 172.21.15.49:0/2157474965 >> 172.21.15.49:0/2157474965 conn(0x564bfddc5dc0 msgr2=0x564bfddc61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 -- 172.21.15.49:0/2157474965 shutdown_connections 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 -- 172.21.15.49:0/2157474965 wait complete. 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 -- start start 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.286+0000 7f8a717d1740 1 Processor -- start 2024-03-18T01:01:06.295 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddcec60 0x564bfddd9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 0x564bfddd96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddd47c0 0x564bfdde06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a717d1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x564bfddd7710 con 0x564bfddd47c0 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a717d1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x564bfdceb230 con 0x564bfddcf650 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a717d1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x564bfddd7890 con 0x564bfddcec60 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a717d1740 10 client.0 _ll_register_callbacks cb 0x564bfdd27010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6c982640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddcec60 0x564bfddd9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6d183640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddd47c0 0x564bfdde06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a67fff640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 0x564bfddd96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6c982640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddcec60 0x564bfddd9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58556/0 (socket says 172.21.15.49:58556) 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6c982640 1 -- 172.21.15.49:0/3747110916 learned_addr learned my addr 172.21.15.49:0/3747110916 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6c982640 1 -- 172.21.15.49:0/3747110916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 msgr2=0x564bfddd96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.296 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6c982640 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 0x564bfddd96b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6c982640 1 -- 172.21.15.49:0/3747110916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddd47c0 msgr2=0x564bfdde06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6c982640 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddd47c0 0x564bfdde06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6c982640 1 -- 172.21.15.49:0/3747110916 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bfdcfa100 con 0x564bfddcec60 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6d183640 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddd47c0 0x564bfdde06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a6c982640 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddcec60 0x564bfddd9170 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8a580ccc30 tx=0x7f8a58003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb955f0aa8d2e7c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a47fff640 10 client.0 ms_handle_connect on v2:172.21.15.107:3300/0 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a47fff640 1 -- 172.21.15.49:0/3747110916 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5809b2a0 con 0x564bfddcec60 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a717d1740 1 -- 172.21.15.49:0/3747110916 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a5809be60 con 0x564bfddcec60 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.287+0000 7f8a47fff640 1 -- 172.21.15.49:0/3747110916 <== mon.1 v2:172.21.15.107:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8a58097dd0 con 0x564bfddcec60 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a47fff640 1 -- 172.21.15.49:0/3747110916 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8a580ba020 con 0x564bfddcec60 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a717d1740 10 client.27340 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a717d1740 1 -- 172.21.15.49:0/3747110916 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564bfda433f0 con 0x564bfddcec60 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a47fff640 1 -- 172.21.15.49:0/3747110916 <== mon.1 v2:172.21.15.107:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a5809be60 con 0x564bfddcec60 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a47fff640 1 -- 172.21.15.49:0/3747110916 <== mon.1 v2:172.21.15.107:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8a5806c070 con 0x564bfddcec60 2024-03-18T01:01:06.297 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a717d1740 10 client.27340 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a717d1740 10 client.27340 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a717d1740 20 client.27340 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a717d1740 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x564bfdde4be0 0x564bfdde6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a717d1740 4 client.27340 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a717d1740 1 -- 172.21.15.49:0/3747110916 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bfdd3c9d0 con 0x564bfdde4be0 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a67fff640 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x564bfdde4be0 0x564bfdde6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a67fff640 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x564bfdde4be0 0x564bfdde6fd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=39ebeb21f7851c8a server_cookie=621b2ee244dae034 in_seq=0 out_seq=0 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.288+0000 7f8a47fff640 10 client.27340 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.289+0000 7f8a47fff640 1 -- 172.21.15.49:0/3747110916 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564bfdd3c9d0 con 0x564bfdde4be0 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.289+0000 7f8a47fff640 10 client.27340 handle_command_reply: tid=0 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.289+0000 7f8a717d1740 1 client.27340 shutdown 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 -- 172.21.15.49:0/3747110916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddcec60 msgr2=0x564bfddd9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.298 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddcec60 0x564bfddd9170 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8a580ccc30 tx=0x7f8a58003f10 comp rx=0 tx=0).stop 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 -- 172.21.15.49:0/3747110916 shutdown_connections 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x564bfdde4be0 0x564bfdde6fd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 0x564bfddd96b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddd47c0 0x564bfdde06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 --2- 172.21.15.49:0/3747110916 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddcec60 0x564bfddd9170 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a47fff640 0 client.27340 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 -- 172.21.15.49:0/3747110916 >> 172.21.15.49:0/3747110916 conn(0x564bfddc5dc0 msgr2=0x564bfddc4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 -- 172.21.15.49:0/3747110916 shutdown_connections 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 1 -- 172.21.15.49:0/3747110916 wait complete. 2024-03-18T01:01:06.299 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.290+0000 7f8a717d1740 20 client.27340 trim_cache size 0 max 16384 2024-03-18T01:01:06.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a717d1740 1 Processor -- start 2024-03-18T01:01:06.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a717d1740 1 -- start start 2024-03-18T01:01:06.308 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 0x564bfddd9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfdde0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfddc9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a717d1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681a6c80 con 0x564bfddcf650 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a717d1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681a8820 con 0x564bfddcec60 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a717d1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681a7350 con 0x564bfddd47c0 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a67fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfdde0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a67fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfdde0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34470/0 (socket says 172.21.15.49:34470) 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a67fff640 1 -- 172.21.15.49:0/1713046541 learned_addr learned my addr 172.21.15.49:0/1713046541 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a6c982640 1 --2- 172.21.15.49:0/1713046541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 0x564bfddd9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.309 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a6d183640 1 --2- 172.21.15.49:0/1713046541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfddc9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a67fff640 1 -- 172.21.15.49:0/1713046541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 msgr2=0x564bfddc9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a67fff640 1 --2- 172.21.15.49:0/1713046541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfddc9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a67fff640 1 -- 172.21.15.49:0/1713046541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 msgr2=0x564bfddd9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a67fff640 1 --2- 172.21.15.49:0/1713046541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 0x564bfddd9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.303+0000 7f8a67fff640 1 -- 172.21.15.49:0/1713046541 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bfdb0d7d0 con 0x564bfddcf650 2024-03-18T01:01:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a67fff640 1 --2- 172.21.15.49:0/1713046541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfdde0680 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x564bfddc6d30 tx=0x7f8a5c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f8ae479d144fa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a4e7fc640 1 -- 172.21.15.49:0/1713046541 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c025070 con 0x564bfddcf650 2024-03-18T01:01:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a4e7fc640 1 -- 172.21.15.49:0/1713046541 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a5c003d80 con 0x564bfddcf650 2024-03-18T01:01:06.310 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a4e7fc640 1 -- 172.21.15.49:0/1713046541 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c010b80 con 0x564bfddcf650 2024-03-18T01:01:06.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a717d1740 1 -- 172.21.15.49:0/1713046541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 msgr2=0x564bfdde0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a717d1740 1 --2- 172.21.15.49:0/1713046541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfdde0680 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x564bfddc6d30 tx=0x7f8a5c005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:06.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a717d1740 1 -- 172.21.15.49:0/1713046541 shutdown_connections 2024-03-18T01:01:06.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a717d1740 1 --2- 172.21.15.49:0/1713046541 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfddc9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a717d1740 1 --2- 172.21.15.49:0/1713046541 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcf650 0x564bfdde0680 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.311 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a717d1740 1 --2- 172.21.15.49:0/1713046541 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcec60 0x564bfddd9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a717d1740 1 -- 172.21.15.49:0/1713046541 >> 172.21.15.49:0/1713046541 conn(0x564bfddc5dc0 msgr2=0x564bfddc9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a717d1740 1 -- 172.21.15.49:0/1713046541 shutdown_connections 2024-03-18T01:01:06.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.304+0000 7f8a717d1740 1 -- 172.21.15.49:0/1713046541 wait complete. 2024-03-18T01:01:06.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a717d1740 1 -- start start 2024-03-18T01:01:06.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a717d1740 1 Processor -- start 2024-03-18T01:01:06.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcec60 0x564bfdddd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.312 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 0x564bfddddb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a717d1740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfde48db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a717d1740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681a8120 con 0x564bfddcec60 2024-03-18T01:01:06.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a717d1740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681a7a20 con 0x564bfddcf650 2024-03-18T01:01:06.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a717d1740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8a681a4c60 con 0x564bfddd47c0 2024-03-18T01:01:06.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a717d1740 10 client.0 _ll_register_callbacks cb 0x7f8a5c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:06.313 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a6c982640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcec60 0x564bfdddd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a6c982640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcec60 0x564bfdddd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34484/0 (socket says 172.21.15.49:34484) 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a6c982640 1 -- 172.21.15.49:0/4276367892 learned_addr learned my addr 172.21.15.49:0/4276367892 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a6d183640 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfde48db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a67fff640 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 0x564bfddddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a6c982640 1 -- 172.21.15.49:0/4276367892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 msgr2=0x564bfde48db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a6c982640 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfde48db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a6c982640 1 -- 172.21.15.49:0/4276367892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 msgr2=0x564bfddddb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a6c982640 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 0x564bfddddb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.305+0000 7f8a6c982640 1 -- 172.21.15.49:0/4276367892 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bfde492f0 con 0x564bfddcec60 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a6c982640 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcec60 0x564bfdddd600 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8a58098130 tx=0x7f8a58005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c935609af9e09caf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.314 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a47fff640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a47fff640 1 -- 172.21.15.49:0/4276367892 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5807f060 con 0x564bfddcec60 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a47fff640 1 -- 172.21.15.49:0/4276367892 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8a5809b9c0 con 0x564bfddcec60 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a717d1740 1 -- 172.21.15.49:0/4276367892 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a5c003d80 con 0x564bfddcec60 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a47fff640 1 -- 172.21.15.49:0/4276367892 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8a5809b2a0 con 0x564bfddcec60 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a717d1740 10 client.18162 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a717d1740 1 -- 172.21.15.49:0/4276367892 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564bfde4a510 con 0x564bfddcec60 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a47fff640 1 -- 172.21.15.49:0/4276367892 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a5c003d80 con 0x564bfddcec60 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a47fff640 1 -- 172.21.15.49:0/4276367892 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f8a5809ad40 con 0x564bfddcec60 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a717d1740 10 client.18162 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a717d1740 10 client.18162 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a717d1740 20 client.18162 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a717d1740 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x564bfde4acb0 0x564bfde4d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a717d1740 4 client.18162 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:01:06.315 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a717d1740 1 -- 172.21.15.49:0/4276367892 --> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564bfdd36d20 con 0x564bfde4acb0 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.306+0000 7f8a67fff640 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x564bfde4acb0 0x564bfde4d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a47fff640 10 client.18162 ms_handle_connect on v2:172.21.15.107:6836/3043451403 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a67fff640 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x564bfde4acb0 0x564bfde4d0d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d2a8159ce7a8fef4 server_cookie=2c04f03340583831 in_seq=0 out_seq=0 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a47fff640 1 -- 172.21.15.49:0/4276367892 <== mds.2 v2:172.21.15.107:6836/3043451403 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564bfdd36d20 con 0x564bfde4acb0 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a47fff640 10 client.18162 handle_command_reply: tid=0 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 client.18162 shutdown 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 -- 172.21.15.49:0/4276367892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcec60 msgr2=0x564bfdddd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcec60 0x564bfdddd600 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f8a58098130 tx=0x7f8a58005a70 comp rx=0 tx=0).stop 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 -- 172.21.15.49:0/4276367892 shutdown_connections 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.107:6836/3043451403,v1:172.21.15.107:6837/3043451403] conn(0x564bfde4acb0 0x564bfde4d0d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x564bfddcf650 0x564bfddddb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x564bfddd47c0 0x564bfde48db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 --2- 172.21.15.49:0/4276367892 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x564bfddcec60 0x564bfdddd600 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a47fff640 0 client.18162 ms_handle_reset on v2:172.21.15.107:6836/3043451403 2024-03-18T01:01:06.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 -- 172.21.15.49:0/4276367892 >> 172.21.15.49:0/4276367892 conn(0x564bfddc5dc0 msgr2=0x564bfddc98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 -- 172.21.15.49:0/4276367892 shutdown_connections 2024-03-18T01:01:06.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 1 -- 172.21.15.49:0/4276367892 wait complete. 2024-03-18T01:01:06.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.307+0000 7f8a717d1740 20 client.18162 trim_cache size 0 max 16384 2024-03-18T01:01:06.317 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a500a2800 msgr2=0x7f8a500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a500a2800 0x7f8a500a4cc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x564bfddcfeb0 tx=0x7f8a60032590 comp rx=0 tx=0).stop 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a68100700 msgr2=0x7f8a68197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a68100700 0x7f8a68197b50 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800e9c0 tx=0x7f8a5800ee80 comp rx=0 tx=0).stop 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/1563937229 shutdown_connections 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8a500a2800 0x7f8a500a4cc0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8a681022b0 0x7f8a6819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8a681018f0 0x7f8a68198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 --2- 172.21.15.49:0/1563937229 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8a68100700 0x7f8a68197b50 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/1563937229 >> 172.21.15.49:0/1563937229 conn(0x7f8a680fbc90 msgr2=0x7f8a680fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/1563937229 shutdown_connections 2024-03-18T01:01:06.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.308+0000 7f8a6ebe6640 1 -- 172.21.15.49:0/1563937229 wait complete. 2024-03-18T01:01:06.320 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T01:01:06.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:06 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/529879248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:01:06.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.431+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/1469057451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c78102290 msgr2=0x7f4c78108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.431+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/1469057451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c78102290 0x7f4c78108b20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4c68009ca0 tx=0x7f4c6802c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.431+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/1469057451 shutdown_connections 2024-03-18T01:01:06.433 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.431+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/1469057451 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c781018d0 0x7f4c78101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.431+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/1469057451 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c78102290 0x7f4c78108b20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.431+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/1469057451 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c781006e0 0x7f4c78100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.431+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/1469057451 >> 172.21.15.49:0/1469057451 conn(0x7f4c780fbc70 msgr2=0x7f4c780fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/1469057451 shutdown_connections 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/1469057451 wait complete. 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 Processor -- start 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 -- start start 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c781006e0 0x7f4c78197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c781018d0 0x7f4c78197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c78102290 0x7f4c7819f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7810d450 con 0x7f4c781018d0 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7810d2d0 con 0x7f4c781006e0 2024-03-18T01:01:06.434 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.432+0000 7f4c7ce0b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7810d5d0 con 0x7f4c78102290 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c75d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c781018d0 0x7f4c78197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c76d9d640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c78102290 0x7f4c7819f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c7659c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c781006e0 0x7f4c78197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c75d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c781018d0 0x7f4c78197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34502/0 (socket says 172.21.15.49:34502) 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c75d9b640 1 -- 172.21.15.49:0/15880056 learned_addr learned my addr 172.21.15.49:0/15880056 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c76d9d640 1 -- 172.21.15.49:0/15880056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c781006e0 msgr2=0x7f4c78197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c76d9d640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c781006e0 0x7f4c78197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c76d9d640 1 -- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c781018d0 msgr2=0x7f4c78197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c76d9d640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c781018d0 0x7f4c78197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c76d9d640 1 -- 172.21.15.49:0/15880056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7819f6c0 con 0x7f4c78102290 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c781018d0 0x7f4c78197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c7659c640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c781006e0 0x7f4c78197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c76d9d640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c78102290 0x7f4c7819f050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4c68037040 tx=0x7f4c68032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.435 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c6f7fe640 1 -- 172.21.15.49:0/15880056 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68041020 con 0x7f4c78102290 2024-03-18T01:01:06.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c6f7fe640 1 -- 172.21.15.49:0/15880056 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c68034cd0 con 0x7f4c78102290 2024-03-18T01:01:06.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/15880056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7819f950 con 0x7f4c78102290 2024-03-18T01:01:06.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c6f7fe640 1 -- 172.21.15.49:0/15880056 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c68039040 con 0x7f4c78102290 2024-03-18T01:01:06.436 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.433+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/15880056 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7819fe10 con 0x7f4c78102290 2024-03-18T01:01:06.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c7f9fc740 1 Processor -- start 2024-03-18T01:01:06.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c6f7fe640 1 -- 172.21.15.49:0/15880056 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4c68044020 con 0x7f4c78102290 2024-03-18T01:01:06.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c7f9fc740 1 -- start start 2024-03-18T01:01:06.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9f93c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f94240 0x5578f9f98d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.442 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f993b0 0x5578f9f9b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5578f9ef8490 con 0x5578f9f93850 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5578f9ea0c90 con 0x5578f9f94240 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5578f9ef4530 con 0x5578f9f993b0 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c75d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9f93c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c75d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9f93c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34518/0 (socket says 172.21.15.49:34518) 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c75d9b640 1 -- 172.21.15.49:0/3542547154 learned_addr learned my addr 172.21.15.49:0/3542547154 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c7659c640 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f94240 0x5578f9f98d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.436+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f993b0 0x5578f9f9b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c6f7fe640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4c640a2800 0x7f4c640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c75d9b640 1 -- 172.21.15.49:0/3542547154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f993b0 msgr2=0x5578f9f9b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f993b0 0x5578f9f9b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c75d9b640 1 -- 172.21.15.49:0/3542547154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f94240 msgr2=0x5578f9f98d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f94240 0x5578f9f98d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.443 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c75d9b640 1 -- 172.21.15.49:0/3542547154 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f9f5b500 con 0x5578f9f93850 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c7659c640 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f94240 0x5578f9f98d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c76d9d640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4c640a2800 0x7f4c640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c6f7fe640 1 -- 172.21.15.49:0/15880056 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4c680f0890 con 0x7f4c78102290 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9f93c70 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4c60013ad0 tx=0x7f4c60009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d57a1f6f3736d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c6cff9640 1 -- 172.21.15.49:0/3542547154 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000fa40 con 0x5578f9f93850 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c6cff9640 1 -- 172.21.15.49:0/3542547154 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c6000fbe0 con 0x5578f9f93850 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.437+0000 7f4c6cff9640 1 -- 172.21.15.49:0/3542547154 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6000e3b0 con 0x5578f9f93850 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3542547154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 msgr2=0x5578f9f93c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c76d9d640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4c640a2800 0x7f4c640a4cc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4c680324c0 tx=0x7f4c680dd040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9f93c70 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f4c60013ad0 tx=0x7f4c60009c10 comp rx=0 tx=0).stop 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3542547154 shutdown_connections 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f94240 0x5578f9f98d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f993b0 0x5578f9f9b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.444 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3542547154 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9f93c70 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3542547154 >> 172.21.15.49:0/3542547154 conn(0x5578f9f8edc0 msgr2=0x5578f9f8f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3542547154 shutdown_connections 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3542547154 wait complete. 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 -- start start 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.438+0000 7f4c7f9fc740 1 Processor -- start 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578f9fa5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x5578f9f9c360 con 0x5578f9f94240 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x5578f9eb4230 con 0x5578f9f993b0 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x5578f9f9c4e0 con 0x5578f9f93850 2024-03-18T01:01:06.445 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7f9fc740 10 client.0 _ll_register_callbacks cb 0x5578f9ef0010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34522/0 (socket says 172.21.15.49:34522) 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7659c640 1 -- 172.21.15.49:0/3788503251 learned_addr learned my addr 172.21.15.49:0/3788503251 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c75d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578f9fa5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7659c640 1 -- 172.21.15.49:0/3788503251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 msgr2=0x5578f9f9ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7659c640 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7659c640 1 -- 172.21.15.49:0/3788503251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 msgr2=0x5578f9fa5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7659c640 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578f9fa5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7659c640 1 -- 172.21.15.49:0/3788503251 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578f9ec3100 con 0x5578f9f94240 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7659c640 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4c70012d00 tx=0x7f4c7000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f846cafa7e58acb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c5dffb640 1 -- 172.21.15.49:0/3788503251 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70012110 con 0x5578f9f94240 2024-03-18T01:01:06.446 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c5dffb640 1 -- 172.21.15.49:0/3788503251 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4c70004050 con 0x5578f9f94240 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.439+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3788503251 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c6000fbe0 con 0x5578f9f94240 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c5dffb640 1 -- 172.21.15.49:0/3788503251 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4c700122b0 con 0x5578f9f94240 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c7f9fc740 10 client.18174 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3788503251 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5578f9c0c3f0 con 0x5578f9f94240 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c5dffb640 1 -- 172.21.15.49:0/3788503251 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c6000fbe0 con 0x5578f9f94240 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c5dffb640 1 -- 172.21.15.49:0/3788503251 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4c70010430 con 0x5578f9f94240 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c7f9fc740 10 client.18174 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c7f9fc740 10 client.18174 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c7f9fc740 20 client.18174 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5578f9fa9d80 0x5578f9fac1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c7f9fc740 4 client.18174 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3788503251 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578f9f059d0 con 0x5578f9fa9d80 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5578f9fa9d80 0x5578f9fac1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.447 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5578f9fa9d80 0x5578f9fac1c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=35f8bb05e19b5fc server_cookie=1e26d27514eb570c in_seq=0 out_seq=0 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.440+0000 7f4c5dffb640 10 client.18174 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c5dffb640 1 -- 172.21.15.49:0/3788503251 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5578f9f059d0 con 0x5578f9fa9d80 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c5dffb640 10 client.18174 handle_command_reply: tid=0 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 client.18174 shutdown 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3788503251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 msgr2=0x5578f9f9e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f4c70012d00 tx=0x7f4c7000f790 comp rx=0 tx=0).stop 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3788503251 shutdown_connections 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5578f9fa9d80 0x5578f9fac1c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578f9fa5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c5dffb640 0 client.18174 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/3788503251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.448 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3788503251 >> 172.21.15.49:0/3788503251 conn(0x5578f9f8edc0 msgr2=0x5578f9f8dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3788503251 shutdown_connections 2024-03-18T01:01:06.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/3788503251 wait complete. 2024-03-18T01:01:06.449 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.441+0000 7f4c7f9fc740 20 client.18174 trim_cache size 0 max 16384 2024-03-18T01:01:06.449 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:06 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/529879248' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:01:06.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7f9fc740 1 Processor -- start 2024-03-18T01:01:06.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7f9fc740 1 -- start start 2024-03-18T01:01:06.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.460 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578f9fa5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4c781a8d30 con 0x5578f9f94240 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4c781aa8d0 con 0x5578f9f993b0 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4c781a9400 con 0x5578f9f93850 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7659c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c75d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c75d9b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:58616/0 (socket says 172.21.15.49:58616) 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c75d9b640 1 -- 172.21.15.49:0/209541133 learned_addr learned my addr 172.21.15.49:0/209541133 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/209541133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578f9fa5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.461 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c75d9b640 1 -- 172.21.15.49:0/209541133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 msgr2=0x5578f9fa5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/209541133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578f9fa5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c75d9b640 1 -- 172.21.15.49:0/209541133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 msgr2=0x5578f9f9e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/209541133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c75d9b640 1 -- 172.21.15.49:0/209541133 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578f9cd67d0 con 0x5578f9f93850 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.455+0000 7f4c7659c640 1 --2- 172.21.15.49:0/209541133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/209541133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4c60005780 tx=0x7f4c60009c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46a1e8b4c4e2f0b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c5ffff640 1 -- 172.21.15.49:0/209541133 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c60010b20 con 0x5578f9f93850 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c5ffff640 1 -- 172.21.15.49:0/209541133 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c60010cc0 con 0x5578f9f93850 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c5ffff640 1 -- 172.21.15.49:0/209541133 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c600316a0 con 0x5578f9f93850 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/209541133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 msgr2=0x5578f9f9ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.462 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/209541133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4c60005780 tx=0x7f4c60009c10 comp rx=0 tx=0).stop 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/209541133 shutdown_connections 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/209541133 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578f9fa5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/209541133 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f94240 0x5578f9f9e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/209541133 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f93850 0x5578f9f9ddf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/209541133 >> 172.21.15.49:0/209541133 conn(0x5578f9f8edc0 msgr2=0x5578f9f8f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/209541133 shutdown_connections 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.456+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/209541133 wait complete. 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7f9fc740 1 -- start start 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7f9fc740 1 Processor -- start 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9fa2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.463 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f94240 0x5578f9fa25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7f9fc740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578fa00e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4c781aa1d0 con 0x5578f9f93850 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4c781a9ad0 con 0x5578f9f993b0 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7f9fc740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4c781a6d10 con 0x5578f9f94240 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7f9fc740 10 client.0 _ll_register_callbacks cb 0x7f4c70009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9fa2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9fa2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34532/0 (socket says 172.21.15.49:34532) 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 -- 172.21.15.49:0/1723069294 learned_addr learned my addr 172.21.15.49:0/1723069294 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c7659c640 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f94240 0x5578f9fa25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.464 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578fa00e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 -- 172.21.15.49:0/1723069294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f94240 msgr2=0x5578f9fa25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f94240 0x5578f9fa25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 -- 172.21.15.49:0/1723069294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 msgr2=0x5578fa00e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578fa00e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 -- 172.21.15.49:0/1723069294 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578f9fa55f0 con 0x5578f9f93850 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c75d9b640 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9fa2080 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009e50 tx=0x7f4c60010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8676accd98bcee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.457+0000 7f4c5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c5dffb640 1 -- 172.21.15.49:0/1723069294 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c60010630 con 0x5578f9f93850 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c5dffb640 1 -- 172.21.15.49:0/1723069294 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4c60031440 con 0x5578f9f93850 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/1723069294 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c60010cc0 con 0x5578f9f93850 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c5dffb640 1 -- 172.21.15.49:0/1723069294 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f4c60031640 con 0x5578f9f93850 2024-03-18T01:01:06.465 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7f9fc740 10 client.18180 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/1723069294 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5578fa00f5c0 con 0x5578f9f93850 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c5dffb640 1 -- 172.21.15.49:0/1723069294 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c60010cc0 con 0x5578f9f93850 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c5dffb640 1 -- 172.21.15.49:0/1723069294 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7f4c600035f0 con 0x5578f9f93850 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7f9fc740 10 client.18180 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7f9fc740 10 client.18180 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7f9fc740 20 client.18180 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5578fa00fdd0 0x5578fa0121f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7f9fc740 4 client.18180 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/1723069294 --> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5578f9effd20 con 0x5578fa00fdd0 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.458+0000 7f4c7659c640 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5578fa00fdd0 0x5578fa0121f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c5dffb640 10 client.18180 ms_handle_connect on v2:172.21.15.107:6838/2909446244 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7659c640 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5578fa00fdd0 0x5578fa0121f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=537363742b4077ba server_cookie=2d721015813458a1 in_seq=0 out_seq=0 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c5dffb640 1 -- 172.21.15.49:0/1723069294 <== mds.1 v2:172.21.15.107:6838/2909446244 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5578f9effd20 con 0x5578fa00fdd0 2024-03-18T01:01:06.466 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c5dffb640 10 client.18180 handle_command_reply: tid=0 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 client.18180 shutdown 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/1723069294 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 msgr2=0x5578f9fa2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9fa2080 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f4c60009e50 tx=0x7f4c60010e90 comp rx=0 tx=0).stop 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/1723069294 shutdown_connections 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.107:6838/2909446244,v1:172.21.15.107:6839/2909446244] conn(0x5578fa00fdd0 0x5578fa0121f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x5578f9f993b0 0x5578fa00e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5578f9f94240 0x5578f9fa25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c5dffb640 0 client.18180 ms_handle_reset on v2:172.21.15.107:6838/2909446244 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 --2- 172.21.15.49:0/1723069294 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x5578f9f93850 0x5578f9fa2080 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/1723069294 >> 172.21.15.49:0/1723069294 conn(0x5578f9f8edc0 msgr2=0x5578f9f957a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/1723069294 shutdown_connections 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 1 -- 172.21.15.49:0/1723069294 wait complete. 2024-03-18T01:01:06.467 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.459+0000 7f4c7f9fc740 20 client.18180 trim_cache size 0 max 16384 2024-03-18T01:01:06.468 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4c640a2800 msgr2=0x7f4c640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4c640a2800 0x7f4c640a4cc0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4c680324c0 tx=0x7f4c680dd040 comp rx=0 tx=0).stop 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/15880056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c78102290 msgr2=0x7f4c7819f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c78102290 0x7f4c7819f050 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4c68037040 tx=0x7f4c68032ea0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/15880056 shutdown_connections 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4c640a2800 0x7f4c640a4cc0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4c78102290 0x7f4c7819f050 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4c781018d0 0x7f4c78197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 --2- 172.21.15.49:0/15880056 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4c781006e0 0x7f4c78197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/15880056 >> 172.21.15.49:0/15880056 conn(0x7f4c780fbc70 msgr2=0x7f4c78104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/15880056 shutdown_connections 2024-03-18T01:01:06.469 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.460+0000 7f4c7ce0b640 1 -- 172.21.15.49:0/15880056 wait complete. 2024-03-18T01:01:06.470 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T01:01:06.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.582+0000 7facea43f640 1 -- 172.21.15.49:0/4168218047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7face4101da0 msgr2=0x7face41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.582+0000 7facea43f640 1 --2- 172.21.15.49:0/4168218047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7face4101da0 0x7face41021c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7facd4009860 tx=0x7facd402bb70 comp rx=0 tx=0).stop 2024-03-18T01:01:06.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.582+0000 7facea43f640 1 -- 172.21.15.49:0/4168218047 shutdown_connections 2024-03-18T01:01:06.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.582+0000 7facea43f640 1 --2- 172.21.15.49:0/4168218047 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7face4101320 0x7face4108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.582+0000 7facea43f640 1 --2- 172.21.15.49:0/4168218047 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face4100960 0x7face4100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.582+0000 7facea43f640 1 --2- 172.21.15.49:0/4168218047 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7face4101da0 0x7face41021c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.582+0000 7facea43f640 1 -- 172.21.15.49:0/4168218047 >> 172.21.15.49:0/4168218047 conn(0x7face40fb410 msgr2=0x7face40fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.584 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.582+0000 7facea43f640 1 -- 172.21.15.49:0/4168218047 shutdown_connections 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.582+0000 7facea43f640 1 -- 172.21.15.49:0/4168218047 wait complete. 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7facea43f640 1 Processor -- start 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7facea43f640 1 -- start start 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7facea43f640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face4100960 0x7face4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7facea43f640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7face4101320 0x7face4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7facea43f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7face4101da0 0x7face419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7facea43f640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7face410c330 con 0x7face4101da0 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7facea43f640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7face410c1b0 con 0x7face4101320 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7facea43f640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7face410c4b0 con 0x7face4100960 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7face89dc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7face4101da0 0x7face419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7face3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face4100960 0x7face4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7face89dc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7face4101da0 0x7face419f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34550/0 (socket says 172.21.15.49:34550) 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7face89dc640 1 -- 172.21.15.49:0/2776485901 learned_addr learned my addr 172.21.15.49:0/2776485901 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.585 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.583+0000 7face37fe640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7face4101320 0x7face4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7face89dc640 1 -- 172.21.15.49:0/2776485901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face4100960 msgr2=0x7face4197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7face89dc640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face4100960 0x7face4197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7face89dc640 1 -- 172.21.15.49:0/2776485901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7face4101320 msgr2=0x7face4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7face89dc640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7face4101320 0x7face4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7face89dc640 1 -- 172.21.15.49:0/2776485901 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7face419f770 con 0x7face4101da0 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7face89dc640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7face4101da0 0x7face419f100 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7facdc00eb10 tx=0x7facdc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7face17fa640 1 -- 172.21.15.49:0/2776485901 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc004b20 con 0x7face4101da0 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7face17fa640 1 -- 172.21.15.49:0/2776485901 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facdc004cc0 con 0x7face4101da0 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7facea43f640 1 -- 172.21.15.49:0/2776485901 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7face419fa60 con 0x7face4101da0 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7face17fa640 1 -- 172.21.15.49:0/2776485901 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facdc010760 con 0x7face4101da0 2024-03-18T01:01:06.586 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.584+0000 7facea43f640 1 -- 172.21.15.49:0/2776485901 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7face419ff60 con 0x7face4101da0 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7face17fa640 1 -- 172.21.15.49:0/2776485901 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7facdc005020 con 0x7face4101da0 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7faced020740 1 Processor -- start 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7faced020740 1 -- start start 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7faced020740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eaf850 0x557b00eafc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7faced020740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb0240 0x557b00eb4d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7faced020740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00eb7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7faced020740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557b00e14490 con 0x557b00eb53b0 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7faced020740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557b00dbcc90 con 0x557b00eb0240 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7faced020740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557b00e10530 con 0x557b00eaf850 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7face37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00eb7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.593 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7face37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00eb7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34566/0 (socket says 172.21.15.49:34566) 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7face37fe640 1 -- 172.21.15.49:0/581225413 learned_addr learned my addr 172.21.15.49:0/581225413 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7face3fff640 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb0240 0x557b00eb4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.587+0000 7face37fe640 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eaf850 0x557b00eafc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face17fa640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faccc0a2800 0x7faccc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face37fe640 1 -- 172.21.15.49:0/581225413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eaf850 msgr2=0x557b00eafc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face37fe640 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eaf850 0x557b00eafc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face37fe640 1 -- 172.21.15.49:0/581225413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb0240 msgr2=0x557b00eb4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face37fe640 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb0240 0x557b00eb4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face37fe640 1 -- 172.21.15.49:0/581225413 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b00e77500 con 0x557b00eb53b0 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face3fff640 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb0240 0x557b00eb4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face89dc640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faccc0a2800 0x7faccc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.594 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face17fa640 1 -- 172.21.15.49:0/2776485901 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7facdc0ceff0 con 0x7face4101da0 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7face37fe640 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00eb7840 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x557b00eb0aa0 tx=0x7facd80095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea30b9a2a13865e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7facca7fc640 1 -- 172.21.15.49:0/581225413 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd8005020 con 0x557b00eb53b0 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7facca7fc640 1 -- 172.21.15.49:0/581225413 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facd80047a0 con 0x557b00eb53b0 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.588+0000 7facca7fc640 1 -- 172.21.15.49:0/581225413 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd800f9f0 con 0x557b00eb53b0 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 -- 172.21.15.49:0/581225413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 msgr2=0x557b00eb7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00eb7840 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x557b00eb0aa0 tx=0x7facd80095b0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7face89dc640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faccc0a2800 0x7faccc0a4cc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7facdc098340 tx=0x7facdc01bb30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 -- 172.21.15.49:0/581225413 shutdown_connections 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb0240 0x557b00eb4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00eb7840 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 --2- 172.21.15.49:0/581225413 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eaf850 0x557b00eafc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 -- 172.21.15.49:0/581225413 >> 172.21.15.49:0/581225413 conn(0x557b00eaadc0 msgr2=0x557b00eab1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 -- 172.21.15.49:0/581225413 shutdown_connections 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 -- 172.21.15.49:0/581225413 wait complete. 2024-03-18T01:01:06.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 -- start start 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.589+0000 7faced020740 1 Processor -- start 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7faced020740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eb9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7faced020740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00eba2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7faced020740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 0x557b00ec1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7faced020740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x557b00eb8360 con 0x557b00eaf850 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7faced020740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x557b00dd0230 con 0x557b00eb53b0 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7faced020740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x557b00eb84e0 con 0x557b00eb0240 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7faced020740 10 client.0 _ll_register_callbacks cb 0x557b00e0c010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eb9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eb9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34578/0 (socket says 172.21.15.49:34578) 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 -- 172.21.15.49:0/3607299328 learned_addr learned my addr 172.21.15.49:0/3607299328 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face3fff640 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00eba2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 0x557b00ec1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 -- 172.21.15.49:0/3607299328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 msgr2=0x557b00eba2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00eba2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 -- 172.21.15.49:0/3607299328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 msgr2=0x557b00ec1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 0x557b00ec1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 -- 172.21.15.49:0/3607299328 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b00ddf100 con 0x557b00eaf850 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face3fff640 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00eba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7face37fe640 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eb9da0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7facd8004eb0 tx=0x7facd8005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66318004ded2ffde server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7facc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7facc37fe640 1 -- 172.21.15.49:0/3607299328 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd8030060 con 0x557b00eaf850 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7facc37fe640 1 -- 172.21.15.49:0/3607299328 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7facd8019040 con 0x557b00eaf850 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7faced020740 1 -- 172.21.15.49:0/3607299328 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facd80047a0 con 0x557b00eaf850 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7facc37fe640 1 -- 172.21.15.49:0/3607299328 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7facd80053b0 con 0x557b00eaf850 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.590+0000 7faced020740 10 client.18192 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7faced020740 1 -- 172.21.15.49:0/3607299328 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557b00b283f0 con 0x557b00eaf850 2024-03-18T01:01:06.597 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7facc37fe640 1 -- 172.21.15.49:0/3607299328 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facd80047a0 con 0x557b00eaf850 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7facc37fe640 1 -- 172.21.15.49:0/3607299328 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7facd801d030 con 0x557b00eaf850 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7faced020740 10 client.18192 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7faced020740 10 client.18192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7faced020740 20 client.18192 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7faced020740 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557b00ec5810 0x557b00ec7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7faced020740 4 client.18192 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7faced020740 1 -- 172.21.15.49:0/3607299328 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b00e219d0 con 0x557b00ec5810 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7face3fff640 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557b00ec5810 0x557b00ec7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7facc37fe640 10 client.18192 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.591+0000 7face3fff640 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557b00ec5810 0x557b00ec7c00 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43eaa399b38c2322 server_cookie=ce07db09135e4f6d in_seq=0 out_seq=0 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7facc37fe640 1 -- 172.21.15.49:0/3607299328 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557b00e219d0 con 0x557b00ec5810 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7facc37fe640 10 client.18192 handle_command_reply: tid=0 2024-03-18T01:01:06.598 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 client.18192 shutdown 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 -- 172.21.15.49:0/3607299328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 msgr2=0x557b00eb9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eb9da0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7facd8004eb0 tx=0x7facd8005a00 comp rx=0 tx=0).stop 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 -- 172.21.15.49:0/3607299328 shutdown_connections 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557b00ec5810 0x557b00ec7c00 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 0x557b00ec1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00eba2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 --2- 172.21.15.49:0/3607299328 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eb9da0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7facc37fe640 0 client.18192 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 -- 172.21.15.49:0/3607299328 >> 172.21.15.49:0/3607299328 conn(0x557b00eaadc0 msgr2=0x557b00ea9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 -- 172.21.15.49:0/3607299328 shutdown_connections 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.592+0000 7faced020740 1 -- 172.21.15.49:0/3607299328 wait complete. 2024-03-18T01:01:06.599 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.593+0000 7faced020740 20 client.18192 trim_cache size 0 max 16384 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7faced020740 1 Processor -- start 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7faced020740 1 -- start start 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7faced020740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eba240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7faced020740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00ec12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7faced020740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 0x557b00eae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7faced020740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7face41a8da0 con 0x557b00eaf850 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7faced020740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7face41aa940 con 0x557b00eb53b0 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7faced020740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7face41a9470 con 0x557b00eb0240 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eba240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eba240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34592/0 (socket says 172.21.15.49:34592) 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face37fe640 1 -- 172.21.15.49:0/878089616 learned_addr learned my addr 172.21.15.49:0/878089616 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face3fff640 1 --2- 172.21.15.49:0/878089616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00ec12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.611 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face37fe640 1 --2- 172.21.15.49:0/878089616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 0x557b00eae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face37fe640 1 -- 172.21.15.49:0/878089616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 msgr2=0x557b00ec12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face37fe640 1 --2- 172.21.15.49:0/878089616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00ec12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face37fe640 1 -- 172.21.15.49:0/878089616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 msgr2=0x557b00eae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face37fe640 1 --2- 172.21.15.49:0/878089616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 0x557b00eae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.606+0000 7face37fe640 1 -- 172.21.15.49:0/878089616 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b00eb1270 con 0x557b00eaf850 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7face37fe640 1 --2- 172.21.15.49:0/878089616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eba240 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7facd802e040 tx=0x7facd8011570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=983c4872da8231f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7facc9ffb640 1 -- 172.21.15.49:0/878089616 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd803a040 con 0x557b00eaf850 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7facc9ffb640 1 -- 172.21.15.49:0/878089616 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facd80047a0 con 0x557b00eaf850 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7facc9ffb640 1 -- 172.21.15.49:0/878089616 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd8030040 con 0x557b00eaf850 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7faced020740 1 -- 172.21.15.49:0/878089616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 msgr2=0x557b00eba240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7faced020740 1 --2- 172.21.15.49:0/878089616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eba240 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7facd802e040 tx=0x7facd8011570 comp rx=0 tx=0).stop 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7faced020740 1 -- 172.21.15.49:0/878089616 shutdown_connections 2024-03-18T01:01:06.612 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7faced020740 1 --2- 172.21.15.49:0/878089616 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eb53b0 0x557b00eae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7faced020740 1 --2- 172.21.15.49:0/878089616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00ec12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7faced020740 1 --2- 172.21.15.49:0/878089616 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eaf850 0x557b00eba240 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7faced020740 1 -- 172.21.15.49:0/878089616 >> 172.21.15.49:0/878089616 conn(0x557b00eaadc0 msgr2=0x557b00eae020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7faced020740 1 -- 172.21.15.49:0/878089616 shutdown_connections 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.607+0000 7faced020740 1 -- 172.21.15.49:0/878089616 wait complete. 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7faced020740 1 -- start start 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7faced020740 1 Processor -- start 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7faced020740 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eaf850 0x557b00ebe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7faced020740 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00ebec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7faced020740 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00ebfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7faced020740 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7face41aa240 con 0x557b00eb53b0 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7faced020740 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7face41a9b40 con 0x557b00eaf850 2024-03-18T01:01:06.613 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7faced020740 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7face41a6d80 con 0x557b00eb0240 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7faced020740 10 client.0 _ll_register_callbacks cb 0x7facdc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7face37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00ebfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7face37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00ebfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:34596/0 (socket says 172.21.15.49:34596) 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7face37fe640 1 -- 172.21.15.49:0/3130797476 learned_addr learned my addr 172.21.15.49:0/3130797476 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7face3fff640 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00ebec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.608+0000 7face37fe640 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eaf850 0x557b00ebe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7face37fe640 1 -- 172.21.15.49:0/3130797476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 msgr2=0x557b00ebec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7face37fe640 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00ebec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7face37fe640 1 -- 172.21.15.49:0/3130797476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eaf850 msgr2=0x557b00ebe6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7face37fe640 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eaf850 0x557b00ebe6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7face37fe640 1 -- 172.21.15.49:0/3130797476 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b00bf27d0 con 0x557b00eb53b0 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7face37fe640 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00ebfc20 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7facd8004fd0 tx=0x7facd8011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f9f3d3f609b75e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7facc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.49:3300/0 2024-03-18T01:01:06.614 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7facc37fe640 1 -- 172.21.15.49:0/3130797476 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facd8030070 con 0x557b00eb53b0 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7facc37fe640 1 -- 172.21.15.49:0/3130797476 <== mon.0 v2:172.21.15.49:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7facd802ba80 con 0x557b00eb53b0 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7faced020740 1 -- 172.21.15.49:0/3130797476 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7facd80047a0 con 0x557b00eb53b0 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7facc37fe640 1 -- 172.21.15.49:0/3130797476 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7facd8009970 con 0x557b00eb53b0 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7faced020740 10 client.18198 fetch_fsmap learned FSMap version 31 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7faced020740 1 -- 172.21.15.49:0/3130797476 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x557b00ec1340 con 0x557b00eb53b0 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7facc37fe640 1 -- 172.21.15.49:0/3130797476 <== mon.0 v2:172.21.15.49:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facd80047a0 con 0x557b00eb53b0 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7facc37fe640 1 -- 172.21.15.49:0/3130797476 <== mon.0 v2:172.21.15.49:3300/0 5 ==== fsmap(e 31) ==== 8362+0+0 (secure 0 0 0) 0x7facd801d070 con 0x557b00eb53b0 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7faced020740 10 client.18198 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7faced020740 10 client.18198 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7faced020740 20 client.18198 populate_metadata read hostname 'smithi049' 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7faced020740 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557b00f2bd50 0x557b00f2e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7faced020740 4 client.18198 mds_command: new command op to 24506 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.609+0000 7faced020740 1 -- 172.21.15.49:0/3130797476 --> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b00e1bd20 con 0x557b00f2bd50 2024-03-18T01:01:06.615 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7face3fff640 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557b00f2bd50 0x557b00f2e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7face3fff640 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557b00f2bd50 0x557b00f2e170 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d15e483a443a37a9 server_cookie=5e85a099b3d0a212 in_seq=0 out_seq=0 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7facc37fe640 10 client.18198 ms_handle_connect on v2:172.21.15.106:6838/2088471086 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7facc37fe640 1 -- 172.21.15.49:0/3130797476 <== mds.0 v2:172.21.15.106:6838/2088471086 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b00e1bd20 con 0x557b00f2bd50 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7facc37fe640 10 client.18198 handle_command_reply: tid=0 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 client.18198 shutdown 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 -- 172.21.15.49:0/3130797476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 msgr2=0x557b00ebfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00ebfc20 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7facd8004fd0 tx=0x7facd8011440 comp rx=0 tx=0).stop 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 -- 172.21.15.49:0/3130797476 shutdown_connections 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.106:6838/2088471086,v1:172.21.15.106:6839/2088471086] conn(0x557b00f2bd50 0x557b00f2e170 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x557b00eb53b0 0x557b00ebfc20 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x557b00eb0240 0x557b00ebec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7facc37fe640 0 client.18198 ms_handle_reset on v2:172.21.15.106:6838/2088471086 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 --2- 172.21.15.49:0/3130797476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x557b00eaf850 0x557b00ebe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.616 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 -- 172.21.15.49:0/3130797476 >> 172.21.15.49:0/3130797476 conn(0x557b00eaadc0 msgr2=0x557b00eba0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 -- 172.21.15.49:0/3130797476 shutdown_connections 2024-03-18T01:01:06.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.610+0000 7faced020740 1 -- 172.21.15.49:0/3130797476 wait complete. 2024-03-18T01:01:06.617 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7faced020740 20 client.18198 trim_cache size 0 max 16384 2024-03-18T01:01:06.617 INFO:teuthology.orchestra.run.smithi049.stdout:[] 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 -- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faccc0a2800 msgr2=0x7faccc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faccc0a2800 0x7faccc0a4cc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7facdc098340 tx=0x7facdc01bb30 comp rx=0 tx=0).stop 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 -- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7face4101da0 msgr2=0x7face419f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7face4101da0 0x7face419f100 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7facdc00eb10 tx=0x7facdc00c2d0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 -- 172.21.15.49:0/2776485901 shutdown_connections 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7faccc0a2800 0x7faccc0a4cc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7face4101320 0x7face4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7face4101da0 0x7face419f100 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 --2- 172.21.15.49:0/2776485901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face4100960 0x7face4197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:06.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 -- 172.21.15.49:0/2776485901 >> 172.21.15.49:0/2776485901 conn(0x7face40fb410 msgr2=0x7face40ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:06.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 -- 172.21.15.49:0/2776485901 shutdown_connections 2024-03-18T01:01:06.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:06.611+0000 7facea43f640 1 -- 172.21.15.49:0/2776485901 wait complete. 2024-03-18T01:01:07.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:07 smithi049 ceph-mon[32130]: pgmap v680: 97 pgs: 97 active+clean; 7.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 284 MiB/s rd, 12 MiB/s wr, 667 op/s 2024-03-18T01:01:07.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:07 smithi107 ceph-mon[36270]: pgmap v680: 97 pgs: 97 active+clean; 7.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 284 MiB/s rd, 12 MiB/s wr, 667 op/s 2024-03-18T01:01:07.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:07 smithi106 ceph-mon[35899]: pgmap v680: 97 pgs: 97 active+clean; 7.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 284 MiB/s rd, 12 MiB/s wr, 667 op/s 2024-03-18T01:01:07.622 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-18T01:01:07.622 INFO:tasks.fwd_scrub:done joining 2024-03-18T01:01:07.623 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-18T01:01:07.633 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:01:07.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.755+0000 7fa39e152640 1 -- 172.21.15.49:0/67152153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa398104c90 msgr2=0x7fa398107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:07.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.755+0000 7fa39e152640 1 --2- 172.21.15.49:0/67152153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa398104c90 0x7fa398107080 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fa38c009900 tx=0x7fa38c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:01:07.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.755+0000 7fa39e152640 1 -- 172.21.15.49:0/67152153 shutdown_connections 2024-03-18T01:01:07.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.755+0000 7fa39e152640 1 --2- 172.21.15.49:0/67152153 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa3981075c0 0x7fa3981099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.755+0000 7fa39e152640 1 --2- 172.21.15.49:0/67152153 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa398104c90 0x7fa398107080 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.755+0000 7fa39e152640 1 --2- 172.21.15.49:0/67152153 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa398102360 0x7fa398104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.755+0000 7fa39e152640 1 -- 172.21.15.49:0/67152153 >> 172.21.15.49:0/67152153 conn(0x7fa3980fbd10 msgr2=0x7fa3980fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:07.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.755+0000 7fa39e152640 1 -- 172.21.15.49:0/67152153 shutdown_connections 2024-03-18T01:01:07.755 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.755+0000 7fa39e152640 1 -- 172.21.15.49:0/67152153 wait complete. 2024-03-18T01:01:07.756 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa39e152640 1 Processor -- start 2024-03-18T01:01:07.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa39e152640 1 -- start start 2024-03-18T01:01:07.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa39e152640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa398102360 0x7fa398101cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:07.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa39e152640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa398104c90 0x7fa398100310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:07.757 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa39e152640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3981075c0 0x7fa398100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa39e152640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fa39810d4e0 con 0x7fa3981075c0 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa39e152640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fa39810d360 con 0x7fa398104c90 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa39e152640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fa39810d660 con 0x7fa398102360 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa397fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3981075c0 0x7fa398100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa3977fe640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa398102360 0x7fa398101cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa396ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa398104c90 0x7fa398100310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa397fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3981075c0 0x7fa398100850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58926/0 (socket says 172.21.15.49:58926) 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.756+0000 7fa397fff640 1 -- 172.21.15.49:0/1438863944 learned_addr learned my addr 172.21.15.49:0/1438863944 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa397fff640 1 -- 172.21.15.49:0/1438863944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa398102360 msgr2=0x7fa398101cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa397fff640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa398102360 0x7fa398101cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa397fff640 1 -- 172.21.15.49:0/1438863944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa398104c90 msgr2=0x7fa398100310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa397fff640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa398104c90 0x7fa398100310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa397fff640 1 -- 172.21.15.49:0/1438863944 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3981010e0 con 0x7fa3981075c0 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa396ffd640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa398104c90 0x7fa398100310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:07.758 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa397fff640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3981075c0 0x7fa398100850 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fa39000eb10 tx=0x7fa39000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:07.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa394ff9640 1 -- 172.21.15.49:0/1438863944 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa390004b20 con 0x7fa3981075c0 2024-03-18T01:01:07.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3981a1c50 con 0x7fa3981075c0 2024-03-18T01:01:07.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa394ff9640 1 -- 172.21.15.49:0/1438863944 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa390004cc0 con 0x7fa3981075c0 2024-03-18T01:01:07.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa394ff9640 1 -- 172.21.15.49:0/1438863944 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa390010760 con 0x7fa3981075c0 2024-03-18T01:01:07.759 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.757+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3981a2170 con 0x7fa3981075c0 2024-03-18T01:01:07.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.758+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa398068a20 con 0x7fa3981075c0 2024-03-18T01:01:07.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.760+0000 7fa394ff9640 1 -- 172.21.15.49:0/1438863944 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa390005000 con 0x7fa3981075c0 2024-03-18T01:01:07.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.760+0000 7fa394ff9640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3800a2800 0x7fa3800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:07.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.760+0000 7fa3977fe640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3800a2800 0x7fa3800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:07.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.760+0000 7fa394ff9640 1 -- 172.21.15.49:0/1438863944 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa3900cf500 con 0x7fa3981075c0 2024-03-18T01:01:07.760 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.761+0000 7fa3977fe640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3800a2800 0x7fa3800a4cc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa388009bd0 tx=0x7fa388005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:07.764 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.764+0000 7fa394ff9640 1 -- 172.21.15.49:0/1438863944 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa39009c580 con 0x7fa3981075c0 2024-03-18T01:01:07.951 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.952+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa3980623e0 con 0x7fa3981075c0 2024-03-18T01:01:07.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.953+0000 7fa394ff9640 1 -- 172.21.15.49:0/1438863944 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7fa39009be80 con 0x7fa3981075c0 2024-03-18T01:01:07.953 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T01:01:07.953 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:01:07.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3800a2800 msgr2=0x7fa3800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:07.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3800a2800 0x7fa3800a4cc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa388009bd0 tx=0x7fa388005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:07.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3981075c0 msgr2=0x7fa398100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3981075c0 0x7fa398100850 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fa39000eb10 tx=0x7fa39000c2d0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 shutdown_connections 2024-03-18T01:01:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fa3800a2800 0x7fa3800a4cc0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fa398104c90 0x7fa398100310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fa3981075c0 0x7fa398100850 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 --2- 172.21.15.49:0/1438863944 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa398102360 0x7fa398101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 >> 172.21.15.49:0/1438863944 conn(0x7fa3980fbd10 msgr2=0x7fa398102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 shutdown_connections 2024-03-18T01:01:07.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:07.956+0000 7fa39e152640 1 -- 172.21.15.49:0/1438863944 wait complete. 2024-03-18T01:01:07.967 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T00:48:04.316609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T00:48:05.271815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T01:01:07.967 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:01:08.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.079+0000 7f1bf7d52640 1 -- 172.21.15.49:0/627872425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01022b0 msgr2=0x7f1bf0108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.079+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/627872425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01022b0 0x7f1bf0108b40 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009ca0 tx=0x7f1be802c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.079+0000 7f1bf7d52640 1 -- 172.21.15.49:0/627872425 shutdown_connections 2024-03-18T01:01:08.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.079+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/627872425 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01022b0 0x7f1bf0108b40 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.081 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/627872425 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bf01018f0 0x7f1bf0101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/627872425 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bf0100700 0x7f1bf0100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 -- 172.21.15.49:0/627872425 >> 172.21.15.49:0/627872425 conn(0x7f1bf00fbc90 msgr2=0x7f1bf00fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 -- 172.21.15.49:0/627872425 shutdown_connections 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 -- 172.21.15.49:0/627872425 wait complete. 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 Processor -- start 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 -- start start 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bf0100700 0x7f1bf006a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01018f0 0x7f1bf019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bf01022b0 0x7f1bf019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf010c380 con 0x7f1bf01018f0 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf010c200 con 0x7f1bf0100700 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf7d52640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf010c500 con 0x7f1bf01022b0 2024-03-18T01:01:08.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf52ed640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01018f0 0x7f1bf019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf52ed640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01018f0 0x7f1bf019c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58944/0 (socket says 172.21.15.49:58944) 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf52ed640 1 -- 172.21.15.49:0/3078307458 learned_addr learned my addr 172.21.15.49:0/3078307458 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.080+0000 7f1bf5aee640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bf0100700 0x7f1bf006a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1bf62ef640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bf01022b0 0x7f1bf019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1bf52ed640 1 -- 172.21.15.49:0/3078307458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bf01022b0 msgr2=0x7f1bf019ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1bf52ed640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bf01022b0 0x7f1bf019ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1bf52ed640 1 -- 172.21.15.49:0/3078307458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bf0100700 msgr2=0x7f1bf006a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1bf52ed640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bf0100700 0x7f1bf006a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1bf52ed640 1 -- 172.21.15.49:0/3078307458 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf019f720 con 0x7f1bf01018f0 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1bf52ed640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01018f0 0x7f1bf019c7f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1be000e9f0 tx=0x7f1be000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1be6ffd640 1 -- 172.21.15.49:0/3078307458 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be000cd80 con 0x7f1bf01018f0 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1be6ffd640 1 -- 172.21.15.49:0/3078307458 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1be0005500 con 0x7f1bf01018f0 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf019f9b0 con 0x7f1bf01018f0 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1be6ffd640 1 -- 172.21.15.49:0/3078307458 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be0010690 con 0x7f1bf01018f0 2024-03-18T01:01:08.083 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.081+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf019ff00 con 0x7f1bf01018f0 2024-03-18T01:01:08.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.082+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf0107a20 con 0x7f1bf01018f0 2024-03-18T01:01:08.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.083+0000 7f1be6ffd640 1 -- 172.21.15.49:0/3078307458 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1be00046c0 con 0x7f1bf01018f0 2024-03-18T01:01:08.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.084+0000 7f1be6ffd640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bdc0a2880 0x7f1bdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.084+0000 7f1bf5aee640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bdc0a2880 0x7f1bdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.084+0000 7f1be6ffd640 1 -- 172.21.15.49:0/3078307458 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f1be00cee20 con 0x7f1bf01018f0 2024-03-18T01:01:08.085 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.085+0000 7f1bf5aee640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bdc0a2880 0x7f1bdc0a4d40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0101750 tx=0x7f1bec005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:08.088 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.089+0000 7f1be6ffd640 1 -- 172.21.15.49:0/3078307458 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1be009bea0 con 0x7f1bf01018f0 2024-03-18T01:01:08.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:08 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/1438863944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:01:08.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.271+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1bf00623e0 con 0x7f1bf01018f0 2024-03-18T01:01:08.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.272+0000 7f1be6ffd640 1 -- 172.21.15.49:0/3078307458 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f1be009b7a0 con 0x7f1bf01018f0 2024-03-18T01:01:08.271 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T01:01:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bdc0a2880 msgr2=0x7f1bdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.273 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bdc0a2880 0x7f1bdc0a4d40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1bf0101750 tx=0x7f1bec005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01018f0 msgr2=0x7f1bf019c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01018f0 0x7f1bf019c7f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1be000e9f0 tx=0x7f1be000edf0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 shutdown_connections 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1bdc0a2880 0x7f1bdc0a4d40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1bf01022b0 0x7f1bf019ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1bf01018f0 0x7f1bf019c7f0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 --2- 172.21.15.49:0/3078307458 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1bf0100700 0x7f1bf006a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 >> 172.21.15.49:0/3078307458 conn(0x7f1bf00fbc90 msgr2=0x7f1bf0104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 shutdown_connections 2024-03-18T01:01:08.274 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.274+0000 7f1bf7d52640 1 -- 172.21.15.49:0/3078307458 wait complete. 2024-03-18T01:01:08.283 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:08 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/1438863944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:01:08.284 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:01:08.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2712992198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4100960 msgr2=0x7f1ae4100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2712992198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4100960 0x7f1ae4100de0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1acc009900 tx=0x7f1acc02bc30 comp rx=0 tx=0).stop 2024-03-18T01:01:08.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2712992198 shutdown_connections 2024-03-18T01:01:08.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2712992198 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ae4101320 0x7f1ae4108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2712992198 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4100960 0x7f1ae4100de0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2712992198 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae4101d10 0x7f1ae4102130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2712992198 >> 172.21.15.49:0/2712992198 conn(0x7f1ae40fb410 msgr2=0x7f1ae40fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:08.399 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2712992198 shutdown_connections 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2712992198 wait complete. 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 Processor -- start 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.397+0000 7f1aea8cc640 1 -- start start 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1aea8cc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ae4100960 0x7f1ae4196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1aea8cc640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4101320 0x7f1ae4196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1aea8cc640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae4101d10 0x7f1ae419dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1aea8cc640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae410c3a0 con 0x7f1ae4101320 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1aea8cc640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae410c220 con 0x7f1ae4101d10 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1aea8cc640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae410c520 con 0x7f1ae4100960 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4101320 0x7f1ae4196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae3fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ae4100960 0x7f1ae4196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4101320 0x7f1ae4196f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58958/0 (socket says 172.21.15.49:58958) 2024-03-18T01:01:08.400 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae37fe640 1 -- 172.21.15.49:0/2800161638 learned_addr learned my addr 172.21.15.49:0/2800161638 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae8e69640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae4101d10 0x7f1ae419dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae3fff640 1 -- 172.21.15.49:0/2800161638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae4101d10 msgr2=0x7f1ae419dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae3fff640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae4101d10 0x7f1ae419dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae3fff640 1 -- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4101320 msgr2=0x7f1ae4196f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae3fff640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4101320 0x7f1ae4196f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae3fff640 1 -- 172.21.15.49:0/2800161638 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae419e660 con 0x7f1ae4100960 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae37fe640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4101320 0x7f1ae4196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae3fff640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ae4100960 0x7f1ae4196a40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400ead0 tx=0x7f1ad400ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.398+0000 7f1ae17fa640 1 -- 172.21.15.49:0/2800161638 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad400bb70 con 0x7f1ae4100960 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.399+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae40ff8b0 con 0x7f1ae4100960 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.399+0000 7f1ae17fa640 1 -- 172.21.15.49:0/2800161638 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad400bd10 con 0x7f1ae4100960 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.399+0000 7f1ae17fa640 1 -- 172.21.15.49:0/2800161638 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4017410 con 0x7f1ae4100960 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.399+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae40ffe50 con 0x7f1ae4100960 2024-03-18T01:01:08.401 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.399+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae4068a20 con 0x7f1ae4100960 2024-03-18T01:01:08.402 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.401+0000 7f1ae17fa640 1 -- 172.21.15.49:0/2800161638 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1ad401e020 con 0x7f1ae4100960 2024-03-18T01:01:08.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.401+0000 7f1ae17fa640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.401+0000 7f1ae37fe640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.401+0000 7f1ae17fa640 1 -- 172.21.15.49:0/2800161638 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f1ad40d0420 con 0x7f1ae4100960 2024-03-18T01:01:08.403 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.402+0000 7f1ae37fe640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4197f50 tx=0x7f1acc02cdb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:08.404 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.405+0000 7f1ae17fa640 1 -- 172.21.15.49:0/2800161638 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ad409d440 con 0x7f1ae4100960 2024-03-18T01:01:08.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:08 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/1438863944' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:01:08.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.592+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1ae4107560 con 0x7f1ae4100960 2024-03-18T01:01:08.592 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.593+0000 7f1ae17fa640 1 -- 172.21.15.49:0/2800161638 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f1ad409cd40 con 0x7f1ae4100960 2024-03-18T01:01:08.593 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 32 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1ad00a2800 msgr2=0x7f1ad00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4197f50 tx=0x7f1acc02cdb0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ae4100960 msgr2=0x7f1ae4196a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ae4100960 0x7f1ae4196a40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400ead0 tx=0x7f1ad400ef90 comp rx=0 tx=0).stop 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 shutdown_connections 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f1ae4101d10 0x7f1ae419dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f1ae4101320 0x7f1ae4196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 --2- 172.21.15.49:0/2800161638 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1ae4100960 0x7f1ae4196a40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.595 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 >> 172.21.15.49:0/2800161638 conn(0x7f1ae40fb410 msgr2=0x7f1ae4104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:08.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 shutdown_connections 2024-03-18T01:01:08.596 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.595+0000 7f1aea8cc640 1 -- 172.21.15.49:0/2800161638 wait complete. 2024-03-18T01:01:08.606 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:01:08.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.717+0000 7f62a43f4640 1 -- 172.21.15.49:0/3513088985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f629c0757a0 msgr2=0x7f629c073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.717+0000 7f62a43f4640 1 --2- 172.21.15.49:0/3513088985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f629c0757a0 0x7f629c073bf0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f6294009900 tx=0x7f629402bcd0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.717+0000 7f62a43f4640 1 -- 172.21.15.49:0/3513088985 shutdown_connections 2024-03-18T01:01:08.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.717+0000 7f62a43f4640 1 --2- 172.21.15.49:0/3513088985 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c0741c0 0x7f629c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.717+0000 7f62a43f4640 1 --2- 172.21.15.49:0/3513088985 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c1096a0 0x7f629c10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.717+0000 7f62a43f4640 1 --2- 172.21.15.49:0/3513088985 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f629c0757a0 0x7f629c073bf0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.719 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.717+0000 7f62a43f4640 1 -- 172.21.15.49:0/3513088985 >> 172.21.15.49:0/3513088985 conn(0x7f629c0fcd80 msgr2=0x7f629c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.717+0000 7f62a43f4640 1 -- 172.21.15.49:0/3513088985 shutdown_connections 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.717+0000 7f62a43f4640 1 -- 172.21.15.49:0/3513088985 wait complete. 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a43f4640 1 Processor -- start 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a43f4640 1 -- start start 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a43f4640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0741c0 0x7f629c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a43f4640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f629c0757a0 0x7f629c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a43f4640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c1096a0 0x7f629c1a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a43f4640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f629c10f650 con 0x7f629c0757a0 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a43f4640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f629c10f4d0 con 0x7f629c1096a0 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a43f4640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f629c10f7d0 con 0x7f629c0741c0 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a2190640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0741c0 0x7f629c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a198f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f629c0757a0 0x7f629c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.720 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a2190640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0741c0 0x7f629c199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:47654/0 (socket says 172.21.15.49:47654) 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a2190640 1 -- 172.21.15.49:0/2408424381 learned_addr learned my addr 172.21.15.49:0/2408424381 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a198f640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f629c0757a0 0x7f629c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58968/0 (socket says 172.21.15.49:58968) 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.718+0000 7f62a2991640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c1096a0 0x7f629c1a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f62a2190640 1 -- 172.21.15.49:0/2408424381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c1096a0 msgr2=0x7f629c1a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f62a2190640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c1096a0 0x7f629c1a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f62a2190640 1 -- 172.21.15.49:0/2408424381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f629c0757a0 msgr2=0x7f629c19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f62a2190640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f629c0757a0 0x7f629c19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f62a2190640 1 -- 172.21.15.49:0/2408424381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629c1a1960 con 0x7f629c0741c0 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f62a2190640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0741c0 0x7f629c199d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6294005ab0 tx=0x7f629402c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f629b7fe640 1 -- 172.21.15.49:0/2408424381 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294041070 con 0x7f629c0741c0 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f629c1a1bf0 con 0x7f629c0741c0 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f629b7fe640 1 -- 172.21.15.49:0/2408424381 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6294004450 con 0x7f629c0741c0 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f629b7fe640 1 -- 172.21.15.49:0/2408424381 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294034440 con 0x7f629c0741c0 2024-03-18T01:01:08.721 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.719+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f629c1a20e0 con 0x7f629c0741c0 2024-03-18T01:01:08.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.720+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f629c108460 con 0x7f629c0741c0 2024-03-18T01:01:08.722 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.721+0000 7f629b7fe640 1 -- 172.21.15.49:0/2408424381 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6294044070 con 0x7f629c0741c0 2024-03-18T01:01:08.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.722+0000 7f629b7fe640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62840a2800 0x7f62840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:08.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.722+0000 7f62a198f640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62840a2800 0x7f62840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:08.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.722+0000 7f629b7fe640 1 -- 172.21.15.49:0/2408424381 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f62940f0710 con 0x7f629c0741c0 2024-03-18T01:01:08.723 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.722+0000 7f62a198f640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62840a2800 0x7f62840a4cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f629c19b250 tx=0x7f6290005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:08.729 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.730+0000 7f629b7fe640 1 -- 172.21.15.49:0/2408424381 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f62940bd170 con 0x7f629c0741c0 2024-03-18T01:01:08.913 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.914+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f629c02d340 con 0x7f629c0741c0 2024-03-18T01:01:08.914 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.914+0000 7f629b7fe640 1 -- 172.21.15.49:0/2408424381 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f62940c1210 con 0x7f629c0741c0 2024-03-18T01:01:08.914 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 64 2024-03-18T01:01:08.916 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62840a2800 msgr2=0x7f62840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62840a2800 0x7f62840a4cc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f629c19b250 tx=0x7f6290005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0741c0 msgr2=0x7f629c199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0741c0 0x7f629c199d40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6294005ab0 tx=0x7f629402c6e0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 shutdown_connections 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f62840a2800 0x7f62840a4cc0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f629c1096a0 0x7f629c1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f629c0757a0 0x7f629c19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 --2- 172.21.15.49:0/2408424381 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f629c0741c0 0x7f629c199d40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 >> 172.21.15.49:0/2408424381 conn(0x7f629c0fcd80 msgr2=0x7f629c105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 shutdown_connections 2024-03-18T01:01:08.917 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:08.917+0000 7f62a43f4640 1 -- 172.21.15.49:0/2408424381 wait complete. 2024-03-18T01:01:08.927 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-18T01:01:09.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.036+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2841085668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b641022b0 msgr2=0x7f8b64108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:09.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.036+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2841085668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b641022b0 0x7f8b64108b40 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009900 tx=0x7f8b5402bc30 comp rx=0 tx=0).stop 2024-03-18T01:01:09.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.036+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2841085668 shutdown_connections 2024-03-18T01:01:09.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.036+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2841085668 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b641022b0 0x7f8b64108b40 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:09.038 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.036+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2841085668 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b641018f0 0x7f8b64101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.036+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2841085668 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b64100700 0x7f8b64100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.036+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2841085668 >> 172.21.15.49:0/2841085668 conn(0x7f8b640fbc70 msgr2=0x7f8b640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2841085668 shutdown_connections 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2841085668 wait complete. 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 Processor -- start 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 -- start start 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b64100700 0x7f8b64197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b641018f0 0x7f8b64198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b641022b0 0x7f8b6419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6410c380 con 0x7f8b64100700 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6410c200 con 0x7f8b641022b0 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b6b6c6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6410c500 con 0x7f8b641018f0 2024-03-18T01:01:09.039 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.037+0000 7f8b69462640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b64100700 0x7f8b64197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b69462640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b64100700 0x7f8b64197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:58990/0 (socket says 172.21.15.49:58990) 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b69462640 1 -- 172.21.15.49:0/2236863130 learned_addr learned my addr 172.21.15.49:0/2236863130 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b68c61640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b641018f0 0x7f8b64198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b69c63640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b641022b0 0x7f8b6419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b69462640 1 -- 172.21.15.49:0/2236863130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b641018f0 msgr2=0x7f8b64198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b69462640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b641018f0 0x7f8b64198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b69462640 1 -- 172.21.15.49:0/2236863130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b641022b0 msgr2=0x7f8b6419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b69462640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b641022b0 0x7f8b6419f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b69462640 1 -- 172.21.15.49:0/2236863130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6419f730 con 0x7f8b64100700 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b69462640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b64100700 0x7f8b64197b10 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009d20 tx=0x7f8b5c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b627fc640 1 -- 172.21.15.49:0/2236863130 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c010630 con 0x7f8b64100700 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b627fc640 1 -- 172.21.15.49:0/2236863130 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b5c010cb0 con 0x7f8b64100700 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b6419fa20 con 0x7f8b64100700 2024-03-18T01:01:09.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b627fc640 1 -- 172.21.15.49:0/2236863130 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c018de0 con 0x7f8b64100700 2024-03-18T01:01:09.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.038+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b6419ff70 con 0x7f8b64100700 2024-03-18T01:01:09.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.039+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b641a0140 con 0x7f8b64100700 2024-03-18T01:01:09.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.040+0000 7f8b627fc640 1 -- 172.21.15.49:0/2236863130 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8b5c01f020 con 0x7f8b64100700 2024-03-18T01:01:09.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.041+0000 7f8b627fc640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8b4c0a2800 0x7f8b4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:09.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.041+0000 7f8b68c61640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8b4c0a2800 0x7f8b4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:09.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.041+0000 7f8b627fc640 1 -- 172.21.15.49:0/2236863130 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f8b5c0ceef0 con 0x7f8b64100700 2024-03-18T01:01:09.042 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.041+0000 7f8b68c61640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8b4c0a2800 0x7f8b4c0a4cc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8b64199020 tx=0x7f8b58005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:09.045 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.045+0000 7f8b627fc640 1 -- 172.21.15.49:0/2236863130 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8b5c09bf70 con 0x7f8b64100700 2024-03-18T01:01:09.282 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.282+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f8b641a03b0 con 0x7f8b64100700 2024-03-18T01:01:09.290 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.291+0000 7f8b627fc640 1 -- 172.21.15.49:0/2236863130 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+739829 (secure 0 0 0) 0x7f8b5c09b870 con 0x7f8b64100700 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "epoch": 18, 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "active_gid": 14150, 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "active_name": "x", 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "active_addrs": { 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.301 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6800", 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1246732627 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v1", 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:6801", 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1246732627 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "active_addr": "172.21.15.49:6801/1246732627", 2024-03-18T01:01:09.302 INFO:teuthology.orchestra.run.smithi049.stdout: "active_change": "2024-03-18T00:41:01.376865+0000", 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "active_mgr_features": 4540701547738038271, 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "available": true, 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "standbys": [ 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "gid": 24106, 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "z", 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_features": 4540701547738038271, 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "available_modules": [ 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "alerts", 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:09.303 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.304 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.305 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.306 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.307 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.308 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_destination": { 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_destination", 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.309 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Email address to send alerts to", 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_from_name": { 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_from_name", 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.310 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "Ceph", 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Email From: name", 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_host": { 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_host", 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.311 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SMTP server", 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_password": { 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_password", 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.312 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Password to authenticate with", 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_port": { 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_port", 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.313 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "465", 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SMTP port", 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_sender": { 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_sender", 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.314 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SMTP envelope sender", 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_ssl": { 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_ssl", 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.315 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_user": { 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_user", 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.316 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "User to authenticate as", 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.317 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "balancer", 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "active": { 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "active", 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T01:01:09.318 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "begin_time": { 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "begin_time", 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0000", 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T01:01:09.319 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "begin_weekday": { 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "begin_weekday", 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "0", 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "6", 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:01:09.320 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_compat_max_iterations": { 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crush_compat_max_iterations", 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "25", 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "250", 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.321 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_compat_metrics": { 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crush_compat_metrics", 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.322 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_compat_step": { 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crush_compat_step", 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0.5", 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "0.001", 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "0.999", 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.323 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "aggressiveness of optimization", 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "end_time": { 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "end_time", 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "2359", 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.324 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "end_weekday": { 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "end_weekday", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "0", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "6", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:01:09.325 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.326 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.327 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.328 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "min_score": { 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "min_score", 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.331 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": { 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mode", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "upmap", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-compat", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "none", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "read", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap", 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap-read" 2024-03-18T01:01:09.332 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Balancer mode", 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_ids": { 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pool_ids", 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.333 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.334 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap_max_deviation": { 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "upmap_max_deviation", 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.335 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap_max_optimizations": { 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "upmap_max_optimizations", 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.336 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cephadm", 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "agent_down_multiplier": { 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "agent_down_multiplier", 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.337 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "3.0", 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "agent_refresh_rate": { 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "agent_refresh_rate", 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.338 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "20", 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.342 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "agent_starting_port": { 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "agent_starting_port", 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "4721", 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.343 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_ptrace": { 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "allow_ptrace", 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "autotune_interval": { 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "autotune_interval", 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.345 INFO:teuthology.orchestra.run.smithi049.stdout: "autotune_memory_target_ratio": { 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0.7", 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.346 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm_log_destination": { 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cephadm_log_destination", 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "file", 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "file,syslog", 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "syslog" 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.347 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "cgroups_split": { 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cgroups_split", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.348 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "config_checks_enabled": { 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "config_checks_enabled", 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.349 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "config_dashboard": { 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "config_dashboard", 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.350 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_alertmanager": { 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_alertmanager", 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_base": { 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_base", 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Container image name, without the tag", 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_elasticsearch": { 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_elasticsearch", 2024-03-18T01:01:09.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "elasticsearch container image", 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_grafana": { 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_grafana", 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.353 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_haproxy": { 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_haproxy", 2024-03-18T01:01:09.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "HAproxy container image", 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_jaeger_agent": { 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_jaeger_agent", 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.355 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Jaeger agent container image", 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_jaeger_collector": { 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_jaeger_collector", 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.356 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Jaeger collector container image", 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_jaeger_query": { 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_jaeger_query", 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.357 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Jaeger query container image", 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_keepalived": { 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_keepalived", 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.358 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Keepalived container image", 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_loki": { 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_loki", 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.359 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Loki container image", 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_node_exporter": { 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_node_exporter", 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.360 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_nvmeof": { 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_nvmeof", 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.361 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Nvme-of container image", 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_prometheus": { 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_prometheus", 2024-03-18T01:01:09.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_promtail": { 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_promtail", 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.363 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Promtail container image", 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_snmp_gateway": { 2024-03-18T01:01:09.364 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_snmp_gateway", 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SNMP Gateway container image", 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "container_init": { 2024-03-18T01:01:09.365 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_init", 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "daemon_cache_timeout": { 2024-03-18T01:01:09.366 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "daemon_cache_timeout", 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "default_cephadm_command_timeout": { 2024-03-18T01:01:09.367 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "900", 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "default_registry": { 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default_registry", 2024-03-18T01:01:09.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io", 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "device_cache_timeout": { 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "device_cache_timeout", 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.369 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "1800", 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "seconds to cache device inventory", 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "device_enhanced_scan": { 2024-03-18T01:01:09.370 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "device_enhanced_scan", 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "facts_cache_timeout": { 2024-03-18T01:01:09.371 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "facts_cache_timeout", 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "seconds to cache host facts data", 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "host_check_interval": { 2024-03-18T01:01:09.372 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "host_check_interval", 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to perform a host check", 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "hw_monitoring": { 2024-03-18T01:01:09.373 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "hw_monitoring", 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "inventory_list_all": { 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "inventory_list_all", 2024-03-18T01:01:09.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.375 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.376 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "log_refresh_metadata": { 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_refresh_metadata", 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.377 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.378 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.379 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.380 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T01:01:09.381 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T01:01:09.382 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "*", 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "max_count_per_host": { 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_count_per_host", 2024-03-18T01:01:09.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "max number of daemons per service per host", 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd_draining_count": { 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_osd_draining_count", 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.384 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "migration_current": { 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "migration_current", 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.385 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "internal - do not modify", 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": { 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mode", 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.386 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "root", 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm-package", 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "root" 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "oob_default_addr": { 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "oob_default_addr", 2024-03-18T01:01:09.387 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "169.254.1.1", 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "prometheus_alerts_path": { 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "prometheus_alerts_path", 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.388 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_insecure": { 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_insecure", 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.389 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_password": { 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_password", 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.390 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_url": { 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_url", 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.391 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_username": { 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_username", 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.392 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "secure_monitoring_stack": { 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "secure_monitoring_stack", 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.393 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "service_discovery_port": { 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "service_discovery_port", 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.394 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8765", 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "cephadm service discovery port", 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "ssh_config_file": { 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssh_config_file", 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.395 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "use_agent": { 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "use_agent", 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.396 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "use_repo_digest": { 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "use_repo_digest", 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.397 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_on_failed_host_check": { 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_on_failed_host_check", 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.398 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_on_stray_daemons": { 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_on_stray_daemons", 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.399 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_on_stray_hosts": { 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_on_stray_hosts", 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.400 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crash", 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.401 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.402 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.403 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.404 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.405 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.406 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "retain_interval": { 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "retain_interval", 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "31536000", 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.407 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_recent_interval": { 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_recent_interval", 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "1209600", 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.408 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "dashboard", 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.409 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.410 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T01:01:09.411 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "AUDIT_API_ENABLED": { 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T01:01:09.412 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T01:01:09.413 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T01:01:09.414 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T01:01:09.415 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T01:01:09.416 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T01:01:09.417 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T01:01:09.418 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.419 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.420 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.421 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.422 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.423 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "admin", 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.424 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_URL": { 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_URL", 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.425 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.426 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "admin", 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.427 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.428 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.429 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.430 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.431 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "{}", 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T01:01:09.432 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T01:01:09.433 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T01:01:09.434 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T01:01:09.435 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T01:01:09.436 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.437 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T01:01:09.438 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.439 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T01:01:09.440 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.441 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.442 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.443 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.444 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8", 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.445 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "45", 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.446 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T01:01:09.447 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "admin", 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.448 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T01:01:09.449 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T01:01:09.450 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T01:01:09.451 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T01:01:09.452 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.453 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "cross_origin_url": { 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cross_origin_url", 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.454 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "crt_file": { 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crt_file", 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.455 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "debug": { 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "debug", 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.456 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Enable/disable debug options", 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "jwt_token_ttl": { 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "jwt_token_ttl", 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "28800", 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "key_file": { 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "key_file", 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.458 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.459 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.460 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.461 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.462 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.463 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "motd": { 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "motd", 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.464 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "The message of the day", 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_resolve_ip_addr": { 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.465 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "server_addr": { 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_addr", 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.466 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "::", 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.467 INFO:teuthology.orchestra.run.smithi049.stdout: "server_port": { 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_port", 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8080", 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "ssl": { 2024-03-18T01:01:09.468 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssl", 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "ssl_server_port": { 2024-03-18T01:01:09.469 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssl_server_port", 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8443", 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.470 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_behaviour": { 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_behaviour", 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "redirect", 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect" 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.471 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_error_status_code": { 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_error_status_code", 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "500", 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "400", 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "599", 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.472 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "url_prefix": { 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "url_prefix", 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.473 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "devicehealth", 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: "enable_monitoring": { 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enable_monitoring", 2024-03-18T01:01:09.474 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "monitor device health metrics", 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.475 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.476 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.477 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.478 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.479 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "mark_out_threshold": { 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mark_out_threshold", 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.480 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "2419200", 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": { 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pool_name", 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.481 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "device_health_metrics", 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "retention_period": { 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "retention_period", 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.482 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "15552000", 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how long to retain device health metrics", 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "scrape_frequency": { 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "scrape_frequency", 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "86400", 2024-03-18T01:01:09.483 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "self_heal": { 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "self_heal", 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.484 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.485 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_threshold": { 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_threshold", 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.486 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "7257600", 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "diskprediction_local", 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.487 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.488 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.489 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.490 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.491 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "predict_interval": { 2024-03-18T01:01:09.492 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "predict_interval", 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "86400", 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "predictor_model": { 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "predictor_model", 2024-03-18T01:01:09.493 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "prophetstor", 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:09.494 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.495 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "influx", 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": false, 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "influxdb python module not found", 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "batch_size": { 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "batch_size", 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5000", 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.496 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "database": { 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "database", 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "ceph", 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.497 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "hostname": { 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "hostname", 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.498 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "InfluxDB server hostname", 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "30", 2024-03-18T01:01:09.499 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "5", 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.500 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.502 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.503 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.504 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "password": { 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "password", 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.505 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "password of InfluxDB server user", 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "port": { 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "port", 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.506 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8086", 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "InfluxDB server port", 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "ssl": { 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssl", 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.507 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "false", 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "threads": { 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "threads", 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.508 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "32", 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "username": { 2024-03-18T01:01:09.509 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "username", 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "username of InfluxDB server user", 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.510 INFO:teuthology.orchestra.run.smithi049.stdout: "verify_ssl": { 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "verify_ssl", 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "true", 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:01:09.511 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "insights", 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.512 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.513 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.514 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.515 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.516 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.517 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "iostat", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.518 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.519 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.520 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.521 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.522 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "k8sevents", 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_event_retention_days": { 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ceph_event_retention_days", 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.523 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "7", 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "config_check_secs": { 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "config_check_secs", 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.524 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "10", 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.525 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.526 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.527 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.528 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.529 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.530 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "localpool", 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "failure_domain": { 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "failure_domain", 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.531 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "host", 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "failure domain for any created local pool", 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.532 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.533 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.534 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.535 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.536 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": { 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "min_size", 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.537 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "default min_size for any created local pool", 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "num_rep": { 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "num_rep", 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.538 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "3", 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "default replica count for any created local pool", 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": { 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pg_num", 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.539 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "128", 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "prefix": { 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "prefix", 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.540 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "name prefix for any created local pool", 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "subtree": { 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "subtree", 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.541 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "rack", 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.542 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mds_autoscaler", 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.543 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.544 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.545 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.546 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.547 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mirroring", 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.548 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.549 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.550 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.551 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.552 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "nfs", 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.554 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.555 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.556 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.557 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.558 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "orchestrator", 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "fail_fs": { 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "fail_fs", 2024-03-18T01:01:09.559 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.560 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.561 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.562 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.563 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.565 INFO:teuthology.orchestra.run.smithi049.stdout: "orchestrator": { 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "orchestrator", 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm", 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "rook", 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "test_orchestrator" 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.566 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Orchestrator backend", 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "osd_perf_query", 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.567 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.568 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.569 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.570 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.571 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.572 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "osd_support", 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.573 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.574 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.575 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.576 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.577 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pg_autoscaler", 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.578 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.579 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.580 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.581 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.582 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.583 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.584 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "threshold": { 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "threshold", 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "3.0", 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1.0", 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "scaling threshold", 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.585 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "progress", 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_pg_recovery_event": { 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "allow_pg_recovery_event", 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.586 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": { 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enabled", 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.587 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.588 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.589 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.591 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.592 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "max_completed_events": { 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_completed_events", 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "50", 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.593 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "number of past completed events to remember", 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.594 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how long the module is going to sleep", 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "prometheus", 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "cache": { 2024-03-18T01:01:09.595 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cache", 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "exclude_perf_counters": { 2024-03-18T01:01:09.596 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "exclude_perf_counters", 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.597 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.598 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.599 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.600 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.601 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.602 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_stats_pools": { 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_stats_pools", 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "300", 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "scrape_interval": { 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "scrape_interval", 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "15.0", 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "server_addr": { 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_addr", 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "::", 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.606 INFO:teuthology.orchestra.run.smithi049.stdout: "server_port": { 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_port", 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "9283", 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "stale_cache_strategy": { 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "stale_cache_strategy", 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "log", 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.608 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_behaviour": { 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_behaviour", 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "default", 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "default", 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "error" 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_error_status_code": { 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_error_status_code", 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "500", 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "400", 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "599", 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.610 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_support", 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.611 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.612 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.613 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.614 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.615 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "max_concurrent_snap_create": { 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_concurrent_snap_create", 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.616 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "mirror_snapshot_schedule": { 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.617 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "trash_purge_schedule": { 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "trash_purge_schedule", 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.618 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "restful", 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.619 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "enable_auth": { 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enable_auth", 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.620 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "key_file": { 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "key_file", 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.621 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.622 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.622 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.622 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.622 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.622 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.622 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.622 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.622 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.622 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.623 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.624 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.625 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.626 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "server_addr": { 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_addr", 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.627 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "server_port": { 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_port", 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.628 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rgw", 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.629 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.630 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.631 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.632 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.633 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rook", 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.634 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.635 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.636 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.637 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.638 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": { 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "storage_class", 2024-03-18T01:01:09.639 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "local", 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.640 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "selftest", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.641 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.642 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.643 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.644 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.645 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "roption1": { 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "roption1", 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.646 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "roption2": { 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "roption2", 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "xyz", 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.647 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption1": { 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption1", 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.648 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption2": { 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption2", 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.649 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption3": { 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption3", 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.650 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption4": { 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption4", 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.651 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption5": { 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption5", 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.652 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption6": { 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption6", 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.653 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption7": { 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption7", 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "42", 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.654 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "testkey": { 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "testkey", 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.655 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "testlkey": { 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "testlkey", 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.656 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "testnewline": { 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "testnewline", 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.657 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "snap_schedule", 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_m_granularity": { 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "allow_m_granularity", 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.658 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "dump_on_update": { 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "dump_on_update", 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.659 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "dump database to debug log on update", 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.660 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.661 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.662 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.663 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.664 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.665 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "stats", 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.666 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.667 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.668 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.669 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.670 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "status", 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.671 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.672 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.673 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.674 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.675 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "telegraf", 2024-03-18T01:01:09.676 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "address": { 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "address", 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.677 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "15", 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.678 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.679 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.680 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.681 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.682 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.683 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "telemetry", 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_basic": { 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_basic", 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.684 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_crash": { 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_crash", 2024-03-18T01:01:09.685 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_device": { 2024-03-18T01:01:09.686 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_device", 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_ident": { 2024-03-18T01:01:09.687 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_ident", 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.688 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_perf": { 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_perf", 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "contact": { 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "contact", 2024-03-18T01:01:09.689 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "description": { 2024-03-18T01:01:09.690 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "description", 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.691 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "device_url": { 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "device_url", 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.692 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": { 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enabled", 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.693 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "24", 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "8", 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.694 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "last_opt_revision": { 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "last_opt_revision", 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "1", 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.695 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "leaderboard": { 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "leaderboard", 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.696 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "leaderboard_description": { 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "leaderboard_description", 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.697 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.698 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.699 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.700 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.701 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.702 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "organization": { 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "organization", 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.703 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "proxy": { 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "proxy", 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.704 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "url": { 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "url", 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.705 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "test_orchestrator", 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.706 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.707 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.708 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.709 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.710 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.711 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "volumes", 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.712 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.713 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.714 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.715 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.716 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "max_concurrent_clones": { 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_concurrent_clones", 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "4", 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T01:01:09.717 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "periodic_async_work": { 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "periodic_async_work", 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.718 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Periodically check for async work", 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "snapshot_clone_delay": { 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "snapshot_clone_delay", 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:09.719 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "snapshot_clone_no_wait": { 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.720 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix", 2024-03-18T01:01:09.721 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "discovery_interval": { 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "discovery_interval", 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "100", 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.722 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "identifier": { 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "identifier", 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.723 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.724 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.725 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.726 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.727 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.728 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.729 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "zabbix_host": { 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix_host", 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.730 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "zabbix_port": { 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix_port", 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10051", 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.731 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "zabbix_sender": { 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix_sender", 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.732 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: "gid": 24113, 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "y", 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: "mgr_features": 4540701547738038271, 2024-03-18T01:01:09.733 INFO:teuthology.orchestra.run.smithi049.stdout: "available_modules": [ 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "alerts", 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.734 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.735 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.736 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.737 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.738 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.739 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.740 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_destination": { 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_destination", 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Email address to send alerts to", 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.741 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_from_name": { 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_from_name", 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "Ceph", 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Email From: name", 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.742 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_host": { 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_host", 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SMTP server", 2024-03-18T01:01:09.743 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_password": { 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_password", 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Password to authenticate with", 2024-03-18T01:01:09.744 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_port": { 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_port", 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "465", 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SMTP port", 2024-03-18T01:01:09.745 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_sender": { 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_sender", 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SMTP envelope sender", 2024-03-18T01:01:09.746 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_ssl": { 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_ssl", 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T01:01:09.747 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.748 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.749 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_user": { 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_user", 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "User to authenticate as", 2024-03-18T01:01:09.750 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "balancer", 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "active": { 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "active", 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.751 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "begin_time": { 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "begin_time", 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.752 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0000", 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "begin_weekday": { 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "begin_weekday", 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:09.753 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "0", 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "6", 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_compat_max_iterations": { 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crush_compat_max_iterations", 2024-03-18T01:01:09.754 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "25", 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "250", 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_compat_metrics": { 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crush_compat_metrics", 2024-03-18T01:01:09.755 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_compat_step": { 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crush_compat_step", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.756 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0.5", 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "0.001", 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "0.999", 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "aggressiveness of optimization", 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "end_time": { 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "end_time", 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.757 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "2359", 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "end_weekday": { 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "end_weekday", 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.758 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "0", 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "6", 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.759 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.760 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.761 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.762 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.763 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "min_score": { 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "min_score", 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.764 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": { 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mode", 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "upmap", 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.765 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-compat", 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "none", 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "read", 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap", 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap-read" 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Balancer mode", 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_ids": { 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pool_ids", 2024-03-18T01:01:09.766 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.767 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap_max_deviation": { 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "upmap_max_deviation", 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.768 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap_max_optimizations": { 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "upmap_max_optimizations", 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.769 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cephadm", 2024-03-18T01:01:09.770 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "agent_down_multiplier": { 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "agent_down_multiplier", 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "3.0", 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.771 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "agent_refresh_rate": { 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "agent_refresh_rate", 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "20", 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T01:01:09.772 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "agent_starting_port": { 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "agent_starting_port", 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "4721", 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.773 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_ptrace": { 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "allow_ptrace", 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T01:01:09.774 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "autotune_interval": { 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "autotune_interval", 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T01:01:09.775 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "autotune_memory_target_ratio": { 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0.7", 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T01:01:09.776 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm_log_destination": { 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cephadm_log_destination", 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "file", 2024-03-18T01:01:09.777 INFO:teuthology.orchestra.run.smithi049.stdout: "file,syslog", 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "syslog" 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "cgroups_split": { 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cgroups_split", 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.778 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "config_checks_enabled": { 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "config_checks_enabled", 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.779 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "config_dashboard": { 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "config_dashboard", 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.780 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_alertmanager": { 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_alertmanager", 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.781 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_base": { 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_base", 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.782 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Container image name, without the tag", 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_elasticsearch": { 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_elasticsearch", 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.783 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "elasticsearch container image", 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_grafana": { 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_grafana", 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.784 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_haproxy": { 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_haproxy", 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.785 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "HAproxy container image", 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_jaeger_agent": { 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_jaeger_agent", 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.786 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Jaeger agent container image", 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_jaeger_collector": { 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_jaeger_collector", 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.787 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Jaeger collector container image", 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_jaeger_query": { 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_jaeger_query", 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.788 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Jaeger query container image", 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_keepalived": { 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_keepalived", 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.789 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Keepalived container image", 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_loki": { 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_loki", 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.790 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Loki container image", 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_node_exporter": { 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_node_exporter", 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.791 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_nvmeof": { 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_nvmeof", 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.792 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Nvme-of container image", 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_prometheus": { 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_prometheus", 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.793 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_promtail": { 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_promtail", 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T01:01:09.794 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Promtail container image", 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_snmp_gateway": { 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_snmp_gateway", 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T01:01:09.795 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SNMP Gateway container image", 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "container_init": { 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_init", 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.796 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "daemon_cache_timeout": { 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "daemon_cache_timeout", 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.797 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "default_cephadm_command_timeout": { 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "900", 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.798 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "default_registry": { 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default_registry", 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io", 2024-03-18T01:01:09.799 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "device_cache_timeout": { 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "device_cache_timeout", 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.800 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "1800", 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "seconds to cache device inventory", 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "device_enhanced_scan": { 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "device_enhanced_scan", 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.801 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "facts_cache_timeout": { 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "facts_cache_timeout", 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.802 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "seconds to cache host facts data", 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "host_check_interval": { 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "host_check_interval", 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.803 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to perform a host check", 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "hw_monitoring": { 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "hw_monitoring", 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.804 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "inventory_list_all": { 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "inventory_list_all", 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.805 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.806 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.807 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "log_refresh_metadata": { 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_refresh_metadata", 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.808 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.809 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.810 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.811 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.812 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.813 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "*", 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T01:01:09.814 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "max_count_per_host": { 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_count_per_host", 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "max number of daemons per service per host", 2024-03-18T01:01:09.815 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd_draining_count": { 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_osd_draining_count", 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T01:01:09.816 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "migration_current": { 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "migration_current", 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "internal - do not modify", 2024-03-18T01:01:09.817 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": { 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mode", 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "root", 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm-package", 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: "root" 2024-03-18T01:01:09.818 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "oob_default_addr": { 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "oob_default_addr", 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "169.254.1.1", 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.819 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "prometheus_alerts_path": { 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "prometheus_alerts_path", 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.820 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_insecure": { 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_insecure", 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.821 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_password": { 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_password", 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.822 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_url": { 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_url", 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.823 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_username": { 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_username", 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.824 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "secure_monitoring_stack": { 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "secure_monitoring_stack", 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.825 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "service_discovery_port": { 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "service_discovery_port", 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8765", 2024-03-18T01:01:09.826 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "cephadm service discovery port", 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "ssh_config_file": { 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssh_config_file", 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.827 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "use_agent": { 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "use_agent", 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.828 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "use_repo_digest": { 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "use_repo_digest", 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.829 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_on_failed_host_check": { 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_on_failed_host_check", 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.830 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_on_stray_daemons": { 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_on_stray_daemons", 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.831 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_on_stray_hosts": { 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_on_stray_hosts", 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.832 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crash", 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.833 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.834 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.835 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.836 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.837 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.838 INFO:teuthology.orchestra.run.smithi049.stdout: "retain_interval": { 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "retain_interval", 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "31536000", 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.839 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_recent_interval": { 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_recent_interval", 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "1209600", 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.840 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "dashboard", 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.841 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.842 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T01:01:09.843 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.844 INFO:teuthology.orchestra.run.smithi049.stdout: "AUDIT_API_ENABLED": { 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.845 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.846 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.847 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.848 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.849 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.850 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.851 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.852 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.853 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.854 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.855 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "admin", 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.856 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.857 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_URL": { 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_URL", 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.858 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "admin", 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.859 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.860 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.861 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.862 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.863 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "{}", 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.864 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.865 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.866 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.867 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.868 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.869 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.870 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.871 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.872 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.873 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.874 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.875 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.876 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8", 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.877 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "45", 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.878 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.879 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "admin", 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.880 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.881 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.882 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.883 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:09.884 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.885 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.886 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "cross_origin_url": { 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cross_origin_url", 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.887 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "crt_file": { 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crt_file", 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.888 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "debug": { 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "debug", 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.889 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Enable/disable debug options", 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "jwt_token_ttl": { 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "jwt_token_ttl", 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "28800", 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.890 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "key_file": { 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "key_file", 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.891 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.892 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.893 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.894 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.895 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.896 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "motd": { 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "motd", 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "The message of the day", 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.897 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_resolve_ip_addr": { 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.898 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "server_addr": { 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_addr", 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "::", 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.899 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "server_port": { 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_port", 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8080", 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.900 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "ssl": { 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssl", 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.901 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "ssl_server_port": { 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssl_server_port", 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8443", 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.902 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_behaviour": { 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_behaviour", 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "redirect", 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.903 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect" 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_error_status_code": { 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_error_status_code", 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "500", 2024-03-18T01:01:09.904 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "400", 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "599", 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "url_prefix": { 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "url_prefix", 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.905 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "devicehealth", 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.906 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "enable_monitoring": { 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enable_monitoring", 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "monitor device health metrics", 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.907 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.908 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.909 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.910 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.911 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.912 INFO:teuthology.orchestra.run.smithi049.stdout: "mark_out_threshold": { 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mark_out_threshold", 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "2419200", 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": { 2024-03-18T01:01:09.913 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pool_name", 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "device_health_metrics", 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "retention_period": { 2024-03-18T01:01:09.914 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "retention_period", 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "15552000", 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how long to retain device health metrics", 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "scrape_frequency": { 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "scrape_frequency", 2024-03-18T01:01:09.915 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "86400", 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "self_heal": { 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "self_heal", 2024-03-18T01:01:09.916 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:09.917 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.918 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_threshold": { 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_threshold", 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "7257600", 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.919 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "diskprediction_local", 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.920 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.922 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.923 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.924 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "predict_interval": { 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "predict_interval", 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "86400", 2024-03-18T01:01:09.925 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "predictor_model": { 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "predictor_model", 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.926 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "prophetstor", 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.927 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "influx", 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": false, 2024-03-18T01:01:09.928 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "influxdb python module not found", 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "batch_size": { 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "batch_size", 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5000", 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.929 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "database": { 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "database", 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "ceph", 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "hostname": { 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "hostname", 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "InfluxDB server hostname", 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.931 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "30", 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "5", 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.932 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.933 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.934 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.935 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "password": { 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "password", 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "password of InfluxDB server user", 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.938 INFO:teuthology.orchestra.run.smithi049.stdout: "port": { 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "port", 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8086", 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "InfluxDB server port", 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.939 INFO:teuthology.orchestra.run.smithi049.stdout: "ssl": { 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssl", 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "false", 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.940 INFO:teuthology.orchestra.run.smithi049.stdout: "threads": { 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "threads", 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "32", 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "username": { 2024-03-18T01:01:09.941 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "username", 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "username of InfluxDB server user", 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.942 INFO:teuthology.orchestra.run.smithi049.stdout: "verify_ssl": { 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "verify_ssl", 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "true", 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.943 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "insights", 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.944 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.945 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.947 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.948 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "iostat", 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.949 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.950 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.951 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.951 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.951 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.951 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.951 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.951 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.951 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.951 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.953 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.953 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.953 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.953 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.954 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.955 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.956 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "k8sevents", 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_event_retention_days": { 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ceph_event_retention_days", 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.957 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "7", 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "config_check_secs": { 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "config_check_secs", 2024-03-18T01:01:09.958 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "10", 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.959 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.960 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.961 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.962 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.963 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.964 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "localpool", 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "failure_domain": { 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "failure_domain", 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "host", 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.965 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "failure domain for any created local pool", 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.966 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.967 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.968 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.969 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.970 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": { 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "min_size", 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.971 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "default min_size for any created local pool", 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "num_rep": { 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "num_rep", 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "3", 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "default replica count for any created local pool", 2024-03-18T01:01:09.972 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": { 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pg_num", 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "128", 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T01:01:09.973 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: "prefix": { 2024-03-18T01:01:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "prefix", 2024-03-18T01:01:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.974 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "name prefix for any created local pool", 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "subtree": { 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "subtree", 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.975 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "rack", 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mds_autoscaler", 2024-03-18T01:01:09.976 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.977 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.978 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.979 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.980 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.981 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mirroring", 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.982 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.983 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.984 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.985 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.986 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.987 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "nfs", 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.988 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.989 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.990 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.991 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.992 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "orchestrator", 2024-03-18T01:01:09.993 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "fail_fs": { 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "fail_fs", 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T01:01:09.994 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.995 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.996 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.997 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.998 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:09.999 INFO:teuthology.orchestra.run.smithi049.stdout: "orchestrator": { 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "orchestrator", 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm", 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "rook", 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "test_orchestrator" 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Orchestrator backend", 2024-03-18T01:01:10.000 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "osd_perf_query", 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.001 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.002 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.003 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.004 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.005 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.006 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "osd_support", 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.007 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.008 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.009 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.010 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.011 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pg_autoscaler", 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.012 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.013 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.014 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.015 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.016 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:10.017 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "threshold": { 2024-03-18T01:01:10.018 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "threshold", 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "3.0", 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1.0", 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "scaling threshold", 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.019 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "progress", 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_pg_recovery_event": { 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "allow_pg_recovery_event", 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.020 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": { 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enabled", 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.021 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.022 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.023 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.024 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.025 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.026 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "max_completed_events": { 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_completed_events", 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.027 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "50", 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "number of past completed events to remember", 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.028 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how long the module is going to sleep", 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "prometheus", 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.029 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "cache": { 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cache", 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.030 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "exclude_perf_counters": { 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "exclude_perf_counters", 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.031 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.032 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.033 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.034 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.035 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.036 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_stats_pools": { 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_stats_pools", 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.037 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "300", 2024-03-18T01:01:10.038 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "scrape_interval": { 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "scrape_interval", 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.039 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "15.0", 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.040 INFO:teuthology.orchestra.run.smithi049.stdout: "server_addr": { 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_addr", 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "::", 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.041 INFO:teuthology.orchestra.run.smithi049.stdout: "server_port": { 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_port", 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "9283", 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.042 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "stale_cache_strategy": { 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "stale_cache_strategy", 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "log", 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.043 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_behaviour": { 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_behaviour", 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "default", 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "default", 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "error" 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.044 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_error_status_code": { 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_error_status_code", 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "500", 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "400", 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "599", 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.045 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_support", 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.046 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.047 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.048 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.049 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.050 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "max_concurrent_snap_create": { 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_concurrent_snap_create", 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.051 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "mirror_snapshot_schedule": { 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.052 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "trash_purge_schedule": { 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "trash_purge_schedule", 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.053 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "restful", 2024-03-18T01:01:10.054 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "enable_auth": { 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enable_auth", 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.055 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "key_file": { 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "key_file", 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.056 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.057 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.058 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.059 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.060 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.061 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "server_addr": { 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_addr", 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.062 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "server_port": { 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_port", 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.063 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rgw", 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.064 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.065 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.066 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.067 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.068 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.069 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rook", 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.070 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.071 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.072 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.073 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.074 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": { 2024-03-18T01:01:10.075 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "storage_class", 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "local", 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.076 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "selftest", 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.077 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.080 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.081 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "roption1": { 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "roption1", 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.082 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "roption2": { 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "roption2", 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "xyz", 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.083 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption1": { 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption1", 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.084 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption2": { 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption2", 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.085 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption3": { 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption3", 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.086 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption4": { 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption4", 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.087 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption5": { 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption5", 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.088 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption6": { 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption6", 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.089 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption7": { 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption7", 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "42", 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.090 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "testkey": { 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "testkey", 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.091 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "testlkey": { 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "testlkey", 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.092 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "testnewline": { 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "testnewline", 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.093 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.094 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "snap_schedule", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_m_granularity": { 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "allow_m_granularity", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.095 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "dump_on_update": { 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "dump_on_update", 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "dump database to debug log on update", 2024-03-18T01:01:10.096 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.097 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.098 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.099 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.100 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.101 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "stats", 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.102 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.103 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.104 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.105 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.106 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.107 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "status", 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.108 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.109 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.110 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.111 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.112 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.113 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "telegraf", 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "address": { 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "address", 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.114 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "15", 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.115 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.116 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.117 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.118 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.119 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.120 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "telemetry", 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_basic": { 2024-03-18T01:01:10.121 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_basic", 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_crash": { 2024-03-18T01:01:10.122 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_crash", 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.123 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_device": { 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_device", 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.124 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_ident": { 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_ident", 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.125 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_perf": { 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_perf", 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.126 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "contact": { 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "contact", 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.127 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "description": { 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "description", 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.128 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "device_url": { 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "device_url", 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.129 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": { 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enabled", 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.130 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "24", 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "8", 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.131 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "last_opt_revision": { 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "last_opt_revision", 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "1", 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.132 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "leaderboard": { 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "leaderboard", 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.133 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "leaderboard_description": { 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "leaderboard_description", 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.134 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.135 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.136 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.137 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.138 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.139 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "organization": { 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "organization", 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.140 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "proxy": { 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "proxy", 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.141 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "url": { 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "url", 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.142 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "test_orchestrator", 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.143 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.144 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.145 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.146 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.147 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.148 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "volumes", 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.149 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.150 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.151 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.152 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.153 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "max_concurrent_clones": { 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_concurrent_clones", 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "4", 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.154 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "periodic_async_work": { 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "periodic_async_work", 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Periodically check for async work", 2024-03-18T01:01:10.155 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "snapshot_clone_delay": { 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "snapshot_clone_delay", 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T01:01:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "snapshot_clone_no_wait": { 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T01:01:10.157 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix", 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "discovery_interval": { 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "discovery_interval", 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:10.158 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "100", 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "identifier": { 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "identifier", 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.159 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.160 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.161 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.162 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.163 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.164 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.165 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "zabbix_host": { 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix_host", 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.166 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "zabbix_port": { 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix_port", 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.167 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10051", 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "zabbix_sender": { 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix_sender", 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.168 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.169 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "modules": [ 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm", 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "dashboard", 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "iostat", 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "nfs", 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "restful" 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "available_modules": [ 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "alerts", 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.170 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.171 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.172 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.173 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.174 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.175 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.176 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_destination": { 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_destination", 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Email address to send alerts to", 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.177 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_from_name": { 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_from_name", 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "Ceph", 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Email From: name", 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.178 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_host": { 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_host", 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SMTP server", 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.179 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_password": { 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_password", 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Password to authenticate with", 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.180 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_port": { 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_port", 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "465", 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SMTP port", 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.181 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_sender": { 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_sender", 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SMTP envelope sender", 2024-03-18T01:01:10.182 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_ssl": { 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_ssl", 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.183 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "smtp_user": { 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "smtp_user", 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "User to authenticate as", 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.184 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "balancer", 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "active": { 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "active", 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.185 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "begin_time": { 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "begin_time", 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.186 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0000", 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "begin_weekday": { 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "begin_weekday", 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.187 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "0", 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "6", 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_compat_max_iterations": { 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crush_compat_max_iterations", 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:10.188 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "25", 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "250", 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_compat_metrics": { 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crush_compat_metrics", 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.189 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "crush_compat_step": { 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crush_compat_step", 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.190 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0.5", 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "0.001", 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "0.999", 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "aggressiveness of optimization", 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "end_time": { 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "end_time", 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.191 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "2359", 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "end_weekday": { 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "end_weekday", 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.192 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "0", 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "6", 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.193 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.194 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.195 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.197 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "min_score": { 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "min_score", 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.198 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": { 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mode", 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.199 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "upmap", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "crush-compat", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "none", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "read", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap-read" 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Balancer mode", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.200 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_ids": { 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pool_ids", 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.201 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.202 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap_max_deviation": { 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "upmap_max_deviation", 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T01:01:10.203 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "upmap_max_optimizations": { 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "upmap_max_optimizations", 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T01:01:10.204 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cephadm", 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "agent_down_multiplier": { 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "agent_down_multiplier", 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.205 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "3.0", 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "agent_refresh_rate": { 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "agent_refresh_rate", 2024-03-18T01:01:10.206 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "20", 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "agent_starting_port": { 2024-03-18T01:01:10.207 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "agent_starting_port", 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "4721", 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.208 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_ptrace": { 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "allow_ptrace", 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T01:01:10.209 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "autotune_interval": { 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "autotune_interval", 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.210 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "autotune_memory_target_ratio": { 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0.7", 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.211 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm_log_destination": { 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cephadm_log_destination", 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.212 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "file", 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "file,syslog", 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "syslog" 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "cgroups_split": { 2024-03-18T01:01:10.213 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cgroups_split", 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.214 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "config_checks_enabled": { 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "config_checks_enabled", 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T01:01:10.215 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "config_dashboard": { 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "config_dashboard", 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.216 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_alertmanager": { 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_alertmanager", 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.217 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_base": { 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_base", 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Container image name, without the tag", 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.218 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_elasticsearch": { 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_elasticsearch", 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "elasticsearch container image", 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.219 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_grafana": { 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_grafana", 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.220 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_haproxy": { 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_haproxy", 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "HAproxy container image", 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.221 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_jaeger_agent": { 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_jaeger_agent", 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Jaeger agent container image", 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.222 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_jaeger_collector": { 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_jaeger_collector", 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Jaeger collector container image", 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.223 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_jaeger_query": { 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_jaeger_query", 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Jaeger query container image", 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.224 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_keepalived": { 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_keepalived", 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Keepalived container image", 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.225 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_loki": { 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_loki", 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Loki container image", 2024-03-18T01:01:10.226 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_node_exporter": { 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_node_exporter", 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.227 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_nvmeof": { 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_nvmeof", 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T01:01:10.228 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Nvme-of container image", 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_prometheus": { 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_prometheus", 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T01:01:10.229 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Prometheus container image", 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_promtail": { 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_promtail", 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T01:01:10.230 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Promtail container image", 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "container_image_snmp_gateway": { 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_image_snmp_gateway", 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T01:01:10.231 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "SNMP Gateway container image", 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "container_init": { 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "container_init", 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.232 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "daemon_cache_timeout": { 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "daemon_cache_timeout", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.233 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "default_cephadm_command_timeout": { 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "900", 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.234 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "default_registry": { 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "default_registry", 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "docker.io", 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.235 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "device_cache_timeout": { 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "device_cache_timeout", 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "1800", 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.236 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "seconds to cache device inventory", 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "device_enhanced_scan": { 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "device_enhanced_scan", 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.237 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "facts_cache_timeout": { 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "facts_cache_timeout", 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.238 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "seconds to cache host facts data", 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "host_check_interval": { 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "host_check_interval", 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.239 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to perform a host check", 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "hw_monitoring": { 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "hw_monitoring", 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.240 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "inventory_list_all": { 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "inventory_list_all", 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.241 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.242 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "log_refresh_metadata": { 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_refresh_metadata", 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.243 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.244 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.245 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.246 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T01:01:10.247 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T01:01:10.248 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "*", 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "max_count_per_host": { 2024-03-18T01:01:10.249 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_count_per_host", 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "max number of daemons per service per host", 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.250 INFO:teuthology.orchestra.run.smithi049.stdout: "max_osd_draining_count": { 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_osd_draining_count", 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "migration_current": { 2024-03-18T01:01:10.251 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "migration_current", 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "internal - do not modify", 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "mode": { 2024-03-18T01:01:10.252 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mode", 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "root", 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm-package", 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "root" 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.253 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "oob_default_addr": { 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "oob_default_addr", 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "169.254.1.1", 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.254 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "prometheus_alerts_path": { 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "prometheus_alerts_path", 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.255 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_insecure": { 2024-03-18T01:01:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_insecure", 2024-03-18T01:01:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.256 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T01:01:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_password": { 2024-03-18T01:01:10.257 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_password", 2024-03-18T01:01:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.258 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T01:01:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.259 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_url": { 2024-03-18T01:01:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_url", 2024-03-18T01:01:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.260 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T01:01:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.261 INFO:teuthology.orchestra.run.smithi049.stdout: "registry_username": { 2024-03-18T01:01:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "registry_username", 2024-03-18T01:01:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.262 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T01:01:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.263 INFO:teuthology.orchestra.run.smithi049.stdout: "secure_monitoring_stack": { 2024-03-18T01:01:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "secure_monitoring_stack", 2024-03-18T01:01:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.264 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T01:01:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.265 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "service_discovery_port": { 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "service_discovery_port", 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8765", 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "cephadm service discovery port", 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.266 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "ssh_config_file": { 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssh_config_file", 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.267 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "use_agent": { 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "use_agent", 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.268 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "use_repo_digest": { 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "use_repo_digest", 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.269 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_on_failed_host_check": { 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_on_failed_host_check", 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T01:01:10.270 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_on_stray_daemons": { 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_on_stray_daemons", 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.271 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_on_stray_hosts": { 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_on_stray_hosts", 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T01:01:10.272 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crash", 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.273 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.274 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.275 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.276 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.277 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "retain_interval": { 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "retain_interval", 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "31536000", 2024-03-18T01:01:10.278 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_recent_interval": { 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_recent_interval", 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.279 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "1209600", 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "dashboard", 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.280 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.281 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.282 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.283 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "AUDIT_API_ENABLED": { 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.284 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.285 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.286 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.287 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.288 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.289 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.290 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.291 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.292 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.293 INFO:teuthology.orchestra.run.smithi049.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.294 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "admin", 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T01:01:10.295 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_URL": { 2024-03-18T01:01:10.296 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_URL", 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T01:01:10.297 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "admin", 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T01:01:10.298 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.299 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T01:01:10.300 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.301 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T01:01:10.302 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "{}", 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T01:01:10.303 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T01:01:10.304 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T01:01:10.305 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.306 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.307 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.308 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.309 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.310 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.311 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.312 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.313 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.314 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8", 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.315 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "45", 2024-03-18T01:01:10.316 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.317 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "admin", 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.318 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.319 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.320 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.321 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.322 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.323 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.324 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "cross_origin_url": { 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cross_origin_url", 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.325 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "crt_file": { 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "crt_file", 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.326 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "debug": { 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "debug", 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.327 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Enable/disable debug options", 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "jwt_token_ttl": { 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "jwt_token_ttl", 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "28800", 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.328 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "key_file": { 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "key_file", 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.329 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.331 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.332 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.333 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.334 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "motd": { 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "motd", 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.335 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "The message of the day", 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect_resolve_ip_addr": { 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.336 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "server_addr": { 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_addr", 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.337 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "::", 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "server_port": { 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_port", 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.338 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8080", 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "ssl": { 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssl", 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.339 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "ssl_server_port": { 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssl_server_port", 2024-03-18T01:01:10.340 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8443", 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.341 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_behaviour": { 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_behaviour", 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "redirect", 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: "redirect" 2024-03-18T01:01:10.342 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_error_status_code": { 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_error_status_code", 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "500", 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "400", 2024-03-18T01:01:10.343 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "599", 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "url_prefix": { 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "url_prefix", 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.344 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "devicehealth", 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.345 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "enable_monitoring": { 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enable_monitoring", 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "monitor device health metrics", 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.346 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.347 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.348 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.349 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.350 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.351 INFO:teuthology.orchestra.run.smithi049.stdout: "mark_out_threshold": { 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mark_out_threshold", 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "2419200", 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.352 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "pool_name": { 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pool_name", 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "device_health_metrics", 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.353 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "retention_period": { 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "retention_period", 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "15552000", 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how long to retain device health metrics", 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.354 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "scrape_frequency": { 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "scrape_frequency", 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "86400", 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.355 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "self_heal": { 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "self_heal", 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.356 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.357 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "warn_threshold": { 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "warn_threshold", 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "7257600", 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.358 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "diskprediction_local", 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.359 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.360 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.361 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.362 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.363 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.364 INFO:teuthology.orchestra.run.smithi049.stdout: "predict_interval": { 2024-03-18T01:01:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "predict_interval", 2024-03-18T01:01:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.366 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "86400", 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "predictor_model": { 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "predictor_model", 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.367 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "prophetstor", 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.368 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "600", 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.369 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "influx", 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": false, 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "influxdb python module not found", 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "batch_size": { 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "batch_size", 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5000", 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.370 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "database": { 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "database", 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "ceph", 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.371 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "hostname": { 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "hostname", 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.372 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "InfluxDB server hostname", 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "30", 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "5", 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.373 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.374 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.375 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.376 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.377 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.378 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "password": { 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "password", 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.379 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "password of InfluxDB server user", 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "port": { 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "port", 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "8086", 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.380 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "InfluxDB server port", 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "ssl": { 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ssl", 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "false", 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.381 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "threads": { 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "threads", 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:10.382 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "32", 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "username": { 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "username", 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.383 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "username of InfluxDB server user", 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "verify_ssl": { 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "verify_ssl", 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "true", 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.384 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "insights", 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.385 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.386 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.387 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.388 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.389 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.390 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "iostat", 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.391 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.392 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.393 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.394 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.395 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.396 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "k8sevents", 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_event_retention_days": { 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "ceph_event_retention_days", 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "7", 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T01:01:10.397 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "config_check_secs": { 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "config_check_secs", 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "10", 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T01:01:10.398 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.399 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.400 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.401 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.402 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.403 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "localpool", 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "failure_domain": { 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "failure_domain", 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.404 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "host", 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "failure domain for any created local pool", 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.405 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.406 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.407 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.408 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.409 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "min_size": { 2024-03-18T01:01:10.410 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "min_size", 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "default min_size for any created local pool", 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "num_rep": { 2024-03-18T01:01:10.411 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "num_rep", 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "3", 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "default replica count for any created local pool", 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.412 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_num": { 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pg_num", 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "128", 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.413 INFO:teuthology.orchestra.run.smithi049.stdout: "prefix": { 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "prefix", 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "name prefix for any created local pool", 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.414 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "subtree": { 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "subtree", 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "rack", 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.415 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mds_autoscaler", 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.416 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.417 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.418 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.419 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.420 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mirroring", 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.421 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.422 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.423 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.424 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.425 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.426 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "nfs", 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.427 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.428 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.429 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.430 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.431 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "orchestrator", 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.432 INFO:teuthology.orchestra.run.smithi049.stdout: "fail_fs": { 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "fail_fs", 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.433 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.434 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.435 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.436 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.437 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "orchestrator": { 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "orchestrator", 2024-03-18T01:01:10.438 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "cephadm", 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "rook", 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "test_orchestrator" 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Orchestrator backend", 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.439 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "osd_perf_query", 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.440 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.441 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.442 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.443 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.444 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.445 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "osd_support", 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.446 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.447 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.448 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.449 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.450 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "pg_autoscaler", 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.451 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.452 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.453 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.454 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.455 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.456 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.457 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "threshold": { 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "threshold", 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "3.0", 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1.0", 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "scaling threshold", 2024-03-18T01:01:10.458 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "progress", 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_pg_recovery_event": { 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "allow_pg_recovery_event", 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.459 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": { 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enabled", 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.460 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.461 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.462 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.463 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.464 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.465 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.466 INFO:teuthology.orchestra.run.smithi049.stdout: "max_completed_events": { 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_completed_events", 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "50", 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "number of past completed events to remember", 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "sleep_interval": { 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "sleep_interval", 2024-03-18T01:01:10.467 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "5", 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "how long the module is going to sleep", 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.468 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "prometheus", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "cache": { 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "cache", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.469 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "exclude_perf_counters": { 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "exclude_perf_counters", 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.470 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.471 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.472 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.473 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.474 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.475 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_stats_pools": { 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_stats_pools", 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.476 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "300", 2024-03-18T01:01:10.477 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "scrape_interval": { 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "scrape_interval", 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "15.0", 2024-03-18T01:01:10.478 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "server_addr": { 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_addr", 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "::", 2024-03-18T01:01:10.479 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "server_port": { 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_port", 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "9283", 2024-03-18T01:01:10.480 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "stale_cache_strategy": { 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "stale_cache_strategy", 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "log", 2024-03-18T01:01:10.481 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_behaviour": { 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_behaviour", 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.482 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "default", 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "default", 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "error" 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "standby_error_status_code": { 2024-03-18T01:01:10.483 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "standby_error_status_code", 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "500", 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "400", 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "599", 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.484 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_support", 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.485 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.486 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.487 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.488 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.489 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "max_concurrent_snap_create": { 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_concurrent_snap_create", 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10", 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.490 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "mirror_snapshot_schedule": { 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.491 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "trash_purge_schedule": { 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "trash_purge_schedule", 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.492 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "restful", 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.493 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "enable_auth": { 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enable_auth", 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.494 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "key_file": { 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "key_file", 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.495 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.496 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.497 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.498 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.499 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.500 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "server_addr": { 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_addr", 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.501 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "server_port": { 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "server_port", 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.502 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rgw", 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.503 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.504 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.505 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.506 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.507 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.508 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rook", 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.509 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.510 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.511 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.512 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.513 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.514 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "storage_class": { 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "storage_class", 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "local", 2024-03-18T01:01:10.515 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "selftest", 2024-03-18T01:01:10.516 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.517 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.518 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.519 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.520 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.521 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "roption1": { 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "roption1", 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.522 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "roption2": { 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "roption2", 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "xyz", 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.523 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption1": { 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption1", 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.524 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption2": { 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption2", 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.525 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption3": { 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption3", 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "float", 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.526 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption4": { 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption4", 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.527 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption5": { 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption5", 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.528 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption6": { 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption6", 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.529 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "rwoption7": { 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rwoption7", 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.530 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "1", 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "42", 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "testkey": { 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "testkey", 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.531 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "testlkey": { 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "testlkey", 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.532 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "testnewline": { 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "testnewline", 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.533 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "snap_schedule", 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.534 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "allow_m_granularity": { 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "allow_m_granularity", 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.535 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "dump_on_update": { 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "dump_on_update", 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "dump database to debug log on update", 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.536 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.537 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.538 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.539 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.540 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.541 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "stats", 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.542 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.544 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.545 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.546 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.547 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "status", 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.548 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.549 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.550 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.551 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.552 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "telegraf", 2024-03-18T01:01:10.553 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "address": { 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "address", 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.554 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "15", 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.555 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.556 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.557 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.558 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.559 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.560 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "telemetry", 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_basic": { 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_basic", 2024-03-18T01:01:10.561 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_crash": { 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_crash", 2024-03-18T01:01:10.562 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_device": { 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_device", 2024-03-18T01:01:10.563 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_ident": { 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_ident", 2024-03-18T01:01:10.564 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "channel_perf": { 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "channel_perf", 2024-03-18T01:01:10.565 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "contact": { 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "contact", 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.566 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "description": { 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "description", 2024-03-18T01:01:10.567 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "device_url": { 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "device_url", 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.568 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "enabled": { 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "enabled", 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.569 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:10.570 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "24", 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "8", 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "last_opt_revision": { 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "last_opt_revision", 2024-03-18T01:01:10.571 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "1", 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "leaderboard": { 2024-03-18T01:01:10.572 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "leaderboard", 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "leaderboard_description": { 2024-03-18T01:01:10.573 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "leaderboard_description", 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.574 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.575 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.576 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.577 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.578 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.579 INFO:teuthology.orchestra.run.smithi049.stdout: "organization": { 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "organization", 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.580 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "proxy": { 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "proxy", 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.581 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "url": { 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "url", 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.582 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "test_orchestrator", 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.583 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.584 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.585 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.586 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.587 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.588 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "volumes", 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.589 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.590 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.591 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.592 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.593 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "max_concurrent_clones": { 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "max_concurrent_clones", 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "4", 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T01:01:10.594 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "periodic_async_work": { 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "periodic_async_work", 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Periodically check for async work", 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.595 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "snapshot_clone_delay": { 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "snapshot_clone_delay", 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "0", 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.596 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "snapshot_clone_no_wait": { 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "True", 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.597 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix", 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "can_run": true, 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "error_string": "", 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "module_options": { 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "discovery_interval": { 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "discovery_interval", 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "uint", 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.598 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "100", 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "identifier": { 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "identifier", 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.599 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "interval": { 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "interval", 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "secs", 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "60", 2024-03-18T01:01:10.600 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "log_level": { 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_level", 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.601 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.602 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster": { 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster", 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.603 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_cluster_level": { 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_cluster_level", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "info", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [ 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "critical", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "debug", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "error", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "info", 2024-03-18T01:01:10.604 INFO:teuthology.orchestra.run.smithi049.stdout: "warning" 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "log_to_file": { 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "log_to_file", 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "bool", 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 1, 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "False", 2024-03-18T01:01:10.605 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "zabbix_host": { 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix_host", 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "", 2024-03-18T01:01:10.606 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "zabbix_port": { 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix_port", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "int", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "10051", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.607 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "zabbix_sender": { 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "zabbix_sender", 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "str", 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "level": "advanced", 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "flags": 0, 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "min": "", 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "max": "", 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "enum_allowed": [], 2024-03-18T01:01:10.608 INFO:teuthology.orchestra.run.smithi049.stdout: "desc": "", 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "long_desc": "", 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "tags": [], 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "see_also": [] 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "services": { 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "dashboard": "https://172.21.15.49:8443/" 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "always_on_modules": { 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "octopus": [ 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "balancer", 2024-03-18T01:01:10.609 INFO:teuthology.orchestra.run.smithi049.stdout: "crash", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "devicehealth", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "orchestrator", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscaler", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "progress", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_support", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "status", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "telemetry", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "volumes" 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "pacific": [ 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "balancer", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "crash", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "devicehealth", 2024-03-18T01:01:10.610 INFO:teuthology.orchestra.run.smithi049.stdout: "orchestrator", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscaler", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "progress", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_support", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "status", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "telemetry", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "volumes" 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "quincy": [ 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "balancer", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "crash", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "devicehealth", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "orchestrator", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscaler", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "progress", 2024-03-18T01:01:10.611 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_support", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "status", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "telemetry", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "volumes" 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "reef": [ 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "balancer", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "crash", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "devicehealth", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "orchestrator", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscaler", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "progress", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_support", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "status", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "telemetry", 2024-03-18T01:01:10.612 INFO:teuthology.orchestra.run.smithi049.stdout: "volumes" 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: ], 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "squid": [ 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "balancer", 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "crash", 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "devicehealth", 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "orchestrator", 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "pg_autoscaler", 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "progress", 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "rbd_support", 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "status", 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "telemetry", 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: "volumes" 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T01:01:10.613 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "last_failure_osd_epoch": 3, 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "active_clients": [ 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "devicehealth", 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:0", 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 270568828 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.614 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "libcephsqlite", 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:0", 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 900173211 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "rbd_support", 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.615 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:0", 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 1950297592 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "name": "volumes", 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "addrvec": [ 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: { 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "type": "v2", 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "addr": "172.21.15.49:0", 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: "nonce": 3078209257 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T01:01:10.616 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:10.617 INFO:teuthology.orchestra.run.smithi049.stdout: ] 2024-03-18T01:01:10.617 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:01:10.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8b4c0a2800 msgr2=0x7f8b4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:10.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8b4c0a2800 0x7f8b4c0a4cc0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f8b64199020 tx=0x7f8b58005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:10.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b64100700 msgr2=0x7f8b64197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:10.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b64100700 0x7f8b64197b10 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c009d20 tx=0x7f8b5c00c6a0 comp rx=0 tx=0).stop 2024-03-18T01:01:10.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 shutdown_connections 2024-03-18T01:01:10.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f8b4c0a2800 0x7f8b4c0a4cc0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:10.618 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f8b641022b0 0x7f8b6419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:10.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b641018f0 0x7f8b64198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:10.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 --2- 172.21.15.49:0/2236863130 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f8b64100700 0x7f8b64197b10 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:10.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 >> 172.21.15.49:0/2236863130 conn(0x7f8b640fbc70 msgr2=0x7f8b640fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:10.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 shutdown_connections 2024-03-18T01:01:10.619 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:09.297+0000 7f8b6b6c6640 1 -- 172.21.15.49:0/2236863130 wait complete. 2024-03-18T01:01:10.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:09 smithi049 ceph-mon[32130]: pgmap v681: 97 pgs: 97 active+clean; 420 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 255 MiB/s rd, 11 MiB/s wr, 768 op/s 2024-03-18T01:01:10.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:09 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3078307458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:01:10.619 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:09 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2800161638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:09 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2408424381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:09 smithi106 ceph-mon[35899]: pgmap v681: 97 pgs: 97 active+clean; 420 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 255 MiB/s rd, 11 MiB/s wr, 768 op/s 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:09 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3078307458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:09 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2800161638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:09 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2408424381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:10 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2236863130' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:09 smithi107 ceph-mon[36270]: pgmap v681: 97 pgs: 97 active+clean; 420 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 255 MiB/s rd, 11 MiB/s wr, 768 op/s 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:09 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3078307458' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:09 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2800161638' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:09 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2408424381' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:01:10.620 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:10 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2236863130' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T01:01:10.626 DEBUG:tasks.check_counter:Getting stats from a 2024-03-18T01:01:10.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-18T01:01:10.627 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:10 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2236863130' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T01:01:10.944 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:01:11.271 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:11 smithi049 ceph-mon[32130]: pgmap v682: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 145 MiB/s rd, 6.4 MiB/s wr, 543 op/s 2024-03-18T01:01:11.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:11 smithi107 ceph-mon[36270]: pgmap v682: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 145 MiB/s rd, 6.4 MiB/s wr, 543 op/s 2024-03-18T01:01:11.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:11 smithi106 ceph-mon[35899]: pgmap v682: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 145 MiB/s rd, 6.4 MiB/s wr, 543 op/s 2024-03-18T01:01:12.913 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:01:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:12.913 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 1009, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 999, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 801136, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 379852, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 5, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 1, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.167475687, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.052716041, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.050186397, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.054420312, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 998, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.148444520, 2024-03-18T01:01:12.914 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000148742 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 539728, 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 132448 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 4, 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 166, 2024-03-18T01:01:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 1550, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 370366, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 5, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 4, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.016694971, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.012978736, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.002030101, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.000076321, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 164, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.030204211, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000184172 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 369344 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 785, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 785, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 208211, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 248807, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 4, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 3, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.107467695, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.025121427, 2024-03-18T01:01:12.917 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.031538745, 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.039149231, 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 784, 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.062474779, 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000079687 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T01:01:12.918 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "cct": { 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "total_workers": 1, 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "unhealthy_workers": 0 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "finisher-MDSRank": { 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "queue_len": 0, 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "complete_latency": { 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 3117, 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.152322997, 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000048868 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:12.919 INFO:teuthology.orchestra.run.smithi049.stdout: "queue_len": 0, 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "complete_latency": { 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "mds": { 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "request": 0, 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "reply": 0, 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "reply_latency": { 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.920 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "slow_reply": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "forward": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_fetch_complete": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_fetch_keys": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_commit": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_split": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_merge": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_top": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_bottom": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_pin_tail": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_pinned": 0, 2024-03-18T01:01:12.921 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_expired": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_with_caps": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "caps": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "subtrees": 1, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_hit": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_forward": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_discover": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_lock": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "load_cent": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "q": 0, 2024-03-18T01:01:12.922 INFO:teuthology.orchestra.run.smithi049.stdout: "exported": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "exported_inodes": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "imported": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "imported_inodes": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "openino_peer_discover": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "root_rfiles": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "root_rbytes": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "root_rsnaps": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:01:12.923 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_file_inodes": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_caps": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_cap_release": 0, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "process_request_cap_release": 0 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache": { 2024-03-18T01:01:12.924 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_update": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_update_receipt": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_try_discover": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_send_discover": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_handle_discover": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "num_strays": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "strays_created": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "strays_enqueued": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "strays_migrated": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:12.925 INFO:teuthology.orchestra.run.smithi049.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_started": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_completed": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_exportdir": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_flush": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:12.926 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_inodestats": 0 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log": { 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "evlrg": 0, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "evadd": 0, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "evex": 0, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "evtrm": 0, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "ev": 2, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "evexg": 0, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "evexd": 0, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "segadd": 0, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "segex": 0, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "segtrm": 0, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "seg": 2, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "segmjr": 2, 2024-03-18T01:01:12.927 INFO:teuthology.orchestra.run.smithi049.stdout: "segexg": 0, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "segexd": 0, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "expos": 4194304, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "wrpos": 4194892, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "rdpos": 4194352, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "jlat": { 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "replayed": 2 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_mem": { 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "ino": 3, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "ino+": 3, 2024-03-18T01:01:12.928 INFO:teuthology.orchestra.run.smithi049.stdout: "ino-": 0, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "dir": 1, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "dir+": 1, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "dir-": 0, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "dn": 0, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "dn+": 0, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "dn-": 0, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "cap": 0, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "cap+": 0, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "cap-": 0, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "rss": 50164, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "heap": 215420 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_server": { 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "dispatch_client_request": 0, 2024-03-18T01:01:12.929 INFO:teuthology.orchestra.run.smithi049.stdout: "dispatch_server_request": 0, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_request": 0, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_session": 0, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_peer_request": 0, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "req_create_latency": { 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "req_getattr_latency": { 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.930 INFO:teuthology.orchestra.run.smithi049.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "req_link_latency": { 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookup_latency": { 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.931 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookupino_latency": { 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookupname_latency": { 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.932 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lssnap_latency": { 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.933 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "req_mkdir_latency": { 2024-03-18T01:01:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "req_mknod_latency": { 2024-03-18T01:01:12.934 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "req_mksnap_latency": { 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "req_open_latency": { 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "req_readdir_latency": { 2024-03-18T01:01:12.935 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "req_rename_latency": { 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:12.936 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "req_rmdir_latency": { 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.937 INFO:teuthology.orchestra.run.smithi049.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setattr_latency": { 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.938 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setlayout_latency": { 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setxattr_latency": { 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.939 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "req_symlink_latency": { 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "req_unlink_latency": { 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:12.940 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_sessions": { 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "session_count": 0, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "session_add": 0, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "session_remove": 0, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "sessions_open": 0, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "sessions_stale": 0, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "total_load": 0, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "average_load": 0, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "avg_session_uptime": 0, 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:12.941 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "mempool": { 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:12.942 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:12.943 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_items": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_anon_bytes": 189463, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_anon_items": 52, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bytes": 0, 2024-03-18T01:01:12.944 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_items": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_items": 162, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "pgmap_items": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_co_bytes": 8464, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_co_items": 13, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_1_items": 0, 2024-03-18T01:01:12.945 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_2_items": 0 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter": { 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "op_active": 0, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "op_laggy": 0, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "op_send": 1562, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "op_send_bytes": 21, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "op_resend": 3, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "op_reply": 1562, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "op_latency": { 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 1562, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 3.150912178, 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.002017229 2024-03-18T01:01:12.946 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "op_inflight": 0, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "oplen_avg": { 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 1562, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 1563 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "op": 1562, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "op_r": 1562, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "op_w": 0, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "op_rmw": 0, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "op_pg": 0, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_stat": 778, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_create": 0, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_read": 782, 2024-03-18T01:01:12.947 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_write": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_writefull": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_writesame": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_append": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_zero": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_truncate": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_delete": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_mapext": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_getxattr": 1, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_setxattr": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:12.948 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_call": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_watch": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_notify": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_pgls": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_other": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "linger_active": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "linger_send": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "linger_resend": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "linger_ping": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "poolop_active": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "poolop_send": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "poolop_resend": 0, 2024-03-18T01:01:12.949 INFO:teuthology.orchestra.run.smithi049.stdout: "poolstat_active": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "poolstat_send": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "poolstat_resend": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "statfs_active": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "statfs_send": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "statfs_resend": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "command_active": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "command_send": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "command_resend": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "map_epoch": 88, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "map_full": 0, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "map_inc": 2, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_sessions": 5, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_session_open": 5, 2024-03-18T01:01:12.950 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_session_close": 0, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_laggy": 0, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_wr": 0, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_rd": 2, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_del": 0 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "oft": { 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_total_objs": 0, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_total_updates": 0, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_total_removes": 0 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "osd-slow-ops": { 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "slow_ops_count": 0 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.951 INFO:teuthology.orchestra.run.smithi049.stdout: "purge_queue": { 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executing": 0, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executing_high_water": 0, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executed_ops": 0, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executed": 0, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 104857600, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 1794, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 834593, 2024-03-18T01:01:12.952 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 1794, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 0, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 0, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 1794, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 834593, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:12.953 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 104857600, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 0, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 0, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 1562, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 593, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 1562, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 593, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:12.954 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 1024, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 0, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 0, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 1562, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 1562, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 1562, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 1562, 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:12.955 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 3758096384, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 0, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 0, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 0, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 0, 2024-03-18T01:01:12.956 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 0, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 0, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-write_buf_throttle-0x55cdcbf6c5a0": { 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 3758096384, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 0, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 0, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:12.957 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 0, 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 0, 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 0, 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 0, 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:12.958 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:01:13.225 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:13 smithi049 ceph-mon[32130]: pgmap v683: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s rd, 6.0 MiB/s wr, 532 op/s 2024-03-18T01:01:13.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:13 smithi107 ceph-mon[36270]: pgmap v683: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s rd, 6.0 MiB/s wr, 532 op/s 2024-03-18T01:01:13.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:13 smithi106 ceph-mon[35899]: pgmap v683: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 136 MiB/s rd, 6.0 MiB/s wr, 532 op/s 2024-03-18T01:01:13.694 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-18T01:01:13.694 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-18T01:01:13.694 DEBUG:tasks.check_counter:Getting stats from d 2024-03-18T01:01:13.694 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-18T01:01:13.959 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:01:15.364 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:15 smithi049 ceph-mon[32130]: pgmap v684: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s rd, 3.2 MiB/s wr, 414 op/s 2024-03-18T01:01:15.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:15 smithi107 ceph-mon[36270]: pgmap v684: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s rd, 3.2 MiB/s wr, 414 op/s 2024-03-18T01:01:15.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:15 smithi106 ceph-mon[35899]: pgmap v684: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s rd, 3.2 MiB/s wr, 414 op/s 2024-03-18T01:01:16.374 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:01:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 0, 2024-03-18T01:01:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 2, 2024-03-18T01:01:16.375 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 0, 2024-03-18T01:01:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 0, 2024-03-18T01:01:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 4, 2024-03-18T01:01:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 0, 2024-03-18T01:01:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.001258660, 2024-03-18T01:01:16.376 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T01:01:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.000248428, 2024-03-18T01:01:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T01:01:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:16.377 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:16.378 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:16.379 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:16.379 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:16.379 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.379 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:01:16.379 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:01:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 227, 2024-03-18T01:01:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 222, 2024-03-18T01:01:16.380 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 480688, 2024-03-18T01:01:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 135552, 2024-03-18T01:01:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 2, 2024-03-18T01:01:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 1, 2024-03-18T01:01:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.042615405, 2024-03-18T01:01:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.015696982, 2024-03-18T01:01:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.018575807, 2024-03-18T01:01:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.007673252, 2024-03-18T01:01:16.381 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 222, 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.062391404, 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000281042 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 480688, 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 135552 2024-03-18T01:01:16.382 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 1, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 167, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 160, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 107056, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 3, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 2, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.015576378, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.012683963, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.001302261, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 165, 2024-03-18T01:01:16.383 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.020011520, 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000121281 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 107056 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "cct": { 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "total_workers": 0, 2024-03-18T01:01:16.384 INFO:teuthology.orchestra.run.smithi049.stdout: "unhealthy_workers": 0 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "mempool": { 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:16.385 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:16.386 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_items": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_anon_bytes": 149047, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_anon_items": 31, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:16.387 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bytes": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_items": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_bytes": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_items": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "pgmap_items": 0, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_co_bytes": 928, 2024-03-18T01:01:16.388 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_co_items": 2, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_1_items": 0, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_2_items": 0 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 104857600, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 224, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 446179, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:16.389 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 224, 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 0, 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 0, 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 224, 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 446179, 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:16.390 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:01:17.113 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-18T01:01:17.113 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-18T01:01:17.113 DEBUG:tasks.check_counter:Getting stats from g 2024-03-18T01:01:17.113 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-18T01:01:17.372 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:01:17.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:17 smithi049 ceph-mon[32130]: pgmap v685: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 226 KiB/s wr, 217 op/s 2024-03-18T01:01:17.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:17 smithi107 ceph-mon[36270]: pgmap v685: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 226 KiB/s wr, 217 op/s 2024-03-18T01:01:17.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:17 smithi106 ceph-mon[35899]: pgmap v685: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 226 KiB/s wr, 217 op/s 2024-03-18T01:01:19.280 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:19 smithi049 ceph-mon[32130]: pgmap v686: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.7 KiB/s wr, 163 op/s 2024-03-18T01:01:19.413 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:01:19.413 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:19.413 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 782, 2024-03-18T01:01:19.413 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 784, 2024-03-18T01:01:19.413 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 262697, 2024-03-18T01:01:19.413 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 248677, 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 6, 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 2, 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.123788561, 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.041430542, 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.030898469, 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.039911025, 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 782, 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.143011781, 2024-03-18T01:01:19.414 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000182879 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 1451, 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 1612, 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 372324, 2024-03-18T01:01:19.415 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 829853, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 5, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 4, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.173180207, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.048574469, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.044395354, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.070584253, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 1611, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.169434414, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000105173 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.416 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 369968 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 237, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 227, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 541892, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 135333, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 4, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 3, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.044866264, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.017814000, 2024-03-18T01:01:19.417 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.018624213, 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.007837599, 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 226, 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.048100922, 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000212835 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 540544, 2024-03-18T01:01:19.418 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 134368 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "cct": { 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "total_workers": 1, 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "unhealthy_workers": 0 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "finisher-MDSRank": { 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "queue_len": 0, 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "complete_latency": { 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 3664, 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.136702567, 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000037309 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.419 INFO:teuthology.orchestra.run.smithi049.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "queue_len": 0, 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "complete_latency": { 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "mds": { 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "request": 0, 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "reply": 0, 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "reply_latency": { 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.420 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "slow_reply": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "forward": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_fetch_complete": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_fetch_keys": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_commit": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_split": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_merge": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_top": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_bottom": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_pin_tail": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_pinned": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_expired": 0, 2024-03-18T01:01:19.421 INFO:teuthology.orchestra.run.smithi049.stdout: "inodes_with_caps": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "caps": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "subtrees": 1, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_hit": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_forward": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_discover": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "traverse_lock": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "load_cent": 0, 2024-03-18T01:01:19.422 INFO:teuthology.orchestra.run.smithi049.stdout: "q": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "exported": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "exported_inodes": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "imported": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "imported_inodes": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "openino_peer_discover": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "root_rfiles": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "root_rbytes": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "root_rsnaps": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:19.423 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "scrub_file_inodes": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_caps": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_cap_release": 0, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "process_request_cap_release": 0 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.424 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_cache": { 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_update": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_update_receipt": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_try_discover": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_send_discover": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "dir_handle_discover": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "num_strays": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "strays_created": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "strays_enqueued": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "strays_migrated": 0, 2024-03-18T01:01:19.425 INFO:teuthology.orchestra.run.smithi049.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_started": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "recovery_completed": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_exportdir": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_flush": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:19.426 INFO:teuthology.orchestra.run.smithi049.stdout: "ireq_inodestats": 0 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_log": { 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "evlrg": 0, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "evadd": 0, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "evex": 0, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "evtrm": 0, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "ev": 3, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "evexg": 0, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "evexd": 0, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "segadd": 0, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "segex": 0, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "segtrm": 0, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "seg": 2, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "segmjr": 2, 2024-03-18T01:01:19.427 INFO:teuthology.orchestra.run.smithi049.stdout: "segexg": 0, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "segexd": 0, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "expos": 4194304, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "wrpos": 4195200, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "rdpos": 4194892, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "jlat": { 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "replayed": 3 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_mem": { 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "ino": 3, 2024-03-18T01:01:19.428 INFO:teuthology.orchestra.run.smithi049.stdout: "ino+": 3, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "ino-": 0, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "dir": 1, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "dir+": 1, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "dir-": 0, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "dn": 0, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "dn+": 0, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "dn-": 0, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "cap": 0, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "cap+": 0, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "cap-": 0, 2024-03-18T01:01:19.429 INFO:teuthology.orchestra.run.smithi049.stdout: "rss": 49236, 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "heap": 215420 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_server": { 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "dispatch_client_request": 0, 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "dispatch_server_request": 0, 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_request": 0, 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_client_session": 0, 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "handle_peer_request": 0, 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "req_create_latency": { 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.430 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "req_getattr_latency": { 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "req_link_latency": { 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.431 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookup_latency": { 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.432 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.433 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.433 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookupino_latency": { 2024-03-18T01:01:19.433 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.433 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.433 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.433 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.433 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookupname_latency": { 2024-03-18T01:01:19.433 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:19.434 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: "req_lssnap_latency": { 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: "req_mkdir_latency": { 2024-03-18T01:01:19.435 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:19 smithi107 ceph-mon[36270]: pgmap v686: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.7 KiB/s wr, 163 op/s 2024-03-18T01:01:19.436 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.436 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.436 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.436 INFO:teuthology.orchestra.run.smithi049.stdout: "req_mknod_latency": { 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "req_mksnap_latency": { 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "req_open_latency": { 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.437 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "req_readdir_latency": { 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "req_rename_latency": { 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.438 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "req_rmdir_latency": { 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.439 INFO:teuthology.orchestra.run.smithi049.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setattr_latency": { 2024-03-18T01:01:19.440 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:19.441 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setlayout_latency": { 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "req_setxattr_latency": { 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.442 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "req_symlink_latency": { 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "req_unlink_latency": { 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:19.443 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_sessions": { 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "session_count": 1, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "session_add": 1, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "session_remove": 0, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "sessions_open": 1, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "sessions_stale": 0, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "total_load": 0, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "average_load": 0, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "avg_session_uptime": 1744, 2024-03-18T01:01:19.444 INFO:teuthology.orchestra.run.smithi049.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "mempool": { 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:19.445 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:19.446 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_items": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_anon_bytes": 197503, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_anon_items": 55, 2024-03-18T01:01:19.447 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bytes": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_items": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_items": 162, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "pgmap_items": 0, 2024-03-18T01:01:19.448 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_co_bytes": 8464, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_co_items": 13, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_1_items": 0, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_2_items": 0 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "objecter": { 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "op_active": 0, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "op_laggy": 0, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "op_send": 2231, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "op_send_bytes": 21, 2024-03-18T01:01:19.449 INFO:teuthology.orchestra.run.smithi049.stdout: "op_resend": 3, 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "op_reply": 2231, 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "op_latency": { 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 2231, 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 7.377794400, 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.003306945 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "op_inflight": 0, 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "oplen_avg": { 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 2231, 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 2232 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.450 INFO:teuthology.orchestra.run.smithi049.stdout: "op": 2231, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "op_r": 2231, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "op_w": 0, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "op_rmw": 0, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "op_pg": 0, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_stat": 1443, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_create": 0, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_read": 786, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_write": 0, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_writefull": 0, 2024-03-18T01:01:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_writesame": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_append": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_zero": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_truncate": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_delete": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_mapext": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_getxattr": 1, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_setxattr": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_call": 0, 2024-03-18T01:01:19.452 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_watch": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_notify": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_pgls": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "osdop_other": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "linger_active": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "linger_send": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "linger_resend": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "linger_ping": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "poolop_active": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "poolop_send": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "poolop_resend": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "poolstat_active": 0, 2024-03-18T01:01:19.453 INFO:teuthology.orchestra.run.smithi049.stdout: "poolstat_send": 0, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "poolstat_resend": 0, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "statfs_active": 0, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "statfs_send": 0, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "statfs_resend": 0, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "command_active": 0, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "command_send": 0, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "command_resend": 0, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "map_epoch": 88, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "map_full": 0, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "map_inc": 2, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_sessions": 6, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_session_open": 6, 2024-03-18T01:01:19.454 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_session_close": 0, 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_laggy": 0, 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_wr": 0, 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_rd": 2, 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_del": 0 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "oft": { 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_total_objs": 0, 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_total_updates": 0, 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "omap_total_removes": 0 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "osd-slow-ops": { 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: "slow_ops_count": 0 2024-03-18T01:01:19.455 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "purge_queue": { 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executing": 0, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executing_high_water": 0, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executed_ops": 0, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_executed": 0, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 104857600, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 2466, 2024-03-18T01:01:19.456 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 939973, 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 2466, 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 0, 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 0, 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 2466, 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 939973, 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:19.457 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 104857600, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 0, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 0, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 2231, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 901, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 2231, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 901, 2024-03-18T01:01:19.458 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 1024, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 0, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 0, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:19.459 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 2231, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 2231, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 2231, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 2231, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 3758096384, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 0, 2024-03-18T01:01:19.460 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 0, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 0, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 0, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 0, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 0, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-write_buf_throttle-0x562066c445a0": { 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:19.461 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 3758096384, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:19.462 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:19.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:19.463 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:19.463 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:01:19.463 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:19 smithi106 ceph-mon[35899]: pgmap v686: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.7 KiB/s wr, 163 op/s 2024-03-18T01:01:20.175 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-18T01:01:20.175 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-18T01:01:20.175 DEBUG:tasks.check_counter:Getting stats from j 2024-03-18T01:01:20.175 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-18T01:01:20.433 INFO:teuthology.orchestra.run.smithi049.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T01:01:21.262 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:21 smithi049 ceph-mon[32130]: pgmap v687: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 KiB/s wr, 17 op/s 2024-03-18T01:01:21.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:21 smithi107 ceph-mon[36270]: pgmap v687: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 KiB/s wr, 17 op/s 2024-03-18T01:01:21.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:21 smithi106 ceph-mon[35899]: pgmap v687: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 KiB/s wr, 17 op/s 2024-03-18T01:01:22.165 INFO:teuthology.orchestra.run.smithi049.stdout:{ 2024-03-18T01:01:22.165 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:22.165 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 0, 2024-03-18T01:01:22.165 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 2, 2024-03-18T01:01:22.165 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 0, 2024-03-18T01:01:22.165 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 0, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 4, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 0, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.001411704, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.000317634, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:22.166 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 230, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 225, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 481504, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 137472, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 2, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 1, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.043976537, 2024-03-18T01:01:22.167 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.015816708, 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.019193359, 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.007785106, 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 225, 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.033844225, 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000150418 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.168 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 481504, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 137472 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_messages": 1, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages": 169, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_bytes": 160, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_bytes": 107408, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_created_connections": 3, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_active_connections": 2, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_total_time": 0.015033862, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_send_time": 0.011214092, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_recv_time": 0.001011632, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 167, 2024-03-18T01:01:22.169 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.010927147, 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000065432 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "msgr_send_encrypted_bytes": 107408 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "cct": { 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "total_workers": 0, 2024-03-18T01:01:22.170 INFO:teuthology.orchestra.run.smithi049.stdout: "unhealthy_workers": 0 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "mempool": { 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:22.171 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:22.172 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_items": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_anon_bytes": 150015, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_anon_items": 31, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:22.173 INFO:teuthology.orchestra.run.smithi049.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_bytes": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_items": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_bytes": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_items": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "pgmap_items": 0, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_co_bytes": 928, 2024-03-18T01:01:22.174 INFO:teuthology.orchestra.run.smithi049.stdout: "mds_co_items": 2, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_1_items": 0, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "unittest_2_items": 0 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: }, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "val": 0, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "max": 104857600, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "get_started": 0, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "get": 227, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "get_sum": 446569, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "get_or_fail_success": 227, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "take": 0, 2024-03-18T01:01:22.175 INFO:teuthology.orchestra.run.smithi049.stdout: "take_sum": 0, 2024-03-18T01:01:22.176 INFO:teuthology.orchestra.run.smithi049.stdout: "put": 227, 2024-03-18T01:01:22.176 INFO:teuthology.orchestra.run.smithi049.stdout: "put_sum": 446569, 2024-03-18T01:01:22.176 INFO:teuthology.orchestra.run.smithi049.stdout: "wait": { 2024-03-18T01:01:22.176 INFO:teuthology.orchestra.run.smithi049.stdout: "avgcount": 0, 2024-03-18T01:01:22.176 INFO:teuthology.orchestra.run.smithi049.stdout: "sum": 0.000000000, 2024-03-18T01:01:22.176 INFO:teuthology.orchestra.run.smithi049.stdout: "avgtime": 0.000000000 2024-03-18T01:01:22.176 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:22.176 INFO:teuthology.orchestra.run.smithi049.stdout: } 2024-03-18T01:01:22.176 INFO:teuthology.orchestra.run.smithi049.stdout:} 2024-03-18T01:01:22.853 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-18T01:01:22.853 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-18T01:01:22.853 DEBUG:tasks.check_counter:Getting stats from b 2024-03-18T01:01:22.853 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-18T01:01:23.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:23 smithi049 ceph-mon[32130]: pgmap v688: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-18T01:01:23.294 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:23 smithi107 ceph-mon[36270]: pgmap v688: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-18T01:01:23.363 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:01:23.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:23 smithi106 ceph-mon[35899]: pgmap v688: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-03-18T01:01:25.196 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:25 smithi106 ceph-mon[35899]: pgmap v689: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-18T01:01:25.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:25 smithi049 ceph-mon[32130]: pgmap v689: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-18T01:01:25.388 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 3901, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 3743, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 1091688, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 324174479, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 22, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 3, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 2.199104084, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 1.285317674, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.151536072, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.720169577, 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:25.389 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 3742, 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 1.849172620, 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000494166 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 2484, 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000633182, 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000254 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T01:01:25.390 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 529591, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 550931, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 164531071, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 486201039, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 14, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 6, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 33.622548810, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 13.867180077, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 62.688732920, 2024-03-18T01:01:25.391 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.608423056, 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 550930, 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 116.360304467, 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000211207 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 448709, 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.286487198, 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000638 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.392 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 633952, 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 195488 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 88632, 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 47076, 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 23963733, 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 261240532, 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 196, 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 9, 2024-03-18T01:01:25.393 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 4.650218242, 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 2.240117051, 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 2.435201562, 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.507977942, 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 47074, 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 2.592098906, 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000055064 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 128238, 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.016565384, 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000129 2024-03-18T01:01:25.394 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 378848 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "cct": { 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "total_workers": 1, 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "unhealthy_workers": 0 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-MDSRank": { 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 2933, 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 483.209213459, 2024-03-18T01:01:25.395 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.164749135 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 30, 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000753908, 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000025130 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "mds": { 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "request": 97, 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "reply": 97, 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "reply_latency": { 2024-03-18T01:01:25.396 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 97, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 637.073863349, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 6.567771787 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_reply": 0, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "forward": 0, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_complete": 5, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_keys": 2, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_commit": 181, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_split": 1, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_merge": 1, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes": 26, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_top": 6, 2024-03-18T01:01:25.397 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_bottom": 7, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pin_tail": 13, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pinned": 17, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_expired": 24, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_with_caps": 2, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "caps": 2, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "subtrees": 3, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse": 186, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_hit": 152, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_forward": 0, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_discover": 0, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_lock": 0, 2024-03-18T01:01:25.398 INFO:teuthology.orchestra.run.smithi106.stdout: "load_cent": 31, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "q": 0, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "exported": 2, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "exported_inodes": 18, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "imported": 1, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "imported_inodes": 6, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_backtrace_fetch": 442, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_peer_discover": 0, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rfiles": 4, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rbytes": 234, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rsnaps": 0, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_fetch": 442, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:25.399 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_inodes": 169, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_base_inodes": 16, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dirfrag_rstats": 153, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_file_inodes": 273, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_inode_file_caps": 3, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_revoke": 21131, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_grant": 24506, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps": 524348, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps_dirty": 500775, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_cap_release": 5, 2024-03-18T01:01:25.400 INFO:teuthology.orchestra.run.smithi106.stdout: "process_request_cap_release": 28 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache": { 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update": 0, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update_receipt": 0, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_try_discover": 0, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_send_discover": 0, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_handle_discover": 40, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays": 1, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_created": 4, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_enqueued": 3, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_migrated": 0, 2024-03-18T01:01:25.401 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_started": 0, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_completed": 0, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_enqueue_scrub": 16, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_exportdir": 2, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_flush": 0, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragmentdir": 2, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_inodestats": 0 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.402 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log": { 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "evlrg": 0, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "evadd": 501649, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "evex": 501634, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "evtrm": 501634, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "ev": 15, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "evexg": 0, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "evexd": 4, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "segadd": 600, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "segex": 599, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "segtrm": 599, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "seg": 1, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "segmjr": 1, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "segexg": 0, 2024-03-18T01:01:25.403 INFO:teuthology.orchestra.run.smithi106.stdout: "segexd": 1, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "expos": 958554778, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "wrpos": 958592486, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "rdpos": 0, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "jlat": { 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 6672, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 2887.347299685, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.432755890 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "replayed": 0 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_mem": { 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "ino": 26, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "ino+": 68, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "ino-": 42, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "dir": 21, 2024-03-18T01:01:25.404 INFO:teuthology.orchestra.run.smithi106.stdout: "dir+": 35, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "dir-": 14, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "dn": 26, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "dn+": 61, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "dn-": 35, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "cap": 2, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "cap+": 41, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "cap-": 39, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "rss": 79136, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "heap": 231804 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_server": { 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_client_request": 168, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_server_request": 17, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_request": 97, 2024-03-18T01:01:25.405 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_session": 17, 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_peer_request": 47, 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "req_create_latency": { 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 22, 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.021804771, 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000991125 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getattr_latency": { 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 20, 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 611.478119143, 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 30.573905957 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.406 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "req_link_latency": { 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookup_latency": { 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 26, 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 25.482831366, 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.980108898 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.407 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupino_latency": { 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1, 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000177237, 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000177237 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupname_latency": { 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.408 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.409 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.409 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:25.409 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.409 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.409 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.409 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.409 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lssnap_latency": { 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mkdir_latency": { 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 10, 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.013370437, 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.001337043 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mknod_latency": { 2024-03-18T01:01:25.410 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mksnap_latency": { 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "req_open_latency": { 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1, 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000227456, 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000227456 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "req_readdir_latency": { 2024-03-18T01:01:25.411 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 3, 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.010266917, 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.003422305 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rename_latency": { 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 2, 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.002759310, 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.001379655 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.412 INFO:teuthology.orchestra.run.smithi106.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmdir_latency": { 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 3, 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.042523373, 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.014174457 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.413 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setattr_latency": { 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1, 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.001606992, 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.001606992 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.414 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setlayout_latency": { 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setxattr_latency": { 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 7, 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.017994423, 2024-03-18T01:01:25.415 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.002570631 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "req_symlink_latency": { 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "req_unlink_latency": { 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1, 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.002181924, 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.002181924 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:25.416 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_sessions": { 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "session_count": 1, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "session_add": 2, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "session_remove": 1, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_open": 1, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_stale": 0, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "total_load": 19, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "average_load": 19, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "avg_session_uptime": 681, 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:25.417 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "mempool": { 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:25.418 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:25.419 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_items": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_bytes": 320816, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_items": 123, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bytes": 0, 2024-03-18T01:01:25.420 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_items": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_items": 162, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_items": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_bytes": 108458, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_items": 769, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:25.421 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_items": 0, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_items": 0 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter": { 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "op_active": 0, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "op_laggy": 0, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send": 8888, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send_bytes": 970038906, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "op_resend": 37, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "op_reply": 8888, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "op_latency": { 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 8888, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 361.775503169, 2024-03-18T01:01:25.422 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.040703814 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "op_inflight": 0, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "oplen_avg": { 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 8888, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 9336 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "op": 8888, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "op_r": 484, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "op_w": 8404, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "op_rmw": 0, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "op_pg": 0, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_stat": 150, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_create": 44, 2024-03-18T01:01:25.423 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_read": 19, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_write": 6675, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writefull": 138, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writesame": 0, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_append": 0, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_zero": 0, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_truncate": 1, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_delete": 977, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_mapext": 0, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_getxattr": 465, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_setxattr": 88, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:25.424 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_call": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_watch": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_notify": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_other": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_active": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_send": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_resend": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_ping": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_active": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_send": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_resend": 0, 2024-03-18T01:01:25.425 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_active": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_send": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_resend": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_active": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_send": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_resend": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "command_active": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "command_send": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "command_resend": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "map_epoch": 88, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "map_full": 0, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "map_inc": 3, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_sessions": 12, 2024-03-18T01:01:25.426 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_open": 12, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_close": 0, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_laggy": 0, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_wr": 750, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_rd": 14, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_del": 15 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "oft": { 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_objs": 1, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_updates": 0, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_removes": 0 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "osd-slow-ops": { 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_ops_count": 6 2024-03-18T01:01:25.427 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "purge_queue": { 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops_high_water": 1, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing": 0, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_high_water": 1, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed_ops": 3, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed": 3, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:25.428 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 622120, 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 141968547, 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 622120, 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 622120, 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 141968547, 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.429 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:25.430 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 8888, 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 970039121, 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 8888, 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 970039121, 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:25.431 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.432 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:25.432 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:25.432 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 1024, 2024-03-18T01:01:25.432 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:25.432 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:25.432 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:25.432 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:25.432 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:25.432 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 8888, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 8888, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 8888, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 8888, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:25.433 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 3, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 303, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 3, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 3, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 303, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle-0x556d2b256500": { 2024-03-18T01:01:25.434 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 501649, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 954398182, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 501649, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 6672, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 954398182, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:25.435 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:25.436 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:25.436 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-18T01:01:25.436 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:25 smithi107 ceph-mon[36270]: pgmap v689: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-18T01:01:26.396 INFO:tasks.check_counter:Daemon mds.b mds.exported=2 2024-03-18T01:01:26.396 INFO:tasks.check_counter:Daemon mds.b mds.imported=1 2024-03-18T01:01:26.396 DEBUG:tasks.check_counter:Getting stats from e 2024-03-18T01:01:26.397 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-18T01:01:26.716 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:01:27.136 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:27 smithi106 ceph-mon[35899]: pgmap v690: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-18T01:01:27.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:27 smithi049 ceph-mon[32130]: pgmap v690: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-18T01:01:27.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:27 smithi107 ceph-mon[36270]: pgmap v690: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-18T01:01:28.961 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-18T01:01:28.961 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:28.961 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 1854, 2024-03-18T01:01:28.961 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 1856, 2024-03-18T01:01:28.961 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 336128136, 2024-03-18T01:01:28.961 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 589582, 2024-03-18T01:01:28.961 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 8, 2024-03-18T01:01:28.961 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 4, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.421267068, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.046676006, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.301692797, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.057628055, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1854, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.093985937, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000050693 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.962 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 1635, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 1625, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 405923241, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 581726, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 6, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 5, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.416237797, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.041540373, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.317329275, 2024-03-18T01:01:28.963 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.047786894, 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1624, 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.082272588, 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000050660 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 542336, 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 138368 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.964 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:28.965 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 769, 2024-03-18T01:01:28.965 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 934, 2024-03-18T01:01:28.965 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 214143808, 2024-03-18T01:01:28.965 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 616715, 2024-03-18T01:01:28.965 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 6, 2024-03-18T01:01:28.965 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 5, 2024-03-18T01:01:28.965 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.232053594, 2024-03-18T01:01:28.965 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.021314916, 2024-03-18T01:01:28.965 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.180641379, 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.022754451, 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 933, 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.041638998, 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000044629 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T01:01:28.966 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 374080 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: "cct": { 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: "total_workers": 1, 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: "unhealthy_workers": 0 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-MDSRank": { 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 5870, 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.155468366, 2024-03-18T01:01:28.967 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000026485 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: "mds": { 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: "request": 0, 2024-03-18T01:01:28.968 INFO:teuthology.orchestra.run.smithi106.stdout: "reply": 0, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "reply_latency": { 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_reply": 0, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "forward": 0, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_complete": 0, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_keys": 0, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_commit": 0, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_split": 0, 2024-03-18T01:01:28.969 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_merge": 0, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes": 19, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_top": 4, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_bottom": 15, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pin_tail": 0, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pinned": 12, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_expired": 0, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_with_caps": 0, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "caps": 0, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "subtrees": 2, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse": 0, 2024-03-18T01:01:28.970 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_hit": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_forward": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_discover": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_lock": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "load_cent": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "q": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "exported": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "exported_inodes": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "imported": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "imported_inodes": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_peer_discover": 0, 2024-03-18T01:01:28.971 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rfiles": 4, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rbytes": 234, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rsnaps": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_file_inodes": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:28.972 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_cap_release": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "process_request_cap_release": 0 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache": { 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update_receipt": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_try_discover": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_send_discover": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_handle_discover": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays": 0, 2024-03-18T01:01:28.973 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_created": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_enqueued": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_migrated": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_started": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_completed": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:01:28.974 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_exportdir": 0, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_flush": 0, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_inodestats": 0 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log": { 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "evlrg": 0, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "evadd": 0, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "evex": 0, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "evtrm": 0, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "ev": 501649, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "evexg": 0, 2024-03-18T01:01:28.975 INFO:teuthology.orchestra.run.smithi106.stdout: "evexd": 0, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "segadd": 0, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "segex": 0, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "segtrm": 0, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "seg": 2, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "segmjr": 67, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "segexg": 0, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "segexd": 0, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "expos": 958554778, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "wrpos": 958592486, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "rdpos": 958588185, 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "jlat": { 2024-03-18T01:01:28.976 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "replayed": 501649 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_mem": { 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "ino": 15, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "ino+": 36, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "ino-": 21, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "dir": 10, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "dir+": 22, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "dir-": 12, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "dn": 19, 2024-03-18T01:01:28.977 INFO:teuthology.orchestra.run.smithi106.stdout: "dn+": 45, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "dn-": 26, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "cap": 0, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "cap+": 0, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "cap-": 0, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "rss": 53152, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "heap": 215420 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_server": { 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_client_request": 0, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_server_request": 0, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_request": 0, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_session": 0, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_peer_request": 0, 2024-03-18T01:01:28.978 INFO:teuthology.orchestra.run.smithi106.stdout: "req_create_latency": { 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getattr_latency": { 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.979 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "req_link_latency": { 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookup_latency": { 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.980 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupino_latency": { 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupname_latency": { 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.981 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lssnap_latency": { 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.982 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mkdir_latency": { 2024-03-18T01:01:28.983 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.983 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.983 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.983 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.983 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mknod_latency": { 2024-03-18T01:01:28.983 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mksnap_latency": { 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "req_open_latency": { 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.984 INFO:teuthology.orchestra.run.smithi106.stdout: "req_readdir_latency": { 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rename_latency": { 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.985 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmdir_latency": { 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.986 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setattr_latency": { 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.987 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setlayout_latency": { 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setxattr_latency": { 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.988 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "req_symlink_latency": { 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "req_unlink_latency": { 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:28.989 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_sessions": { 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "session_count": 1, 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "session_add": 2, 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "session_remove": 1, 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_open": 1, 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_stale": 0, 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "total_load": 0, 2024-03-18T01:01:28.990 INFO:teuthology.orchestra.run.smithi106.stdout: "average_load": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "avg_session_uptime": 2113, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "mempool": { 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:28.991 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:28.992 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_items": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_bytes": 238718, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_items": 79, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:28.993 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bytes": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_items": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_items": 162, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_items": 0, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_bytes": 59762, 2024-03-18T01:01:28.994 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_items": 358, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_items": 0, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_items": 0 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter": { 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "op_active": 0, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "op_laggy": 0, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send": 4011, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send_bytes": 26, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "op_resend": 4, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "op_reply": 4011, 2024-03-18T01:01:28.995 INFO:teuthology.orchestra.run.smithi106.stdout: "op_latency": { 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 4011, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 17.620586599, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.004393065 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "op_inflight": 0, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "oplen_avg": { 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 4011, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 4012 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "op": 4011, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "op_r": 4011, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "op_w": 0, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "op_rmw": 0, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "op_pg": 0, 2024-03-18T01:01:28.996 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_stat": 2598, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_create": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_read": 1410, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_write": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writefull": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writesame": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_append": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_zero": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_truncate": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_delete": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_mapext": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_getxattr": 2, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_setxattr": 0, 2024-03-18T01:01:28.997 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_call": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_watch": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_notify": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_other": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_active": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_send": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_resend": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_ping": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_active": 0, 2024-03-18T01:01:28.998 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_send": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_resend": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_active": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_send": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_resend": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_active": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_send": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_resend": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "command_active": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "command_send": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "command_resend": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "map_epoch": 88, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "map_full": 0, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "map_inc": 2, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_sessions": 11, 2024-03-18T01:01:28.999 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_open": 11, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_close": 0, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_laggy": 0, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_wr": 0, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_rd": 2, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_del": 0 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "oft": { 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_objs": 0, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_updates": 0, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_removes": 0 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "osd-slow-ops": { 2024-03-18T01:01:29.000 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_ops_count": 0 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "purge_queue": { 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing": 0, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_high_water": 0, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed_ops": 0, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed": 0, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:29.001 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 4254, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 955796967, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 4254, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 4254, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 955796967, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:29.002 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 4011, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 954561555, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 4011, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 954561555, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:29.003 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 1024, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 4011, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 4011, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 4011, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 4011, 2024-03-18T01:01:29.004 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:29.005 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 0, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 0, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle-0x55e9f28705a0": { 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:29.006 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 0, 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 0, 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:29.007 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-18T01:01:29.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:29 smithi106 ceph-mon[35899]: pgmap v691: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-18T01:01:29.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:29 smithi049 ceph-mon[32130]: pgmap v691: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-18T01:01:29.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:29 smithi107 ceph-mon[36270]: pgmap v691: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 12 op/s 2024-03-18T01:01:29.803 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-18T01:01:29.804 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-18T01:01:29.804 DEBUG:tasks.check_counter:Getting stats from h 2024-03-18T01:01:29.804 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-18T01:01:30.085 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:01:31.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:31 smithi049 ceph-mon[32130]: pgmap v692: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-18T01:01:31.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:31 smithi107 ceph-mon[36270]: pgmap v692: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-18T01:01:31.438 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:31 smithi106 ceph-mon[35899]: pgmap v692: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-03-18T01:01:31.878 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-18T01:01:31.879 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:31.879 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 977, 2024-03-18T01:01:31.879 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 1037, 2024-03-18T01:01:31.879 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 826528, 2024-03-18T01:01:31.879 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 375833, 2024-03-18T01:01:31.879 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 8, 2024-03-18T01:01:31.879 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 2, 2024-03-18T01:01:31.879 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.098421554, 2024-03-18T01:01:31.879 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.039248945, 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.052270349, 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.009334250, 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1043, 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.085681240, 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000082148 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1394, 2024-03-18T01:01:31.880 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000325190, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000233 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 562960, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 142016 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 82, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 249, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 68132, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 445424, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 8, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 5, 2024-03-18T01:01:31.881 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.015471579, 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.008001371, 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.006775122, 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.000256613, 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 247, 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.036808520, 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000149022 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 117, 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000043943, 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000375 2024-03-18T01:01:31.882 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 375072 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 116, 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 124, 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 72883, 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 493443, 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 45, 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 8, 2024-03-18T01:01:31.883 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.023916493, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.002883746, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.020457170, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.000200790, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 116, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.018981302, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000163631 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 156, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000032259, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000206 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.884 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "cct": { 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "total_workers": 1, 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "unhealthy_workers": 0 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-MDSRank": { 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 41, 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000903111, 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000022027 2024-03-18T01:01:31.885 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:31.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.886 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:31.886 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-18T01:01:31.886 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-18T01:01:31.886 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 2, 2024-03-18T01:01:31.886 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000006279, 2024-03-18T01:01:31.886 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000003139 2024-03-18T01:01:31.886 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:31.886 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: "mds": { 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: "request": 0, 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: "reply": 0, 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: "reply_latency": { 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_reply": 0, 2024-03-18T01:01:31.887 INFO:teuthology.orchestra.run.smithi106.stdout: "forward": 0, 2024-03-18T01:01:31.888 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_complete": 0, 2024-03-18T01:01:31.888 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_keys": 0, 2024-03-18T01:01:31.888 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_commit": 11, 2024-03-18T01:01:31.888 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_split": 0, 2024-03-18T01:01:31.888 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_merge": 0, 2024-03-18T01:01:31.888 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes": 15, 2024-03-18T01:01:31.888 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_top": 0, 2024-03-18T01:01:31.888 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_bottom": 15, 2024-03-18T01:01:31.888 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pin_tail": 0, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pinned": 13, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_expired": 6, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_with_caps": 0, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "caps": 0, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "subtrees": 3, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse": 12, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_hit": 11, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_forward": 0, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_discover": 1, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:31.889 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_lock": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "load_cent": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "q": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "exported": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "exported_inodes": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "imported": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "imported_inodes": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_peer_discover": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rfiles": 22, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rbytes": 10738150101, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rsnaps": 0, 2024-03-18T01:01:31.890 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_file_inodes": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:31.891 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps": 0, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_cap_release": 0, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "process_request_cap_release": 0 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache": { 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update": 0, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update_receipt": 21, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_try_discover": 11, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_send_discover": 22, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_handle_discover": 0, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays": 0, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:31.892 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_created": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_enqueued": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_migrated": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_started": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_completed": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_exportdir": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_flush": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:01:31.893 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_inodestats": 0 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log": { 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "evlrg": 0, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "evadd": 2, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "evex": 0, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "evtrm": 0, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "ev": 2, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "evexg": 0, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "evexd": 0, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "segadd": 2, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "segex": 0, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "segtrm": 0, 2024-03-18T01:01:31.894 INFO:teuthology.orchestra.run.smithi106.stdout: "seg": 2, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "segmjr": 2, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "segexg": 0, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "segexd": 0, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "expos": 4194304, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "wrpos": 4194892, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "rdpos": 0, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "jlat": { 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.001391631, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.001391631 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "replayed": 0 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_mem": { 2024-03-18T01:01:31.895 INFO:teuthology.orchestra.run.smithi106.stdout: "ino": 19, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "ino+": 22, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "ino-": 3, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "dir": 18, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "dir+": 31, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "dir-": 13, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "dn": 15, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "dn+": 21, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "dn-": 6, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "cap": 0, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "cap+": 0, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "cap-": 0, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "rss": 50236, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "heap": 223612 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.896 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_server": { 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_client_request": 0, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_server_request": 1, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_request": 0, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_session": 0, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_peer_request": 2, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "req_create_latency": { 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getattr_latency": { 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.897 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "req_link_latency": { 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookup_latency": { 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.898 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupino_latency": { 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupname_latency": { 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.899 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lssnap_latency": { 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.900 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.901 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mkdir_latency": { 2024-03-18T01:01:31.901 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.901 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.901 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.901 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mknod_latency": { 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mksnap_latency": { 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "req_open_latency": { 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.902 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "req_readdir_latency": { 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rename_latency": { 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.903 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmdir_latency": { 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:31.904 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setattr_latency": { 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:31.905 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setlayout_latency": { 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setxattr_latency": { 2024-03-18T01:01:31.906 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "req_symlink_latency": { 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "req_unlink_latency": { 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:31.907 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_sessions": { 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "session_count": 0, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "session_add": 0, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "session_remove": 0, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_open": 0, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_stale": 0, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "total_load": 0, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "average_load": 0, 2024-03-18T01:01:31.908 INFO:teuthology.orchestra.run.smithi106.stdout: "avg_session_uptime": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "mempool": { 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:31.909 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:31.910 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_items": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_bytes": 226031, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_items": 73, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:31.911 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bytes": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_items": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_items": 162, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_items": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_bytes": 76373, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_items": 302, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:31.912 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_items": 0, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_items": 0 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter": { 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "op_active": 0, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "op_laggy": 0, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send": 35, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send_bytes": 10710, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "op_resend": 28, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "op_reply": 35, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "op_latency": { 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 35, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.198995300, 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.005685580 2024-03-18T01:01:31.913 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "op_inflight": 0, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "oplen_avg": { 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 35, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 57 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "op": 35, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "op_r": 0, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "op_w": 35, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "op_rmw": 0, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "op_pg": 0, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_stat": 0, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_create": 10, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_read": 0, 2024-03-18T01:01:31.914 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_write": 1, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writefull": 5, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writesame": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_append": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_zero": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_truncate": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_delete": 6, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_mapext": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_getxattr": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_setxattr": 20, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:31.915 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_call": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_watch": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_notify": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_other": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_active": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_send": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_resend": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_ping": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_active": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_send": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_resend": 0, 2024-03-18T01:01:31.916 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_active": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_send": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_resend": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_active": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_send": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_resend": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "command_active": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "command_send": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "command_resend": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "map_epoch": 88, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "map_full": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "map_inc": 2, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_sessions": 8, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_open": 8, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_close": 0, 2024-03-18T01:01:31.917 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_laggy": 0, 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_wr": 14, 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_rd": 0, 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_del": 1 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "oft": { 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_objs": 1, 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_updates": 0, 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_removes": 0 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "osd-slow-ops": { 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_ops_count": 0 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.918 INFO:teuthology.orchestra.run.smithi106.stdout: "purge_queue": { 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing": 0, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_high_water": 0, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed_ops": 0, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed": 0, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 1171, 2024-03-18T01:01:31.919 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 845565, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 1171, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 1171, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 845565, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:31.920 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 35, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 10710, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 35, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 10710, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:31.921 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 1024, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 35, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 35, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 35, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 35, 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:31.922 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:31.923 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 0, 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 0, 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle-0x5628915bc500": { 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 2, 2024-03-18T01:01:31.924 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 588, 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 2, 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 1, 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 588, 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:31.925 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-18T01:01:32.776 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-18T01:01:32.776 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-18T01:01:32.776 DEBUG:tasks.check_counter:Getting stats from k 2024-03-18T01:01:32.777 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-18T01:01:33.075 INFO:teuthology.orchestra.run.smithi106.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T01:01:33.354 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:33 smithi106 ceph-mon[35899]: pgmap v693: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-18T01:01:33.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:33 smithi049 ceph-mon[32130]: pgmap v693: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-18T01:01:33.375 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:33 smithi107 ceph-mon[36270]: pgmap v693: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-18T01:01:34.931 INFO:teuthology.orchestra.run.smithi106.stdout:{ 2024-03-18T01:01:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 799, 2024-03-18T01:01:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 801, 2024-03-18T01:01:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 210549, 2024-03-18T01:01:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 254146, 2024-03-18T01:01:34.931 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 6, 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 2, 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.070019243, 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.023156159, 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.016925065, 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.022397699, 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 799, 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.040449884, 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000050625 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:34.932 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 3, 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 174, 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 674, 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 376417, 2024-03-18T01:01:34.933 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 4, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 3, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.009000378, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.006721427, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.000955153, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.000044250, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 172, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.007235877, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000042069 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.934 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 375776 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_messages": 1043, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages": 1031, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_bytes": 813376, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_bytes": 394026, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_created_connections": 3, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_active_connections": 2, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_total_time": 0.110014368, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_send_time": 0.032002459, 2024-03-18T01:01:34.935 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_recv_time": 0.037442144, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_running_fast_dispatch_time": 0.035763216, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1031, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.059395873, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000057609 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_recv_encrypted_bytes": 546256, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "msgr_send_encrypted_bytes": 141216 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.936 INFO:teuthology.orchestra.run.smithi106.stdout: "cct": { 2024-03-18T01:01:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "total_workers": 1, 2024-03-18T01:01:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "unhealthy_workers": 0 2024-03-18T01:01:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-MDSRank": { 2024-03-18T01:01:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-18T01:01:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-18T01:01:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 3185, 2024-03-18T01:01:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.098451510, 2024-03-18T01:01:34.937 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000030910 2024-03-18T01:01:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "queue_len": 0, 2024-03-18T01:01:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "complete_latency": { 2024-03-18T01:01:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.938 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "mds": { 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "request": 0, 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "reply": 0, 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "reply_latency": { 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_reply": 0, 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "forward": 0, 2024-03-18T01:01:34.939 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_complete": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_fetch_keys": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_commit": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_split": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_merge": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_top": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_bottom": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pin_tail": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_pinned": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_expired": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "inodes_with_caps": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "caps": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "subtrees": 1, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse": 0, 2024-03-18T01:01:34.940 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_hit": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_forward": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_discover": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "traverse_lock": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "load_cent": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "q": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "exported": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "exported_inodes": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "imported": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "imported_inodes": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:01:34.941 INFO:teuthology.orchestra.run.smithi106.stdout: "openino_peer_discover": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rfiles": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rbytes": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "root_rsnaps": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "scrub_file_inodes": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:01:34.942 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps": 0, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_cap_release": 0, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "process_request_cap_release": 0 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_cache": { 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update": 0, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_update_receipt": 0, 2024-03-18T01:01:34.943 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_try_discover": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_send_discover": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "dir_handle_discover": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_created": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_enqueued": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "strays_migrated": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_started": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "recovery_completed": 0, 2024-03-18T01:01:34.944 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_exportdir": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_flush": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "ireq_inodestats": 0 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_log": { 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "evlrg": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "evadd": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "evex": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "evtrm": 0, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "ev": 2, 2024-03-18T01:01:34.945 INFO:teuthology.orchestra.run.smithi106.stdout: "evexg": 0, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "evexd": 0, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "segadd": 0, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "segex": 0, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "segtrm": 0, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "seg": 2, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "segmjr": 2, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "segexg": 0, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "segexd": 0, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "expos": 4194304, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "wrpos": 4194892, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "rdpos": 4194352, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "jlat": { 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.946 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "replayed": 2 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_mem": { 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "ino": 3, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "ino+": 3, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "ino-": 0, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "dir": 1, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "dir+": 1, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "dir-": 0, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "dn": 0, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "dn+": 0, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "dn-": 0, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "cap": 0, 2024-03-18T01:01:34.947 INFO:teuthology.orchestra.run.smithi106.stdout: "cap+": 0, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "cap-": 0, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "rss": 48936, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "heap": 215420 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_server": { 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_client_request": 0, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "dispatch_server_request": 0, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_request": 0, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_client_session": 0, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "handle_peer_request": 0, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "req_create_latency": { 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.948 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getattr_latency": { 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "req_link_latency": { 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.949 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookup_latency": { 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupino_latency": { 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.950 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupname_latency": { 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.951 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "req_lssnap_latency": { 2024-03-18T01:01:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mkdir_latency": { 2024-03-18T01:01:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.952 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mknod_latency": { 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "req_mksnap_latency": { 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.953 INFO:teuthology.orchestra.run.smithi106.stdout: "req_open_latency": { 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "req_readdir_latency": { 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rename_latency": { 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.954 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:34.955 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmdir_latency": { 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.956 INFO:teuthology.orchestra.run.smithi106.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setattr_latency": { 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.957 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setlayout_latency": { 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "req_setxattr_latency": { 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.958 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "req_symlink_latency": { 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "req_unlink_latency": { 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:34.959 INFO:teuthology.orchestra.run.smithi106.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_sessions": { 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "session_count": 0, 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "session_add": 0, 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "session_remove": 0, 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_open": 0, 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "sessions_stale": 0, 2024-03-18T01:01:34.960 INFO:teuthology.orchestra.run.smithi106.stdout: "total_load": 0, 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "average_load": 0, 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "avg_session_uptime": 0, 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "mempool": { 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:34.961 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:34.962 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:34.963 INFO:teuthology.orchestra.run.smithi106.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_items": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_bytes": 181337, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_anon_items": 48, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_bytes": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_items": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:34.964 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_items": 162, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "pgmap_items": 0, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_bytes": 8464, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "mds_co_items": 13, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_1_items": 0, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:34.965 INFO:teuthology.orchestra.run.smithi106.stdout: "unittest_2_items": 0 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "objecter": { 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "op_active": 0, 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "op_laggy": 0, 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send": 1596, 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "op_send_bytes": 21, 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "op_resend": 3, 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "op_reply": 1596, 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "op_latency": { 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1596, 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 8.537799835, 2024-03-18T01:01:34.966 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.005349498 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: "op_inflight": 0, 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: "oplen_avg": { 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 1596, 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 1597 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: "op": 1596, 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: "op_r": 1596, 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: "op_w": 0, 2024-03-18T01:01:34.967 INFO:teuthology.orchestra.run.smithi106.stdout: "op_rmw": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "op_pg": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_stat": 795, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_create": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_read": 799, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_write": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writefull": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_writesame": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_append": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_zero": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_truncate": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_delete": 0, 2024-03-18T01:01:34.968 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_mapext": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_getxattr": 1, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_setxattr": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_call": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_watch": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_notify": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:34.969 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "osdop_other": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_active": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_send": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_resend": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "linger_ping": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_active": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_send": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "poolop_resend": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_active": 0, 2024-03-18T01:01:34.970 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_send": 0, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "poolstat_resend": 0, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_active": 0, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_send": 0, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "statfs_resend": 0, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "command_active": 0, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "command_send": 0, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "command_resend": 0, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "map_epoch": 88, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "map_full": 0, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "map_inc": 2, 2024-03-18T01:01:34.971 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_sessions": 4, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_open": 4, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_session_close": 0, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "osd_laggy": 0, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_wr": 0, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_rd": 2, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_del": 0 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "oft": { 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_objs": 0, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_updates": 0, 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: "omap_total_removes": 0 2024-03-18T01:01:34.972 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "osd-slow-ops": { 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "slow_ops_count": 0 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "purge_queue": { 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing": 0, 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executing_high_water": 0, 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed_ops": 0, 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_executed": 0, 2024-03-18T01:01:34.973 INFO:teuthology.orchestra.run.smithi106.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 1841, 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 843389, 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 1841, 2024-03-18T01:01:34.974 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 1841, 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 843389, 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:34.975 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 104857600, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 1596, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 593, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 1596, 2024-03-18T01:01:34.976 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 593, 2024-03-18T01:01:34.977 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:34.977 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.977 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.977 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.977 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:34.977 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.977 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:34.977 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:34.977 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 1024, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 1596, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 1596, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 1596, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 1596, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.978 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 0, 2024-03-18T01:01:34.979 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 0, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: }, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "throttle-write_buf_throttle-0x55578879e5a0": { 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "val": 0, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "max": 3758096384, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "get_started": 0, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "get": 0, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "get_sum": 0, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:34.980 INFO:teuthology.orchestra.run.smithi106.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: "take": 0, 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: "take_sum": 0, 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: "put": 0, 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: "put_sum": 0, 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: "wait": { 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: "avgcount": 0, 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: "sum": 0.000000000, 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: "avgtime": 0.000000000 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout: } 2024-03-18T01:01:34.981 INFO:teuthology.orchestra.run.smithi106.stdout:} 2024-03-18T01:01:35.195 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:35 smithi106 ceph-mon[35899]: pgmap v694: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T01:01:35.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:35 smithi049 ceph-mon[32130]: pgmap v694: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T01:01:35.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:35 smithi107 ceph-mon[36270]: pgmap v694: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-18T01:01:35.765 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-18T01:01:35.766 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-18T01:01:35.766 DEBUG:tasks.check_counter:Getting stats from c 2024-03-18T01:01:35.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-18T01:01:36.196 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:01:37.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:37 smithi107 ceph-mon[36270]: pgmap v695: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-18T01:01:37.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:37 smithi049 ceph-mon[32130]: pgmap v695: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-18T01:01:37.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:37 smithi106 ceph-mon[35899]: pgmap v695: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-18T01:01:38.161 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-03-18T01:01:38.161 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:38.161 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 409, 2024-03-18T01:01:38.161 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 393, 2024-03-18T01:01:38.162 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 645675, 2024-03-18T01:01:38.162 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 633437, 2024-03-18T01:01:38.162 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 44, 2024-03-18T01:01:38.162 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 3, 2024-03-18T01:01:38.162 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.060980605, 2024-03-18T01:01:38.162 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.016381125, 2024-03-18T01:01:38.162 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.037221096, 2024-03-18T01:01:38.162 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.006286976, 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 392, 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.030679067, 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000078262 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 198, 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000039195, 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000197 2024-03-18T01:01:38.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 565872, 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 143008 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 88, 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 255, 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 71623, 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 455889, 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 8, 2024-03-18T01:01:38.164 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 4, 2024-03-18T01:01:38.165 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.016098438, 2024-03-18T01:01:38.165 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.008431801, 2024-03-18T01:01:38.165 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.006743608, 2024-03-18T01:01:38.165 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000489991, 2024-03-18T01:01:38.165 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:38.165 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 254, 2024-03-18T01:01:38.165 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.030602612, 2024-03-18T01:01:38.165 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000120482 2024-03-18T01:01:38.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 127, 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000047215, 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000371 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 379104 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 690, 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 777, 2024-03-18T01:01:38.166 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 259657, 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 228008, 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 8, 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 4, 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.062601125, 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.022397357, 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.039816885, 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.004255805, 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 775, 2024-03-18T01:01:38.167 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.055263185, 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000071307 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1381, 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000403129, 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000291 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 1, 2024-03-18T01:01:38.168 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-MDSRank": { 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 40, 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.001711403, 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000042785 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 2, 2024-03-18T01:01:38.169 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000009188, 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000004594 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": { 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "request": 0, 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "reply": 0, 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "reply_latency": { 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_reply": 0, 2024-03-18T01:01:38.170 INFO:teuthology.orchestra.run.smithi107.stdout: "forward": 0, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_complete": 0, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_keys": 0, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_commit": 11, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_split": 0, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_merge": 0, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes": 15, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_top": 0, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_bottom": 15, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pin_tail": 0, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pinned": 13, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_expired": 6, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_with_caps": 0, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": 0, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "subtrees": 3, 2024-03-18T01:01:38.171 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse": 12, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_hit": 11, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_forward": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_discover": 1, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_lock": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "load_cent": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "q": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "exported": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "exported_inodes": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "imported": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "imported_inodes": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:01:38.172 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_peer_discover": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rfiles": 22, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rbytes": 10738150101, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rsnaps": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_file_inodes": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:01:38.173 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps": 0, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_cap_release": 0, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "process_request_cap_release": 0 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache": { 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update": 0, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update_receipt": 21, 2024-03-18T01:01:38.174 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_try_discover": 11, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_send_discover": 22, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_handle_discover": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_created": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_enqueued": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_migrated": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_started": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_completed": 0, 2024-03-18T01:01:38.175 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_exportdir": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_flush": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_inodestats": 0 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log": { 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "evlrg": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "evadd": 2, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "evex": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "evtrm": 0, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "ev": 2, 2024-03-18T01:01:38.176 INFO:teuthology.orchestra.run.smithi107.stdout: "evexg": 0, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "evexd": 0, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "segadd": 2, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "segex": 0, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "segtrm": 0, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "seg": 2, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "segmjr": 2, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "segexg": 0, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "segexd": 0, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "expos": 4194304, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "wrpos": 4194892, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "rdpos": 0, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "jlat": { 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1, 2024-03-18T01:01:38.177 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.001283733, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.001283733 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "replayed": 0 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mem": { 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "ino": 19, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "ino+": 22, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "ino-": 3, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "dir": 18, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "dir+": 31, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "dir-": 13, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "dn": 15, 2024-03-18T01:01:38.178 INFO:teuthology.orchestra.run.smithi107.stdout: "dn+": 21, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "dn-": 6, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "cap": 0, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "cap+": 0, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "cap-": 0, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "rss": 52620, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "heap": 223612 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_server": { 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_client_request": 0, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_server_request": 1, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_request": 0, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_session": 0, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_peer_request": 2, 2024-03-18T01:01:38.179 INFO:teuthology.orchestra.run.smithi107.stdout: "req_create_latency": { 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getattr_latency": { 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "req_link_latency": { 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookup_latency": { 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupino_latency": { 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupname_latency": { 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.182 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lssnap_latency": { 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.183 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mkdir_latency": { 2024-03-18T01:01:38.184 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.184 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.184 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mknod_latency": { 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mksnap_latency": { 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "req_open_latency": { 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.185 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: "req_readdir_latency": { 2024-03-18T01:01:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rename_latency": { 2024-03-18T01:01:38.186 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.187 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmdir_latency": { 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setattr_latency": { 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setlayout_latency": { 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setxattr_latency": { 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "req_symlink_latency": { 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.190 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "req_unlink_latency": { 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.191 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessions": { 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "session_count": 0, 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "session_add": 0, 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "session_remove": 0, 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_open": 0, 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_stale": 0, 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "total_load": 0, 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "average_load": 0, 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "avg_session_uptime": 0, 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:38.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:38.193 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:38.194 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:38.195 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 250143, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 81, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:38.196 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 162, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 76373, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 302, 2024-03-18T01:01:38.197 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter": { 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: "op_active": 0, 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: "op_laggy": 0, 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send": 35, 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send_bytes": 10710, 2024-03-18T01:01:38.198 INFO:teuthology.orchestra.run.smithi107.stdout: "op_resend": 29, 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: "op_reply": 35, 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: "op_latency": { 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 35, 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.194996914, 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.005571340 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: "op_inflight": 0, 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: "oplen_avg": { 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 35, 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 57 2024-03-18T01:01:38.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "op": 35, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "op_r": 0, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "op_w": 35, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "op_rmw": 0, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "op_pg": 0, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_stat": 0, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_create": 10, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_read": 0, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_write": 1, 2024-03-18T01:01:38.200 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writefull": 5, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writesame": 0, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_append": 0, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_zero": 0, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_truncate": 0, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_delete": 6, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_mapext": 0, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_getxattr": 0, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_setxattr": 20, 2024-03-18T01:01:38.201 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_call": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_watch": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_notify": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_other": 0, 2024-03-18T01:01:38.202 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_active": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_send": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_resend": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_ping": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_active": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_send": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_resend": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_active": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_send": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_resend": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_active": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_send": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_resend": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "command_active": 0, 2024-03-18T01:01:38.203 INFO:teuthology.orchestra.run.smithi107.stdout: "command_send": 0, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "command_resend": 0, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 88, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "map_full": 0, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "map_inc": 2, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_sessions": 8, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_open": 8, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_close": 0, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_laggy": 0, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_wr": 14, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_rd": 0, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_del": 1 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "oft": { 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_objs": 1, 2024-03-18T01:01:38.204 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_updates": 0, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_removes": 0 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "osd-slow-ops": { 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_ops_count": 0 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "purge_queue": { 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing": 0, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_high_water": 0, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed_ops": 0, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed": 0, 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 1183, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 853651, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 1183, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 1183, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 853651, 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:38.206 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:38.207 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 35, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 10710, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 35, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 10710, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:38.208 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 1024, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 35, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 35, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 35, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 35, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.209 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 0, 2024-03-18T01:01:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 0, 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle-0x56101c13c500": { 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 2, 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 588, 2024-03-18T01:01:38.211 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 2, 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 1, 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 588, 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:38.212 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-03-18T01:01:39.180 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:39 smithi107 ceph-mon[36270]: pgmap v696: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-18T01:01:39.276 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-18T01:01:39.276 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-18T01:01:39.276 DEBUG:tasks.check_counter:Getting stats from f 2024-03-18T01:01:39.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-18T01:01:39.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:39 smithi049 ceph-mon[32130]: pgmap v696: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-18T01:01:39.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:39 smithi106 ceph-mon[35899]: pgmap v696: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-18T01:01:39.597 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:01:41.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:41 smithi049 ceph-mon[32130]: pgmap v697: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T01:01:41.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:41 smithi107 ceph-mon[36270]: pgmap v697: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T01:01:41.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:41 smithi106 ceph-mon[35899]: pgmap v697: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T01:01:41.727 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 177, 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 183, 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 140924, 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 186123, 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 15, 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 5, 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.018775051, 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.005399618, 2024-03-18T01:01:41.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.010449463, 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000410155, 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 177, 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.023437906, 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000132417 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 296, 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000085422, 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000288 2024-03-18T01:01:41.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 953, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 1017, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 811248, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 490709, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 7, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 4, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.092388553, 2024-03-18T01:01:41.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.035528873, 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.051751140, 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.010044846, 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1022, 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.087548218, 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000085663 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1354, 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000433556, 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000320 2024-03-18T01:01:41.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 554640, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 144128 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 64, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 234, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 13498, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 758968, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 41, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 7, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.030741408, 2024-03-18T01:01:41.732 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.009277563, 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.018790759, 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.000487109, 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 231, 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.020294561, 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000087855 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 44, 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000013800, 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000313 2024-03-18T01:01:41.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 379280 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 1, 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-MDSRank": { 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-03-18T01:01:41.734 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 37, 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.001501713, 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000040586 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 2, 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000005254, 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000002627 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:41.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": { 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "request": 2, 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "reply": 1, 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "reply_latency": { 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1, 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000083481, 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000083481 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_reply": 0, 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "forward": 1, 2024-03-18T01:01:41.736 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_complete": 0, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_keys": 0, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_commit": 11, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_split": 0, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_merge": 0, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes": 15, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_top": 0, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_bottom": 15, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pin_tail": 0, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pinned": 13, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_expired": 6, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_with_caps": 1, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": 1, 2024-03-18T01:01:41.737 INFO:teuthology.orchestra.run.smithi107.stdout: "subtrees": 3, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse": 12, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_hit": 11, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_forward": 0, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_discover": 1, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_lock": 0, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "load_cent": 1000, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "q": 1, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "exported": 0, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "exported_inodes": 0, 2024-03-18T01:01:41.738 INFO:teuthology.orchestra.run.smithi107.stdout: "imported": 0, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "imported_inodes": 0, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_peer_discover": 1, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rfiles": 22, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rbytes": 10738150101, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rsnaps": 0, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:41.739 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_file_inodes": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:01:41.740 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_cap_release": 0, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "process_request_cap_release": 0 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache": { 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update": 0, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update_receipt": 21, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_try_discover": 11, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_send_discover": 22, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_handle_discover": 0, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays": 0, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:41.741 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_created": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_enqueued": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_migrated": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_started": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_completed": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:01:41.742 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_exportdir": 0, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_flush": 0, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_inodestats": 0 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log": { 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "evlrg": 0, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "evadd": 3, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "evex": 0, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "evtrm": 0, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "ev": 3, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "evexg": 0, 2024-03-18T01:01:41.743 INFO:teuthology.orchestra.run.smithi107.stdout: "evexd": 0, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "segadd": 2, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "segex": 0, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "segtrm": 0, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "seg": 2, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "segmjr": 2, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "segexg": 0, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "segexd": 0, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "expos": 4194304, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "wrpos": 4195200, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "rdpos": 0, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "jlat": { 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 2, 2024-03-18T01:01:41.744 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.003078933, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.001539466 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "replayed": 0 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mem": { 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "ino": 19, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "ino+": 22, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "ino-": 3, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "dir": 18, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "dir+": 31, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "dir-": 13, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "dn": 15, 2024-03-18T01:01:41.745 INFO:teuthology.orchestra.run.smithi107.stdout: "dn+": 21, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "dn-": 6, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "cap": 1, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "cap+": 1, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "cap-": 0, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "rss": 51052, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "heap": 223612 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_server": { 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_client_request": 2, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_server_request": 1, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_request": 2, 2024-03-18T01:01:41.746 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_session": 10, 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_peer_request": 2, 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "req_create_latency": { 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getattr_latency": { 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.747 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "req_link_latency": { 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookup_latency": { 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupino_latency": { 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1, 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000083481, 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000083481 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupname_latency": { 2024-03-18T01:01:41.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.751 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lssnap_latency": { 2024-03-18T01:01:41.751 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.751 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.751 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.751 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mkdir_latency": { 2024-03-18T01:01:41.752 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.752 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.752 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.752 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mknod_latency": { 2024-03-18T01:01:41.752 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.752 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mksnap_latency": { 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: "req_open_latency": { 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: "req_readdir_latency": { 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rename_latency": { 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.754 INFO:teuthology.orchestra.run.smithi107.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmdir_latency": { 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.755 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.756 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setattr_latency": { 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.757 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setlayout_latency": { 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setxattr_latency": { 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "req_symlink_latency": { 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "req_unlink_latency": { 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:41.759 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessions": { 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "session_count": 1, 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "session_add": 1, 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "session_remove": 0, 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_open": 1, 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_stale": 0, 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "total_load": 1, 2024-03-18T01:01:41.760 INFO:teuthology.orchestra.run.smithi107.stdout: "average_load": 1, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "avg_session_uptime": 687, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:41.761 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:41.762 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-03-18T01:01:41.763 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 250477, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 85, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:41.764 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 162, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 76749, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 305, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:41.765 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter": { 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "op_active": 0, 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "op_laggy": 0, 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send": 37, 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send_bytes": 11108, 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "op_resend": 28, 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "op_reply": 37, 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "op_latency": { 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 37, 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.199996859, 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.005405320 2024-03-18T01:01:41.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "op_inflight": 0, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "oplen_avg": { 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 37, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 59 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "op": 37, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "op_r": 0, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "op_w": 37, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "op_rmw": 0, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "op_pg": 0, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_stat": 0, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_create": 10, 2024-03-18T01:01:41.767 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_read": 0, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_write": 2, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writefull": 6, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writesame": 0, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_append": 0, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_zero": 0, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_truncate": 0, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_delete": 6, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_mapext": 0, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_getxattr": 0, 2024-03-18T01:01:41.768 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_setxattr": 20, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_call": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_watch": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_notify": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_other": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_active": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_send": 0, 2024-03-18T01:01:41.769 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_resend": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_ping": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_active": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_send": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_resend": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_active": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_send": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_resend": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_active": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_send": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_resend": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "command_active": 0, 2024-03-18T01:01:41.770 INFO:teuthology.orchestra.run.smithi107.stdout: "command_send": 0, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "command_resend": 0, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 88, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "map_full": 0, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "map_inc": 2, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_sessions": 9, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_open": 9, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_close": 0, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_laggy": 0, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_wr": 14, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_rd": 0, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_del": 1 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.771 INFO:teuthology.orchestra.run.smithi107.stdout: "oft": { 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_objs": 1, 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_updates": 0, 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_removes": 0 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "osd-slow-ops": { 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_ops_count": 0 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "purge_queue": { 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:01:41.772 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing": 0, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_high_water": 0, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed_ops": 0, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed": 0, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 1190, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 842052, 2024-03-18T01:01:41.773 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 1190, 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 1190, 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 842052, 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:41.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 37, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 11108, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 37, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 11108, 2024-03-18T01:01:41.775 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 1024, 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:41.776 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:41.777 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:41.777 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 37, 2024-03-18T01:01:41.777 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 37, 2024-03-18T01:01:41.777 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 37, 2024-03-18T01:01:41.777 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 37, 2024-03-18T01:01:41.777 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:41.777 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.777 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.777 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:41.778 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 0, 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 0, 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle-0x557f18aac500": { 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-03-18T01:01:41.779 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 3, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 896, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 3, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 2, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 896, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:41.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:41.781 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:41.781 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-03-18T01:01:42.612 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-18T01:01:42.613 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-18T01:01:42.613 DEBUG:tasks.check_counter:Getting stats from i 2024-03-18T01:01:42.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-18T01:01:42.928 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:01:43.306 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:43 smithi107 ceph-mon[36270]: pgmap v698: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-18T01:01:43.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:43 smithi049 ceph-mon[32130]: pgmap v698: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-18T01:01:43.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:43 smithi106 ceph-mon[35899]: pgmap v698: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-18T01:01:44.810 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 67159, 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 96829, 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 18936468, 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 42981342, 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 47, 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 7, 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 3.828049537, 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 1.880743077, 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 2.073404535, 2024-03-18T01:01:44.811 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.287917641, 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 96823, 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 3.032045631, 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000031315 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 89423, 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.020505435, 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000229 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 1584, 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 1440 2024-03-18T01:01:44.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 46061, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 87685, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 9942372, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 24000694, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 9, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 5, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 3.671700512, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 1.517542146, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 2.299290995, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.048544561, 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:44.813 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 87689, 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 3.057036296, 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000034862 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 127626, 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.026885229, 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000210 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 578224, 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 156560 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 9795, 2024-03-18T01:01:44.814 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 9964, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 2475243, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 13298727, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 11, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 7, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.959416170, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.417481890, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.201354545, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.272823942, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 9962, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.431622524, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000043326 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.815 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 167, 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000030265, 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000181 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 380944 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 1, 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-MDSRank": { 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-03-18T01:01:44.816 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 19037, 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 11.897566354, 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000624970 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 4643, 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.193051331, 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000041579 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.817 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": { 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "request": 23233, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "reply": 23231, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "reply_latency": { 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 23231, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 159.029147750, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.006845557 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_reply": 0, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "forward": 2, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_complete": 0, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_keys": 0, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_commit": 11, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_split": 0, 2024-03-18T01:01:44.818 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_merge": 0, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes": 15, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_top": 0, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_bottom": 1, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pin_tail": 14, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pinned": 14, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_expired": 12, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_with_caps": 1, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": 1, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "subtrees": 3, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse": 46471, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_hit": 46442, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_forward": 0, 2024-03-18T01:01:44.819 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_discover": 3, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_lock": 0, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "load_cent": 1000, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "q": 1, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "exported": 1, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "exported_inodes": 6, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "imported": 2, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "imported_inodes": 18, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_backtrace_fetch": 44, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_peer_discover": 0, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rfiles": 22, 2024-03-18T01:01:44.820 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rbytes": 10738150101, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rsnaps": 0, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_fetch": 44, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_inodes": 1, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dirfrag_rstats": 1, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_file_inodes": 43, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_inode_file_caps": 10, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_revoke": 35055, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_grant": 21138, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:44.821 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps": 38187, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps_dirty": 11633, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_cap_release": 12, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "process_request_cap_release": 33 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache": { 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update": 30, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update_receipt": 0, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_try_discover": 0, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_send_discover": 4, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_handle_discover": 30, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays": 0, 2024-03-18T01:01:44.822 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_created": 18, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_enqueued": 18, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_migrated": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_started": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_completed": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_exportdir": 1, 2024-03-18T01:01:44.823 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_flush": 0, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_inodestats": 0 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log": { 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "evlrg": 0, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "evadd": 11718, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "evex": 0, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "evtrm": 0, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "ev": 11718, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "evexg": 0, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "evexd": 0, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "segadd": 15, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "segex": 0, 2024-03-18T01:01:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: "segtrm": 0, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "seg": 15, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "segmjr": 2, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "segexg": 0, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "segexd": 0, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "expos": 4194304, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "wrpos": 18101950, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "rdpos": 0, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "jlat": { 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 11648, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 61.260981683, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.005259356 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "replayed": 0 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.825 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mem": { 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "ino": 19, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "ino+": 40, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "ino-": 21, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "dir": 18, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "dir+": 32, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "dir-": 14, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "dn": 15, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "dn+": 57, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "dn-": 42, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "cap": 1, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "cap+": 26, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "cap-": 25, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "rss": 53740, 2024-03-18T01:01:44.826 INFO:teuthology.orchestra.run.smithi107.stdout: "heap": 223612 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_server": { 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_client_request": 44380, 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_server_request": 6, 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_request": 23233, 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_session": 14, 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_peer_request": 29, 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "req_create_latency": { 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getattr_latency": { 2024-03-18T01:01:44.827 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 23200, 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 158.966772691, 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.006852016 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "req_link_latency": { 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.828 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookup_latency": { 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 11, 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.003981743, 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000361976 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupino_latency": { 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.829 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupname_latency": { 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.831 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lssnap_latency": { 2024-03-18T01:01:44.831 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.831 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.831 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.831 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mkdir_latency": { 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mknod_latency": { 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mksnap_latency": { 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "req_open_latency": { 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "req_readdir_latency": { 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 2, 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000880283, 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000440141 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rename_latency": { 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.833 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmdir_latency": { 2024-03-18T01:01:44.834 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1, 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.009014936, 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.009014936 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setattr_latency": { 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:44.836 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setlayout_latency": { 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setxattr_latency": { 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.837 INFO:teuthology.orchestra.run.smithi107.stdout: "req_symlink_latency": { 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "req_unlink_latency": { 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 17, 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.048498097, 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.002852829 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.838 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessions": { 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "session_count": 1, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "session_add": 2, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "session_remove": 1, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_open": 1, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_stale": 0, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "total_load": 17412, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "average_load": 17412, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "avg_session_uptime": 698, 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:44.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:44.840 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:44.841 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 298192, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 124, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:44.842 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 162, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 77197, 2024-03-18T01:01:44.843 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 341, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter": { 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "op_active": 0, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "op_laggy": 0, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send": 16921, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send_bytes": 13924947, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "op_resend": 33, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "op_reply": 16921, 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "op_latency": { 2024-03-18T01:01:44.844 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 16921, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 92.678630784, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.005477136 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "op_inflight": 0, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "oplen_avg": { 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 16921, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 16954 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "op": 16921, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "op_r": 49, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "op_w": 16872, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "op_rmw": 0, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "op_pg": 0, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_stat": 0, 2024-03-18T01:01:44.845 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_create": 10, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_read": 5, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_write": 11653, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writefull": 20, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writesame": 0, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_append": 0, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_zero": 0, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_truncate": 0, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_delete": 5153, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_mapext": 0, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_getxattr": 44, 2024-03-18T01:01:44.846 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_setxattr": 20, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_call": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_watch": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_notify": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_other": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_active": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_send": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_resend": 0, 2024-03-18T01:01:44.847 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_ping": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_active": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_send": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_resend": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_active": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_send": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_resend": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_active": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_send": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_resend": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "command_active": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "command_send": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "command_resend": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 88, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "map_full": 0, 2024-03-18T01:01:44.848 INFO:teuthology.orchestra.run.smithi107.stdout: "map_inc": 2, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_sessions": 12, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_open": 12, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_close": 0, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_laggy": 0, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_wr": 43, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_rd": 0, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_del": 6 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "oft": { 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_objs": 1, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_updates": 0, 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_removes": 0 2024-03-18T01:01:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "osd-slow-ops": { 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_ops_count": 0 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "purge_queue": { 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops_high_water": 1026, 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing": 0, 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_high_water": 6, 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed_ops": 5138, 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed": 18, 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:44.850 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 123011, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 21247536, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 123011, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 123011, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 21247536, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.851 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 16921, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 13926765, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 16921, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 13926765, 2024-03-18T01:01:44.852 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 1024, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:44.853 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 16921, 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 16921, 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 16921, 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 16921, 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-03-18T01:01:44.854 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 18, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 1818, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 18, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 5, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 1818, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:44.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle-0x56132fe46500": { 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 11718, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 13907646, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 11718, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 11648, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 13907646, 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:44.856 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:44.857 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:44.857 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:44.857 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:44.857 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:44.857 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-03-18T01:01:45.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:45 smithi049 ceph-mon[32130]: pgmap v699: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T01:01:45.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:45 smithi107 ceph-mon[36270]: pgmap v699: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T01:01:45.450 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:45 smithi106 ceph-mon[35899]: pgmap v699: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T01:01:45.632 INFO:tasks.check_counter:Daemon mds.i mds.exported=1 2024-03-18T01:01:45.632 INFO:tasks.check_counter:Daemon mds.i mds.imported=2 2024-03-18T01:01:45.632 DEBUG:tasks.check_counter:Getting stats from l 2024-03-18T01:01:45.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-18T01:01:45.918 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T01:01:47.307 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:47 smithi107 ceph-mon[36270]: pgmap v700: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-03-18T01:01:47.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:47 smithi049 ceph-mon[32130]: pgmap v700: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-03-18T01:01:47.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:47 smithi106 ceph-mon[35899]: pgmap v700: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 9 op/s 2024-03-18T01:01:47.726 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 957, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 957, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 4507616, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 303509, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 6, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 2, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.089652211, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.025873921, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.025283327, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.031060060, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 956, 2024-03-18T01:01:47.727 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.048060661, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000050272 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 1412, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 1585, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 8746595, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 829594, 2024-03-18T01:01:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 5, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 4, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.110164601, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.033711513, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.028831205, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.041134585, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 1583, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.056136874, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000035462 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.729 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 381152 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_messages": 313, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages": 303, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_bytes": 1885872, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_bytes": 163676, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_created_connections": 4, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_active_connections": 3, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_total_time": 0.039081704, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_send_time": 0.012731229, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_recv_time": 0.018711161, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_running_fast_dispatch_time": 0.007831525, 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T01:01:47.730 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 302, 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.015055327, 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000049852 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_handle_ack_lat": { 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_recv_encrypted_bytes": 544080, 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "msgr_send_encrypted_bytes": 142688 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "cct": { 2024-03-18T01:01:47.731 INFO:teuthology.orchestra.run.smithi107.stdout: "total_workers": 1, 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "unhealthy_workers": 0 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-MDSRank": { 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 4059, 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.107291441, 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000026432 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "finisher-PurgeQueue": { 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "queue_len": 0, 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "complete_latency": { 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.732 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": { 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "request": 0, 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "reply": 0, 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "reply_latency": { 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_reply": 0, 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "forward": 0, 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_complete": 0, 2024-03-18T01:01:47.733 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_fetch_keys": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_commit": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_split": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_merge": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes": 19, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_top": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_bottom": 19, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pin_tail": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_pinned": 19, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_expired": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "inodes_with_caps": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "caps": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "subtrees": 1, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_hit": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_forward": 0, 2024-03-18T01:01:47.734 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_discover": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_dir_fetch": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_remote_ino": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "traverse_lock": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "load_cent": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "q": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "exported": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "exported_inodes": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "imported": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "imported_inodes": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_dir_fetch": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_backtrace_fetch": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "openino_peer_discover": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rfiles": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rbytes": 0, 2024-03-18T01:01:47.735 INFO:teuthology.orchestra.run.smithi107.stdout: "root_rsnaps": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_set_tag": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_inotable_repaired": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_inodes": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "scrub_file_inodes": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_inode_file_caps": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_grant": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps": 0, 2024-03-18T01:01:47.736 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_caps_dirty": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_cap_release": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "process_request_cap_release": 0 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache": { 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_update_receipt": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_try_discover": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_send_discover": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "dir_handle_discover": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_delayed": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "num_strays_enqueuing": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_created": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_enqueued": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_reintegrated": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "strays_migrated": 0, 2024-03-18T01:01:47.737 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_processing": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_enqueued": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "num_recovering_prioritized": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_started": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "recovery_completed": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_path": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_quiesce_inode": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_exportdir": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_flush": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragmentdir": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_fragstats": 0, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "ireq_inodestats": 0 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log": { 2024-03-18T01:01:47.738 INFO:teuthology.orchestra.run.smithi107.stdout: "evlrg": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "evadd": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "evex": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "evtrm": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "ev": 11718, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "evexg": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "evexd": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "segadd": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "segex": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "segtrm": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "seg": 15, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "segmjr": 2, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "segexg": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "segexd": 0, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "expos": 4194304, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "wrpos": 18101950, 2024-03-18T01:01:47.739 INFO:teuthology.orchestra.run.smithi107.stdout: "rdpos": 18100812, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "jlat": { 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "replayed": 11718 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mem": { 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "ino": 4, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "ino+": 31, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "ino-": 27, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "dir": 2, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "dir+": 13, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "dir-": 11, 2024-03-18T01:01:47.740 INFO:teuthology.orchestra.run.smithi107.stdout: "dn": 19, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "dn+": 46, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "dn-": 27, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "cap": 0, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "cap+": 0, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "cap-": 0, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "rss": 49852, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "heap": 215420 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_server": { 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_client_request": 0, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "dispatch_server_request": 0, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_request": 0, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_client_session": 0, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "handle_peer_request": 0, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "req_create_latency": { 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.741 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getattr_latency": { 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getfilelock_latency": { 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.742 INFO:teuthology.orchestra.run.smithi107.stdout: "req_link_latency": { 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookup_latency": { 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookuphash_latency": { 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.743 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupino_latency": { 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupname_latency": { 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupparent_latency": { 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lookupsnap_latency": { 2024-03-18T01:01:47.744 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "req_lssnap_latency": { 2024-03-18T01:01:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.745 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mkdir_latency": { 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mknod_latency": { 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "req_mksnap_latency": { 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.746 INFO:teuthology.orchestra.run.smithi107.stdout: "req_open_latency": { 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "req_readdir_latency": { 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rename_latency": { 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.747 INFO:teuthology.orchestra.run.smithi107.stdout: "req_renamesnap_latency": { 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "req_snapdiff_latency": { 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmdir_latency": { 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmsnap_latency": { 2024-03-18T01:01:47.748 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "req_rmxattr_latency": { 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setattr_latency": { 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setdirlayout_latency": { 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.749 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setfilelock_latency": { 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setlayout_latency": { 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "req_setxattr_latency": { 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.750 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "req_symlink_latency": { 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "req_unlink_latency": { 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_revoke_eviction": 0, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "cap_acquisition_throttle": 0, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "req_getvxattr_latency": { 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.751 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessions": { 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "session_count": 1, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "session_add": 2, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "session_remove": 1, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_open": 1, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "sessions_stale": 0, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "total_load": 0, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "average_load": 0, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "avg_session_uptime": 2070, 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: "mdthresh_evicted": 0 2024-03-18T01:01:47.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool": { 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_bytes": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bloom_filter_items": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_items": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_data_items": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_other_items": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T01:01:47.753 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_bytes": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_items": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_bytes": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blob_items": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_items": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_bytes": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_txc_items": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_bytes": 0, 2024-03-18T01:01:47.754 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_writing_items": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_bytes": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_items": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_reader_items": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_file_writer_items": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_bytes": 197463, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_anon_items": 54, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_bytes": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "buffer_meta_items": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bytes": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_items": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_bytes": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mapbl_items": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_bytes": 0, 2024-03-18T01:01:47.755 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pglog_items": 0, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_bytes": 7720, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_items": 162, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "osdmap_mapping_items": 0, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_bytes": 0, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "pgmap_items": 0, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_bytes": 25360, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_co_items": 104, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_bytes": 0, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_1_items": 0, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_bytes": 0, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "unittest_2_items": 0 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter": { 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "op_active": 0, 2024-03-18T01:01:47.756 INFO:teuthology.orchestra.run.smithi107.stdout: "op_laggy": 0, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send": 2429, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "op_send_bytes": 21, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "op_resend": 3, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "op_reply": 2429, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "op_latency": { 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 2429, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 3.738942240, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.001539292 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "op_inflight": 0, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "oplen_avg": { 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 2429, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 2430 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "op": 2429, 2024-03-18T01:01:47.757 INFO:teuthology.orchestra.run.smithi107.stdout: "op_r": 2429, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "op_w": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "op_rmw": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "op_pg": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_stat": 1544, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_create": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_read": 883, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_write": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writefull": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_writesame": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_append": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_zero": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_truncate": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_delete": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_mapext": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_sparse_read": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_clonerange": 0, 2024-03-18T01:01:47.758 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_getxattr": 1, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_setxattr": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_cmpxattr": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_rmxattr": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_resetxattrs": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_call": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_watch": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_notify": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_pgls_filter": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "osdop_other": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_active": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_send": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_resend": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "linger_ping": 0, 2024-03-18T01:01:47.759 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_active": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_send": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "poolop_resend": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_active": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_send": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "poolstat_resend": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_active": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_send": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "statfs_resend": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "command_active": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "command_send": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "command_resend": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "map_epoch": 88, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "map_full": 0, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "map_inc": 2, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_sessions": 6, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_open": 6, 2024-03-18T01:01:47.760 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_session_close": 0, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_laggy": 0, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_wr": 0, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_rd": 2, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_del": 0 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "oft": { 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_objs": 0, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_kv_pairs": 0, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_updates": 0, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "omap_total_removes": 0 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "osd-slow-ops": { 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "slow_ops_count": 0 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "purge_queue": { 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops": 0, 2024-03-18T01:01:47.761 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing": 0, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executing_high_water": 0, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed_ops": 0, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_executed": 0, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "pq_item_in_journal": 0 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 2678, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 14883387, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:47.762 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 2678, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 2678, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 14883387, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_bytes": { 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 104857600, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:47.763 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 2429, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 13907651, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 2429, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 13907651, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-objecter_ops": { 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 1024, 2024-03-18T01:01:47.764 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 2429, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 2429, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 2429, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 2429, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle": { 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:47.765 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-03-18T01:01:47.766 INFO:teuthology.orchestra.run.smithi107.stdout: "throttle-write_buf_throttle-0x558d268145a0": { 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "val": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "max": 3758096384, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "get_started": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "get": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "get_sum": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_fail": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "get_or_fail_success": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "take": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "take_sum": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "put": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "put_sum": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "wait": { 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "avgcount": 0, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "sum": 0.000000000, 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: "avgtime": 0.000000000 2024-03-18T01:01:47.767 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-03-18T01:01:47.768 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-03-18T01:01:48.576 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-18T01:01:48.577 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-18T01:01:48.577 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-18T01:01:48.587 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-18T01:01:48.587 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T01:01:48.587 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:01:48.621 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-03-18T01:01:48.621 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T01:01:48.680 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:01:48.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.844+0000 7f0c09f53640 1 -- 172.21.15.49:0/322065595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c041075c0 msgr2=0x7f0c041099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:48.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.844+0000 7f0c09f53640 1 --2- 172.21.15.49:0/322065595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c041075c0 0x7f0c041099b0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009ca0 tx=0x7f0bfc02c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:48.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.844+0000 7f0c09f53640 1 -- 172.21.15.49:0/322065595 shutdown_connections 2024-03-18T01:01:48.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.844+0000 7f0c09f53640 1 --2- 172.21.15.49:0/322065595 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c04104c90 0x7f0c04107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:48.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.844+0000 7f0c09f53640 1 --2- 172.21.15.49:0/322065595 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c041075c0 0x7f0c041099b0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:48.846 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.844+0000 7f0c09f53640 1 --2- 172.21.15.49:0/322065595 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c04102360 0x7f0c04104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.844+0000 7f0c09f53640 1 -- 172.21.15.49:0/322065595 >> 172.21.15.49:0/322065595 conn(0x7f0c040fbd10 msgr2=0x7f0c040fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.845+0000 7f0c09f53640 1 -- 172.21.15.49:0/322065595 shutdown_connections 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.845+0000 7f0c09f53640 1 -- 172.21.15.49:0/322065595 wait complete. 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.845+0000 7f0c09f53640 1 Processor -- start 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.845+0000 7f0c09f53640 1 -- start start 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c09f53640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04102360 0x7f0c04197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c09f53640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c04104c90 0x7f0c04197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c09f53640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c041075c0 0x7f0c0419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c09f53640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0410d490 con 0x7f0c04102360 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c09f53640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0410d310 con 0x7f0c04104c90 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c09f53640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f0c0410d610 con 0x7f0c041075c0 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c037fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04102360 0x7f0c04197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c02ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c04104c90 0x7f0c04197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:48.847 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c03fff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c041075c0 0x7f0c0419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c037fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04102360 0x7f0c04197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60588/0 (socket says 172.21.15.49:60588) 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c02ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c04104c90 0x7f0c04197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48570/0 (socket says 172.21.15.49:48570) 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c037fe640 1 -- 172.21.15.49:0/261685628 learned_addr learned my addr 172.21.15.49:0/261685628 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c037fe640 1 -- 172.21.15.49:0/261685628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c041075c0 msgr2=0x7f0c0419f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c037fe640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c041075c0 0x7f0c0419f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c037fe640 1 -- 172.21.15.49:0/261685628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c04104c90 msgr2=0x7f0c04197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c037fe640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c04104c90 0x7f0c04197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c037fe640 1 -- 172.21.15.49:0/261685628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0419f6d0 con 0x7f0c04102360 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c037fe640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04102360 0x7f0c04197ab0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400e9f0 tx=0x7f0bf400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c00ff9640 1 -- 172.21.15.49:0/261685628 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf400cd80 con 0x7f0c04102360 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c00ff9640 1 -- 172.21.15.49:0/261685628 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bf4010910 con 0x7f0c04102360 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.846+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0419f9c0 con 0x7f0c04102360 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.847+0000 7f0c00ff9640 1 -- 172.21.15.49:0/261685628 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4018990 con 0x7f0c04102360 2024-03-18T01:01:48.848 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.847+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c0419ff10 con 0x7f0c04102360 2024-03-18T01:01:48.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.848+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c04068a20 con 0x7f0c04102360 2024-03-18T01:01:48.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.849+0000 7f0c00ff9640 1 -- 172.21.15.49:0/261685628 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0bf4010ac0 con 0x7f0c04102360 2024-03-18T01:01:48.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.849+0000 7f0c00ff9640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0bec0a2800 0x7f0bec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:48.849 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.850+0000 7f0c02ffd640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0bec0a2800 0x7f0bec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:48.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.850+0000 7f0c00ff9640 1 -- 172.21.15.49:0/261685628 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f0bf40cf190 con 0x7f0c04102360 2024-03-18T01:01:48.850 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.850+0000 7f0c02ffd640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0bec0a2800 0x7f0bec0a4cc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0c04198fc0 tx=0x7f0bf8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:48.852 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:48.853+0000 7f0c00ff9640 1 -- 172.21.15.49:0/261685628 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0bf409bbc0 con 0x7f0c04102360 2024-03-18T01:01:49.040 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.040+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c040623e0 con 0x7f0c04102360 2024-03-18T01:01:49.041 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.042+0000 7f0c00ff9640 1 -- 172.21.15.49:0/261685628 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7f0bf409fc60 con 0x7f0c04102360 2024-03-18T01:01:49.042 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T01:01:49.042 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0bec0a2800 msgr2=0x7f0bec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0bec0a2800 0x7f0bec0a4cc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0c04198fc0 tx=0x7f0bf8005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04102360 msgr2=0x7f0c04197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04102360 0x7f0c04197ab0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400e9f0 tx=0x7f0bf400eeb0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 shutdown_connections 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f0bec0a2800 0x7f0bec0a4cc0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f0c04104c90 0x7f0c04197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c041075c0 0x7f0c0419f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 --2- 172.21.15.49:0/261685628 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f0c04102360 0x7f0c04197ab0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 >> 172.21.15.49:0/261685628 conn(0x7f0c040fbd10 msgr2=0x7f0c04102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 shutdown_connections 2024-03-18T01:01:49.044 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.044+0000 7f0c09f53640 1 -- 172.21.15.49:0/261685628 wait complete. 2024-03-18T01:01:49.055 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T00:43:27.910351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T00:48:04.316609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T00:48:05.271815+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T01:01:49.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T01:01:49.170 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.168+0000 7fef0f07e640 1 -- 172.21.15.49:0/3499523223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 msgr2=0x7fef08101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.168+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3499523223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 0x7fef08101c20 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7feefc009900 tx=0x7feefc02bc30 comp rx=0 tx=0).stop 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.168+0000 7fef0f07e640 1 -- 172.21.15.49:0/3499523223 shutdown_connections 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.168+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3499523223 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef081033b0 0x7fef08109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.168+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3499523223 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef081029f0 0x7fef08102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.168+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3499523223 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 0x7fef08101c20 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.168+0000 7fef0f07e640 1 -- 172.21.15.49:0/3499523223 >> 172.21.15.49:0/3499523223 conn(0x7fef080fcd70 msgr2=0x7fef080ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.168+0000 7fef0f07e640 1 -- 172.21.15.49:0/3499523223 shutdown_connections 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.168+0000 7fef0f07e640 1 -- 172.21.15.49:0/3499523223 wait complete. 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0f07e640 1 Processor -- start 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0f07e640 1 -- start start 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0f07e640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 0x7fef08199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.171 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0f07e640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef081029f0 0x7fef0819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0f07e640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef081033b0 0x7fef081a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0f07e640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fef0810d470 con 0x7fef08101800 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0f07e640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fef0810d2f0 con 0x7fef081029f0 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0f07e640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fef0810d5f0 con 0x7fef081033b0 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0ce1a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 0x7fef08199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0ce1a640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 0x7fef08199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60604/0 (socket says 172.21.15.49:60604) 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0ce1a640 1 -- 172.21.15.49:0/3052636898 learned_addr learned my addr 172.21.15.49:0/3052636898 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0d61b640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef081033b0 0x7fef081a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef07fff640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef081029f0 0x7fef0819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0ce1a640 1 -- 172.21.15.49:0/3052636898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef081033b0 msgr2=0x7fef081a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.172 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0ce1a640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef081033b0 0x7fef081a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0ce1a640 1 -- 172.21.15.49:0/3052636898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef081029f0 msgr2=0x7fef0819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0ce1a640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef081029f0 0x7fef0819a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0ce1a640 1 -- 172.21.15.49:0/3052636898 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef081a1990 con 0x7fef08101800 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.169+0000 7fef0ce1a640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 0x7fef08199e10 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7feefc000c00 tx=0x7feefc02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.170+0000 7fef05ffb640 1 -- 172.21.15.49:0/3052636898 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc041070 con 0x7fef08101800 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.170+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef081a1c20 con 0x7fef08101800 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.170+0000 7fef05ffb640 1 -- 172.21.15.49:0/3052636898 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feefc004450 con 0x7fef08101800 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.170+0000 7fef05ffb640 1 -- 172.21.15.49:0/3052636898 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feefc003df0 con 0x7fef08101800 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.170+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef081a2110 con 0x7fef08101800 2024-03-18T01:01:49.173 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.171+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef081a22e0 con 0x7fef08101800 2024-03-18T01:01:49.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.173+0000 7fef05ffb640 1 -- 172.21.15.49:0/3052636898 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7feefc044070 con 0x7fef08101800 2024-03-18T01:01:49.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.173+0000 7fef05ffb640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feef80a2800 0x7feef80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.173+0000 7fef07fff640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feef80a2800 0x7feef80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.174 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.173+0000 7fef05ffb640 1 -- 172.21.15.49:0/3052636898 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7feefc0f0740 con 0x7fef08101800 2024-03-18T01:01:49.175 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.174+0000 7fef07fff640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feef80a2800 0x7feef80a4cc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fef0819b280 tx=0x7feef4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:49.176 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.177+0000 7fef05ffb640 1 -- 172.21.15.49:0/3052636898 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feefc0bd850 con 0x7fef08101800 2024-03-18T01:01:49.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.361+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fef081a25d0 con 0x7fef08101800 2024-03-18T01:01:49.361 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.362+0000 7fef05ffb640 1 -- 172.21.15.49:0/3052636898 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7feefc0bd150 con 0x7fef08101800 2024-03-18T01:01:49.362 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 1 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feef80a2800 msgr2=0x7feef80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feef80a2800 0x7feef80a4cc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fef0819b280 tx=0x7feef4005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 msgr2=0x7fef08199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 0x7fef08199e10 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7feefc000c00 tx=0x7feefc02c640 comp rx=0 tx=0).stop 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 shutdown_connections 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7feef80a2800 0x7feef80a4cc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fef081029f0 0x7fef0819a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef081033b0 0x7fef081a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 --2- 172.21.15.49:0/3052636898 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fef08101800 0x7fef08199e10 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 >> 172.21.15.49:0/3052636898 conn(0x7fef080fcd70 msgr2=0x7fef08073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:49.364 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 shutdown_connections 2024-03-18T01:01:49.365 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.364+0000 7fef0f07e640 1 -- 172.21.15.49:0/3052636898 wait complete. 2024-03-18T01:01:49.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:49 smithi049 ceph-mon[32130]: pgmap v701: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T01:01:49.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:49 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/261685628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:01:49.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T01:01:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:49 smithi107 ceph-mon[36270]: pgmap v701: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T01:01:49.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:49 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/261685628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:01:49.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:49 smithi106 ceph-mon[35899]: pgmap v701: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T01:01:49.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:49 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/261685628' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:01:49.488 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 -- 172.21.15.49:0/3800283651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 msgr2=0x7f2454108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 --2- 172.21.15.49:0/3800283651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 0x7f2454108b40 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f244c009ca0 tx=0x7f244c02c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 -- 172.21.15.49:0/3800283651 shutdown_connections 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 --2- 172.21.15.49:0/3800283651 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f24541018f0 0x7f2454101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 --2- 172.21.15.49:0/3800283651 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 0x7f2454108b40 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 --2- 172.21.15.49:0/3800283651 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2454100700 0x7f2454100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 -- 172.21.15.49:0/3800283651 >> 172.21.15.49:0/3800283651 conn(0x7f24540fbc90 msgr2=0x7f24540fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 -- 172.21.15.49:0/3800283651 shutdown_connections 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 -- 172.21.15.49:0/3800283651 wait complete. 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.487+0000 7f24599f3640 1 Processor -- start 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24599f3640 1 -- start start 2024-03-18T01:01:49.489 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24599f3640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2454100700 0x7f24541969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24599f3640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24541018f0 0x7f2454196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24599f3640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 0x7f245419df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24599f3640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f245410c380 con 0x7f24541022b0 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24599f3640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f245410c200 con 0x7f2454100700 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24599f3640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f245410c500 con 0x7f24541018f0 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24537fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 0x7f245419df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24527fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24541018f0 0x7f2454196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f2452ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2454100700 0x7f24541969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24527fc640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24541018f0 0x7f2454196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38530/0 (socket says 172.21.15.49:38530) 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24537fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 0x7f245419df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60622/0 (socket says 172.21.15.49:60622) 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f2452ffd640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2454100700 0x7f24541969c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.106:3300/0 says I am v2:172.21.15.49:48614/0 (socket says 172.21.15.49:48614) 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24527fc640 1 -- 172.21.15.49:0/552193104 learned_addr learned my addr 172.21.15.49:0/552193104 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24527fc640 1 -- 172.21.15.49:0/552193104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2454100700 msgr2=0x7f24541969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.490 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24527fc640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2454100700 0x7f24541969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24527fc640 1 -- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 msgr2=0x7f245419df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24527fc640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 0x7f245419df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24527fc640 1 -- 172.21.15.49:0/552193104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f245419e5e0 con 0x7f24541018f0 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f24537fe640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 0x7f245419df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.488+0000 7f2452ffd640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2454100700 0x7f24541969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.489+0000 7f24527fc640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24541018f0 0x7f2454196f00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f244400ea60 tx=0x7f244400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.489+0000 7f24589f1640 1 -- 172.21.15.49:0/552193104 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f244400cde0 con 0x7f24541018f0 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.489+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f245419e8d0 con 0x7f24541018f0 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.489+0000 7f24589f1640 1 -- 172.21.15.49:0/552193104 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24440035f0 con 0x7f24541018f0 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.489+0000 7f24589f1640 1 -- 172.21.15.49:0/552193104 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444005220 con 0x7f24541018f0 2024-03-18T01:01:49.491 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.489+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f245419ee70 con 0x7f24541018f0 2024-03-18T01:01:49.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.490+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24541079c0 con 0x7f24541018f0 2024-03-18T01:01:49.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.491+0000 7f24589f1640 1 -- 172.21.15.49:0/552193104 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f24440037a0 con 0x7f24541018f0 2024-03-18T01:01:49.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.492+0000 7f24589f1640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f24400a2800 0x7f24400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.492+0000 7f2452ffd640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f24400a2800 0x7f24400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.492+0000 7f24589f1640 1 -- 172.21.15.49:0/552193104 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f24440cf150 con 0x7f24541018f0 2024-03-18T01:01:49.492 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.492+0000 7f2452ffd640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f24400a2800 0x7f24400a4cc0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2454101750 tx=0x7f243c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:49.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.499+0000 7f24589f1640 1 -- 172.21.15.49:0/552193104 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f244409c170 con 0x7f24541018f0 2024-03-18T01:01:49.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.685+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f245402d340 con 0x7f24541018f0 2024-03-18T01:01:49.685 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.685+0000 7f24589f1640 1 -- 172.21.15.49:0/552193104 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f244409ba70 con 0x7f24541018f0 2024-03-18T01:01:49.685 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 32 2024-03-18T01:01:49.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f24400a2800 msgr2=0x7f24400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f24400a2800 0x7f24400a4cc0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2454101750 tx=0x7f243c005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:49.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24541018f0 msgr2=0x7f2454196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24541018f0 0x7f2454196f00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f244400ea60 tx=0x7f244400ef20 comp rx=0 tx=0).stop 2024-03-18T01:01:49.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 shutdown_connections 2024-03-18T01:01:49.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f24400a2800 0x7f24400a4cc0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.687 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f24541022b0 0x7f245419df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f24541018f0 0x7f2454196f00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 --2- 172.21.15.49:0/552193104 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f2454100700 0x7f24541969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 >> 172.21.15.49:0/552193104 conn(0x7f24540fbc90 msgr2=0x7f2454104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:49.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 shutdown_connections 2024-03-18T01:01:49.688 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.687+0000 7f24599f3640 1 -- 172.21.15.49:0/552193104 wait complete. 2024-03-18T01:01:49.698 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T01:01:49.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 -- 172.21.15.49:0/2042844565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43e41006e0 msgr2=0x7f43e4100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.813 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/2042844565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43e41006e0 0x7f43e4100b00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009be0 tx=0x7f43d402bef0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 -- 172.21.15.49:0/2042844565 shutdown_connections 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/2042844565 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43e41018d0 0x7f43e4101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/2042844565 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43e4102290 0x7f43e4108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/2042844565 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43e41006e0 0x7f43e4100b00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 -- 172.21.15.49:0/2042844565 >> 172.21.15.49:0/2042844565 conn(0x7f43e40fbc70 msgr2=0x7f43e40fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 -- 172.21.15.49:0/2042844565 shutdown_connections 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 -- 172.21.15.49:0/2042844565 wait complete. 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.812+0000 7f43e9e4b640 1 Processor -- start 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e9e4b640 1 -- start start 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e9e4b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43e41006e0 0x7f43e4195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e9e4b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43e41018d0 0x7f43e4195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e9e4b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43e4102290 0x7f43e419cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e9e4b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f43e410b260 con 0x7f43e41006e0 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e9e4b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f43e410b0e0 con 0x7f43e41018d0 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e9e4b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f43e410b3e0 con 0x7f43e4102290 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43e41006e0 0x7f43e4195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e37fe640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43e41006e0 0x7f43e4195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60646/0 (socket says 172.21.15.49:60646) 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e37fe640 1 -- 172.21.15.49:0/327970120 learned_addr learned my addr 172.21.15.49:0/327970120 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e2ffd640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43e41018d0 0x7f43e4195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e3fff640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43e4102290 0x7f43e419cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e37fe640 1 -- 172.21.15.49:0/327970120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43e4102290 msgr2=0x7f43e419cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e37fe640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43e4102290 0x7f43e419cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e37fe640 1 -- 172.21.15.49:0/327970120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43e41018d0 msgr2=0x7f43e4195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e37fe640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43e41018d0 0x7f43e4195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e37fe640 1 -- 172.21.15.49:0/327970120 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e419d580 con 0x7f43e41006e0 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.813+0000 7f43e37fe640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43e41006e0 0x7f43e4195960 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f43d4036040 tx=0x7f43d4033580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:49.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.814+0000 7f43e0ff9640 1 -- 172.21.15.49:0/327970120 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4043020 con 0x7f43e41006e0 2024-03-18T01:01:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.814+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e419d810 con 0x7f43e41006e0 2024-03-18T01:01:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.814+0000 7f43e0ff9640 1 -- 172.21.15.49:0/327970120 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43d4003e50 con 0x7f43e41006e0 2024-03-18T01:01:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.814+0000 7f43e0ff9640 1 -- 172.21.15.49:0/327970120 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d4034690 con 0x7f43e41006e0 2024-03-18T01:01:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.814+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e419ddb0 con 0x7f43e41006e0 2024-03-18T01:01:49.816 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.815+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e4107b20 con 0x7f43e41006e0 2024-03-18T01:01:49.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.816+0000 7f43e0ff9640 1 -- 172.21.15.49:0/327970120 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f43d4046020 con 0x7f43e41006e0 2024-03-18T01:01:49.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.816+0000 7f43e0ff9640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43cc0a2800 0x7f43cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:49.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.817+0000 7f43e2ffd640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43cc0a2800 0x7f43cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:49.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.817+0000 7f43e0ff9640 1 -- 172.21.15.49:0/327970120 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f43d40f1000 con 0x7f43e41006e0 2024-03-18T01:01:49.817 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.817+0000 7f43e2ffd640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43cc0a2800 0x7f43cc0a4cc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f43e4196e70 tx=0x7f43d800d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:49.820 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:49.821+0000 7f43e0ff9640 1 -- 172.21.15.49:0/327970120 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f43d404d020 con 0x7f43e41006e0 2024-03-18T01:01:50.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.005+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f43e40623e0 con 0x7f43e41006e0 2024-03-18T01:01:50.004 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.005+0000 7f43e0ff9640 1 -- 172.21.15.49:0/327970120 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7f43d40be120 con 0x7f43e41006e0 2024-03-18T01:01:50.005 INFO:teuthology.orchestra.run.smithi049.stdout:pg_num: 64 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43cc0a2800 msgr2=0x7f43cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43cc0a2800 0x7f43cc0a4cc0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f43e4196e70 tx=0x7f43d800d040 comp rx=0 tx=0).stop 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43e41006e0 msgr2=0x7f43e4195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43e41006e0 0x7f43e4195960 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f43d4036040 tx=0x7f43d4033580 comp rx=0 tx=0).stop 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 shutdown_connections 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f43cc0a2800 0x7f43cc0a4cc0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f43e41018d0 0x7f43e4195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f43e4102290 0x7f43e419cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 --2- 172.21.15.49:0/327970120 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f43e41006e0 0x7f43e4195960 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 >> 172.21.15.49:0/327970120 conn(0x7f43e40fbc70 msgr2=0x7f43e4104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 shutdown_connections 2024-03-18T01:01:50.007 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.007+0000 7f43e9e4b640 1 -- 172.21.15.49:0/327970120 wait complete. 2024-03-18T01:01:50.018 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T01:01:50.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.128+0000 7f4a53052640 1 -- 172.21.15.49:0/3967051037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 msgr2=0x7f4a4c109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.128+0000 7f4a53052640 1 --2- 172.21.15.49:0/3967051037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 0x7f4a4c109990 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f4a44009ca0 tx=0x7f4a4402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.130 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.128+0000 7f4a53052640 1 -- 172.21.15.49:0/3967051037 shutdown_connections 2024-03-18T01:01:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.128+0000 7f4a53052640 1 --2- 172.21.15.49:0/3967051037 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a4c104c70 0x7f4a4c107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.128+0000 7f4a53052640 1 --2- 172.21.15.49:0/3967051037 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 0x7f4a4c109990 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.128+0000 7f4a53052640 1 --2- 172.21.15.49:0/3967051037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c102340 0x7f4a4c104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.128+0000 7f4a53052640 1 -- 172.21.15.49:0/3967051037 >> 172.21.15.49:0/3967051037 conn(0x7f4a4c0fbc90 msgr2=0x7f4a4c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.128+0000 7f4a53052640 1 -- 172.21.15.49:0/3967051037 shutdown_connections 2024-03-18T01:01:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.128+0000 7f4a53052640 1 -- 172.21.15.49:0/3967051037 wait complete. 2024-03-18T01:01:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a53052640 1 Processor -- start 2024-03-18T01:01:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a53052640 1 -- start start 2024-03-18T01:01:50.131 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a53052640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a4c102340 0x7f4a4c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a53052640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c104c70 0x7f4a4c198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a53052640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 0x7f4a4c19f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a53052640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c10d4b0 con 0x7f4a4c1075a0 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a53052640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c10d330 con 0x7f4a4c102340 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a53052640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4c10d630 con 0x7f4a4c104c70 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a515ef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 0x7f4a4c19f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a50dee640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a4c102340 0x7f4a4c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a4bfff640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c104c70 0x7f4a4c198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a515ef640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 0x7f4a4c19f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60666/0 (socket says 172.21.15.49:60666) 2024-03-18T01:01:50.132 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a515ef640 1 -- 172.21.15.49:0/4148257877 learned_addr learned my addr 172.21.15.49:0/4148257877 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a515ef640 1 -- 172.21.15.49:0/4148257877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c104c70 msgr2=0x7f4a4c198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a515ef640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c104c70 0x7f4a4c198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a515ef640 1 -- 172.21.15.49:0/4148257877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a4c102340 msgr2=0x7f4a4c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a515ef640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a4c102340 0x7f4a4c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a515ef640 1 -- 172.21.15.49:0/4148257877 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4c19f740 con 0x7f4a4c1075a0 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a50dee640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a4c102340 0x7f4a4c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.129+0000 7f4a4bfff640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c104c70 0x7f4a4c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.130+0000 7f4a515ef640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 0x7f4a4c19f0d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f4a44037040 tx=0x7f4a44032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.130+0000 7f4a49ffb640 1 -- 172.21.15.49:0/4148257877 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44041020 con 0x7f4a4c1075a0 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.130+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4c19f9d0 con 0x7f4a4c1075a0 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.130+0000 7f4a49ffb640 1 -- 172.21.15.49:0/4148257877 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a44034cd0 con 0x7f4a4c1075a0 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.130+0000 7f4a49ffb640 1 -- 172.21.15.49:0/4148257877 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a44039040 con 0x7f4a4c1075a0 2024-03-18T01:01:50.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.130+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a4c19ff10 con 0x7f4a4c1075a0 2024-03-18T01:01:50.134 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.131+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a4c0fd780 con 0x7f4a4c1075a0 2024-03-18T01:01:50.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.132+0000 7f4a49ffb640 1 -- 172.21.15.49:0/4148257877 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4a44044020 con 0x7f4a4c1075a0 2024-03-18T01:01:50.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.133+0000 7f4a49ffb640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:50.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.133+0000 7f4a49ffb640 1 -- 172.21.15.49:0/4148257877 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4a440f0920 con 0x7f4a4c1075a0 2024-03-18T01:01:50.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.133+0000 7f4a50dee640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:50.135 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.133+0000 7f4a50dee640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4a40009fc0 tx=0x7f4a40005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:50.137 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.137+0000 7f4a49ffb640 1 -- 172.21.15.49:0/4148257877 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4a4404b020 con 0x7f4a4c1075a0 2024-03-18T01:01:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.333+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4a4c0623e0 con 0x7f4a4c1075a0 2024-03-18T01:01:50.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.333+0000 7f4a49ffb640 1 -- 172.21.15.49:0/4148257877 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) ==== 81+0+589 (secure 0 0 0) 0x7f4a440bda30 con 0x7f4a4c1075a0 2024-03-18T01:01:50.333 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:0/3253512442 2024-03-19T00:41:01.376745+0000 2024-03-18T01:01:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:6801/1899027445 2024-03-19T00:41:01.376745+0000 2024-03-18T01:01:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:0/378341991 2024-03-19T00:41:01.376745+0000 2024-03-18T01:01:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:6800/1899027445 2024-03-19T00:41:01.376745+0000 2024-03-18T01:01:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:0/1473779746 2024-03-19T00:40:34.217610+0000 2024-03-18T01:01:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:0/620710678 2024-03-19T00:40:34.217610+0000 2024-03-18T01:01:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:6800/1672783865 2024-03-19T00:40:34.217610+0000 2024-03-18T01:01:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:0/3065759231 2024-03-19T00:40:34.217610+0000 2024-03-18T01:01:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:0/406635835 2024-03-19T00:41:01.376745+0000 2024-03-18T01:01:50.334 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49:6801/1672783865 2024-03-19T00:40:34.217610+0000 2024-03-18T01:01:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:listed 10 entries 2024-03-18T01:01:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4a3c0a2800 msgr2=0x7f4a3c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4a40009fc0 tx=0x7f4a40005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 msgr2=0x7f4a4c19f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 0x7f4a4c19f0d0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f4a44037040 tx=0x7f4a44032f40 comp rx=0 tx=0).stop 2024-03-18T01:01:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 shutdown_connections 2024-03-18T01:01:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4a3c0a2800 0x7f4a3c0a4cc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4a4c1075a0 0x7f4a4c19f0d0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4a4c104c70 0x7f4a4c198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 --2- 172.21.15.49:0/4148257877 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4a4c102340 0x7f4a4c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 >> 172.21.15.49:0/4148257877 conn(0x7f4a4c0fbc90 msgr2=0x7f4a4c102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 shutdown_connections 2024-03-18T01:01:50.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.335+0000 7f4a53052640 1 -- 172.21.15.49:0/4148257877 wait complete. 2024-03-18T01:01:50.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:50 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3052636898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:01:50.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:50 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/552193104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:01:50.345 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:50 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/327970120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:01:50.346 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-18T01:01:50.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:01:50.346 DEBUG:teuthology.orchestra.run.smithi049:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-18T01:01:50.365 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:50 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3052636898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:01:50.365 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:50 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/552193104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:01:50.365 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:50 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/327970120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:01:50.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:50 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3052636898' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T01:01:50.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:50 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/552193104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T01:01:50.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:50 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/327970120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T01:01:50.486 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi049.front.sepia.ceph.com 2024-03-18T01:01:50.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:01:50.486 DEBUG:teuthology.orchestra.run.smithi049:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T01:01:50.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:01:50.528 DEBUG:teuthology.orchestra.run.smithi049:> ip netns list 2024-03-18T01:01:50.611 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T01:01:50.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:01:50.611 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T01:01:50.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T01:01:50.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo ip link delete ceph-brx 2024-03-18T01:01:50.782 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T01:01:50.793 INFO:tasks.cephadm:Teardown begin 2024-03-18T01:01:50.793 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T01:01:50.793 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:01:50.903 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 -- 172.21.15.49:0/1896649279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc1022b0 msgr2=0x7fb1fc108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 --2- 172.21.15.49:0/1896649279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc1022b0 0x7fb1fc108b40 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009ca0 tx=0x7fb1f402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 -- 172.21.15.49:0/1896649279 shutdown_connections 2024-03-18T01:01:50.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 --2- 172.21.15.49:0/1896649279 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1fc1018f0 0x7fb1fc101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 --2- 172.21.15.49:0/1896649279 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc1022b0 0x7fb1fc108b40 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 --2- 172.21.15.49:0/1896649279 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc100700 0x7fb1fc100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 -- 172.21.15.49:0/1896649279 >> 172.21.15.49:0/1896649279 conn(0x7fb1fc0fbc70 msgr2=0x7fb1fc0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:50.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 -- 172.21.15.49:0/1896649279 shutdown_connections 2024-03-18T01:01:50.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 -- 172.21.15.49:0/1896649279 wait complete. 2024-03-18T01:01:50.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 Processor -- start 2024-03-18T01:01:50.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.904+0000 7fb20350c640 1 -- start start 2024-03-18T01:01:50.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb20350c640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc100700 0x7fb1fc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:50.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb20350c640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1fc1018f0 0x7fb1fc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:50.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb20350c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc1022b0 0x7fb1fc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:50.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb20350c640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc10d4a0 con 0x7fb1fc100700 2024-03-18T01:01:50.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb20350c640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc10d320 con 0x7fb1fc1018f0 2024-03-18T01:01:50.906 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb20350c640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc10d620 con 0x7fb1fc1022b0 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb2012a8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc100700 0x7fb1fc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb2012a8640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc100700 0x7fb1fc197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60690/0 (socket says 172.21.15.49:60690) 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb2012a8640 1 -- 172.21.15.49:0/559495117 learned_addr learned my addr 172.21.15.49:0/559495117 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb200aa7640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1fc1018f0 0x7fb1fc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb201aa9640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc1022b0 0x7fb1fc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb2012a8640 1 -- 172.21.15.49:0/559495117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc1022b0 msgr2=0x7fb1fc19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb2012a8640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc1022b0 0x7fb1fc19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb2012a8640 1 -- 172.21.15.49:0/559495117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1fc1018f0 msgr2=0x7fb1fc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb2012a8640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1fc1018f0 0x7fb1fc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb2012a8640 1 -- 172.21.15.49:0/559495117 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc19f6f0 con 0x7fb1fc100700 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb2012a8640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc100700 0x7fb1fc197ad0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00e9c0 tx=0x7fb1ec00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb1f27fc640 1 -- 172.21.15.49:0/559495117 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec00cd80 con 0x7fb1fc100700 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb1f27fc640 1 -- 172.21.15.49:0/559495117 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1ec0035f0 con 0x7fb1fc100700 2024-03-18T01:01:50.907 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1fc19f9e0 con 0x7fb1fc100700 2024-03-18T01:01:50.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb1f27fc640 1 -- 172.21.15.49:0/559495117 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec010690 con 0x7fb1fc100700 2024-03-18T01:01:50.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.905+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1fc19ff30 con 0x7fb1fc100700 2024-03-18T01:01:50.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.907+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1fc107a20 con 0x7fb1fc100700 2024-03-18T01:01:50.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.908+0000 7fb1f27fc640 1 -- 172.21.15.49:0/559495117 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb1ec0037a0 con 0x7fb1fc100700 2024-03-18T01:01:50.908 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.908+0000 7fb1f27fc640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e80a2880 0x7fb1e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:50.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.908+0000 7fb200aa7640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e80a2880 0x7fb1e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:50.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.908+0000 7fb1f27fc640 1 -- 172.21.15.49:0/559495117 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb1ec0ced50 con 0x7fb1fc100700 2024-03-18T01:01:50.909 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.909+0000 7fb200aa7640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e80a2880 0x7fb1e80a4d40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc198fe0 tx=0x7fb1f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:50.912 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:50.913+0000 7fb1f27fc640 1 -- 172.21.15.49:0/559495117 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb1ec0d2030 con 0x7fb1fc100700 2024-03-18T01:01:51.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.139+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1fc02d1b0 con 0x7fb1fc100700 2024-03-18T01:01:51.139 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.140+0000 7fb1f27fc640 1 -- 172.21.15.49:0/559495117 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11936 (secure 0 0 0) 0x7fb1ec09bdd0 con 0x7fb1fc100700 2024-03-18T01:01:51.140 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T01:01:51.140 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T01:01:01.508865+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":178,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24470,24463,24481,24506,24499]},"id":1}]} 2024-03-18T01:01:51.140 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 31 2024-03-18T01:01:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e80a2880 msgr2=0x7fb1e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e80a2880 0x7fb1e80a4d40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb1fc198fe0 tx=0x7fb1f8005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc100700 msgr2=0x7fb1fc197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc100700 0x7fb1fc197ad0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec00e9c0 tx=0x7fb1ec00ee80 comp rx=0 tx=0).stop 2024-03-18T01:01:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 shutdown_connections 2024-03-18T01:01:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fb1e80a2880 0x7fb1e80a4d40 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fb1fc1018f0 0x7fb1fc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.142 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb1fc1022b0 0x7fb1fc19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 --2- 172.21.15.49:0/559495117 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fb1fc100700 0x7fb1fc197ad0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 >> 172.21.15.49:0/559495117 conn(0x7fb1fc0fbc70 msgr2=0x7fb1fc104bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:51.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 shutdown_connections 2024-03-18T01:01:51.143 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.142+0000 7fb20350c640 1 -- 172.21.15.49:0/559495117 wait complete. 2024-03-18T01:01:51.153 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T01:01:51.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.265+0000 7f4fa8787640 1 -- 172.21.15.49:0/3970630476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa0104c90 msgr2=0x7f4fa0107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.265+0000 7f4fa8787640 1 --2- 172.21.15.49:0/3970630476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa0104c90 0x7f4fa0107080 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4f90009860 tx=0x7f4f9002bb70 comp rx=0 tx=0).stop 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.265+0000 7f4fa8787640 1 -- 172.21.15.49:0/3970630476 shutdown_connections 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.265+0000 7f4fa8787640 1 --2- 172.21.15.49:0/3970630476 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa0104c90 0x7f4fa0107080 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.265+0000 7f4fa8787640 1 --2- 172.21.15.49:0/3970630476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa01075c0 0x7f4fa01099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.265+0000 7f4fa8787640 1 --2- 172.21.15.49:0/3970630476 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4fa0102360 0x7f4fa0104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.265+0000 7f4fa8787640 1 -- 172.21.15.49:0/3970630476 >> 172.21.15.49:0/3970630476 conn(0x7f4fa00fbd10 msgr2=0x7f4fa00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.265+0000 7f4fa8787640 1 -- 172.21.15.49:0/3970630476 shutdown_connections 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.265+0000 7f4fa8787640 1 -- 172.21.15.49:0/3970630476 wait complete. 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa8787640 1 Processor -- start 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa8787640 1 -- start start 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa8787640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa0102360 0x7f4fa0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa8787640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa0104c90 0x7f4fa0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa8787640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4fa01075c0 0x7f4fa019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.268 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa8787640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa010d460 con 0x7f4fa01075c0 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa8787640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa010d2e0 con 0x7f4fa0102360 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa8787640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f4fa010d5e0 con 0x7f4fa0104c90 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa6d24640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4fa01075c0 0x7f4fa019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa6d24640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4fa01075c0 0x7f4fa019f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60704/0 (socket says 172.21.15.49:60704) 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa6d24640 1 -- 172.21.15.49:0/2059030262 learned_addr learned my addr 172.21.15.49:0/2059030262 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa5d22640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa0104c90 0x7f4fa0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa6523640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa0102360 0x7f4fa0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa6d24640 1 -- 172.21.15.49:0/2059030262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa0104c90 msgr2=0x7f4fa0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa6d24640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa0104c90 0x7f4fa0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa6d24640 1 -- 172.21.15.49:0/2059030262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa0102360 msgr2=0x7f4fa0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa6d24640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa0102360 0x7f4fa0197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.266+0000 7f4fa6d24640 1 -- 172.21.15.49:0/2059030262 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa019f750 con 0x7f4fa01075c0 2024-03-18T01:01:51.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.267+0000 7f4fa6d24640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4fa01075c0 0x7f4fa019f0e0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4f9800efd0 tx=0x7f4f9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:51.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.267+0000 7f4f977fe640 1 -- 172.21.15.49:0/2059030262 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f98003b70 con 0x7f4fa01075c0 2024-03-18T01:01:51.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.267+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fa019fa40 con 0x7f4fa01075c0 2024-03-18T01:01:51.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.267+0000 7f4f977fe640 1 -- 172.21.15.49:0/2059030262 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4f98003d10 con 0x7f4fa01075c0 2024-03-18T01:01:51.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.267+0000 7f4f977fe640 1 -- 172.21.15.49:0/2059030262 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f980109f0 con 0x7f4fa01075c0 2024-03-18T01:01:51.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.267+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fa019ff60 con 0x7f4fa01075c0 2024-03-18T01:01:51.270 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.268+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fa00fd800 con 0x7f4fa01075c0 2024-03-18T01:01:51.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.269+0000 7f4f977fe640 1 -- 172.21.15.49:0/2059030262 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4f98010b90 con 0x7f4fa01075c0 2024-03-18T01:01:51.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.270+0000 7f4f977fe640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f8c0a2800 0x7f4f8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.270+0000 7f4fa6523640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f8c0a2800 0x7f4f8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.270+0000 7f4f977fe640 1 -- 172.21.15.49:0/2059030262 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4f980cf280 con 0x7f4fa01075c0 2024-03-18T01:01:51.271 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.270+0000 7f4fa6523640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f8c0a2800 0x7f4f8c0a4cc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009530 tx=0x7f4f9c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:51.272 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.273+0000 7f4f977fe640 1 -- 172.21.15.49:0/2059030262 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4f9809c300 con 0x7f4fa01075c0 2024-03-18T01:01:51.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:51 smithi049 ceph-mon[32130]: pgmap v702: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-18T01:01:51.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:51 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/4148257877' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:01:51.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:51 smithi107 ceph-mon[36270]: pgmap v702: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-18T01:01:51.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:51 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/4148257877' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:01:51.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:51 smithi106 ceph-mon[35899]: pgmap v702: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-18T01:01:51.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:51 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/4148257877' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T01:01:51.496 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.497+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fa002d1b0 con 0x7f4fa01075c0 2024-03-18T01:01:51.497 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.497+0000 7f4f977fe640 1 -- 172.21.15.49:0/2059030262 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11936 (secure 0 0 0) 0x7f4f9809bc00 con 0x7f4fa01075c0 2024-03-18T01:01:51.497 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T01:01:51.498 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6835/1901706381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6834","nonce":1901706381},{"type":"v1","addr":"172.21.15.49:6835","nonce":1901706381}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.49:6837/535636420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6836","nonce":535636420},{"type":"v1","addr":"172.21.15.49:6837","nonce":535636420}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T00:48:05.881816+0000","modified":"2024-03-18T01:01:01.508865+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24499,"mds_2":24481,"mds_3":24470,"mds_4":24463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6839/1964983089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6838","nonce":1964983089},{"type":"v1","addr":"172.21.15.49:6839","nonce":1964983089}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14625":{"gid":14625,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.49:6841/3327527918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6840","nonce":3327527918},{"type":"v1","addr":"172.21.15.49:6841","nonce":3327527918}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24404":{"gid":24404,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6833/701158707","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6832","nonce":701158707},{"type":"v1","addr":"172.21.15.106:6833","nonce":701158707}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24421":{"gid":24421,"name":"l","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/1442063907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":1442063907},{"type":"v1","addr":"172.21.15.107:6833","nonce":1442063907}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.106:6835/353176461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6834","nonce":353176461},{"type":"v1","addr":"172.21.15.106:6835","nonce":353176461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":9,"addr":"172.21.15.107:6835/863606455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":863606455},{"type":"v1","addr":"172.21.15.107:6835","nonce":863606455}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24470":{"gid":24470,"name":"h","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.106:6837/3542707642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6836","nonce":3542707642},{"type":"v1","addr":"172.21.15.106:6837","nonce":3542707642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24481":{"gid":24481,"name":"c","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.107:6837/3043451403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":3043451403},{"type":"v1","addr":"172.21.15.107:6837","nonce":3043451403}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24499":{"gid":24499,"name":"i","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.107:6839/2909446244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":2909446244},{"type":"v1","addr":"172.21.15.107:6839","nonce":2909446244}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":178,"addr":"172.21.15.106:6839/2088471086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6838","nonce":2088471086},{"type":"v1","addr":"172.21.15.106:6839","nonce":2088471086}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24506,"qdb_cluster":[24470,24463,24481,24506,24499]},"id":1}]} 2024-03-18T01:01:51.498 INFO:teuthology.orchestra.run.smithi049.stderr:dumped fsmap epoch 31 2024-03-18T01:01:51.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f8c0a2800 msgr2=0x7f4f8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.499 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f8c0a2800 0x7f4f8c0a4cc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009530 tx=0x7f4f9c009260 comp rx=0 tx=0).stop 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4fa01075c0 msgr2=0x7f4fa019f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4fa01075c0 0x7f4fa019f0e0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f4f9800efd0 tx=0x7f4f9800c6a0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 shutdown_connections 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f4f8c0a2800 0x7f4f8c0a4cc0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f4fa01075c0 0x7f4fa019f0e0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4fa0104c90 0x7f4fa0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 --2- 172.21.15.49:0/2059030262 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f4fa0102360 0x7f4fa0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 >> 172.21.15.49:0/2059030262 conn(0x7f4fa00fbd10 msgr2=0x7f4fa0102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 shutdown_connections 2024-03-18T01:01:51.500 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.500+0000 7f4fa8787640 1 -- 172.21.15.49:0/2059030262 wait complete. 2024-03-18T01:01:51.511 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T01:01:51.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.619+0000 7fd5eb753640 1 -- 172.21.15.49:0/3664146554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5e41022b0 msgr2=0x7fd5e4108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.619+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3664146554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5e41022b0 0x7fd5e4108b40 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4009ca0 tx=0x7fd5d402c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.619+0000 7fd5eb753640 1 -- 172.21.15.49:0/3664146554 shutdown_connections 2024-03-18T01:01:51.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.619+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3664146554 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5e41022b0 0x7fd5e4108b40 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.619+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3664146554 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5e41018f0 0x7fd5e4101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.619+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3664146554 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e4100700 0x7fd5e4100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.621 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.619+0000 7fd5eb753640 1 -- 172.21.15.49:0/3664146554 >> 172.21.15.49:0/3664146554 conn(0x7fd5e40fbc70 msgr2=0x7fd5e40fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 -- 172.21.15.49:0/3664146554 shutdown_connections 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 -- 172.21.15.49:0/3664146554 wait complete. 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 Processor -- start 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 -- start start 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e4100700 0x7fd5e4197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5e41018f0 0x7fd5e4198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5e41022b0 0x7fd5e419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e410c380 con 0x7fd5e41022b0 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e410c200 con 0x7fd5e41018f0 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5eb753640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e410c500 con 0x7fd5e4100700 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5e9cf0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5e41022b0 0x7fd5e419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5e94ef640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e4100700 0x7fd5e4197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5e9cf0640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5e41022b0 0x7fd5e419f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60714/0 (socket says 172.21.15.49:60714) 2024-03-18T01:01:51.622 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.620+0000 7fd5e9cf0640 1 -- 172.21.15.49:0/3135644250 learned_addr learned my addr 172.21.15.49:0/3135644250 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e8cee640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5e41018f0 0x7fd5e4198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e9cf0640 1 -- 172.21.15.49:0/3135644250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e4100700 msgr2=0x7fd5e4197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e9cf0640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e4100700 0x7fd5e4197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e9cf0640 1 -- 172.21.15.49:0/3135644250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5e41018f0 msgr2=0x7fd5e4198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e9cf0640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5e41018f0 0x7fd5e4198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e9cf0640 1 -- 172.21.15.49:0/3135644250 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e419f730 con 0x7fd5e41022b0 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e9cf0640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5e41022b0 0x7fd5e419f0c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4037040 tx=0x7fd5d4032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e27fc640 1 -- 172.21.15.49:0/3135644250 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4041020 con 0x7fd5e41022b0 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e27fc640 1 -- 172.21.15.49:0/3135644250 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5d4034cd0 con 0x7fd5e41022b0 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e419f9c0 con 0x7fd5e41022b0 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5e27fc640 1 -- 172.21.15.49:0/3135644250 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d4039040 con 0x7fd5e41022b0 2024-03-18T01:01:51.623 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.621+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e419feb0 con 0x7fd5e41022b0 2024-03-18T01:01:51.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.622+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e41a00e0 con 0x7fd5e41022b0 2024-03-18T01:01:51.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.623+0000 7fd5e27fc640 1 -- 172.21.15.49:0/3135644250 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd5d4044020 con 0x7fd5e41022b0 2024-03-18T01:01:51.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.624+0000 7fd5e27fc640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd5cc0a2800 0x7fd5cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.624+0000 7fd5e94ef640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd5cc0a2800 0x7fd5cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.624 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.625+0000 7fd5e27fc640 1 -- 172.21.15.49:0/3135644250 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd5d40f0d70 con 0x7fd5e41022b0 2024-03-18T01:01:51.625 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.625+0000 7fd5e94ef640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd5cc0a2800 0x7fd5cc0a4cc0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4101750 tx=0x7fd5dc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:51.628 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.628+0000 7fd5e27fc640 1 -- 172.21.15.49:0/3135644250 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd5d404b020 con 0x7fd5e41022b0 2024-03-18T01:01:51.814 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.814+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd5e41a0470 con 0x7fd5e41022b0 2024-03-18T01:01:51.815 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.816+0000 7fd5e27fc640 1 -- 172.21.15.49:0/3135644250 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22734 (secure 0 0 0) 0x7fd5d40bde80 con 0x7fd5e41022b0 2024-03-18T01:01:51.816 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-03-18T01:01:51.816 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":88,"fsid":"ec390cde-e4bf-11ee-95c9-87774f69a715","created":"2024-03-18T00:40:03.036875+0000","modified":"2024-03-18T00:49:03.081599+0000","last_up_change":"2024-03-18T00:47:15.551397+0000","last_in_change":"2024-03-18T00:46:55.689657+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T00:43:27.910351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T00:48:04.316609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T00:48:05.271815+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba2a7ba2-663e-47ea-99f4-f190dece19d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6803","nonce":2159617645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6805","nonce":2159617645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6809","nonce":2159617645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":2159617645},{"type":"v1","addr":"172.21.15.49:6807","nonce":2159617645}]},"public_addr":"172.21.15.49:6803/2159617645","cluster_addr":"172.21.15.49:6805/2159617645","heartbeat_back_addr":"172.21.15.49:6809/2159617645","heartbeat_front_addr":"172.21.15.49:6807/2159617645","state":["exists","up"]},{"osd":1,"uuid":"e16bd7a7-f97d-4c1c-b014-c3252101dd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6800","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6801","nonce":3847617930}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6802","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6803","nonce":3847617930}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6806","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6807","nonce":3847617930}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6804","nonce":3847617930},{"type":"v1","addr":"172.21.15.106:6805","nonce":3847617930}]},"public_addr":"172.21.15.106:6801/3847617930","cluster_addr":"172.21.15.106:6803/3847617930","heartbeat_back_addr":"172.21.15.106:6807/3847617930","heartbeat_front_addr":"172.21.15.106:6805/3847617930","state":["exists","up"]},{"osd":2,"uuid":"2fc48643-7a8d-4a34-9dcc-8b5b9098abe9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6801","nonce":4213760131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6803","nonce":4213760131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6807","nonce":4213760131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":4213760131},{"type":"v1","addr":"172.21.15.107:6805","nonce":4213760131}]},"public_addr":"172.21.15.107:6801/4213760131","cluster_addr":"172.21.15.107:6803/4213760131","heartbeat_back_addr":"172.21.15.107:6807/4213760131","heartbeat_front_addr":"172.21.15.107:6805/4213760131","state":["exists","up"]},{"osd":3,"uuid":"0adbb9e1-881b-4d18-8933-6ac789b3d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6811","nonce":725028995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6813","nonce":725028995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6816","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6817","nonce":725028995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6814","nonce":725028995},{"type":"v1","addr":"172.21.15.49:6815","nonce":725028995}]},"public_addr":"172.21.15.49:6811/725028995","cluster_addr":"172.21.15.49:6813/725028995","heartbeat_back_addr":"172.21.15.49:6817/725028995","heartbeat_front_addr":"172.21.15.49:6815/725028995","state":["exists","up"]},{"osd":4,"uuid":"d95b4131-8533-4e43-b25c-c081adff6639","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6808","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6809","nonce":2755514783}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6810","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6811","nonce":2755514783}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6814","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6815","nonce":2755514783}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6812","nonce":2755514783},{"type":"v1","addr":"172.21.15.106:6813","nonce":2755514783}]},"public_addr":"172.21.15.106:6809/2755514783","cluster_addr":"172.21.15.106:6811/2755514783","heartbeat_back_addr":"172.21.15.106:6815/2755514783","heartbeat_front_addr":"172.21.15.106:6813/2755514783","state":["exists","up"]},{"osd":5,"uuid":"41055562-3e13-4785-9ba2-de16fcd415f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6809","nonce":391404983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6811","nonce":391404983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6815","nonce":391404983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":391404983},{"type":"v1","addr":"172.21.15.107:6813","nonce":391404983}]},"public_addr":"172.21.15.107:6809/391404983","cluster_addr":"172.21.15.107:6811/391404983","heartbeat_back_addr":"172.21.15.107:6815/391404983","heartbeat_front_addr":"172.21.15.107:6813/391404983","state":["exists","up"]},{"osd":6,"uuid":"8945dfe3-e4b2-49bc-81bd-ed8ef849eab6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6818","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6819","nonce":24691072}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6820","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6821","nonce":24691072}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6824","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6825","nonce":24691072}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6822","nonce":24691072},{"type":"v1","addr":"172.21.15.49:6823","nonce":24691072}]},"public_addr":"172.21.15.49:6819/24691072","cluster_addr":"172.21.15.49:6821/24691072","heartbeat_back_addr":"172.21.15.49:6825/24691072","heartbeat_front_addr":"172.21.15.49:6823/24691072","state":["exists","up"]},{"osd":7,"uuid":"dc773d75-ff12-45ce-ac11-27de406961c7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6816","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6817","nonce":4213628543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6818","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6819","nonce":4213628543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6822","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6823","nonce":4213628543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6820","nonce":4213628543},{"type":"v1","addr":"172.21.15.106:6821","nonce":4213628543}]},"public_addr":"172.21.15.106:6817/4213628543","cluster_addr":"172.21.15.106:6819/4213628543","heartbeat_back_addr":"172.21.15.106:6823/4213628543","heartbeat_front_addr":"172.21.15.106:6821/4213628543","state":["exists","up"]},{"osd":8,"uuid":"0e583e9c-fd87-4cdf-9b71-ced791a6419e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6817","nonce":3032692345}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6819","nonce":3032692345}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6823","nonce":3032692345}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3032692345},{"type":"v1","addr":"172.21.15.107:6821","nonce":3032692345}]},"public_addr":"172.21.15.107:6817/3032692345","cluster_addr":"172.21.15.107:6819/3032692345","heartbeat_back_addr":"172.21.15.107:6823/3032692345","heartbeat_front_addr":"172.21.15.107:6821/3032692345","state":["exists","up"]},{"osd":9,"uuid":"a435579c-62d2-409c-83ce-6c44d620a592","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6826","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6827","nonce":1286831410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6828","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6829","nonce":1286831410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6832","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6833","nonce":1286831410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6830","nonce":1286831410},{"type":"v1","addr":"172.21.15.49:6831","nonce":1286831410}]},"public_addr":"172.21.15.49:6827/1286831410","cluster_addr":"172.21.15.49:6829/1286831410","heartbeat_back_addr":"172.21.15.49:6833/1286831410","heartbeat_front_addr":"172.21.15.49:6831/1286831410","state":["exists","up"]},{"osd":10,"uuid":"f13f5e88-b234-4c16-a11b-b65a9100334b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6824","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6825","nonce":3868441447}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6826","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6827","nonce":3868441447}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6830","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6831","nonce":3868441447}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.106:6828","nonce":3868441447},{"type":"v1","addr":"172.21.15.106:6829","nonce":3868441447}]},"public_addr":"172.21.15.106:6825/3868441447","cluster_addr":"172.21.15.106:6827/3868441447","heartbeat_back_addr":"172.21.15.106:6831/3868441447","heartbeat_front_addr":"172.21.15.106:6829/3868441447","state":["exists","up"]},{"osd":11,"uuid":"95663d7f-7dd2-4eb8-bcf1-e7f5923105d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6825","nonce":1232543995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6827","nonce":1232543995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6831","nonce":1232543995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1232543995},{"type":"v1","addr":"172.21.15.107:6829","nonce":1232543995}]},"public_addr":"172.21.15.107:6825/1232543995","cluster_addr":"172.21.15.107:6827/1232543995","heartbeat_back_addr":"172.21.15.107:6831/1232543995","heartbeat_front_addr":"172.21.15.107:6829/1232543995","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:42:35.326390+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:00.006372+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:23.770819+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:43:49.817268+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:14.635464+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:44:39.008991+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:04.479325+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:30.125742+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:45:55.420675+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:21.796578+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:46:47.852526+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T00:47:13.987163+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":8,"to":4}]},{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":6}]},{"pgid":"2.1b","mappings":[{"from":5,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.49:0/3253512442":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/378341991":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6800/1899027445":"2024-03-19T00:41:01.376745+0000","172.21.15.49:0/1473779746":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/620710678":"2024-03-19T00:40:34.217610+0000","172.21.15.49:6800/1672783865":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/3065759231":"2024-03-19T00:40:34.217610+0000","172.21.15.49:0/406635835":"2024-03-19T00:41:01.376745+0000","172.21.15.49:6801/1672783865":"2024-03-19T00:40:34.217610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd5cc0a2800 msgr2=0x7fd5cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd5cc0a2800 0x7fd5cc0a4cc0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4101750 tx=0x7fd5dc005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5e41022b0 msgr2=0x7fd5e419f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5e41022b0 0x7fd5e419f0c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4037040 tx=0x7fd5d4032f40 comp rx=0 tx=0).stop 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 shutdown_connections 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd5cc0a2800 0x7fd5cc0a4cc0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd5e41018f0 0x7fd5e4198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd5e41022b0 0x7fd5e419f0c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 --2- 172.21.15.49:0/3135644250 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd5e4100700 0x7fd5e4197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 >> 172.21.15.49:0/3135644250 conn(0x7fd5e40fbc70 msgr2=0x7fd5e40fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:51.818 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 shutdown_connections 2024-03-18T01:01:51.819 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.818+0000 7fd5eb753640 1 -- 172.21.15.49:0/3135644250 wait complete. 2024-03-18T01:01:51.829 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T01:01:51.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.940+0000 7fd31d2c6640 1 -- 172.21.15.49:0/1388360436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181022b0 msgr2=0x7fd318108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.940+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/1388360436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181022b0 0x7fd318108b40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fd310009ca0 tx=0x7fd31002c100 comp rx=0 tx=0).stop 2024-03-18T01:01:51.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.940+0000 7fd31d2c6640 1 -- 172.21.15.49:0/1388360436 shutdown_connections 2024-03-18T01:01:51.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.940+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/1388360436 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3181018f0 0x7fd318101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.940+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/1388360436 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181022b0 0x7fd318108b40 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.940+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/1388360436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd318100700 0x7fd318100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.940+0000 7fd31d2c6640 1 -- 172.21.15.49:0/1388360436 >> 172.21.15.49:0/1388360436 conn(0x7fd3180fbc90 msgr2=0x7fd3180fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:51.942 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.940+0000 7fd31d2c6640 1 -- 172.21.15.49:0/1388360436 shutdown_connections 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.940+0000 7fd31d2c6640 1 -- 172.21.15.49:0/1388360436 wait complete. 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd31d2c6640 1 Processor -- start 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd31d2c6640 1 -- start start 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd31d2c6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd318100700 0x7fd318197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd31d2c6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181018f0 0x7fd3181980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd31d2c6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3181022b0 0x7fd31819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd31d2c6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fd31810c380 con 0x7fd3181018f0 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd31d2c6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fd31810c200 con 0x7fd3181022b0 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd31d2c6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fd31810c500 con 0x7fd318100700 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd30ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181018f0 0x7fd3181980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd316ffd640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd318100700 0x7fd318197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd30ffff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181018f0 0x7fd3181980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60732/0 (socket says 172.21.15.49:60732) 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd30ffff640 1 -- 172.21.15.49:0/70834489 learned_addr learned my addr 172.21.15.49:0/70834489 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd3177fe640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3181022b0 0x7fd31819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.943 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd30ffff640 1 -- 172.21.15.49:0/70834489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd318100700 msgr2=0x7fd318197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd30ffff640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd318100700 0x7fd318197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd30ffff640 1 -- 172.21.15.49:0/70834489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3181022b0 msgr2=0x7fd31819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd30ffff640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3181022b0 0x7fd31819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd30ffff640 1 -- 172.21.15.49:0/70834489 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31819f770 con 0x7fd3181018f0 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.942+0000 7fd3177fe640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3181022b0 0x7fd31819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.941+0000 7fd316ffd640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd318100700 0x7fd318197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.942+0000 7fd30ffff640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181018f0 0x7fd3181980e0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fd30000ea60 tx=0x7fd30000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.942+0000 7fd314ff9640 1 -- 172.21.15.49:0/70834489 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30000cde0 con 0x7fd3181018f0 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.942+0000 7fd314ff9640 1 -- 172.21.15.49:0/70834489 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3000035f0 con 0x7fd3181018f0 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.942+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31819fa60 con 0x7fd3181018f0 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.942+0000 7fd314ff9640 1 -- 172.21.15.49:0/70834489 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd300010670 con 0x7fd3181018f0 2024-03-18T01:01:51.944 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.942+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31819ffb0 con 0x7fd3181018f0 2024-03-18T01:01:51.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.943+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3181a0240 con 0x7fd3181018f0 2024-03-18T01:01:51.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.944+0000 7fd314ff9640 1 -- 172.21.15.49:0/70834489 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd3000037a0 con 0x7fd3181018f0 2024-03-18T01:01:51.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.945+0000 7fd314ff9640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3040a2800 0x7fd3040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:51.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.945+0000 7fd316ffd640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3040a2800 0x7fd3040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:51.945 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.945+0000 7fd314ff9640 1 -- 172.21.15.49:0/70834489 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd3000cee20 con 0x7fd3181018f0 2024-03-18T01:01:51.946 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.945+0000 7fd316ffd640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3040a2800 0x7fd3040a4cc0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd318101750 tx=0x7fd308005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:51.948 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:51.949+0000 7fd314ff9640 1 -- 172.21.15.49:0/70834489 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd30009bf30 con 0x7fd3181018f0 2024-03-18T01:01:52.124 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/559495117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:01:52.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/2059030262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:01:52.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3135644250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:01:52.177 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.178+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd3181a0410 con 0x7fd3181018f0 2024-03-18T01:01:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.180+0000 7fd314ff9640 1 -- 172.21.15.49:0/70834489 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) ==== 161+0+0 (secure 0 0 0) 0x7fd30009b830 con 0x7fd3181018f0 2024-03-18T01:01:52.180 INFO:teuthology.orchestra.run.smithi049.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T01:01:52.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.183+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3040a2800 msgr2=0x7fd3040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:52.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.183+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3040a2800 0x7fd3040a4cc0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd318101750 tx=0x7fd308005f90 comp rx=0 tx=0).stop 2024-03-18T01:01:52.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.183+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181018f0 msgr2=0x7fd3181980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:52.183 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.183+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181018f0 0x7fd3181980e0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fd30000ea60 tx=0x7fd30000ef20 comp rx=0 tx=0).stop 2024-03-18T01:01:52.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.184+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 shutdown_connections 2024-03-18T01:01:52.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.184+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fd3040a2800 0x7fd3040a4cc0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:52.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.184+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fd3181022b0 0x7fd31819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:52.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.184+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fd3181018f0 0x7fd3181980e0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:52.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.184+0000 7fd31d2c6640 1 --2- 172.21.15.49:0/70834489 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd318100700 0x7fd318197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:52.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.184+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 >> 172.21.15.49:0/70834489 conn(0x7fd3180fbc90 msgr2=0x7fd3180fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:52.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.184+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 shutdown_connections 2024-03-18T01:01:52.184 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.184+0000 7fd31d2c6640 1 -- 172.21.15.49:0/70834489 wait complete. 2024-03-18T01:01:52.196 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T01:01:52.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.315+0000 7f8130b59640 1 -- 172.21.15.49:0/3485885044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 msgr2=0x7f812c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:52.316 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.315+0000 7f8130b59640 1 --2- 172.21.15.49:0/3485885044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 0x7f812c100b00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f811c009900 tx=0x7f811c02bc30 comp rx=0 tx=0).stop 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.315+0000 7f8130b59640 1 -- 172.21.15.49:0/3485885044 shutdown_connections 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.315+0000 7f8130b59640 1 --2- 172.21.15.49:0/3485885044 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f812c1018d0 0x7f812c101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.315+0000 7f8130b59640 1 --2- 172.21.15.49:0/3485885044 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c102290 0x7f812c108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.315+0000 7f8130b59640 1 --2- 172.21.15.49:0/3485885044 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 0x7f812c100b00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.315+0000 7f8130b59640 1 -- 172.21.15.49:0/3485885044 >> 172.21.15.49:0/3485885044 conn(0x7f812c0fbc70 msgr2=0x7f812c0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.315+0000 7f8130b59640 1 -- 172.21.15.49:0/3485885044 shutdown_connections 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.315+0000 7f8130b59640 1 -- 172.21.15.49:0/3485885044 wait complete. 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8130b59640 1 Processor -- start 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8130b59640 1 -- start start 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8130b59640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 0x7f812c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8130b59640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f812c1018d0 0x7f812c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:52.317 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8130b59640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c102290 0x7f812c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8130b59640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7f812c10c350 con 0x7f812c102290 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8130b59640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7f812c10c1d0 con 0x7f812c1018d0 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8130b59640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7f812c10c4d0 con 0x7f812c1006e0 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f812a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 0x7f812c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f812ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c102290 0x7f812c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8129d9b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f812c1018d0 0x7f812c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f812a59c640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 0x7f812c197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.49:38714/0 (socket says 172.21.15.49:38714) 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f812ad9d640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c102290 0x7f812c19f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60800/0 (socket says 172.21.15.49:60800) 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f812a59c640 1 -- 172.21.15.49:0/3210077957 learned_addr learned my addr 172.21.15.49:0/3210077957 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8129d9b640 1 -- 172.21.15.49:0/3210077957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 msgr2=0x7f812c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8129d9b640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 0x7f812c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:52.318 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8129d9b640 1 -- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c102290 msgr2=0x7f812c19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8129d9b640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c102290 0x7f812c19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f8129d9b640 1 -- 172.21.15.49:0/3210077957 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f812c19f790 con 0x7f812c1018d0 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f812ad9d640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c102290 0x7f812c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.316+0000 7f812a59c640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 0x7f812c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.317+0000 7f8129d9b640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f812c1018d0 0x7f812c1980b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f812000e9f0 tx=0x7f812000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.317+0000 7f811b7fe640 1 -- 172.21.15.49:0/3210077957 <== mon.2 v2:172.21.15.106:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812000ccc0 con 0x7f812c1018d0 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.317+0000 7f811b7fe640 1 -- 172.21.15.49:0/3210077957 <== mon.2 v2:172.21.15.106:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8120005020 con 0x7f812c1018d0 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.317+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f812c19fa80 con 0x7f812c1018d0 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.317+0000 7f811b7fe640 1 -- 172.21.15.49:0/3210077957 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81200049e0 con 0x7f812c1018d0 2024-03-18T01:01:52.319 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.317+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f812c19ffd0 con 0x7f812c1018d0 2024-03-18T01:01:52.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.318+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f812c1a0260 con 0x7f812c1018d0 2024-03-18T01:01:52.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.319+0000 7f811b7fe640 1 -- 172.21.15.49:0/3210077957 <== mon.2 v2:172.21.15.106:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8120005560 con 0x7f812c1018d0 2024-03-18T01:01:52.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.319+0000 7f811b7fe640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81140a2880 0x7f81140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:52.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.320+0000 7f812a59c640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81140a2880 0x7f81140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:52.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.320+0000 7f811b7fe640 1 -- 172.21.15.49:0/3210077957 <== mon.2 v2:172.21.15.106:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8699+0+0 (secure 0 0 0) 0x7f81200ceaf0 con 0x7f812c1018d0 2024-03-18T01:01:52.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.320+0000 7f812a59c640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81140a2880 0x7f81140a4d40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f811c02c8e0 tx=0x7f811c005b50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:52.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.323+0000 7f811b7fe640 1 -- 172.21.15.49:0/3210077957 <== mon.2 v2:172.21.15.106:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81200051d0 con 0x7f812c1018d0 2024-03-18T01:01:52.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/559495117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:01:52.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/2059030262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:01:52.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3135644250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:01:52.431 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c[67104]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.431 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c[67104]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.431 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-mds[67110]: -- 172.21.15.107:0/570256910 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555a97171a40 con 0x555a97145000 2024-03-18T01:01:52.431 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c[67104]: starting mds.c at 2024-03-18T01:01:52.431 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f[66788]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.431 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f[66788]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.431 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-mds[66792]: -- 172.21.15.107:0/2532323434 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dd28eb1a40 con 0x55dd28e84800 2024-03-18T01:01:52.431 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f[66788]: starting mds.f at 2024-03-18T01:01:52.431 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i[67415]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.432 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i[67415]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.432 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-mds[67419]: -- 172.21.15.107:0/2235176928 <== mon.2 v2:172.21.15.106:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56306bd424e0 con 0x56306bd18c00 2024-03-18T01:01:52.432 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-mds[67419]: -- 172.21.15.107:0/2235176928 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56306bd45a40 con 0x56306bd18c00 2024-03-18T01:01:52.432 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i[67415]: starting mds.i at 2024-03-18T01:01:52.432 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l[66476]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.432 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l[66476]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.432 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-mds[66480]: -- 172.21.15.107:0/2940023134 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5579c3473a40 con 0x5579c3446800 2024-03-18T01:01:52.432 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:01:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l[66476]: starting mds.l at 2024-03-18T01:01:52.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/559495117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:01:52.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/2059030262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T01:01:52.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3135644250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T01:01:52.446 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h[66542]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.446 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h[66542]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.446 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-mds[66551]: -- 172.21.15.106:0/533862866 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56363725da40 con 0x563637231000 2024-03-18T01:01:52.446 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h[66542]: starting mds.h at 2024-03-18T01:01:52.446 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k[65907]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.447 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k[65907]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.447 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-mds[65911]: -- 172.21.15.106:0/1027644511 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56061f953a40 con 0x56061f926c00 2024-03-18T01:01:52.447 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k[65907]: starting mds.k at 2024-03-18T01:01:52.447 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b[66856]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.447 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b[66856]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.448 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-mds[66860]: -- 172.21.15.106:0/1787819693 <== mon.2 v2:172.21.15.106:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5650cf427a40 con 0x5650cf3fa800 2024-03-18T01:01:52.448 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b[66856]: starting mds.b at 2024-03-18T01:01:52.448 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e[66233]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.448 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e[66233]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.448 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-mds[66237]: -- 172.21.15.106:0/2626426290 <== mon.1 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55769625ba40 con 0x55769622f000 2024-03-18T01:01:52.448 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:01:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e[66233]: starting mds.e at 2024-03-18T01:01:52.482 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a[73529]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.482 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a[73529]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.482 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-mds[73533]: -- 172.21.15.49:0/2510424192 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5602fde79a40 con 0x5602fde4cc00 2024-03-18T01:01:52.482 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a[73529]: starting mds.a at 2024-03-18T01:01:52.483 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g[73175]: ignoring --setuser ceph since I am not root 2024-03-18T01:01:52.483 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g[73175]: ignoring --setgroup ceph since I am not root 2024-03-18T01:01:52.483 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-mds[73179]: -- 172.21.15.49:0/4193577894 <== mon.0 v2:172.21.15.49:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560275571a40 con 0x560275544800 2024-03-18T01:01:52.483 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g[73175]: starting mds.g at 2024-03-18T01:01:52.483 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a[32126]: 2024-03-18T01:01:52.179+0000 7f49d8a3b700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:01:52.547 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:52.548+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f812c1a0430 con 0x7f812c1018d0 2024-03-18T01:01:53.187 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.188+0000 7f811b7fe640 1 -- 172.21.15.49:0/3210077957 <== mon.2 v2:172.21.15.106:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) ==== 104+0+0 (secure 0 0 0) 0x7f812009b9c0 con 0x7f812c1018d0 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81140a2880 msgr2=0x7f81140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81140a2880 0x7f81140a4d40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f811c02c8e0 tx=0x7f811c005b50 comp rx=0 tx=0).stop 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f812c1018d0 msgr2=0x7f812c1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f812c1018d0 0x7f812c1980b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f812000e9f0 tx=0x7f812000eeb0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 shutdown_connections 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7f81140a2880 0x7f81140a4d40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7f812c1018d0 0x7f812c1980b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7f812c102290 0x7f812c19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 --2- 172.21.15.49:0/3210077957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f812c1006e0 0x7f812c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 >> 172.21.15.49:0/3210077957 conn(0x7f812c0fbc70 msgr2=0x7f812c0fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 shutdown_connections 2024-03-18T01:01:53.190 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.190+0000 7f8130b59640 1 -- 172.21.15.49:0/3210077957 wait complete. 2024-03-18T01:01:53.203 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 -- 172.21.15.49:0/4145459113 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb8100a10 msgr2=0x7fcdb8100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/4145459113 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb8100a10 0x7fcdb8100e70 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fcdac009900 tx=0x7fcdac02bc30 comp rx=0 tx=0).stop 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 -- 172.21.15.49:0/4145459113 shutdown_connections 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/4145459113 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb81013b0 0x7fcdb8101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/4145459113 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb8100a10 0x7fcdb8100e70 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/4145459113 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdb81000b0 0x7fcdb81004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 -- 172.21.15.49:0/4145459113 >> 172.21.15.49:0/4145459113 conn(0x7fcdb80fbc50 msgr2=0x7fcdb80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 -- 172.21.15.49:0/4145459113 shutdown_connections 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 -- 172.21.15.49:0/4145459113 wait complete. 2024-03-18T01:01:53.320 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.318+0000 7fcdbe11b640 1 Processor -- start 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdbe11b640 1 -- start start 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdbe11b640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb81000b0 0x7fcdb8199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdbe11b640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdb8100a10 0x7fcdb819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdbe11b640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb81013b0 0x7fcdb81a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdbe11b640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb810f6a0 con 0x7fcdb81013b0 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdbe11b640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb810f520 con 0x7fcdb8100a10 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdbe11b640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb810f820 con 0x7fcdb81000b0 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb81013b0 0x7fcdb81a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb81013b0 0x7fcdb81a11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60816/0 (socket says 172.21.15.49:60816) 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb77fe640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb81000b0 0x7fcdb8199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb7fff640 1 -- 172.21.15.49:0/3655619259 learned_addr learned my addr 172.21.15.49:0/3655619259 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb6ffd640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdb8100a10 0x7fcdb819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb77fe640 1 -- 172.21.15.49:0/3655619259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdb8100a10 msgr2=0x7fcdb819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:53.321 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb77fe640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdb8100a10 0x7fcdb819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb77fe640 1 -- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb81013b0 msgr2=0x7fcdb81a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb77fe640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb81013b0 0x7fcdb81a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb77fe640 1 -- 172.21.15.49:0/3655619259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb81a1890 con 0x7fcdb81000b0 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb7fff640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb81013b0 0x7fcdb81a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.319+0000 7fcdb77fe640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb81000b0 0x7fcdb8199c80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcda800c970 tx=0x7fcda800ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.320+0000 7fcdb4ff9640 1 -- 172.21.15.49:0/3655619259 <== mon.1 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda8007bb0 con 0x7fcdb81000b0 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.320+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb81a1b80 con 0x7fcdb81000b0 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.320+0000 7fcdb4ff9640 1 -- 172.21.15.49:0/3655619259 <== mon.1 v2:172.21.15.107:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcda8007d50 con 0x7fcdb81000b0 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.320+0000 7fcdb4ff9640 1 -- 172.21.15.49:0/3655619259 <== mon.1 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda8018820 con 0x7fcdb81000b0 2024-03-18T01:01:53.322 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.320+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb81a2120 con 0x7fcdb81000b0 2024-03-18T01:01:53.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.321+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdb810af10 con 0x7fcdb81000b0 2024-03-18T01:01:53.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.322+0000 7fcdb4ff9640 1 -- 172.21.15.49:0/3655619259 <== mon.1 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcda80189c0 con 0x7fcdb81000b0 2024-03-18T01:01:53.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.323+0000 7fcdb4ff9640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcda00a2800 0x7fcda00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:53.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.323+0000 7fcdb6ffd640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcda00a2800 0x7fcda00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:53.323 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.323+0000 7fcdb4ff9640 1 -- 172.21.15.49:0/3655619259 <== mon.1 v2:172.21.15.107:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8699+0+0 (secure 0 0 0) 0x7fcda80cef80 con 0x7fcdb81000b0 2024-03-18T01:01:53.324 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.323+0000 7fcdb6ffd640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcda00a2800 0x7fcda00a4cc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fcdac02c240 tx=0x7fcdac02df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:53.325 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.326+0000 7fcdb4ff9640 1 -- 172.21.15.49:0/3655619259 <== mon.1 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcda8014030 con 0x7fcdb81000b0 2024-03-18T01:01:53.374 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:53 smithi049 ceph-mon[32130]: pgmap v703: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-18T01:01:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:53 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/70834489' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T01:01:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:53 smithi049 ceph-mon[32130]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T01:01:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:53 smithi049 ceph-mon[32130]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:01:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:53 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/70834489' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T01:01:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:53 smithi049 ceph-mon[32130]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T01:01:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:53 smithi049 ceph-mon[32130]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T01:01:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:53 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3210077957' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:01:53.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:53 smithi049 ceph-mon[32130]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:01:53.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:53 smithi107 ceph-mon[36270]: pgmap v703: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-18T01:01:53.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:53 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/70834489' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T01:01:53.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:53 smithi107 ceph-mon[36270]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T01:01:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:53 smithi107 ceph-mon[36270]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:01:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:53 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/70834489' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T01:01:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:53 smithi107 ceph-mon[36270]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T01:01:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:53 smithi107 ceph-mon[36270]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T01:01:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:53 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3210077957' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:01:53.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:53 smithi107 ceph-mon[36270]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:01:53.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:53 smithi106 ceph-mon[35899]: pgmap v703: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-03-18T01:01:53.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:53 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/70834489' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T01:01:53.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:53 smithi106 ceph-mon[35899]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T01:01:53.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:53 smithi106 ceph-mon[35899]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T01:01:53.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:53 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/70834489' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T01:01:53.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:53 smithi106 ceph-mon[35899]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T01:01:53.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:53 smithi106 ceph-mon[35899]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T01:01:53.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:53 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3210077957' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:01:53.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:53 smithi106 ceph-mon[35899]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T01:01:53.509 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:53.510+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fcdb802d1b0 con 0x7fcdb81000b0 2024-03-18T01:01:54.191 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.191+0000 7fcdb4ff9640 1 -- 172.21.15.49:0/3655619259 <== mon.1 v2:172.21.15.107:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v90) ==== 181+0+0 (secure 0 0 0) 0x7fcda809be50 con 0x7fcdb81000b0 2024-03-18T01:01:54.191 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'cephfs_metadata' removed 2024-03-18T01:01:54.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcda00a2800 msgr2=0x7fcda00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:54.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcda00a2800 0x7fcda00a4cc0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fcdac02c240 tx=0x7fcdac02df90 comp rx=0 tx=0).stop 2024-03-18T01:01:54.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb81000b0 msgr2=0x7fcdb8199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:54.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb81000b0 0x7fcdb8199c80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcda800c970 tx=0x7fcda800ce30 comp rx=0 tx=0).stop 2024-03-18T01:01:54.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 shutdown_connections 2024-03-18T01:01:54.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fcda00a2800 0x7fcda00a4cc0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.194 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fcdb8100a10 0x7fcdb819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fcdb81013b0 0x7fcdb81a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 --2- 172.21.15.49:0/3655619259 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcdb81000b0 0x7fcdb8199c80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 >> 172.21.15.49:0/3655619259 conn(0x7fcdb80fbc50 msgr2=0x7fcdb80fd2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:54.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 shutdown_connections 2024-03-18T01:01:54.195 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.194+0000 7fcdbe11b640 1 -- 172.21.15.49:0/3655619259 wait complete. 2024-03-18T01:01:54.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T01:01:54.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.327+0000 7fc3053e6640 1 -- 172.21.15.49:0/2783458683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3001022b0 msgr2=0x7fc300108b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:54.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.327+0000 7fc3053e6640 1 --2- 172.21.15.49:0/2783458683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3001022b0 0x7fc300108b40 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8009ca0 tx=0x7fc2f802c0f0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.327+0000 7fc3053e6640 1 -- 172.21.15.49:0/2783458683 shutdown_connections 2024-03-18T01:01:54.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.327+0000 7fc3053e6640 1 --2- 172.21.15.49:0/2783458683 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3001018f0 0x7fc300101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.327+0000 7fc3053e6640 1 --2- 172.21.15.49:0/2783458683 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc3001022b0 0x7fc300108b40 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.329 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.327+0000 7fc3053e6640 1 --2- 172.21.15.49:0/2783458683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc300100700 0x7fc300100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.327+0000 7fc3053e6640 1 -- 172.21.15.49:0/2783458683 >> 172.21.15.49:0/2783458683 conn(0x7fc3000fbc90 msgr2=0x7fc3000fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.327+0000 7fc3053e6640 1 -- 172.21.15.49:0/2783458683 shutdown_connections 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.327+0000 7fc3053e6640 1 -- 172.21.15.49:0/2783458683 wait complete. 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc3053e6640 1 Processor -- start 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc3053e6640 1 -- start start 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc3053e6640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc300100700 0x7fc300197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc3053e6640 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3001018f0 0x7fc300197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc3053e6640 1 --2- >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3001022b0 0x7fc30019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc3053e6640 1 -- --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_getmap magic: 0 -- 0x7fc30010c380 con 0x7fc300100700 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc3053e6640 1 -- --> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] -- mon_getmap magic: 0 -- 0x7fc30010c200 con 0x7fc3001022b0 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc3053e6640 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 -- 0x7fc30010c500 con 0x7fc3001018f0 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc2feffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc300100700 0x7fc300197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:54.330 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc2feffd640 1 --2- >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc300100700 0x7fc300197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.49:3300/0 says I am v2:172.21.15.49:60822/0 (socket says 172.21.15.49:60822) 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.328+0000 7fc2feffd640 1 -- 172.21.15.49:0/3689295074 learned_addr learned my addr 172.21.15.49:0/3689295074 (peer_addr_for_me v2:172.21.15.49:0/0) 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2ff7fe640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3001022b0 0x7fc30019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2fe7fc640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3001018f0 0x7fc300197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2feffd640 1 -- 172.21.15.49:0/3689295074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3001018f0 msgr2=0x7fc300197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2feffd640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3001018f0 0x7fc300197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2feffd640 1 -- 172.21.15.49:0/3689295074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3001022b0 msgr2=0x7fc30019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2feffd640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3001022b0 0x7fc30019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2feffd640 1 -- 172.21.15.49:0/3689295074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30019f6c0 con 0x7fc300100700 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2feffd640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc300100700 0x7fc300197aa0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000d950 tx=0x7fc2f000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2e7fff640 1 -- 172.21.15.49:0/3689295074 <== mon.0 v2:172.21.15.49:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0017070 con 0x7fc300100700 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30019f9b0 con 0x7fc300100700 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2e7fff640 1 -- 172.21.15.49:0/3689295074 <== mon.0 v2:172.21.15.49:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc2f0009c30 con 0x7fc300100700 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc2e7fff640 1 -- 172.21.15.49:0/3689295074 <== mon.0 v2:172.21.15.49:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f000f600 con 0x7fc300100700 2024-03-18T01:01:54.331 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.329+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30019feb0 con 0x7fc300100700 2024-03-18T01:01:54.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.330+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3001a0080 con 0x7fc300100700 2024-03-18T01:01:54.332 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.331+0000 7fc2e7fff640 1 -- 172.21.15.49:0/3689295074 <== mon.0 v2:172.21.15.49:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc2f001a070 con 0x7fc300100700 2024-03-18T01:01:54.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.332+0000 7fc2e7fff640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2e80a2800 0x7fc2e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T01:01:54.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.332+0000 7fc2fe7fc640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2e80a2800 0x7fc2e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T01:01:54.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.332+0000 7fc2e7fff640 1 -- 172.21.15.49:0/3689295074 <== mon.0 v2:172.21.15.49:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8116+0+0 (secure 0 0 0) 0x7fc2f00ce3f0 con 0x7fc300100700 2024-03-18T01:01:54.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.333+0000 7fc2fe7fc640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2e80a2800 0x7fc2e80a4cc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc300198fb0 tx=0x7fc2f400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T01:01:54.336 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.337+0000 7fc2e7fff640 1 -- 172.21.15.49:0/3689295074 <== mon.0 v2:172.21.15.49:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc2f0009de0 con 0x7fc300100700 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: Cluster is now healthy 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.49:6838/1999640807,v1:172.21.15.49:6839/1999640807] up:boot 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6836/2804265801,v1:172.21.15.107:6838/2804265801] up:boot 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6832/2346859285,v1:172.21.15.107:6834/2346859285] up:boot 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.49:6840/1028693926,v1:172.21.15.49:6841/1028693926] up:boot 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6833/2862883345,v1:172.21.15.107:6835/2862883345] up:boot 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6837/678262597,v1:172.21.15.106:6839/678262597] up:boot 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6833/2700756738,v1:172.21.15.106:6835/2700756738] up:boot 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6832/3742797587,v1:172.21.15.106:6834/3742797587] up:boot 2024-03-18T01:01:54.375 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.107:6837/1365841604,v1:172.21.15.107:6839/1365841604] up:boot 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: mds.? [v2:172.21.15.106:6836/3912308395,v1:172.21.15.106:6838/3912308395] up:boot 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: fsmap 12 up:standby 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T01:01:54.376 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3655619259' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:01:54.377 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:54 smithi049 ceph-mon[32130]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:01:54.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T01:01:54.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: Cluster is now healthy 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.49:6838/1999640807,v1:172.21.15.49:6839/1999640807] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6836/2804265801,v1:172.21.15.107:6838/2804265801] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6832/2346859285,v1:172.21.15.107:6834/2346859285] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.49:6840/1028693926,v1:172.21.15.49:6841/1028693926] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6833/2862883345,v1:172.21.15.107:6835/2862883345] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6837/678262597,v1:172.21.15.106:6839/678262597] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6833/2700756738,v1:172.21.15.106:6835/2700756738] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6832/3742797587,v1:172.21.15.106:6834/3742797587] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.107:6837/1365841604,v1:172.21.15.107:6839/1365841604] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: mds.? [v2:172.21.15.106:6836/3912308395,v1:172.21.15.106:6838/3912308395] up:boot 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: fsmap 12 up:standby 2024-03-18T01:01:54.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3655619259' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:01:54.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:54 smithi107 ceph-mon[36270]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: Cluster is now healthy 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.49:6838/1999640807,v1:172.21.15.49:6839/1999640807] up:boot 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6836/2804265801,v1:172.21.15.107:6838/2804265801] up:boot 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6832/2346859285,v1:172.21.15.107:6834/2346859285] up:boot 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.49:6840/1028693926,v1:172.21.15.49:6841/1028693926] up:boot 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6833/2862883345,v1:172.21.15.107:6835/2862883345] up:boot 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6837/678262597,v1:172.21.15.106:6839/678262597] up:boot 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6833/2700756738,v1:172.21.15.106:6835/2700756738] up:boot 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6832/3742797587,v1:172.21.15.106:6834/3742797587] up:boot 2024-03-18T01:01:54.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.107:6837/1365841604,v1:172.21.15.107:6839/1365841604] up:boot 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: mds.? [v2:172.21.15.106:6836/3912308395,v1:172.21.15.106:6838/3912308395] up:boot 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: fsmap 12 up:standby 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3655619259' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:01:54.447 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:54 smithi106 ceph-mon[35899]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:01:54.526 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:54.526+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 --> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc3001a0250 con 0x7fc300100700 2024-03-18T01:01:55.188 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.188+0000 7fc2e7fff640 1 -- 172.21.15.49:0/3689295074 <== mon.0 v2:172.21.15.49:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v91) ==== 169+0+0 (secure 0 0 0) 0x7fc2f009ae60 con 0x7fc300100700 2024-03-18T01:01:55.188 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'cephfs_data' removed 2024-03-18T01:01:55.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2e80a2800 msgr2=0x7fc2e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:55.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2e80a2800 0x7fc2e80a4cc0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc300198fb0 tx=0x7fc2f400b390 comp rx=0 tx=0).stop 2024-03-18T01:01:55.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc300100700 msgr2=0x7fc300197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T01:01:55.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc300100700 0x7fc300197aa0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000d950 tx=0x7fc2f000de10 comp rx=0 tx=0).stop 2024-03-18T01:01:55.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 shutdown_connections 2024-03-18T01:01:55.192 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:6800/1246732627,v1:172.21.15.49:6801/1246732627] conn(0x7fc2e80a2800 0x7fc2e80a4cc0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:55.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] conn(0x7fc3001022b0 0x7fc30019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:55.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc3001018f0 0x7fc300197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:55.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 --2- 172.21.15.49:0/3689295074 >> [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] conn(0x7fc300100700 0x7fc300197aa0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T01:01:55.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.192+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 >> 172.21.15.49:0/3689295074 conn(0x7fc3000fbc90 msgr2=0x7fc300069db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T01:01:55.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.193+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 shutdown_connections 2024-03-18T01:01:55.193 INFO:teuthology.orchestra.run.smithi049.stderr:2024-03-18T01:01:55.193+0000 7fc3053e6640 1 -- 172.21.15.49:0/3689295074 wait complete. 2024-03-18T01:01:55.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:01:55.272 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:01:55.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:01:55.332 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T01:01:55.332 DEBUG:teuthology.orchestra.run.smithi049:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T01:01:55.348 DEBUG:teuthology.orchestra.run.smithi106:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T01:01:55.363 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T01:01:55.387 INFO:tasks.cephadm:Stopping all daemons... 2024-03-18T01:01:55.388 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-18T01:01:55.388 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a 2024-03-18T01:01:55.625 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:55 smithi049 ceph-mon[32130]: pgmap v705: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 409 B/s wr, 10 op/s 2024-03-18T01:01:55.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:55 smithi049 ceph-mon[32130]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:01:55.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:55 smithi049 ceph-mon[32130]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T01:01:55.675 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:55 smithi049 ceph-mon[32130]: from='client.? 172.21.15.49:0/3689295074' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:01:55.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:55 smithi049 ceph-mon[32130]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:01:55.676 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:55 smithi049 systemd[1]: Stopping Ceph mon.a for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:01:55.676 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:55 smithi106 ceph-mon[35899]: pgmap v705: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 409 B/s wr, 10 op/s 2024-03-18T01:01:55.676 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:55 smithi106 ceph-mon[35899]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:01:55.676 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:55 smithi106 ceph-mon[35899]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T01:01:55.677 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:55 smithi106 ceph-mon[35899]: from='client.? 172.21.15.49:0/3689295074' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:01:55.677 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:55 smithi106 ceph-mon[35899]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:01:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:55 smithi107 ceph-mon[36270]: pgmap v705: 97 pgs: 97 active+clean; 16 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 409 B/s wr, 10 op/s 2024-03-18T01:01:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:55 smithi107 ceph-mon[36270]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T01:01:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:55 smithi107 ceph-mon[36270]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T01:01:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:55 smithi107 ceph-mon[36270]: from='client.? 172.21.15.49:0/3689295074' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T01:01:55.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:55 smithi107 ceph-mon[36270]: from='mgr.14150 172.21.15.49:0/1107003799' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T01:01:56.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a[32126]: 2024-03-18T01:01:55.735+0000 7f49de246700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:01:56.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a[32126]: 2024-03-18T01:01:55.735+0000 7f49de246700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-18T01:01:56.125 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:55 smithi049 podman[99480]: 2024-03-18 01:01:55.789739452 +0000 UTC m=+0.181615568 container died fdaeca8c59356a016730a979fb4d4f2bee1498b4002376b708941295218b679f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:01:56.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:56 smithi049 podman[99480]: 2024-03-18 01:01:56.345603302 +0000 UTC m=+0.737479411 container cleanup fdaeca8c59356a016730a979fb4d4f2bee1498b4002376b708941295218b679f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-18T01:01:56.563 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:56 smithi049 bash[99480]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a 2024-03-18T01:01:56.792 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a.service' 2024-03-18T01:01:56.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:56 smithi049 podman[99513]: 2024-03-18 01:01:56.563958054 +0000 UTC m=+0.770261488 container remove fdaeca8c59356a016730a979fb4d4f2bee1498b4002376b708941295218b679f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:01:56.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:56 smithi049 systemd[1]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T01:01:56.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:56 smithi049 systemd[1]: Stopped Ceph mon.a for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T01:01:56.821 INFO:journalctl@ceph.mon.a.smithi049.stdout:Mar 18 01:01:56 smithi049 systemd[1]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.a.service: Consumed 24.948s CPU time. 2024-03-18T01:01:56.837 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:01:56.838 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-18T01:01:56.838 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-18T01:01:56.838 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.b 2024-03-18T01:01:57.186 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:56 smithi106 systemd[1]: Stopping Ceph mon.b for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:01:57.445 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:57 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-b[35894]: 2024-03-18T01:01:57.185+0000 7f8ed701d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:01:57.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:57 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-b[35894]: 2024-03-18T01:01:57.185+0000 7f8ed701d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-18T01:01:57.446 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:57 smithi106 podman[71721]: 2024-03-18 01:01:57.400349409 +0000 UTC m=+0.339842223 container died d10ddbeaacf39c8f59374f81656b5250e78b76277c62985423e14fb7b0812c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:01:58.122 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:57 smithi106 podman[71721]: 2024-03-18 01:01:57.820113823 +0000 UTC m=+0.759606637 container cleanup d10ddbeaacf39c8f59374f81656b5250e78b76277c62985423e14fb7b0812c68 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T01:01:58.122 INFO:journalctl@ceph.mon.b.smithi106.stdout:Mar 18 01:01:57 smithi106 bash[71721]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-b 2024-03-18T01:01:58.334 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.b.service' 2024-03-18T01:01:58.386 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:01:58.386 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-18T01:01:58.387 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-18T01:01:58.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.c 2024-03-18T01:01:58.680 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:58 smithi107 systemd[1]: Stopping Ceph mon.c for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:01:59.024 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:58 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-c[36253]: 2024-03-18T01:01:58.731+0000 7fbd7b1e8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:01:59.024 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:58 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-c[36253]: 2024-03-18T01:01:58.731+0000 7fbd7b1e8700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-18T01:01:59.024 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:58 smithi107 podman[71947]: 2024-03-18 01:01:58.752596159 +0000 UTC m=+0.145255170 container died 92247214c5cb42348a7fcc0a404471892ead07faa9b3a8519846ded5bc03e235 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-03-18T01:01:59.399 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:59 smithi107 podman[71947]: 2024-03-18 01:01:59.115638723 +0000 UTC m=+0.508297752 container cleanup 92247214c5cb42348a7fcc0a404471892ead07faa9b3a8519846ded5bc03e235 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-c, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-18T01:01:59.399 INFO:journalctl@ceph.mon.c.smithi107.stdout:Mar 18 01:01:59 smithi107 bash[71947]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mon-c 2024-03-18T01:01:59.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mon.c.service' 2024-03-18T01:01:59.583 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:01:59.583 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-18T01:01:59.583 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-18T01:01:59.583 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.x 2024-03-18T01:01:59.875 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 01:01:59 smithi049 systemd[1]: Stopping Ceph mgr.x for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:00.260 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 01:01:59 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T01:01:59.916+0000 7fc7efff8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:02:00.260 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 01:01:59 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x[32370]: 2024-03-18T01:01:59.916+0000 7fc7efff8700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-18T01:02:00.260 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 01:01:59 smithi049 podman[99623]: 2024-03-18 01:01:59.948582711 +0000 UTC m=+0.160973897 container died 440c098891eb0f0fb99964bff7cd5d8463034c8989277afefdf55ef266a7e049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:02:00.624 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 01:02:00 smithi049 podman[99623]: 2024-03-18 01:02:00.352168151 +0000 UTC m=+0.564559380 container cleanup 440c098891eb0f0fb99964bff7cd5d8463034c8989277afefdf55ef266a7e049 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:02:00.625 INFO:journalctl@ceph.mgr.x.smithi049.stdout:Mar 18 01:02:00 smithi049 bash[99623]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-x 2024-03-18T01:02:00.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-18T01:02:00.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:02:00.785 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-18T01:02:00.786 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-18T01:02:00.786 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.y 2024-03-18T01:02:01.149 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 01:02:00 smithi106 systemd[1]: Stopping Ceph mgr.y for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:01.446 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 01:02:01 smithi106 podman[71843]: 2024-03-18 01:02:01.150140998 +0000 UTC m=+0.174097834 container died c2df8d4435b81f3cba5a8c64947ab1fd563fdaacd5001ab50c6a3c1d7111f15e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-03-18T01:02:01.813 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 01:02:01 smithi106 podman[71843]: 2024-03-18 01:02:01.553894669 +0000 UTC m=+0.577851492 container cleanup c2df8d4435b81f3cba5a8c64947ab1fd563fdaacd5001ab50c6a3c1d7111f15e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-18T01:02:01.813 INFO:journalctl@ceph.mgr.y.smithi106.stdout:Mar 18 01:02:01 smithi106 bash[71843]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-y 2024-03-18T01:02:01.965 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-18T01:02:02.002 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:02:02.003 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-18T01:02:02.003 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-18T01:02:02.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.z 2024-03-18T01:02:02.349 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 01:02:02 smithi107 systemd[1]: Stopping Ceph mgr.z for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:02.646 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 01:02:02 smithi107 podman[72069]: 2024-03-18 01:02:02.350239293 +0000 UTC m=+0.155523742 container died 2d7aad30a0be14c5744966df721e0ddc434248c30a84a2c1f4c6c87c4575ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:02:02.930 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 01:02:02 smithi107 podman[72069]: 2024-03-18 01:02:02.746326464 +0000 UTC m=+0.551610960 container cleanup 2d7aad30a0be14c5744966df721e0ddc434248c30a84a2c1f4c6c87c4575ff99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-18T01:02:02.930 INFO:journalctl@ceph.mgr.z.smithi107.stdout:Mar 18 01:02:02 smithi107 bash[72069]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mgr-z 2024-03-18T01:02:03.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-18T01:02:03.154 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:02:03.154 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-18T01:02:03.154 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-18T01:02:03.154 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.0 2024-03-18T01:02:03.498 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:03 smithi049 systemd[1]: Stopping Ceph osd.0 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:03.874 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:03 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0[46641]: 2024-03-18T01:02:03.498+0000 7f4917da0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:02:03.874 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:03 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0[46641]: 2024-03-18T01:02:03.498+0000 7f4917da0700 -1 osd.0 91 *** Got signal Terminated *** 2024-03-18T01:02:03.875 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:03 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0[46641]: 2024-03-18T01:02:03.498+0000 7f4917da0700 -1 osd.0 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:02:09.183 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:08 smithi049 podman[99746]: 2024-03-18 01:02:08.910026137 +0000 UTC m=+5.537376025 container died f4ecaa5a576c4196f626a2a418266327f71cebe0bfa98b9fc106a92e7cebef73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-18T01:02:09.592 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:09 smithi049 podman[99746]: 2024-03-18 01:02:09.274456514 +0000 UTC m=+5.901806402 container cleanup f4ecaa5a576c4196f626a2a418266327f71cebe0bfa98b9fc106a92e7cebef73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-18T01:02:09.592 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:09 smithi049 bash[99746]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0 2024-03-18T01:02:09.874 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:09 smithi049 podman[99766]: 2024-03-18 01:02:09.592987469 +0000 UTC m=+0.679417268 container remove f4ecaa5a576c4196f626a2a418266327f71cebe0bfa98b9fc106a92e7cebef73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-18T01:02:10.187 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:09 smithi049 podman[99856]: 2024-03-18 01:02:09.800317392 +0000 UTC m=+0.017982110 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:02:10.187 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:09 smithi049 podman[99856]: 2024-03-18 01:02:09.918863238 +0000 UTC m=+0.136527956 container create 846871af59612702dc433172fa2e0961ac521560817155c1817e76114f88696e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-18T01:02:10.565 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:10 smithi049 podman[99856]: 2024-03-18 01:02:10.188171421 +0000 UTC m=+0.405836139 container init 846871af59612702dc433172fa2e0961ac521560817155c1817e76114f88696e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:02:10.565 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:10 smithi049 podman[99856]: 2024-03-18 01:02:10.192417845 +0000 UTC m=+0.410082582 container start 846871af59612702dc433172fa2e0961ac521560817155c1817e76114f88696e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:02:10.565 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:10 smithi049 podman[99856]: 2024-03-18 01:02:10.288391434 +0000 UTC m=+0.506056170 container attach 846871af59612702dc433172fa2e0961ac521560817155c1817e76114f88696e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T01:02:10.857 INFO:journalctl@ceph.osd.0.smithi049.stdout:Mar 18 01:02:10 smithi049 podman[99856]: 2024-03-18 01:02:10.566534858 +0000 UTC m=+0.784199579 container died 846871af59612702dc433172fa2e0961ac521560817155c1817e76114f88696e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-18T01:02:11.197 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.0.service' 2024-03-18T01:02:11.248 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:02:11.248 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-18T01:02:11.249 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-18T01:02:11.249 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.1 2024-03-18T01:02:11.583 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:11 smithi106 systemd[1]: Stopping Ceph osd.1 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:11.945 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:11 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1[40489]: 2024-03-18T01:02:11.582+0000 7fd99bccd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:02:11.946 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:11 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1[40489]: 2024-03-18T01:02:11.582+0000 7fd99bccd700 -1 osd.1 91 *** Got signal Terminated *** 2024-03-18T01:02:11.946 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:11 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1[40489]: 2024-03-18T01:02:11.582+0000 7fd99bccd700 -1 osd.1 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:02:17.333 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:17 smithi106 podman[71967]: 2024-03-18 01:02:17.044203982 +0000 UTC m=+5.587383616 container died 8c41a7ba83067803ce81da83e27591df4b42d0500530da5639319ae7a1c60816 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T01:02:17.659 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:17 smithi106 podman[71967]: 2024-03-18 01:02:17.408621122 +0000 UTC m=+5.951800752 container cleanup 8c41a7ba83067803ce81da83e27591df4b42d0500530da5639319ae7a1c60816 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:02:17.659 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:17 smithi106 bash[71967]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1 2024-03-18T01:02:17.945 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:17 smithi106 podman[71984]: 2024-03-18 01:02:17.660343271 +0000 UTC m=+0.605526560 container remove 8c41a7ba83067803ce81da83e27591df4b42d0500530da5639319ae7a1c60816 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T01:02:18.373 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:18 smithi106 podman[72078]: 2024-03-18 01:02:17.964857838 +0000 UTC m=+0.017755875 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:02:18.373 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:18 smithi106 podman[72078]: 2024-03-18 01:02:18.095340313 +0000 UTC m=+0.148238341 container create b2255d9eac7e764151c818f3ebb6b56f9bb2e269ab88adbfb43bd1bb3a297efd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-03-18T01:02:18.696 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:18 smithi106 podman[72078]: 2024-03-18 01:02:18.373726165 +0000 UTC m=+0.426624204 container init b2255d9eac7e764151c818f3ebb6b56f9bb2e269ab88adbfb43bd1bb3a297efd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-18T01:02:18.696 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:18 smithi106 podman[72078]: 2024-03-18 01:02:18.377441089 +0000 UTC m=+0.430339118 container start b2255d9eac7e764151c818f3ebb6b56f9bb2e269ab88adbfb43bd1bb3a297efd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-18T01:02:18.696 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:18 smithi106 podman[72078]: 2024-03-18 01:02:18.499452898 +0000 UTC m=+0.552350935 container attach b2255d9eac7e764151c818f3ebb6b56f9bb2e269ab88adbfb43bd1bb3a297efd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T01:02:19.311 INFO:journalctl@ceph.osd.1.smithi106.stdout:Mar 18 01:02:18 smithi106 podman[72078]: 2024-03-18 01:02:18.950631505 +0000 UTC m=+1.003529535 container died b2255d9eac7e764151c818f3ebb6b56f9bb2e269ab88adbfb43bd1bb3a297efd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T01:02:19.650 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.1.service' 2024-03-18T01:02:19.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:02:19.692 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-18T01:02:19.692 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-18T01:02:19.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.2 2024-03-18T01:02:20.007 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:19 smithi107 systemd[1]: Stopping Ceph osd.2 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:20.430 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:20 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2[41435]: 2024-03-18T01:02:20.006+0000 7f5ef7c10700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:02:20.430 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:20 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2[41435]: 2024-03-18T01:02:20.006+0000 7f5ef7c10700 -1 osd.2 91 *** Got signal Terminated *** 2024-03-18T01:02:20.430 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:20 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2[41435]: 2024-03-18T01:02:20.006+0000 7f5ef7c10700 -1 osd.2 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:02:25.836 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:25 smithi107 podman[72191]: 2024-03-18 01:02:25.574249934 +0000 UTC m=+5.682604927 container died 1930f3f0c0e95a3b45ec1e58a4539cf52ff870040fb42343eb8145048a9de06f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-18T01:02:26.115 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:25 smithi107 podman[72191]: 2024-03-18 01:02:25.940184742 +0000 UTC m=+6.048539728 container cleanup 1930f3f0c0e95a3b45ec1e58a4539cf52ff870040fb42343eb8145048a9de06f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-18T01:02:26.115 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:25 smithi107 bash[72191]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2 2024-03-18T01:02:26.430 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:26 smithi107 podman[72208]: 2024-03-18 01:02:26.116404383 +0000 UTC m=+0.537579579 container remove 1930f3f0c0e95a3b45ec1e58a4539cf52ff870040fb42343eb8145048a9de06f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-03-18T01:02:26.811 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:26 smithi107 podman[72302]: 2024-03-18 01:02:26.429365056 +0000 UTC m=+0.017393599 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:02:26.811 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:26 smithi107 podman[72302]: 2024-03-18 01:02:26.559199222 +0000 UTC m=+0.147227801 container create 1b2a3f2b8e73318b507b5f914c6b1fcc974b99c81b317fa8990384224c678662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:02:27.180 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:26 smithi107 podman[72302]: 2024-03-18 01:02:26.811712052 +0000 UTC m=+0.399740602 container init 1b2a3f2b8e73318b507b5f914c6b1fcc974b99c81b317fa8990384224c678662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-18T01:02:27.180 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:26 smithi107 podman[72302]: 2024-03-18 01:02:26.814925844 +0000 UTC m=+0.402954375 container start 1b2a3f2b8e73318b507b5f914c6b1fcc974b99c81b317fa8990384224c678662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-03-18T01:02:27.180 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:26 smithi107 podman[72302]: 2024-03-18 01:02:26.911699849 +0000 UTC m=+0.499728401 container attach 1b2a3f2b8e73318b507b5f914c6b1fcc974b99c81b317fa8990384224c678662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T01:02:27.485 INFO:journalctl@ceph.osd.2.smithi107.stdout:Mar 18 01:02:27 smithi107 podman[72302]: 2024-03-18 01:02:27.223348076 +0000 UTC m=+0.811376612 container died 1b2a3f2b8e73318b507b5f914c6b1fcc974b99c81b317fa8990384224c678662 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-18T01:02:27.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.2.service' 2024-03-18T01:02:27.894 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:02:27.894 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-18T01:02:27.894 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-18T01:02:27.895 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.3 2024-03-18T01:02:28.228 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:27 smithi049 systemd[1]: Stopping Ceph osd.3 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:28.624 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:28 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3[53315]: 2024-03-18T01:02:28.228+0000 7fd545d8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:02:28.625 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:28 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3[53315]: 2024-03-18T01:02:28.228+0000 7fd545d8d700 -1 osd.3 91 *** Got signal Terminated *** 2024-03-18T01:02:28.625 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:28 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3[53315]: 2024-03-18T01:02:28.228+0000 7fd545d8d700 -1 osd.3 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:02:29.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:29 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:29.453+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:30.124 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:29 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3[53315]: 2024-03-18T01:02:29.644+0000 7fd542381700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:04.717280+0000 front 2024-03-18T01:02:04.717316+0000 (oldest deadline 2024-03-18T01:02:29.417475+0000) 2024-03-18T01:02:30.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:30 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:30.111+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:30.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:30 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:30.498+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:30.931 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:30 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:30.682+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:31.125 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:30 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3[53315]: 2024-03-18T01:02:30.675+0000 7fd542381700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:04.717280+0000 front 2024-03-18T01:02:04.717316+0000 (oldest deadline 2024-03-18T01:02:29.417475+0000) 2024-03-18T01:02:31.195 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:31 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:31.102+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:31.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:31.726+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:31.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:31.510+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:32.124 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:31 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3[53315]: 2024-03-18T01:02:31.697+0000 7fd542381700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:04.717280+0000 front 2024-03-18T01:02:04.717316+0000 (oldest deadline 2024-03-18T01:02:29.417475+0000) 2024-03-18T01:02:32.445 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:32 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:32.109+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:32.744 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:32 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:32.463+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:33.026 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:32 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3[53315]: 2024-03-18T01:02:32.666+0000 7fd542381700 -1 osd.3 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:04.717280+0000 front 2024-03-18T01:02:04.717316+0000 (oldest deadline 2024-03-18T01:02:29.417475+0000) 2024-03-18T01:02:33.180 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:32 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:32.744+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:33.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:33 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:33.077+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:33.593 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:33 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:33.338+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:33.773 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:33 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:33.507+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:33.874 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:33 smithi049 podman[99963]: 2024-03-18 01:02:33.755223111 +0000 UTC m=+5.653498654 container died 6b0a81a592fa2555e091b462a63f8d58db4b1df9d539ddad2eb434846ace6463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T01:02:33.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:33 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:33.592+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:34.180 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:33 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:33.773+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:34.315 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:34 smithi049 podman[99963]: 2024-03-18 01:02:34.162472212 +0000 UTC m=+6.060747762 container cleanup 6b0a81a592fa2555e091b462a63f8d58db4b1df9d539ddad2eb434846ace6463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:02:34.315 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:34 smithi049 bash[99963]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3 2024-03-18T01:02:34.445 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:34 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:34.049+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:34.580 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:34 smithi049 podman[99992]: 2024-03-18 01:02:34.414324076 +0000 UTC m=+0.657379143 container remove 6b0a81a592fa2555e091b462a63f8d58db4b1df9d539ddad2eb434846ace6463 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T01:02:34.581 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:34 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:34.378+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:34.866 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:34 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:34.579+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:34.866 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:34 smithi049 podman[100080]: 2024-03-18 01:02:34.642483794 +0000 UTC m=+0.018425312 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:02:34.866 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:34 smithi049 podman[100080]: 2024-03-18 01:02:34.740059128 +0000 UTC m=+0.116000651 container create b5c58673851266527f2bf5e7fda5de0bdcbd3fc2464d8c014036478b5a23b036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:02:34.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:34.795+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:34.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:34.554+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:35.124 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:34 smithi049 podman[100080]: 2024-03-18 01:02:34.984512831 +0000 UTC m=+0.360454349 container init b5c58673851266527f2bf5e7fda5de0bdcbd3fc2464d8c014036478b5a23b036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-18T01:02:35.125 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:34 smithi049 podman[100080]: 2024-03-18 01:02:34.987691609 +0000 UTC m=+0.363633128 container start b5c58673851266527f2bf5e7fda5de0bdcbd3fc2464d8c014036478b5a23b036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T01:02:35.125 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:35 smithi049 podman[100080]: 2024-03-18 01:02:35.076229738 +0000 UTC m=+0.452171272 container attach b5c58673851266527f2bf5e7fda5de0bdcbd3fc2464d8c014036478b5a23b036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-18T01:02:35.445 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:35 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:35.076+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:35.589 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:35 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:35.341+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:35.589 INFO:journalctl@ceph.osd.3.smithi049.stdout:Mar 18 01:02:35 smithi049 podman[100080]: 2024-03-18 01:02:35.287288715 +0000 UTC m=+0.663230239 container died b5c58673851266527f2bf5e7fda5de0bdcbd3fc2464d8c014036478b5a23b036 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-03-18T01:02:35.874 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:35 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:35.588+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:35.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.3.service' 2024-03-18T01:02:35.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:35.795+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:35.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:35.567+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:35.971 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:02:35.972 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-18T01:02:35.972 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-18T01:02:35.972 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.4 2024-03-18T01:02:36.282 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:36 smithi106 systemd[1]: Stopping Ceph osd.4 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:36.282 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:36 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:36.100+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:36.624 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:36 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:36.322+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:36.625 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:36 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:36.552+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:36.695 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:36 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4[47489]: 2024-03-18T01:02:36.282+0000 7fd3c9383700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:02:36.696 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:36 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4[47489]: 2024-03-18T01:02:36.282+0000 7fd3c9383700 -1 osd.4 91 *** Got signal Terminated *** 2024-03-18T01:02:36.696 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:36 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4[47489]: 2024-03-18T01:02:36.282+0000 7fd3c9383700 -1 osd.4 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:02:36.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:36.769+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:36.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:36.523+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:37.445 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:37 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:37.140+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:37.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:37 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:37.140+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:37.586 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:37 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:37.332+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:37.779 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:37 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:37.474+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:37.874 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:37 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:37.586+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:37.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:37 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:37.586+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:38.180 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:37 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:37.779+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:38.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:38 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:38.163+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:38.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:38 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:38.163+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:38.607 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:38.354+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:38.607 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:38.354+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:38.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:38 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:38.447+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:38.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:38 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:38.383+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:38.874 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:38.607+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:38.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:38 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:38.607+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:38.946 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:38 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4[47489]: 2024-03-18T01:02:38.646+0000 7fd3c5977700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:11.845599+0000 front 2024-03-18T01:02:11.845658+0000 (oldest deadline 2024-03-18T01:02:37.745674+0000) 2024-03-18T01:02:39.180 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:38 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:38.778+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:39.180 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:38 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:38.778+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:39.445 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:39 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:39.191+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:39.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:39 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:39.191+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:39.624 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:39 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:39.328+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:39.625 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:39 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:39.328+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:39.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:39 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:39.388+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:39.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:39 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:39.456+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:39.945 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:39 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4[47489]: 2024-03-18T01:02:39.665+0000 7fd3c5977700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:11.845599+0000 front 2024-03-18T01:02:11.845658+0000 (oldest deadline 2024-03-18T01:02:37.745674+0000) 2024-03-18T01:02:39.946 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:39 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:39.691+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:40.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:39 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:39.634+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:40.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:39 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:39.634+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:40.180 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:39 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:39.759+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:40.180 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:39 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:39.759+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:40.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:40 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:40.158+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:40.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:40 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:40.158+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:40.624 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:40 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:40.309+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:40.625 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:40 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:40.309+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:40.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:40 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:40.404+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:40.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:40 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:40.493+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:40.945 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:40 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:40.739+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:40.946 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:40 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4[47489]: 2024-03-18T01:02:40.686+0000 7fd3c5977700 -1 osd.4 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:11.845599+0000 front 2024-03-18T01:02:11.845658+0000 (oldest deadline 2024-03-18T01:02:37.745674+0000) 2024-03-18T01:02:41.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:40 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:40.634+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:41.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:40 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:40.634+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:41.180 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:40 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:40.715+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:41.180 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:40 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:40.715+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:41.623 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:41 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:41.355+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:41.623 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:41 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:41.355+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:41.675 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:41 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:41.421+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:41.675 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:41 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:41.491+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:41.695 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:41 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:41.202+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:41.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:41 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:41.202+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:41.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:41 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:41.623+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:41.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:41 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:41.623+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:41.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:41 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:41.675+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:41.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:41 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:41.675+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:42.150 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:41 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:41.767+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:42.150 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:41 smithi106 podman[72196]: 2024-03-18 01:02:41.846210733 +0000 UTC m=+5.683964468 container died 352b3ce8d8778aa0d9042f3e88645c20e11007f3e7ce7cb53e4e3ec9d1895559 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:02:42.442 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:42 smithi106 podman[72196]: 2024-03-18 01:02:42.24969769 +0000 UTC m=+6.087451427 container cleanup 352b3ce8d8778aa0d9042f3e88645c20e11007f3e7ce7cb53e4e3ec9d1895559 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T01:02:42.443 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:42 smithi106 bash[72196]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4 2024-03-18T01:02:42.443 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:42 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:42.207+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:42.443 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:42 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:42.207+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:42.696 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:42 smithi106 podman[72213]: 2024-03-18 01:02:42.442579174 +0000 UTC m=+0.593648377 container remove 352b3ce8d8778aa0d9042f3e88645c20e11007f3e7ce7cb53e4e3ec9d1895559 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-18T01:02:42.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:42 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:42.608+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:42.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:42 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:42.608+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:42.877 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:42 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:42.380+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:42.877 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:42 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:42.380+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:42.930 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:42 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:42.468+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:42.931 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:42 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:42.701+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:42.931 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:42 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:42.701+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:42.931 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:42 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:42.469+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:42.931 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:42 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:42.469+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:43.078 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:42 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:42.730+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:43.079 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:42 smithi106 podman[72305]: 2024-03-18 01:02:42.750783445 +0000 UTC m=+0.017571234 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:02:43.079 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:42 smithi106 podman[72305]: 2024-03-18 01:02:42.860194319 +0000 UTC m=+0.126982092 container create 93d8bb5a278313632686254923172863aaa1071a1ce137421826516736430bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-18T01:02:43.079 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:43 smithi106 podman[72305]: 2024-03-18 01:02:43.079343319 +0000 UTC m=+0.346131099 container init 93d8bb5a278313632686254923172863aaa1071a1ce137421826516736430bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T01:02:43.370 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:43 smithi106 podman[72305]: 2024-03-18 01:02:43.082614253 +0000 UTC m=+0.349402035 container start 93d8bb5a278313632686254923172863aaa1071a1ce137421826516736430bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T01:02:43.370 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:43 smithi106 podman[72305]: 2024-03-18 01:02:43.196426662 +0000 UTC m=+0.463214444 container attach 93d8bb5a278313632686254923172863aaa1071a1ce137421826516736430bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4-deactivate, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T01:02:43.371 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:43 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:43.236+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:43.371 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:43 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:43.236+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:43.682 INFO:journalctl@ceph.osd.4.smithi106.stdout:Mar 18 01:02:43 smithi106 podman[72305]: 2024-03-18 01:02:43.37123925 +0000 UTC m=+0.638027040 container died 93d8bb5a278313632686254923172863aaa1071a1ce137421826516736430bbf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-18T01:02:43.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:43 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:43.600+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:43.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:43 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:43.600+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:43.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:43 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:43.409+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:43.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:43 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:43.409+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:43.930 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:43 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:43.440+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:43.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:43 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:43.672+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:43.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:43 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:43.673+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:43.931 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:43 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:43.448+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:43.931 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:43 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:43.448+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:43.946 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:43 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:43.716+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:44.036 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.4.service' 2024-03-18T01:02:44.077 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:02:44.077 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-18T01:02:44.078 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-18T01:02:44.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.5 2024-03-18T01:02:44.390 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:44 smithi107 systemd[1]: Stopping Ceph osd.5 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:44.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:44 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:44.227+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:44.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:44 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:44.227+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:44.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:44 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:44.390+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:44.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:44 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:44.477+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:44.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:44 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:44.477+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:44.681 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:44 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:44.408+0000 7f2c09a8c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:02:44.681 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:44 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:44.408+0000 7f2c09a8c700 -1 osd.5 91 *** Got signal Terminated *** 2024-03-18T01:02:44.681 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:44 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:44.408+0000 7f2c09a8c700 -1 osd.5 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:02:44.681 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:44 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:44.628+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:44.681 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:44 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:44.628+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:44.874 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:44 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:44.559+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:44.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:44 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:44.559+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:44.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:44 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:44.428+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:44.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:44 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:44.428+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:44.945 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:44 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:44.695+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:45.642 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:45 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:45.367+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:45.642 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:45 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:45.456+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:45.642 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:45 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:45.456+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:45.673 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:45 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:45.261+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:45.673 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:45 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:45.261+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:45.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:45 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:45.456+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:45.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:45 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:45.456+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:45.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:45 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:45.556+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:45.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:45 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:45.556+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:45.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:45 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:45.642+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:45.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:45 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:45.642+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:45.945 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:45 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:45.672+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:46.621 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:46 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:46.327+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:46.621 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:46 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:46.460+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:46.621 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:46 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:46.460+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:46.632 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:46 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:46.277+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:46.632 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:46 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:46.277+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:46.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:46 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:46.580+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:46.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:46 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:46.580+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:46.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:46 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:46.448+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:46.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:46 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:46.448+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:46.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:46 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:46.448+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:46.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:46 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:46.621+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:46.930 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:46 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:46.621+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:46.945 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:46 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:46.631+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:47.625 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:47 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:47.297+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:47.625 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:47 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:47.297+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:47.625 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:47 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:47.297+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:47.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:47 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:47.362+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:47.680 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:47 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:47.593+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:47.680 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:47 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:47.593+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:47.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:47 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:47.458+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:47.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:47 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:47.458+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:47.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:47 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:47.555+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:47.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:47 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:47.555+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:47.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:47 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:47.416+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:47.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:47 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:47.416+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:47.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:47 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:47.416+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:47.945 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:47 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:47.625+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:48.641 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:48 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:48.252+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:48.642 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:48 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:48.252+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:48.642 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:48 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:48.252+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:48.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:48 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:48.334+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:48.680 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:48 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:48.566+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.351284+0000 front 2024-03-18T01:02:07.351222+0000 (oldest deadline 2024-03-18T01:02:30.251335+0000) 2024-03-18T01:02:48.680 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:48 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5[47989]: 2024-03-18T01:02:48.566+0000 7f2c06080700 -1 osd.5 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:13.152484+0000 front 2024-03-18T01:02:13.152515+0000 (oldest deadline 2024-03-18T01:02:37.852420+0000) 2024-03-18T01:02:48.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:48 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:48.465+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:48.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:48 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:48.465+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:48.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:48 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:48.465+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:48.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:48 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:48.437+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:48.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:48 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:48.437+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:48.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:48 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:48.437+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:48.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:48 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:48.536+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:48.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:48 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:48.536+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:48.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:48 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:48.536+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:48.945 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:48 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:48.641+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:49.638 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:49 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:49.263+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:49.638 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:49 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:49.263+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:49.638 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:49 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:49.263+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:49.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:49 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:49.304+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:49.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:49 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:49.506+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:49.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:49 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:49.506+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:49.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:49 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:49.506+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:49.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:49 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:49.484+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:49.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:49 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:49.484+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:49.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:49 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:49.484+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:49.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:49 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:49.499+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:49.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:49 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:49.499+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:49.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:49 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:49.499+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:49.945 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:49 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:49.638+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:49.946 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:49 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:49.638+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:50.100 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:49 smithi107 podman[72418]: 2024-03-18 01:02:49.813358447 +0000 UTC m=+5.540439035 container died fd7d36ad148c83abf2de056c6cbde1b4411ba1fe1c71187a9d37f11b99efe63e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-18T01:02:50.352 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:50 smithi107 podman[72418]: 2024-03-18 01:02:50.199734638 +0000 UTC m=+5.926815234 container cleanup fd7d36ad148c83abf2de056c6cbde1b4411ba1fe1c71187a9d37f11b99efe63e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:02:50.352 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:50 smithi107 bash[72418]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5 2024-03-18T01:02:50.637 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:50 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:50.254+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:50.637 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:50 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:50.254+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:50.637 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:50 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:50.254+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:50.680 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:50 smithi107 podman[72435]: 2024-03-18 01:02:50.493197735 +0000 UTC m=+0.676636419 container remove fd7d36ad148c83abf2de056c6cbde1b4411ba1fe1c71187a9d37f11b99efe63e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-18T01:02:50.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:50 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:50.353+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:50.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:50 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:50.353+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:50.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:50 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:50.512+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:50.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:50 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:50.512+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:50.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:50 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:50.512+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:50.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:50 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:50.481+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:50.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:50 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:50.481+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:50.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:50 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:50.481+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:50.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:50 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:50.492+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:50.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:50 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:50.492+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:50.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:50 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:50.492+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:50.945 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:50 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:50.637+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:50.946 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:50 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:50.637+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:51.079 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:50 smithi107 podman[72523]: 2024-03-18 01:02:50.718679967 +0000 UTC m=+0.017823363 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:02:51.079 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:50 smithi107 podman[72523]: 2024-03-18 01:02:50.835628451 +0000 UTC m=+0.134771856 container create 0851440b276cfc6bd955289e3c29862501a6eb6e85281e97d6bc3608e0ef1a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:02:51.080 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:51 smithi107 podman[72523]: 2024-03-18 01:02:51.079760986 +0000 UTC m=+0.378904402 container init 0851440b276cfc6bd955289e3c29862501a6eb6e85281e97d6bc3608e0ef1a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:02:51.339 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:51 smithi107 podman[72523]: 2024-03-18 01:02:51.082855288 +0000 UTC m=+0.381998690 container start 0851440b276cfc6bd955289e3c29862501a6eb6e85281e97d6bc3608e0ef1a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-18T01:02:51.339 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:51 smithi107 podman[72523]: 2024-03-18 01:02:51.171625357 +0000 UTC m=+0.470768768 container attach 0851440b276cfc6bd955289e3c29862501a6eb6e85281e97d6bc3608e0ef1a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:02:51.595 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:51 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:51.302+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:51.595 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:51 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:51.302+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:51.595 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:51 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:51.302+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:51.632 INFO:journalctl@ceph.osd.5.smithi107.stdout:Mar 18 01:02:51 smithi107 podman[72523]: 2024-03-18 01:02:51.369908963 +0000 UTC m=+0.669052383 container died 0851440b276cfc6bd955289e3c29862501a6eb6e85281e97d6bc3608e0ef1a2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-18T01:02:51.633 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:51 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:51.339+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:51.633 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:51 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:51.339+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:51.633 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:51 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:51.486+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:51.633 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:51 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:51.486+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:51.634 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:51 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:51.486+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:51.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:51 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:51.475+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:51.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:51 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:51.475+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:51.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:51 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:51.475+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:51.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:51 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:51.459+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:51.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:51 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:51.459+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:51.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:51 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:51.459+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:51.946 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:51 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:51.595+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:51.946 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:51 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:51.595+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:51.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.5.service' 2024-03-18T01:02:52.002 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:02:52.003 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-18T01:02:52.003 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-18T01:02:52.003 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.6 2024-03-18T01:02:52.332 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:52 smithi049 systemd[1]: Stopping Ceph osd.6 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:02:52.333 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:52.332+0000 7fa0cb6a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:02:52.568 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:52.286+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:52.568 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:52.286+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:52.568 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:52.286+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:52.625 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:52.332+0000 7fa0cb6a3700 -1 osd.6 91 *** Got signal Terminated *** 2024-03-18T01:02:52.625 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:52.332+0000 7fa0cb6a3700 -1 osd.6 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:02:52.625 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:52.465+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:52.625 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:52.465+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:52.625 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:52.465+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:52.625 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:52.454+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:52.625 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:52.454+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:52.625 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:52 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:52.454+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:52.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:52.296+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:52.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:52.296+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:52.682 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:52.520+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:52.682 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:52.520+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:52.682 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:52 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:52.520+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:52.946 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:52.568+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:52.946 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:52 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:52.568+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:53.511 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:53 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:53.250+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:53.511 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:53 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:53.250+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:53.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:53 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:53.554+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:53.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:53 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:53.554+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:53.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:53 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:53.314+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:53.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:53 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:53.314+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:53.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:53 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:53.314+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:53.874 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:53 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:53.431+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:53.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:53 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:53.431+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:53.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:53 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:53.431+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:53.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:53 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:53.445+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:53.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:53 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:53.445+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:53.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:53 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:53.445+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:53.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:53 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:53.511+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:53.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:53 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:53.511+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:53.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:53 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:53.511+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:54.519 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:54 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:54.205+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:54.520 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:54 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:54.205+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:54.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:54 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:54.543+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:54.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:54 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:54.543+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:54.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:54 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:54.364+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:54.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:54 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:54.364+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:54.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:54 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:54.364+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:54.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:54.418+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:54.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:54.418+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:54.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:54.418+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:54.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:54.404+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:54.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:54.404+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:54.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:54 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:54.404+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:54.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:54 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:54.520+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:54.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:54 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:54.520+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:54.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:54 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:54.520+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:55.430 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:55 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:55.165+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:55.430 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:55 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:55.165+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:55.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:55 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:55.539+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:55.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:55 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:55.539+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:55.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:55 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:55.375+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:55.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:55 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:55.375+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:55.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:55 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:55.375+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:55.874 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:55.420+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:55.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:55.420+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:55.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:55.420+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:55.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:55.455+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:55.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:55.455+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:55.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:55 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:55.455+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:55.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:55 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:55.475+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:55.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:55 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:55.475+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:55.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:55 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:55.475+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:56.455 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:56 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:56.197+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:56.455 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:56 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:56.197+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:56.455 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:56 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:56.432+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:56.455 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:56 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:56.432+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:56.456 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:56 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:56.432+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:56.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:56 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:56.564+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:56.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:56 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:56.564+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:56.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:56 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:56.564+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:30.293603+0000 front 2024-03-18T01:02:30.293700+0000 (oldest deadline 2024-03-18T01:02:55.593406+0000) 2024-03-18T01:02:56.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:56 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:56.393+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:56.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:56 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:56.393+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:56.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:56 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:56.393+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:56.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:56 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:56.393+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:02:56.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:56.381+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:07.109076+0000 front 2024-03-18T01:02:07.109105+0000 (oldest deadline 2024-03-18T01:02:33.009240+0000) 2024-03-18T01:02:56.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:56.381+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.510864+0000 front 2024-03-18T01:02:16.510580+0000 (oldest deadline 2024-03-18T01:02:38.210688+0000) 2024-03-18T01:02:56.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:56.381+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.412580+0000 front 2024-03-18T01:02:23.412866+0000 (oldest deadline 2024-03-18T01:02:46.312647+0000) 2024-03-18T01:02:56.875 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6[60279]: 2024-03-18T01:02:56.381+0000 7fa0c7c97700 -1 osd.6 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.613545+0000 front 2024-03-18T01:02:31.613486+0000 (oldest deadline 2024-03-18T01:02:55.614012+0000) 2024-03-18T01:02:56.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:56.436+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:56.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:56.436+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:56.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:56 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:56.436+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:57.426 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:57 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:57.165+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:57.426 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:57 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:57.165+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:57.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:57 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:57.426+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:57.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:57 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:57.426+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:57.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:57 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:57.426+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:57.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:57 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:57.537+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:57.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:57 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:57.537+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:57.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:57 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:57.537+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:30.293603+0000 front 2024-03-18T01:02:30.293700+0000 (oldest deadline 2024-03-18T01:02:55.593406+0000) 2024-03-18T01:02:57.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:57 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:57.438+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:57.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:57 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:57.438+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:57.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:57 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:57.438+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:57.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:57 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:57.438+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:02:57.852 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:57 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:57.450+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:57.852 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:57 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:57.450+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:57.852 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:57 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:57.450+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:57.852 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:57 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:57.450+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:02:58.124 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:57 smithi049 podman[100185]: 2024-03-18 01:02:57.853197266 +0000 UTC m=+5.637940550 container died 25ca0569e2289646a533b9231a225f601e0cdc5d614726d3c18f11e9bf942ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T01:02:58.374 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:58 smithi049 podman[100185]: 2024-03-18 01:02:58.207780711 +0000 UTC m=+5.992523995 container cleanup 25ca0569e2289646a533b9231a225f601e0cdc5d614726d3c18f11e9bf942ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-18T01:02:58.375 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:58 smithi049 bash[100185]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6 2024-03-18T01:02:58.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:58 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:58.187+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:58.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:58 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:58.187+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:58.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:58 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:58.187+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:02:58.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:58 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:58.420+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:58.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:58 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:58.420+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:58.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:58 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:58.420+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:58.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:58 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:58.579+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:58.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:58 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:58.579+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:58.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:58 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:58.579+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:30.293603+0000 front 2024-03-18T01:02:30.293700+0000 (oldest deadline 2024-03-18T01:02:55.593406+0000) 2024-03-18T01:02:58.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:58 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:58.394+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:58.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:58 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:58.394+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:58.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:58 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:58.394+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:58.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:58 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:58.394+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:02:58.850 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:58 smithi049 podman[100202]: 2024-03-18 01:02:58.542822353 +0000 UTC m=+0.684314161 container remove 25ca0569e2289646a533b9231a225f601e0cdc5d614726d3c18f11e9bf942ced (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-18T01:02:58.850 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:58 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:58.440+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:58.850 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:58 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:58.440+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:58.850 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:58 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:58.440+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:58.850 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:58 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:58.440+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:02:59.112 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:58 smithi049 podman[100289]: 2024-03-18 01:02:58.754887443 +0000 UTC m=+0.019275187 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:02:59.112 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:58 smithi049 podman[100289]: 2024-03-18 01:02:58.868680574 +0000 UTC m=+0.133068324 container create 3679d581aca8b654ba0a5c51792bedaf9670c788cfe3b889e801044fad3a14af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:02:59.112 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:59 smithi049 podman[100289]: 2024-03-18 01:02:59.113094539 +0000 UTC m=+0.377482296 container init 3679d581aca8b654ba0a5c51792bedaf9670c788cfe3b889e801044fad3a14af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-18T01:02:59.375 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:59 smithi049 podman[100289]: 2024-03-18 01:02:59.116464752 +0000 UTC m=+0.380852493 container start 3679d581aca8b654ba0a5c51792bedaf9670c788cfe3b889e801044fad3a14af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:02:59.375 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:59 smithi049 podman[100289]: 2024-03-18 01:02:59.196378059 +0000 UTC m=+0.460765817 container attach 3679d581aca8b654ba0a5c51792bedaf9670c788cfe3b889e801044fad3a14af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True) 2024-03-18T01:02:59.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:59 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:59.185+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:02:59.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:59 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:59.185+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:02:59.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:02:59 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:02:59.185+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:02:59.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:59 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:59.435+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:02:59.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:59 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:59.435+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:02:59.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:02:59 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:02:59.435+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:02:59.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:59 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:59.541+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:02:59.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:59 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:59.541+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:02:59.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:02:59 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:02:59.541+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:30.293603+0000 front 2024-03-18T01:02:30.293700+0000 (oldest deadline 2024-03-18T01:02:55.593406+0000) 2024-03-18T01:02:59.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:59 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:59.381+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:02:59.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:59 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:59.381+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:02:59.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:59 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:59.381+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:02:59.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:02:59 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:02:59.381+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:02:59.731 INFO:journalctl@ceph.osd.6.smithi049.stdout:Mar 18 01:02:59 smithi049 podman[100289]: 2024-03-18 01:02:59.425388031 +0000 UTC m=+0.689775773 container died 3679d581aca8b654ba0a5c51792bedaf9670c788cfe3b889e801044fad3a14af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:02:59.732 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:59 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:59.486+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:02:59.732 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:59 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:59.486+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:02:59.732 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:59 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:59.486+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:02:59.732 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:02:59 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:02:59.486+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:00.042 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.6.service' 2024-03-18T01:03:00.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:00.090 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-18T01:03:00.090 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-18T01:03:00.090 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.7 2024-03-18T01:03:00.402 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:00 smithi106 systemd[1]: Stopping Ceph osd.7 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:00.403 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:00.342+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:00.403 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:00.342+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:00.403 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:00.342+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:00.403 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:00.342+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:00.666 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:00.402+0000 7f3b70072700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:00.666 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:00.402+0000 7f3b70072700 -1 osd.7 91 *** Got signal Terminated *** 2024-03-18T01:03:00.666 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:00.402+0000 7f3b70072700 -1 osd.7 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:03:00.666 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:00.500+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:03:00.666 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:00.500+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:03:00.666 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:00 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:00.500+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:30.293603+0000 front 2024-03-18T01:02:30.293700+0000 (oldest deadline 2024-03-18T01:02:55.593406+0000) 2024-03-18T01:03:00.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:00 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:00.198+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:00.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:00 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:00.198+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:00.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:00 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:00.198+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:00.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:00 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:00.407+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:00.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:00 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:00.407+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:00.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:00 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:00.407+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:00.874 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:00 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:00.466+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:00.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:00 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:00.466+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:00.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:00 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:00.466+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:00.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:00 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:00.466+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:01.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:01 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:01.206+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:01.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:01 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:01.206+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:01.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:01 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:01.206+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:01.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:01 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:01.390+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:01.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:01 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:01.390+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:01.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:01 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:01.390+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:01.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:01.517+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:03:01.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:01.517+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:03:01.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:01.517+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:30.293603+0000 front 2024-03-18T01:02:30.293700+0000 (oldest deadline 2024-03-18T01:02:55.593406+0000) 2024-03-18T01:03:01.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:01.352+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:01.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:01.352+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:01.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:01.352+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:01.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:01.352+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:01.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:01 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:01.515+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:01.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:01 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:01.515+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:01.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:01 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:01.515+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:01.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:01 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:01.515+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:02.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:02 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:02.371+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:02.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:02 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:02.371+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:02.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:02 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:02.371+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:02.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:02 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:02.233+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:02.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:02 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:02.233+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:02.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:02 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:02.233+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:02.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:02 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:02.233+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:02.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:02 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:02.532+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:03:02.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:02 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:02.532+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:03:02.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:02 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:02.532+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:30.293603+0000 front 2024-03-18T01:02:30.293700+0000 (oldest deadline 2024-03-18T01:02:55.593406+0000) 2024-03-18T01:03:02.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:02 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:02.532+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:35.593982+0000 front 2024-03-18T01:02:35.593882+0000 (oldest deadline 2024-03-18T01:03:01.994548+0000) 2024-03-18T01:03:02.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:02 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:02.322+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:02.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:02 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:02.322+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:02.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:02 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:02.322+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:02.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:02 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:02.322+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:02.874 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:02 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:02.543+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:02.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:02 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:02.543+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:02.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:02 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:02.543+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:02.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:02 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:02.543+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:03.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:03 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:03.218+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:03.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:03 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:03.218+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:03.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:03 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:03.218+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:03.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:03 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:03.218+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:03.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:03 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:03.379+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:03.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:03 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:03.379+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:03.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:03 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:03.379+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:03.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:03 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:03.379+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:03.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:03 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:03.487+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:03:03.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:03 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:03.487+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:03:03.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:03 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:03.487+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:30.293603+0000 front 2024-03-18T01:02:30.293700+0000 (oldest deadline 2024-03-18T01:02:55.593406+0000) 2024-03-18T01:03:03.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:03 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:03.487+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:35.593982+0000 front 2024-03-18T01:02:35.593882+0000 (oldest deadline 2024-03-18T01:03:01.994548+0000) 2024-03-18T01:03:03.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:03 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:03.276+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:03.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:03 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:03.276+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:03.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:03 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:03.276+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:03.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:03 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:03.276+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:03.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:03 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:03.276+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:03.874 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:03 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:03.578+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:03.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:03 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:03.578+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:03.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:03 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:03.578+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:03.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:03 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:03.578+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:04.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:04 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:04.234+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:04.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:04 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:04.234+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:04.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:04 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:04.234+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:04.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:04 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:04.234+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:04.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:04 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:04.337+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:04.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:04 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:04.337+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:04.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:04 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:04.337+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:04.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:04 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:04.337+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:04.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:04 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:04.450+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.189513+0000 front 2024-03-18T01:02:14.189471+0000 (oldest deadline 2024-03-18T01:02:38.889648+0000) 2024-03-18T01:03:04.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:04 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:04.450+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:22.191919+0000 front 2024-03-18T01:02:22.191898+0000 (oldest deadline 2024-03-18T01:02:49.192383+0000) 2024-03-18T01:03:04.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:04 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:04.450+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:30.293603+0000 front 2024-03-18T01:02:30.293700+0000 (oldest deadline 2024-03-18T01:02:55.593406+0000) 2024-03-18T01:03:04.696 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:04 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7[54062]: 2024-03-18T01:03:04.450+0000 7f3b6865e700 -1 osd.7 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:35.593982+0000 front 2024-03-18T01:02:35.593882+0000 (oldest deadline 2024-03-18T01:03:01.994548+0000) 2024-03-18T01:03:04.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:04 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:04.230+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:04.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:04 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:04.230+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:04.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:04 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:04.230+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:04.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:04 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:04.230+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:04.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:04 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:04.230+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:04.874 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:04 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:04.552+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:04.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:04 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:04.552+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:04.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:04 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:04.552+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:04.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:04 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:04.552+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:05.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:05 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:05.218+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:05.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:05 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:05.218+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:05.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:05 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:05.218+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:05.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:05 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:05.218+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:05.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:05 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:05.360+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:05.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:05 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:05.360+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:05.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:05 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:05.360+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:05.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:05 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:05.360+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:05.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:05 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:05.214+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:05.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:05 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:05.214+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:05.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:05 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:05.214+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:05.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:05 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:05.214+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:05.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:05 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:05.214+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:05.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:05 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:05.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:05.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:05 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:05.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:05.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:05 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:05.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:05.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:05 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:05.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:06.046 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:05 smithi106 podman[72411]: 2024-03-18 01:03:05.775530319 +0000 UTC m=+5.499648049 container died 42dad86d6f94e249637fa20ecc24ff77a1bd0bcaf738e21e6eb577b2fcde7ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:03:06.412 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:06 smithi106 podman[72411]: 2024-03-18 01:03:06.153740333 +0000 UTC m=+5.877858115 container cleanup 42dad86d6f94e249637fa20ecc24ff77a1bd0bcaf738e21e6eb577b2fcde7ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-18T01:03:06.412 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:06 smithi106 bash[72411]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7 2024-03-18T01:03:06.413 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:06 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:06.168+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:06.413 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:06 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:06.168+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:06.413 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:06 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:06.168+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:06.413 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:06 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:06.168+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:06.413 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:06 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:06.168+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:06.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:06 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:06.224+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:06.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:06 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:06.224+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:06.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:06 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:06.224+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:06.681 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:06 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:06.224+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:06.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:06 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:06.360+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:06.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:06 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:06.360+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:06.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:06 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:06.360+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:06.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:06 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:06.360+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:06.695 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:06 smithi106 podman[72440]: 2024-03-18 01:03:06.413020202 +0000 UTC m=+0.640889314 container remove 42dad86d6f94e249637fa20ecc24ff77a1bd0bcaf738e21e6eb577b2fcde7ad6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:03:06.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:06 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:06.588+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:06.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:06 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:06.588+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:06.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:06 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:06.588+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:06.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:06 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:06.588+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:06.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:06 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:06.588+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:07.024 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:06 smithi106 podman[72530]: 2024-03-18 01:03:06.65852943 +0000 UTC m=+0.017237758 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:03:07.025 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:06 smithi106 podman[72530]: 2024-03-18 01:03:06.780638704 +0000 UTC m=+0.139347033 container create af378e0cc7974755ea9ebac754731ce661d3bf3382e2fd6556f958e2e272f92e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-18T01:03:07.025 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:07 smithi106 podman[72530]: 2024-03-18 01:03:07.025205487 +0000 UTC m=+0.383913828 container init af378e0cc7974755ea9ebac754731ce661d3bf3382e2fd6556f958e2e272f92e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-03-18T01:03:07.322 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:07 smithi106 podman[72530]: 2024-03-18 01:03:07.028595897 +0000 UTC m=+0.387304220 container start af378e0cc7974755ea9ebac754731ce661d3bf3382e2fd6556f958e2e272f92e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-18T01:03:07.322 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:07 smithi106 podman[72530]: 2024-03-18 01:03:07.116399452 +0000 UTC m=+0.475107803 container attach af378e0cc7974755ea9ebac754731ce661d3bf3382e2fd6556f958e2e272f92e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:03:07.322 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:07 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:07.204+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:07.323 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:07 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:07.204+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:07.323 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:07 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:07.204+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:07.323 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:07 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:07.204+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:07.323 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:07 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:07.204+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:07.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:07 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:07.184+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:07.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:07 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:07.184+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:07.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:07 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:07.184+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:07.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:07 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:07.184+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:07.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:07 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:07.345+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:07.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:07 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:07.345+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:07.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:07 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:07.345+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:07.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:07 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:07.345+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:07.660 INFO:journalctl@ceph.osd.7.smithi106.stdout:Mar 18 01:03:07 smithi106 podman[72530]: 2024-03-18 01:03:07.32329854 +0000 UTC m=+0.682006877 container died af378e0cc7974755ea9ebac754731ce661d3bf3382e2fd6556f958e2e272f92e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T01:03:07.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:07 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:07.578+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:07.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:07 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:07.578+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:07.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:07 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:07.578+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:07.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:07 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:07.578+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:07.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:07 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:07.578+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:08.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.7.service' 2024-03-18T01:03:08.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:08.047 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-18T01:03:08.047 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-18T01:03:08.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.8 2024-03-18T01:03:08.360 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:08.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:08.360 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:08.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:08.360 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:08.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:08.360 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:08.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:08.360 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:08 smithi107 systemd[1]: Stopping Ceph osd.8 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:08.360 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:08.167+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:08.360 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:08.167+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:08.360 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:08.167+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:08.361 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:08.167+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:08.361 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:08.359+0000 7fcc18bfc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:08.361 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:08.359+0000 7fcc18bfc700 -1 osd.8 91 *** Got signal Terminated *** 2024-03-18T01:03:08.361 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:08 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:08.359+0000 7fcc18bfc700 -1 osd.8 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:03:08.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:08 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:08.176+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:08.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:08 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:08.176+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:08.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:08 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:08.176+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:08.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:08 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:08.176+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:08.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:08 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:08.176+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:08.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:08 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:08.586+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:08.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:08 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:08.586+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:08.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:08 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:08.586+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:08.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:08 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:08.586+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:08.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:08 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:08.586+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:09.430 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:09 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:09.128+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:09.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:09 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:09.128+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:09.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:09 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:09.128+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:09.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:09 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:09.128+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:09.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:09 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:09.291+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:09.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:09 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:09.291+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:09.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:09 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:09.291+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:09.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:09 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:09.291+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:09.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:09 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:09.203+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:09.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:09 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:09.203+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:09.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:09 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:09.204+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:09.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:09 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:09.204+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:09.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:09 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:09.204+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:09.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:09 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:09.589+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:09.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:09 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:09.589+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:09.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:09 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:09.589+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:09.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:09 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:09.589+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:09.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:09 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:09.589+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:10.430 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:10 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:10.144+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:10.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:10 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:10.144+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:10.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:10 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:10.144+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:10.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:10 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:10.144+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:10.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:10 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:10.243+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:10.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:10 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:10.243+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:10.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:10 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:10.243+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:10.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:10 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:10.243+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:10.445 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:10 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:10.182+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:10.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:10 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:10.182+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:10.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:10 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:10.182+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:10.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:10 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:10.182+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:10.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:10 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:10.182+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:10.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:10.593+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:10.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:10.593+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:10.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:10.593+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:10.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:10.593+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:10.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:10 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:10.593+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:11.430 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:11.142+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:11.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:11.142+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:11.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:11.142+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:11.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:11.142+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:11.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:11.142+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:46.152117+0000 front 2024-03-18T01:02:46.152142+0000 (oldest deadline 2024-03-18T01:03:10.852227+0000) 2024-03-18T01:03:11.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:11.234+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:11.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:11.234+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:11.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:11.234+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:11.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:11.234+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:11.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:11 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:11.234+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:11.445 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:11 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:11.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:11.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:11 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:11.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:11.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:11 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:11.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:11.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:11 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:11.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:11.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:11 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:11.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:11.876 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:11.618+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:11.876 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:11.618+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:11.876 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:11.618+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:11.876 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:11.618+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:11.876 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:11 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:11.618+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:12.430 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:12.158+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:12.430 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:12.158+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:12.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:12.158+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:12.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:12.158+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:12.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:12.158+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:46.152117+0000 front 2024-03-18T01:02:46.152142+0000 (oldest deadline 2024-03-18T01:03:10.852227+0000) 2024-03-18T01:03:12.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:12.268+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:12.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:12.268+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:12.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:12.268+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:12.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:12.268+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:12.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:12 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:12.268+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:12.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:12 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:12.148+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:12.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:12 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:12.148+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:12.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:12 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:12.148+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:12.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:12 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:12.148+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:12.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:12 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:12.148+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:12.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:12 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:12.576+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:12.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:12 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:12.576+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:12.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:12 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:12.576+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:12.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:12 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:12.576+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:12.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:12 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:12.576+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:13.430 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:13.118+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.547336+0000 front 2024-03-18T01:02:14.547233+0000 (oldest deadline 2024-03-18T01:02:37.447325+0000) 2024-03-18T01:03:13.430 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:13.118+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:23.849082+0000 front 2024-03-18T01:02:23.849056+0000 (oldest deadline 2024-03-18T01:02:49.749252+0000) 2024-03-18T01:03:13.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:13.118+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:32.650533+0000 front 2024-03-18T01:02:32.650545+0000 (oldest deadline 2024-03-18T01:02:57.350536+0000) 2024-03-18T01:03:13.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:13.118+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.351159+0000 front 2024-03-18T01:02:37.351142+0000 (oldest deadline 2024-03-18T01:03:02.051139+0000) 2024-03-18T01:03:13.431 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8[54976]: 2024-03-18T01:03:13.118+0000 7fcc111e8700 -1 osd.8 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:46.152117+0000 front 2024-03-18T01:02:46.152142+0000 (oldest deadline 2024-03-18T01:03:10.852227+0000) 2024-03-18T01:03:13.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:13.307+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:13.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:13.307+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:13.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:13.307+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:13.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:13.307+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:13.431 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:13.307+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:13.445 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:13 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:13.099+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:13.445 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:13 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:13.099+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:13.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:13 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:13.099+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:13.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:13 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:13.099+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:13.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:13 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:13.099+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:13.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:13.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:13.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:13.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:13.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:13.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:13.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:13.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:13.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:13.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:13.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:13 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:13.539+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.593480+0000 front 2024-03-18T01:02:47.593609+0000 (oldest deadline 2024-03-18T01:03:13.493385+0000) 2024-03-18T01:03:14.085 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:13 smithi107 podman[72640]: 2024-03-18 01:03:13.803632602 +0000 UTC m=+5.556868371 container died e2ebf7f994ab747f1e97403042e30751affb66b5eb180b21e4e919e99b5e1370 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T01:03:14.346 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:14 smithi107 podman[72640]: 2024-03-18 01:03:14.176459532 +0000 UTC m=+5.929695301 container cleanup e2ebf7f994ab747f1e97403042e30751affb66b5eb180b21e4e919e99b5e1370 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T01:03:14.346 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:14 smithi107 bash[72640]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8 2024-03-18T01:03:14.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:14 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:14.122+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:14.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:14 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:14.122+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:14.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:14 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:14.122+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:14.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:14 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:14.122+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:14.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:14 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:14.122+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:14.680 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:14 smithi107 podman[72658]: 2024-03-18 01:03:14.477786966 +0000 UTC m=+0.671934311 container remove e2ebf7f994ab747f1e97403042e30751affb66b5eb180b21e4e919e99b5e1370 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-18T01:03:14.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:14.346+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:14.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:14.346+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:14.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:14.346+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:14.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:14.346+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:14.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:14 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:14.346+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:14.874 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:14.498+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:14.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:14.498+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:14.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:14.498+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:14.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:14.498+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:14.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:14.498+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:14.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:14 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:14.498+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.593480+0000 front 2024-03-18T01:02:47.593609+0000 (oldest deadline 2024-03-18T01:03:13.493385+0000) 2024-03-18T01:03:15.038 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:14 smithi107 podman[72750]: 2024-03-18 01:03:14.761975674 +0000 UTC m=+0.106952332 container create 2ee1b77c11b4e3bc8583436cca76043c0127af0becbdf6e674517f83108a2c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:03:15.039 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:14 smithi107 podman[72750]: 2024-03-18 01:03:14.672945563 +0000 UTC m=+0.017922199 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:03:15.039 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:15 smithi107 podman[72750]: 2024-03-18 01:03:15.039460051 +0000 UTC m=+0.384436682 container init 2ee1b77c11b4e3bc8583436cca76043c0127af0becbdf6e674517f83108a2c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-18T01:03:15.298 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:15 smithi107 podman[72750]: 2024-03-18 01:03:15.042729843 +0000 UTC m=+0.387706476 container start 2ee1b77c11b4e3bc8583436cca76043c0127af0becbdf6e674517f83108a2c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T01:03:15.298 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:15 smithi107 podman[72750]: 2024-03-18 01:03:15.131150519 +0000 UTC m=+0.476127153 container attach 2ee1b77c11b4e3bc8583436cca76043c0127af0becbdf6e674517f83108a2c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-18T01:03:15.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:15.169+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:15.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:15.169+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:15.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:15.169+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:15.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:15.169+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:15.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:15.169+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:15.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:15 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:15.169+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:15.608 INFO:journalctl@ceph.osd.8.smithi107.stdout:Mar 18 01:03:15 smithi107 podman[72750]: 2024-03-18 01:03:15.325479765 +0000 UTC m=+0.670456398 container died 2ee1b77c11b4e3bc8583436cca76043c0127af0becbdf6e674517f83108a2c48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T01:03:15.608 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:15 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:15.298+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:15.608 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:15 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:15.298+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:15.608 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:15 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:15.298+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:15.608 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:15 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:15.298+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:15.608 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:15 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:15.298+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:15.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:15 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:15.540+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:15.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:15 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:15.540+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:15.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:15 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:15.540+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:15.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:15 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:15.540+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:15.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:15 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:15.540+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:15.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:15 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:15.540+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.593480+0000 front 2024-03-18T01:02:47.593609+0000 (oldest deadline 2024-03-18T01:03:13.493385+0000) 2024-03-18T01:03:15.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.8.service' 2024-03-18T01:03:15.976 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:15.976 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-18T01:03:15.977 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-18T01:03:15.977 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.9 2024-03-18T01:03:16.294 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 systemd[1]: Stopping Ceph osd.9 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:16.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:16.184+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:16.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:16.184+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:16.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:16.184+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:16.447 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:16.184+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:16.447 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:16.184+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:16.447 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:16 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:16.184+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:16.564 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:16.293+0000 7f9af1aee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:16.564 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:16.293+0000 7f9af1aee700 -1 osd.9 91 *** Got signal Terminated *** 2024-03-18T01:03:16.564 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:16.293+0000 7f9af1aee700 -1 osd.9 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:03:16.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:16 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:16.259+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:16.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:16 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:16.259+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:16.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:16 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:16.259+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:16.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:16 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:16.259+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:16.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:16 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:16.259+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:16.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:16.564+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:16.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:16.564+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:16.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:16.564+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:16.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:16.564+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:16.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:16.564+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:16.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:16 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:16.564+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.593480+0000 front 2024-03-18T01:02:47.593609+0000 (oldest deadline 2024-03-18T01:03:13.493385+0000) 2024-03-18T01:03:17.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:17 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:17.161+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:17.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:17 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:17.161+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:17.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:17 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:17.161+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:17.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:17 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:17.161+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:17.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:17 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:17.161+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:17.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:17 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:17.161+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:17.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:17 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:17.282+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:17.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:17 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:17.282+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:17.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:17 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:17.282+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:17.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:17 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:17.282+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:17.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:17 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:17.282+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:17.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:17 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:17.611+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:17.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:17 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:17.611+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:17.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:17 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:17.611+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:17.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:17 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:17.611+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:17.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:17 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:17.611+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:17.875 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:17 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:17.611+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.593480+0000 front 2024-03-18T01:02:47.593609+0000 (oldest deadline 2024-03-18T01:03:13.493385+0000) 2024-03-18T01:03:18.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:18.113+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:18.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:18.113+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:18.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:18.113+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:18.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:18.113+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:18.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:18.113+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:18.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:18 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:18.113+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:18.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:18 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:18.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:18.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:18 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:18.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:18.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:18 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:18.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:18.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:18 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:18.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:18.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:18 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:18.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:19.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:18 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:18.660+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:19.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:18 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:18.660+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:19.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:18 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:18.660+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:19.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:18 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:18.660+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:19.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:18 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:18.660+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:19.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:18 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:18.660+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.593480+0000 front 2024-03-18T01:02:47.593609+0000 (oldest deadline 2024-03-18T01:03:13.493385+0000) 2024-03-18T01:03:19.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:18 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:18.660+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:56.294826+0000 front 2024-03-18T01:02:56.294850+0000 (oldest deadline 2024-03-18T01:03:18.594875+0000) 2024-03-18T01:03:19.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:19.139+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:19.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:19.139+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:19.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:19.139+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:19.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:19.139+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:19.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:19.139+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:19.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:19 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:19.139+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:19.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:19 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:19.229+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:19.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:19 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:19.229+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:19.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:19 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:19.229+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:19.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:19 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:19.229+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:19.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:19 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:19.229+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:19.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:19 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:19.229+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:20.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:19 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:19.667+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:20.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:19 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:19.667+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:20.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:19 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:19.667+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:20.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:19 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:19.667+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:20.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:19 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:19.667+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:20.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:19 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:19.667+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.593480+0000 front 2024-03-18T01:02:47.593609+0000 (oldest deadline 2024-03-18T01:03:13.493385+0000) 2024-03-18T01:03:20.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:19 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:19.667+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:56.294826+0000 front 2024-03-18T01:02:56.294850+0000 (oldest deadline 2024-03-18T01:03:18.594875+0000) 2024-03-18T01:03:20.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:20 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:20.189+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:20.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:20 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:20.189+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:20.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:20 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:20.189+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:20.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:20 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:20.189+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:20.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:20 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:20.189+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:20.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:20 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:20.189+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:20.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:20 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:20.185+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:20.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:20 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:20.185+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:20.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:20 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:20.185+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:20.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:20 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:20.185+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:20.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:20 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:20.185+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:20.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:20 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:20.185+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:21.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:20 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:20.668+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:03.886080+0000 front 2024-03-18T01:02:03.886132+0000 (oldest deadline 2024-03-18T01:02:32.686895+0000) 2024-03-18T01:03:21.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:20 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:20.668+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.988035+0000 front 2024-03-18T01:02:14.988058+0000 (oldest deadline 2024-03-18T01:02:37.287959+0000) 2024-03-18T01:03:21.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:20 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:20.668+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.789958+0000 front 2024-03-18T01:02:21.790104+0000 (oldest deadline 2024-03-18T01:02:47.690164+0000) 2024-03-18T01:03:21.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:20 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:20.668+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:27.690763+0000 front 2024-03-18T01:02:27.690793+0000 (oldest deadline 2024-03-18T01:02:56.491354+0000) 2024-03-18T01:03:21.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:20 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:20.668+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:36.491766+0000 front 2024-03-18T01:02:36.491690+0000 (oldest deadline 2024-03-18T01:03:05.892488+0000) 2024-03-18T01:03:21.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:20 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:20.668+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.593480+0000 front 2024-03-18T01:02:47.593609+0000 (oldest deadline 2024-03-18T01:03:13.493385+0000) 2024-03-18T01:03:21.125 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:20 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9[66861]: 2024-03-18T01:03:20.668+0000 7f9aee0e2700 -1 osd.9 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:56.294826+0000 front 2024-03-18T01:02:56.294850+0000 (oldest deadline 2024-03-18T01:03:18.594875+0000) 2024-03-18T01:03:21.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:21 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:21.233+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:21.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:21 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:21.233+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:21.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:21 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:21.233+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:21.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:21 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:21.233+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:21.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:21 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:21.233+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:21.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:21 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:21.233+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:21.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:21 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:21.198+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:21.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:21 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:21.198+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:21.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:21 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:21.198+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:21.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:21 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:21.198+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:21.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:21 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:21.198+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:21.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:21 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:21.198+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:22.109 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:21 smithi049 podman[100405]: 2024-03-18 01:03:21.799901164 +0000 UTC m=+5.618126616 container died d8c9d417f6357473c608b141451427bf72d836ec8736f7b2962d0d2914cbc2c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T01:03:22.109 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:22 smithi049 podman[100405]: 2024-03-18 01:03:22.109995195 +0000 UTC m=+5.928220646 container cleanup d8c9d417f6357473c608b141451427bf72d836ec8736f7b2962d0d2914cbc2c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:03:22.375 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:22 smithi049 bash[100405]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9 2024-03-18T01:03:22.375 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:22 smithi049 podman[100422]: 2024-03-18 01:03:22.311295019 +0000 UTC m=+0.505334058 container remove d8c9d417f6357473c608b141451427bf72d836ec8736f7b2962d0d2914cbc2c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T01:03:22.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:22 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:22.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:22.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:22 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:22.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:22.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:22 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:22.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:22.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:22 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:22.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:22.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:22 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:22.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:22.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:22 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:22.151+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:22.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:22 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:22.228+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:22.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:22 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:22.228+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:22.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:22 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:22.228+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:22.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:22 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:22.228+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:22.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:22 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:22.228+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:22.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:22 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:22.229+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:22.920 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:22 smithi049 podman[100511]: 2024-03-18 01:03:22.553918692 +0000 UTC m=+0.031588896 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:03:22.920 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:22 smithi049 podman[100511]: 2024-03-18 01:03:22.662253248 +0000 UTC m=+0.139923491 container create dfdfb176fd3305e42dd3c0396bd7129ee23f7574c8c2d6495992f1bec00082a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T01:03:22.921 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:22 smithi049 podman[100511]: 2024-03-18 01:03:22.921337175 +0000 UTC m=+0.399007375 container init dfdfb176fd3305e42dd3c0396bd7129ee23f7574c8c2d6495992f1bec00082a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T01:03:23.221 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:22 smithi049 podman[100511]: 2024-03-18 01:03:22.925015734 +0000 UTC m=+0.402685940 container start dfdfb176fd3305e42dd3c0396bd7129ee23f7574c8c2d6495992f1bec00082a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:03:23.222 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:23 smithi049 podman[100511]: 2024-03-18 01:03:23.139939813 +0000 UTC m=+0.617610018 container attach dfdfb176fd3305e42dd3c0396bd7129ee23f7574c8c2d6495992f1bec00082a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-18T01:03:23.505 INFO:journalctl@ceph.osd.9.smithi049.stdout:Mar 18 01:03:23 smithi049 podman[100511]: 2024-03-18 01:03:23.222654084 +0000 UTC m=+0.700324278 container died dfdfb176fd3305e42dd3c0396bd7129ee23f7574c8c2d6495992f1bec00082a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:03:23.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:23 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:23.226+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:23.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:23 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:23.226+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:23.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:23 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:23.226+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:23.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:23 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:23.226+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:23.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:23 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:23.226+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:23.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:23 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:23.226+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:23.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:23 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:23.200+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:23.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:23 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:23.200+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:23.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:23 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:23.200+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:23.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:23 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:23.200+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:23.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:23 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:23.200+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:23.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:23 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:23.200+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:23.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:23 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:23.200+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:57.257747+0000 front 2024-03-18T01:02:57.257632+0000 (oldest deadline 2024-03-18T01:03:23.157472+0000) 2024-03-18T01:03:23.787 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.9.service' 2024-03-18T01:03:23.831 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:23.832 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-18T01:03:23.832 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-18T01:03:23.832 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.10 2024-03-18T01:03:24.147 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:23 smithi106 systemd[1]: Stopping Ceph osd.10 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:24.147 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.146+0000 7f3d847c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:24.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.146+0000 7f3d847c0700 -1 osd.10 91 *** Got signal Terminated *** 2024-03-18T01:03:24.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.146+0000 7f3d847c0700 -1 osd.10 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:03:24.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.197+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:24.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.197+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:24.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.197+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:24.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.197+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:24.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.197+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:24.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.197+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:24.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:24 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:24.197+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:57.257747+0000 front 2024-03-18T01:02:57.257632+0000 (oldest deadline 2024-03-18T01:03:23.157472+0000) 2024-03-18T01:03:24.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:24 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:24.187+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:24.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:24 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:24.187+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:24.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:24 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:24.187+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:24.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:24 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:24.187+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:24.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:24 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:24.187+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:24.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:24 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:24.187+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:25.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:25 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:25.165+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:25.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:25 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:25.165+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:25.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:25 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:25.165+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:25.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:25 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:25.165+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:25.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:25 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:25.165+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:25.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:25 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:25.165+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:25.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:25 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:25.165+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:57.257747+0000 front 2024-03-18T01:02:57.257632+0000 (oldest deadline 2024-03-18T01:03:23.157472+0000) 2024-03-18T01:03:25.582 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:25 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:25.231+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:25.582 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:25 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:25.231+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:25.582 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:25 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:25.231+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:25.582 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:25 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:25.231+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:25.582 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:25 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:25.231+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:25.582 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:25 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:25.231+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:26.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:26 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:26.179+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:26.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:26 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:26.179+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:26.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:26 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:26.179+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:26.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:26 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:26.179+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:26.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:26 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:26.179+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:26.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:26 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:26.179+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:26.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:26 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:26.179+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:57.257747+0000 front 2024-03-18T01:02:57.257632+0000 (oldest deadline 2024-03-18T01:03:23.157472+0000) 2024-03-18T01:03:26.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:26 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:26.215+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:26.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:26 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:26.215+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:26.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:26 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:26.215+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:26.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:26 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:26.215+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:26.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:26 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:26.215+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:26.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:26 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:26.215+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:27.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:27 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:27.256+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:27.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:27 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:27.256+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:27.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:27 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:27.256+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:27.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:27 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:27.256+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:27.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:27 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:27.256+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:27.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:27 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:27.256+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:27.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:27 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:27.199+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:27.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:27 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:27.199+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:27.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:27 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:27.199+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:27.696 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:27 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:27.199+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:27.697 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:27 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:27.199+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:27.697 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:27 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:27.199+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:27.697 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:27 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:27.199+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:57.257747+0000 front 2024-03-18T01:02:57.257632+0000 (oldest deadline 2024-03-18T01:03:23.157472+0000) 2024-03-18T01:03:28.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:28 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:28.193+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.950314+0000 front 2024-03-18T01:02:05.950225+0000 (oldest deadline 2024-03-18T01:02:30.050119+0000) 2024-03-18T01:03:28.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:28 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:28.193+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:14.751222+0000 front 2024-03-18T01:02:14.751246+0000 (oldest deadline 2024-03-18T01:02:37.051481+0000) 2024-03-18T01:03:28.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:28 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:28.193+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.752739+0000 front 2024-03-18T01:02:21.752781+0000 (oldest deadline 2024-03-18T01:02:47.052610+0000) 2024-03-18T01:03:28.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:28 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:28.193+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6814 osd.3 since back 2024-03-18T01:02:31.554524+0000 front 2024-03-18T01:02:31.554628+0000 (oldest deadline 2024-03-18T01:02:56.254475+0000) 2024-03-18T01:03:28.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:28 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:28.193+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:37.355215+0000 front 2024-03-18T01:02:37.355229+0000 (oldest deadline 2024-03-18T01:03:03.255251+0000) 2024-03-18T01:03:28.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:28 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:28.193+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:45.556372+0000 front 2024-03-18T01:02:45.556352+0000 (oldest deadline 2024-03-18T01:03:14.956581+0000) 2024-03-18T01:03:28.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:28 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10[61137]: 2024-03-18T01:03:28.193+0000 7f3d80db4700 -1 osd.10 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:57.257747+0000 front 2024-03-18T01:02:57.257632+0000 (oldest deadline 2024-03-18T01:03:23.157472+0000) 2024-03-18T01:03:28.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:28 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:28.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:28.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:28 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:28.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:28.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:28 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:28.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:28.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:28 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:28.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:28.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:28 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:28.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:28.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:28 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:28.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:28.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:28 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:28.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-18T01:03:04.034617+0000 front 2024-03-18T01:03:04.034629+0000 (oldest deadline 2024-03-18T01:03:27.434867+0000) 2024-03-18T01:03:29.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:29 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:29.296+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:29.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:29 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:29.296+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:29.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:29 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:29.296+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:29.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:29 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:29.296+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:29.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:29 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:29.296+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:29.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:29 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:29.296+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:29.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:29 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:29.296+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-18T01:03:04.034617+0000 front 2024-03-18T01:03:04.034629+0000 (oldest deadline 2024-03-18T01:03:27.434867+0000) 2024-03-18T01:03:29.789 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:29 smithi106 podman[72636]: 2024-03-18 01:03:29.500821681 +0000 UTC m=+5.477753078 container died 09a4e019cad4d82415a156a6c43086a0366a7407bced51173033d73de3f3a1e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T01:03:30.139 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:29 smithi106 podman[72636]: 2024-03-18 01:03:29.897205944 +0000 UTC m=+5.874137351 container cleanup 09a4e019cad4d82415a156a6c43086a0366a7407bced51173033d73de3f3a1e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True) 2024-03-18T01:03:30.139 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:29 smithi106 bash[72636]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10 2024-03-18T01:03:30.446 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:30 smithi106 podman[72651]: 2024-03-18 01:03:30.139885492 +0000 UTC m=+0.635632788 container remove 09a4e019cad4d82415a156a6c43086a0366a7407bced51173033d73de3f3a1e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-03-18T01:03:30.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:30 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:30.260+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:30.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:30 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:30.260+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:30.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:30 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:30.260+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:30.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:30 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:30.260+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:30.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:30 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:30.260+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:30.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:30 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:30.260+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:30.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:30 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:30.260+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-18T01:03:04.034617+0000 front 2024-03-18T01:03:04.034629+0000 (oldest deadline 2024-03-18T01:03:27.434867+0000) 2024-03-18T01:03:30.768 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:30 smithi106 podman[72747]: 2024-03-18 01:03:30.367025622 +0000 UTC m=+0.017947216 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:03:30.768 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:30 smithi106 podman[72747]: 2024-03-18 01:03:30.468044453 +0000 UTC m=+0.118966031 container create 50c2043bfac7959796b17fb43e7ac6ee46880719bb4c453fd600b5b5e385cc9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T01:03:30.768 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:30 smithi106 podman[72747]: 2024-03-18 01:03:30.685472055 +0000 UTC m=+0.336393633 container init 50c2043bfac7959796b17fb43e7ac6ee46880719bb4c453fd600b5b5e385cc9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-18T01:03:30.768 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:30 smithi106 podman[72747]: 2024-03-18 01:03:30.688572038 +0000 UTC m=+0.339493617 container start 50c2043bfac7959796b17fb43e7ac6ee46880719bb4c453fd600b5b5e385cc9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True) 2024-03-18T01:03:30.768 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:30 smithi106 podman[72747]: 2024-03-18 01:03:30.768623419 +0000 UTC m=+0.419545017 container attach 50c2043bfac7959796b17fb43e7ac6ee46880719bb4c453fd600b5b5e385cc9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-18T01:03:31.196 INFO:journalctl@ceph.osd.10.smithi106.stdout:Mar 18 01:03:30 smithi106 podman[72747]: 2024-03-18 01:03:30.975743883 +0000 UTC m=+0.626665460 container died 50c2043bfac7959796b17fb43e7ac6ee46880719bb4c453fd600b5b5e385cc9a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-10-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:03:31.578 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.10.service' 2024-03-18T01:03:31.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:31.622 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-18T01:03:31.622 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-18T01:03:31.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.11 2024-03-18T01:03:31.654 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.263+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:31.654 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.263+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:31.654 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.263+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:31.654 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.263+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:31.654 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.263+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:31.654 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.263+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:31.654 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.263+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-18T01:03:04.034617+0000 front 2024-03-18T01:03:04.034629+0000 (oldest deadline 2024-03-18T01:03:27.434867+0000) 2024-03-18T01:03:31.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 systemd[1]: Stopping Ceph osd.11 for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:32.279 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.993+0000 7f65a04e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:32.279 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.993+0000 7f65a04e0700 -1 osd.11 91 *** Got signal Terminated *** 2024-03-18T01:03:32.279 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:31 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:31.993+0000 7f65a04e0700 -1 osd.11 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T01:03:32.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:32 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:32.279+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:32.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:32 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:32.279+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:32.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:32 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:32.279+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:32.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:32 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:32.279+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:32.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:32 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:32.279+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:32.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:32 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:32.279+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:32.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:32 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:32.279+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-18T01:03:04.034617+0000 front 2024-03-18T01:03:04.034629+0000 (oldest deadline 2024-03-18T01:03:27.434867+0000) 2024-03-18T01:03:33.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:33 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:33.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:33.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:33 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:33.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:33.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:33 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:33.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:33.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:33 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:33.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:33.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:33 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:33.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:33.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:33 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:33.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:33.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:33 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:33.254+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-18T01:03:04.034617+0000 front 2024-03-18T01:03:04.034629+0000 (oldest deadline 2024-03-18T01:03:27.434867+0000) 2024-03-18T01:03:34.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:34.251+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:34.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:34.251+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:34.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:34.251+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:34.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:34.251+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:34.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:34.251+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:34.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:34.251+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:34.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:34.251+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-18T01:03:04.034617+0000 front 2024-03-18T01:03:04.034629+0000 (oldest deadline 2024-03-18T01:03:27.434867+0000) 2024-03-18T01:03:34.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:34 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:34.251+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6820 osd.8 since back 2024-03-18T01:03:09.735910+0000 front 2024-03-18T01:03:09.735938+0000 (oldest deadline 2024-03-18T01:03:33.836075+0000) 2024-03-18T01:03:35.637 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:35.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:35.637 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:35.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:35.637 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:35.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:35.637 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:35.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:35.637 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:35.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:35.637 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:35.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:35.637 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:35.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-18T01:03:04.034617+0000 front 2024-03-18T01:03:04.034629+0000 (oldest deadline 2024-03-18T01:03:27.434867+0000) 2024-03-18T01:03:35.637 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:35 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:35.281+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6820 osd.8 since back 2024-03-18T01:03:09.735910+0000 front 2024-03-18T01:03:09.735938+0000 (oldest deadline 2024-03-18T01:03:33.836075+0000) 2024-03-18T01:03:36.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:36.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6806 osd.0 since back 2024-03-18T01:02:05.622588+0000 front 2024-03-18T01:02:05.622650+0000 (oldest deadline 2024-03-18T01:02:29.122756+0000) 2024-03-18T01:03:36.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:36.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6804 osd.1 since back 2024-03-18T01:02:16.525721+0000 front 2024-03-18T01:02:16.525617+0000 (oldest deadline 2024-03-18T01:02:41.825637+0000) 2024-03-18T01:03:36.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:36.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6804 osd.2 since back 2024-03-18T01:02:21.826075+0000 front 2024-03-18T01:02:21.826122+0000 (oldest deadline 2024-03-18T01:02:47.726204+0000) 2024-03-18T01:03:36.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:36.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6812 osd.4 since back 2024-03-18T01:02:39.729937+0000 front 2024-03-18T01:02:39.729955+0000 (oldest deadline 2024-03-18T01:03:03.229941+0000) 2024-03-18T01:03:36.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:36.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6812 osd.5 since back 2024-03-18T01:02:47.831361+0000 front 2024-03-18T01:02:47.831383+0000 (oldest deadline 2024-03-18T01:03:10.731559+0000) 2024-03-18T01:03:36.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:36.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.49:6822 osd.6 since back 2024-03-18T01:02:54.833042+0000 front 2024-03-18T01:02:54.832901+0000 (oldest deadline 2024-03-18T01:03:18.332721+0000) 2024-03-18T01:03:36.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:36.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.106:6820 osd.7 since back 2024-03-18T01:03:04.034617+0000 front 2024-03-18T01:03:04.034629+0000 (oldest deadline 2024-03-18T01:03:27.434867+0000) 2024-03-18T01:03:36.681 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:36 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11[62046]: 2024-03-18T01:03:36.316+0000 7f659cad4700 -1 osd.11 91 heartbeat_check: no reply from 172.21.15.107:6820 osd.8 since back 2024-03-18T01:03:09.735910+0000 front 2024-03-18T01:03:09.735938+0000 (oldest deadline 2024-03-18T01:03:33.836075+0000) 2024-03-18T01:03:37.680 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:37 smithi107 podman[72867]: 2024-03-18 01:03:37.398029632 +0000 UTC m=+5.581957282 container died 05804a5b1443b984a8d0dcefe7d54df97e2803eababd25afc9f3a75a464e877e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T01:03:38.180 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:37 smithi107 podman[72867]: 2024-03-18 01:03:37.70156375 +0000 UTC m=+5.885491416 container cleanup 05804a5b1443b984a8d0dcefe7d54df97e2803eababd25afc9f3a75a464e877e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-18T01:03:38.180 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:37 smithi107 bash[72867]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11 2024-03-18T01:03:38.180 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:37 smithi107 podman[72884]: 2024-03-18 01:03:37.910868586 +0000 UTC m=+0.516639684 container remove 05804a5b1443b984a8d0dcefe7d54df97e2803eababd25afc9f3a75a464e877e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-03-18T01:03:38.505 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:38 smithi107 podman[72978]: 2024-03-18 01:03:38.138848166 +0000 UTC m=+0.017440095 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T01:03:38.505 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:38 smithi107 podman[72978]: 2024-03-18 01:03:38.236519976 +0000 UTC m=+0.115111902 container create 44456d1a557b88c51dc16cfe9edbeed2cf5acaedc9a6b9ffc45521cd448667ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-18T01:03:38.505 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:38 smithi107 podman[72978]: 2024-03-18 01:03:38.422130864 +0000 UTC m=+0.300722803 container init 44456d1a557b88c51dc16cfe9edbeed2cf5acaedc9a6b9ffc45521cd448667ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-18T01:03:38.505 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:38 smithi107 podman[72978]: 2024-03-18 01:03:38.425186186 +0000 UTC m=+0.303778116 container start 44456d1a557b88c51dc16cfe9edbeed2cf5acaedc9a6b9ffc45521cd448667ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-18T01:03:38.505 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:38 smithi107 podman[72978]: 2024-03-18 01:03:38.505196274 +0000 UTC m=+0.383788222 container attach 44456d1a557b88c51dc16cfe9edbeed2cf5acaedc9a6b9ffc45521cd448667ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:03:38.930 INFO:journalctl@ceph.osd.11.smithi107.stdout:Mar 18 01:03:38 smithi107 podman[72978]: 2024-03-18 01:03:38.714581046 +0000 UTC m=+0.593172971 container died 44456d1a557b88c51dc16cfe9edbeed2cf5acaedc9a6b9ffc45521cd448667ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-18T01:03:39.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@osd.11.service' 2024-03-18T01:03:39.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:39.396 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-18T01:03:39.396 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-18T01:03:39.396 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.a 2024-03-18T01:03:39.720 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:03:39 smithi049 systemd[1]: Stopping Ceph mds.a for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:40.125 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:03:39 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a[73529]: 2024-03-18T01:03:39.719+0000 7f4ef9783700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:40.125 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:03:39 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a[73529]: 2024-03-18T01:03:39.719+0000 7f4ef9783700 -1 mds.a *** got signal Terminated *** 2024-03-18T01:03:44.004 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:03:43 smithi049 podman[100631]: 2024-03-18 01:03:43.731013092 +0000 UTC m=+4.128460605 container died 240276f0c2668c71d6efce021b307f213b95d1c1dc0d312566d8de1ec9839b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T01:03:44.375 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:03:44 smithi049 podman[100631]: 2024-03-18 01:03:44.112013548 +0000 UTC m=+4.509461062 container cleanup 240276f0c2668c71d6efce021b307f213b95d1c1dc0d312566d8de1ec9839b0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T01:03:44.375 INFO:journalctl@ceph.mds.a.smithi049.stdout:Mar 18 01:03:44 smithi049 bash[100631]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-a 2024-03-18T01:03:44.536 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.a.service' 2024-03-18T01:03:44.579 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:44.579 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-18T01:03:44.579 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-18T01:03:44.579 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.d 2024-03-18T01:03:44.898 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 01:03:44 smithi049 systemd[1]: Stopping Ceph mds.d for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:45.268 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 01:03:44 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-d[72579]: 2024-03-18T01:03:44.896+0000 7f875f597700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:45.268 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 01:03:44 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-d[72579]: 2024-03-18T01:03:44.896+0000 7f875f597700 -1 mds.d *** got signal Terminated *** 2024-03-18T01:03:49.188 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 01:03:48 smithi049 podman[100751]: 2024-03-18 01:03:48.911487739 +0000 UTC m=+4.102802409 container died 60f6373da18496ef6fa041bf969cc8005ec08a603062be6b29dd59f573463980 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-d, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-18T01:03:49.189 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 01:03:49 smithi049 podman[100751]: 2024-03-18 01:03:49.189523478 +0000 UTC m=+4.380838156 container cleanup 60f6373da18496ef6fa041bf969cc8005ec08a603062be6b29dd59f573463980 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:03:49.481 INFO:journalctl@ceph.mds.d.smithi049.stdout:Mar 18 01:03:49 smithi049 bash[100751]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-d 2024-03-18T01:03:49.618 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.d.service' 2024-03-18T01:03:49.658 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:49.658 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-18T01:03:49.658 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-18T01:03:49.658 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.g 2024-03-18T01:03:50.124 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:03:49 smithi049 systemd[1]: Stopping Ceph mds.g for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:50.125 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:03:49 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g[73175]: 2024-03-18T01:03:49.983+0000 7ff9e892d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:50.125 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:03:49 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g[73175]: 2024-03-18T01:03:49.983+0000 7ff9e892d700 -1 mds.g *** got signal Terminated *** 2024-03-18T01:03:52.519 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:03:52 smithi049 podman[100874]: 2024-03-18 01:03:52.25795382 +0000 UTC m=+2.374235220 container died 9ebcde9fa222f9a71bdf241662530e2254ab3e6238d4c2318056d3f134b18e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-18T01:03:52.875 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:03:52 smithi049 podman[100874]: 2024-03-18 01:03:52.618173839 +0000 UTC m=+2.734455245 container cleanup 9ebcde9fa222f9a71bdf241662530e2254ab3e6238d4c2318056d3f134b18e5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T01:03:52.875 INFO:journalctl@ceph.mds.g.smithi049.stdout:Mar 18 01:03:52 smithi049 bash[100874]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-g 2024-03-18T01:03:53.055 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.g.service' 2024-03-18T01:03:53.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:53.093 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-18T01:03:53.093 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-18T01:03:53.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.j 2024-03-18T01:03:53.430 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 01:03:53 smithi049 systemd[1]: Stopping Ceph mds.j for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:53.875 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 01:03:53 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-j[72224]: 2024-03-18T01:03:53.430+0000 7f97e5ae9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:53.875 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 01:03:53 smithi049 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-j[72224]: 2024-03-18T01:03:53.430+0000 7f97e5ae9700 -1 mds.j *** got signal Terminated *** 2024-03-18T01:03:57.722 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 01:03:57 smithi049 podman[100994]: 2024-03-18 01:03:57.439978639 +0000 UTC m=+4.116943361 container died ae2cab6fbc2187453bcf57976df56d53555b8f0aa2ea996c86c0a6d2fd5c9efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-j, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T01:03:58.105 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 01:03:57 smithi049 podman[100994]: 2024-03-18 01:03:57.813474248 +0000 UTC m=+4.490438972 container cleanup ae2cab6fbc2187453bcf57976df56d53555b8f0aa2ea996c86c0a6d2fd5c9efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-j, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T01:03:58.105 INFO:journalctl@ceph.mds.j.smithi049.stdout:Mar 18 01:03:57 smithi049 bash[100994]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-j 2024-03-18T01:03:58.251 DEBUG:teuthology.orchestra.run.smithi049:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.j.service' 2024-03-18T01:03:58.289 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:03:58.289 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-18T01:03:58.289 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-18T01:03:58.289 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.b 2024-03-18T01:03:58.606 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:03:58 smithi106 systemd[1]: Stopping Ceph mds.b for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:03:58.945 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:03:58 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b[66856]: 2024-03-18T01:03:58.605+0000 7f4a59034700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:03:58.946 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:03:58 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b[66856]: 2024-03-18T01:03:58.605+0000 7f4a59034700 -1 mds.b *** got signal Terminated *** 2024-03-18T01:04:00.582 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:04:00 smithi106 podman[72867]: 2024-03-18 01:04:00.266250727 +0000 UTC m=+1.782363573 container died e291b8ea36c847a09c42cab7d772a0075c9564652e7c681de8e6f5afc1b4ca2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T01:04:00.898 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:04:00 smithi106 podman[72867]: 2024-03-18 01:04:00.681543615 +0000 UTC m=+2.197656480 container cleanup e291b8ea36c847a09c42cab7d772a0075c9564652e7c681de8e6f5afc1b4ca2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-18T01:04:00.899 INFO:journalctl@ceph.mds.b.smithi106.stdout:Mar 18 01:04:00 smithi106 bash[72867]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-b 2024-03-18T01:04:01.005 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.b.service' 2024-03-18T01:04:01.043 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:04:01.044 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-18T01:04:01.044 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-18T01:04:01.044 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.e 2024-03-18T01:04:01.195 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:04:01 smithi106 systemd[1]: Stopping Ceph mds.e for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:04:01.696 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:04:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e[66233]: 2024-03-18T01:04:01.368+0000 7f74c5ace700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:04:01.696 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:04:01 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e[66233]: 2024-03-18T01:04:01.368+0000 7f74c5ace700 -1 mds.e *** got signal Terminated *** 2024-03-18T01:04:04.527 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:04:04 smithi106 podman[72988]: 2024-03-18 01:04:04.265935885 +0000 UTC m=+2.998372404 container died ebdb9d656f990f1472ad02baf492a5e5037af5efee922ece87fb0b484ac02a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-18T01:04:04.886 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:04:04 smithi106 podman[72988]: 2024-03-18 01:04:04.619038902 +0000 UTC m=+3.351475428 container cleanup ebdb9d656f990f1472ad02baf492a5e5037af5efee922ece87fb0b484ac02a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-18T01:04:04.886 INFO:journalctl@ceph.mds.e.smithi106.stdout:Mar 18 01:04:04 smithi106 bash[72988]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-e 2024-03-18T01:04:05.021 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.e.service' 2024-03-18T01:04:05.058 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:04:05.059 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-18T01:04:05.059 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-18T01:04:05.059 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.h 2024-03-18T01:04:05.446 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:04:05 smithi106 systemd[1]: Stopping Ceph mds.h for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:04:05.446 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:04:05 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h[66542]: 2024-03-18T01:04:05.381+0000 7fb4a0921700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:04:05.446 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:04:05 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h[66542]: 2024-03-18T01:04:05.381+0000 7fb4a0921700 -1 mds.h *** got signal Terminated *** 2024-03-18T01:04:08.673 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:04:08 smithi106 podman[73122]: 2024-03-18 01:04:08.274262746 +0000 UTC m=+3.000909742 container died c065f4492ff73440b01601298facb29169b9eeeefba6bede87ac2f97f4a463cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-18T01:04:08.945 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:04:08 smithi106 podman[73122]: 2024-03-18 01:04:08.772703207 +0000 UTC m=+3.499350196 container cleanup c065f4492ff73440b01601298facb29169b9eeeefba6bede87ac2f97f4a463cc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:04:08.946 INFO:journalctl@ceph.mds.h.smithi106.stdout:Mar 18 01:04:08 smithi106 bash[73122]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-h 2024-03-18T01:04:09.152 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.h.service' 2024-03-18T01:04:09.187 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:04:09.187 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-18T01:04:09.187 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-18T01:04:09.187 DEBUG:teuthology.orchestra.run.smithi106:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.k 2024-03-18T01:04:09.445 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:04:09 smithi106 systemd[1]: Stopping Ceph mds.k for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:04:09.946 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:04:09 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k[65907]: 2024-03-18T01:04:09.534+0000 7f01810fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:04:09.946 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:04:09 smithi106 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k[65907]: 2024-03-18T01:04:09.534+0000 7f01810fd700 -1 mds.k *** got signal Terminated *** 2024-03-18T01:04:12.632 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:04:12 smithi106 podman[73247]: 2024-03-18 01:04:12.353260454 +0000 UTC m=+2.950110095 container died 6a9c5dec50936d490c57ac3a4897fee359f753998521d0bc11577d98f2e069d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:04:12.945 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:04:12 smithi106 podman[73247]: 2024-03-18 01:04:12.709608202 +0000 UTC m=+3.306457833 container cleanup 6a9c5dec50936d490c57ac3a4897fee359f753998521d0bc11577d98f2e069d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T01:04:12.946 INFO:journalctl@ceph.mds.k.smithi106.stdout:Mar 18 01:04:12 smithi106 bash[73247]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-k 2024-03-18T01:04:13.142 DEBUG:teuthology.orchestra.run.smithi106:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.k.service' 2024-03-18T01:04:13.177 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:04:13.177 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-18T01:04:13.177 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-18T01:04:13.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.c 2024-03-18T01:04:13.680 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:13 smithi107 systemd[1]: Stopping Ceph mds.c for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:04:13.680 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c[67104]: 2024-03-18T01:04:13.446+0000 7f0d9735c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:04:13.680 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:13 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c[67104]: 2024-03-18T01:04:13.446+0000 7f0d9735c700 -1 mds.c *** got signal Terminated *** 2024-03-18T01:04:16.597 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:16 smithi107 podman[73083]: 2024-03-18 01:04:16.26488078 +0000 UTC m=+2.897220650 container died b6e77d53936fa32630a9731807ce3d70d9cfed233643fdc192947fc50639e8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-18T01:04:16.888 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:16 smithi107 podman[73083]: 2024-03-18 01:04:16.688305018 +0000 UTC m=+3.320644857 container cleanup b6e77d53936fa32630a9731807ce3d70d9cfed233643fdc192947fc50639e8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-18T01:04:16.888 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:16 smithi107 bash[73083]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c 2024-03-18T01:04:17.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.c.service' 2024-03-18T01:04:17.138 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:16 smithi107 podman[73110]: 2024-03-18 01:04:16.889285761 +0000 UTC m=+0.543781666 container remove b6e77d53936fa32630a9731807ce3d70d9cfed233643fdc192947fc50639e8be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-c, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T01:04:17.139 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:17 smithi107 systemd[1]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.c.service: Deactivated successfully. 2024-03-18T01:04:17.139 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:17 smithi107 systemd[1]: Stopped Ceph mds.c for ec390cde-e4bf-11ee-95c9-87774f69a715. 2024-03-18T01:04:17.139 INFO:journalctl@ceph.mds.c.smithi107.stdout:Mar 18 01:04:17 smithi107 systemd[1]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.c.service: Consumed 4min 24.065s CPU time. 2024-03-18T01:04:17.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:04:17.153 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-18T01:04:17.153 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-18T01:04:17.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.f 2024-03-18T01:04:17.430 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:04:17 smithi107 systemd[1]: Stopping Ceph mds.f for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:04:17.930 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:04:17 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f[66788]: 2024-03-18T01:04:17.483+0000 7f7d68df4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:04:17.931 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:04:17 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f[66788]: 2024-03-18T01:04:17.483+0000 7f7d68df4700 -1 mds.f *** got signal Terminated *** 2024-03-18T01:04:20.557 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:04:20 smithi107 podman[73219]: 2024-03-18 01:04:20.269317515 +0000 UTC m=+2.891031609 container died 72008fe08c4a5764b32c0df63a3b8f0f5099f1cb9c5371a0fa9ba90c22145ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:04:20.558 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:04:20 smithi107 podman[73219]: 2024-03-18 01:04:20.558437809 +0000 UTC m=+3.180151920 container cleanup 72008fe08c4a5764b32c0df63a3b8f0f5099f1cb9c5371a0fa9ba90c22145ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T01:04:20.845 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:04:20 smithi107 bash[73219]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f 2024-03-18T01:04:20.845 INFO:journalctl@ceph.mds.f.smithi107.stdout:Mar 18 01:04:20 smithi107 podman[73233]: 2024-03-18 01:04:20.751032744 +0000 UTC m=+0.478286429 container remove 72008fe08c4a5764b32c0df63a3b8f0f5099f1cb9c5371a0fa9ba90c22145ce5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-f, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-18T01:04:20.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.f.service' 2024-03-18T01:04:20.919 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:04:20.919 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-18T01:04:20.919 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-18T01:04:20.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.i 2024-03-18T01:04:21.180 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:04:21 smithi107 systemd[1]: Stopping Ceph mds.i for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:04:21.680 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:04:21 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i[67415]: 2024-03-18T01:04:21.260+0000 7f366f1f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:04:21.681 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:04:21 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i[67415]: 2024-03-18T01:04:21.260+0000 7f366f1f8700 -1 mds.i *** got signal Terminated *** 2024-03-18T01:04:24.536 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:04:24 smithi107 podman[73338]: 2024-03-18 01:04:24.261168324 +0000 UTC m=+3.116100017 container died eea5ffc66de3490d39701764637c8d68956368162d6f33148babe15743b1bfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-18T01:04:24.903 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:04:24 smithi107 podman[73338]: 2024-03-18 01:04:24.627881351 +0000 UTC m=+3.482813044 container cleanup eea5ffc66de3490d39701764637c8d68956368162d6f33148babe15743b1bfda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T01:04:24.903 INFO:journalctl@ceph.mds.i.smithi107.stdout:Mar 18 01:04:24 smithi107 bash[73338]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-i 2024-03-18T01:04:25.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.i.service' 2024-03-18T01:04:25.100 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:04:25.100 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-18T01:04:25.101 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-18T01:04:25.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.l 2024-03-18T01:04:25.423 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:04:25 smithi107 systemd[1]: Stopping Ceph mds.l for ec390cde-e4bf-11ee-95c9-87774f69a715... 2024-03-18T01:04:25.680 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:04:25 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l[66476]: 2024-03-18T01:04:25.422+0000 7ff870ec5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T01:04:25.680 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:04:25 smithi107 ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l[66476]: 2024-03-18T01:04:25.422+0000 7ff870ec5700 -1 mds.l *** got signal Terminated *** 2024-03-18T01:04:28.523 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:04:28 smithi107 podman[73457]: 2024-03-18 01:04:28.254043027 +0000 UTC m=+2.937334799 container died 88fa35199912406a4d8442a03c45e49967cac5650384b31bb71e3b5dd84ac945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:04:28.786 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:04:28 smithi107 podman[73457]: 2024-03-18 01:04:28.623212544 +0000 UTC m=+3.306504321 container cleanup 88fa35199912406a4d8442a03c45e49967cac5650384b31bb71e3b5dd84ac945 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T01:04:28.787 INFO:journalctl@ceph.mds.l.smithi107.stdout:Mar 18 01:04:28 smithi107 bash[73457]: ceph-ec390cde-e4bf-11ee-95c9-87774f69a715-mds-l 2024-03-18T01:04:28.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec390cde-e4bf-11ee-95c9-87774f69a715@mds.l.service' 2024-03-18T01:04:29.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T01:04:29.017 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-18T01:04:29.017 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T01:04:29.222 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T01:04:32.427 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T01:04:32.617 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T01:04:35.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T01:04:35.373 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T01:04:37.815 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:04:37.855 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:04:37.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T01:04:37.919 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T01:04:37.921 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/remote/smithi049/crash 2024-03-18T01:04:37.922 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/crash -- . 2024-03-18T01:04:37.957 INFO:teuthology.orchestra.run.smithi049.stderr:tar: /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T01:04:37.957 INFO:teuthology.orchestra.run.smithi049.stderr:tar: Error is not recoverable: exiting now 2024-03-18T01:04:37.959 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/remote/smithi106/crash 2024-03-18T01:04:37.960 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/crash -- . 2024-03-18T01:04:37.999 INFO:teuthology.orchestra.run.smithi106.stderr:tar: /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T01:04:38.000 INFO:teuthology.orchestra.run.smithi106.stderr:tar: Error is not recoverable: exiting now 2024-03-18T01:04:38.002 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/remote/smithi107/crash 2024-03-18T01:04:38.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/crash -- . 2024-03-18T01:04:38.069 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T01:04:38.070 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2024-03-18T01:04:38.071 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T01:04:38.071 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T01:04:38.163 INFO:teuthology.orchestra.run.smithi049.stdout:2024-03-18T00:42:01.490162+0000 mon.a (mon.0) 212 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] is down (out of quorum) 2024-03-18T01:04:38.164 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-18T01:04:38.164 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[SEC\]' /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T01:04:38.205 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]' /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T01:04:38.287 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[WRN\]' /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T01:04:38.367 INFO:teuthology.orchestra.run.smithi049.stdout:2024-03-18T00:42:01.490162+0000 mon.a (mon.0) 212 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] is down (out of quorum) 2024-03-18T01:04:38.367 INFO:tasks.cephadm:Compressing logs... 2024-03-18T01:04:38.367 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T01:04:38.409 DEBUG:teuthology.orchestra.run.smithi106:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T01:04:38.413 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T01:04:38.450 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T01:04:38.452 INFO:teuthology.orchestra.run.smithi049.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-03-18T01:04:38.452 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-03-18T01:04:38.459 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T01:04:38.459 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-18T01:04:38.462 INFO:teuthology.orchestra.run.smithi107.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T01:04:38.462 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mgr.z.log: -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T01:04:38.463 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-18T01:04:38.465 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-18T01:04:38.465 INFO:teuthology.orchestra.run.smithi049.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-18T01:04:38.465 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T01:04:38.465 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-18T01:04:38.465 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-18T01:04:38.466 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mon.c.log: 92.8% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-18T01:04:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-18T01:04:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-18T01:04:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-18T01:04:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T01:04:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log 2024-03-18T01:04:38.468 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T01:04:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-18T01:04:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log: /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-18T01:04:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-18T01:04:38.469 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.d.log: /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.a.log: 86.7% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T01:04:38.470 INFO:teuthology.orchestra.run.smithi106.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T01:04:38.470 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T01:04:38.470 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-18T01:04:38.471 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T01:04:38.472 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-18T01:04:38.472 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-18T01:04:38.472 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-18T01:04:38.472 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-18T01:04:38.473 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-18T01:04:38.474 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-18T01:04:38.474 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-18T01:04:38.474 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.7.log: --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-18T01:04:38.474 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T01:04:38.474 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log 2024-03-18T01:04:38.474 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T01:04:38.475 INFO:teuthology.orchestra.run.smithi106.stderr:gzip -5 --verbose -- /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-18T01:04:38.478 INFO:teuthology.orchestra.run.smithi106.stderr:/var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log: /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mon.b.log: 92.7% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-18T01:04:38.485 INFO:teuthology.orchestra.run.smithi106.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T01:04:38.487 INFO:teuthology.orchestra.run.smithi107.stderr: 82.4% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T01:04:38.489 INFO:teuthology.orchestra.run.smithi107.stderr: 90.5% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T01:04:38.495 INFO:teuthology.orchestra.run.smithi107.stderr: 86.6% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T01:04:38.511 INFO:teuthology.orchestra.run.smithi107.stderr: 92.6% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-18T01:04:38.513 INFO:teuthology.orchestra.run.smithi106.stderr: 94.5% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-18T01:04:38.515 INFO:teuthology.orchestra.run.smithi107.stderr: 92.7% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-18T01:04:38.518 INFO:teuthology.orchestra.run.smithi049.stderr: 82.8% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T01:04:38.522 INFO:teuthology.orchestra.run.smithi106.stderr: 92.7% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-18T01:04:38.554 INFO:teuthology.orchestra.run.smithi049.stderr: 94.4% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-18T01:04:38.556 INFO:teuthology.orchestra.run.smithi106.stderr: 82.5% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T01:04:38.571 INFO:teuthology.orchestra.run.smithi106.stderr: 91.0% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T01:04:38.595 INFO:teuthology.orchestra.run.smithi049.stderr: 94.5% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-18T01:04:38.605 INFO:teuthology.orchestra.run.smithi107.stderr: 91.0% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T01:04:38.618 INFO:teuthology.orchestra.run.smithi049.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T01:04:38.668 INFO:teuthology.orchestra.run.smithi049.stderr: 91.0% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T01:04:38.698 INFO:teuthology.orchestra.run.smithi106.stderr: 90.4% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T01:04:38.715 INFO:teuthology.orchestra.run.smithi106.stderr: 86.6% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T01:04:38.750 INFO:teuthology.orchestra.run.smithi049.stderr: 94.4% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-18T01:04:38.901 INFO:teuthology.orchestra.run.smithi049.stderr: 90.3% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T01:04:39.051 INFO:teuthology.orchestra.run.smithi107.stderr: 91.8% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-18T01:04:39.054 INFO:teuthology.orchestra.run.smithi107.stderr: 96.9% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-18T01:04:39.080 INFO:teuthology.orchestra.run.smithi049.stderr: 94.6% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-18T01:04:39.508 INFO:teuthology.orchestra.run.smithi049.stderr: 90.2% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-18T01:04:40.654 INFO:teuthology.orchestra.run.smithi106.stderr: 91.8% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-18T01:04:41.941 INFO:teuthology.orchestra.run.smithi049.stderr: 90.3% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-18T01:05:21.514 INFO:teuthology.orchestra.run.smithi107.stderr: 94.0% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-18T01:05:40.300 INFO:teuthology.orchestra.run.smithi107.stderr: 93.3% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-18T01:05:55.741 INFO:teuthology.orchestra.run.smithi107.stderr: 93.3% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-18T01:06:03.536 INFO:teuthology.orchestra.run.smithi107.stderr: 93.2% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-18T01:06:06.738 INFO:teuthology.orchestra.run.smithi107.stderr: 93.1% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-18T01:06:06.739 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-18T01:06:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:real 1m28.312s 2024-03-18T01:06:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:user 3m49.132s 2024-03-18T01:06:06.739 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m8.228s 2024-03-18T01:06:33.571 INFO:teuthology.orchestra.run.smithi049.stderr: 93.3% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-18T01:06:50.448 INFO:teuthology.orchestra.run.smithi049.stderr: 93.1% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-18T01:06:54.134 INFO:teuthology.orchestra.run.smithi049.stderr: 93.2% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-18T01:07:10.214 INFO:teuthology.orchestra.run.smithi049.stderr: 93.2% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-18T01:07:10.214 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-03-18T01:07:10.214 INFO:teuthology.orchestra.run.smithi049.stderr:real 2m31.792s 2024-03-18T01:07:10.215 INFO:teuthology.orchestra.run.smithi049.stderr:user 3m26.396s 2024-03-18T01:07:10.215 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m7.550s 2024-03-18T01:10:12.261 INFO:teuthology.orchestra.run.smithi106.stderr: 97.4% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-18T01:11:54.605 INFO:teuthology.orchestra.run.smithi106.stderr: 93.2% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-18T01:12:27.541 INFO:teuthology.orchestra.run.smithi106.stderr: 93.4% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-18T01:14:00.873 INFO:teuthology.orchestra.run.smithi106.stderr: 93.3% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-18T01:15:04.855 INFO:teuthology.orchestra.run.smithi106.stderr: 93.2% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-18T01:17:12.597 INFO:teuthology.orchestra.run.smithi106.stderr: 93.9% -- replaced with /var/log/ceph/ec390cde-e4bf-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-18T01:17:12.629 INFO:teuthology.orchestra.run.smithi106.stderr: 2024-03-18T01:17:12.629 INFO:teuthology.orchestra.run.smithi106.stderr:real 12m34.191s 2024-03-18T01:17:12.629 INFO:teuthology.orchestra.run.smithi106.stderr:user 6m27.953s 2024-03-18T01:17:12.629 INFO:teuthology.orchestra.run.smithi106.stderr:sys 0m18.303s 2024-03-18T01:17:12.630 INFO:tasks.cephadm:Archiving logs... 2024-03-18T01:17:12.631 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/remote/smithi049/log 2024-03-18T01:17:12.632 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T01:17:43.782 DEBUG:teuthology.misc:Transferring archived files from smithi106:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/remote/smithi106/log 2024-03-18T01:17:43.784 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T01:18:44.089 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/remote/smithi107/log 2024-03-18T01:18:44.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T01:19:19.742 INFO:tasks.cephadm:Removing cluster... 2024-03-18T01:19:19.743 DEBUG:teuthology.orchestra.run.smithi049:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 --force 2024-03-18T01:19:19.948 INFO:teuthology.orchestra.run.smithi049.stdout:Deleting cluster with fsid: ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T01:19:20.444 DEBUG:teuthology.orchestra.run.smithi106:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 --force 2024-03-18T01:19:20.849 INFO:teuthology.orchestra.run.smithi106.stdout:Deleting cluster with fsid: ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T01:19:21.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ec390cde-e4bf-11ee-95c9-87774f69a715 --force 2024-03-18T01:19:21.722 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: ec390cde-e4bf-11ee-95c9-87774f69a715 2024-03-18T01:19:22.235 INFO:tasks.cephadm:Removing cephadm ... 2024-03-18T01:19:22.235 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T01:19:22.255 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T01:19:22.272 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T01:19:22.287 INFO:tasks.cephadm:Teardown complete 2024-03-18T01:19:22.288 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T01:19:22.299 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T01:19:22.299 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T01:19:22.302 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T01:19:22.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T01:19:22.427 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T01:19:22.439 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T01:19:22.440 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T01:19:22.443 DEBUG:teuthology.orchestra.run.smithi106:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T01:19:22.445 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T01:19:22.458 INFO:teuthology.orchestra.run.smithi106.stderr:bash: line 1: ntpq: command not found 2024-03-18T01:19:22.459 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-18T01:19:22.472 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-03-18T01:19:22.476 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T01:19:22.476 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-03-18T01:19:22.476 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +2816us[+2816us] +/- 121ms 2024-03-18T01:19:22.476 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv02.front.sepia.ceph.com 3 7 377 65 -1035us[-1151us] +/- 50ms 2024-03-18T01:19:22.476 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -234us[ -234us] +/- 69ms 2024-03-18T01:19:22.476 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:19:22.510 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T01:19:22.511 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-18T01:19:22.511 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 +2833us[+2833us] +/- 121ms 2024-03-18T01:19:22.511 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 58 -1040us[-1031us] +/- 50ms 2024-03-18T01:19:22.511 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv03.front.sepia.ceph.com 3 6 377 58 -211us[ -203us] +/- 69ms 2024-03-18T01:19:22.511 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:19:22.544 INFO:teuthology.orchestra.run.smithi106.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T01:19:22.544 INFO:teuthology.orchestra.run.smithi106.stdout:=============================================================================== 2024-03-18T01:19:22.544 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +2813us[+2813us] +/- 121ms 2024-03-18T01:19:22.544 INFO:teuthology.orchestra.run.smithi106.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -1069us[-1064us] +/- 50ms 2024-03-18T01:19:22.544 INFO:teuthology.orchestra.run.smithi106.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 -241us[ -241us] +/- 69ms 2024-03-18T01:19:22.544 INFO:teuthology.orchestra.run.smithi106.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T01:19:22.546 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T01:19:22.558 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T01:19:22.559 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T01:19:22.583 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T01:19:22.729 DEBUG:teuthology.orchestra.run.smithi106:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T01:19:22.844 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T01:19:22.962 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T01:19:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:19:23.000 DEBUG:teuthology.orchestra.run.smithi106:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T01:19:23.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:19:23.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T01:19:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:19:23.063 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T01:19:23.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T01:19:23.107 INFO:teuthology.task.internal:Duration was 2802.392936 seconds 2024-03-18T01:19:23.107 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T01:19:23.143 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T01:19:23.144 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T01:19:23.148 DEBUG:teuthology.orchestra.run.smithi106:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T01:19:23.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T01:19:23.210 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T01:19:23.222 INFO:teuthology.orchestra.run.smithi106.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T01:19:23.239 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T01:19:23.541 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T01:19:23.542 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2024-03-18T01:19:23.542 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T01:19:23.564 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi106.front.sepia.ceph.com 2024-03-18T01:19:23.565 DEBUG:teuthology.orchestra.run.smithi106:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T01:19:23.598 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-03-18T01:19:23.599 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T01:19:23.625 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T01:19:23.626 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T01:19:23.628 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T01:19:23.642 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T01:19:23.694 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T01:19:23.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T01:19:23.703 DEBUG:teuthology.orchestra.run.smithi106:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T01:19:23.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T01:19:24.111 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T01:19:24.123 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T01:19:24.123 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T01:19:24.162 DEBUG:teuthology.orchestra.run.smithi106:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T01:19:24.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T01:19:24.227 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T01:19:24.239 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T01:19:24.242 DEBUG:teuthology.orchestra.run.smithi106:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T01:19:24.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T01:19:24.285 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2024-03-18T01:19:24.288 INFO:teuthology.orchestra.run.smithi106.stdout:kernel.core_pattern = core 2024-03-18T01:19:24.296 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-03-18T01:19:24.319 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T01:19:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:19:24.334 DEBUG:teuthology.orchestra.run.smithi106:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T01:19:24.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:19:24.348 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T01:19:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T01:19:24.375 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T01:19:24.388 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T01:19:24.390 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/remote/smithi049 2024-03-18T01:19:24.390 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T01:19:24.443 DEBUG:teuthology.misc:Transferring archived files from smithi106:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/remote/smithi106 2024-03-18T01:19:24.443 DEBUG:teuthology.orchestra.run.smithi106:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T01:19:24.483 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544/remote/smithi107 2024-03-18T01:19:24.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T01:19:24.523 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T01:19:24.524 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T01:19:24.526 DEBUG:teuthology.orchestra.run.smithi106:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T01:19:24.528 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T01:19:24.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T01:19:24.584 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T01:19:24.584 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T01:19:24.625 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T01:19:24.625 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T01:19:24.628 DEBUG:teuthology.orchestra.run.smithi106:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T01:19:24.630 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T01:19:24.640 INFO:teuthology.orchestra.run.smithi049.stdout: 265750 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 01:19 /home/ubuntu/cephtest 2024-03-18T01:19:24.642 INFO:teuthology.orchestra.run.smithi106.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 01:19 /home/ubuntu/cephtest 2024-03-18T01:19:24.644 INFO:teuthology.orchestra.run.smithi107.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 01:19 /home/ubuntu/cephtest 2024-03-18T01:19:24.645 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T01:19:24.658 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T01:19:24.726 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T01:19:24.751 DEBUG:teuthology.nuke:shortname: smithi049 2024-03-18T01:19:24.751 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T01:19:24.775 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:22:17.613197', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHcrMVErWx8rMuitlnLnLqf/0THqc6mmQOVGlNFjE7UMOVSDBdBwKxCn/qjSqBgIJYUAcuF0mbdep72qRPRxmzI='} 2024-03-18T01:19:24.779 DEBUG:teuthology.nuke:shortname: smithi106 2024-03-18T01:19:24.779 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T01:19:24.803 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi106.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:22:17.615069', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO8hZd4SIuuzDj1ZLb0BOj23GeXKiYy6xYrzbQwJi23Zt956HjsWRSC4ue6hMgic2Cif1ehIpryh3kqf8+tsaUk='} 2024-03-18T01:19:24.808 DEBUG:teuthology.nuke:shortname: smithi107 2024-03-18T01:19:24.808 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T01:19:24.832 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-17_23:17:27-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608544', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 00:22:17.611476', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO/cQoSbhCAXJ2Ioh5Z+nED/+TevQTyQp4fE+LLukTLLljCaNNXqFPFhudd8be5/9kFWDRzy8WOUvnEBFhb5hxM='} 2024-03-18T01:19:24.841 INFO:teuthology.orchestra.console.smithi049:Power off 2024-03-18T01:19:24.841 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T01:19:24.859 INFO:teuthology.orchestra.console.smithi106:Power off 2024-03-18T01:19:24.860 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T01:19:24.875 DEBUG:teuthology.orchestra.console.smithi049:power off output: Chassis Power Control: Down/Off 2024-03-18T01:19:24.875 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:24.887 DEBUG:teuthology.orchestra.console.smithi106:power off output: Chassis Power Control: Down/Off 2024-03-18T01:19:24.888 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:24.900 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-03-18T01:19:24.914 INFO:teuthology.orchestra.console.smithi107:Power off 2024-03-18T01:19:24.914 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T01:19:24.930 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-18T01:19:24.945 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-03-18T01:19:24.945 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:24.970 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-18T01:19:28.902 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:28.931 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:28.971 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:29.014 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is on 2024-03-18T01:19:29.041 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-18T01:19:29.081 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-18T01:19:33.018 DEBUG:teuthology.orchestra.console.smithi049:pexpect command: ipmitool -H smithi049.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:33.042 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:33.082 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:33.131 DEBUG:teuthology.orchestra.console.smithi049:check power output: Chassis Power is off 2024-03-18T01:19:33.153 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is on 2024-03-18T01:19:33.191 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-03-18T01:19:33.232 INFO:teuthology.orchestra.console.smithi049:Power off completed 2024-03-18T01:19:33.292 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-03-18T01:19:37.157 DEBUG:teuthology.orchestra.console.smithi106:pexpect command: ipmitool -H smithi106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T01:19:37.266 DEBUG:teuthology.orchestra.console.smithi106:check power output: Chassis Power is off 2024-03-18T01:19:37.367 INFO:teuthology.orchestra.console.smithi106:Power off completed 2024-03-18T01:19:37.472 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} duration: 2802.3929357528687 failure_reason: '"2024-03-18T00:42:01.490162+0000 mon.a (mon.0) 212 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.106:3300/0,v1:172.21.15.106:6789/0] is down (out of quorum)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-18T01:19:37.472 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T01:19:37.566 INFO:teuthology.run:FAIL